Question

A new version of the operating system is being planned for installation into your department’s production environment. What sort of testing would you recommend is done before your department goes live with the new version? Identify each type of testing and describe what is tested. Explain the rationale for performing each type of testing.

[ your answer goes here ]

Would the amount of testing and types of testing to be done be different if you were installing a security patch instead of a new operating system version? Explain in detail your reasons?

[ your answer goes here ]

Would the amount of testing and types of testing to be done be different if you were installing an application patch instead of a new operating system version? Explain in detail your reasons?

[ your answer goes here ]

What sort of preparations would you make for an operating system upgrade on your home PC? Provide some detail.

[ your answer goes here ]

Introduction In the previous sections we discussed various operational issues. With many of these operational activities the issue of testing and verification comes into play. Whenever processes or procedures are changed or added to a system it is important to verify they work as planned. Of even more importance is to ensure the new functionality does not cause any regression in how the system operates. It can be very costly for a client to install a new system that does not work as planned. If a client needs to go off line to rollback changes that is one cost. Having upset customers is another cost. For the manufacturer that shipped the new system that does not work as planned there is the cost to re-engineer it to get it right. Also, the credibility of a manufacturer that ships a poorly tested system can be damaged; sometimes irreparably if the new system resulted in damage that could not be reversed, such as if the untested system resulted in a damaging security breach of data. The idea or excuse that companies don’t have the time to test because of schedule demands is ridiculous in my opinion. It demonstrates that leadership does not understand how to develop systems and the important role testing plays in the “System Development Life Cycle (SDLC)”.

Description Testing is the investigation of functionality or behavior or a system to determine if the system behaves in a predictable manner. The functionality of a system is measured by comparing the current behavior or outputs of the system under test to known good behaviors or outputs. Good or correct system behaviors can be expressed with documentation, stored benchmarks or other “oracles” or repositories of correct results that can be used to judge if the system being tested is producing correct results. There are numerous levels of testing that can be performed. Each level of testing focuses on a particular aspect of a systems behavior. Testing should be performed at various points in the development, deployment or running of a system. Many organizations will deploy incremental testing methodologies to ensure that any change in system behavior is detected as early in the development, deployment or operational process as possible. There are numerous testing methodologies and automated testing tools that support the testing process. Test engineering is a field onto itself that could comprise a semester topic. What I want to focus on in this section is a sampling of testing concepts and how they relate to security. An important point to remember when building a system is that the earlier a problem is found the cheaper it is to fix. Or stated differently, the longer a problem exists in the system the more expensive it is to fix. The following table is based on a NIST study. It provides a cost attached to finding a problem at various times in the SDLC. For example, if a problem is found in the requirements and design phase it has a cost of 1X the cost to repair it. If however, a problem is found after the system has been released; the cost of repairing the problem can be extremely high. Having clearly defined requirements and a comprehensive testing strategy is the most cost effective way to build a system. Software Development Stage Cost Requirements and Design 1X Code and Unit Testing 5X Integration and System Testing 10X Beta Testing 15X Post Release 30X-100X Testing efforts are directly related to the number of security vulnerabilities found in a system. The more comprehensive the testing that is performed the more chance of finding security vulnerabilities early in the SDLC. There are numerous estimates for the cost of a security incident. There is much empirical data showing the cost of a security incident can quickly get into the millions of dollars. Preventing a security incident means finding security vulnerabilities. Finding security vulnerabilities early in the SDLC is desirable. The earlier the vulnerability is found in the SDLC the cheaper it is to fix. Once a system is released the cost of a security vulnerability being discovered in the field quickly mounts. If the vulnerability is exploited into an incident, the cost further increases.

Relationship of Testing to Quality and Security In order to ensure a product is of high quality it must perform as documented and advertised. By utilizing various testing methodologies the behavior of a product can be verified. The importance of testing to security is paramount. The tenets of security are expressed through the attributes of the CIA (Confidentiality, Integrity, Availability) triad. If any of these attributes are not adhered to the security of the system is threatened. Ensuring the tenets of the CIA triad remain intact is done by testing. The tenets of the CIA triad can be threatened in the following two ways:  New functionality is released resulting in new vulnerabilities.  New vulnerabilities are discovered and exploited in existing functionality. Whenever, new functionality is deployed security testing needs to take place to ensure that all attributes of the CIA triad are maintained. New functionality can be introduced into a system by:  new features implemented in hardware, software, firmware;  patches implemented in the system  configuration changes to the system

Testing Methods There are various types of testing methods. The waterfall model for developing systems and testing is a traditional incremental model. In some cases iterative models of development and testing are more appropriate. Or in some cases formal testing models utilizing proofs are desirable. The clarity of the requirements, skill and proximity of the development team and need for prototype systems are a few of the criteria that can dictate using one methodology over another. As part of each programming methodology there are various techniques used to implement the larger methodology. Following are three general types of testing techniques that can be applied to the overall testing strategy.

Black Box Black box testing tests the functionality of a system. It requires no understanding of the internal working or implementation of the system. The tester provides various valid and invalid inputs to the system and based on the documentation they examine the outputs determining if the system is performing correctly.

White Box White box testing requires some understanding of the internal workings and algorithms of the system. Inputs for the test cases are selected based on this internal knowledge and therefore specific paths can be tested. White box testing is a technique that is used in penetration testing which is discussed below.

Gray Box Gray box testing involves some understanding of the internal workings of the system. For example, without knowing detailed algorithm implementation you might have some understanding of the components that are used, the configuration of these components and the default assumptions made in some of the existing black box style tests. While detailed knowledge is not known there are clearly some assumptions that could be made in the testing effort to find possible vulnerabilities in the system.

Levels of Testing There are numerous levels of testing that can be used throughout the SDLC (System Development Life Cycle). A testing level can be thought of as the focus of the testing effort. Following are a few examples of the levels of testing. There are several more to consider.

Functionality Testing This is performed to ensure that functionality works according to accepted expectations. The expectations of correct behavior could be based on: design documentation, user documentation or other published material.

Regression Testing This is performed to ensure that there is no negative change or regression in behavior of systems functionality when: new functionality is deployed, patches are applied or configuration changes are made to the environment. Regression testing is used for ensuring there is no regression in: functionality, performance, system load, security, usability or any other attribute that one wants to test for.

Performance Testing This is performed to test how the system performs under various conditions. The conditions could be based on industry benchmarks, peak loading, average loading or a variety of other conditions.

Usability Testing This is performed to test how the system performs from a usability perspective. Often times a system may perform as designed and documented but it is not usable from the perspective of the end user. Perhaps the required commands are awkward to enter, or the menus and screens difficult to understand. This is sometimes referred to as testing a system to determine if it is “user friendly”.

Test Targets Test targets refer to the part of the system that is being tested. Note that each test target could be subjected to a particular level of testing. For example, unit testing could be executed to test for functionality, regression or performance. These are just a few of the examples of test targets.

Unit Test This is performed in the implementation or development phase of a system. As software routines, hardware or firmware components are being developed they are incrementally tested. Once they are ready for integration into the larger system they are said to be ready for unit test. Unit testing may involve the interfacing and integration of various routines into a program, run time library or some larger construct. Unit testing would test for items such as: a routine accepting the correct number of parameters, are the data types correct; are the outputs of the routine consistent with inputs.

Integration Test This is performed when various parts of a system are ready to come together or interface with one another. The components being integrated are larger functional units compared to the routines or procedures that are tested during unit test.

System Test System test is the testing of a completely integrated system. Ideally, the system has been incrementally built up from unit test to integration test with numerous rounds of tests focusing on the various levels of; functionality, regression, performance and usability.

Security Test The importance of security has resulted in a separate branch of testing called security testing. There are various process and procedures that are employed in a comprehensive security testing strategy. It is important to note that security testing is an ongoing process that needs to occur throughout the life cycle of a system. In other words, when a system is developed and deployed there should be numerous iterations of security testing to ensure and validate that the CIA of the data and the system are intact. Once the system is deployed and in production it is important that security testing continues. It is not enough to declare a system has been tested and certified prior to deployment. The testing effort must be ongoing, since attackers will be diligent in searching for new vulnerabilities that were not discovered during development of the system. Some of the security testing activities that must be done in an ongoing fashion are: vulnerability assessment and penetration testing. Vulnerability Assessment Vulnerability assessment is determining how vulnerable your system is to attack. There are numerous tools that are used to scan your system to determine the level of vulnerability. These tools can scan your system for items such as: open ports, running services, dormant accounts, high traffic applications and other criteria. Some vulnerability assessment tools will scan log files looking for certain activities. These tools then produce reports that can be reviewed to determine if the vulnerabilities need to be dealt with. There are also real time vulnerability assessment tools that monitor a running system for indications of vulnerabilities as well as incidents. These tools can be programmed to provide real time alerts in the form of: text messages, mail messages, phone calls, and process or account shutdowns when vulnerabilities as well as incidents are detected. Many different types of tools can be used to detect vulnerabilities in systems. As security related tools evolve there continues to be a blurring of functionality between tools such as vulnerability scanners (e.g. Nessus, Nmap), malware scanners (e.g., Symantec, Avast), intelligent firewalls and IDS (e.g. Snort) systems.

Penetration Testing Penetration testing or “pen testing” is a type of testing that simulates what an attacker would do to break into your system. The attack being simulated can be an outside attack or an inside attack. Assumptions made for the inside attack may be that the attacker possesses detailed knowledge of the system and environment being attacked. They may have access to user documentation or design material that an outsider attacker may not have. An insider may also have more knowledge of the configuration of the systems under attack that an outsider would not have. Both the insider and outsider can make use of any tool they want. Network protocol analyzers such as Wireshark could be used to network traffic and packets. Vulnerability scanners such as Nessus or Nmap could be used, particularly by an insider to probe for vulnerabilities on various systems. Dumpsters could be searched by an outsider for acquiring important information that missed the shredder, such as a phone directory, or an insider might have ready access to this information which would support social engineering attacks such as phishing to gain passwords or other forms of access. All information is fair game to simulate an attack. Anything that the simulator(s) of the attack can obtain will support keeping the attack real.

Penetration testing is an example of testing that needs to be done in an ongoing, active and real time fashion. All of the vulnerabilities in a system are seldom known. As defects in hardware, software or operational weaknesses are found new vulnerabilities are also found. These vulnerabilities need to found by the “good guys” before an attacker finds them and exploits the vulnerability resulting in a security incident. It is important that penetration testing is done in an ongoing fashion. New vulnerabilities are found as system configurations change, employees change and processes and procedures evolve. The attack surface is constantly changing and growing. With the changing attack surface the attack vectors are growing. It is important to actively monitor your systems for vulnerabilities. Note: You should never run a penetration test on a system without the permission of the owner or responsible party for the system. A penetration test being run without proper permissions is considered a security breach and can have undesirable consequences for the system being tested since vulnerabilities will be exposed and valuable data placed at risk. There can also be undesirable consequences for the person running the penetration test that does not have the proper permissions.

Summary In this section we discussed some aspects of testing and how testing relates to information security. We tied together the importance of testing and how it relates to the tenets of CIA. Testing is a topic that could take

Answer 1

1) In our case, as new Operating system is introduced, so first type of testing should be Regression Testing, which will check whether the introduction of new operting system affected the correct functionality of the system.

Regression Testing This is performed to ensure that there is no negative change or regression in behavior of systems functionality when: new functionality is deployed, patches are applied or configuration changes are made to the environment. Regression testing is used for ensuring there is no regression in: functionality, performance, system load, security, usability or any other attribute that one wants to test for.

After the regression testing, one must go for Performance testing to see whether the implementation of new operating system meets the industry benchmarks, performs as expected and meeting the industry standards.

Performance Testing This is performed to test how the system performs under various conditions. The conditions could be based on industry benchmarks, peak loading, average loading or a variety of other conditions.

After performing Performance Testing and getting positive results, we should go for following testing:

Security Test The importance of security has resulted in a separate branch of testing called security testing. There are various process and procedures that are employed in a comprehensive security testing strategy. It is important to note that security testing is an ongoing process that needs to occur throughout the life cycle of a system. In other words, when a system is developed and deployed there should be numerous iterations of security testing to ensure and validate that the CIA of the data and the system are intact. Once the system is deployed and in production it is important that security testing continues. It is not enough to declare a system has been tested and certified prior to deployment. The testing effort must be ongoing, since attackers will be diligent in searching for new vulnerabilities that were not discovered during development of the system. Some of the security testing activities that must be done in an ongoing fashion are: vulnerability assessment and penetration testing. Vulnerability Assessment Vulnerability assessment is determining how vulnerable your system is to attack. There are numerous tools that are used to scan your system to determine the level of vulnerability. These tools can scan your system for items such as: open ports, running services, dormant accounts, high traffic applications and other criteria. Some vulnerability assessment tools will scan log files looking for certain activities. These tools then produce reports that can be reviewed to determine if the vulnerabilities need to be dealt with. There are also real time vulnerability assessment tools that monitor a running system for indications of vulnerabilities as well as incidents. These tools can be programmed to provide real time alerts in the form of: text messages, mail messages, phone calls, and process or account shutdowns when vulnerabilities as well as incidents are detected. Many different types of tools can be used to detect vulnerabilities in systems. As security related tools evolve there continues to be a blurring of functionality between tools such as vulnerability scanners (e.g. Nessus, Nmap), malware scanners (e.g., Symantec, Avast), intelligent firewalls and IDS (e.g. Snort) systems.

If the system after the changes being employed passes the security test, next test must be Penetration Test to check for the vulnerabilities that attackers can exploit.

Penetration Testing Penetration testing or “pen testing” is a type of testing that simulates what an attacker would do to break into your system. The attack being simulated can be an outside attack or an inside attack. Assumptions made for the inside attack may be that the attacker possesses detailed knowledge of the system and environment being attacked. They may have access to user documentation or design material that an outsider attacker may not have. An insider may also have more knowledge of the configuration of the systems under attack that an outsider would not have. Both the insider and outsider can make use of any tool they want. Network protocol analyzers such as Wireshark could be used to network traffic and packets. Vulnerability scanners such as Nessus or Nmap could be used, particularly by an insider to probe for vulnerabilities on various systems. Dumpsters could be searched by an outsider for acquiring important information that missed the shredder, such as a phone directory, or an insider might have ready access to this information which would support social engineering attacks such as phishing to gain passwords or other forms of access. All information is fair game to simulate an attack. Anything that the simulator(s) of the attack can obtain will support keeping the attack real.

Penetration testing is an example of testing that needs to be done in an ongoing, active and real time fashion. All of the vulnerabilities in a system are seldom known. As defects in hardware, software or operational weaknesses are found new vulnerabilities are also found. These vulnerabilities need to found by the “good guys” before an attacker finds them and exploits the vulnerability resulting in a security incident. It is important that penetration testing is done in an ongoing fashion. New vulnerabilities are found as system configurations change, employees change and processes and procedures evolve. The attack surface is constantly changing and growing. With the changing attack surface the attack vectors are growing. It is important to actively monitor your systems for vulnerabilities. Note: You should never run a penetration test on a system without the permission of the owner or responsible party for the system. A penetration test being run without proper permissions is considered a security breach and can have undesirable consequences for the system being tested since vulnerabilities will be exposed and valuable data placed at risk. There can also be undesirable consequences for the person running the penetration test that does not have the proper permissions.

If we are technically satisfied with the performace of the system after doing all the tests, we must do a Usability test, which checks the usabilty of the system i.e., how much user friendly the system is.

Usability Testing This is performed to test how the system performs from a usability perspective. Often times a system may perform as designed and documented but it is not usable from the perspective of the end user. Perhaps the required commands are awkward to enter, or the menus and screens difficult to understand. This is sometimes referred to as testing a system to determine if it is “user friendly”.

After all the tests, a final test i.e., Full System Test must take place to ensure the accepted expected functionality of the given product or software.

System Test System test is the testing of a completely integrated system. Ideally, the system has been incrementally built up from unit test to integration test with numerous rounds of tests focusing on the various levels of; functionality, regression, performance and usability.

2) In this case, we don't need to perform all of the above testings. As, security patch is a feature, which is introduced in the operating system to remove on of the vulnerabilities present earlier in the operating system. After installing the security patch, one must do the following testings.

i) Security test - This test should be done to check whether the installation of the security patch actually eliminated the security risk or not .

ii) Penetration Testing - This test is done to check whether the introduction of the new security patch in the system hasn't opened the ports for other vulnerabilities and security loopholes.

iii) System Test - A final sytem test is must to check the desirde functionality of the system.

Thank you. If any queries, ask in comment sections.