Software Testing

Software testing is an examination that is supervised to provide teams and stakeholders the details of a software product being tested. It is also considered to be a validation process for software quality evaluation and enhancement.

People look for quality in each simulated artifact. Worldwide competition, off-shoring, outsourcing and rising customer expectations have brought the quality revolution. Building quality software products with strict deadlines is very critical for an organization to be successful in the global economy.

Software Quality

Quality is context dependent and it implies different things to different people. Kitchenham and Pfleeger broadly discuss about the five quality views that include:

Transcendental View

Transcendental view is not definite to software quality and it visualizes software quality as something which should be recognized but not defined.

User View

User view visualizes software quality as readiness for motive. It verifies whether the software product satisfies user needs and expectations or not at the time of software quality evaluation.

Manufacturing View

Manufacturing view perceives the software quality as compliance with the specification. The software product’s quality is defined by the scope to which the software meets the specification.

Product View

In product view, software quality is visualized as intrinsic features of the software product.

Value Based View

In value based view, software quality is perceived in term of the amount a customer is willing to pay for the software.

Role of Testing

Testing plays a key role in assessing a software quality. Testing improves the quality of a software product by repeating tests, finding and fixing bugs and assesses the system’s performance by executing the system level tests before releasing a software product. Basically software quality evaluation is broken into two major categories that include:

Static Analysis

As the name suggests, static analysis is dependent on the analysis of various documents such as requirements specification, design documents, software models and source code. Static analysis does not demand for the code execution under development. This analysis tests the code and finds out the possible outcomes or behaviors that may appear during the execution of the code.

Dynamic Analysis

Dynamic analysis of software product entails the real code execution to exhibit predictable program failures. Code is executed with carefully selected input values and the performance properties of the code are detected.

By executing the above two analyses, maximum number of bugs and defects are identified so that those can be fixed at an early stage of software development lifecycle. Static and dynamic analysis complement each other and should be performed for better effectiveness.

Basic Concepts of Software Testing

There are two frequently used basic concepts for software testing as explained below:

Verification

Verification activities enable users to evaluate software product by discovering if the software product satisfies the stated requirements before the beginning of the development phase. The activities that verifies the precision of development phase are known as verification activities.

Validation

Validation activities enable users to affirm that a software product meets its expected use. These activities primarily focus on the customer’s expectations and confirm whether the expectations of customers are met by the software product. The end product is tested substantially from a customer’s point of view to fulfill overall expectations.

Comparison between Verification and Validation Activities

Objectives of Testing

Stakeholders are the people who are influenced by the system or determine the system’s behavior. Multiple stakeholders have different perspectives for testing process which include:

• Reduce the Risk of Failure

Major software systems or products consists of various flaws that cause the systems to become vulnerable to failures again and again. These failures give rise to the failure rate of the systems. To reduce the risk of frequent failures of the systems, these faults are found and fixed by performing recurrent testing. So, the main objective is to perform tests to bring down the failure rate to an admissible level.

• Reduce the cost of Testing

There are various costs associated with testing which include the cost of:

• Design, Maintenance and execution of test cases
  • Analysis of each test case outcome
  • Documentation of test cases
  • Actual execution of the system and its documentation

So, if the less number of test cases are designed, there will be lesser cost associated with testing. The main objective of performing testing activities is to develop a low-threat software or a system with fewer number test cases.

• Whether the System Works or Not

At the time of implementation of a code or a program unit, the team performs testing to check if the unit of a code is working in standard conditions. In the same manner, the team tests whether the whole system is performing its basic functions or not. Therefore, the main objective of performing the test is to convey that the system works.

• Know When Testing is Completed

This is obvious that achieving hundred percent completion of testing is difficult or not possible. A tester is supposed to create a list of prioritization of test items in the test plan to decide the point at which testing can be stopped. Based on the requirement specification, the tester should complete the positive test items first to ensure that the development requirements are satisfied.

Testing Activities

With the purpose of testing a software or a product, testing activities are performed in a sequence by a test engineer. The major testing activities are explained below:

• Identify the Objective

The first and foremost testing activity is to identify the objective that should be tested and specifies the purpose of designing test cases. Each test case has to be associated with an explicit purpose.

A test case is a pair of input and expected result.

• Set Input

The second testing activity is to set the inputs based on the testing objective. Setting of inputs is dependent on various products such as requirements documents, source code and the expectations.

• Determine the Expected Outcome

The third testing activity is to calculate the expected outcome considering the already set inputs. Primarily, the calculation of the expected outcome is carried out from the high level understanding of the testing objective and the specification being evaluated.  

• Setup the Execution Environment

The fourth testing activity is to establish the right environment for the program execution. In performing this step every assumption external to the program must be satisfied. For example, initializing a local system includes network connection and correct database system availability.

• Execute the Program

The fifth testing activity includes executing the program with the set inputs and checking the real outcomes.

• Analyze the Test Result

The final testing activity is to analyze the end result of the execution. The real outcome is compared with the expected outcome. Based on the comparison, a test judgment is assigned to the program at the end of the analysis step. There are three types of test judgments that include:

• Pass – If the execution creates the expected outcome and the objective of the test case is satisfied, a ‘Pass’ judgment is assigned.
  • Fail – If the execution does not create the expected outcome, a ‘Fail’ Judgment is assigned.
  • Inconclusive – In some cases where neither ‘Pass’ nor ‘Fail’ judgment is possible to assign, an ‘Inconclusive’ judgment is assigned to the program.

Testing Levels

Testing is executed at multiple levels considering the whole system throughout the complete software development life cycle. There are four stages that are applied to a software system or a product before the actual deployment of the system or product. The said four stages are Unit testing, Integration testing, System testing and Acceptance testing. The first three levels of testing are executed by different teams or multiple stakeholders in the organization at different levels of Software Development Life Cycle of a system or product. The final level of testing that is the Acceptance testing is performed by the customers. Generally, the different levels or stages of testing are explained in the form of classical V model as shown below:

• Unit Testing

In unit testing, individual program units of source code, sets of one or more programming modules are tested by the programmers. The individual units are functions, methods, classes and procedures etc.

• Integration Testing

Once the individual units of source code are tested and functional, the individual programming modules are collected together and integration testing is performed on the constructed sub systems. Integration testing is executed by the programmers and integration testing team. The major goal of performing the integration testing is to make sure that a steady system is developed that can stand up to the correctness of system level testing.

• System Testing

System testing consists of multiple different activities that include: test plan creation, test suite design, test environments preparation, test strategy execution and test execution process monitoring.

• Acceptance Testing

Once the system level testing is completed, the product is handed over to the customer and customer performs a series of tests that is called acceptance testing. The main objective of acceptance testing is to test the quality of the product delivered instead of finding the bugs and defects. Acceptance testing is of two types that include:

• User Acceptance Testing

User acceptance testing is executed by the customer to make sure that the system meets the acceptance criteria before signing off.

• Business Acceptance Testing

Business acceptance testing is executed at the supplier’s development organization to make sure that the system will pass the user acceptance test.

There is one more level of testing known as Regression testing that is executed throughout the Software Development Life Cycle of a system. Whenever any component or components of a system are modified, regression testing comes into picture. The main concern during regression testing is to ensure that the changes made to the system have not caused any new defects in the remaining portions of the system. In a nutshell, regression testing is not separately identified as a definite level of testing rather it is a sub-phase of unit, integration and system testing.

White Box and Black Box Testing

The above are the two broad concepts of testing information sources for test design. The black box testing techniques are known as functional testing techniques and in black box testing the program is assumed as a black box. In black box testing, the internal details of the program do not have to be known to the tester. The tester only concerns with what is visible and accessible from outside the program that is input and outcome. The tester gives the input to the program, detects the outcome visible to the outside of the program and verify whether it is the expected outcome or not.

White box testing techniques are known as structural testing techniques and in white box testing the source code is examined with primary focus on data and control flow. Data flow is referred to the distribution of values from one variable to another and control flow is referred to the flow of control from one instruction to the other instruction.

Principles of Good Testing

• Reduce business risk by finding issues or defects.
• Reduce risk by performing positive and negative testing.
• Reduce risk by performing static and execution testing.
• Reduce risk by using automated test tools.
• Set the first testing priority for the highest risks.
• Set the second testing priority for the most frequent business activities.
• Test the system in a manner as the customers are going to use it.
• Forecast testing completion by performing statistical analyses of defect arrival patterns and defect characteristics.
• Assuming that the defects are the outcome of process and not the personality.
• Testing for defects is an investment and a cost too.