What is Test Plan?

test-plan

If you have any quarries/feed back send me on - knowthetesting@gmail.com
Thank you
Ram

Testing Process

In order to make the testing process as effective as possible it needs to be viewed as one with the development process (Fig 2.1.). In many organizations, testing is normally an ad-hoc process being performed in the last stage of a project, if performed at all.

Figure 2.1. Testing should be conducted throughout the development process.

Studies show that testing often represents between 30-50% of the software development cost (RUP).

In order to reduce testing costs, a structured and well defined way of testing needs to be implemented.

Certain projects may appear too small to justify extensive testing. However, one should consider the impact of errors and not the size of the project.

Important to remember is that, unfortunately, testing only shows the presence of errors, not the absence of them.

The figure below (Fig 2.2.) gives a general explanation about both the development process and the testing process but it also shows the relation between these two processes. In real life these two processes, as stated earlier, should be viewed and handled as one. The colors in the figure are to show relations between phases. For example, system testing of an application is to check so that it meets the requirements specified at the start of the development. A large part of integration testing is to check the logical design done in the design phase of the development. The relationships between the phases are based partly on the V-model as presented by Mark Fewster and Dorothy Graham (Software Test Automation, 1999).

Figure 2.2. Test phases relations with Development phases

As mentioned above, a well defined and understood way of testing is essential to make the process of testing as effective as possible. In order to produce software products with high quality one has to view the testing process as a planned and systematic way of performing activities. The activities included are Test Management, Test Planning, Test Case Design & Implementation and Test Execution & Evaluation. Test management will not be further discussed in this report.

1.1 Test planning

As for test planning, the purpose is to plan and organize resources and information as well as describing the objectives, scope and focus for the test effort. The first step is to identify and gather the requirements for the test. In order for the requirements to be of use for the test, they need to be verifiable or measurable. Within test planning an important part is risk analysis. When assigning a certain risk factor, one must examine the likelihood of errors occurring, the effect of the errors and the cost caused by the errors. To make the analysis as exhaustive as possible, each requirement should be reviewed. The purpose of the risk analysis is to identify what is needed to prioritize when performing the test. Risk analysis will be further discussed later in this report.

In order to be able to create a complete test plan, resources needs to be identified and allocated. Resources include

• Human – Who and how many are needed
• Knowledge - What skills are needed
• Time - How much time needs to be asserted
• Economic - What is the estimated cost
• Tools - What kind of tools are needed (hardware, software, platforms, etc.)

A good test plan should also include stop criterias. These can be very intricate to define, since the actual quality of the software is difficult to determine. Some common criterias used are (Rick Hower, 2000):

• Deadlines (release deadlines, testing deadlines, etc.)
• Test cases completed with certain percentage passed
• Test budget depleted
• Coverage of requirements reaches a specified point
• Bug rate falls below a certain level
• Beta or alpha testing period ends

Since every error in today’s complex software very rarely is found, one can go on testing forever if stop criterias aren’t used. Specific criterias should therefore be defined for each separate test case in the process.

The outcome of test planning should of course be the test plan, which will function as the backbone, providing the strategies to be used throughout the test process.

2 Test case design & implementation

The main objective with test case design is to identify the different test cases or scenarios for every software build. These cases shall describe the test procedures and how to perform the test in order to reach the goal of each case. For each test case, the particular conditions for that test shall be described as well as the specific data needed and the supposed result. If there have been testing done on the subject before, the use of old cases becomes important.

The design of test cases is based on what is to be tested. Features to be tested often present a unique need and the testing should be done in small sections to cope with the differences in test case design that occur due to this. When testing a single feature there are a number of things to consider; how does it work, what may cause it to crash, what are the possible variables?

Both data input and user actions should be done in ways that test the designed logic so that we get answers to the questions; do we get the expected answer, what happens when wrong input is used? If, for instance, you are prompted to write your age in a field, the logic behind this may expect a number between 0 and 100, but of course you might by mistake put a letter instead. What happens then? If the application is not designed to see this mistake it will cause the application to crash or at least give an unexpected answer. This makes it important to test this feature so that it does not accept the wrong input but still, of course, accepts the expected input.

Considering the many ways to make similar mistakes there are thousands of different inputs that should be tested. It is not possible to test all these potential inputs, instead one should choose input that in a good way represent all possible groups of input. This procedure is often referred to as boundary testing.

For the results of the test to be useful and to know when stop criterias are reached, the completeness of the test cases is very important.

When creating tests based on the test cases, certain objectives should be addressed.

• Make your tests as reusable as possible
• Make your tests easy to maintain
• Use existing tests when possible

3 Test execution & evaluation

When running your tests, the results have to be taken care of in a defined manner. Was the test completed or did it halt. Are the results of the test the expected ones and how can it be verified that the results originate from a correct run of a test.

When the actual results from a test do not match the expected, certain actions have to be taken. The first is to determine why the actual and the expected results differ. Does the error lie in the tested application or e.g. the test script?

When errors are found, they need to be properly reported. Information on the bug needs to be communicated so that developers and programmers can solve the problem. These reports should include, among others, application name and version, test date, tester’s name and description on the occurred error.

When a bug has been properly taken care of, the software needs to be re-tested to verify that the problems have been solved, and that the corrections made did not create new conflicts.

During the process of development and testing of software, many changes will surely be made to the software as well as its environments. When such changes are made, there will be a need to assure that the program still functions as required. This kind of testing is called regression testing. The difference between a re-test and regression test is that the latter is done when changes has been made to the program regarding, for example, its functionality, whereas re-test is done to test the software after bug fixing.

4 Test phases

As the testing process should be viewed as parallel with the development, it will go through certain phases. When the development moves forward the scope and targets for testing changes, starting with each piece and ends with the complete system. The goal is that every part is tested and fully functioning before being integrated with other parts. Important in the development process is that neither of these phases is completely separated from each other. There is no definite border between when the different phases starts or ends. They can be seen as an overall approximate guideline for how to perform a successful test. Several authors, including Bill Hetzel and Hans Schaefer, describe the test process as consisting of the following phases:

• Unit testing
• Integration testing
• System testing
• Acceptance testing

Unit testing

Also called module test. The testing done on this stage is on the isolated unit.

Integration testing

When units interact with others, one must assure that the communication between them works.

Conflicts often occur when units are developed separately or if the syntax to be used is not communicated in a sufficient way.

System testing

When the system is complete, testing on the system as a whole can commence. Test cases with actual user behaviour can be implemented and non-functional tests, such as usability and performance, may be made.

Acceptance testing

The purpose is to let end users or customers decide whether to accept the system or not. Are the users feeling comfortable with the product and does it perform the activities as required?

5 Test types

The previous text describes the general guidelines for testing, whether it is software applications or web applications. But as the title of this report implies, the scope is centred on how to perform successful testing on web applications and how this process differs from the general test process.

In order to proceed to this area certain issues have to be addressed first. One must be acquainted with the different types of tests that are performed within the different stages throughout the process. The text that follows describes, in short, the most commonly used terms for test types used, with aim on the medium of the web.

There are no definite borders between the types and several of them can seem overlapping with adjacent areas. Needless to say, there are several opinions on this matter and we base the following descriptions on authors such as Hans Schaefer, Bill Hetzel, Tim Van Tongeren and Hung Q. Nguyen.

Functionality testing

The purpose of this type of test is to ensure that every function is working according to the specifications. Functions apply to a complete system as well as a separated unit. Within the context of the web, functionality testing can, for example, include testing links, forms, Java Applets or ActiveX applications.

Performance testing

To ensure that the system has the capability that is requested the performance have to be tested for. The characteristics normally measured are execution time, response time etc. In order to identify bottlenecks, the system or application have to be tested under various conditions. Varying the number of users and what the users are doing helps identify weak areas that are not shown during normal use. When testing applications for the web this kind of testing becomes very important. Since the users are not normally known and the number of users can vary dramatically, web applications have to be tested thoroughly. The general way of performance testing should not vary, but the importance of this kind of test varies (Schaefer, 2000). Testing web applications for extreme conditions is done by load- and stress testing. These two are performed to ensure that the application can withstand, for example, a large amount of users simultaneously or large amount of data from each user. Other characteristics of the web that is important can be download time, network speed etc.

Usability

To ensure that the product will be accepted on the market it has to appeal to users. There are several ways to measure usability and user response. For the web, this is often very important due to the users low acceptance level for glitches in the interface or navigation. Due to the nearly complete lack of standards for web layout this area is dependent on actual usage of the site to receive as useful information as possible. Microsoft has extensive standards down to pixel level on where, for example, buttons are to be placed when designing programs for windows. The situation on the web is exceptionally different with almost no standards at all on how a site layout should be designed.

Compatibility testing

This refers to different settings or configuration of, for example, client machine, server or external databases. When looking at web this can be a very intricate area to test due to the total lack of control over the client machine configuration or an external database. Will your site be compatible with different browser versions, operating systems or external interfaces? Testing every combination is normally not possible, so identifying the most likely used combinations is usually how it’s done.

Security testing

In order to persuade customers to use Internet banking services or shop over the web, security must be high enough. One must feel safe when posting personal information on a site in order to use it. Typical areas to test are directory setup, SSL, logins, firewalls and logfiles.

Security is an area of great importance as well as great extent, not least for the web. A lot of literature has been written on this subject and more will come. Due to the complexity and size of this particular subject, we will not cover this area more than the basic features and where one should put in extra effort.

---

SDLC & STLC

STLC – Software Testing Life Cycle

• Preparation of Testing Project Plan which includes Test Strategy.
• Preparation of Test Scripts which contains Test Scenarios.
• Preparation of Testing Bed. i.e.: Setting up the Test Environment
• Executing the Test Scripts (Automated as well as Manual Tests)..
• Defect Tracking with any bug tracking tools.
• Preparation of Test Completion Report and Test Incident Report.
• Preparation of Test Metrics for Continuous Process Improvement.

Models of SDLC & STLC

There are a number of different models for software development life cycle. One thing which all

models have in common is that at some point in the life cycle, software has to be tested. This

paper outlines some of the more commonly used software development life cycle, with particular

emphasis on the testing activities in each model.

1. V-Model

The figure shows the brief description of the V-Model kind of testing. Every phase of the STLC in

this model corresponds to some activity in the SDLC. The Requirement Analysis would

correspondingly have an acceptance testing activity at the end. The design has Integration

Testing (IT) and the System Integration Testing (SIT) and so on.

• V model is model in which testing is done parallel with development. Left side of v model, reflect development input for the corresponding testing activities.
• V model is the classic software development model. It encapsulates the steps in Verification and Validation phases for each step in the SDLC. For each phase, the subsequent phase becomes the verification (QA) phase and the corresponding testing phase in the other arm of the V becomes the validating (Testing) phase
• In the Software Development Life Cycle, both the Development activity and the testing activities start almost at the same time with the same information in their hands. The development team will apply "do-procedures" to achieve the goals and the testing team will apply "Check-procedures" to verify that. Its a parallel process and finally arrives to the product with almost no bugs or errors
  
• V-model is one of the SDLC STLC; it includes testing from the unit level to business level. That is after completing the coding tester starts testing the code by keeping the design phase documents that all the modules had been integrated or not, after that he will verify for system is according to the requirements or not, and at last he will go for business scenarios where he can validate by the customer and he can do the alpha testing and beta testing. And at last he decides to have the complete stable product.
• The V model shows the Development Cycle Stages and Maps it to Testing Cycles, but it fails to address how to start for all these test levels in parallel to development. It is a parallel activity which would give the tester the domain knowledge and perform more value added, high quality testing with greater efficiency. Also it reduces time since the test plans, test cases, test strategy are prepared during the development stage itself.

2.W-Model
From the view of testing, all of the models presented previously are deficient in various ways. The
test activities first start after the implementation:

• The connection between the various test stages and the basis for the test is not clear
• The tight link between test, debug and change tasks during the test phase is not clear

In the following, the W-model is presented. This is based on the general V-model and the
disadvantages previously mentioned are removed.
3.Waterfall Model
One of the first models for software development is the so-called waterfall-model by B.W.Boehm.
The individual phases i.e. activities that were defined here are to be found in nearly all models
proposed since. In this it was set out that each of the activities in the software development must
be completed before the next phase begins. A return in the development process was only
possible to an immediate previous phase.


In the waterfall-model, testing directly follows the implementation. By this model it was suggested that activities for testing could first be started after the implementation. Preparatory tasks for the testing were not clear. A further disadvantage is that testing, as the last activity before release, could be relatively easily shortened or omitted altogether. This, in practice, is unfortunately all too common. In this model, the expense of the removal of faults and defects found is only recognizable through a return to the implementation phase.


4.Extreme Programming Model




5.Spiral Model

In the spiral-model a cyclical and prototyping view of software development was shown. Tests
were explicitly mentioned (risk analysis, validation of requirements and of the development) and
the test phase was divided into stages. The test activities included module, integration and
acceptance tests. However, in this model the testing also follows the coding. The exception to this
is that the test plan should be constructed after the design of the system. The spiral model also
identifies no activities associated with the removal of defects

Software Testing Life Cycle

Software testing life cycle identifies what test activities to carry out and when (what is the best time) to accomplish those test activities. Even though testing differs between organizations, there is a testing life cycle.

Software Testing Life Cycle consists of six (generic) phases:

• Test Planning,
• Test Analysis,
• Test Design,
• Construction and verification,
• Testing Cycles,
• Final Testing and Implementation and
• Post Implementation.

Software testing has its own life cycle that intersects with every stage of the SDLC. The basic requirements in software testing life cycle is to control/deal with software testing – Manual, Automated and Performance.

Test Planning

This is the phase where Project Manager has to decide what things need to be tested, do I have the appropriate budget etc. Naturally proper planning at this stage would greatly reduce the risk of low quality software. This planning will be an ongoing process with no end point.

Activities at this stage would include preparation of high level test plan-(according to IEEE test plan template The Software Test Plan (STP) is designed to prescribe the scope, approach, resources, and schedule of all testing activities. The plan must identify the items to be tested, the features to be tested, the types of testing to be performed, the personnel responsible for testing, the resources and schedule required to complete testing, and the risks associated with the plan.). Almost all of the activities done during this stage are included in this software test plan and revolve around a test plan.

Test Analysis

Once test plan is made and decided upon, next step is to delve little more into the project and decide what types of testing should be carried out at different stages of SDLC, do we need or plan to automate, if yes then when the appropriate time to automate is, what type of specific documentation I need for testing.

Proper and regular meetings should be held between testing teams, project managers, development teams, Business Analysts to check the progress of things which will give a fair idea of the movement of the project and ensure the completeness of the test plan created in the planning phase, which will further help in enhancing the right testing strategy created earlier. We will start creating test case formats and test cases itself. In this stage we need to develop Functional validation matrix based on Business Requirements to ensure that all system requirements are covered by one or more test cases, identify which test cases to automate, begin review of documentation, i.e. Functional Design, Business Requirements, Product Specifications, Product Externals etc. We also have to define areas for Stress and Performance testing.

Test Design

Test plans and cases which were developed in the analysis phase are revised. Functional validation matrix is also revised and finalized. In this stage risk assessment criteria is developed. If you have thought of automation then you have to select which test cases to automate and begin writing scripts for them. Test data is prepared. Standards for unit testing and pass / fail criteria are defined here. Schedule for testing is revised (if necessary) & finalized and test environment is prepared.

Construction and verification

In this phase we have to complete all the test plans, test cases, complete the scripting of the automated test cases, Stress and Performance testing plans needs to be completed. We have to support the development team in their unit testing phase. And obviously bug reporting would be done as when the bugs are found. Integration tests are performed and errors (if any) are reported.

Testing Cycles

In this phase we have to complete testing cycles until test cases are executed without errors or a predefined condition is reached. Run test cases --> Report Bugs --> revise test cases (if needed) --> add new test cases (if needed) --> bug fixing --> retesting (test cycle 2, test cycle 3….).

Final Testing and Implementation

In this we have to execute remaining stress and performance test cases, documentation for testing is completed / updated, provide and complete different matrices for testing. Acceptance, load and recovery testing will also be conducted and the application needs to be verified under production conditions.

Post Implementation

In this phase, the testing process is evaluated and lessons learnt from that testing process are documented. Line of attack to prevent similar problems in future projects is identified. Create plans to improve the processes. The recording of new errors and enhancements is an ongoing process. Cleaning up of test environment is done and test machines are restored to base lines in this stage.

Software Testing Life Cycle
Phase	Activities	Outcome
Planning	Create high level test plan	Test plan, Refined Specification
Analysis	Create detailed test plan, Functional Validation Matrix, test cases	Revised Test Plan, Functional validation matrix, test cases
Design	test cases are revised; select which test cases to automate	revised test cases, test data sets, sets, risk assessment sheet
Construction	scripting of test cases to automate,	test procedures/Scripts, Drivers, test results, Bugreports.
Testing cycles	complete testing cycles	Test results, Bug Reports
Final testing	execute remaining stress and performance tests, complete documentation	Test results and different metrics on test efforts
Post implementation	Evaluate testing processes	Plan for improvement of testing process

What is Software testing?

According to Wikipedia "Testing is a process of technical investigation, performed on behalf of stakeholders, that is intended to reveal quality-related information about the product with respect to the context in which it is intended to operate. This includes, but is not limited to, the process of executing a program or application with the intent of finding errors."

According to the classic definition of Myers "Software Testing is the process of executing a program or system with the intent of finding errors."

According to the definition given by Hetzel, "Testing involves any activity aimed at evaluating an attribute or capability of a program or system and determining that it meets its required results"


In simple words, software testing is the process of finding deviation from the requirements and user expectations. As a tester, you will be the first person to look at the product from a user point of view and the last person to give objective information about its quality prior to its release.