Testing Standards

Testing of software is defined very differently by different people and different corporations. You have process standards bodies, like ISO, SPICE, IEEE, etc. that attempt to impose a process to whatever types of development projects you do (be it hardware, software, embedded systems, etc.) and some of that will, by proxy, speak to testing. However, these are more there to guide the process and not the testing, such as it is. So, for example, IEEE will give you ideas for templates for such things as test case specifications, test plans, etc. That may help you out. On the other hand, those IEEE templates tell you nothing about actually testing the product itself. They basically just show you how to document that you are testing the product. The same thing pretty much applies with ISO. ISO is the standard for international projects and yet it, like IEEE, does not really force or even advocate a certain "testing standard." You also have other process and project oriented concepts out there like the Capability Maturity Model (CMM)

Some of the organization that define testing standards are

• · BS – British Standards
• · ISO- International Organization of Standards
• · CMM- Capability Maturity Model
• · SPICE- Software Process Improvement and Capability Determination
• · NIST-National institute of Standards and Technology
• · DoD-Department of Defense

1. SW – CMM:

SEI - Software Engineering Institute, Carnegie Mellon University.

CMM - Capability Maturity Model

Software Process

A software process can be defined as a set of activities, methods, practices, and transformations that people use to develop and maintain software and the associated products Software Process Capability

Software Process Capability describes the range of expected results that can be achieved by following a software process.The software process capability of an organization provides one means of predicting the most likely outcomes to be expected from the next software project the organization undertakes.

Software Process Maturity

Software Process Maturity is the extent to which a specific process is explicitly defined, managed, measured, controlled, and effective. Maturity implies a potential growth in capability and indicates both the richness of an organization’s software process and the consistency with which it is applied in projects throughout the organization

The five levels of SW- CMM

Leel 1: Initial

Level 2: Repeatable

Level 3: Managed

Level 4: Defined

Level 5: Optimum

2. SW – TMM

SW-TMM is a testing process improvement tool that can be used either in conjunction with the SW-CMM or as a stand-alone tool.

2.1. Levels of SW –TMM

2.1.1. Level 1: Initial

• · A chaotic process
• · Not distinguished from debugging and ill defined
• · The tests are developed ad hoc after coding is complete
• · Usually lack a trained professional testing staff and testing tools
• · The objective of testing is to show that the system and software work

2.1.2. Level 2: Phase Definition

• · Identify testing as a separate function from debugging
• · Testing becomes a defined phase following coding
• · Standardize their process to the point where basic testing techniques and methods are in place
• · The objective of testing is to show that the system and software meets specifications

2.1.3. Level 3: Integration

• · Integrate testing into the entire life cycle
• · Establish a formal testing organization
• · establishes formal testing technical trainings
• · controls and monitors the testing process
• · begins to consider using automated test tools
• · Te objective of testing is based on system requirements
• · Major milestone reached at this level: management
• · recognizes testing as a professional activity

2.1.4. Level 4: Management and Measurement

• · Testing is a measured and quantified process
• · Development products are now tested for quality attributes such as Reliability, Usability and Maintainability.
• · Test cases are collected and recorded in a test database for reuse and regression testing
• · Defects found during testing are now logged, given a severity level, and assigned a priority for correction

2.1.5. Level 5: Optimization / Defect Prevention and Quality Control

• · Testing is institutionalized within the organization
• · Testing process is well defined and managed
• · Testing costs and effectiveness are monitored
• · Automated tools are a primary part of the testing process
• · There is an established procedure for selecting and evaluating testing tools

2.2. Need to use SW-TMM

• · easy to understand and use
• · provide a methodology to baseline the current test
• · process maturity
• · designed to guide organization
• · selecting process improvement strategies
• · identifying critical issues to test process maturity
• · provide a road map for continuous test process improvement
• · provide a method for measuring progress
• · allow organizations to perform their own assessment

Organizations that are using SW-CMM

• · SW-TMM fulfills the design objective of being an excellent companion to SW-CMM
• · SW-TMM is just another assessment tool and easily incorporated into the software process assessment
  
  

Organizations that are not using SW-CMM

• · provide an unbiased assessment of the current testing process
• · provide a road map for incremental improvements
• · save testing cost as the testing process moves up the maturity levels

2.3. SW-TMM Assessment Process

• · Prepare for the assessment
• · choose team leader and members
• · choose evaluation tools (e.g. questionnaire)
• · training and briefing
• · Conduct the assessment
• · Document the findings
• · Analyze the findings
• · Develop the action plan
• · Write the final report
• · Implement the improvements best to implement the improvements either in a pilot project or in phases track progress and achievements prior to expanding organization wide also good in a limited application easier to fine-tune the new process prior to expanded implementation

2.4.SW-TMM Summary

• · baseline the current testing process level of maturity
• · identify areas that can be improved
• · identify testing processes that can be adopted organization-wide
• · provide a road map for implementing the improvements
• · provide a method for measuring the improvement results
• · provide a companion tool to be used in conjunction with the SW-TMM

3. ISO : International Organisation for Standardisation

• · Q9001 – 2000 – Quality Management System : Requirements
• · Q9000 – 2000 – Quality Management System : Fundamentals and Vocabulary
• · Q9004 – 2000 – Quality Management System : Guidelines for performance improvements

4.4.ANSI / IEEE Standards

ANSI - ‘American National Standards Institute’ IEEE Standards: Institute of Electrical and electronics Engineers (Founded in 1884) Have an entire set of standards devoted to software. Testers should be familiar with all the standards mentioned in IEEE.

1. 610.12-1990 IEEE Standard Glossary of Software Engineering Terminology

2. 730-1998 IEEE Standard for Software Quality Assurance plans

3. 828-1998 IEEE Standard for Software Configuration Management

4. 829-1998 IEEE Standard for Software Test Documentation

5. 830-1998 IEEE Recommended Practice for Software Requirement Specifications.

6. 1008-1987(R1993) IEEE Standard for Software Unit Testing

7. 1012-1998 IEEE Standard for Software Verification and validation

8. 1012a-1998 IEEE Standard for Software Verification and validation –Supplement to 1012-1998 Content

9. 1016-1998- IEEE Recommended Practice for Software Design description

10. 1028-1997 IEEE Standard for Software Reviews

11. 1044-1993 IEEE Standard Classification for Software Anomalies

12. 1045-1992 IEEE Standard for Software Productivity metrics

13. 1058-1998 IEEE Standard for Software Project Management Plans

14. 1058.1-1987 IEEE Standard for Software Management

15. 1061-1998.1 IEEE Standard for Software Quality Metrics Methodology.

4.5.BCS - SIGIST

A meeting of the Specialist Interest Group on Software Testing was held in January 1989 (this group was later to affiliate with the British Computer Society). This meeting agreed that existing testing standards are generally good standards within the scope which they cover, but they describe the importance of good test case selection, without being specific about how to choose and develop test cases.

The SIG formed a subgroup to develop a standard which addresses the quality of testing performed. Draft 1.2 was completed by November 1990 and this was made a semi-public release for comment. A few members of the subgroup trialed this draft of the standard within their own organizations. Draft 1.3 was circulated in July 1992 (it contained only the main clauses) to about 20 reviewers outside of the subgroup. Much of the feedback from this review suggested that the approach to the standard needed re-consideration.