Defect Reporting Guidelines

The key to making a good report is providing the development staff with as much information as necessary to reproduce the bug. This can be broken down into 5 points:

1. Give a brief description of the problem
2. List the steps that are needed to reproduce the bug or problem
3. Supply all relevant information such as version, project and data used.
   
4. Supply a copy of all relevant reports and data including copies of the expected results.
5. Summarize what you think the problem is.

When you are reporting a defect the more information you supply, the easier it will be for the developers to determine the problem and fix it.

Simple problems can have a simple report, but the more complex the problem– the more information the developer is going to need.

For example: cosmetic errors may only require a brief description of the screen, how to get it and what needs to be changed.

However, an error in processing will require a more detailed description, such as:

1) The name of the process and how to get to it.

2) Documentation on what was expected. (Expected results)

3) The source of the expected results, if available. This includes spread sheets, an earlier version of the software and any formulas used)

4) Documentation on what actually happened. (Perceived results)

5) An explanation of how the results differed.

6) Identify the individual items that are wrong.

7) If specific data is involved, a copy of the data both before and after the process should be included.

8) Copies of any output should be included.

As a rule the detail of your report will increase based on a) the severity of the bug, b) the level of the processing, c) the complexity of reproducing the bug.

Anatomy of a bug report

Bug reports need to do more than just describe the bug. They have to give developers something to work with so that they can successfully reproduce the problem.

In most cases the more information– correct information– given the better. The report should explain exactly how to reproduce the problem and an explanation of exactly what the problem is.

The basic items in a report are as follows:

Version: This is very important. In most cases the product is not static, developers will have been working on it and if they’ve found a bug– it may already have been reported or even fixed. In either case, they need to know which version to use when testing out the bug.

Product: If you are developing more than one product– Identify the product in question.

Data: Unless you are reporting something very simple, such as a cosmetic error on a screen, you should include a dataset that exhibits the error.

If you’re reporting a processing error, you should include two versions of the dataset, one before the process and one after. If the dataset from before the process is not included, developers will be forced to try and find the bug based on forensic evidence. With the data, developers can trace what is happening.

Steps: List the steps taken to recreate the bug. Include all proper menu names, don’t abbreviate and don’t assume anything.

After you’ve finished writing down the steps, follow them - make sure you’ve included everything you type and do to get to the problem. If there are parameters, list them. If you have to enter any data, supply the exact data entered. Go through the process again and see if there are any steps that can be removed.

When you report the steps they should be the clearest steps to recreating the bug.

Description: Explain what is wrong - Try to weed out any extraneous information, but detail what is wrong. Include a list of what was expected. Remember report one problem at a time; don’t combine bugs in one report.

Supporting documentation:

If available, supply documentation. If the process is a report, include a copy of the report with the problem areas highlighted. Include what you expected. If you have a report to compare against, include it and its source information (if it’s a printout from a previous version, include the version number and the dataset used)

This information should be stored in a centralized location so that Developers and Testers have access to the information. The developers need it to reproduce the bug, identify it and fix it. Testers will need this information for later regression testing and verification.

Test Report

A Test Report is a document that is prepared once the testing of a software product is complete and the delivery is to be made to the customer. This document would contain a summary of the entire project and would have to be presented in a way that any person who has not worked on the project would also get a good overview of the testing effort.

Contents of a Test Report

The contents of a test report are as follows:

Executive Summary

Overview

Application Overview

Testing Scope

Test Details

Test Approach

Types of testing conducted

Test Environment

Tools Used

Metrics

Test Results

Test Deliverables

Recommendations

These sections are explained as follows:

1. Executive Summary

This section would comprise of general information regarding the project, the client, the application, tools and people involved in such a way that it can be taken as a summary of the Test Report itself (i.e.) all the topics mentioned here would be elaborated in the various sections of the report.

1. Overview

This comprises of 2 sections – Application Overview and Testing Scope.

Application Overview – This would include detailed information on the application under test, the end users and a brief outline of the functionality as well.

Testing Scope – This would clearly outline the areas of the application that would / would not be tested by the QA team. This is done so that there would not be any misunderstandings between customer and QA as regards what needs to be tested and what does not need to be tested.

This section would also contain information of Operating System / Browser combinations if Compatibility testing is included in the testing effort.

2. Test Details

This section would contain the Test Approach, Types of Testing conducted, Test Environment and Tools Used.

Test Approach – This would discuss the strategy followed for executing the project. This could include information on how coordination was achieved between Onsite and Offshore teams, any innovative methods used for automation or for reducing repetitive workload on the testers, how information and daily / weekly deliverables were delivered to the client etc.

Types of testing conducted – This section would mention any specific types of testing performed (i.e.) Functional, Compatibility, Performance, Usability etc along with related specifications.

Test Environment – This would contain information on the Hardware and Software requirements for the project (i.e.) server configuration, client machine configuration, specific software installations required etc.

Tools used – This section would include information on any tools that were used for testing the project. They could be functional or performance testing automation tools, defect management tools, project tracking tools or any other tools which made the testing work easier.

3. Metrics

This section would include details on total number of test cases executed in the course of the project, number of defects found etc. Calculations like defects found per test case or number of test cases executed per day per person etc would also be entered in this section. This can be used in calculating the efficiency of the testing effort.

4. Test Results

This section is similar to the Metrics section, but is more for showcasing the salient features of the testing effort. Incase many defects have been logged for the project, graphs can be generated accordingly and depicted in this section. The graphs can be for Defects per build, Defects based on severity, Defects based on Status (i.e.) how many were fixed and how many rejected etc.

5. Test Deliverables

This section would include links to the various documents prepared in the course of the testing project (i.e.) Test Plan, Test Procedures, Test Logs, Release Report etc.

6. Recommendations

This section would include any recommendations from the QA team to the client on the product tested. It could also mention the list of known defects which have been logged by QA but not yet fixed by the development team so that they can be taken care of in the next release of the application.

hOW TO wRITE gOOD tEST CASES

What is a good test case?

• Accurate - tests what it’s designed to test
• Economica - no unnecessary steps
• Repeatabe, reusabe - keeps on going
• Traceabe - to a requirement
• Appropriate - for test environment, testers
• Sef standing - independent of the writer
• Sef ceaning - picks up after itsef

Types of test cases

• A rose by any other name smells as sweet
• Step-by-step or word/action instructions
• Table, matrix
• Script for record/playback or performance test
• All need the same structure (anatomy)

Anatomy of a test case

• Statement of purpose, what is being tested
• Method, how it will be tested
• Setup, environment, data
• Steps - actions and expected results, implied in atable or matrix
• Proofs, files, printouts, reports, screen grabs(optional)

Improving testability - language

• Testability = easy to test
• Use active case, do this, do that
• System displays this, does that
• Simple, conversational language
• Exact, consistent names of fields, not generic
• Don’t explain Windows basics

Improving testability - length

• 10-15 steps or less, unless user cannot savework
• Uniform time to test
• Wide range of testers
• Pros and cons of cumulative cases
• Order of cases follows business scenarios

The seven most common mistakes

• Making cases too long
• Incomplete, incorrect, or incoherent setup
• Leaving out a step
• Naming fields that changed or no longer exist
• Unclear whether tester or system does action
• Unclear what is a pass or fail result
• Failure to clean up

Test Case Checklist

Quality Attributes

• Accurate - tests what the description says it will test.
• Economical - has only the steps needed for its purpose
• Repeatable, self standing - same results no matter who tests it.
• Appropriate - for both immediate and future testers
• Traceable - to a requirement
• Self cleaning - returns the test environment to clean state

Structure and testability

• Has a name and number
• Has a stated purpose that includes what requirement is being tested
• Has a description of the method of testing
• Specifies setup information - environment, data, prerequisite tests, security access
• Has actions and expected results
• States if any proofs, such as reports or screen grabs, need to be saved
• Leaves the testing environment clean
• Uses active case language
• Does not exceed 15 steps
• Matrix does not take longer than 20 minutes to test
• Automated script is commented with purpose, inputs, expected results
• Setup offers alternative to prerequisite tests, if possible
• Is in correct business scenario order with other tests

Configuration management

• Employs naming and numbering conventions
• Saved in specified formats, file types
• Is versioned to match software under test
• Includes test objects needed by the case, such as databases
• Stored as read
• Stored with controlled access
• Stored where network backup operates
• Archived off-site

How to write a good bug report?

Why good Bug report?
If your bug report is effective, chances are higher that it will get fixed. So fixing a bug depends on how effectively you report it. Reporting a bug is nothing but a skill and I will tell you how to achieve this skill.

“The point of writing problem report(bug report) is to get bugs fixed” - By Cem Kaner. If tester is not reporting bug correctly, programmer will most likely reject this bug stating as irreproducible. This can hurt testers moral and some time ego also. (I suggest do not keep any type of ego. Ego’s like “I have reported bug correctly”, “I can reproduce it”, “Why he/she has rejected the bug?”, “It’s not my fault” etc etc..)

What are the qualities of a good software bug report?
Anyone can write a bug report. But not everyone can write a effective bug report. You should be able to distinguish between average bug report and a good bug report. How to distinguish a good or bad bug report? It’s simple, apply following characteristics and techniques to report a bug.

1) Having clearly specified bug number:
Always assign a unique number to each bug report. This will help to identify the bug record. If you are using any automated bug-reporting tool then this unique number will be generated automatically each time you report the bug. Note the number and brief description of each bug you reported.

2) Reproducible:
If your bug is not reproducible it will never get fixed. You should clearly mention the steps to reproduce the bug. Do not assume or skip any reproducing step. Step by step described bug problem is easy to reproduce and fix.

3) Be Specific:
Do not write a essay about the problem. Be Specific and to the point. Try to summarize the problem in minimum words yet in effective way. Do not combine multiple problems even they seem to be similar. Write different reports for each problem.

How to Report a Bug?

Use following simple Bug report template:
This is a simple bug report format. It may vary on the bug report tool you are using. If you are writing bug report manually then some fields need to specifically mention like Bug number which should be assigned manually.

Reporter: Your name and email address.

Product: In which product you found this bug.

Version: The product version if any.

Component: These are the major sub modules of the product.

Platform: Mention the hardware platform where you found this bug. The various platforms like ‘PC’, ‘MAC’, ‘HP’, ‘Sun’ etc.

Operating system: Mention all operating systems where you found the bug. Operating systems like Windows, Linux, Unix, SunOS, Mac OS. Mention the different OS versions also if applicable like Windows NT, Windows 2000, Windows XP etc.

Priority:
When bug should be fixed? Priority is generally set from P1 to P5. P1 as “fix the bug with highest priority” and P5 as ” Fix when time permits”.

Severity:
This describes the impact of the bug.
Types of Severity:

• Blocker: No further testing work can be done.
• Critical: Application crash, Loss of data.
• Major: Major loss of function.
• Minor: minor loss of function.
• Trivial: Some UI enhancements.
• Enhancement: Request for new feature or some enhancement in existing one.

Status:
When you are logging the bug in any bug tracking system then by default the bug status is ‘New’.
Later on bug goes through various stages like Fixed, Verified, Reopen, Won’t Fix etc.

Assign To:
If you know which developer is responsible for that particular module in which bug occurred, then you can specify email address of that developer. Else keep it blank this will assign bug to module owner or Manger will assign bug to developer. Possibly add the manager email address in CC list.

URL:
The page url on which bug occurred.

Summary:
A brief summary of the bug mostly in 60 or below words. Make sure your summary is reflecting what the problem is and where it is.

Description:
A detailed description of bug. Use following fields for description field:

• Reproduce steps: Clearly mention the steps to reproduce the bug.
• Expected result: How application should behave on above mentioned steps.
• Actual result: What is the actual result on running above steps i.e. the bug behavior.

These are the important steps in bug report. You can also add the “Report type” as one more field which will describe the bug type.

The report types are typically:
1) Coding error
2) Design error
3) New suggestion
4) Documentation issue
5) Hardware problem

Some Bonus tips to write a good bug report:

1) Report the problem immediately:If you found any bug while testing, do not wait to write detail bug report later. Instead write the bug report immediately. This will ensure a good and reproducible bug report. If you decide to write the bug report later on then chances are high to miss the important steps in your report.

2) Reproduce the bug three times before writing bug report:Your bug should be reproducible. Make sure your steps are robust enough to reproduce the bug without any ambiguity. If your bug is not reproducible every time you can still file a bug mentioning the periodic nature of the bug.

3) Test the same bug occurrence on other similar module:
Sometimes developer use same code for different similar modules. So chances are high that bug in one module can occur in other similar modules as well. Even you can try to find more severe version of the bug you found.

4) Write a good bug summary:
Bug summary will help developers to quickly analyze the bug nature. Poor quality report will unnecessarily increase the development and testing time. Communicate well through your bug report summary. Keep in mind bug summary is used as a reference to search the bug in bug inventory.

5) Read bug report before hitting Submit button:
Read all sentences, wording, steps used in bug report. See if any sentence is creating ambiguity that can lead to misinterpretation. Misleading words or sentences should be avoided in order to have a clear bug report.

6) Do not use Abusive language:
It’s nice that you did a good work and found a bug but do not use this credit for criticizing developer or to attack any individual.

Conclusion:
No doubt that your bug report should be a high quality document. Focus on writing good bug reports, spend some time on this task because this is main communication point between tester, developer and manager. Mangers should make aware to their team that writing a good bug report is primary responsibility of any tester. Your efforts towards writing good bug report will not only save company resources but also create a good relationship between you and developers.

For better productivity write a better bug report.