|
|
What happens to a bug from start to finish.
While attending testing seminars, I noticed that there was a gap in what was being taught. There’s a lot of theory presented, a lot of ‘why test’ classes and a lot of classes on specific techniques but nothing on a couple of practices that will go a long way towards improving the testing process in a company, specifically setting up a defect tracking system and enforcing policies and procedures to resolve those defects. Setting up these two things, more than anything else, will put a company on the road to organizing its testing and QA effort. To fill that gap, I’ve come up with the ‘Bug Life Cycle’ presentation. While I can’t claim it as my own, it is what I’ve learned over the years as a tester; many of you will find it familiar.
What is a bug?
In computer technology, a bug is a coding error in a computer program. Myers defined it by saying that “A software error is present when the program does not do what its end user reasonably expects it to do.” (Myers, 1976.). I tell my testers if you don’t like it, it’s a bug.
Over the years, my colleagues and I have decided that there are as many definitions for the term “bug” as there are testers. “There can never be an absolute definition for bugs, nor an absolute determination of their existence. The extent to which a program has bugs is measured by the extent to which it fails to be useful. This is a fundamentally human measure.” (Beizer, 1984.). For a more definitive list of many types of bugs refer to Software Testing by Cem Kaner, et. al., pages 363-432.
Who can report a bug?
Anyone who can figure out that the software isn’t working properly can report a bug. The more people who critique a product, the better it’s going to be. However, here’s a short list of people expected to report bugs:
Testers / QA personnel
Developers
Technical Support
Beta sites
End users
Sales and marketing staff (especially when interacting with customers).
When do you report a bug?
When you find it! When in doubt, write it up. Waiting means that you’ll forget to write it altogether or important details about the bug will be forgotten. Writing it now also gives you a ‘scratch pad’ to make notes on as you do more investigation and work on the bug.
Also, writing the bug when you find it makes that information instantly available to everyone. You don’t have to run around the building telling everyone about the bug; a simple phone call or email will alert everyone that the bug exists. Additionally, the information about the bug doesn’t change or get forgotten with every telling of the story.
Bugs are tracked in a database
The easiest way to keep track of defect reports is in a database. Paper is an ok way to record defect reports on but pieces of paper can get lost or destroyed; a database is more reliable and can be backed up on a regular basis.
You can purchase many commercially available defect tracking databases or you can build your own. It’s up to you. I’ve always built my own with something small like Microsoft Access or SQL Server. The decision then was that it was cheaper to build and maintain it on site than it was to purchase it. You’ll have to run the numbers for your situation when you make that decision.
The rule of thumb is one and only one defect per report (or record) when writing a bug report. If more than one defect is put into a report, the human tendency is to deal with the first problem and forget the rest of them. Also, defects are not always fixed at the same time. With one defect per report, as the defects get fixed, they will be tested individually instead of in a group where the chance that a defect is overlooked or forgotten is greater.
You may hear the term “bugfile” used by people when referring to a defect database. The name bugfile is a slang term from the old WordPerfect Corporation. “Bugs” were first logged into a flat file database called DataPerfect; a file of bugs, hence the word “bugfile”.
A good bug reports include the following items:
Put the Reporter’s Name on the bug. If there are questions we need to know who originated this report.
Specify the Build or Version number of the code being worked on. Is this the shipping version or a build done in-house for testing and development? Some bugs may only occur in the shipping version; if this is the case, the version number is a crucial piece of information.
Specify the Feature or Specification or part of the code. This facilitates assigning the bug to a developer assigned to that part of the product.
Include a Brief Description of what the problem is. For example, “Fatal error when printing landscape.” is a good description; short and to the point.
List Details including how to duplicate the bug and any other relevant data or clues about the bug. Start with how the computer and software is setup. List each and every step (don’t leave any out) to produce the bug. Sometimes a minor detail can make all the difference in duplicating or not duplicating a bug. For example, using the keyboard versus using the mouse may product very different results when duplicating a bug.
If the status isn’t ‘Submitted’ by default, change it to Submitted. This is a flag to the bug verifier that a new bug has been created and needs to be verified and assigned.
Things to remember…
Keep the text of the bug impersonal. Bug reports will be read by a variety of people including those outside the department and even the company. Please don’t insult people’s ancestors or the company they work for or the state they live in or make any other impulsive or insensitive comment. Be careful with humorous remarks; one person’s humor is another person’s insult. Keep the writing professional.
Be as specific as possible in describing the current state of the bug along with the steps to get into that state. Don’t make assumptions that the reader of the bug will be in the same frame of mind as you are. Please don’t make people guess where you are or how you got into that situation. Not everyone is thinking along the same lines as you are.
Rating Bugs
While it is important to know how many bugs are in a product, it is even more useful to know how many of those bugs are severe, ship stopping bugs compared to the number of inconvenient bugs. To aid in assessing the state of the product and to prioritize bug fixes, bugs are ranked. The easiest way to rank or rate bugs is to assign each bug a severity rating and a likelihood rating. This assignment is done by the bug reporter when the bug is created. The bug’s rating is a combination of the severity and likelihood ratings.
Severity
The severity tells the reader of the bug how bad the problem is. Or in other words, say what the results of the bug are. Here’s a common list for judging the severity of bugs. There is sometimes disagreement about how bad a bug is. This list takes the guess work out of assigning a severity to bugs.
Rating Value
Blue screen 1
Loss without a work around 2
Loss with a work around 3
Inconvenient 4
Enhancement 5
Likelihood
Put yourself in the average user’s place. How likely is a user to encounter this bug? While the tester may encounter this bug every day with every build, if the user isn’t likely to see it, how bad can the bug be?
Rating Value
Always 1
Usually 2
Sometimes 3
Rarely 4
Never 5
Severity * Likelihood = Rating
Computing the rating of a bug is done by multiplying the numeric value given to the severity and likelihood status’. Do the math by hand or let your defect tracker do it for you.
The trick is to remember that the lower the number, the more severe the bug is. The highest rating is a 25 (5 X 5), the lowest is 1 (1 X 1). The bug with a 1 rating should be fixed first while the bug with a 25 rating may never get fixed.
Looking at a list of these bugs ordered by rating means the most important ones will be at the top of the list to be dealt with first. Sorting bugs this way also lets management know whether the product is ready to ship or not. If the number of severe (1) bugs is zero, the product can ship. If there are any severe bugs, then bug fixing must continue.
Other useful information
Who’s the bug Assigned to; who’s going to be responsible for the bug and do the work on the bug?
What Platform was the bug found on – Windows, Linux, etc. Is the bug specific to one platform or does it occur on all platforms?
What Product was the bug found in? If your company is doing multiple products this is a good way to track those products.
What Company would be concerned about this bug? If your company is working with multiple companies either as an OEM or as customer this is a good way to track that information.
Whatever else you want or need to keep track of. Some of these fields will also have value to marketing and sales. It’s a useful way to track information about companies and clients.
An example of a bug report:
Figure 1 The status tells us the state of the bug. The severity tells us how bad the bug is. The likelihood tells us how often the average user will see the bug. The Assigned To field tells us who is responsible for resolving the bug. The Feature tells us what part of the product the bug is in. The Problem Description gives a brief (very brief) summation of the problem. This description is used when compiling reports or lists of bugs. The details tell us the current setup or situation, the steps to duplicate the problem and any other essential information that will allow someone else to duplicate the bug. Once the bug is submitted, this field cannot be changed. Any additional information will go in the notes field. The notes field contains any discussions about the bug. For example, when and why the status was changed and by whom; additional information about how to duplicate the bug that will aid in resolving the bug; and opinions about the bug. This is a “free-forum” area where people should feel free to express their opinions without censure or criticism. Once comments are placed on a bug, they cannot be changed or deleted by anyone else or even the author. The comments, like the details, stand. Anyone reading the bug after the fact should be able to understand not only what the bug was and how bad a bug it was but also how it was resolved and why it was resolved that way.
Examples of poorly written bugs:
Please keep in mind that I didn’t make any of these up!
“My computer crashed.” We are sorry for your loss, can you give us more information?
“It’s kinda “sucky”.” This one violates all sorts of rules. What’s kind of “sucky”. For that matter, define “sucky.”. Better yet, don’t use the work “sucky” it’s not in the dictionary and most certainly not in good taste.
“It don’t.” This bug doesn’t provide enough information. What don’t? Don’t what?
“Product needs a “speel” checker.” It goes without saying that “spelling counts”!
Now we have a bug…
The first step is Verification. A bug verifier searches the database looking for all ‘Submitted’ bugs assigned to him. He then duplicates the bug by following the steps listed in the details section of the bug. If the bug is reproduced and has all the proper information, the assigned to field is changed to the appropriate person who will be fixing the bug. If the bug is not written clearly, is missing some steps or can’t be reproduced, it will be sent back to the bug reporter for additional work.
The Assigned To field contains the name of the person responsible for that area of the product or code. It is important to note that from this point onward, the developer’s name stays on the bug. Why? There are usually more developers than there are testers. Developers have a set of features to work on. Developers look at bugs from a stand point of “what is assigned to me?”. Testers have multiple sets of features to test. Testers look at bugs from a stand point of “what needs to be tested?”; testers may also change what features they are assigned to test. Because of the different way testers and developers work, developers sort bugs by the Assigned To field and testers sort bugs by the Status field. Leaving the developer’s name on the bug also makes it easier to send the bug back to the developer for more work. The tester simply changes the status field to Verified and it automatically goes back to the developer.
The Developer works on the bug…
The first thing the developer does is give the bug a ‘In Progress’ status indicating that he has seen the bug and is aware that it his responsibility to resolve. The developer works on the bug and based on his conclusions assigns a status to the bug indicating what the next step should be.
Remember, the developer does NOT change the Assigned To field. His name stays on the bug so if the bug has to go back to him, it will make back onto his list. This procedure ensures that bugs don’t fall between the cracks.
The following is a list of status’ that a developer can assign to a bug.
Fixed
The Fixed status indicates that a change was made to the code and will be available in the next build. Testers search the database on a daily basis looking for all Fixed status bugs. Then the bug reporter or tester assigned to the feature retests the bug duplicating the original circumstances. If the bug is fixed and it is now working properly, another test with slightly different circumstances is performed to confirm the fix. If the bug passes both tests, it gets a Tested status.
If the bug doesn’t pass the test, the bug is given a Verified status and sent back to the developer. Notice here that since the bug’s Assigned To field has retained the developer’s name, it’s an easy process for the tester to send the bug back by simply changing the status to Submitted.
Duplicate
The Duplicate status bug is the same as a previously reported bug. Sometimes only the developer or person looking at the code can tell that the bug is a duplicate. It’s not always obvious from the surface. A note indicating the previous bug number is placed on the duplicate bug. A note is also placed on the original bug indicating that a duplicate bug exists. When the original bug is fixed and tested, the duplicate bug will be tested also. If the bug really is a duplicate of previous bug then the when the previous bug is fixed, the duplicate bug will also be fixed. If this the case then both bugs get a Tested status.
If the duplicate is still a bug, while the original bug is working properly, the duplicate bug is no longer has a duplicate status. It gets a Submitted status and is sent back to the developer. This is a “fail-safe” built into the bug life cycle. It’s a check and balance that prevents legitimate bugs from being swept under the carpet or falling between the cracks.
A note of warning. Writing lots of duplicate bugs will get a tester a reputation for being an “airhead”. It pays to set time aside daily to read all the new bugs written the previous day.
Resolved
Resolved means that the problem has been taken care of but no code has been changed. For example, bugs can be resolved by getting new device drivers or third party software. Resolved bugs are tested to make sure that the problem really has been resolved with the new situation. If the problem no longer occurs, the bug gets a Tested status. If the Resolved bug still occurs, it is sent back to the developer with a Submitted status.
Need More Information
Need More Information or “NMI” indicates that the bug verifier or developer does not have enough information to duplicate or fix the bug; for example, the steps to duplicate the bug may be unclear or incomplete. The developer changes the status to ‘Need More Information’ and includes a question or comments to the reporter of the bug. This status is a flag to the bug reporter to supply the necessary information or a demonstration of the problem. After updating the bug information (in the Notes field), the status is put back to Verified so the developer can continue working on the bug. If the bug reporter can not duplicate the bug, it is given a Can’t Duplicate status along with a note indicating the circumstances.
The only person who can put “Can’t Duplicate” on a bug is the person who reported it (or the person testing it). The developer can NOT use this status, he must put Need More Information on it to give the bug reporter a chance to work on the bug.
This is another example of a “fail-safe” built into the database. It is vital at this stage that the bug be given a second chance. The developer should never give a bug a ‘Can’t Duplicate’ status. The bug reporter needs an opportunity to clarify or add information to the bug or to retire it.
Working as Designed
The developer has examined the bug, the product requirements and the design documents and determined that the bug is not a bug, it is Working as Designed. What the product or code is doing is intentional as per the design. Or as someone more aptly pointed out it’s “working as coded”! It’s doing exactly what the code said to do.
This bug can go several directions after being assigned this status. If the tester agrees with the status, then the status stands and the bug is finished. The bug may be sent to documentation for inclusion in help files and the manual. If the tester disagrees with the status then the bug can be appealed by putting a Submitted status on it to send the bug back through the process again. The tester should include in the notes a reason why, although it is Working as Designed, it should be changed now. The bug may also be sent back to the design committee so that the design can be improved.
This is a dangerous status. It’s an easy way to sweep bugs under the carpet by giving them this status. It’s up to the bug reporter to make sure the bug doesn’t get forgotten in this manner. Product managers may also review lists of bugs recently assigned Working as Designed.
Enhancement
Enhancement means that while the suggested change is great idea because of technical reasons, time constraints or other factors, it won’t be incorporated into the code until the next version of the product.
This status may also be appealed by changing the status to Submitted and adding a note specifying why it should be fixed now.
Defer
Defer is almost the same status as Enhancement. This status implies that the cost of fixing the bug is too great given the benefits that it would produce. If the fix is a one liner to one file that doesn’t influence other files, it might be ok to fix the bug. On the other hand, if the fix will force the rebuild of many files which would force the re-testing of the product and there’s no time left to test the fix before shipping the product, then the fix would be unacceptable and the bug would get a Defer status. To appeal the status, send it back through the process again by putting a Submitted status on it and add a note saying why it should be fixed now.
Not to be Fixed
You may see the Not to be Fixed status although I don’t recommend making this status available for use. There may be extenuating circumstances where a bug will not be fixed because of technology, time constraints, a risk of destabilizing the code or other factors. A better status to use is Not to be Fixed. To appeal the status, send it back through the process again by putting a Submitted status on it and add a note saying why it should be fixed now.
This is similar to the Working as Designed status in that its use can be dangerous. Be on the watch for this one. Sometimes developers call this status “You can’t make me”.
Tested
The Tested status is used only by testers on Fixed, Resolved and Duplicate bugs. This status is a “end of the road” status indicating that the bug has reached the end of its life cycle.
Pending
The Pending status is used only by testers on Fixed bugs when a bug cannot be immediately tested. The tester may be waiting on hardware, device drivers, a build or additional information necessary to test the bug. When the necessary items have been obtained, the bug status is changed back to Fixed and then it is tested. Make sure the testing part isn’t skipped.
Can’t Duplicate
This status is used only by the bug reporter; developers or managers cannot use this status. If a bug isn’t reproducible by the assigned developer or bug verifier the bug reporter needs a chance to clarify or add to the bug. There may be a hardware setup or situation or particular way of producing a bug that is peculiar to only this computer or bug reporter and he needs a chance to explain what the circumstances are. Limiting this status to bug reporters only prevents bugs from slipping between the cracks and not getting fixed.
The product has shipped, what happens next?
First of all, to ship a product the number of bugs rated 5 or less, bugs with a Fixed, Resolved, Pending or Need More Information status must be zero. We’ll assume that the product has shipped and all these bugs have been taken care of. This means the bugfile is full of bugs that have reached the end of their life cycle. Proper database maintenance takes place at this point. Archiving or hiding all these bugs will make the database easier to use and read.
All bugs with a Tested or Can’t Duplicate bugs are archived. This means that the records are either removed and placed in an archive database or flagged to be hidden from the current view of the database. Never delete any bug records; it may be necessary to do some historical research in the bugfile (‘What did we ship when?’ or ‘Why did we ship with this bug?’).
Enhancement and Defer bugs are either moved to the new bugfile or retained in the current bugfile. The status of these bugs is then changed back to Verified.
Advanced defect database techniques
There are things you can do with your database to make it more than just a to-do list for developers and testers. The bugfile is basically raw data, sorting and filtering it makes the data information that is useful in the decision making process done by management. A couple of these things to do with the data is to create Reports and Customized views.
Reports
The data in the defect database is not very useful until it is sorted and presented in a organized fashion, then it becomes information. For example, sorting by developer, the information becomes a ‘to do’ list sorted by rating. Sorting by status lets the reader know how many bugs are submitted or in progress; i.e. how many bugs are currently being worked on? By feature – how many open bugs are there for a particular feature? What feature needs more work and what feature is stable? Sorting by product is useful when more than one product is being worked on simultaneously.
Be aware that there are certain metrics or reports that should not be used. If you use these reports you will destroy the credibility of your bug file and it will be reduced to a “laundry-list” for developers. One of these reports is “how many bugs did a tester report” and the other is “how many bugs did a developer fix”. Neither one of these has any useful purpose except to beat up people uselessly. See Software Testing by Cem Kaner, et. al.
Examples of Reports
The Product Manager wants to see a list of bugs to be fixed before shipping that is sorted by rating to determine how ready the product is for shipping. He needs to know what work is left to be done so that an appropriate schedule can be set. The Test Lead needs a list of bugs to be tested before shipping, sorted by tester to make sure all the bugs get tested in the allotted amount of time. The Development Lead has a list of bugs to be resolved before shipping, sorted by developer so that adjustments in work load can be made to ensure that everything is taken care of by the deadline. Technical Support likes a list of bugs that were fixed and not fixed in the product before shipping so they can adequately prepare to support the product. Technical support can resolve customer problems faster by knowing what has been fixed in a product release (the new release or upgrade will fix that problem) and what bugs are still in the product.
A defect database that has all these fields built into it is able to sort defect data and make it useful information.
Customized views of the database
If your defect tracking database supports it, you can limit the information seen based on who the person logging in is. For example, the database that I built with SQL Server 7 was web browser based. Each person was assigned a job category such as ‘Tester’, ‘Developer’ or ‘Manager’. The web page that displayed was then the web page designed for that job category. The Developer would see Submitted and In Progress bugs assigned to him while Testers would see all Fixed, NMI, Pending, Resolved, and Duplicate bugs regardless of who is assigned to the bug. The Product Manager view sees all bugs assigned to him, all newly reported bugs, and newly updated bugs.
The reason for this customized view is that each job views the defect database in a different way based on their needs and the work that has to be accomplished. Testers don’t care who the bug is assigned to, just what the status is. Testers want to see all the bugs, not just bugs written by them, with a ‘Fixed’ status so the bugs can be tested with the latest build. Developers only care about what current and active bugs are assigned to them; developers aren’t concerned about any other person’s bugs. Product managers are concerned that bugs don’t get ‘swept under the carpet’ with Enhancement or Defer status’ so managers want to see all bugs with that particular status.
References
Software Testing by Cem Kaner, et.al.
Software Testing in the Real World by Edward Kit
Software Test Documentation (IEEE 829-1983)
The Complete Guide to Software Testing by Bill Hetzel
The Art of Software Testing by Glenford J. Myers
Software Testing and Quality Assurance by Boris Beizer
Software Quality: Analysis and Guidelines for Success by Capers Jones
Software Testing Techniques by Boris Beizer
Software Testing by Ron Patton (2000, Sams, ISBN 0672319837). See chapters 18 and 19.
“Organize Your Problem Tracking System Cleaning up your bug database can be as easy as organizing your sock drawer” By Barry Mirrer. |
|
/1 
关于我们
发表于 2006-5-12 22:07:30
QQ好友和群
QQ空间
腾讯微博
腾讯朋友
微信
收藏
置顶服务
换色服务
