The frequency of bug reporting over the life of a project is somewhat reminiscent of the view of Pike’s Peak from Denver. Why compare it to the Rocky Mountains and a Peak that appears to pop out into the middle of the Colorado plains? Sometimes the defect saturation timeline closely resembles a series of peaks that gradually dwindle to rolling hills with a random peak somewhere near the end.
Typically, the amount of defects being reported are expected to skyrocket at the start of the project then decrease as all the existing defects are discovered and reported. The decrease in defects over time is a good indication of a product that is being thoroughly tested and becoming more stable. However, this is only true if the reported defects are being resolved. While the discovery of new defects may decrease over the lifetime of a project it is a dangerous misconception to perceive that because fewer defects are being reported that the product is bug free. Therefore, taking into account the number of open and unresolved defects in the bug database, the number of open defects must also steadily decrease over time.
With each build deployment that contains fixes, there also contains the potential for more defects to be found. Immediately after each new build there may be a temporary increase in the number of defects reported. This is frequently been the case in instances where ‘Critical’ or ‘Blocker’ defects are resolved, since often untested areas within the product become available almost always contain undiscovered defects. Even with the most minor of defect resolutions, it is always important to retest any area of the product that had been touched by code changes.
Towards the conclusion of development, as the frequency of bug-fixes increases in shorter intervals, it is not uncommon to see a drastic increase in the number of new defects being reported as a result. Consequently, it is not uncommon for rare, often ‘Critical’, defects to emerge shortly before the end of a project. This is why it is important to test the final bug-fixes in pre-release products despite being confident that the fixes ‘just work’. Maintaining a consistent QA presence over the lifeline of the product is key to a successful release free of surprise defects.