From: Andreia Shibayama <deia@conectiva.com.br> To: lwn@lwn.net Subject: Snapshot quality factor explained Date: Wed, 2 May 2001 19:05:33 -0300 (EST) Snapshot Quality Factor (draft) Wed May 2 17:05:03 BRT 2001 Introduction The snapshot webpage (http://snapshot.conectiva.com/) presents a quality percentile of the current snapshot. By standard means, there's no way to quantitatively measure the quality of a software product; there are many variables involved, and not all of them are known. However, it is possible to evaluate this quality based on certain parameters, and different choice of parameters can result in different quality values. For instance, a high quality product from the developer's point of view can be deemed a product of questionable quality from the end user's point of view. The "snapshot index" is a way to assign a value that gives a rough idea of the current snapshot quality from the user's point of view. It is based on a few premises: 1. More bugs implies in lower quality 2. Critical bugs are worse than minor bugs 3. Prioritary bugs should be solved first 4. Bugs that are not noticed by the user don't affect quality 5. Users affect by bugs will report them to the bug tracking system 6. Quality is not taken as an instantaneous value, it is affected by extended periods of low quality (see more about this item in the "Bug aging" section below") Note that item 5. is the weakest link in the chain: the whole system depends on user feedback to produce accurate values. The basic formula In the approach we adopted for the snapshot distribution the normalized quality is given by the following equation: n(i) ____ ____ \ | \ | Q = 1 - [ > > A(i,p) * (a(i,j) + 1) * w(i) ] / N (1) /___| /___| sev i j = 1 where 'a' is the bug ticket age, 'p' is the ticket priority, 'A' is the aging coefficient, 'w' is a weight given to each severity i in the bugzilla severities domain (currently { enhancement, trivial, minor, normal, major, critical, blocker }) and N is the number of bugzilla components (packages) in the distribution. The aging coefficients and severity weights are currently given by the following tables: Table 1: bug weights Table 2: aging coefficients +------------------+ +-----------------------+ |Severity Weight| |Priority Aging Coeff.| +------------------+ +-----------------------+ | enh 0.01 | | P1 4 | | tri 0.70 | | P2 2 | | min 1.00 | | P3 1 | | nor 2.50 | | P4 0.5 | | maj 5.00 | | P5 0 | | cri 10.00 | +-----------------------+ | blo 10.00 | +------------------+ These values are available at the bottom of the daily snapshot page. An extra value (severity aging) is also used, but not shown in equation (1) for clarity. It's currently equal to 0.01 for all severities except "enhancement", which has value 0. Severity weights The weight given to each open bug ticket according to its severity is listed in Table 1 and it can be explained as follows: a "critical" bug open is ten times worse (for the "snapshot index") than a minor bug. Note that these values are pushed to meet very rigid standard: the base value is given by a "minor" bug, not "normal". Additionaly, from equation (1) we can see that a flawless distribution can have a 100% limit value, but its lower limit is not 0% -- in fact it can get much lower than that! To make it even more rigorous, a bad veredict is issued when quality drops below 50%. We consider that 80-85% are very good values for production systems, values higher than that are not realistic for real systems with a reasonably large user base. (A 100% value could be easily obtained by a system with no users, or at least no bug reports). Bug aging For distribution quality purposes, we consider that having an old open bug ticket is worse than having a recently open bug ticket of the same severity (that means that the distribution had this ticket open for a larger amount of time). Standard P3 bugs have their final quality-affecting value increased by 1% per day of bug age. Enhancements won't age. If nobody touches the distribution, and there are non-enhancement bug tickets open, the quality factor will drop with time. To maintain it at good levels permanent work is needed. Implementation The current snapshot system updates the quality factor hourly and compares it to the index of the snapshot distribution available in the ftp site. One might have noticed that the values at the top and bottom of the snapshot status web page don't match. That's a bug I'll work on later, and is caused by differences between the instant index value and the immediately previous one. If you want accuracy with hour precision, use the values at the bottom of the page.