From:	 Andreia Shibayama <deia@conectiva.com.br>
To:	 lwn@lwn.net
Subject: Snapshot quality factor explained
Date:	 Wed, 2 May 2001 19:05:33 -0300 (EST)

Snapshot Quality Factor (draft)
Wed May  2 17:05:03 BRT 2001


Introduction

The snapshot webpage (http://snapshot.conectiva.com/) presents a quality
percentile of the current snapshot. By standard means, there's no way to
quantitatively measure the quality of a software product; there are many
variables involved, and not all of them are known. However, it is possible
to evaluate this quality based on certain parameters, and different choice
of parameters can result in different quality values. For instance, a high
quality product from the developer's point of view can be deemed a
product of questionable quality from the end user's point of view.

The "snapshot index" is a way to assign a value that gives a rough idea
of the current snapshot quality from the user's point of view. It is
based on a few premises:

1. More bugs implies in lower quality
2. Critical bugs are worse than minor bugs
3. Prioritary bugs should be solved first
4. Bugs that are not noticed by the user don't affect quality
5. Users affect by bugs will report them to the bug tracking system
6. Quality is not taken as an instantaneous value, it is affected by
   extended periods of low quality (see more about this item in the
   "Bug aging" section below")

Note that item 5. is the weakest link in the chain: the whole system
depends on user feedback to produce accurate values.


The basic formula

In the approach we adopted for the snapshot distribution the normalized
quality is given by the following equation:

                        n(i)
                ____   ____
                \   |  \   |
     Q =  1 - [  >      >    A(i,p) * (a(i,j) + 1) * w(i) ] / N		(1)
                /___|  /___|
                sev i  j = 1


where 'a' is the bug ticket age, 'p' is the ticket priority, 'A' is the
aging coefficient, 'w' is a weight given to each severity i in the bugzilla
severities domain (currently { enhancement, trivial, minor, normal, major,
critical, blocker }) and N is the number of bugzilla components (packages)
in the distribution.

The aging coefficients and severity weights are currently given by the
following tables:

    Table 1: bug weights                  Table 2: aging coefficients
    +------------------+                   +-----------------------+
    |Severity    Weight|                   |Priority   Aging Coeff.|
    +------------------+                   +-----------------------+
    |  enh        0.01 |                   |   P1            4     |
    |  tri        0.70 |                   |   P2            2     |
    |  min        1.00 |                   |   P3            1     |
    |  nor        2.50 |                   |   P4           0.5    |
    |  maj        5.00 |                   |   P5            0     |
    |  cri       10.00 |                   +-----------------------+
    |  blo       10.00 |
    +------------------+

These values are available at the bottom of the daily snapshot page.
An extra value (severity aging) is also used, but not shown in
equation (1) for clarity. It's currently equal to 0.01 for all
severities except "enhancement", which has value 0.


Severity weights

The weight given to each open bug ticket according to its severity is
listed in Table 1 and it can be explained as follows: a "critical" bug
open is ten times worse (for the "snapshot index") than a minor bug.
Note that these values are pushed to meet very rigid standard: the
base value is given by a "minor" bug, not "normal". Additionaly, from
equation (1) we can see that a flawless distribution can have a 100%
limit value, but its lower limit is not 0% -- in fact it can get much
lower than that! To make it even more rigorous, a bad veredict is
issued when quality drops below 50%.

We consider that 80-85% are very good values for production systems,
values higher than that are not realistic for real systems with a
reasonably large user base. (A 100% value could be easily obtained by
a system with no users, or at least no bug reports).


Bug aging

For distribution quality purposes, we consider that having an old open
bug ticket is worse than having a recently open bug ticket of the same
severity (that means that the distribution had this ticket open for a
larger amount of time). Standard P3 bugs have their final quality-affecting
value increased by 1% per day of bug age. Enhancements won't age.

If nobody touches the distribution, and there are non-enhancement bug
tickets open, the quality factor will drop with time. To maintain it at
good levels permanent work is needed.


Implementation

The current snapshot system updates the quality factor hourly and compares
it to the index of the snapshot distribution available in the ftp site.
One might have noticed that the values at the top and bottom of the
snapshot status web page don't match. That's a bug I'll work on later,
and is caused by differences between the instant index value and the
immediately previous one. If you want accuracy with hour precision, use
the values at the bottom of the page.