Test Day metrics

Well, first of all, those of you who’ve noticed the schedule around here may be wondering why I’m not advertising this week’s Test Day. That would be because it’s on UEFI, a technology that’s basically only available to motherboard manufacturer’s or companies with access to motherboard manufacturers’ advance hardware test programs, so the only people who can contribute to this test day are really folks working for Red Hat or motherboard manufacturers – and all of those already know about it. So, that’s that. 🙂

Secondly, I was asked by James Laska if I could come up with some metrics for the video driver triage days we’ve run for Fedora 11. After an absorbing hour or two whacking at the data set by grepping through the test day wiki pages for bug IDs, feeding them into python-bugzilla and then sorting the result via OpenOffice.org (why yes, I AM a fan of Heath Robinson…oh, and of course, I use nedit for its awesome ability to do rectangular selections), here we go for the Nouveau and Radeon test days. Unfortunately I can’t get data for Intel yet because of a bug in python-bugzilla, but once Will’s fixed that, I shall.

So! To start with Nouveau:

40 bugs were reported as a result of the test day. As of now, 14 (35%) are in ASSIGNED state, 10 (25%) are in NEW state, and 16 (40%) are in CLOSED state. Of the CLOSED bugs, 4 were closed as DUPLICATE, 2 were closed as NOTABUG, and the other 10 were closed as RAWHIDE (i.e. fixed).

And, Radeon:

46 bugs were reported as a result of the test day. As of now, 26 (57%) are in ASSIGNED state, 14 (30%) are in NEW state, and 6 (13%) are in CLOSED state. Of the CLOSED bugs, 5 were closed as DUPLICATE, and one was closed as RAWHIDE (i.e. fixed).

So the Nouveau test day has already been pretty successful: just two weeks after it happened, it’s already resulted in ten fixed bugs. Radeon test day has only resulted in one fixed bug so far, but plenty of valid reports which should hopefully be addressed as work continues on the driver. Also, a quarter or so of the bugs filed haven’t yet been triaged (that’s the difference between NEW and ASSIGNED), so Matej, François and myself should get those done.

Intel data coming as soon as the python-bugzilla bug’s fixed!

5 Responses

  1. jspaleta
    jspaleta April 9, 2009 at 6:33 am | | Reply

    This is great start on metrics.

    If I may let me give you a personal challenge. Before the test day process for f12 begins can you do a few summary graphs concerning f11 test days.

    I’m thinking a stacked line graphs showing the evolution of bug states over time, showing exactly the information you wrote here..but in graph form and covering more of the test day items.

    Are bugs filed as part of test days tagged as such in bugzilla for easy culling?


  2. jspaleta
    jspaleta April 9, 2009 at 1:24 pm | | Reply

    I can do plots stupid easy with python-matplotlib..its what I do.

    If you want pie charts shoot me something I can process into a chart. A well labelled csv spreadsheet is fine. and I’ll hand you back a script for you to use again.

    The really interesting thing I’d like to see is if on average test day bugs get resolved more quickly compared to bugs that just dribble in. Is there an enhancement in efficiency not just in testing..but all the way into resolution? If there is that’s going to be a pretty good argument to engage upstream developers about, as they’ll have a more self-serving reason to want a test day for their codebase.


  3. AdamW on Linux and more » Blog Archive » Intel Test Day statistics

    […] up my earlier post, the python-bugzilla was fixed, so here’s the Intel […]

Leave a Reply

Your email address will not be published. Required fields are marked *