Fedora 22 (and later) nightly testing

Right around the release of Fedora 21, I talked to the anaconda team a bit about the idea of doing more organized early validation testing for Fedora 22. We later discussed and approved the idea at a QA meeting, and since then I've been working on the bits to make it all run smoothly.

We had an early prototype with the Fedora 21 nightly testing, which used pages created every month and asked folks to enter the date of the image they tested along with their report. But since then, I'd set up the template system for generating result pages, and written relval, reviving the testcase-stats output, and I thought it would probably make sense to change the approach a bit. It was impractical to produce new result pages very often with the old manual-cut-n-paste method - it was just too much of a pain - but it's very easy with relval, and having new pages regularly (but not too regularly) results in better testcase-stats output and an easier experience for testers.

So after quite a bit of work on relval and the wiki templates, as of last night, we should have a completely automated process in place for 'nominating' nightly composes for testing - creating result pages for them, and sending an announcement mail to test-announce. Possibly starting tomorrow (it depends on some checks), you might see the first fully-automated nightly compose nomination, with a subject something like Fedora 22 20141225 nightly compose nominated for testing.

Every few days - it won't ever be more than every three days, and should rarely or never be less than every 14 - a new nightly compose will be nominated, the full set of result pages will be created, and the announcement mail will go out. The results pages should provide accurate instructions and download links for testing the nightly composes (right now there are direct links to network install images, and Koji searches for lives and disk images - providing direct links to those would be a bit messy, though not impossible, though some improvements to Koji's web UI might help). The Fedora 22 testcase-stats pages are being updated every hour, to provide an up-to-date overview of the current test coverage. Remember, you can report results with relval report-results these days, and you may find it a bit more convenient than editing the wiki (although it doesn't yet allow you to add comments - I'll add that in a later version). Please use the latest relval, 1.8.3 as I write this, for reporting results against nightly composes.

Of course, there's no strict requirement for the amount of nightly testing that gets done. The only strict requirement in the validation test plan is that we get full test coverage for the release candidates. The idea of the nightly testing is we try our best to cover as much testing as we can, which will help identify blocker bugs earlier and help the developers - particularly the installer developers - space out the work on fixing them, and reduce the amount of churn we experience trying to fix blocker bugs during the TC/RC windows. The same pay-off applies for testers - we can spread out the work of identifying bugs across the cycle instead of starting at Alpha TC1.

As mentioned this idea started in discussion with the anaconda folks, but it may well be of interest to all the WGs. I hope the WGs will take advantage of the opportunity to help test their bits as early as possible and get things into shape early!


adamw wrote on 2014-12-25 20:33:

For the record, 20141225 didn't get nominated because the package version comparison check decides there's no significant new packages since 20141222. Whenever one of the 'significant' packages changes (currently the list is anaconda, python-blivet, pyparted, lorax, pungi, parted, pykickstart), the next nightly which composes successfully will get nominated.