Project Mojito, my long-term work goal, involves us making the Fedora QA process sufficiently well-oiled and automated that the squishy humans can spend all our time on the beach with mojitos. (Continuing to draw a paycheck in this glorious future is a somewhat tricky problem which I've yet to solve, but give me time!) Here's just one small step on the path to that promised land.
We run many Test Days for Fedora. Each cycle, I organize a set of three, one for each commonly-used X graphics driver. Together, these are known as the X Test Week. Back in the dark ages of Fedora 11, when this started, the three test days were somewhat more separated, and the particular areas tested for each driver were quite different. Each driver ended up with its own range of test cases, some of which were very similar, some of which aimed to test the same thing but were written quite differently, and some of which were unique to one driver and not present on the other.
Over time the set of test cases for each driver got more and more similar, but all the test case pages were still completely independent. Little differences grew up between otherwise identical test cases - errors fixed in one but not another, additional instructions given for Intel but not Radeon. It was silly.
No more! The brand new world is here!
If you go to the page for a specific graphics test case now - for instance, the Intel XVideo test - you will see a boring old test case, much as it has looked for years. But look at the page source, and you will find only the category, and this mysterious invocation:
{{Template:Testcase_video_xv|driver=intel|module=i915}}
This is the beauty of abstraction, laid scandalously bare to your inspection. The use of templates is pretty common in mediawiki-based wikis; if you edit the Fedora wiki at all you're likely used to using 'small' templates like {{package|packagename}} to format (packagename) with a neat little standard format that makes it obvious you're referring to a package, for instance. But you can use templates for much bigger stuff, if you like. For this purpose, I converted each of the various X test cases into one gigantic template. Template:Testcase_video_xv is the template for the XVideo test case: you can see it here. You can see that it simply contains the entire test case, with variables named driver and module. When you include this template with the driver and module parameters specified - as in the mysterious invocation seen above - they are inserted into the template, so now it refers to the X driver 'intel' and the kernel module 'i915', and we have an Intel-specific X video test case. Simply by changing the driver= and module= parameters we can create the Radeon and Nouveau XVideo test cases, and all three will now always be in sync; any change to the test process itself is made to the template, and will of course appear in each of the three driver-specific test case pages.
You can nest templates, as deeply as you like - so each of the X test case templates uses several other templates, mostly typical utility templates like {{package}} and {{command}}. I also created yet another new template to add one more step of abstraction to the X test cases. I noticed just about each one starts out by instructing you to check the X and kernel configuration, and do a clean boot. Instead of having this same text in each test case template, I made it into its own little template and included that template into each test case template. Now if the general procedure for preparing the system properly to run an X test changes, we only have to change that template once, not change each test case template separately.
You can also play neat little tricks. There are still occasions where the test procedure may differ slightly between packages. For instance, when testing OpenGL, you have to install the mesa-dri-drivers-experimental package for nouveau, but not for the other two drivers. I could have just put this step into the generic template - it wouldn't really hurt radeon or intel users to install it - but instead I found a more elegant method. I added this variable to the start of the setup step of the OpenGL test case template:
{{{extrasetup|}}}
that is, it's a variable called 'extrasetup'. The neat bit is the pipe with nothing after it; that sets the content of the variable, if it's not specified when the template is called, as nothing at all (the default is to fill it in with the variable's name). So if you call the template and specify content for the 'extrasetup' variable, that content will be included as steps at the start of the 'setup' process. If you don't specify the extrasetup variable, everything will look fine, there will be no extra setup steps. So you can see that the nouveau OpenGL test case specifies content for the 'extrasetup' variable to add an extra setup step instructing the user to install mesa-dri-drivers-experimental; the Radeon and Intel OpenGL test cases don't specify the 'extrasetup' variable at all, and they simply get the normal setup steps from the template.
This was an afternoon's work to set up, but it allowed me to standardize the test case set across all three drivers, and it means that it'll be much easier to adjust the test cases in future, and they will not fall out of synchronization.
