Yep, this here is a sequel to my most recent best-seller, Automated critical path update functional testing for Fedora :)
When I first thought about running update tests with openQA, I wasn't actually thinking about testing critical path packages. I just made that the first implementation because it was easy. But I first thought about doing it when we added the FreeIPA tests to openQA - it seemed pretty obvious that it'd be handy to run the same tests on FreeIPA-related updates as well as running them on the nightly development release composes. So all along, I was planning to come up with a way to do that too.
Funnily enough, right after I push out the critpath update testing stuff, a FreeIPA-related update that broke FreeIPA showed up, and Stephen Gallagher poked me on IRC and said "hey, it sure would be nice if we could run the openQA tests on FreeIPA-related updates!", so I said "funny you should ask..."
I bumped the topic up my todo list a bit, and wrote it that afternoon, and now it's deployed in production. For now, it's pretty simple: we just have a hand-written list of packages that we want to run some of the update tests for, whenever an update shows up with one of those packages in it. Simple enough, but it works: whenever an update containing one of those packages is submitted or edited, the server update tests (including the FreeIPA tests) will get run, and the results will be visible in Bodhi.
Here's a run on the staging instance that was triggered using the new code; since I sent it to the production instance no relevant updates have been submitted or edited, but it should work just the same there. So from now on whenever our FreeIPA-ish overlords submit an update, we'll get an idea of whether it breaks everything right away.
We can extend this system to other packages, but I couldn't think of any (besides postgresql, which I threw in there) which would really benefit from the current update tests but aren't already in the critical path (all the important bits of GNOME or in the critical path, for example, so all the desktop update tests get run on all GNOME updates already). If you can think of any, go ahead and let us know.