AdamW on Linux and more (Posts about relval)

openQA and Autocloud result submission to ResultsDB

2017-02-06T21:18:23Z

So I've just arrived back from a packed two weeks in Brno, and I'll probably have some more stuff to post soon. But let's lead with some big news!

One of the big topics at Devconf and around the RH offices was the ongoing effort to modernize both Fedora and RHEL's overall build processes to be more flexible and involve a lot more testing (or, as some people may have put it, "CI CI CI"). A lot of folks wearing a lot of hats are involved in different bits of this effort, but one thing that seems to stay constant is that ResultsDB will play a significant role.

ResultsDB started life as the result storage engine for AutoQA, and the concept and name was preserved as AutoQA was replaced by Taskotron. Its current version, however, is designed to be a scalable, capable and generic store for test results from any test system, not just Taskotron. Up until last week, though, we'd never quite got around to hooking up any other systems to it to demonstrate this.

Well, that's all changed now! In the course of three days, Jan Sedlak and I got both Fedora's openQA instance and Autocloud reporting to ResultsDB. As results come out of both those systems, fedmsg consumers take the results, process them into a common format, and forward them to ResultsDB. This means there are groups with results from both systems for the same compose together, and you'll find metadata in very similar format attached to the results from both systems. This is all deployed in production right now - the results from every daily compose from both openQA and Autocloud are being forwarded smoothly to ResultsDB.

To aid in this effort I wrote a thing we're calling resultsdb_conventions for now. I think of it as being a code representation of some 'conventions' for formatting and organizing results in ResultsDB, as well as a tool for conveniently reporting results in line with those conventions. The attraction of ResultsDB is that it's very little more than a RESTful API for a database; it enforces a pretty bare minimum in terms of required data for each result. A result must provide only a test name, an 'item' that was tested, and a status ('outcome') from a choice of four. ResultsDB allows a result to include as much more data as it likes, in the form of a freeform key:value data store, but it does not require any extra data to be provided, or impose any policy on its form.

This makes ResultsDB flexible, but also means we will need to establish conventions where appropriate to ensure related results can be conveniently located and reasoned about. resultsdb_conventions is my initial contribution to this effort, originally written just to reduce duplication between the openQA and Autocloud result submitters and ensure they used a common layout, but intended to perhaps cover far more use cases in the future.

Having this data in ResultsDB is likely to be practically useful either immediately or in the very near future, but we're also hoping it acts as a demonstration that using ResultsDB to consolidate results from multiple test sources is not only possible but quite easy. And I'm hoping resultsdb_conventions can be a starting point for a discussion and some consensus around what metadata we provide, and in what format, for various types of result. If all goes well, we're hoping to hook up manual test result submission to ResultsDB next, via the relval-ng project that's had some discussion on the QA mailing lists. Stay tuned for more on that!

wikitcms, relval, fedfind and testdays moved to Pagure

2016-10-14T14:54:56Z

Today I moved several of my pet projects from the cgit instance on this server to Pagure. You can now find them here:

The home page URLs for each project on this server - e.g. https://www.happyassassin.net/fedfind - also now redirect to the Pagure project pages.

I also deleted some other repos that were hosted in my cgit instance entirely, because I don't think they were any longer of interest to anyone and I didn't want to maintain them. Those were mostly related to Fedlet, which I haven't been working on for 2-3 years now.

For now the repos for the three main projects - wikitcms, relval and fedfind - remain in my cgit instance, containing just a single text file documenting the move to Pagure; in a month or so I will remove these repositories and decommission the cgit instance. So, update your checkouts! :)

This saves me maintaining the repos, provides pull review and issue mechanisms, and it's a good thing to have all Fedora-ish code projects in Pagure in general, I think.

Many thanks to pingou and everyone else who works on Pagure, it's a great project!

Fedora 24 and Rawhide: What's goin' on (aka why is everything awful)

2016-02-29T16:32:10Z

Hi folks!

Welp, I was doing a Fedora 24 status update in the QA meeting this morning, and figured a quick(ish) summary of what all is going on in Fedora 24 and Rawhide right now might also be of interest to a wider audience.

So, uh, the executive summary is: stuff's busted. Lots of stuff is busted. We are aware of this, and fixing it. Hold onto your hats.

glibc langpacks

A rather big change landed in Fedora 24 and Rawhide last week. glibc locales are now split into subpackages using the 'langpack' mechanism that yum introduced and dnf supports. This lets us drop a somewhat ugly hack we were using to remove unneeded locales from space-sensitive images (Cloud and container images). However, it broke...quite a lot of stuff.

Locales lost on update/upgrade

As it initially landed, Fedora 24 / Rawhide users who updated to the new glibc packages lose all their locales; after the update you may have no locales except C and C.UTF-8. Various apps have trouble when you have a locale configured but not available, including but probably not limited to ssh and gnome-terminal. If you've hit this, you probably will want to do something like dnf install glibc-langpack-en at least (substitute your actual locale group for en). If you just want to have all locales back (so you can test apps in other locales and so on), you can do dnf install glibc-all-langpacks.

Installer doesn't run any more

anaconda tries to set os.environ["LANG"] as the default locale when starting up. There's also no dependency or lorax configuration to pull any glibc langpacks into the installer environment. The result is that in recent Rawhide and F24 nightly installer images, anaconda blows up during startup, trying to set a locale that isn't installed.

Live and cloud images don't build

This is actually a consequence of the previous issue. Cloud images are built via anaconda. As of last week, with the Pungi 4 switchover, live images are also now built using anaconda (via livemedia-creator). anaconda hits the locale bug in both those workflows, blows up, and consequently we get no live or cloud images in current Rawhide and Fedora 24 nightly composes.

Pungi 4

Speaking of Pungi 4...yep, as of the middle of last week, Fedora 24 and Rawhide composes are being done with that tool, as I've been talking about for a while. You can see evidence of this in the Rawhide and Branched trees. They now look more like release trees, with variant directories at the top level and all the regular images produced daily (well, they should be, except see above for why half of them are missing). If you've got scripts or anything which expect a certain layout of these trees, you're probably going to have to update them.

Up until glibc threw a spanner in the works this seems to have turned out quite well, but there are a few known consequences so far. There was a bug with the Server DVD installer image not booting properly due to an incorrect inst.stage2 kernel parameter, but that seems to be fixed now.

No name resolution on live images

If you manage to find a Pungi 4-created live image (from the few days before glibc broke 'em) and get it to boot, you'll probably find networking is busted. In fact basic connectivity works, but name resolution doesn't. This is because /etc/resolv.conf is a dangling symlink. This is the latest incarnation of a longstanding...disagreement among the systemd developers, NetworkManager developers, and everyone else unfortunate enough to get caught up in the crossfire. No doubt it'll get bodged up somehow this time, too, soon enough. You can easily resolve the problem manually with rm -f /etc/resolv.conf; ln -s /var/run/NetworkManager/resolv.conf /etc/resolv.conf. The change here isn't Pungi 4 per se, but the fact that under the Pungi 4 regime, live images are now created by livemedia-creator rather than livecd-creator. livecd-creator stuffed a /etc/resolv.conf into the live image it created, which avoided this bug by preventing systemd-tmpfiles from creating it as a dangling symlink on boot. livemedia-creator does not do this, so when the live image boots /etc/resolv.conf does not exist, systemd creates it as a dangling symlink, and NetworkManager refuses to replace the dangling symlink with its own symlink.

Rawhide / Branched reports missing depcheck information

The 'Rawhide report' and 'Branched report' emails are still going out, but they're now generated by a new tool and look a bit different. I kinda like the added information, but some people don't like the new format so much; send patches ;) It is known that at present the new reports are missing information on broken dependencies, and releng are working to get this back ASAP.

compose check report emails not appearing

I've mentioned this before, but briefly, the 'compose check report' emails sent out by my tool aren't happening at all at the moment. The process for producing them runs through fedfind and needed rather a lot of rework for the new Pungi 4-ish world. I have code that works now and am aiming to get it deployed this week. Right now all the reports would basically say "all the tests failed and half the images are missing" due to the above-mentioned problems anyhow.

Long term I'd like to move the image checks from check-compose into compose-utils and thus have the 'missing expected images' and 'image diff to previous compose' bits appear in the 'Rawhide report' / 'Branched report' emails; check-compose would then just generate an openQA test report, basically. Doing that cleanly requires a change to the productmd metadata format, though, which I need to work through with pungi and productmd folks.

Release validation test events not happening

Also due to the compose process changes, we can't really create release validation events at present. Well, we could create nightly ones, but the image download tables would be missing, and we'd have to do it manually; the stuff for creating them automatically is kind of outdated now (it relied on some assumptions about the compose process which no longer really hold true). We can't do Alpha TCs and RCs (and thus the events for them) until we work out with releng how we want to handle TCs and RCs with Pungi 4.

This week I'm aiming to at least update python-wikitcms and relval so we can have proper nightly validation events again and they'll have correct download links. Probably this will just involve changing the page names a bit to add the 'respin' component of Pungi 4 nightly compose IDs (so we'll have e.g. Test Results:Fedora 24 Branched 20160301.0 Installation or Test Results:Fedora 24 Branched 20160301.n.0 Installation instead of Test Results:Fedora 24 Branched 20160301 Installation) and tweaking wikitcms a bit to add the 'respin' concept to its event/page versioning design, and writing a fedmsg consumer which replaces the relval nightly --if-needed mode to create the nightly events every so often.

It'll probably take a bit longer to figure out what we want to do for non-nightly composes.

Other bits: Wayland and SELinux

We also have a couple of other fairly prominent issues related to other changes.

Lives don't boot or don't work properly with SELinux in enforcing mode

A change to systemd seems to result in several things in /run being mislabelled in Fedora live images. (Yeah, yeah, systemd and SELinux...please put down the comment box and step away from the keyboard, trolls, moderation is in effect). With SELinux in enforcing mode (the default), this seems to result in Workstation lives not booting (it sits there looping over failing to set up the live user's session, basically). KDE lives boot, but then lots of stuff is broken (you can't reboot, for instance, and probably lots of other bits, that's just the one our tests noticed). I didn't check other desktops yet.

You can work around this one quite easily by booting with enforcing=0.

Installer doesn't run on Workstation lives

Workstation live images for F24 and Rawhide were flipped over to running on Wayland by default (in most cases) quite recently. Unfortunately, the live installer relies on using consolehelper to run as root, but consolehelper doesn't work on Wayland. So if you find a recent Rawhide / F24 Workstation nightly live, and you get it to boot, and you ignore the fact that networking is busted, you won't be able to install it (bet you were just dying to do that after this blog post, weren't you?) unless you just run /usr/sbin/liveinst directly as root. Well, I mean, I'm not guaranteeing you'll actually be able to install it if you do that. I haven't got that far in testing yet.

IN CONCLUSION

So, um, yeah. We know everything's busted. We know! We're sorry. It's all gettin' fixed. Return to your homes, and your Fedora 23 installs. :)

Identifying Fedora media redux: What is Atomic?

2015-09-18T11:11:18Z

You may remember my post from yesterday, on whether a 'Fedora Atomic' flavor made sense, and how I think about identifying Fedora media for my fedfind project.

Well, after talking it over with a few folks this morning and thinking some more, I've come to the conclusion that post was kind of wrong, on the major question: what is Atomic?

In that last post I came to the conclusion that Atomic was a deployment method. I still think deployment method is a perfectly viable concept, but I no longer think Atomic really is one. Rather, ostree is a deployment method, and Atomic really is - as I previously was considering it - something more like a payload.

Reading the Project Atomic site, it seems I was kind of working from an outdated understanding of the 'Atomic' concept. I'd always understood it as being more or less a branding of ostree, i.e. an 'Atomic' system was just one that used the ostree deployment method. But it seems the concept has now been somewhat changed upstream. To quote the Introduction to Project Atomic:

"The core of Project Atomic is the Project Atomic Host. This is a lightweight operating system that has been assembled out of upstream RPM content. ...

Project Atomic builds on these features, using the following components, which have been tailored for containerized-application management:

Docker, an open source project for creating lightweight, portable, self-sufficient application containers.
Kubernetes, an open source project that allows you to manage a cluster of Linux containers as a single system.
rpm-ostree, an open source tool for managing bootable, immutable, versioned filesystem trees from upstream RPM content.
systemd, an open source system and service manager for Linux."

There is still some conceptual confusion about exactly what Atomic means - for instance, there's a concept called Atomic Apps, and it is apparently fine to deploy Atomic Apps on systems which are not Atomic Hosts. But after discussing it with some folks this morning, I think it's reasonable to take the following as a principle:

Any Fedora image with 'Atomic' in its name deploys as an Atomic Host as defined by Project Atomic.

Assuming that rule holds, for 'image identification' purposes under the fedfind concepts covered in the previous post, atomic really does work best as something in the line of a flavor, loadout and/or payload. It also implies the deployment method ostree, but I think I'm OK with leaving that concept out of fedfind for now as it's functionally useless: as all Atomic images have ostree deployment and all non-Atomic images have rpm deployment, distinguishing the properties is no use at this point. I'm going to hold it in reserve for the future, in case we get other images that use non-RPM deployment methods but are not Atomic hosts.

For now I think I'll more or less go back to the subflavor concept I initially threw out in favour of deployment, and for current purposes, all Atomic images will have flavor cloud and subflavor atomic. The alternative is to have a concept that's a bit more parallel to flavor and loadout which feeds into payload, but for now keeping the interface stable seems better.

Identifying Fedora media

2015-09-17T23:49:33Z

EDIT: With the immensely greater wisdom that comes with being a day older, I'm now more or less convinced this whole post is wrong. But instead of editing it and confusing people who read the original, I've left it here and written a follow-up. Please read that.

Thanks to mizmo for inspiring this post.

On 'Fedora Atomic' as a flavor

Mo wrote that the Cloud WG has decided that, from Fedora 24 onwards, they will focus on Atomic-based images as their primary deliverables. In response to this, Mo and Matt Miller were kicking around the idea of - as I understand it - effectively rebranding the 'Cloud' flavor of Fedora as the 'Atomic' flavor.

This immediately struck me as problematic, and after a bit of thought I was able to identify why.

The current Fedora flavors - Cloud, Server, and Workstation - can be characterized as contexts in which you might want to deploy Fedora. "I want to deploy Fedora on a Cloud!", or "I want to deploy Fedora on a Workstation!", or "I want to deploy Fedora on a Server!" Yes, there's arguably a bit of overlap between 'Cloud' and 'Server', but we do have a reasonable answer to that ('Cloud' is about new-fangled cattle, 'Server' is about old-fashioned pets).

Atomic is not a context in which you might deploy Fedora. You can't say "I want to deploy Fedora on an Atomic!", it just doesn't work. Atomic is rather what I'm referring to as a deployment method. It's a mechanism by which the system is deployed and updated. Another deployment method - the counterpart to 'Atomic' - would be 'RPM'. Atomic happens to a deployment method that's quite appropriate for Cloud deployments, but that doesn't mean you can just do s/Cloud/Atomic/g and everything will make sense.

So for me the idea is simply not conceptually compatible with how we define the flavors. We might even, for instance, want to build 'Atomic Workstation' instances of Fedora. I believe there's even been interest in doing that, before.

Mo's post suggests that we might treat 'cloud' rather like we currently treat 'ARM', as a sort of orthogonal concept to the flavors, with a kind of parallel identity on the download site. I would suggest that it would make more sense to do exactly the opposite: it's Atomic that should be given a sort of parallel existence. We might want to have an Atomic landing page which focused on all the Atomic implementations of Fedora. But it's not, for me, a flavor.

Thinking about identifying Fedora media

So you might be aware, I have a little project called fedfind, which is broadly about finding Fedora media. About half of the work fedfind does is finding the media. The other half of the work it does is identifying them. Fedora 23 Beta RC1 has 59 'images', approximately (depending on whether you count boot.iso files separately from their hardlinked alternate names). Trying to come up with a set of attributes that would serve as a sort of conceptual framework for identifying images has been an interesting (and ongoing!) challenge. In fact, as a result of thinking about the 'Atomic' proposal, I've just revised fedfind's approach a little. So, I thought I'd write quickly about how fedfind does it.

I first thought about this stuff before starting fedfind, when I wrote an image naming policy. fedfind initially followed that policy precisely. Events have overtaken it a bit, though, so the current implementation isn't quite the same. And today, I've added the deployment method concept as an attribute in fedfind's system. The changes described here are sitting in my editor right now, but should show up in git master soon!

Some of the attributes fedfind uses are fairly obvious (arch is just the arch, and release, milestone and compose are just version identifiers), so I'll focus on the more interesting ones. fedfind provides a couple of properties - desc and shortdesc - for images which can be used as descriptions; taken together, all these properties allow us to give a unique identity to every one of the 59 images in 23 Beta RC1, as well as covering all historical releases (in my checks anyway). shortdesc contains payload, deployment (if not 'rpm'), imagetype, and imagesubtype (if present); desc adds arch. For any given release's images, there should be no duplicated descs.

imagetype and imagesubtype

These identify the 'type' of the image specifically. What's the difference between the Fedora 22 x86_64 Workstation live image and the Fedora 22 x86_64 Workstation network install image? They're the same version, same arch, same payload (see below) - but different imagetype. A few images also have an imagesubtype. For instance, we provide two types of Vagrant image, for use with libvirt and virtualbox; these have vagrant as the imagetype and libvirt or virtualbox as the imagesubtype.

flavor, loadout and payload

It's intended that most every fedfind image should have either a flavor or a loadout; for convenience, whichever one it has is also available as the payload. This can be thought of as 'what's the actual content of the image'. Flavors are workstation, server, and cloud; loadouts are things like kde, xfce, mate etc. (for live images and ARM disk images), plus some oddballs. There is a minimal for ARM disk images, and source and desktop for older releases where we had source CDs/DVDs and 'Desktop' live images. There's one fairly special case: pre-Fedora.next DVDs and network install images, boot.isos, and current nightly images have no flavor or loadout, but have the special payload generic.

Current fedfind releases have a concept of subflavor, which is only used to give some Cloud images a subflavor of atomic. After thinking about this (above) it seemed completely wrong, so today I've changed things around so the subflavor property is gone, and instead we have...

deployment

As discussed above, this is the 'deployment method'. Currently it's either rpm or atomic.

Flock 2015 report, and Fedora nightly compose testing

2015-08-21T16:44:41Z

Hi, folks! I've been waiting to write my post-Flock report until I had some fun stuff to show off, because that's more exciting than just a bunch of 'I went to this talk and then talked to this person', right?

Fedora nightly compose testing

So let me get to the shiny first! Without further ado:

Cool, right? That's what I've been working on this whole week since Flock. All the bits are now basically in place such that, each night, openQA will run on the Branched and Rawhide nightly composes when they're done, and when openQA is done, the compose reports will be mailed out.

Flock report

The details behind that get quite long, so before I hit that, here's a quick round-up of other stuff I did at Flock! I'm not going to cover the talks and sessions many others have already blogged about (the keynotes, etc.) as it seems redundant, but I'll mention some stuff that hasn't really been covered yet.

Josef ran a workshop on getting started with openQA. It was a bit tricky, though, due to poor networking on site; the people trying to follow along and deploy their own Docker-based openQA instances couldn't quite get all the way. So we turned the last bit of the talk into a live demo using my openQA instance instead, and created a new test case LIVE ON STAGE. We didn't quite get it all the way done before getting kicked out by a wedding party, but I finished it up shortly after the session. Josef did a great job of explaining the basics of setting up openQA and creating tests, and I hope we'll have a few more people following the openQA stuff now.

Mike McLean did a great talk on Koji 2.0, which has been kinda under the radar (at least for me) compared to Bodhi 2, but sounds like it'll come with a lot of really significant improvements and a better design. As someone who's spent a lot of time staring at kojihub.py lately, I can only say it'd be welcome...

Denise Dumas gave the now-traditional What Red Hat Wants talk, which I'm really glad is happening now. I'm totally behind the idea that we're up-front about the relationship between Red Hat and Fedora, instead of some silly arrangement where Red Hat pretends Fedora just 'happens' and is a totally community-based distro; it's much better for RH to be saying a couple of years in advance 'hey, this is where we'd like to see things going', rather than every so often a bunch of Features/Changes 'mysteriously' appearing four months out from a release and lots of people suddenly caring a lot about them (but it just all being a BIG COINCIDENCE!)

Paul Frields did a nice talk on working remotely, which had a lot of great ideas that I don't do at all (hi Paul, it's 4:30pm and I'm writing this in my dressing gown...) - but it was great to compare notes with a bunch of other folks and think about other ways of doing things.

I did a lightning talk on Fedlet, showing it off running and talking a bit about what Fedlet involves and how well (or not) it runs. Folks seemed interested, and a few people came by to play with my fedlet afterwards.

Stephen Gallagher ran a rolekit hackfest. I was hoping to use it to come up with an openQA role, but failed for a couple of reasons: Stephen doesn't recommend creating new roles right now as the format is likely to change a lot quite soon, and since I last worked on the package openQA has added a few more dependencies which need packaging. But I did manage to move forward with work on the package a bit, which was useful. In the session Stephen explained the rolekit design and current state to people, and talked about various work that needs doing on it; hopefully he'll get some more help with it soon!

Of course, as always, there was lots of hallway track and social stuff. We had a couple of excellent poker games - good to see the FUDCon/Flock poker tradition continues strong - and played some Exploding Kittens, which is a lot of fun. My favourite bit is the NOPE cards. As many others have said, the Strong Museum was awesome - got to play a bunch of pinball, and see Will Wright's notebooks(!) and John Romero's Apple ][(!!!!).

Fedora compose testing: development details and The Future

So, back to the 'compose CI' stuff I spent a lot of time talking about/working on!

A lot of what I did at Flock centred around the big topic you can call 'CI for Fedora'. We still have lots of plans afoot for big, serious test and task automation based on Taskotron, which is now getting really close to the point where you'll see a lot more cool stuff using it. But in the meantime, the 'skunkworks' openQA project we spun up during the Fedora 22 cycle has grown quite a bit, and the fedfind project I mostly built to back openQA has grown quite a lot of interesting capabilities.

So while we were talking about properly engineered plans for the future, I realized I could probably hack together some stupidly-engineered stuff that would work right now! In Kevin Fenzi's Rawhide session I threw out a few ideas and then figured that, hell, I should just do them.

So I started out by teaching fedfind some new tricks. It can now 'diff' two releases: that is, it can tell you what images are in one, but not the other. It can also check a release for 'expected' images - basically it has some knowledge about what images we'd most want to be present all the time, and it can tell you if any are missing. (FIXME: I didn't know which of the Cloud images were the most important, so right now it has no 'expected' Cloud images: if some Cloud-y people want to tell me which images are most important, I can add them).

Then I wrote a little script called check-compose which produces a handy report from that information. It also looks for openQA tests for the compose it's checking, and includes a list of failures if it finds any. It can email the report and also write the results in JSON format (which seemed like a good idea in case we want to look back at them in some programmatic way later on). The 'compose check reports' that have been showing up this week (and that I linked above) are the output of the script.

I had all of that basically done by Tuesday, so what have I been wasting the rest of my week on? Read on!

What was missing was the 'C' part of 'CI'. There was nothing that would actually run the compose report at appropriate times, and we weren't actually running openQA tests nightly. For the past few days I've been kind of faking things up by manually kicking off openQA jobs and firing off the compose report when they're done. This kind of mechanical Turk CI doesn't really work in the long run! So for the last few days I've worked on that.

We were not actually scheduling nightly openQA runs at all. The openQA trigger script has an all mode which is intended to do that, but we weren't running it. I suggested we turn it back on, but I also wanted to fix one big problem it had: it didn't know whether the composes were actually done. It just got today's date and tried to run on the nightlies for it. If they weren't actually done whenever the script ran, you got no tests.

This definitely hooks in with one of the big topics at Flock: Pungi 4, which is the pending major revision. Pungi is the tool which runs Fedora composes. Well, that's not quite right: there's actually a couple of releng scripts which produce the composes (the first of those is for nightlies, the second is for TCs/RCs). They run pungi and do lots of other stuff too, because currently pungi only actually does some of the work involved in a compose (a lot of the images are just built by the trigger scripts firing off Koji tasks and other...stuff). The current revision of the compose process is something of a mess (it's grown chaotically as we added ARM images and Cloud images and Docker images and Atomic images and flavors and all the rest of it). With the Pungi 4 revision and associated changes to the releng process, it should be trivial to follow the compose process.

Right now, though, it isn't. Nightly composes and TC/RC composes are very different. TCs/RCs don't emit information on their progress really at all. Nightlies emit some fedmsg signals, but crucially, there's no signal when the Koji builds complete: you get a signal when they start, but not when they're done.

So it was time to teach fedfind some new tricks! I decided not to go the fedmsg route yet since it's not sufficient at present. Instead I taught it to tell if composes are complete in lower-tech ways. For the Pungi part of the process it looks for a file the script creates when it's done. For Koji tasks, it finds all the Koji tasks that looks like they're a part of this nightly, and only considers the nightly 'done' when there are at least some tasks (so it doesn't report 'done' before the process starts at all) and none of the tasks is 'open' (meaning running or not yet started).

So now we could make the openQA trigger script or the compose-check script wait for a compose to actually exist before running against it! Great. Only now I had a different problem: the openQA trigger script was set up to run for both nightlies. This is fine if it's not waiting - it just goes ahead and fires one, then the other. But how to make it work with waiting?

This one had to go through a couple of revisions. My first thought was "I have a problem. I know! I'll use threads", and we all know how that joke goes. Sure enough, all three of the revisions of this approach (using threading, multiprocessing and multiprocessing.dummy) turned out to have problems. I eventually decided it wasn't worth carrying on fighting with that, and came up with some different approaches. One is a low-tech round-robin waiting approach, where the trigger script alternates between checking for Branched and Rawhide. The other is even simpler: by just adding a few capabilities to the mode where the trigger runs on a single compose, we can simply schedule two separate runs of that mode each night, one for Rawhide, one for Branched. That keeps the code simple and means either one can get all the way through the 'find compose, schedule jobs, run jobs, run compose report' process without waiting for the other.

And that, finally, is about where we're at right now! I'm hoping one or the other openQA change will be approved on Monday and then we can have this whole process running unattended each night - which will more or less finally implement some more of the near-legendary is Rawhide broken? proposal. Up till then I'll keep running the compose reports by hand.

Along the way I did some other messing around in fedfind, mostly to do with optimizing how it does Koji queries (and fixing some bugs). For all of a day or so, it used multiprocessing to run queries in parallel; I decided multithreading just wasn't worth it for the moderate performance increase, though, so I switched to using a batched query mode provided by xmlrpclib, which speeds things up a little less but keeps the code simpler. I also implemented a query cache, and spent an entire goddamn afternoon coming up with a reasonable way to make it handle fuzzy matches (when e.g. we run a query for 'all open or successful tasks', then run a query for 'all successful live CD tasks', we can derive the results for the latter from the former and not waste time talking to the server again). But I got there in the end, I think.

It was quite a lot of work, in the end, but I'm pretty happy with the result. I'm really, really looking forward to the releng improvements, though. fedfind is more or less just the things releng is aiming to do, only implemented (unavoidably) stupidly and from the wrong end. As I understand it, releng's medium-term goals are:

all composes to contain sufficient metadata on what's actually in them
compose processes for nightlies to be the same as that for TCs/RCs
compose process to notify properly at all stages via fedmsg
ComposeDB to track what composes actually exist and where they are

right now we don't really have any of those things, and so fedfind exists to reconstruct all that information painfully, from the other end. It will definitely be a relief when we can get all that information out of sane systems, and I don't have to maintain a crazy ball of magic knowledge, Koji queries and rsync scrapes any longer. For now, though, the whole crazy ball of wax seems to actually work. I'm really glad that folks like Kevin, Dennis, Peter, Ralph, Adam and others are all thinking down the same general lines: I'm hopeful that with Pungi, ComposeDB (when it happens), and further work on Taskotron and openQA and even my stupid little scripts, we'll have continuously (see what I did there?!) better stories to tell as we move on for the next few releases.

Testdays: an app for dealing with Test Day wiki pages

2015-07-02T15:42:18Z

Just wanted to quickly announce another app in the ever-growing wikitcms/relval conglomerate: testdays. testdays is to Test Day pages as relval is to release validation pages: it's a CLI app for interacting with Test Day pages, using the python-wikitcms module.

Right now it only does one thing, really - generates statistics. You can't (yet?) use it to create Test Day pages or report results. But you can generate some fairly simple statistics about a single Test Day page or some group of them, with various ways of specifying which pages you want to operate on.

For each page it will give you a count of testers, tests, and bug references, a ratio of bugs referred per tester, and an attempt to count what percentage of valid, unique bugs referred to in the page has been fixed. If you use it on a set of pages, it'll also give you overall statistics for all the pages combined, and a list of the top testers by number of results and bug references.

I'd like to thank Chris Ward for prodding me into making this - he asked for some statistics about the Fedora 22 Test Day cycle, and I figured I may as well write a tool to produce them...

testdays is available from the wikitcms/relval repository - if you have it set up, you can do (yum|dnf) install testdays.

Post-F22 plans

2015-06-09T19:21:57Z

Hi, folks! I've been away on vacation for a few weeks, but I'm back now - if you'd been holding off bugging me about something, please commence bugging now. Of course, I have a metric assload of email backlog to dig out from under.

I'll probably have lots more on the list fairly soon, but just thought I'd kick off with a quick list of stuff I'm intending to do in the post-F22 timeframe:

Replace python-wikitcms' homegrown, regex-based mediawiki syntax parsing with mwparserfromhell (which will imply packaging it)
Improve python-wikitcms' handling of Test Days and add a new tool (a relval analog for Test Days) - initial focus on getting useful stats out, but I may set up a similar wiki template-based Test Day page creation system so you can create the initial page for a Test Day with a single command
Write openQA tests for more of the release validation test cases

I should also add better logging and unit tests to python-wikitcms, fedfind and relval, but let's be honest, I'll probably wind up doing shiny exciting things instead...

I'm also going to update the ownCloud packages to 8.0.4 (everything except EPEL 6) and 7.0.6 (EPEL 6) soon.

LinuxFest NorthWest 2015, ownCloud 8 for stable Fedora / EPEL

2015-05-01T18:09:49Z

LinuxFest NorthWest 2015

As I have for many of the last few years, I attended LinuxFest NorthWest this year. It's been fun to watch this conference grow from a couple hundred people to nearly 2,000 while retaining its community-based and casual atmosphere - I'd like to congratulate and thank the organizers and volunteers who work tirelessly on it each year (and a certain few of them for being kind enough to drive me around and entertain me on Sunday evenings!)

The Fedora booth was extra fun this year. As well as the OLPC XO systems we usually have there (which always do a great job of attracting attention), Brian Monroe set up a whole music recording system running out of a Fedora laptop, with a couple of guitars, bass, keyboard, and even a little all-in-one electronic drum...thing. He had multitrack recording via Ardour and guitar effects from Guitarix. This was a great way to show off the capabilities of Fedora Jam, and was very popular all weekend - sometimes it seemed like every third person who came by was ready to crank out a few guitar chords, and we had several bass players and drummers too. I spent a lot of time away from the booth, but even when I was there we had pretty much a full band going quite often.

It was good to meet Brian and also Pete Travis, who does fantastic things for Fedora Docs, as well as Jeff Fitzmaurice. Jeff Sandys was there as usual as well, so we got to catch up over lunch.

I didn't have a talk this year (I proposed one but it didn't make it through the voting process), so I was able to take it nice and easy and just meet up with folks and watch talks. In between all of that I also got myself 3D scanned by Dianne Mueller, who had herself set up in a trailer with a big lazy Susan and a Kinect and software I know nothing at all about which managed to produce a scarily accurate model of me and my terrible posture. She's promised I'll get a tiny plastic bust of myself in the mail sometime soon, though I'm not sure exactly what to do with it...so thanks, Dianne!

Hard to mention everyone else I ran into or met, but of course there was the (thankfully) inimitable Bryan Lunduke and James Mason of openSUSE fame, who took up their traditional spot opposite us and cried all weekend as they watched the huge crowds flock our booth...

There were a lot of really good presentations. I particularly enjoyed Frances Hocutt's A developer's-eye view of API client libraries, which sounds a little dry but was very well presented and full of good notes for API client library producers and consumers. Frances wrangles API client libraries for the Wikimedia Foundation, so it was good to get to thank her for her work on the list of Mediawiki client libraries and the Gold standard set of guidelines for Mediawiki client library designers and all the other things she's done to improve API client libraries for Mediawiki - obviously this has been invaluable to wikitcms development.

It was also great to meet Frank Karlitschek at his Crushing data silos with ownCloud talk. I've been packaging ownCloud and making small upstream contributions for a while now so I've chatted with several of the devs on IRC and GitHub - I didn't even know Frank was going to LFNW, so it was an unexpected bonus to be able to say 'hi' in real life, and inspired me to do some work on OC, of which more later!

Diane's presentation on the latest bleeding-edge bits of the OpenShift stack went partly over my head - my todo list includes items like 'learn what the hell is going on with all this cloudockershifty stuff already' - but she always presents effectively and it was interesting to learn that OpenShift 3.0 is a real production thing built on Project Atomic, which is a bit astonishing since in my head Atomic is still 'this weird experimental thing Colin Walters keeps bugging me about'. These cloud people sure do move fast. Kids these days, I don't know.

Finally it was great to see Seth Schoen present Let's Encrypt. I'd heard a little about this awesome project but it was good to get some details on exactly how it's being implemented and how it will work. Their goal is, pretty simply, to make it possible to get and install a TLS certificate that will be trusted by all clients for any web server by running a single command. They're basically automating domain validation: the server comes up with a set of actions that will demonstrate control of the domain, the script running on your web server box (the 'client' in this transaction) demonstrates the ability to make those changes, the server checks they were done and issues a certificate, the script installs it. None of it is rocket science, but it's so immeasurably superior to doing it all one awkward step at a time with openssl-generated CSRs and janky web interfaces that the only wish I have is for it to be in production already. The real goal is to enable a web where unencrypted traffic simply doesn't happen - make it sufficiently easy to get a trusted certificate that, simply, everyone does it. It was pretty cool that at the end of the talk Seth was mobbed by Red Hat / Fedora folks offering help with integration - I'm guessing you'll be able to use this stuff on RHEL and Fedora servers from day 1.

All that and we had the now-traditional Friday night games night (beer, pizza, M&Ms, and Cards Against Humanity - really all you need on a Friday night!) too. It was a very enjoyable event as always.

ownCloud 8 for Fedora 21, Fedora 20, and EPEL 7

The other big news I have: ownCloud 8.0.3 came out recently, and it seemed like an appropriate time to kick most of the still-maintained stable Fedora and EPEL releases over to it. So there are now Fedora 21, Fedora 20 and EPEL 7 testing updates providing ownCloud 8.0.3 for those releases. Please do read the update notes carefully, and back everything up before trying the update!

EPEL 6 is still on ownCloud 7.0.5 for now; I'd have to bump even more OC dependencies to new versions in order to have ownCloud 8 on EPEL 6. That might still happen, but I decided to get it done for the more up-to-date releases first.

This build also includes some fixes to the packaged nginx configuration from Igor Gnatenko, so I'd like to thank him very much for those! I still haven't got around to testing OC on nginx, but Igor has been running it, so hopefully with these changes it'll now work out of the box.

Fedora 22 Alpha, Ipsilon Test Day, openQA progress, and ownCloud status

2015-03-11T19:23:44Z

Fedora 22 Alpha

The big news this week is definitely that Fedora 22 Alpha is released! We even managed to ship this one on time, though as it's an Alpha, it is of course not bug-free. For me the nicest thing about Fedora 22 is all the improvements in GNOME 3.16, which Alpha has a pre-release of. The new notification system is a huge improvement. I'm fighting some bugs in Evolution's new composer, but mostly it's working out.

Ipsilon Test Day

We also have the next Fedora 22 Test Day coming up tomorrow, Ipsilon Test Day. This is one of those pretty specialized events that requires some interest in and probably knowledge of a specialist area; it also requires a fairly powerful test system with a decent amount of spare disk space.

Ipsilon provides SSO services together with separate identity management systems; it's being implemented into Fedora's identity/authentication system to replace Fedoauth (or this may have already happened, I'm not quite sure on the timelines!). It's one of those things where you go to some website, and you need to log in, and you get redirected through an authentication system that might be shared with lots of other websites - like when you log in to various Fedora sites through the shared Fedora authentication process.

If you're already interested in FreeIPA and might be interested in looking at web application-focused SSO on top of it, you may well be interested in this Test Day. So come join us in #fedora-test-day on Freenode IRC and help out!

openQA automated install testing for Fedora

Personally, I've been working on our openQA-based automated install testing system for the last few days.

Waaaaaaaaait, I hear you say, Fedora has an automated install testing system?

Well, it does now. We are still expecting Taskotron to be our long-term story in this area, but after the awesome Richard Brown demonstrated the feasibility of getting a quick-and-dirty Fedora deployment of openQA running, some of the Red Hat folks working on Fedora QA in our Brno office - Jan Sedlak and Josef Skladanka - threw together an implementation in literally a few days, as a stopgap measure until Taskotron is ready.

The original couple of deployments of the system are behind the Red Hat firewall, just because it works out easier for the devs that way, but I now have my own instance which is running on my own servers and is entirely public; you can look in on all my experiments there. (Please don't download any really huge files from it - all the images I'm testing are available from the public Fedora servers and mirrors, and I have limited bandwidth).

The Fedora tests and the dispatcher and other miscellaneous bits are available, and there's a deployment guide - click on InstallGuide.txt, I'm not giving a direct link because BitBucket doesn't seem to make it possible to do a direct link to the current master revision of an individual file - if you're feeling adventurous and want to set up your own deployment. We are running our instances on openSUSE at the moment, because packaging openQA for Fedora and making any necessary adjustments to its layout would sort of defeat the object of a quick-and-dirty system. And hey, openSUSE is a pretty cool distro too! We're all friends.

The first thing I did was make the system use fedfind for locating, downloading and identifying (in terms of version and payload) images. This had the benefit of making it work for builds other than Rawhide nightlies.

I have my own forks of the tests and tools where I'm keeping topic branches for the work I'm doing. Currently I'm working on the live branches, where I'm implementing testing of live images (and really just uncoupling the system from some expectations it has about only testing one image per build per arch in general). It's mostly done, and I'm hoping to get it merged in soon. After that I'm probably going to work on getting the system to report bugs when it encounters crashes. It's painful sometimes figuring out how to do stuff in perl, but mostly it's been pretty fun working with the system; you can argue that a test system based on screenshot recognition is silly, but really, just about any form of automated testing for interactive interfaces is silly in some way or another, and at least this one's out there and working.

The openQA devs have been very open to questions and suggestions, so thanks to them! I have a couple of trivial commits upstream to fix issues I noticed while running the development packages on my instance.

When you look at Fedora 22 (and later?) release validation pages, results you see from 'coconut' and 'colada' are from the openQA instances - 'coconut' results are from the deployments managed by the main devs, and 'colada' results are from my deployment.

ownCloud updates

Finally, before I dived into openQA I did find a bit of time to work on ownCloud. I got ownCloud 8 into good enough shape (I think!) to go in Rawhide and Fedora 22: it's now in Rawhide, and there's an update pending for Fedora 22.

The other distro which gets a major update is EPEL 6; there's an update pending for 7.0.4 (current stable for EPEL 6 is OC 6).

7.0.4-3 is also pending as an update for the stable Fedora releases and EPEL 7; for those releases it's a minor update which mostly tweaks the Apache HTTP configuration file handling. I came up with a new layout which, as well as making the App Store actually work, should make it easier to 'fire and forget' a typical ownCloud deployment's HTTP configuration; after you do the initial setup, you can run ln -s /etc/httpd/conf.d/owncloud-access.conf.avail /etc/httpd/conf.d/z-owncloud-access.conf to open up access from anywhere, and you won't have to manually check config files on future upgrades, because I'll keep owncloud-access.conf.avail up to date with any directory/path changes that crop up in future releases. If you have an existing ownCloud install and you'd like to 'migrate', I recommend:

Move any customizations you have to the HTTP configuration into an override file, e.g. /etc/httpd/conf.d/z-owncloud-local.conf
Restore the clean packaged copy of /etc/httpd/conf.d/owncloud.conf
Run ln -s /etc/httpd/conf.d/owncloud-access.conf.avail /etc/httpd/conf.d/z-owncloud-access.conf (assuming you want to allow access from any host!)
Never edit any of the packaged files in future (including the .inc and .avail file(s)), always do customizations in override files

Happy Clouding!