Yeah, in a way it’s anaconda test day every day in Fedoraland, as we focus a lot of our day-to-day testing on the installer. But the folks running this Test Day have come up with some good extended test cases that go beyond what we cover in the release validation testing, so we should be able to find some more interesting bugs (it’s always fun).

We’ll be testing the current Beta candidate build, Beta RC2. As always, full instructions for testing are available in the Test Day page, and there will be QA folks on hand in #fedora-test-day on Freenode IRC for discussion and any help you need in testing or debugging. If you’re not sure what IRC is or how to use it, we have instructions here, and you can also simply click here to join the chat through a Web front end.

If you haven’t tried out the improved anaconda in Fedora 19 yet, this event is a great excuse to take a look at it: it is much more polished and stable now than it was in Fedora 18, so come try it for yourself and help us to improve it further before the final release of Fedora 19!

I’ve been really enjoying work on Fedora 19 lately. After the somewhat hellish Fedora 18 cycle where we worked more or less 24×7 on stabilising anaconda, Fedora 19 has been a pretty well-behaved release so far, and so I’ve had time to look around and try to investigate and fix up some other things. Among other things, today, I was looking into a couple of bugs which turned out to be very interesting (to weird people like me, anyhow) to debug. I don’t know if anyone else feels like this, but I absolutely love this stuff – it’s my favourite thing about my job, really, the process of pinning down exactly why something is going wrong, and how to fix it. I also love that in a project like Fedora, you rarely do it alone: it’s very cool to work alongside (or just watch) a big group of extremely smart and knowledgeable people, both inside RH and in the Fedora community. I just had the urge to do a post mortem on a specific bug here, because I really love this process. Those of you who do this every day can safely move on.

Since upgrading to Fedora 19, I hit a strange problem three or four times where there was a long delay when using the ‘su’ command to become root. I’d run ‘su’, enter my password, and then it would sit there apparently thinking about it for 15-20 seconds before returning a root prompt; it’s usually instant. This was different from the delay that usually happens when you get the password wrong – it happens when you get the password right, and is longer than that delay.

So when I finally got around to trying to figure out what was going on recently, I started out by trying to ‘strace’ the su process.

Now, strace is rarely the tool you really want to use for debugging. It’s a very unwieldy thing to use, for most cases. There’s usually a better way of doing it. But it can be invaluable for your real mystery bugs, like this one, where there is no other obvious way to attack the problem – no logs to sink your teeth into or obvious triage techniques you can use. It turned out to be invaluable for this bug.

At first I was stymied because if you just run the su process through strace from the outset, authentication always fails – strace interferes with it somehow. Tomas Mraz pointed out the workaround which really should’ve been obvious to me in hindsight: just run su first, then attach the strace to it (the ‘strace -p [pid]‘ command does this). That works fine. So, now I had the strace output, and it had a reasonably obvious smoking gun:


stat("/root/.xauth/import", 0x7fff660f8b70) = -1 ENOENT (No such file or directory)
getgid() = 1001
getuid() = 1001
pipe([4, 6]) = 0
pipe([7, 8]) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f2a174b9a10) = 15192
close(4) = 0
close(8) = 0
close(6) = 0
read(7,

"", 2048) = 0
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=15192, si_status=1, si_utime=0, si_stime=0} ---
close(7)


The pause happened at “PAUSE HERE”: the strace output sat right in the middle of that line for 15 seconds before moving on. So what does that bunch of gibberish mean?

Extracting meaning from voluminous gibberish is an invaluable skill for the kind of stuff we’re working on here, which you only really develop with experience. But in this particular case, here’s how that stuff reads to me:

blahblahblahwordswordswords CHILD CHILD SIGCHLD 15192 blahblahblahwords…pause…SIGCHLD CLD_EXITED wordswordswords

The key thing to know is that, just before the pause, the su process created a child process. Basically, it ran some other command that it needed to run. This happens all the time: say you were writing a very unsophisticated file manager, if the user clicked the Create Directory button, your program would run a child ‘mkdir’ process. (That’s a horrible way to write a file manager, but you get the idea).

So what this means is that we’re not going to get any further staring at this strace, because the data just isn’t there. The pause happens right after a child process is forked off, and right when the pause ends, we see that the child process finished. That makes it pretty damn obvious that the pause is happening in the child process, somehow.

So what next? Well, we find out what the child process is. That ’15192′ is the pid of the child process. How did I know that? I didn’t (though a real developer probably would have), but it seemed like a reasonable guess – it’s a pid-ish number, it’s clearly associated with the creation of the child process, and if I were the developer of strace, then when a child process was created, I’d bloody well make sure it logged the pid. So it seemed a pretty safe bet, and it was easy enough to check: re-do the strace and check ‘ps’ output for the apparent pid while the pause was happening. And lo, it came to pass that I was right. So ps told me that the child process was this:

/usr/bin/xauth -f /run/gdm/auth-for-adamw-szT11D/database nlist :0

it’s trying to read some kind of authentication information from a database provided by gdm. Okay. Now we’re making progress.

The next step is probably pretty obvious: let’s strace the child process. It’s easy enough to watch the su strace output and fire off an ‘strace -p 15192′ (or whatever the new number was) right when the pause happens.

That yielded some nice gratifying results. The xauth strace showed this happening, seven times:

open(“/run/gdm/auth-for-adamw-szT11D/database-c”, O_WRONLY|O_CREAT|O_EXCL, 0600) = -1 EACCES (Permission denied)
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
nanosleep({2, 0}, 0x7fff8b56d5c0) = 0

it tries to access (actually, create) a file, gets ‘Permission denied’, WORDSWORDSWORDS, then goes away to sulk in the corner for a bit: after that ‘nanosleep’ line, there was a two second pause. Multiply that by 7 (actually I think it’s probably 8, and the first happened before I got the strace started) and you get the long pause. Eureka!

Of course, we’re not done yet. When you run up against something like this, there are two key questions:

a) Why is it trying to do that in the first place?
b) Why is it failing?

a) can sometimes be an interesting question, but then, I always find it’s a good idea to consider a third question: ‘do the people who did this know better than I do?’, to which the answer is often ‘yes’. I’m certainly not going to start questioning the motives of the people who maintain util-linux unless it seems to be urgently necessary. So in a case like this, it’s probably prudent to assume that there’s a good and valid answer to a), and slide right along to b).

At this point it was necessary to call in the cavalry again, what with me still being the idiot monkey and all. So I got on IRC and bleated piteously for halfline – Ray Strode – the maintainer of GDM. He did most of the work from there on; I just sat around coming up with silly ideas.

It was still a pretty wide open problem area to start with; there are always multiple possibilities when you’re looking at a problem like this. We had to consider, for instance, the possibility that this file creation always failed, but didn’t always cause a delay. Or that the attempt to create the file itself isn’t a ‘normal’ thing but some kind of failure path; perhaps in the ‘success’ case, the file creation isn’t attempted. So it seemed prudent to run an strace on the ‘success’ case as well as the failure case. ajax – Adam Jackson – had a handy trick for achieving this. In the ‘success’ case the xauth command returns pretty much instantaneously, so you can’t catch it in strace just by being handy with the copy/paste shortcuts. Noo, we have to be smarter than that. So, we do this:

cp /usr/bin/xauth /usr/bin/xauth-real
cat > /usr/bin/xauth << EOF
strace -o /tmp/$$.strace /usr/bin/xauth-real "$@"
EOF

What we did there was replace the xauth binary with a wrapper script that runs the 'real' xauth (renamed to xauth-real) through strace. (The cat > << EOF...EOF stuff is just a way to create a file with any contents you like; you could just as easily create a file in gedit, and type in that strace line to it). The "$$" means "the pid of the process", so you wind up with /tmp/18104.strace or whatever.

After that nifty trick we run our 'su', and take a look at the strace. It shows that the creation of the file is indeed attempted in the success case, and works immediately. So as it turns out, this isn't a curveball: the problem really is that xauth should be able to create that file, but it's being denied permission. Still - always check for curveballs. Don’t get blinkers on when you’ve caught a hint of what the problem is: at every stage of debugging, try and consider all the possibilities.

With that knowledge we could focus in a little bit. After looking at various things we worked our way around to what turned out to be the problem: on the system where I was seeing the bug, /run/gdm/ had the permissions 1770 (and ownership root.gdm). Well, that’s clear enough.

We still weren’t done, though! The question is always why: why did the directory have those permissions? Remember, the bug isn’t permanent. Checking a normally-working system, the permissions were 0711, which is what halfline expected to see. Since I didn’t see the bug all the time, it looked like the permissions were usually right, but somehow, sometimes, they got set to the wrong thing. So what would be changing the permissions in the middle of a running session?

Tracking that down was a bit tricky. We poked through the gdm codebase looking for calls to chmod or other functions for setting permissions and couldn’t find anything. Finally, though, when we were getting pretty stumped, halfline thought to check the gdm package spec file. Eureka once again:

%attr(1770, root, gdm) %dir /run/gdm

And there, the prosecution rests: we have our culprit. On package installation, the permission on /run/gdm were set wrong. As we’d discovered while poking the gdm codebase, gdm has code to set the correct permissions – 0711 – at runtime, so most of the time, that was disguising this bug; when I booted up my system, gdm would fix the incorrect permissions. But what would trigger the bug would be any update to the ‘gdm’ package that I installed on the fly, without rebooting or restarting GNOME: the package install would set the permissions back to 1770, and boom, bug. A reboot would trigger gdm fixing the permissions and the bug went away again.

I don’t know, maybe I’m just weird, but I enjoy the whole process of narrowing something like that down, and it’s a great feeling when you get to the end and you’re satisfied that you know exactly what broke and how to fix it. Even if someone else wound up doing most of the work! Thanks much to Ray, Adam and Tomas for that.

I actually intended to also work through in similar fashion the other interesting thing I figured out today, which is why most of anaconda’s icons are missing when you run it from the new MATE Fedora spin, but this has gotten very long, so I guess I’ll save that for another time! Thanks for reading, if you got this far.

The awesome Chris Lumens made this much easier back in 2011 by adding memory use logging functionality to anaconda. He wrote up his findings at the time – based on a Fedora 15 install – in this blog post.

I ran a very similar test to Chris’ using F19 Beta TC3. We both did a default package set install from a DVD image. Only difference is that I did not switch language to German in the anaconda UI, as Chris did. Here’s my graph:

As Chris did, I’ve annotated the spikes (although much more messily than he did).

So, what do we see? It’s a bit hard to see in this graph as I blew through the install process very quickly, but I just ran another test, and if I complete the necessary spokes and leave the UI sitting at the hub screen before actually kicking off an install, the ‘background level’ of anaconda itself is 329MB. In Chris’ graph, this ‘background level’ before package installation kicks in looks like it’s around 200MB. However, oldUI was very different from newUI, and most of the difference there may be accounted for by the fact that we’ve already fired up storage and repository configuration code at this point.

The ‘background level’ during package installation – where the line sort of mostly sits aside from spikes – is around 590MB in my data. In Chris’ F15 test, it looks like it was around 520MB-530MB (eyeballing it). So usage has increased about 60-70MB there.

The spikes are as follows:

A – sepolgen-ifgen running during install of policycoreutils-devel package (spikes to 796MB)
B and C – both gtk-update-icon-cache, running during %posttrans I believe. See icon cache packaging guidelines.
D – this whole ramp is yum’s “VerifyTransaction” action, which is new since Fedora 15 (I forget which release introduced it)

The shorter unlabelled spikes I haven’t looked into, but they’re probably update-mime-databases, like the similar small spikes in Chris’ F15 run.

It’s rather odd that gtk-update-icon-cache runs multiple times in %posttrans, the policy seems designed to prevent this happening. From the raw data it looks like it runs three times, once at 08:29-31, once at 08:36-38, and once at 09:17. The memory usage spikes in all three cases. edit: Thanks to kalev who came up with a nifty bit of RPM magic to inspect all scripts that do gtk-update-icon-cache, it looks like this makes sense: there are three icon themes of significant size that are updated in %posttrans of various packages, hicolor, gnome and HighContrast. So the three spikes associated with gtk-update-icon-cache runs make sense.

So it looks like since F15, the general ‘background’ usage of memory during an anaconda run has increased modestly, and the spikes in usage have changed somewhat.

The good news is that the highest spike in a default F19 run is lower than the highest spike in a default F15 run; Chris’ graph spikes clear over 800MB, while mine just touches it. The big spike in F15 was “from the selinux-policy-targeted package’s %post running semodule”; that’s gone completely in F19, there is no spike at all associated with selinux-policy-targeted. However, the biggest spike in F19 appears to be a new one: it is a different SELinux spike, this time sepolgen-ifgen running during install of policycoreutils-devel. It is not quite as big as the F15 spike, but obviously something we should try and optimize if possible.

gtk-update-icon-cache caused spikes both times. However, during F15 it ran only once (according to Chris), and was a somewhat smaller spike. It’s labelled K in Chris’ graph, and is a spike of 70-80MB, to 600MB total. In F19Btc3, the spike is of 110MB, to 700MB total.

The ‘depmod’ spike from Chris’ run seems to have disappeared now.

Finally, the ‘ramp’ associated with the new VerifyTransaction step in yum is a new issue, and possibly worrying: I haven’t tested with a larger package set, but this test indicates that the memory usage never seems to level off, which would indicate that memory usage during VerifyTransaction is directly proportionate to the number of packages being installed. So at least theoretically, if you did a really big package set install, the memory usage during VerifyTransaction might be the highest of all, and could go quite high. I’ll try doing an ‘as much stuff as possible’ install and see how high it gets.

So we have a mixed picture from this small comparison; in fact the highest memory usage of a stock DVD install seems to have gone down slightly since F15, but there’s clearly room for improvement, and there may be issues with larger package sets that we didn’t have in F15.

ABRT is the Fedora tool for catching and reporting crashes. If you’ve been running Fedora 19, or you’ve updated with updates-testing in Fedora 18 in the last few days, you may have noticed some major changes to ABRT and libreport, including a completely new graphical tool for reporting crashes called gnome-abrt. We’ll be testing out these big changes at the ABRT Test Day. ABRT gets better every Fedora release, but the more broad-based testing we get the more issues we can squish, so please, come along and help us test!

The SSSD improvements and Active Directory integration Test Day will focus on Fedora 19 enhancements to our enterprise authentication tools. In particular, we’ll be testing integrating Fedora 19 systems into Active Directory domains. This probably won’t be of interest to some of you, but if you use or even help to admin a FreeIPA or AD shop, you might well want to come along and help check if we have things working properly for your deployment.

As always, full instructions for taking part in each Test Day are available on the Wiki page, and we’ll be making live images available so you can do as much of the testing as possible without needing to install a pre-release Fedora. QA and development folks will be present in the #fedora-test-day channel on Freenode IRC for discussion and any help you might need in testing. If you’re not sure what IRC is or how to use it, we have instructions here, and you can also simply click here to join the chat through a Web front end.

Thanks to all in advance!

I have come across some truly crazy bugs in my time – buy me a beer at a conference and I’ll tell you about them – but I don’t think anything tops Trey Harris’ Case of the 500 Mile Email. That’s amazing.

Bryan Lunduke does a recurring talk there which is called “Why Linux Sucks” or something like it. It’s a great talk, where Bryan gives his tongue-in-cheek opinion on what’s holding back desktop (and, more recently, mobile) Linux adoption. There’s a video of this year’s talk.

Bryan’s awesome, and we get along great. I love his talks (in the last couple of years he’s started following the “Why Linux Sucks” talk with a “Why Linux Rocks” talk which uses almost the exact same slide deck, which is an awesome idea). But I think he’s wrong on one point he brings up. It’s hard to do a good job of explaining why concisely during the talk – you can watch me doing a bad job of it in the video! – because it’s a complex issue that’s hard to distil. So I thought I’d write it down instead.

The problem Bryan identifies affects third parties providing Linux applications directly to users: Bryan trying to provide his games to users of different distributions, or Google trying to provide Chrome, or Mozilla trying to provide Firefox, and so on and so forth.

If you’re one of those third parties, you can put your application in a package, or you can put it in a tarball.

If you put it in a package and provide a repository, your users can take advantage of the features of package management. Your app will show up in their package management applications, it will be updated when they do a system update, and a few other things. But there’s a problem: different distributions use different package formats. So if you just do one package (and repository), you won’t be covering all your potential users. If you do RPM and DEB you’ll cover 90+% of them, but there are still a few who’ll be left out.

If you put it in a tarball, you can just make one, and give it to users of all distributions. They’ll all be able to extract your app and run it. But it won’t show up in their package manager. You’ll have to tell them to download a new tarball, or write your own update mechanism.

Bryan wants for third parties to be able to get all the neat stuff that comes with using a packaging format, without the inconvenience of doing multiple builds.

If distributions all used the same packaging format, would Bryan’s goal be achieved? Absolutely. Is it a worthwhile goal? Sure. I don’t think it’d change the world, but we agree it’d be a benefit.

So Bryan’s right? I’m an idiot? Well…not so fast there!

Switching package formats for a distribution is a huge pile of work. I can explain why if anyone needs me to, but for now I’m going to assume you’re willing to take it on trust.

If all distributions used the same package format, that wouldn’t help them directly at all. A lot of people believe that if, say, Fedora, OpenSUSE and Ubuntu all used the same package format, you could just mix and match packages between them, and each distro wouldn’t have to package everything separately.

This is not true, we can’t ever make it true, and we don’t want it to be true. I’ll try to explain why as quickly as I can: distributions are very different, and they all are strongly committed to using shared resources.

When you get Chrome from Google, you get a static build: it uses very little of the shared resources the distribution provides, stuff that’s present and compatible on virtually any distro.

When you download a package from your distro vendor, it’s not like that. Distros use dynamic builds: distro packages use all the shared resources they can. Just take it on trust that the benefits are huge and distributions will never start using static linking outside of the cases where it’s unavoidable (ask me for details if you want). Distros differ in terms of the shared resources they include, and this isn’t just being ornery, it’s a key part of what differentiates distributions from each other, and why we have them at all. Fedora 18 has openssl 1.0.1e. Ubuntu has 1.0.1c. RHEL 6 has 1.0.0. Debian stable has 0.9.8o. This is because Fedora is bleeding edge, Ubuntu is a bit more conservative, RHEL 6 is a lot more conservative, and Debian stable is even more so. We couldn’t all get together and ‘agree to compromise on 1.0.0′. Then none of those distributions except RHEL would be serving the needs of its users.

Multiply that by the thousands of packages in a distribution. You can’t take a Fedora 18 package and install it on RHEL 6, even though they are both RPM distributions, because the package expects stuff from Fedora 18 that just isn’t in RHEL 6.

So we’ve got a cost/benefit. The benefit? It’s there. The cost? It’s huge, and it falls on groups that don’t directly benefit at all. This sucks. But fortunately, we don’t need to pay that cost!

We don’t need all the distributions to agree on a common package format for their packages so third parties can provide applications with most of the benefits of package management. All we need is for there to be a package management framework that third parties can rely on to be present on all distributions. It doesn’t matter at all whether the distribution packages use it or not.

That’s a much easier problem to solve. All we’d need to do is agree to all provide support for one existing package format out of the box. (Sidebar: LSB already actually tries this. It requires RPM.) That’s one big bunfight at a conference and maybe like a week or so of work. No big deal. If Bryan pushed for this instead of saying ‘all distributions should use the same package format!’, I’d have no disagreement with him.

You can stop reading here if you like! But if you’re thinking ‘hey, waitaminute’…

To pre-emptively address one objection: what if a third party wants to provide a package that depends on something in the core distribution?

Bryan’s idea would go a bit further to addressing that than mine would. But it’s not so important, Bryan’s idea still doesn’t totally solve it, and you can modify mine to solve it quite easily.

It’s not so important because most significant third party providers just want to make their package as independent as possible. They don’t want to worry about making sure their app works with all the possible versions of the shared resources they want to use that are shipped by the various distributions. They usually just link almost everything statically. Here are the requirements of Google’s Chrome RPM, for instance:

libatk-1.0.so.0()(64bit)
libc.so.6(GLIBC_2.11)(64bit)
libcurl.so.4()(64bit)
libgconf-2.so.4()(64bit)
libnss3.so(NSS_3.12.3)(64bit)
libbz2.so.1()(64bit)
libXss.so.1()(64bit)
libXcomposite.so.1()(64bit)
libXfixes.so.3()(64bit)
wget
xdg-utils
zlib

Honestly, you really don’t need to specify any of those. If someone’s running a Linux distribution with a desktop (which they are if they want to install Chrome), it’s about 99 to 1 they have all of that installed.

Bryan’s idea doesn’t fully address this requirement because distributions don’t all agree on what different packages should be called, or how library dependencies should be written, etc etc etc. It’s slightly more likely that we could all agree on that than that we could all switch to one package format, but it’s still a hell of a long shot. Even if we all used the same package format, you couldn’t just write “Requires: foo” and be done. One of the distros probably calls it “libfoo”. You’d still have problems.

And you could pretty easily achieve this with my idea, too. Better, we don’t have to do it all at once. We could just get the format in place first. If that was a success, then ‘I want to be able to specify some dependencies!’ is an obvious enhancement request, and now we’re building on an existing idea, not just throwing impractical proposals around.

Maybe we can’t get all the distros to agree on the names of absolutely all their packages, but we could get them all to agree to have their GTK+ 3.0 package provide gtk+-3.0 under the shared packaging system. The names could still differ under the native packaging systems – it wouldn’t matter. But it’s plausible to see us agreeing on a set of commonly-required core components, and doing the work to have the distro packages express those provides in the shared package manager as well as their native package manager, using an agreed nomenclature. And it could be incremental – we could just start by doing 50 packages, or 10, or just one; however many we could agree on. And then build it out as we went along. And none of this would in any way disturb the functioning of the distro’s native package system and repositories.

Of course, being at LFNW means yesterday I travelled to LFNW – it’s just a train ride down from Vancouver. Being a giant idiot, I decided that ‘five hours before the train to LFNW leaves’ would be a great time to upgrade the CPU and memory in my husband’s system. Yeah…turns out, not so much. Windows managed once more to stun me with its incompetence: after an hour of hardware wrangling I had everything hooked up and booting, but Windows was just stuck in a boot loop. It turns out that for some insane reason, a Windows install is kind of tied to the CPU: if you upgrade the CPU to one that’s sufficiently different (though still the same architecture!), Windows stops booting.

I mean….why. WHY?!

It is apparently possible to escape from this situation, but it was now 30 minutes before the train to LFNW left, so I thought it was possibly not the best moment to be figuring out Windows’ system recovery options. So followed a very hasty 30 minutes of further hardware wrangling, getting the old CPU and RAM back in. So I wasted an entire stressful afternoon to ultimately put everything back exactly the way it was before and rush to catch my train. D’oh.

Tomorrow, Tuesday 2013-04-23, is Intel graphics Test Day. Wednesday 2013-04-24 will be Nouveau Test Day. And Thursday 2013-04-25 will be Radeon Test Day.

As always, we’ll be looking to test out the widest possible range of hardware and see how well it works with the very up-to-date graphics stacks in Fedora 19. As Fedora uses very recent builds of the relevant components and sends all its work upstream, contributing to these Test Days can help out all other distributions, not just Fedora – so please, even if you’re not a Fedora user, consider coming and contributing your testing! We provide comprehensive instructions and live images for testing, so you won’t need to replace your current distribution or do a permanent installation of Fedora at all if you don’t want to. You can easily write a Fedora live image to a USB stick, so you don’t even have to waste a DVD.

We always want to get as much data as we can in these events, so please, if you have a few minutes, help us out and perform the tests for your system(s). If you can’t make the correct date for your hardware, no problems – you can file results early or late and we’ll still be able to use them. It’s also fine to come to the IRC channel on the ‘wrong day’ and ask questions – we’ll have folks in the channel all week who will answer your questions if they can. The testing is very easy, and if you don’t have time to run through all the test cases, partial results are still very useful – if all you have time to do is boot the live image and check whether the desktop appears on your system, even that is useful.

As always, the full instructions and live images are on the Wiki pages: Intel, Nouveau, Radeon. Fedora QA team members and graphics developers will be hanging out in the #fedora-test-day channel on Freenode IRC to help out with testing and debugging any problems you come across, so please come join us there if you’re taking part! If you’re not sure what IRC is or how to use it, we have instructions here, and you can also simply click here to join the channel through a Web front end – all you need to know is that IRC is a chat system.

Sadly, we had our first Fedora 19 delay last week. I actually was trying pretty hard to get this to be the first release where we didn’t slip at all, but alas, this was not to be.

The good news is this is nothing like the Fedora 18 situation. In F18, when we hit the first go/no-go for Alpha we were barely within shouting distance of having anything releasable, and Beta was worse: we didn’t just have a few stray bugs, we were still writing bits of the installer.

What’s holding up Fedora 19 Alpha, in contrast, is two bugs in UEFI installation, and that’s it. (Note for the haters: none of the bugs has anything to do with Secure Boot). The installer is in fine shape, except for an issue in the custom partitioning screen which we’ll try and slip a fix in for. All the code that was meant to be written by now is actually written, it’s all working pretty well, and most of the functionality of the installer is pretty solid. There have been a ton of UI improvements since Fedora 18 based on both online feedback and real-world usability testing and observation, as well.

So it sucks that we had to slip, but it’s a much different situation from Fedora 18, and it’s been a lot lower stress – we’re not running around trying to keep tabs on 15 bugs and 5 features that aren’t written yet, right now we’re really just waiting on upstream review of a patch for the last UEFI issue (the fix will need to go into the upstream kernel, and we won’t put it in Fedora until it’s in an approvable state for upstream merging).

We made some fairly significant changes to the release validation process prior to the Fedora 19 cycle, and we’ve been happy with how they’re working out so far. At FUDCon Lawrence, the QA members present came up with a plan to revise the “nice-to-have” process by which we track fixes that we want to take through the Alpha, Beta and Final freezes. That proposal was sent to the mailing list, where in turn, a lot of the group members who weren’t at FUDCon contributed improvements to it. We rolled out the changes to what is now called the freeze exception process – because, you know, it’s for freeze exceptions! – ahead of the F19 Alpha validation process, and it’s been working out well so far. It really boils down to renaming “nice-to-have” to “freeze exception”, but there are some devils in the details, and it makes what was a somewhat poorly-understood process much more understandable.

I drafted up some substantial revisions to the layout and content of the release criteria just ahead of the Alpha release. The changes have been put into practice for the Alpha release criteria (compare with the F18 Alpha criteria to see the changes), and will go into place for the Beta and Final criteria before we reach those milestones. The idea is to have a shorter ‘main’ text for each criterion with details and ‘commentary’ as expandable extra paragraphs, and to lay out the criteria better so it’s easier to read the page and to refer to specific criteria. So far the changes seem to have been integrated into the process smoothly enough.

Finally, we made some tweaks to the blocker bug process and blocker review meeting process to try and expedite blocker (and freeze exception) review a bit. We introduced the concept of automatic blockers to try and cut down a bit on unnecessary review of issues that are very obviously blockers, and tried to set things up so blocker review meetings don’t run on forever. Again, those changes seem to be working out well.

So I’m feeling good about the Fedora 19 cycle so far! I’m hopeful there’ll be very few (or no) further delays after this first one, and we’ll wind up with a very solid release, with significant improvements to the new installer.