wikitcms and ownCloud stuff

For my salary this week I mostly worked on relval/wikitcms - I shipped 1.7 yesterday which is a major rewrite, on top of recent updates which added support for nightly build testing. relval nightly can create the validation pages for nightly composes, relval report-results can report results in them, and relval testcase-stats and relval user-stats can handle them in statistics generation.

Of course there was some work on the magic wiki templates to go along with that, including some little finicky details like getting valid download URLs for the boot.iso images for Rawhide and Branched nightly composes. It all seems to work pretty well, and we've 'nominated' two Fedora 22 nightly composes for testing so far. The nightly currently being tested is 2014-12-16 - here's the results summary page, and I'm hosting hourly-updated testcase-stats info.

(holy crap, I've got the Maple Leafs / Flyers game going on TV while I write this post - I've missed six goals since I started writing it and it's only half way through the first period...)

Today I spent a bit of time working on my packages, mainly ownCloud. Well, I updated roundcubemail to 1.0.4 and sent out updates for that, too. For ownCloud I backported the upstream updates to its Google Drive support (which, er, I wrote...) which make it compatible with v1.x of Google's API library, updated the client library package to 1.0.6, and dropped the ownCloud package's bundled copy of the 0.6 version of the library, which I've been working on forever - good to finally get it done.

Of course, in the mean time the library went up to 1.1.2 and added an autoloader - unfortunately, not in a way that's very consumable for downstreams. So I sent a pull request to improve that, and I've done a scratch build of 1.1.2 with that patch and verified ownCloud works fine with it. I sent a PR for ownCloud to update its bundled copy to 1.1.2.

If anyone who's using this stuff can test the updates:

that'd be a great help.

Fedora 21 greatest hits: non-Server non-live installs, fedup product behaviour

Hi, Fedora-ites!

So here's a couple of things I've seen popping up multiple times with the Fedora 21 release. I thought I'd note them down here for my readers, Planet Fedora, and also as a handy link target for answering them in future.

Traditional installs - how do I install something other than Server? Where's the network install bits? Where's the DVD?!

So, the answer to all of these is both simple and kinda dumb: use Server.

Trying to do a network install of something other than Server? Use the Server network install image (yup, really). Trying to do a kickstart install? Use the Server network install image or tree (doesn't matter what package set you want to install). Trying to do a direct kernel boot of the installer (e.g. for PXE)? Use the Server/ tree on the mirrors as your inst.stage2 parameter: inst.stage2=https://download.fedoraproject.org/pub/fedora/linux/releases/21/Server/x86_64/os/, for e.g. (If you use inst.repo instead of inst.stage2 it will restrict you to the Server package set.)

The Server network install image is really, practically speaking, a generic network install image. It's not much different from the netinst ISO in previous releases. It has Server visual branding and it defaults to the Server package set, but that's really all that's Server-y about it. It can happily install any package set, even with its default repository configuration. You can pick any of the other package groups graphically, and your kickstarts will work as before. Basically, for anything you would previously have used Fedora-netinst for, you can use Fedora-Server-netinst for instead. It'll be fine.

For bonus points, though it's hidden from the user quite well, if you poke around behind the scenes, you'd find that fedup also gets its upgrade.img (which is a custom initramfs containing all the bits to make fedup's upgrade step work) from the Server tree.

The reasons why things are this way for Fedora 21 are extremely long-winded and boring - if you really want the story, poke me or Dennis Gilmore or Matthew Miller on IRC and we can probably explain. For Fedora 22 onwards, I very much hope this will be cleaned up somehow (probably by reviving the Fedora/ tree just for building the generic installer images, or by making it possible to build images out of the Everything/ tree, which isn't currently possible.)

The Fedora DVD, though, really is gone with Fedora 21. There's no direct replacement. The Server DVD is the same type of image, but doesn't contain a cross-section of package sets like the old DVD did, it really is strictly a Server product - it only contains the Server packages. You can use the Server netinst to do generic non-live deployments with a network connection (see above), and you can use the various live images to do offline live deployments with various package sets, but there is no way to do an offline non-live deployment of anything but Server. Sorry about that, but something had to give in the whole Product plan.

What does fedup --product really do?

This is one I'm not sure we're entirely clearly communicating, so here goes:

When you run fedup --product foo, as part of the upgrade, fedup will try and install the foo-product-environment package group as well as upgrading your existing installed packages.

In practice this only affects Server and Workstation, as they're the only products which have those package groups. If you pick nonproduct or cloud, no additional packages will be installed for Product purposes as part of the upgrade (new packages might still come in as deps or as members of installed package groups).

But I suspect people might be thinking 'if I'm running anything that's vaguely like a desktop I should pick workstation' and 'if I'm running anything that's vaguely like a server I should pick server', and it's probably not quite that straightforward.

If you're running a desktop other than GNOME and you don't want GNOME, then don't go for --product workstation. Go for --product nonproduct. If your existing system has GNOME, --product workstation probably makes sense, but be aware it might install a few additional packages.

If you're running a server-ish system, you may want --product server, but then again, you may not. The Server product isn't strictly minimal, it's aimed at providing a sort of 'comfortable' default environment. Particularly, it includes the new rolekit bits, Cockpit, and various other bits. You can see what it includes right from the source - where you see the <environment> with <id>server-product-environment</id>, all the groups in the <grouplist> entry a few lines down are what get installed as part of server-product-environment. You can see what packages are in those groups in the same file - just look for their <group> entries, e.g. the <group> with <id>container-management</id> will pull in package docker-io. Note that packages listed as optional won't be included.

So if you've got a fairly task-focused server install and you're really happy with its configuration and you want to update it to Fedora 21-level packages but you don't want the new shiny bits of the Fedora Server product added on, you will want to pass --product nonproduct to fedup, not --product server.

There's also another rather large gotcha with upgrades to Server and Workstation: they will replace any existing firewall configuration with their default.

Downtime today: upgrade day

Just a heads-up there'll probably be downtime on this site (and other happyassassin servers, though that mostly matters only to me :>) today - it's upgrade day! Everything's going to F21 (except my desktop, which is going to F22). If you hear explosions from Canada, you'll know why.

Fedora 21 comes out tomorrow, Fedora 22 testing: yup, we're on it

Fedora 21 will be released tomorrow - warm up your downloader! You can read all about what's coming in the release announcement, and you'll be able to grab it from the brand new download page. Overall, I'm pretty impressed with what all the great Fedora contributors have managed to pull off for the Fedora 21 release. The new site is great, the Product split has been implemented pretty well, the new bits in Cloud and Server products are really cool (especially Cockpit!), and the Workstation release should be the best GNOME 3 distribution yet.

But all that's old hat, now - we signed off on it on Thursday, and I mean, it's been like four whole days since then. So we in Fedora QA, being the cutting-edge folks we are, are already moving on to Fedora 22 planning - and, yes, testing.

We had the first QA meeting of the Fedora 22 cycle today, and did some really good planning for improvements to our process. I hunkered down and did some work on the mediawiki template magic and on wikitcms and relval. And now we have...the first Fedora 22 validation testing!

There's a few more little bits to write, but throughout Fedora 22, every few days a new nightly compose will be 'nominated for testing' and we'll have results pages available. The testcase-stats page for Fedora 22 will shortly be live, and it'll give us a nice overview of what we've covered in Fedora 22 testing so far.

The goal here is to try and provide earlier testing to the development teams, and particularly the Anaconda team. We're also looking at ways to streamline the release blocker process, so that ultimately blocker bugs will be found, reported, nominated and reviewed as early as possible, to give developers more time to fix them. We already have (as I write this) three proposed blockers for Fedora 22 Alpha.

Roshi and pschindl will be working to co-ordinate the Test Days for the Fedora 22 cycle, so look out for a call for Test Days soonish.

Of course, there's still testing to be done on Fedora 21 (and Fedora 20 and, for the next month, Fedora 19) - we won't be neglecting work on reviewing test updates, and the more people helping out with that, the better!

Onwards and upwards, to ever newer and more exciting bugs...

Fedora 21 undergoing final testing

We're now hopefully in the final straight of Fedora 21 testing! We're actually relatively on top of the schedule this time around, as compared to previous releases - by Thursday we'll have had nearly 48 hours to test the current (and hopefully final!) package set (across the RC2 and RC4 composes).

If you want to chip in and help out with validation testing, all the info you need is in the RC4 testing availability announcement. Please do be aware that TC and RC Fedora builds are strictly release validation testing builds, these aren't public pre-releases, so don't ask why they're not on the mirrors or why the announcements aren't nice and PR-friendly, this is part of the Fedora testing process :) If you just want to use Fedora 21, sit tight and wait - right now signs look good that it'll be released on Tuesday, as per the planned schedule.

There's a couple of bugs I wish we'd had time to fix and a few rough edges left on the Product stuff which in an ideal world we'd have had time to smooth out, but I think 21 is a really nice release in general and a pretty solid first attempt at the Product work overall. We're looking forward to getting it out there!

It's a good thing they pay me to break stuff

Had an interesting couple of days deep-diving into juicy issues.

Yesterday's was triggered by me messing up the Fedora kernel package git repository - whoops. I keep Fedlet's kernel as a branch that only exists in my checkout; it's not pushed anywhere (I should just push it out on my git server now I have one, but I keep forgetting). I accidentally ran git push from that branch yesterday, and it promptly pushed all the changes on it to master, effectively turning Fedora's kernel into the Fedlet kernel for a few glorious hours until Josh reverted it.

Obviously at least 50% of the blame there lies squarely on me for having set an upstream for my branch and running git push from it at all, but it seemed odd to me that git would run such a push in a fairly 'default' configuration. So I did some digging.

It turns out it's not meant to. I don't have git's push.default setting set locally. So every time I push anything anywhere, I see this message:

warning: push.default is unset; its implicit value has changed in
Git 2.0 from 'matching' to 'simple'. To squelch this message
and maintain the traditional behavior, use:

  git config --global push.default matching

To squelch this message and adopt the new behavior now, use:

  git config --global push.default simple

When push.default is set to 'matching', git will push local branches
to the remote branches that already exist with the same name.

Since Git 2.0, Git defaults to the more conservative 'simple'
behavior, which only pushes the current branch to the corresponding
remote branch that 'git pull' uses to update the current branch.

See 'git help config' and search for 'push.default' for further information.
(the 'simple' mode was introduced in Git 1.7.11. Use the similar mode
'current' instead of 'simple' if you sometimes use older versions of Git)

Yeah, I've been staring at that for like a year or two instead of just picking a default like it tells me to. What can I say, I'm lazy. Anyway, as that clearly says, the intent in this case is that git should use its simple push behaviour. If you go and look at git help config, like it says, you find this interesting snippet:

       ·   upstream - push the current branch back to the branch whose changes
           are usually integrated into the current branch (which is called
           @{upstream}). This mode only makes sense if you are pushing to the
           same repository you would normally pull from (i.e. central
           workflow).

       ·   simple - in centralized workflow, work like upstream with an added
           safety to refuse to push if the upstream branch’s name is different
           from the local one.

That is, simple is supposed to reject the push if the source and target branches don't have the same name. But clearly, in my case, it didn't.

I was now pretty convinced I could blame git for the other 50% of my booboo, so I did some digging, and eventually wound up figuring out what was going on and filing a 'bug report' (mailing list post) with the git folks. Basically, it turns out that they sort of got their wires crossed with a couple of different people working on this code when they implemented the behaviour, and in the case where push.default is not explicitly set, this has been broken all along: in effect, it's been behaving as upstream, not simple, in the case where the default isn't explicitly set. If you explicitly set the default to simple it works as expected - the push is rejected.

I'm mildly surprised more people haven't hit this; I guess not many people do idiotic mistaken pushes like me, and/or most people read the message saying to pick a default behaviour and actually pick one, rather than saying 'meh, maybe next decade'. (Or, other people have been hit by this but didn't feel so convinced git should've saved them from themselves, they just accepted the blame and moved on...)

Anyway, it looks like that'll wind up getting fixed in git upstream, so at least I'll hopefully have saved the next idiot monkey who both procrastinates needlessly and doesn't double check his git pushes. And if you're an idiot monkey like me and have been staring at that same advisory message for months, I'd advise you go right ahead and run git config --global push.default simple right now, and/or be very careful with your git pushes.

Today has been interesting as on the one hand we're in Fedora 21 Final rush mode but on the other hand all the Americans are off for Thanksgiving. Thanks to some heroic work by Kamil Paral and Vratislav Podzimek I was able to file the 21 Final RC1 compose request, which is great news for the chances of getting F21 out on time - it's quite likely we'll wind up needing more RCs, but for a while I was worried we wouldn't even make RC1 until Monday, and with Go/No-Go next Thursday, that would've been yet another week of crazy overnight test runs and lack of sleep. With RC1 composed this afternoon we have tomorrow and the weekend to do really exhaustive testing on it, and if there do turn out to be any more bits to clean up, we'll have much fewer moving parts to keep track of and a much more solid test base to work off next week.

So aside from getting that done, I spent most of the day digging into a bug that was niggling at me. It had come up in blocker review and been rejected because at the time it looked like it was related to installing on a Mac with both the internal storage and an SD card as target disks, which is a pretty obscure corner case, but I wasn't entirely sure that was the whole story.

After an hour or two of running scenarios and several more hours laboriously picking through anaconda code, I'm pretty sure I've figured out what's going on and provided at least a candidate fix - I think the bug in fact affects any install with a 'platform' that uses a partition as the bootloader stage1 target (mainly UEFI, but that also covers one ARM variant, and possibly a few other things like BIOS + GPT and PPC), when you install to multiple disks and want to put the bootloader stage1 partition on a disk other than the first. I'm pretty sure that just doesn't work, and I know more or less why.

That might sound like, you know, kind of a bad bug. You might be sitting there wondering how something that broken happened, and why we hadn't really spotted it until now. But you know what...this is an ideal illustration of the thing we keep banging on about, that writing OS installers is really goddamn complicated. See where I mentioned 'platforms' up there? anaconda supports installing to 10 different 'platforms' - platforms are things like x86 BIOS, x86 UEFI, Mac UEFI, ARM, various ARM variants, PPC and its variants, S390 - all of which require drastically different bootloader configurations. It has (depending on how exactly you count) like 4-6 different interfaces / interface 'workflows' in which partitioning could be defined - kickstart with explicit partition layout. kickstart autopart, text UI, GUI guided, GUI custom+'create partitions for me', GUI custom. The intersection of partitioning and bootloader code has to somehow manage to create valid configurations when using 'guided' or 'automatic' partition creation, and correctly validate the partition layout plus bootloader configuration for all platforms on all workflows, both accepting valid configurations and rejecting invalid ones. I spent like four hours just picking through anaconda and blivet code to even understand the flow from partitioning to bootloader configuration validation before I could touch a line of code, and I'm sure I don't completely grok all the subtleties of it even now. Then I had to try and figure out where exactly in the 15+ source files I have lying around in gedit was the right place to fix it, and try to think through whether the fix not only fixed the case I was trying to fix but didn't break any other cases (with that combinatorial explosion of possible paths I just mentioned to think about). Then I had to test it with a half dozen installs just to feel comfortable in at least submitting it to Bugzilla. And even then I'm only like 70% sure I more or less got things right and my patch is a decent first starting point. I don't know how anyone stands working on anaconda for a living, I'd go insane(r).

And the thing is, you can't even say 'well that's obviously too complex, no-one can deal with that mess, let's simplify it' because, well, Fedora/RH really wants to work on all those platforms and offer all those partitioning workflows. If we don't, people scream. Hell, there are still people complaining about us not letting you install grub to a partition header any more - that was one small thing the devs did to simplify this whole mess (a platform for which the stage1 target can be either a disk or a partition just makes the whole thing even more of a damn nightmare), and just that small change resulted in like weeks of contentious bug reports and ML threads. So we're basically stuck with it. The code is a complex mess, but it's not unnecessarily or egregiously so, it's inevitably so - it's trying to do so much stuff it really can't be otherwise....

Fedlet 20141124: rotation and backlight on Venue 8 Pro, wifi on T100(?)

Hi folks! Just a quick note that I've sent out a new release of Fedlet, my Fedora remix for 32-bit Bay Trail-based tablets like the Dell Venue 8 Pro, Asus T100, and Lenovo Miix 2. The new release, 20141124, can be found on the Fedlet page. It adds sensor-based rotation for the Venue 8 Pro (with many thanks to Jan-Michael Brummer and Bastien Nocera who got that working between them), backlight brightness control on the Venue 8 Pro if booted with i915.force_backlight_pmic=1, and possibly OOTB wifi on the Asus T100. Good luck!

Stupid Fedora tricks: prune livecd-creator cache directory

Here's a really dumb python script that prunes the livecd-creator cache directory - or, any directory with a big lump of RPM files in it, really. It finds (roughly, but we're not going for the World Accuracy Prize here) multiple versions of the same package and removes all but the newest one (it relies on ls to order the files by date after ordering them by name, which it does here, but YMMV). Run it from the directory you want to prune. No guarantees given at all, may eat your babies.

#!/bin/python

from subprocess import check_output
import re
import os

sep = re.compile('-\d')

output = check_output('ls *x86_64.rpm', shell=True)
lastfile = ''
for rpmfile in output.split('\n'):
    if sep.split(rpmfile)[0] == sep.split(lastfile)[0]:
        os.remove(lastfile)
    lastfile = rpmfile

output = check_output('ls *noarch.rpm', shell=True)
lastfile = ''
for rpmfile in output.split('\n'):
    if sep.split(rpmfile)[0] == sep.split(lastfile)[0]:
        os.remove(lastfile)
    lastfile = rpmfile

output = check_output('ls *i686.rpm', shell=True)
lastfile = ''
for rpmfile in output.split('\n'):
    if sep.split(rpmfile)[0] == sep.split(lastfile)[0]:
        os.remove(lastfile)
    lastfile = rpmfile

New Fedlet release, 20141111, a tip for KMS on the Venue 8 Pro, and dealing with Fedlet 'updating' to Fedora 22

I cut a new Fedlet release today, 20141111. You can get it from the Fedlet page.

The big improvement is working internal wifi on the Venue 8 Pro - many thanks to Jan-Michael Brummer for that. It has kernel 3.18rc4, which may possibly help other devices out, I don't know. It bumps the userland to ~Fedora 21 Final TC1 or so.

I figured out a way to make KMS graphics work on my Venue 8 Pro, too. For me, it works when I boot through the boot device menu (hold down volume up on boot). When I boot that way, early boot happens in a low resolution, then when KMS kicks in, it switches to native resolution. When I boot through the firmware UI, or just normally, early boot happens in native resolution, and KMS activation blanks the screen.

For me, graphics are extremely slow and wifi doesn't work after a reboot - the workaround is to power off fully and power back on, not just reboot.

If you have an earlier version of Fedlet installed, you may have wound up getting Rawhide - Fedora 22 - packages installed, instead of tracking Fedora 21. This is to do with Fedlet being a non-official build of Fedora - I didn't bother making a 'fedlet-release' package, I just use the 'generic-release' package that's shipped in Fedora, but it doesn't get updated very quickly and it wound up pulling in Rawhide repositories when it shouldn't have.

To fix it up, you basically need to get up to at least generic-release-21-7 from the F21 repo, and install a 'Product' subpackage - generic-release-workstation or generic-release-nonproduct are the obvious choices. Remove generic-release-rawhide. Then you can run yum --releasever 21 distro-sync, which should pull everything back into line with Fedora 21.

The new release has the current generic-release, so it shouldn't have this problem. I forgot to explicitly pull in a Product sub-package, though, so it got generic-release-cloud, which is silly but shouldn't have any particular consequences. I'll clean it up in the next release.