AdamW on Linux and more (Posts about Fedlet)

Fedora 28, broken comments, QA goings on...

2018-04-27T16:40:20Z

Long time no blog, once more!

Fedora 28

So of course, the big news of the week is that Fedora 28 was signed off yesterday and will be coming out on 2018-05-01. If you examine the Fedora 28 schedule, you will observe that this was in fact the originally targeted date for the release. The earliest targeted date.

Yes. It's a Fedora release. Coming out on time. That noise you hear is the approaching meteor that will wipe out all life on Earth. You're welcome. ;)

We've always said the schedules for Fedora are really estimates and we don't consider it a problem if there's a week or two delay to fix up bugs, and that's still the case. We may well wind up slipping again for F29. But hey, it's nice to get it done "on time" just once. I did, in fact, check, and this really is the very first time a Fedora release has ever been entirely on time. Fedora 8 was close - it was only a day late, if you discount a very early draft schedule - but still, a day's a day!

There are, as always, a few bugs I really wish we'd been able to fix for the release. But that's pretty much always the case, and these are no worse than ones we've shipped before. We have to draw a line somewhere, for a distro that releases as often as Fedora. This should be another pretty solid release. My desktop and main laptop are running it already, and it's pretty fine.

Comments: Yes, They're Broken

Quick note for people who keep emailing me: yes, posting comments on this blog appears to be broken. No, I'm not particularly bothered. I actually have been meaning to convert this into an entirely static blog with no commenting for years, I just don't want to deal with Wordpress or any dynamic blog framework really any more. But I never have time to do it, as I want to include existing comments in the conversion, which isn't straightforward. I'm gonna get it done one of these days, though.

openQA news: upgrade tests for updates, aarch64 testing...

I've been doing a lot of miscellaneous stuff I haven't blogged about lately, but here's one thing I'm pretty proud of: Simo and Rob from the FreeIPA team asked if it would be possible to test whether Fedora package updates broke FreeIPA upgrades, as Simo had noticed a case where upgrading a server to Fedora 27 didn't work. We already had tests that test deploying a FreeIPA server and client on one Fedora release, then upgrading both to the next Fedora release and seeing if things still worked - but we weren't running it on updates, we only ran it on nightly composes of Branched and Rawhide. So effectively we know all the way up until a given release comes out whether upgrading works for it, but once it comes out, we didn't know if upgrading is suddenly broken by a later update.

These tests are some of the longest-running we have, so I was a bit worried about whether we'd have the resources to run them on updates, but I figured I'd go ahead and try it, and after a day or two of bashing was able to get it running in staging. After a week, staging seemed to be keeping up with the load, so I've pushed this out into production today. If you look at recent openQA update tests, like this one, you'll see an updates-server-upgrade flavor with a couple of tests in it: these are testing that installing the previous Fedora release, deploying a FreeIPA server and client, then upgrading them to the release the update is for, with the update included, works OK. I'm quite happy with that! I may extend this basic mechanism to also run the Workstation upgrade test as well. Note that these tests don't run for updates that are for the oldest current stable Fedora, as we don't support upgrades from EOL releases (and openQA doesn't keep the necessary base disk images for EOL releases around, so we actually couldn't run the tests).

Aside from that, the biggest openQA news lately is that we got the staging instance testing on aarch64. Here's the aarch64 tests for Fedora 28 Final, for instance. This isn't perfect yet - there are several spurious failures each time the tests run. I think this is because the workers are kind of overloaded, they're a bit short on RAM and especially on storage bandwidth (they each just have a single consumer-grade 7200RPM hard disk). I'm working with infra to try and improve that situation before we consider pushing this into production.

Other QA goings on

One thing that's been quite pleasant for me lately is I'm no longer trying to do quite so much of...everything (and inevitably missing some things). Sumantro and coremodule have done a great job of taking over Test Day co-ordination and some other community-ish tasks, so I don't have to worry about trying to keep up with those any more. Sumantro has been bringing a whole bundle of energy to organizing Test Days and onboarding events, so we've had lots more Test Days these last two cycles, and more people to take part in them, which is great. We've also had more folks taking part in validation testing. It's made life a lot less stressful around here!

I've been mostly concentrating on co-ordinating things like release validation testing, doing a bit of mentoring for the newer team members, and keeping openQA ticking over. It's nice to be able to focus a bit more.

wikitcms, relval, fedfind and testdays moved to Pagure

2016-10-14T14:54:56Z

Today I moved several of my pet projects from the cgit instance on this server to Pagure. You can now find them here:

The home page URLs for each project on this server - e.g. https://www.happyassassin.net/fedfind - also now redirect to the Pagure project pages.

I also deleted some other repos that were hosted in my cgit instance entirely, because I don't think they were any longer of interest to anyone and I didn't want to maintain them. Those were mostly related to Fedlet, which I haven't been working on for 2-3 years now.

For now the repos for the three main projects - wikitcms, relval and fedfind - remain in my cgit instance, containing just a single text file documenting the move to Pagure; in a month or so I will remove these repositories and decommission the cgit instance. So, update your checkouts! :)

This saves me maintaining the repos, provides pull review and issue mechanisms, and it's a good thing to have all Fedora-ish code projects in Pagure in general, I think.

Many thanks to pingou and everyone else who works on Pagure, it's a great project!

Flock 2015 report, and Fedora nightly compose testing

2015-08-21T16:44:41Z

Hi, folks! I've been waiting to write my post-Flock report until I had some fun stuff to show off, because that's more exciting than just a bunch of 'I went to this talk and then talked to this person', right?

Fedora nightly compose testing

So let me get to the shiny first! Without further ado:

Cool, right? That's what I've been working on this whole week since Flock. All the bits are now basically in place such that, each night, openQA will run on the Branched and Rawhide nightly composes when they're done, and when openQA is done, the compose reports will be mailed out.

Flock report

The details behind that get quite long, so before I hit that, here's a quick round-up of other stuff I did at Flock! I'm not going to cover the talks and sessions many others have already blogged about (the keynotes, etc.) as it seems redundant, but I'll mention some stuff that hasn't really been covered yet.

Josef ran a workshop on getting started with openQA. It was a bit tricky, though, due to poor networking on site; the people trying to follow along and deploy their own Docker-based openQA instances couldn't quite get all the way. So we turned the last bit of the talk into a live demo using my openQA instance instead, and created a new test case LIVE ON STAGE. We didn't quite get it all the way done before getting kicked out by a wedding party, but I finished it up shortly after the session. Josef did a great job of explaining the basics of setting up openQA and creating tests, and I hope we'll have a few more people following the openQA stuff now.

Mike McLean did a great talk on Koji 2.0, which has been kinda under the radar (at least for me) compared to Bodhi 2, but sounds like it'll come with a lot of really significant improvements and a better design. As someone who's spent a lot of time staring at kojihub.py lately, I can only say it'd be welcome...

Denise Dumas gave the now-traditional What Red Hat Wants talk, which I'm really glad is happening now. I'm totally behind the idea that we're up-front about the relationship between Red Hat and Fedora, instead of some silly arrangement where Red Hat pretends Fedora just 'happens' and is a totally community-based distro; it's much better for RH to be saying a couple of years in advance 'hey, this is where we'd like to see things going', rather than every so often a bunch of Features/Changes 'mysteriously' appearing four months out from a release and lots of people suddenly caring a lot about them (but it just all being a BIG COINCIDENCE!)

Paul Frields did a nice talk on working remotely, which had a lot of great ideas that I don't do at all (hi Paul, it's 4:30pm and I'm writing this in my dressing gown...) - but it was great to compare notes with a bunch of other folks and think about other ways of doing things.

I did a lightning talk on Fedlet, showing it off running and talking a bit about what Fedlet involves and how well (or not) it runs. Folks seemed interested, and a few people came by to play with my fedlet afterwards.

Stephen Gallagher ran a rolekit hackfest. I was hoping to use it to come up with an openQA role, but failed for a couple of reasons: Stephen doesn't recommend creating new roles right now as the format is likely to change a lot quite soon, and since I last worked on the package openQA has added a few more dependencies which need packaging. But I did manage to move forward with work on the package a bit, which was useful. In the session Stephen explained the rolekit design and current state to people, and talked about various work that needs doing on it; hopefully he'll get some more help with it soon!

Of course, as always, there was lots of hallway track and social stuff. We had a couple of excellent poker games - good to see the FUDCon/Flock poker tradition continues strong - and played some Exploding Kittens, which is a lot of fun. My favourite bit is the NOPE cards. As many others have said, the Strong Museum was awesome - got to play a bunch of pinball, and see Will Wright's notebooks(!) and John Romero's Apple ][(!!!!).

Fedora compose testing: development details and The Future

So, back to the 'compose CI' stuff I spent a lot of time talking about/working on!

A lot of what I did at Flock centred around the big topic you can call 'CI for Fedora'. We still have lots of plans afoot for big, serious test and task automation based on Taskotron, which is now getting really close to the point where you'll see a lot more cool stuff using it. But in the meantime, the 'skunkworks' openQA project we spun up during the Fedora 22 cycle has grown quite a bit, and the fedfind project I mostly built to back openQA has grown quite a lot of interesting capabilities.

So while we were talking about properly engineered plans for the future, I realized I could probably hack together some stupidly-engineered stuff that would work right now! In Kevin Fenzi's Rawhide session I threw out a few ideas and then figured that, hell, I should just do them.

So I started out by teaching fedfind some new tricks. It can now 'diff' two releases: that is, it can tell you what images are in one, but not the other. It can also check a release for 'expected' images - basically it has some knowledge about what images we'd most want to be present all the time, and it can tell you if any are missing. (FIXME: I didn't know which of the Cloud images were the most important, so right now it has no 'expected' Cloud images: if some Cloud-y people want to tell me which images are most important, I can add them).

Then I wrote a little script called check-compose which produces a handy report from that information. It also looks for openQA tests for the compose it's checking, and includes a list of failures if it finds any. It can email the report and also write the results in JSON format (which seemed like a good idea in case we want to look back at them in some programmatic way later on). The 'compose check reports' that have been showing up this week (and that I linked above) are the output of the script.

I had all of that basically done by Tuesday, so what have I been wasting the rest of my week on? Read on!

What was missing was the 'C' part of 'CI'. There was nothing that would actually run the compose report at appropriate times, and we weren't actually running openQA tests nightly. For the past few days I've been kind of faking things up by manually kicking off openQA jobs and firing off the compose report when they're done. This kind of mechanical Turk CI doesn't really work in the long run! So for the last few days I've worked on that.

We were not actually scheduling nightly openQA runs at all. The openQA trigger script has an all mode which is intended to do that, but we weren't running it. I suggested we turn it back on, but I also wanted to fix one big problem it had: it didn't know whether the composes were actually done. It just got today's date and tried to run on the nightlies for it. If they weren't actually done whenever the script ran, you got no tests.

This definitely hooks in with one of the big topics at Flock: Pungi 4, which is the pending major revision. Pungi is the tool which runs Fedora composes. Well, that's not quite right: there's actually a couple of releng scripts which produce the composes (the first of those is for nightlies, the second is for TCs/RCs). They run pungi and do lots of other stuff too, because currently pungi only actually does some of the work involved in a compose (a lot of the images are just built by the trigger scripts firing off Koji tasks and other...stuff). The current revision of the compose process is something of a mess (it's grown chaotically as we added ARM images and Cloud images and Docker images and Atomic images and flavors and all the rest of it). With the Pungi 4 revision and associated changes to the releng process, it should be trivial to follow the compose process.

Right now, though, it isn't. Nightly composes and TC/RC composes are very different. TCs/RCs don't emit information on their progress really at all. Nightlies emit some fedmsg signals, but crucially, there's no signal when the Koji builds complete: you get a signal when they start, but not when they're done.

So it was time to teach fedfind some new tricks! I decided not to go the fedmsg route yet since it's not sufficient at present. Instead I taught it to tell if composes are complete in lower-tech ways. For the Pungi part of the process it looks for a file the script creates when it's done. For Koji tasks, it finds all the Koji tasks that looks like they're a part of this nightly, and only considers the nightly 'done' when there are at least some tasks (so it doesn't report 'done' before the process starts at all) and none of the tasks is 'open' (meaning running or not yet started).

So now we could make the openQA trigger script or the compose-check script wait for a compose to actually exist before running against it! Great. Only now I had a different problem: the openQA trigger script was set up to run for both nightlies. This is fine if it's not waiting - it just goes ahead and fires one, then the other. But how to make it work with waiting?

This one had to go through a couple of revisions. My first thought was "I have a problem. I know! I'll use threads", and we all know how that joke goes. Sure enough, all three of the revisions of this approach (using threading, multiprocessing and multiprocessing.dummy) turned out to have problems. I eventually decided it wasn't worth carrying on fighting with that, and came up with some different approaches. One is a low-tech round-robin waiting approach, where the trigger script alternates between checking for Branched and Rawhide. The other is even simpler: by just adding a few capabilities to the mode where the trigger runs on a single compose, we can simply schedule two separate runs of that mode each night, one for Rawhide, one for Branched. That keeps the code simple and means either one can get all the way through the 'find compose, schedule jobs, run jobs, run compose report' process without waiting for the other.

And that, finally, is about where we're at right now! I'm hoping one or the other openQA change will be approved on Monday and then we can have this whole process running unattended each night - which will more or less finally implement some more of the near-legendary is Rawhide broken? proposal. Up till then I'll keep running the compose reports by hand.

Along the way I did some other messing around in fedfind, mostly to do with optimizing how it does Koji queries (and fixing some bugs). For all of a day or so, it used multiprocessing to run queries in parallel; I decided multithreading just wasn't worth it for the moderate performance increase, though, so I switched to using a batched query mode provided by xmlrpclib, which speeds things up a little less but keeps the code simpler. I also implemented a query cache, and spent an entire goddamn afternoon coming up with a reasonable way to make it handle fuzzy matches (when e.g. we run a query for 'all open or successful tasks', then run a query for 'all successful live CD tasks', we can derive the results for the latter from the former and not waste time talking to the server again). But I got there in the end, I think.

It was quite a lot of work, in the end, but I'm pretty happy with the result. I'm really, really looking forward to the releng improvements, though. fedfind is more or less just the things releng is aiming to do, only implemented (unavoidably) stupidly and from the wrong end. As I understand it, releng's medium-term goals are:

all composes to contain sufficient metadata on what's actually in them
compose processes for nightlies to be the same as that for TCs/RCs
compose process to notify properly at all stages via fedmsg
ComposeDB to track what composes actually exist and where they are

right now we don't really have any of those things, and so fedfind exists to reconstruct all that information painfully, from the other end. It will definitely be a relief when we can get all that information out of sane systems, and I don't have to maintain a crazy ball of magic knowledge, Koji queries and rsync scrapes any longer. For now, though, the whole crazy ball of wax seems to actually work. I'm really glad that folks like Kevin, Dennis, Peter, Ralph, Adam and others are all thinking down the same general lines: I'm hopeful that with Pungi, ComposeDB (when it happens), and further work on Taskotron and openQA and even my stupid little scripts, we'll have continuously (see what I did there?!) better stories to tell as we move on for the next few releases.

Fedlet 2015-08-10 available

2015-08-10T23:01:52Z

So here's a lil' pre-Flock present for all you crazy Baytrail-ers - a new Fedlet image. Get your 2015-08-10 while it's hot, over at the fedlet page.

This is pretty much just a rebase of the kernel to 4.2rc6 in an F23 Alpha userland. I built a GTK+ package with a patch from Jan-Michael to use a popover for URL histories, which should improve OSK interaction, but Firefox seems to have some issue in this image where it never pops up an OSK at all, and the GTK+ I built is actually older than the current F23 build so it's not included. You can 'downgrade' to it after installing, though, and it seems to work in Epiphany.

Seems to be working pretty well otherwise on the V8P - sound and wifi are both good, and even survive a suspend/resume. Installing worked fine for me (my old install got a bit messed up so I reinstalled), though I hit some anaconda bugs deleting the old install, so I did that manually with fdisk. Except for one bug: the top bar doesn't render properly, so you can't see the 'Done' buttons. They're there, though, so just click blindly in the right place (bottom left corner of the blue top bar). You can boot up a regular Fedora image in a VM or something to see what you're missing.

ownCloud 8 and Fedlet 22(?) updates

2015-02-22T01:54:20Z

Hi there folks!

Between testing and working on my current work-related-ish pet projects fedfind and python-wikitcms/relval, I haven't had much time for Fedlet or the ownCloud package lately. However, I finally managed to squeeze in a few hours to work on them this week.

Fedlet

The Fedlet repository now has a 3.20-pre kernel build that more or less works, for me, in testing. It seems like perhaps it's buggier than 3.18 was, but it at least built and it boots and it has sound and wifi and power status on the V8P. I did throw in a patch which may help with the RPMB bugs. For me it seems slower than 3.18 and touch input is rather worse than it was before, but I haven't had time to look into that in any detail yet. I have built a test Fedora 22-based image which boots, but given the apparent state of 3.20 and the pre-Alpha state of F22 at present I'm not going to release it.

I also set up a public copy of the actual repo from which the Fedlet kernel is built. You can clone that repo and check out the baytrail branch and you have the actual build bits right there. There's one small caveat: I make a few changes right before actually building which are never checked into the repo. They are:

Add the changelog
make config-release
sed -i -e 's,%define debugbuildsenabled 1,%define debugbuildsenabled 0,g' kernel.spec

Step 2 disables debugging in the kernel build (which you really want, on slow Fedlet-ish hardware) and step 3 says not to build a separate kernel-debug package with debugging turned on. Obviously if you actually want kernel debugging to do some...kernel debugging...you can skip 2 and/or 3.

I don't check those changes in because they tend to make merging from upstream (Fedora kernel package) much messier. Aside from that, though, all the substantial stuff is checked in. Mostly the delta is down to patch files now; there's only a couple of changed CONFIG settings any more.

ownCloud

I'm working on getting the ownCloud packages up to the recently-released 8.0. Major OC releases always require quite a bit of gardening downstream. I now have a side repo for OC 8 available; for now there's only a repo for Fedora 22, but you could use it for Rawhide too, and it'll be easy to add at least Fedora 21 soon.

The repo contains OC 8 packages plus Pimple 3.0 (which OC 8 needs) and Doctrine DBAL 2.5.1 with a change that seems to break ownCloud partly reverted.

So far it's at the point where a clean deployment with sqlite as the database works, and you can install apps from the 'app store'. This is important as some apps that were previously part of the core - notably Contacts, Calendar, and Documents - have been moved to the store. I am not sure whether to provide packages for these or even roll them back into the main owncloud package, but for now I'm following upstream.

I haven't yet tested any other database, and I have not tested upgrading an existing install. It goes without saying that you should only do anything at all with this repository on a test setup. :)

You can use oc8-httpd-sqlite.ks to do a full turnkey test install; as usual for my OC test kickstarts, running an F22 install with that kickstart will give you a box with OC running and accessible from any system right out of the box, with insecure admin account (admin/admin) and database credentials, never use my test kickstarts unmodified in production or on any box which is publicly accessible. The kickstarts for other databases need a quick tweak which I'll get around to tomorrow as I test them.

As part of the OC 8 work I wound up overhauling the Apache config layout quite heavily; I needed to fix a few bugs and update it with the latest upstream changes, and took the opportunity to use some Include directives to make it rather cleaner. I've sent a modest proposal about standardizing config snippets intended for inclusion to devel@.

And with that, to bed!

It's a good thing they pay me to break stuff

2014-11-27T18:17:28Z

Had an interesting couple of days deep-diving into juicy issues.

Yesterday's was triggered by me messing up the Fedora kernel package git repository - whoops. I keep Fedlet's kernel as a branch that only exists in my checkout; it's not pushed anywhere (I should just push it out on my git server now I have one, but I keep forgetting). I accidentally ran git push from that branch yesterday, and it promptly pushed all the changes on it to master, effectively turning Fedora's kernel into the Fedlet kernel for a few glorious hours until Josh reverted it.

Obviously at least 50% of the blame there lies squarely on me for having set an upstream for my branch and running git push from it at all, but it seemed odd to me that git would run such a push in a fairly 'default' configuration. So I did some digging.

It turns out it's not meant to. I don't have git's push.default setting set locally. So every time I push anything anywhere, I see this message:

warning: push.default is unset; its implicit value has changed in
Git 2.0 from 'matching' to 'simple'. To squelch this message
and maintain the traditional behavior, use:

  git config --global push.default matching

To squelch this message and adopt the new behavior now, use:

  git config --global push.default simple

When push.default is set to 'matching', git will push local branches
to the remote branches that already exist with the same name.

Since Git 2.0, Git defaults to the more conservative 'simple'
behavior, which only pushes the current branch to the corresponding
remote branch that 'git pull' uses to update the current branch.

See 'git help config' and search for 'push.default' for further information.
(the 'simple' mode was introduced in Git 1.7.11. Use the similar mode
'current' instead of 'simple' if you sometimes use older versions of Git)

Yeah, I've been staring at that for like a year or two instead of just picking a default like it tells me to. What can I say, I'm lazy. Anyway, as that clearly says, the intent in this case is that git should use its simple push behaviour. If you go and look at git help config, like it says, you find this interesting snippet:

       ·   upstream - push the current branch back to the branch whose changes
           are usually integrated into the current branch (which is called
           @{upstream}). This mode only makes sense if you are pushing to the
           same repository you would normally pull from (i.e. central
           workflow).

       ·   simple - in centralized workflow, work like upstream with an added
           safety to refuse to push if the upstream branch’s name is different
           from the local one.

That is, simple is supposed to reject the push if the source and target branches don't have the same name. But clearly, in my case, it didn't.

I was now pretty convinced I could blame git for the other 50% of my booboo, so I did some digging, and eventually wound up figuring out what was going on and filing a 'bug report' (mailing list post) with the git folks. Basically, it turns out that they sort of got their wires crossed with a couple of different people working on this code when they implemented the behaviour, and in the case where push.default is not explicitly set, this has been broken all along: in effect, it's been behaving as upstream, not simple, in the case where the default isn't explicitly set. If you explicitly set the default to simple it works as expected - the push is rejected.

I'm mildly surprised more people haven't hit this; I guess not many people do idiotic mistaken pushes like me, and/or most people read the message saying to pick a default behaviour and actually pick one, rather than saying 'meh, maybe next decade'. (Or, other people have been hit by this but didn't feel so convinced git should've saved them from themselves, they just accepted the blame and moved on...)

Anyway, it looks like that'll wind up getting fixed in git upstream, so at least I'll hopefully have saved the next idiot monkey who both procrastinates needlessly and doesn't double check his git pushes. And if you're an idiot monkey like me and have been staring at that same advisory message for months, I'd advise you go right ahead and run git config --global push.default simple right now, and/or be very careful with your git pushes.

Today has been interesting as on the one hand we're in Fedora 21 Final rush mode but on the other hand all the Americans are off for Thanksgiving. Thanks to some heroic work by Kamil Paral and Vratislav Podzimek I was able to file the 21 Final RC1 compose request, which is great news for the chances of getting F21 out on time - it's quite likely we'll wind up needing more RCs, but for a while I was worried we wouldn't even make RC1 until Monday, and with Go/No-Go next Thursday, that would've been yet another week of crazy overnight test runs and lack of sleep. With RC1 composed this afternoon we have tomorrow and the weekend to do really exhaustive testing on it, and if there do turn out to be any more bits to clean up, we'll have much fewer moving parts to keep track of and a much more solid test base to work off next week.

So aside from getting that done, I spent most of the day digging into a bug that was niggling at me. It had come up in blocker review and been rejected because at the time it looked like it was related to installing on a Mac with both the internal storage and an SD card as target disks, which is a pretty obscure corner case, but I wasn't entirely sure that was the whole story.

After an hour or two of running scenarios and several more hours laboriously picking through anaconda code, I'm pretty sure I've figured out what's going on and provided at least a candidate fix - I think the bug in fact affects any install with a 'platform' that uses a partition as the bootloader stage1 target (mainly UEFI, but that also covers one ARM variant, and possibly a few other things like BIOS + GPT and PPC), when you install to multiple disks and want to put the bootloader stage1 partition on a disk other than the first. I'm pretty sure that just doesn't work, and I know more or less why.

That might sound like, you know, kind of a bad bug. You might be sitting there wondering how something that broken happened, and why we hadn't really spotted it until now. But you know what...this is an ideal illustration of the thing we keep banging on about, that writing OS installers is really goddamn complicated. See where I mentioned 'platforms' up there? anaconda supports installing to 10 different 'platforms' - platforms are things like x86 BIOS, x86 UEFI, Mac UEFI, ARM, various ARM variants, PPC and its variants, S390 - all of which require drastically different bootloader configurations. It has (depending on how exactly you count) like 4-6 different interfaces / interface 'workflows' in which partitioning could be defined - kickstart with explicit partition layout. kickstart autopart, text UI, GUI guided, GUI custom+'create partitions for me', GUI custom. The intersection of partitioning and bootloader code has to somehow manage to create valid configurations when using 'guided' or 'automatic' partition creation, and correctly validate the partition layout plus bootloader configuration for all platforms on all workflows, both accepting valid configurations and rejecting invalid ones. I spent like four hours just picking through anaconda and blivet code to even understand the flow from partitioning to bootloader configuration validation before I could touch a line of code, and I'm sure I don't completely grok all the subtleties of it even now. Then I had to try and figure out where exactly in the 15+ source files I have lying around in gedit was the right place to fix it, and try to think through whether the fix not only fixed the case I was trying to fix but didn't break any other cases (with that combinatorial explosion of possible paths I just mentioned to think about). Then I had to test it with a half dozen installs just to feel comfortable in at least submitting it to Bugzilla. And even then I'm only like 70% sure I more or less got things right and my patch is a decent first starting point. I don't know how anyone stands working on anaconda for a living, I'd go insane(r).

And the thing is, you can't even say 'well that's obviously too complex, no-one can deal with that mess, let's simplify it' because, well, Fedora/RH really wants to work on all those platforms and offer all those partitioning workflows. If we don't, people scream. Hell, there are still people complaining about us not letting you install grub to a partition header any more - that was one small thing the devs did to simplify this whole mess (a platform for which the stage1 target can be either a disk or a partition just makes the whole thing even more of a damn nightmare), and just that small change resulted in like weeks of contentious bug reports and ML threads. So we're basically stuck with it. The code is a complex mess, but it's not unnecessarily or egregiously so, it's inevitably so - it's trying to do so much stuff it really can't be otherwise....

Fedlet 20141124: rotation and backlight on Venue 8 Pro, wifi on T100(?)

2014-11-24T21:57:47Z

Hi folks! Just a quick note that I've sent out a new release of Fedlet, my Fedora remix for 32-bit Bay Trail-based tablets like the Dell Venue 8 Pro, Asus T100, and Lenovo Miix 2. The new release, 20141124, can be found on the Fedlet page. It adds sensor-based rotation for the Venue 8 Pro (with many thanks to Jan-Michael Brummer and Bastien Nocera who got that working between them), backlight brightness control on the Venue 8 Pro if booted with i915.force_backlight_pmic=1, and possibly OOTB wifi on the Asus T100. Good luck!

New Fedlet release, 20141111, a tip for KMS on the Venue 8 Pro, and dealing with Fedlet 'updating' to Fedora 22

2014-11-11T22:06:28Z

I cut a new Fedlet release today, 20141111. You can get it from the Fedlet page.

The big improvement is working internal wifi on the Venue 8 Pro - many thanks to Jan-Michael Brummer for that. It has kernel 3.18rc4, which may possibly help other devices out, I don't know. It bumps the userland to ~Fedora 21 Final TC1 or so.

I figured out a way to make KMS graphics work on my Venue 8 Pro, too. For me, it works when I boot through the boot device menu (hold down volume up on boot). When I boot that way, early boot happens in a low resolution, then when KMS kicks in, it switches to native resolution. When I boot through the firmware UI, or just normally, early boot happens in native resolution, and KMS activation blanks the screen.

For me, graphics are extremely slow and wifi doesn't work after a reboot - the workaround is to power off fully and power back on, not just reboot.

If you have an earlier version of Fedlet installed, you may have wound up getting Rawhide - Fedora 22 - packages installed, instead of tracking Fedora 21. This is to do with Fedlet being a non-official build of Fedora - I didn't bother making a 'fedlet-release' package, I just use the 'generic-release' package that's shipped in Fedora, but it doesn't get updated very quickly and it wound up pulling in Rawhide repositories when it shouldn't have.

To fix it up, you basically need to get up to at least generic-release-21-7 from the F21 repo, and install a 'Product' subpackage - generic-release-workstation or generic-release-nonproduct are the obvious choices. Remove generic-release-rawhide. Then you can run yum --releasever 21 distro-sync, which should pull everything back into line with Fedora 21.

The new release has the current generic-release, so it shouldn't have this problem. I forgot to explicitly pull in a Product sub-package, though, so it got generic-release-cloud, which is silly but shouldn't have any particular consequences. I'll clean it up in the next release.

Fedora 21 Beta, wikitcms/relval improvements, more stupid mediawiki tricks, working on laptops...

2014-11-04T14:52:12Z

I really should blog more, shouldn't I?

Today's big news, of course, is that Fedora 21 Beta is out!

Fedora 21 Beta

It was an interesting release to test, with a lot of the practical consequences of the Fedora.next 'Product-ization' effort getting shaken out. There are still some fudges and rough bits here and there in the practical implementation of that vision, but it's not that bad, for a first cut. It's definitely great to see the Cloud images showing up and the awesome Roshi co-ordinating test efforts there - I think the Product approach is really paying off in terms of highlighting the Fedora Cloud story.

The Server bits are also coming together - there's still further to go, but in Beta you can use the current Rolekit code successfully to set up a Server system as a 'domain controller' (a FreeIPA server) pretty painlessly, and Cockpit is pretty great. It's neat to do a Server installation and go straight to its Cockpit interface to manage it.

I'm pretty happy with the quality of 21 Beta as well, especially given all the moving parts we had to keep track of. The big emergency - there's always one - turned out to be a somewhat nasty upgrade issue we discovered after we'd already signed off on the Beta images, but we've ben able to come up with a pretty good strategy for dealing with that, and the only practical consequence for most folks should be that trying to upgrade to 21 with fedup will fail (safely, and intentionally) until we've got images we're happy are safe from this bug lined up later today. No-one should be able to mess up their system due to it. Of course, upgrading is inherently dangerous and upgrading to a pre-release is more so, and you should never do it without solid backups and a recovery plan.

Relval / wikitcms

Since I last blogged about wikitcms/relval I've cut a few more releases. 1.2 added a 'post-processor' to the testcase-stats sub-command which basically cleans up its output in a couple of ways; it handles things better when tables are renamed or test cases moved around or added or removed between composes. I'm hosting testcase_stats output for all Fedora releases from 12 onwards - the output for the current release is updated daily or hourly, depending on how hot testing for the release is at present (right now I'm on daily updates for 21).

relval 1.3 added a report-results sub-command: you can now report release validation results using a rather 1980s-style TUI rather than by editing wiki pages. It's a bit silly, but I found at least for me with 21 Beta validation it was rather more efficient than editing the wiki pages directly. It's pretty easy to try out, so give it a shot.

Currently I'm working on making the whole project more generic. wikitcms 1.4 added some preliminary support for QATracker, Ubuntu's manual test management system - not because I'm specifically interested in QA Tracker, but because it was a handy real-world implementation of something similar to Fedora release validation using a different system.

After 1.4 I decided my initial approach - adding more generic object types within wikitcms - wasn't quite the right one. Instead I'm planning to revert wikitcms to being strictly a client for the Fedora 'TCMS', and instead add a third component, perhaps called python-libtcms, whose job will be to provide a consistent interface to multiple TCMSes. libtcms will have concepts like 'unique test instances' and 'results' implemented in such a way that libtcms consumers just pick a particular TCMS to interface with, and can then use the same methods and so on regardless of what TCMS they picked.

I've already done a proof of concept where a minimally-modified version of relval was able to generate testcase-stats results for Ubuntu instead of Fedora; I'm hoping to get the production implementation done and probably released as 1.5 or 2.0 some time soon. Then I'll look at adding additional backends, perhaps for Moztrap next (I picked QATracker as the first non-wiki system that got implemented because it's a real system with real-world data and it already has a nice simple Python module for talking to its API).

Stupid Mediawiki Tricks

Along the way with Fedora 21 validation and relval work, I added a couple of refinements to the wiki magic bits of the Fedora release validation complex. I'm still trying to decide if the extent to which we've managed to abuse mediawiki for these purposes is cool, terrifying, or both. Anyhow, there's now a new CurrentFedoraCompose template which simply identifies the 'current' TC or RC. This is updated automatically by relval compose when invoked with --current in recent versions. I initially wanted to edit the Test Results:Current (testtype) Test redirect pages to use that template, so they didn't each have to be edited each time a new compose is run, but turns out you can't use templates in redirects, unfortunately.

Still, there's a couple of other places so far where the template has come in handy. When you run relval report-results without specifying the compose you want to file results for, it checks the contents of the template and defaults to filing results against that compose (so it usually defaults to the right one).

The stupid mediawiki trick I was thinking of, though, is on the validation instructions template. This is transcluded in every results page, with variables specifying the compose version and the test type. I took advantage of that and added a warning that's only displayed conditionally. If the page's compose version matches CurrentFedoraCompose, it isn't displayed. Otherwise, it is. The warning says that the page is not for the current compose, and links to the correct page for the current compose.

This uses a mediawiki extension called ParserFunctions, which lets you do clever stuff like that - it has a few special templates which let you do conditionals and evaluate expressions and stuff. This particular trick uses the {{#ifeq:}} function which tests whether some string (in this case, the combination of the page's {{{release}}} {{{milestone}}} and {{{compose}}} variables) matches some other string (in this case, the current contents of the {{CurrentFedoraCompose}} template), and produces different content depending on whether it does or not - in this case the content if it matches is nothing at all, and the content if it doesn't match is the "Warning! Not the current compose" note.

This saves poor Andre Robatino going through all the old compose pages adding the same warning manually!

Working on laptops

Kevin Fenzi's on a mission to blog daily at present, and today he explained why he uses a laptop as his primary work machine. I found it interesting because I have more or less the opposite experience - I use a desktop full-time. I usually go to the UK for a few weeks a year and have to use a laptop as my main system there, and I find it terrible and can't wait to get back to my desk.

I post pictures of my desk occasionally, the latest is still pretty much accurate. I think the main thing that makes me more productive at the desktop is the two large displays, followed by a real mouse and a real mechanical keyboard. I think the display issue varies a lot depending on your work; my work tends to involve a lot of context switching and writing one thing with reference to another thing, which is probably why I find the dual displays so valuable. I can imagine it'd matter much less if you focused on one task at a time and didn't need reference material so much, or if a lot of your work was done through remote console sessions in any case.

My typical dual display workflows are having mail open on one head and a browser on the other (often writing an email with reference to a web page), having a text editor on one head and a browser on the other (lately the text editor usually has some Python that doesn't work, and the browser has sixteen tabs open on either StackOverflow or the Python docs...), having test VMs open on one head and IRC running on the other, stuff like that. On a laptop I just never feel like I have enough room to keep an eye on everything at once.

I've never found any kind of laptop-y pointing device which could get anywhere close to a real mouse. You can carry a mouse around with a laptop, of course, but then you always have to find the damn thing and set it up. My desktop mouse sits on my desk and is always plugged into my desktop, it's an attractively simple scenario. :)

And more!

Well, I guess there's not a lot more? I spent some time polishing the ownCloud updates for Fedora 19 and EPEL 6 after I first blogged about them - I think they should both be good to go, though I'm hoping for at least some Bodhi feedback before I push them. I'm nearly in a position to do an ownCloud 7 build for EPEL 7, as well - I'm just waiting on an EPEL 7 branch for php-phpseclib-crypt-rijndael, then I can do a few builds / tags and a bit of testing and send a build out. I'm pretty sure all the necessary major bits are available for the build to work. I need to find a few hours to clean up some upstream submissions and ideas I have lying around for OC, as well.

Fedlet is sort of sitting third on my passion project list behind wikitcms and ownCloud ATM - apologies to Fedlet folks waiting impatiently for new bits :) I will try and do a Fedora 21 Betaish build with a 3.18 kernel some time soon, though.

Fedlet 2014-09-29 released

2014-09-29T10:10:31Z

Hi, folks - announcing a new Fedlet image. sha256sum is aa2f1150e40965471fc2888db6aad7da52d98f36ce1224b630ba5ed99b28fd5e . This has been rigorously tested by me booting it and pressing a few buttons.

It's built on a 3.17 rc6 kernel and Fedora 21 userland. A couple of patches from Jan-Michael Brummer: one to make the 'Home' button on the Venue 8 Pro act as a Super (Start) key - I tested that, it works, thanks Jan! - and one which should enable microphone input. I haven't tried that one myself yet.

I think this one should fix the problem which caused the last one not to install, but I haven't tested, so give it a shot and let me know.