Hi there, folks! What's up in Fedora-land?

Well, we're well into the Fedora 21 cycle, but the Fedora.next efforts are making it more than usually difficult to compose test images. Alpha TC1 is not complete yet, though release engineering are doing yeoman's work trying to sort out both Fedora.next changes and the usual breakage that shows up when we come to do the first builds for a cycle.

So we in QA have been working on the blocker bug list using nightly live and netinst images for now. Fingers crossed we'll get some test images up soon for full-bore validation testing.

In the mean time I spent a few hours yesterday working on OwnCloud. I've been working on the various packaging changes needed to get OwnCloud 7 into Fedora. I have some very very experimental 7.0.0 RC1 packages up in this repo - only F20 packages and .src.rpms, but I suspect the 'F20' packages should be OK with F21 and Rawhide for testing. PLEASE PLEASE do not install these in anything resembling production. All I've tested is that they work for a new deployment with an sqlite database in a Fedora 20 VM; I've done exactly zero testing of updates or real databases yet, and it's very likely there's more stuff to fix there. So if you upgrade your production deployment and it goes sideways, don't come crying to me.

I was planning to test those packages against a clone of my production OC 7 server today, but I got distracted instead by a new toy - a Logitech Harmony Smart Control.

I have something of a history with programmable remote controls, as long-term readers may remember. Since that last post, I did indeed go back to my Harmony 880. Then the buttons on it really started dying, so I bought another off eBay. One of the buttons on that one turned out to be bad, so I lived with that for a while. Then I created a sort of hideous Frankenstein's monster combination of both handsets, which worked for a while, but it seems like it's fundamentally not possible to stave off Dying Button Syndrome (to say nothing of Dying Charging Contacts Syndrome) on the 880 forever.

I've kept an eye on the several generations of Harmony device which followed the 880 with increasing trepidation, as I tried to keep my 880(s) on life support. The executive summary seems to have been 'they're all bad and they keep getting worse' (through the Harmony One, the Harmony Touch, the Harmony Link, and whatever other stuff they released in the interim). But finally it was time to give up on the 880, so I rather reluctantly bought myself the Smart Control.

It's part of the 2013 Harmony range, which is broadly based on the concept shared by the Harmony Link and the Redeye I discussed in some of my earlier posts: there's a central sort of 'server' box which transmits the actual IR (or sometimes Bluetooth) signals to the devices to be controlled, and the actual remote control(s) communicate with the hub (not directly with the controlled devices). The advantages of this setup are that it's good for non-line-of-sight control (as long as the hub has line of sight - or IR blasters connected to out-of-sight devices - things will work, as the remotes talk to the hub by RF or wifi, which don't require line of sight) and you can have multiple remotes that stay 'in sync'. As the hub speaks wifi, it can also be programmed without a USB cable.

You can buy the Hub alone (with the idea that you use an app running on a smartphone as the remote control). The Smart Link package I got bundles the hub with a very simple hardware remote, and you can also use the smartphone apps. The Ultimate package includes the hub and an 'advanced' hardware remote with a touchscreen. You can also buy the 'advanced' hardware remote alone, as it has its own IR transmitter, in which case it works more like an old-school Harmony, no hub, you have to plug it in to program it.

Anyway, enough of the background, how's the practice?

With the Redeye (and the Harmony 890 I used briefly, which used a very early version of the hub concept) I liked the idea of the non-line-of-sight communication, but setup tended to be finicky and I didn't ever get either hub to operate reliably enough to be happy with it. With the Redeye - which like the Hub-only package has no hardware remote, it relies on a smartphone app as the client - I found that I don't really like using a smartphone as a remote control much. For a start, while you're using it as a remote control, you can't really use it as a smartphone, and vice versa. For a second, no physical buttons makes it very difficult to use one handed and without looking. Remote controls are built to be remote controls, smartphones...not so much.

The intent of the Smart Control package is that the smartphone apps are the primary control device and the physical remote is more a kind of backup. However, it turns out the dumb, simple, cheap physical remote is pretty much the best Harmony remote they've made for a while. It's got all the buttons you really need for 98% of the time. It's got an awesome tactile back. Because it's dumb, simple and cheap it doesn't use a fancy rechargeable battery pack and charger (which have always been the thing that seems to die first on Harmony products); it just uses a simple coin cell. That lasts for a year. The internet consensus is that the 'advanced' touchscreen remote is more trouble than it's worth, and my experience of the Smart Control remote so far certainly doesn't contradict that.

The Hub also seems to be more reliable than the 890 transmitter or the Redeye were, for me. I wasn't able to sit it in a really clear view of all my devices, but it seems very powerful - the 'more or less' view it has seems to be fine, it hasn't missed a transmission yet. It responds reliably to input from the hardware remote and multiple phone clients, even when they're all being used interchangeably. A big advantage over all previous generations and competing systems is that it has a Bluetooth transmitter. At first this could only control Playstations and Wiis (which is a big feature already; previous Harmony generations could control the PS 3 through a dedicated IR to Bluetooth hardware adapter you had to buy for like $50, and couldn't control Wiis at all), but they've now updated the software so you can use it as a Bluetooth keyboard, which lets it control media center PCs without an IR dongle (and more reliably than IR). It would be nice if it could support IP-based remote control - for devices like XBMC computers, or (so I hear) Tivos - but I guess they're saving that for a future generation.

The smartphone app is...fine, for what it is. You get more controls than will fit on the hardware remote, obviously. The button layouts is customizable, but much less so than the Redeye one, which let you put anything anywhere, use custom button sizes and even shapes and graphics, and stuff. Several elements of the Harmony layouts are fixed and the UI wastes quite a bit of space, so you can only ever see a relatively small number of buttons at once and have to swipe between screens to reach other ones. But it's fine, it's not unworkable.

I was still programming my 880s via the very old website, because that method works with the Linux software, concordance. Logitech shipped a Windows-only app to replace that old interface, then shipped a new app for OS X and Windows, MyHarmony, to replace that app. I used that for programming; I was able to transfer my 880 config into the old app, and then transfer it from the old app to the new app and apply it to the Hub. A few settings got lost in translation somewhere along the way, but mostly it worked fine. You can also do the configuration from the smartphone app. Once you've made config changes, you can program the hub via a USB link with the MyHarmony software, or trigger a sync from the mobile app, in which case the config is transmitted to the Hub via wifi. (It seems a mystery why you can't trigger a wifi config update from the PC app, but oh well).

Logitech stores your configuration in The Cloud somewhere, and the concordance team has reverse-engineered the communications between the MyHarmony app and the Logitech servers and written a Linux client for the MyHarmony system called mhgui. This seems to work fine - I can edit my config in mhgui and then use the smartphone app to trigger a sync of the config to the hub. The concordance developers haven't fully figured out the sync process for the recent generations of Harmony devices, though, so you can't program the hub via a USB link to a Linux PC yet.

So far I'm pleasantly surprised - I thought this would be a just-bearable sideways/backwards move from a fully-functional 880, but actually it feels like a decent upgrade. The hub is actually more reliable than the 880's transmitter at long last, the whole setup works reliably without any breakage or much quirkiness, the programming software is fine, the Android app is decent, and the hardware remote is (as everyone says) so simple it's awesome. It feels like it should stand up to long use pretty well as well, but only time will tell.

These days, if you want something like a Harmony, a Harmony is pretty much your only choice. There are programmable remote apps for a few smartphones with IR transmitters, but they're kind of a joke compared to Harmony; relatively tiny device databases, and generally pretty bad at 'Activity'-type control where they have to set multiple devices to different states and control different devices as part of a single remote 'profile'. Thinkflood, the company behind the Redeye, went out of business. Samsung has a thing, but again, it's not close to Harmony standards. So it was kind of a problem that for a few years, Harmony devices seemed to be a long way below Harmony standards. But it seems like they've gone a long way to righting the ship again with the current gen of hardware and software, thankfully. I'd certainly recommend the Smart Control, and you can get it for a pretty damn reasonable price at present (I paid about CAN$120 for mine).

Today, Fedora 21 branched from Rawhide, and Rawhide became what will ultimately be Fedora 22. (See Branched and Rawhide wiki pages for more details on this process). For folks currently running Rawhide, I wrote up a quick guide on what to do if you want to follow Fedora 21 (if you do nothing, you'll wind up following Rawhide / 22).

All official Fedora 21 pre-releases and candidate builds (TCs / RCs) will come configured for the Fedora 21 repositories, so if you've been waiting for Alpha TC1 or Alpha to jump on the Fedora 21 train, you've got nothing to worry about. I filed the Alpha TC1 compose request today, so hang onto your hats for the first ever Fedora.next compose, and help us test it when it comes along!

In readiness for Alpha TC1, I did rather a lot of work on the release validation process late last week and early this week; writing new release criteria and test cases, revising the results matrix pages, and overhauling and consolidating the release validation test plan which documents the process. I wrote a few mailing list posts explaining the work as I went along, so if you want to brush up, there you go! The release validation test plan should serve as a pretty comprehensive overview/introduction to the process as it now stands, if you just want to know how it's going to work rather than see how it changed from before.

Hi folks!

So I've been working on Miscellaneous Stuff this week - a little release criteria drafting here, a little dealing with NetworkManager package splits there, a little discovering significant systemd bugs, yadda yadda - and still trying to get a Fedlet 3.16 build that works for me. 3.16 kernels still won't light up the screen on my Venue 8 Pro, but one of my Intel contacts says the latest kernel in the fedlet repo works for him. The difference may possibly be the firmware (he has updated to the latest firmware from Dell, I have not because the upgrade requires Windows and I blew my Windows install away).

I got a mail from a very nice person asking how he could help with the Baytrail hardware support work, and as part of our discussion, I wrote up a rather lengthy description of the kernel development process - mainly intending to explain how he can find the work being done on Baytrail hardware, but en passant it sort of turned into a general description of how kernel development gets done. I figured since I wrote it, I may as well put it up here in case anyone else finds it useful.

Bear in mind this is just one idiot outsider monkey's interpretation of how things work, and it very likely contains errors. Corrections happily welcomed!

For me, anyway, learning how big projects are put together was mostly just a matter of experience, but you can find actual guides for some prominent projects, certainly for the kernel.

So, let's look at Linus' kernel tree:

Let's take graphics first. The kernel graphics drivers handle quite a lot of work these days, more than they used to; as a rule of thumb anything to do with actually initializing the card and setting a mode is going to be in the kernel, for e.g. So, all the kernel graphics drivers live in drivers/gpu . Just about all the ones you're likely to care about live in drivers/gpu/drm.

The three Intel drivers there are i810 (for very ancient hardware), gma500 (for the horrible Poulsbo chipset that shipped with some low-power hardware a few years ago), and i915 (for just about all the Intel graphics hardware you're very likely to care about). So all the stuff for Baytrail graphics is in i915.

The code for detecting the displays on our hardware, for instance, is mostly in the intel_dsi files there, since the displays in our tablets are connected via DSI. That code all landed in the mainstream kernel in 3.16.

One thing that isn't so easily documented (or at least not that I've found) is where all the stuff that winds up in Linus' tree actually comes from. The setup is that there are maintainers for various areas of responsibility in the kernel, who maintain their own kernel git repos with their own set of branches. Periodically they'll tag one of their branches for merging into upstream (or just note a particular commit ID on a particular branch), mail Linus to request the merge, and he'll review the merge request and (usually) pull it in. (understanding the kernel maintenance setup is quite handy for understanding git, as git was written explicitly to back how kernel development works.) then all the subsidiary trees get synced back against the upstream masters and start working on the next batch of changes.

So, the guy who's in charge of graphics drivers is Dave Airlie, and here's his repo. Stuff from his 'drm-next' branch gets merged up into the mainline kernel during the 'development' / 'merge' phase, between the release of kernel 3.xx and the release of 3.(xx+1)rc1. Stuff from his 'drm-fixes' branch gets merged into mainline during the 'stabilization' phase, between rc1 and release. So right now, Linus is merging from Dave's 'drm-fixes' branch - here's the last such merge, for instance. Those were the DRM fixes between 3.16rc1 and 3.16rc2.

Now Dave, in turn, basically subcontracts development of each driver, so the whole setup repeats with sub-SUB-repos pushing changes up to Dave. So actual development of the Intel drivers ultimately happens here. Again, you see the '-fixes' and '-next' split. I can't quite remember why they have both 'drm-intel' and 'for-linux' branches, but they're usually in sync anyway. So more or less, the stuff that'll ultimately wind up as bug fixes for 3.16, you'll see going into for-linux-next-fixes, from where it'll get pushed up to Dave, from where it'll get pushed up to Linus. You can see right now a juicy change as the last commit to that tree.

'vlv' is one of the keywords you'll want to look out for in dealing with our hardware: it's short for "Valleyview", and is the codename usually used to refer to our stuff. You'll see 'baytrail' sometimes, but usually 'vlv'.

The other piece of the kernel development puzzle is the mailing lists. The big hairy one is lkml, the general kernel development list. You can find its archives in a wonderfully Web 1.0 format here. People all have different methods of interacting with it; you can of course subscribe directly to it, make sure you have a good email setup though, it's a high volume list. Some people like to read it through a gateway called GMANE that converts it into a Usenet newsgroup. Slightly confusingly, GMANE has a web interface, so you can also find LKML there, though that's an even worse interface than the lkml.org one in my opinion.

If all you're interested in is patches, there is an awesome shortcut I only found out about myself recently, called Patchwork. Patchwork basically monitors the mailing lists where kernel development discussion happens, sucks out all the patches, and gives you an interface where you can browse and query the mails with patches attached, and conveniently download them in either bare patch form or 'mbox' form (with mail headers attached, as many git commands expect).

There's a list for Intel graphics development, and that's where all the patches for our graphics hardware get posted for review. So you can find them in Patchwork here. By default it shows all patches sent to the list that haven't yet been committed to a tree Patchwork knows form a part of the whole kernel development setup. You can change the query in various ways by clicking "Filters". So we can search for 'vlv' patches and find, for e.g., this one - and finally we more or less have an entire kernel development workflow! Patches are mailed to this list, reviewed and discussed, committed from there to the drm-intel repo (either 'fixes' or 'next', depending on where we are in the cycle and the nature of the patch), then up to Dave then up to Linus and finally out to the world.

Several of the lists are interesting to follow to watch Baytrail dev work. Some patches just get sent straight to LKML. 'alsa-devel' is the list where all sound development happens (for our hardware and all other sound hardware; very high traffic). 'linux-acpi' is for, not surprisingly, ACPI; 'linux-pm' is for power management (you'd think there'd be an overlap with ACPI there, I'm not quite sure how that works, I just watch both).

So, you asked about sound. Sound used to be a bit different from everything else, as ALSA was technically an independent project, but these days it works more or less like everything else. Patch review on alsa-devel. There's an ALSA git repo but I believe it's not used for the kernel drivers any more. Instead, there's a repo that's approximately the equivalent of Dave Airlie's DRM repo, owned by Takashi Iwai, who has overall responsibility for audio.

Changes either go directly to tiwai, or to various subsidiary repos for different drivers and then up to tiwai. AIUI, the sound hardware in the Baytrail tablets is part of the 'ASoC' subproject (that's another keyword to look out for). tiwai pulls ASoC work up from Mark Brown's repo. Mark in turn pulls stuff for various specific chipsets from other folks at Intel. It seems that a lot of the work on our hardware comes from Liam Girdwood. I don't know precisely how they handle their merges, but I believe that's the general flow of work. I think some work on our stuff goes straight into Mark's tree or comes from somewhere other than Liam's, but I'm not entirely sure of the details.

Ryan Lerch points out that LibreOffice is running a "bug hunt" for its development version, and wants Fedora users to help out! You can get the official packages for the bug hunt here - I don't know if they'll accept results from the Rawhide 4.3.0 builds.

In other news - I know Fedlet has been quiet for a while, sorry about that. I basically wrote off the 3.15 kernel series. Graphics for the Bay Trail tablets only ever worked "by accident" in kernels prior to 3.16. In the 3.15 series, something regressed in that "by accident" support which meant the screen would never come back once it hit an idle timeout. This wasn't a priority for upstream because the support was accidental in the first place, and no-one managed to bisect it close enough for anyone to feel like fixing, so 3.15 was just a bad kernel series for us.

3.16 is the first series which should have official, on-purpose support for the displays in Bay Trail tablets (the problematic bit, if anyone's interested, is detecting the display itself - they're hooked up in an odd and previously unsupported way on these devices). So now 3.16 builds are hitting Rawhide, I'm back to working on fedlet. There is a 3.16 build in the repo already. Unfortunately, for me at least, it didn't work - the display just blanks when modesetting should kick in.

I'm now building a newer 3.16 (rc1+), and Rawhide got a new build of xorg-x11-drv-intel since my last attempt, so we'll see if that makes things better. If I get a kernel build where things work at least as well as on 3.14, I'll do new live images.

In Actual Work News, I've mostly been catching up with a huge backlog of devel@ list posts, and working on release criteria for Fedora Server (I'll also try to do the same for Workstation, and roshi should be working on Cloud). Here's my current draft. Release criteria work is always good for pointing up slightly problematic / undecided / questionable aspects of product design, so we've had a few list threads clarifying various aspects of the Server tech spec in relation to issues that came up as I drafted the criteria, and we had a very useful WG meeting this morning which gave me the next few revisions needed to the draft. Once we get a solid set of criteria nailed down, the next step is to write a manual test plan before we hit Fedora 21 Alpha. Of course, I hope we'll be able to get some automated tests done once Taskotron is in production.

I was poking a longstanding minor niggle in anaconda today, and it made me think about one of the principles I've come up with over the years I've worked on bugs, in various capacities.

To put it simply: it's often important to know the story of why something came to be the way it is. Just knowing that something is a certain way, and perhaps believing it should be otherwise, is frequently an incomplete position.

In the case of the bug above, without the historical context which is somewhat covered in the bug, maybe you'd read the dialog, and notice that the options mentioned in the text and the buttons at the bottom don't really seem to match up. You might even make a guess at why. But it helps a lot to understand the full context.

In this case, the text is from a previous incarnation of the same dialog, when the flow through the various possible paths of the partitioning UI was somewhat different, and different buttons were available in the dialog. It's pretty useful to know that the workflow changed around the text, rather than vice versa, or the text simply being written wrong, or a bug preventing the right buttons being shown, or any one of a thousand other ways the same situation could result.

I've come across many other cases like this; this is just one example. These days, when I come across any bug which is broadly a question of some kind of designed behaviour being odd - not simply a straightforward 'this app crashes when I do X' type of bug - my approach looks something like this:

1. Check if someone already reported the issue and if so, pick up that report, don't start a new one (this is always step 1).
2. Check the documentation. (This is always step 2).
3. Find the code that implements the behaviour, and read it. Yes, even though I'm really not a hacker. Always try this. Frequently, the kind of code that implements behaviour choices will be understandable, at least broadly speaking, to anyone with a rough understanding of how computers work, and if you're lucky, there might be an explicit comment explaining exactly what's going on, put there the last time someone asked.
4. Check the context and the history of the code. git blame is one of the greatest inventions ever, for this - not literally because it tells you who made the change (usually), but because it gives you the commit. Use git blame, read the complete commit, and read the commit message. Check the date on the commit, and go read the project's mailing list (or whatever publicly archived discussion format the project uses) from around that time; there may well be ancillary discussion there. Read around the area of code in question, it may matter. Follow the trail wherever it leads - go check other functions up/downstream of the one you started at if necessary, look back through multiple git revisions, just keep asking "why does this behave the way it does?" until you think you have a complete answer to the question.
5. If you can't manage to work it out from the code and the public records you can find, go find the relevant people, and just ask. If you know them already (and they don't hate you), this step should be easy. If you don't, it's still easy. Just drop an email to the address associated with the commit, say "hi, I'm Bob, I'm looking into this bug (URL) for this project (URL), it seems to be related to this code change you made back in 2008 (commit link), and I couldn't quite figure out why you wanted it to behave this way - could you help me out?" Usually, if you're polite and provide a full explanation of why you're asking, people will be happy to help.

I always find it fun when disciplines overlap in interesting/unexpected ways. As I've probably mentioned a few times, what school background I have (a glorious bachelor's degree!) is in history, which doesn't seem like it ought to be that relevant to the work I do now. Frequently, though, there's more of an overlap than you'd expect. This is one obvious example.

I've been a bad boy not blogging again; Google+ is just too easy. But it's also evil, so I've signed up for Diaspora again and am hoping it becomes less of a wasteland at some point. It's gotten a lot smoother in the last two years everyone forgot about it, so give it a look. I'm adamw@diasp.eu.

At the end of last month, I went to the always-awesome LinuxFest NorthWest in Bellingham. I always look forward to LFNW: it's organized extremely well and always seems to provide a really interesting bunch of attendees and talks. The always-fun-to-talk-to Bryan Lunduke was there, and I had fun hanging out with him and his motley crew (hi, James Mason!) and attending his long-running talk on why Linux sucks - watch the thing before you flame. One of the longest-standing and hardest-working Fedora QA contributors, Thomas 'satellit' Gilliard, dropped by on Saturday, it was a pleasure to meet him at last and thank him for all his work. Jeff Sandys did sterling work manning the Fedora booth for most of the weekend, Joe Brockmeier was there and was responsible for the Fedora sponsorship of the first evening games night (which was a great success), and I managed to make it to the legendary Jesse Keating's talk on Rackspace deployment techniques. Jakob Perry has been involved in running the event I think since the very first one way back when, and I was as impressed as always by his organizational skills and seemingly infinite energy supply, and honoured to extend my one-day-per-year guest membership in his Sunday afternoon board game club!

I went to several talks, but a couple stick out in my memory apart from Jesse's and Bryan's. Alex Jordan's introduction to Arch, which I went along to because I knew very little about Arch and figured it'd be good to know more, was excellent - he managed to concisely and clearly present a very large chunk of information in the time available, and I came out feeling like I knew a lot more about the Arch design and approach than I knew going in. Owen DeLong's session on IPv6 was also very well executed. Every time I read about or attend a presentation on IPv6 I feel like I've got about another 10% of the way towards actually understanding it...just another five or six and I might have a shot. :)

The Saturday night party at the Spark Museum was also great - I'd missed the previous LFNWs where there were events here, so it was my first time. The museum's collections are fascinating, not just for hardcore electrical/electronic nerds but also for liberal arts dilettantes like myself; there are some really interesting elements of social and historical context to the gadgets, and they have some really fun stuff like Rudy Vallée's first 'big' megaphone, signed and inscribed to Dick Clark! (There's a photo of it in the Wikipedia article). Of course, the main attraction is their giant four million volt Tesla coil, which they demonstrate by letting a few insane volunteers stand in a Faraday cage which they proceed to zap with the coil while the volunteers either cower on the other side or grab hold of the internal bars closest to the coil and laugh maniacally, depending on whether they're called Bryan Lunduke or not. I have no idea how the museum ever got the safety clearance - especially to do it with an audience well-lubricated with free (both ways!) beer courtesy of the WiseAss beer folks (whose open source beer is fantastic, and whose funding campaign you should definitely donate to).

I did a presentation of my own on UEFI, aiming to cover both the 'theoretical background' material from my blog post on the topic and the 'practical' material from the Fedora wiki page I wrote. It turns out to be a lot to try and communicate in an hour, even with a Real Actual Honest To God Slide Deck, and I felt like I rather skated over some of the theoretical material in particular, but I had a pretty full room of folks (including Jon Hall, so no pressure there) and none of them started throwing things or left in disgust, so I think it went OK. I'm going to try doing it again at Flock, if I got enough votes, so we'll see how that goes.

The news on Fedlet is kinda mixed. I've been building 3.15 kernels periodically. A lot of the changes I had to carry as patches have been merged upstream in 3.15, which is great. There is also now some work by a guy named Doug Johnson in this bug on getting the SDIO bus operational on the Venue 8 Pro (and probably other devices affected by a known Intel chipset bug (VLT37 in this PDF)). However, 3.15 causes a rather major other problem with X - the display will die apparently on any mode switch, or RandR operation (like a rotation), or if you just let it time out due to inactivity and then try to wake it up. This shouldn't be a problem once the actual 'correct' support for the Baytrail tablet displays is merged probably in 3.16, but it sounds like it may be difficult to fix for 3.15 unless someone like me pulls their finger out and does some git bisections. Due to this X issue, I haven't done any new image builds for a while, as it just seems like too big a bug to 'ship a release' with. If you're willing to live with it, the 3.15 kernel builds are available from the fedlet repo (I need to do a new one soon).

In other Side Project news, I'm still meaning to stand up an OpenTripPlanner server for Vancouver Real Soon Now(tm), I just never quite seem to get around to it. The LFNW IPv6 talk I mentioned above has kind of added to my desire to do an overhaul on my network infrastructure, too. Right now I have a kinda circa-2006 'everything as 192.168.1.x NATted behind a WRT150N running dd-wrt' system (with a WNDR3700v2 acting as a wireless AP for $REASONS) which more or less works, but...I kinda feel like moving to an 802.11ac-capable router running OpenWRT, using more internal subnets a la CeroWRT's default setup, making sure all my 'private' services have secure authentication of some kind and making the whole shebang directly globally addressable via IPv6. As I'm on a business account I may be able to get native IPv6 from my ISP (have to call them and ask), or else I can use a tunnelling service of some kind. This would be more F/OSS-y, much more Awesome Bleeding Edge Cool, faster for 802.11ac clients, and probably more secure than my current setup if I got it right (I'm getting less and less happy about how I have some services where I just trust Windows and Android boxes connected 'inside the router'). Plus at least in theory I could access all my stuff/services from anywhere, no more 'things that only work behind the router'. I should probably also split up the mixed bag of public and private stuff I have running on this box (www) onto multiple server VMs. Or containers! I hear containers are cool now.

I took apart my older laptop and replaced its dying SSD with one of these the other day. Lots of fun with interfaces there - the laptop predates not just mSATA but also the now-more-or-less obsolete micro SATA so its original drive used some oddball connector, so you have to replace the motherboard-to-drive ribbon cable with one that has a micro SATA connector (there's no cable available with an mSATA one), then connect the mSATA SSD to an mSATA to micro SATA adapter so you can plug the ribbon cable into it. Whew. But after doing all that, the number of times the system spews a bunch of ATA error messages, remounts root as read-only, and then refuses to boot up cleanly again has dropped to around about zero, so I'm happy (and the modern SSD does seem a bit faster than the old one the system came with, even though there's probably some interface bottlenecking going on). As part of the Great Network Infrastructure Upgrade I may push my luck and try upgrading both laptops with 802.11ac-capable mini-PCIe wifi adapters - i.e. this one, which seems to be more or less the only game in town.

Finally and much the least interestingly, boring old actual-work news...

I've been kind of in grazing-around mode for the last few weeks, doing little things here and there. Running Rawhide on a couple of boxes, fixing up builds, filing miscellaneous bugs. We're at the point where Wayland actually works on my newer laptop, which is a bit of a shock...at least it fails to wake up from an idle timeout and has glacially sluggish touchpad response, I'd be worried if it was actually usable or anything.

I spent some time revising several of the storage-related test cases, and implemented Kamil's idea of doing entirely optional monthly Rawhide validation matrices - here are the May 2014 Installation and Base matrices. These will let us at least keep an eye on the status of Rawhide during the extended period between Fedora 20 release and Fedora 21 Alpha, make sure nothing's gone too far sideways in the interim. I wrote up a 'test outline' for the Fedora.next Server product, and have been spending quite a bit of time thinking about how we should re-examine the role of Fedora QA in the Fedora.next context. I'm broadly aiming to take the 'test outlines' for Server, Desktop and Cloud products and combine them with our existing validation matrices into a comprehensive formal test plan for Fedora 21 (we've never really had a full test plan before) and use that as a jumping-off point for the debate about how various testing responsibilities should be organized between 'Fedora QA' and the various product WGs. So, interesting times!

Wow, this makes me happy!

edit: If you have an XPS 13 of the current-gen Haswell variety - or an XPS 12, or possibly other similar Dell systems - you aren't affected by this issue. You may also have trackpad weirdness on such systems, but it's not this bug, it's a different one. If you have the Haswell XPS 13, try blacklisting the i2c-hid module.

tl;dr version: if you have a Dell XPS 13 of the last-gen Ivy Bridge (not current-gen Haswell) variety, or other system with a 'CyPS/2 Cypress Trackpad' clickpad, and you have trouble trying to middle-click or use custom synaptics configurations or even with normal clicks with the new synaptics 1.8 code, stick something like this as /etc/X11/xorg.conf.d/99-clickpad.conf:

Section "InputClass"
        Identifier "Disable clickpad for touchpad"
        MatchDriver "synaptics"
        Option "ClickPad" "no"
EndSection

and you should be happy: everything should work properly with synaptics 1.8, and middle-click emulation (by pressing left and right softbuttons together) should work!

I spent the last couple of days tracking down a frustrating problem with the trackpad on my laptop, with the extremely kind and patient assistance of the awesome Peter Hutterer. The debugging was tedious and annoying, but once we figured out what was going on, it was sure worth it.

I've hated the trackpad on the system since I bought it, mainly because I've never been able to get a middle click working at all. With clickpads - trackpads with no physical buttons, where you can click the pad itself - you're supposed to be able to get a middle-click by doing a three-finger click, or you can configure various other things (like a particular corner of the pad or something) to do it. But none of those seemed to work right for me.

I was managing to live with that, but with the recent changes to the 'synaptics' X driver (leading up to the release of 1.8), things seemed to get a lot more broken. Clicking at the bottom left of the pad didn't always seem to generate a left click, and with some builds, right clicks weren't working at all. It was pretty frustrating.

In the end, in debugging the new issue, we figured out the underlying cause and it turns out to have been causing the older issues too. This particular hardware is smarter than your average clickpad: its firmware tries to do a lot of the typical clickpad tricks which other clickpads leave to the OS driver.

Apparently most clickpads are pretty dumb devices: they just track the number of fingers, finger location and pressure across the whole pad, and send out an event when you click the pad. It's up to the OS driver to implement stuff like 'make it a right click if the click was in the bottom-right hand corner of the pad', 'make it a middle click if it was a three finger click', etc etc.

However, it turns out that the firmware on this clickpad does rather more. First, it implements its own soft button area - let's call it a 'firmware button area' - at the bottom of the pad, which behaves rather differently from the rest of the pad. If you click in the bottom right it actually sends out a right click event (other clickpads don't differentiate, they leave it to the driver to do the clever stuff). If you make initial finger contact within this area, the driver doesn't register the 'tap' or pressure or even the finger location at all until you move the finger around a bit (outside this 'firmware button area', it starts tracking those things immediately). The firmware also implements 'click with multiple fingers sends a right click event'. Again, other clickpads don't do that.

So ultimately all the bugs turned out to essentially be symptoms of the same underlying problem: the pad's firmware and the synaptics driver were kind of fighting each other. Synaptics 'knows' this pad is a clickpad, so it tries to do all the clickpad 'clever stuff' - but the firmware is already doing it too. Their implementations of button emulation were clashing, causing the problem with left clicks not working. Trying to configure a multi-finger click as a middle click didn't work because the firmware already detects multi-finger clicks and sends a right-click event, which synaptics doesn't expect. And so on.

Ultimately, the solution turned out to be simple: just tell synaptics not to treat the pad as a clickpad at all, and rely on the firmware implementation. That way they're not fighting. You lose all the flexibility/configurability of synaptics' behaviour - you can't use all the neat synaptics config options relating to clickpads - but at least the basic stuff works.

And most awesomely (for me), with synaptics not treating the pad as a clickpad, middle-button emulation works! If I click the left and right 'buttons' implemented by the firmware together, I get a middle click. This makes me very happy.

So yeah, if you have a similar 'smart' clickpad (my system is the previous generation of Dell XPS 13 developer edition, I'd guess other XPS 13s have the same hardware but can't be sure, and I expect there are other laptops with the same pads in them out there), try the xorg config snippet at the top of this post. It might make you happy.

Peter will probably implement a quirk for my pad in future synaptics releases. If you have a different pad that behaves similarly, it might be good to file a bug at Freedesktop to get a quirk added for that, too.

Long time no blog, but I had to blog my admiration for the work Marie Nordin has done on Fedora Badges as part of her OPW internship. Just look at that list - and all amazingly high quality work, too. Congratulations and thanks a lot to her. I wish I'd had more time to help with Badges lately, but I just seem to keep piling up the hobby projects, inside and outside Fedora...

Speaking of which, I haven't got around to blogging about it, but I have been keeping Fedlet up to date. The major improvement recently has probably been sound support; it's a bit experimental and it doesn't work out of the box as the required firmware has not yet been released under a sufficiently redistributable license, so you have to go out and find the firmware for yourself and put it in the place the driver expects. Once you do that, though, sound does work.

Quite a few folks at Intel are working hard on the Baytrail support, but it's mostly groundwork right now, no obvious Big Leaps Forward aside from the sound support. However, the groundwork should lead to stuff like the sensors and hardware buttons working and battery monitoring working quite soon. I'm still hoping someone's going to Do Something about SDIO device enumeration on the Venue so I can maybe get the internal wifi going, but there's so much work going on in so many different trees it's getting tricky to keep track!

In "actual work" news, things lately have mostly been focused around firming up the Fedora.next plans. I've been doing my best to be productive on the Server working group and keep track of the Desktop and FESCo discussions also. We've thrashed out some useful bits about default filesystems and stuff lately, and are currently considering tricky questions like how to handle variant default configurations between Products. I'm pretty sure things are moving in a good direction in general, but it'll be nice to get some concrete Stuff built soon.

In the mean time I've been fiddling around on various minor themes - today I built a Rawhide live image with efibootmgr 0.7 to test it for pjones, for instance, and incidentally caught a couple of other bugs. I've also been testing our planned OwnCloud 6.0 update for Fedora 20 - it looks like we still have a bug with migration of PostgreSQL-based instances from 5.x to 6.0 to fix before that can go out.

I also spent a couple of days looking at updates with dependency issues - specifically, looking through updates with AutoQA depcheck failures and seeing how many were false alarms, and how many of the genuine failures caught by AutoQA had actually been fixed up prior to the update going out. It turns out there've been several cases beyond the more high-profile recent ones where updates went out with dependency errors, so I got the ones that were still broken cleaned up, and got FESCo to sign off on a plan to disable karma automatism for updates where AutoQA checks fail. We just need the Bodhi devs to implement that now. We're hoping to have Taskotron in place relatively soon, with an improved depcheck test that will be sufficiently reliable that we can simply block updates from being released if it fails, but until then this will serve as a stopgap.

So, today's headline news: I finally stopped foot-dragging and put up a public image of Fedlet, my Fedora remix for Bay Trail based tablets. If you have a Dell Venue 8 Pro, Lenovo Miix2, or Toshiba Encore, you can go to that link, grab the live image, and give it a shot. If you have an Asus T100 or something I haven't heard of yet that does not have 800x1280 as its native resolution, hold your horses - that image forces the 800x1280 resolution that is correct for the V8P etc. I'll build an image with 1366x768 resolution forcing (for the T100) tomorrow, probably.

You get a clean(ish) boot to GNOME Shell with working touchscreen and 3D acceleration (and video playback acceleration, if you can legally install the necessary driver - details on the page) and a silly little tool to do screen rotation. No wireless, or bluetooth, or sound, or power status, or accelerometer.

You can get a USB OTG dongle and a small USB wireless adapter to get wifi without making it too un-tablet-ish. I recommend an Asus USB-10N, works out of the box for me.

It's based on Rawhide and is hilariously experimental in all ways, no guarantees whatsoever, lock up your babies. But it works for me!

All the backing code, patches, ugly hacks and packages can be found in this github repo and this package repo. I forgot to put a repo definition in v1 of the image - I'll probably tweak that tomorrow - but it's probably a good idea to enable my repo and skip kernel updates from the official repos, as stock kernels won't boot on these devices yet.

Some very smart folks at Intel and RH and GNOME and elsewhere are working to fix up all the various other bits that aren't quite right yet. At least, if they ever want an end to my incessant bug reports, they are. ;) I'll try and put out updated packages and images regularly.

In other news, this week is more or less my fifth anniversary working for Red Hat. I wasn't going to mention it, but I loved Ricky Elrod's post on the same topic. I second everything he has to say there.

In addition to that - in the last couple of weeks I've:

• Penned epic blog posts and wiki pages on UEFI and fonts and Pete knows what else
• Done a lot of fiddling with the Fedlet
• Ported OwnCloud's Google Drive plugin to a new version of the Google PHP library
• Taken part in lots of enthusiastic discussions on Fedora.next
• Worked on OpenStreetMap
• Contributed some small bits to the installer
• Helped out with bits of the GNOME 3.11.5 rebuild for Rawhide
• Worked on Fedora's OwnCloud packages

and that's just the stuff I remember. And my job title is 'Quality Assurance Engineer'. I've been incredibly lucky to find, in Mandriva and Red Hat, two employers where all this comes under the heading of 'work', or at least doesn't get you fired. I wake up most mornings and don't eat breakfast for two hours because I'm having too much fun at work, and I know that's a heck of a blessing. So thanks very much for putting up with me, Red Hat and the Fedora community, and I intend to stick around until they drag me away by the feet. :)

Yesterday I penned another tome on UEFI. In contrast to my blog post, this is more of a practical guide to installing Fedora on UEFI-capable systems: it doesn't go into the technical details of how UEFI works, it's more just about what you should and shouldn't do when installing Fedora to UEFI-capable systems in various circumstances.

On a related topic, if I can put in a word on so-called "universal" tools for writing bootable images - like Fedora live or install images, for instance - to USB sticks: they're never actually "universal", and it's usually a bad idea to use them.

I'm talking about tools like UNetbootin and Universal USB Installer. Don't use them, and don't let your friends use them. Mail them to your enemies.

I haven't looked into them all very deeply, but everyone who's done support for a Linux distribution can probably tell you that they're often the cause of "mysterious" failures in booting USB sticks. A lot of them, as far as I can tell, seem to just dump the contents of the image onto the stick and then stick a syslinux bootloader onto it.

This comes with myriad problems. I haven't seen a "universal" USB stick writer which came close to writing something that would boot in native UEFI mode, for instance, never mind boot properly on a Mac. By contrast, distributions like Fedora and SUSE and Arch quite carefully tune their images so that if you write them with a dd-style tool, or the distribution's supported USB stick writing tools, you will get a stick that does boot in all these ways.

Although they usually manage to write a stick that boots in BIOS-native mode on a PC, you can't rely on it booting properly, or being reliable for doing installations. For instance, the way non-live Fedora installation images are set up to boot from USB is for the early init stage to look for a filesystem with a particular label to find the installer itself - if you look at the kernel parameters when booting a Fedora DVD or netinst image written to a USB stick, you'll notice something like inst.stage2=hd:LABEL=Fedora\x2020\x20x86_64, and that's what that means.

When you write a Fedora image to a USB stick with a correct tool, it will apply the correct label to the correct filesystem to make this work. If you use a "universal" tool, it may well not, and you'll be left wondering why the boot fails with an error like Warning: /dev/root does not exist. This is why.

So: don't use these "universal" tools, or at least if they fail, don't report it to your distributor, report it to the tool. For Fedora, the guide to writing images to USB sticks the safe way is here (it's also covered in the Installation Guide, though that's usually a bit behind the wiki page). SUSE has three pages: Linux, Windows and OS X. Arch's page is here. Ubuntu's is here. Mageia covers it along with other media types here - their images are hybrid like Fedora's, SUSE's and Arch's, so the graphical direct write utilities described in the Fedora, SUSE and Arch instructions should work for Mageia as well.