Welp, turns out there was one last (I hope!) brown paper bag bug for Fedora 20: fedup 0.7, which was in the Fedora 18 and 19 stable repos today, can't successfully upgrade to Fedora 20 any more. It certainly could last time we tested it, so this has blindsided us a bit, or else we'd have been better prepared! Fedup 0.8, which is currently in updates-testing but will be in stable for Fedora 19 tomorrow and for Fedora 18 just as soon as we can get it pushed, can upgrade to Fedora 20 just fine. If you want to upgrade but haven't got around to it yet, just make sure you use fedup 0.8, and you'll be fine. Run 'yum --enablerepo=updates-testing update fedup' to get 0.8, or you can use the graphical package management tools to enable updates-testing, install fedup, then disable it again. If you already tried with 0.7 and it failed, just upgrade to 0.8 and try again, but you may want to do 'mv /var/lib/fedora-upgrade /var/lib/system-upgrade' and 'mv /var/tmp/fedora-upgrade /var/tmp/system-upgrade' first (fedup's file download locations changed in 0.8, and this will save it needlessly re-downloading the upgrade packages, and make sure it cleans up after itself properly when it's done). If you already had a failure with 0.7 and a success with 0.8, you might want to check for /var/lib/fedora-upgrade and /var/tmp/fedora-upgrade and wipe them if they exist - they won't hurt anything, but you don't need them and they're just wasting disk space. Sorry for the mess, folks!

Well, it's nearly the big day: Fedora 20 will be released tomorrow. It was another somewhat chaotic release here in Fedora QA, and I'm currently firing off proposals to try and improve things for us in future at a high rate of knots, but if you don't peer into the factory I think you'll probably find the sausage delicious :) Yes, the week of all-nighters and last-minute fixes mostly paid off, and Fedora 20's looking like a pretty solid release. There are one or two 'morning after' bugs, but then there usually are, with something as big as a distribution release. If you want to try the new LVM Thin Provisioning support, be prepared for your system failing to boot after installation - it's not too difficult to work around, but I wish we'd caught it. On the positive side, network installs shouldn't be affected, as we sent the update to fix the bug out already. There's also a somewhat complex bug for people who want to use an ISO-based remote installation repository (Fedora's very flexible with installation repositories, you can install from a 'repository' which is just an NFS, FTP or HTTP server with a copy of the DVD ISO file sitting on it). That one is easy to work around, too, and not that many people really use that installation method anyhow. There's a bunch of yum bugs causing it to print various spurious error messages about non-existent groups; you don't really need to worry about these, they're not terribly harmful, but they do look a tad embarrassing. All the gory details are in the common bugs note, so read that if you're interested in the issue and the workarounds. Oh, and don't run 'yum groups mark convert', whatever yum suggests to you. These bugs also exist in Fedora 19, so at least 20's no worse :) And finally, you probably don't want to deploy a production 389 Directory Server / FreeIPA on Fedora 20 until a few different bugs are fixed, again all described on the common bugs page. The devs are working to fix those as fast as possible, updates will be available soon, and after that FreeIPA on F20 will be just fine. The client-side stuff in F20 seems fine, I have two F20 systems running as FreeIPA domain members. That might sound like a bit of a laundry list, but yep, a Linux distribution's a big thing, and this is nothing out of the ordinary! For most people in most use cases, F20 will be a great release, I think. So far the reports from early adopters have been very positive, and I think there are some worthwhile improvements in various bits of F20. GNOME 3.10 is a nice improvement over 3.8 (unless you really loved having a wired network status icon...), particularly in the Online Accounts support - if you haven't checked out that feature, please do! We've made anaconda much better at handling keyboard configuration for non-U.S. English users with this release - it's probably the first newUI that's better than Fedora 17 overall for this. Systemd and anaconda and a lot of the other "new bits" from recent Fedora releases have got bug fixes and useful feature improvements, and there aren't any huge messy changes to introduce big problems. Of course, now it's done, it's old and boring - I already upgraded my desktop to Fedora 21! But if you're not crazy, do grab Fedora 20 tomorrow and give it a shot, we think you'll like it. Fedora 21 promises to be an interesting cycle, with the Fedora Next changes... Whatever they turn out to be. We're waiting on January, when the working groups report back to fesco, to see what will happen.

An update to my prior post's numbers on memory consumption: it seems a large chunk of the apparent increase in GNOME Shell's memory usage in Fedora 20 is, somehow, related to the fact that I tested in a VM. Testing on metal does not reproduce the same numbers: I see 173MB for Shell in top, only slightly higher than F19's 160MB in a VM. I haven't yet tried F19 on metal, but I will. So far the theory is this is due to llvmpipe-based software rendering of the Shell being used in the VM. F19 was already using llvmpipe, so this would mean there must have been some kind of change in its memory use or memory use accounting between F19 and F20, but the situation does seem to be more complicated than just 'Shell uses a ton more memory in F20 than F19'. Many thanks to Jasper St. Pierre and Matthias Clasen for guiding me through a closer investigation of this, and our work has already borne some fruit: I also did some valgrind profiling, and that led to Jasper finding some issues with desktop background loading that should save 12MB or so in my test configuration. I'll keep on poking at this if I can find time, but right now work on F20 is taking priority. Unfortunately, it's looking a bit like we may have to slip the Final release by a week, but we'll make the final decision on Thursday...

It's all a bit quiet around Fedoraland today with U.S. Thanksgiving happening. So I took some hours out today to do some comparisons of a few key things between a bunch of Fedora releases: 13, 15, 17, 19 and 20. Let's take a look at the numbers, Jim! All the raw data is available here. For every release, I used the same KVM VM (specced with 2 CPUs and 2GB of RAM), running on a Fedora 20 host. For Fedora 13 I used cirrus graphics with VNC (the old default), as it cannot handle qxl/SPICE; for all the others I used qxl/SPICE. I did an install using all default choices from the DVD image. Fedora 20 used Final TC3, all others were Final releases.

'free'-reported memory in use from console after boot to runlevel 3

Fedora 13: 129948 Fedora 15: 144788 Fedora 17: 119616 Fedora 19: 153644 Fedora 20: 148720

'free'-reported memory in use from GNOME terminal after boot to runlevel 5, create user, log in

Fedora 13 (GNOME 2): 276104 Fedora 15 (GNOME 3 fallback): 297680 Fedora 17: 389672 Fedora 19: 534376 Fedora 20: 691872 We can see that the memory used when you simply boot to a console and log in has changed very little all the way back to Fedora 13, released 2010-05-25. We're doing a fairly good job of keeping our base system from bloating excessively. 19 and 20 are both 30MB worse than 17, but then, 17 was 25MB better than 15. UPDATE 2013-12-03: Note regarding the following section about GNOME's memory use: the situation may be more complex than it seems. Testing on bare metal does not produce the same results as testing in a VM, and the use of llvmpipe-based software rendering in the VM may be throwing the numbers off; it's possible F20 isn't significantly worse than F19 in this respect after all. See this post. The same certainly doesn't hold true for the graphical desktop, though. Just sitting at a mostly-idle desktop with a terminal open, our memory usage has gone from 275MB under the ancien GNOME 2 regime to 300MB with GNOME 3's 'fallback mode' (which was more or less GNOME 2), then rocketed to nearly 400MB, 535MB, and nearly 700MB in subsequent releases. I haven't yet looked in detail at the changes, but I did take screenshots of 'top' ordered by memory usage for each install. Fedora 13: Xorg 24M, gnote 21M, python 19M, nautilus 17M, gnome-panel 16M, clock-applet 15M, wmck-applet 13M... Fedora 15: Xorg 53M, systemd 22M, gnome-settings-daemon 18M, clock-applet 15M, gnome-panel 15M, gnome-session 14M, gnome-terminal 14M, wnck-applet 12M... Fedora 17: gnome-shell 115M, Xorg 53M, systemd 24M, gnome-settings-daemon 21M, gjs-console 19M, gnome-terminal 17M, tracker-store 16M, libsocialweb-something 16M, goa-daemon 15M, nm-applet 14M... Fedora 19: gnome-shell 160M, Xorg 26M, evolution-data-server 26M/25M/23M/22M (4 processes), gnome-settings-daemon 22M, goa-daemon 22M, gnome-shell 20M, firewalld 20M, libvirtd 18M, gnome-terminal 18M, tracker-store 17M... Fedora 20: gnome-shell 328M(!), evolution-data-server 56M/28M/22M (3 processes), gnome-settings-daemon 30M, Xorg 26M, systemd-journald 22M, gnome-shell 21M, firewalld 21M, goa-daemon 20M, gnome-terminal 19M, libvirtd 19M, tracker-store 17M... these are all the raw 'RES' numbers, which don't really tell the whole story, but they're close enough for a ballpark. The obvious things that jump out are that we don't really see the memory usage of particular elements of the desktop increasing over time - where the same processes occur in multiple releases, memory usage is usually pretty similar - but GNOME seems to be running more and more stuff by default over time (there's a couple of new basesystem bits too, but we can tell from the runlevel 3 numbers that the net impact there is small), some of it adding significantly to the memory burden. GNOME 3 is also clearly heavier than GNOME 2. The obvious big exception is that the memory usage of the Shell itself grew massively - in fact, more than doubled - from Fedora 19 to Fedora 20, accounting for all the overall increase in memory usage from 19 to 20 by itself. gnome-settings-daemon has also been growing on a more modest scale, from 18MB at F15 to 30MB in F20. I don't know if anyone in GNOME land is focused on efficiency, but these numbers suggest that it might be a good idea to spend some time on it. Some of the increase is probably unavoidable with GNOME's increase in capabilities over time, but some of it is probably fixable.

Size of installed system

Fedora 13: 3GB Fedora 15: 3.1GB Fedora 17: 3.5GB Fedora 19: 3.4GB Fedora 20: 3.5GB Well, nothing much interesting going on there.

Loaded services at runlevel 3

Fedora 15: 56 Fedora 17: 42 Fedora 19: 38 Fedora 20: 47 Seems like we were gradually reducing the number of services loaded at boot from 15 through 19, then reversed course for 20. This is one where the raw numbers are a blunt instrument, though, and you have to look at details. Manually comparing the 19 and 20 lists doesn't reveal any huge oddities; it's mostly just that more things which were happening at startup anyway appear as discrete systemd units now. One that does seem a little odd is that 19 did not have ModemManager.service running by default, but 20 does.

Package set

It's kinda fun to diff the default package set from release to release; I saved a full list of packages installed (without versions) in each release. Fedora 13: 1204 packages Fedora 15: 1204 packages Fedora 17: 1209 packages Fedora 19: 1237 packages Fedora 20: 1309 packages There's a lot of detail in there, of course. For fun, here is a diff between the Fedora 13 and Fedora 20 package sets. A bit more usefully, here is a diff between the Fedora 19 and Fedora 20 package sets. The additions in 20 look to be kinda split between new functionality and added deps for existing things.

Kernel size

Fedora 13: kernel 3.5MB, initramfs 123MB Fedora 15: kernel 3.9MB, initramfs 147MB Fedora 17: kernel 4.7MB, initramfs 174MB Fedora 19: kernel 5.1MB, initramfs (generic) 266MB, initramfs (stripped) 118MB Fedora 20: kernel 5.1MB, initramfs (generic) 376MB, initramfs (stripped) 121MB In F18 or F19 (I forget which) we stopped using a generic initramfs - one with every possible thing in it - by default, and started stripping the initramfs to contain only stuff needed for the installed system. We install a 'rescue' initramfs (and boot menu entry) which is generic at install time, so if you change your system's hardware in a way that the stripped initramfs can't cope with, you have something to boot from to generate a new stripped initramfs. That's what the "generic" and "stripped" numbers are. As we can see, the kernel+initramfs have sure gotten bigger over time; I'm not qualified to speculate on how much is additional capability and how much is bloat, but it might have been nice to compare an 'lsinitrd' for each release (unfortunately, I didn't think to save one). The 'stripped' initramfs plan was a bit controversial, but these numbers at least illustrate that it certainly serves a need. Remember, the initramfs is loaded into memory at boot time, so these numbers translate directly into unavoidable memory consumption for the booted system. It's interesting that F20's generic (rescue) initramfs is so much larger than F19's.

Boot speed

Only have numbers for 19 and 20 here, as systemd's handy-dandy 'analyze' capability didn't exist for 17 and earlier, and I wasn't going to bother manually doing bootchart. Fedora 19: 16.720s (console), 18.809s (graphical) Fedora 20: 14.977s (console), 17.571s (graphical) Fedora 19 console bootchart Fedora 20 console bootchart 20 got a second or so faster than 19, which is nice. Both are pretty snappy. Of course, I'm running on a host with lots of RAM. The guest disk images are stored on my fairly fast NAS.

Installer memory consumption

For 17, 19 and 20 I ran the installer memory profiler that Chris Lumens wrote and I blogged about before. Here are the charts. Fedora 17 raw memory usage data Fedora 19 raw memory usage data Fedora 20 raw memory usage data The broad shapes of each graph are very similar; it's obvious that there haven't been really major changes in the way the installer uses memory between these releases. We did kill the thing that generated the huge peaks early in install in Fedora 17 (it was an SELinux policy thing, IIRC). The later peaks in all three graphs are packages running gtk-update-icon-cache and update-mime-database in their %post or %posttrans. It's somewhat interesting that the peaks seem to flatten out at the tops in the F20 graph, but I don't know yet what's behind that exactly. Fedora 20 is slightly heavier on memory usage than F19 was overall. There is no obvious smoking gun behind this, though - as noted, the overall shapes of the graphs are very similar. Fedora 19's installer initramfs is 328MB, Fedora 20's is 343MB; this probably accounts for 15MB of the difference. The absolute peak usage for Fedora 19 is 704156 (704MB) during the last gtk-update-icon-cache run (the previous one hits 703780), for Fedora 20 is 775364 (775MB) during the second-to-last gtk-update-icon-cache run (the last hits 761724). Right after the final gtk-update-icon-cache run, and before the gradual ramp-up in usage that happens during yum's 'verification' step, F19 is at 583MB, F20 is at 642MB. So that's about 60MB, minus the 15MB initramfs, that's gone somewhere between 19 and 20. So...there's a bunch of data, I guess. =) Hope it was interesting!

Getting really tired of the 'systemd is Red Hat's plot to...(anything you like here)' meme. Look, folks. Here is the canonical location for systemd source, the fount of the project. You will note that it is on freedesktop.org - not a location owned by Red Hat. Let's look at a core systemd file. What does the header say?

"This file is part of systemd. Copyright 2010 Lennart Poettering Copyright 2013 Marc-Antoine Perennou"

Do you see "Red Hat" there? No, you do not. Let's compare with Upstart. Well, it lives here. launchpad.net is a domain owned by Canonical. Here's a core upstart file. What does the header say?

"upstart Copyright © 2010 Canonical Ltd. Author: Scott James Remnant [email protected]"

Notice the rather large and significant difference? systemd is copyright Lennart Poettering. It is not copyright Red Hat. Upstart is copyright Canonical.

Lennart works for Red Hat - right now. He is free to leave at any point. If he does, he still owns systemd. Red Hat does not. systemd would still live at freedesktop.org and Lennart would still control the project. He could go work for SUSE or Canonical or anyone else he liked, and systemd would still be his project. Not Red Hat's.

Upstart is Canonical's project. It lives in Canonical's domain and is copyrighted by the company. I believe, if Scott leaves Canonical, he does not retain control of upstart.

So no. systemd is not Red Hat's secret plot to...(anything at all) because Red Hat does not own systemd. Lennart does.

I've always thought it'd be kinda cool to be able to run Fedora on a tablet. There are some others who'd like this too, and we've had the 'Fedora Mini' and 'Fedora Mobility' sub-projects sort of targeting this in the past, but we never really quite got there. I call this mythical tablet on which Fedora works well 'the Fedlet'. We still haven't quite got there...but there are some interesting new possibilities arising. There's a new Intel platform commonly referred to as 'Bay Trail'. It's basically a very low power Atom CPU with a chipset and GPU all optimized for low power consumption, aimed at tablet use. It's a proper x86 CPU and, in contrast to earlier low-power Intel platforms, it uses Intel's own graphics architecture, not a nasty PowerVR chip (remember those things?) This means it's at least theoretically feasible that Bay View-based systems could be viable targets to run mainstream Linux distros on. In the last month or two, Bay Trail-based systems have started showing up in the market (all pre-loaded with Windows 8.1). One that a lot of people seem to be playing with is the Asus T100 - Phoronix took a shot at it, this guy kinda got it working, and there's discussion on XDA. But that's not the only one out there. There's also a crop of 8" tablets: the Toshiba Encore, the Lenovo Miix 2, and the Dell Venue 8 Pro. I read a review of the Venue 8 Pro a few days ago and was intrigued, and did a bit of research. Today I happened to notice that my local Best Buy had it in stock. So, pretty impulsively, I bought one. I've spent this afternoon trying to boot Fedora on it. Executive summary: it doesn't really work. But it's interestingly close. The first and biggest bear trap with, as far as I can tell, all the Bay Trail-based hardware available so far is that they use 32-bit UEFI firmwares. Yes, they didn't listen to Matthew. (I've checked; he's weeping over a bottle of gin, like he does every day of the week with a 'y' in it.) Up till now the only other systems with 32-bit UEFI firmwares are some very old Macs, and Fedora has had a policy of just not doing UEFI on 32-bit at all. We're now going to have to change that, but it'll take time (the plan is to set things up so a 32-bit UEFI grub2 can boot a 64-bit kernel, and ship images with a 32-bit bootloader but 64-bit system, AIUI). In the mean time, I have hacked it up very very dirtily myself, by whacking on livecd-tools with a hammer until it did something that appears to work. I cut shim out of the loop entirely and just convinced it to create 32-bit live images with an EFI bootloader included; this involves editing livecd-tools' live.py, taking out a check for shim, dropping the shim files and switching up a couple of file names, more or less. After I got that working, with a bit of trial and error, I had images that I can actually boot in the Venue 8 Pro. So that's pretty cool. The V8P (and also the Miix2, I believe) only has a single micro-USB port and doesn't appear to be able to boot from a micro SD card, so you need a USB OTG converter (which is a full-size USB female port on one end and a micro-USB male connector on the other end) to boot from a USB stick. I write my hacked-up live image to the USB stick with livecd-iso-to-disk --format --reset-mbr --efi , plug it into the OTG adapter, plug that into the tablet, press the power button and hold down the 'volume down' key for a couple of seconds (this is how you get into the firmware), go to the Boot tab, disable Secure Boot, and promote the USB stick to #1 in the boot order. Then I quit the firmware interface and it goes ahead and boots the stick. Glorious success!... ...well, not really. I haven't yet hit on an image that will make it to a graphical desktop, or even a console. I've built images with kernels 3.13.0-0.rc1.git0.1.fc21 and 3.12.1-2.fc21 (from rawhide-nodebug) so far. With the 3.12 kernel and default settings it actually boots all the way into GNOME, but the display is very garbled; it looks like modesetting doesn't quite work right for the system, picks the wrong output mode or something. With the 3.13 kernel and default settings, the screen just goes completely blank as soon as modesetting kicks in. If I edit the grub.cfg to pass 'nomodeset' and boot either kernel, it hits a kernel trace quite a long way into the boot process, but before making it either to X or to a console (if I try runlevel 3). I'm trying an image with kernel 3.11 right now, but I doubt that'll help. I guess I'll start bugging kernel people with reports tomorrow. I'll need to pick up a USB hub so I can have both the USB stick and a keyboard plugged in at the same time, too. Still, I'm kinda optimistic that I might be able to get something that actually works quite soon, if I can find people to fix the kernel issues. It looks like it's pretty close to working. Then we'll see just how much GNOME 3 is built for tablets ;) As a piece of hardware, it's pretty nice - unlike previous generation Intel-based tablets, which felt a lot like early lab prototypes which someone had accidentally released into the wild, it seems credible. It's light and thin and has a decent screen and doesn't have any heat issues I've heard about. It's exactly the right form factor to fit in my little travel bag I carry around on trips and stuff. So I'm hoping I'll be able to make practical use of it pretty soon. Edit: With kernel 3.11, modesetting is no better, but it boots successfully with 'nomodeset' and gets to a non-garbled X: The touchscreen doesn't work, so I can't really do anything until I get a USB hub. But I got something to work with, at least.

I spent some time this week trying to help clean up the behaviour of Fedora 20 in regard to keyboard layouts, so I thought I'd write it up here. And write a (Fairly) Short History of Fedora Keyboard Input, while I'm at it. In Linux, keyboard input at the console and X levels is actually handled differently. Console keyboard input basically happens in the kernel, with some very simple userspace utilities available to configure things in the 'kbd' package. The 'loadkeys' utility loads keyboard layout maps of a given format from a given location. There is no standard daemon or anything for switching between different layout map files, in this system: it's expected you pretty much load a single layout file and stick with it. (The way 'configuration' works for kbd is really bone simple: somewhere during init there's a very trivial function which reads a config file, gets a layout name from it, and runs 'loadkeys (layout)'. That's it.) At the X level, keyboard input is handled by xkb, about the complexity of which I have written before. But for the purposes of this, the key point is that it uses layout maps of a different format, stored in a different location, and in the xkb world, switching between layouts is normal and expected behaviour. Prior to Fedora 20, we had two entirely separate sets of keyboard layout maps: one for kbd, one for xkb. They had separate upstreams and separate histories; even maps which happened to have the same name in both schemes were not necessarily the same. Prior to Fedora 18, you were expected to configure your keyboard via the 'system-config-keyboard' utility, which had a table of a limited number of maps for which it knew about roughly corresponding kbd and xkb configurations; you hoped your layout or one like it was in s-c-keyboard's list, you picked it, and s-c-k handled setting the kbd and xkb configurations. During installation, we just ran s-c-keyboard for you to configure your keyboard layout. That kinda worked, but it meant maintaining a tool no-one was particularly excited about maintaining, and we were basically stuck with a fairly arbitrary subset of all possible keyboard layouts which had got written quite a long time ago and more or less bit-rotted since then. So in the new installer UI introduced in F18, we changed things up: we didn't run s-c-keyboard in the installer any more. Instead, we gave anaconda the ability to offer every available xkb layout as a possible option, and wrote some fairly trivial heuristics for guessing what layout you might want to use based on your choice of language at the start of the installation process, so we could pick a sane default layout. Around the same time, systemd's localed sub-system gained the ability to sort of do what system-config-keyboard used to do: localed contains a copy of s-c-k's xkb/kbd layout matching data. So the design was that you'd pick an xkb layout in anaconda, and systemd would transparently take care of figuring out a 'matching' kbd layout and configuring it at first boot. Well, we had various teething issues with that design, but it's not worth going into here. But the new design did bring some urgency to something we'd had in the works for a long time. Instead of having systemd do contortions to try and figure out a 'matching' kbd layout for your chosen xkb layout, it'd seem a lot simpler if we could magically make all the xkb layouts available to kbd, right? Turns out we were actually planning that all along. And so early in Fedora 20, we introduced a change to the kbd package. It now used a couple of neat tools to generate kbd-format layout maps based on all the available xkb maps. For every xkb map present in Fedora, we now had a matching kbd map. So things are really simple now, right? We should just drop the localed clever code for 'translating' xkb layouts into kbd layouts and simply tell it to use whatever xkb layout you got from the installation process for kbd as well. Well...turns out, not so fast. I actually had bug reports in asking for these changes to be put in place for a while, without it happening. By the time I got around to requesting those changes with a bit more urgency last month, I was starting to suspect we'd missed a rather large problem, and it turned out we had. It goes back to the issue of layout switching which I mentioned earlier. I said that there's no mechanism for switching between multiple kbd layout maps. This is true, but what you can do with a kbd-style layout is switch between different layouts within a single map file. In both kbd-style and xkb-style layouts, each key can produce up to four characters: one when you press it alone, one when you press it with shift, one when you press it with alt-gr, and one when you press it with shift and alt-gr. What you can also do in a kbd-style layout is define a key or key combination that effectively does for alt-gr what caps lock does for shift: toggles it. So if you write a kbd layout which defines alt-gr mappings for a lot of the keys, and has a key combo for toggling altr-gr, you've effectively got a switched layout. The layout I usually use to illustrate it is Russian: if you load the 'ru' kbd layout and start typing with the letter keys, you'll notice it seems just like a US layout. If you then hit the left ctrl and shift keys together, the letter keys 'switch': they now output Cyrillic characters. That's the alt-gr toggle. In a Russian layout, the letter keys output Latin characters when pressed alone, Cyrillic when pressed with alt-gr. This is how Russians expect their keyboards to behave. Now in xkb, you can achieve basically the same result, but by a different method. A typical Russian xkb configuration would be to have one of the Russian layouts and US English both enabled, and define a key combination to switch between them. xkb can have as many layouts enabled as you like, and can designate a key combination that switches between layouts. So the user experience is much the same: you can type Latin characters, then hit a key combination and switch to typing Cyrillic. But the implementation mechanism is different. Russian's one example of this, but there are actually dozens of layouts of this type, where users expect them to be switched, so they can input both Latin and 'native' character sets. It's not like a UK English, or French, or German, or whatever, layout, which is an alternative arrangement of keys but mostly inputs the same set of characters. So the big problem we hadn't really accounted for was simply this 'switching problem'. No kbd layout which has been tool-generated from an xkb layout will have a set of alt-gr characters and a switch combo defined, because that's just not how xkb layouts are designed to work: you're supposed to switch between xkb layouts, not within them. So any kbd layout produced from an xkb layout whose users would expect to load it in combination with a Latin-capable layout and switch between them becomes a booby trap, because if you load it, you cannot input Latin characters at all. This is obviously not good. Originally, the expectation was that everyone would use an xkb-converted layout, and so the 'old' kbd layouts were moved to a 'legacy' subdirectory and taken out of the loadkeys path: you could only load them with an explicit path. Really, we were just keeping them around in case we'd screwed something up. Which, of course, we had. Obviously that couldn't really work, though. (We also set up symlinks for all the old kbd names which pointed to a similar xkb layout; this was intended to handle people upgrading from older releases. Of course, this meant anyone with a 'switched' layout who upgraded to F20 suddenly lost their switching, which I imagine they weren't happy about...) So in the last week, after thinking this through, we've come up with a set of changes to kbd and systemd. As I'd originally intended, we changed localed's logic, so instead of always trying to find a 'matching' kbd layout for the selected xkb layout by using its table, if a kbd layout of the same name as the selected xkb layout exists, it will use that. But we kept the table of 'corresponding' layouts around, and the logic: it's called only if a layout of the exact same name isn't found. We changed up the kbd package so the legacy layouts are back in the loadkeys path, but after the xkb ones, and dropped the symlinks (the bug report is about a problem I didn't actually mention in this post - console layout loading frequently didn't seem to work at all. We think these changes incidentally fix that bug too). And I came up with a kinda-clever, kinda-gross hack: now, after it generates the full set of xkb-derived kbd layouts, the kbd package finds all the layouts which have no mapping for the character 'A' (an upper-case Latin 'a') and deletes them (this is a trivial zgrep | xargs rm pipeline). This is acting as a proxy for 'cannot input Latin characters' - there are 400+ xkb layouts and 150+ of them cannot input Latin characters, so trying to keep track of them in a 'curated' fashion seems inappropriate. Doing it this way at first felt like a hack, but to be honest, now it seems like the most sensible approach. I did do some manual sanity checks with layouts we know to be either capable or not capable of Latin input. Now if you followed all that, award yourself a gold star, and you should be able to figure out the eventual result. If not, here's how it works: If you pick an xkb layout which is capable of inputting Latin characters, then the kbd package will contain a kbd layout generated from that xkb layout with the same name. localed will use that layout as your console layout, and you'll have precisely the same layout in X and at the console. If you pick an xkb layout which is not capable of inputting Latin characters, the kbd package will not contain a kbd layout generated from it, because you wouldn't want to use that (you wouldn't be able to type any Latin characters). localed will notice this, and use its table to try and find a matching 'legacy' kbd layout. If it finds one - for instance, if you picked Russian - it will use that, and you'll have a proper, native, switchable layout. If it can't, you'll just wind up with the default kernel layout, which is, inevitably, US English. And at least you'll be able to type Latin characters, which is the most important thing to be able to do at a console. It's a saner result than a 'native' layout which is functionally useless. People who upgrade should also wind up with something sane - they may wind up with a converted or a legacy layout (depending on the names), but they shouldn't wind up with anything that doesn't work usably. One other change we put into place between Fedora 18 and Fedora 20 was to use langtable in the installer to try and be better about picking an 'appropriate' X layout for your language by default. langtable is a fairly new project aiming to store this kind of 'if X, then Y' information for i18n stuff: it's intended to be a source of information like 'what keyboard layouts are people who pick Language X, Variant Y most likely to want to use?' It has a list of keyboard layouts, languages, territories, and timezones, and all sorts of mappings between them. It's a work in progress, but it already contains a lot of the knowledge that system-config-keyboard had and localed inherited. So the ultimate upshot of this: if we lined everything up right, after various teething issues in Fedora 18 and Fedora 19, Fedora 20 should be getting pretty good at this keyboard layout stuff. Most users should have a sensible default layout (or pair of layouts - when we know your 'native' xkb layout can't input Latin characters, we automatically configure it alongside US English, with a layout switch combo) selected in the installer after they pick their language and locale. If you want to pick a different one in the installer, you can - you can set any number of layouts you like, and customize the layout switch key combo. And we should do a decent job of selecting the best available console layout to 'match' whatever xkb layout you set to be at the top of your list in the installer. All these changes are in kbd-1.15.5-11.fc20 and systemd-208-6.fc20. They are not included in Final TC2, but will be in Final TC3. Testing with these packages would be greatly appreciated, to see if there's anything we still didn't account for. Well, except for changing console layouts post-install - we know there's still a problem with that. Around the same timeframe, GNOME has been similarly improving and refining its input handling. While there've been various transition pains and teething issues at all kinds of levels over the last few releases, I feel like we're kind of reaching a point where we're getting back to the level of correctness we had prior to Fedora 18, but with much more flexibility: you're now not limited to a fairly arbitrary and bit-rotting subset of keyboard layouts and tied into the Fedora-specific system-config-keyboard (and even older system-setup-keyboard, which systemd killed). You have the whole set of xkb layouts to choose from, and at the GNOME level, with 3.8 and 3.10, we have really pretty awesome support for more complex input methods via ibus. There is much less Fedora-specific infrastructure around, and we're at least more converged between xkb and kbd. I'm now very much looking forward to the introduction of a new thing called kmscon to Fedora. kmscon is a user-space replacement for the kernel console, intended to replace the kernel console plus all the userspace bits that go with it (gettys, kbd and all the rest of it). kmscon actually uses xkb - via libxkbcommon, with no X dependency, of course! - for keyboard input. So in the Glorious Future when we can switch to kmscon for our consoles, we really will be able to forget all about this kbd vs. xkb mess, and have a single system-wide keyboard input framework, data and configuration. How I look forward to it...

Coming up tomorrow - or today, depending on your region! - we have the final scheduled Test Day of the Fedora 20 cycle, power management Test Day! This is another recurring event, where we test out power management functions across as broad a range of hardware as possible. Just about anyone can easily contribute to this event, so please come along and help! As usual, live images are available for performing the testing, and QA and development folks should be on hand at various times through the day to help out with testing in #fedora-test-day on Freenode IRC. If you don't know how to use IRC, read these instructions, or just use WebIRC. Please come and help us test if you have a bit of spare time!

So as we finally signed off on Fedora 20 Beta this morning (with so much handwaving I felt like my shoulder was about to come off at the joint), and I looked forward to a slightly quiet morning... ...my router spontaneously decided to reset itself to factory settings. That. That. Did not. Amuse. Me. This illustrated a hitherto undiscovered weakness in my backup strategy: I didn't have the router configuration backed up. PSA: folks, back up your router configuration. Seriously. A morning spent remembering and re-entering static leases, port forwards and FreeIPA DNS records does not rank highly among my Most Entertaining Of All Time. I think the show is back on the road now, but there will inevitably be that one thing I forgot to do which will come back and bite me in the ass in two weeks...

First off, we have another Test Day coming up: tomorrow (or today, for a few of you!), Thursday 2013-11-07, is KDE 4.11 Test Day. We have a good set of test cases ready in the awesome Test Day results app, we have a custom live image with KDE 4.11.3 packages included, and we'll have QA and KDE team members on hand to help you with any testing, so what are you waiting for? Grab a live image and come help us test tomorrow! Even if you're not a Fedora user, you can test without needing to install Fedora, and your tests will help improve upstream KDE. Join us in #fedora-test-day on Freenode IRC to discuss bugs or get help with testing. If you don't know how to use IRC, read these instructions, or just use WebIRC. It's a busy time, because tomorrow is also the (third :Install * Base * Desktop