Keyboard layouts in Fedora 20 (and previously)

I spent some time this week trying to help clean up the behaviour of Fedora 20 in regard to keyboard layouts, so I thought I'd write it up here. And write a (Fairly) Short History of Fedora Keyboard Input, while I'm at it. In Linux, keyboard input at the console and X levels is actually handled differently. Console keyboard input basically happens in the kernel, with some very simple userspace utilities available to configure things in the 'kbd' package. The 'loadkeys' utility loads keyboard layout maps of a given format from a given location. There is no standard daemon or anything for switching between different layout map files, in this system: it's expected you pretty much load a single layout file and stick with it. (The way 'configuration' works for kbd is really bone simple: somewhere during init there's a very trivial function which reads a config file, gets a layout name from it, and runs 'loadkeys (layout)'. That's it.) At the X level, keyboard input is handled by xkb, about the complexity of which I have written before. But for the purposes of this, the key point is that it uses layout maps of a different format, stored in a different location, and in the xkb world, switching between layouts is normal and expected behaviour. Prior to Fedora 20, we had two entirely separate sets of keyboard layout maps: one for kbd, one for xkb. They had separate upstreams and separate histories; even maps which happened to have the same name in both schemes were not necessarily the same. Prior to Fedora 18, you were expected to configure your keyboard via the 'system-config-keyboard' utility, which had a table of a limited number of maps for which it knew about roughly corresponding kbd and xkb configurations; you hoped your layout or one like it was in s-c-keyboard's list, you picked it, and s-c-k handled setting the kbd and xkb configurations. During installation, we just ran s-c-keyboard for you to configure your keyboard layout. That kinda worked, but it meant maintaining a tool no-one was particularly excited about maintaining, and we were basically stuck with a fairly arbitrary subset of all possible keyboard layouts which had got written quite a long time ago and more or less bit-rotted since then. So in the new installer UI introduced in F18, we changed things up: we didn't run s-c-keyboard in the installer any more. Instead, we gave anaconda the ability to offer every available xkb layout as a possible option, and wrote some fairly trivial heuristics for guessing what layout you might want to use based on your choice of language at the start of the installation process, so we could pick a sane default layout. Around the same time, systemd's localed sub-system gained the ability to sort of do what system-config-keyboard used to do: localed contains a copy of s-c-k's xkb/kbd layout matching data. So the design was that you'd pick an xkb layout in anaconda, and systemd would transparently take care of figuring out a 'matching' kbd layout and configuring it at first boot. Well, we had various teething issues with that design, but it's not worth going into here. But the new design did bring some urgency to something we'd had in the works for a long time. Instead of having systemd do contortions to try and figure out a 'matching' kbd layout for your chosen xkb layout, it'd seem a lot simpler if we could magically make all the xkb layouts available to kbd, right? Turns out we were actually planning that all along. And so early in Fedora 20, we introduced a change to the kbd package. It now used a couple of neat tools to generate kbd-format layout maps based on all the available xkb maps. For every xkb map present in Fedora, we now had a matching kbd map. So things are really simple now, right? We should just drop the localed clever code for 'translating' xkb layouts into kbd layouts and simply tell it to use whatever xkb layout you got from the installation process for kbd as well. Well...turns out, not so fast. I actually had bug reports in asking for these changes to be put in place for a while, without it happening. By the time I got around to requesting those changes with a bit more urgency last month, I was starting to suspect we'd missed a rather large problem, and it turned out we had. It goes back to the issue of layout switching which I mentioned earlier. I said that there's no mechanism for switching between multiple kbd layout maps. This is true, but what you can do with a kbd-style layout is switch between different layouts within a single map file. In both kbd-style and xkb-style layouts, each key can produce up to four characters: one when you press it alone, one when you press it with shift, one when you press it with alt-gr, and one when you press it with shift and alt-gr. What you can also do in a kbd-style layout is define a key or key combination that effectively does for alt-gr what caps lock does for shift: toggles it. So if you write a kbd layout which defines alt-gr mappings for a lot of the keys, and has a key combo for toggling altr-gr, you've effectively got a switched layout. The layout I usually use to illustrate it is Russian: if you load the 'ru' kbd layout and start typing with the letter keys, you'll notice it seems just like a US layout. If you then hit the left ctrl and shift keys together, the letter keys 'switch': they now output Cyrillic characters. That's the alt-gr toggle. In a Russian layout, the letter keys output Latin characters when pressed alone, Cyrillic when pressed with alt-gr. This is how Russians expect their keyboards to behave. Now in xkb, you can achieve basically the same result, but by a different method. A typical Russian xkb configuration would be to have one of the Russian layouts and US English both enabled, and define a key combination to switch between them. xkb can have as many layouts enabled as you like, and can designate a key combination that switches between layouts. So the user experience is much the same: you can type Latin characters, then hit a key combination and switch to typing Cyrillic. But the implementation mechanism is different. Russian's one example of this, but there are actually dozens of layouts of this type, where users expect them to be switched, so they can input both Latin and 'native' character sets. It's not like a UK English, or French, or German, or whatever, layout, which is an alternative arrangement of keys but mostly inputs the same set of characters. So the big problem we hadn't really accounted for was simply this 'switching problem'. No kbd layout which has been tool-generated from an xkb layout will have a set of alt-gr characters and a switch combo defined, because that's just not how xkb layouts are designed to work: you're supposed to switch between xkb layouts, not within them. So any kbd layout produced from an xkb layout whose users would expect to load it in combination with a Latin-capable layout and switch between them becomes a booby trap, because if you load it, you cannot input Latin characters at all. This is obviously not good. Originally, the expectation was that everyone would use an xkb-converted layout, and so the 'old' kbd layouts were moved to a 'legacy' subdirectory and taken out of the loadkeys path: you could only load them with an explicit path. Really, we were just keeping them around in case we'd screwed something up. Which, of course, we had. Obviously that couldn't really work, though. (We also set up symlinks for all the old kbd names which pointed to a similar xkb layout; this was intended to handle people upgrading from older releases. Of course, this meant anyone with a 'switched' layout who upgraded to F20 suddenly lost their switching, which I imagine they weren't happy about...) So in the last week, after thinking this through, we've come up with a set of changes to kbd and systemd. As I'd originally intended, we changed localed's logic, so instead of always trying to find a 'matching' kbd layout for the selected xkb layout by using its table, if a kbd layout of the same name as the selected xkb layout exists, it will use that. But we kept the table of 'corresponding' layouts around, and the logic: it's called only if a layout of the exact same name isn't found. We changed up the kbd package so the legacy layouts are back in the loadkeys path, but after the xkb ones, and dropped the symlinks (the bug report is about a problem I didn't actually mention in this post - console layout loading frequently didn't seem to work at all. We think these changes incidentally fix that bug too). And I came up with a kinda-clever, kinda-gross hack: now, after it generates the full set of xkb-derived kbd layouts, the kbd package finds all the layouts which have no mapping for the character 'A' (an upper-case Latin 'a') and deletes them (this is a trivial zgrep | xargs rm pipeline). This is acting as a proxy for 'cannot input Latin characters' - there are 400+ xkb layouts and 150+ of them cannot input Latin characters, so trying to keep track of them in a 'curated' fashion seems inappropriate. Doing it this way at first felt like a hack, but to be honest, now it seems like the most sensible approach. I did do some manual sanity checks with layouts we know to be either capable or not capable of Latin input. Now if you followed all that, award yourself a gold star, and you should be able to figure out the eventual result. If not, here's how it works: If you pick an xkb layout which is capable of inputting Latin characters, then the kbd package will contain a kbd layout generated from that xkb layout with the same name. localed will use that layout as your console layout, and you'll have precisely the same layout in X and at the console. If you pick an xkb layout which is not capable of inputting Latin characters, the kbd package will not contain a kbd layout generated from it, because you wouldn't want to use that (you wouldn't be able to type any Latin characters). localed will notice this, and use its table to try and find a matching 'legacy' kbd layout. If it finds one - for instance, if you picked Russian - it will use that, and you'll have a proper, native, switchable layout. If it can't, you'll just wind up with the default kernel layout, which is, inevitably, US English. And at least you'll be able to type Latin characters, which is the most important thing to be able to do at a console. It's a saner result than a 'native' layout which is functionally useless. People who upgrade should also wind up with something sane - they may wind up with a converted or a legacy layout (depending on the names), but they shouldn't wind up with anything that doesn't work usably. One other change we put into place between Fedora 18 and Fedora 20 was to use langtable in the installer to try and be better about picking an 'appropriate' X layout for your language by default. langtable is a fairly new project aiming to store this kind of 'if X, then Y' information for i18n stuff: it's intended to be a source of information like 'what keyboard layouts are people who pick Language X, Variant Y most likely to want to use?' It has a list of keyboard layouts, languages, territories, and timezones, and all sorts of mappings between them. It's a work in progress, but it already contains a lot of the knowledge that system-config-keyboard had and localed inherited. So the ultimate upshot of this: if we lined everything up right, after various teething issues in Fedora 18 and Fedora 19, Fedora 20 should be getting pretty good at this keyboard layout stuff. Most users should have a sensible default layout (or pair of layouts - when we know your 'native' xkb layout can't input Latin characters, we automatically configure it alongside US English, with a layout switch combo) selected in the installer after they pick their language and locale. If you want to pick a different one in the installer, you can - you can set any number of layouts you like, and customize the layout switch key combo. And we should do a decent job of selecting the best available console layout to 'match' whatever xkb layout you set to be at the top of your list in the installer. All these changes are in kbd-1.15.5-11.fc20 and systemd-208-6.fc20. They are not included in Final TC2, but will be in Final TC3. Testing with these packages would be greatly appreciated, to see if there's anything we still didn't account for. Well, except for changing console layouts post-install - we know there's still a problem with that. Around the same timeframe, GNOME has been similarly improving and refining its input handling. While there've been various transition pains and teething issues at all kinds of levels over the last few releases, I feel like we're kind of reaching a point where we're getting back to the level of correctness we had prior to Fedora 18, but with much more flexibility: you're now not limited to a fairly arbitrary and bit-rotting subset of keyboard layouts and tied into the Fedora-specific system-config-keyboard (and even older system-setup-keyboard, which systemd killed). You have the whole set of xkb layouts to choose from, and at the GNOME level, with 3.8 and 3.10, we have really pretty awesome support for more complex input methods via ibus. There is much less Fedora-specific infrastructure around, and we're at least more converged between xkb and kbd. I'm now very much looking forward to the introduction of a new thing called kmscon to Fedora. kmscon is a user-space replacement for the kernel console, intended to replace the kernel console plus all the userspace bits that go with it (gettys, kbd and all the rest of it). kmscon actually uses xkb - via libxkbcommon, with no X dependency, of course! - for keyboard input. So in the Glorious Future when we can switch to kmscon for our consoles, we really will be able to forget all about this kbd vs. xkb mess, and have a single system-wide keyboard input framework, data and configuration. How I look forward to it...