Keyboard layouts in Fedora 20 (and previously)

I spent some time this week trying to help clean up the behaviour of Fedora 20 in regard to keyboard layouts, so I thought I’d write it up here. And write a (Fairly) Short History of Fedora Keyboard Input, while I’m at it.

In Linux, keyboard input at the console and X levels is actually handled differently. Console keyboard input basically happens in the kernel, with some very simple userspace utilities available to configure things in the ‘kbd’ package. The ‘loadkeys’ utility loads keyboard layout maps of a given format from a given location. There is no standard daemon or anything for switching between different layout map files, in this system: it’s expected you pretty much load a single layout file and stick with it. (The way ‘configuration’ works for kbd is really bone simple: somewhere during init there’s a very trivial function which reads a config file, gets a layout name from it, and runs ‘loadkeys (layout)’. That’s it.)

At the X level, keyboard input is handled by xkb, about the complexity of which I have written before. But for the purposes of this, the key point is that it uses layout maps of a different format, stored in a different location, and in the xkb world, switching between layouts is normal and expected behaviour.

Prior to Fedora 20, we had two entirely separate sets of keyboard layout maps: one for kbd, one for xkb. They had separate upstreams and separate histories; even maps which happened to have the same name in both schemes were not necessarily the same. Prior to Fedora 18, you were expected to configure your keyboard via the ‘system-config-keyboard’ utility, which had a table of a limited number of maps for which it knew about roughly corresponding kbd and xkb configurations; you hoped your layout or one like it was in s-c-keyboard’s list, you picked it, and s-c-k handled setting the kbd and xkb configurations. During installation, we just ran s-c-keyboard for you to configure your keyboard layout. That kinda worked, but it meant maintaining a tool no-one was particularly excited about maintaining, and we were basically stuck with a fairly arbitrary subset of all possible keyboard layouts which had got written quite a long time ago and more or less bit-rotted since then.

So in the new installer UI introduced in F18, we changed things up: we didn’t run s-c-keyboard in the installer any more. Instead, we gave anaconda the ability to offer every available xkb layout as a possible option, and wrote some fairly trivial heuristics for guessing what layout you might want to use based on your choice of language at the start of the installation process, so we could pick a sane default layout. Around the same time, systemd’s localed sub-system gained the ability to sort of do what system-config-keyboard used to do: localed contains a copy of s-c-k’s xkb/kbd layout matching data. So the design was that you’d pick an xkb layout in anaconda, and systemd would transparently take care of figuring out a ‘matching’ kbd layout and configuring it at first boot.

Well, we had various teething issues with that design, but it’s not worth going into here. But the new design did bring some urgency to something we’d had in the works for a long time. Instead of having systemd do contortions to try and figure out a ‘matching’ kbd layout for your chosen xkb layout, it’d seem a lot simpler if we could magically make all the xkb layouts available to kbd, right? Turns out we were actually planning that all along.

And so early in Fedora 20, we introduced a change to the kbd package. It now used a couple of neat tools to generate kbd-format layout maps based on all the available xkb maps. For every xkb map present in Fedora, we now had a matching kbd map.

So things are really simple now, right? We should just drop the localed clever code for ‘translating’ xkb layouts into kbd layouts and simply tell it to use whatever xkb layout you got from the installation process for kbd as well.

Well…turns out, not so fast. I actually had bug reports in asking for these changes to be put in place for a while, without it happening. By the time I got around to requesting those changes with a bit more urgency last month, I was starting to suspect we’d missed a rather large problem, and it turned out we had. It goes back to the issue of layout switching which I mentioned earlier.

I said that there’s no mechanism for switching between multiple kbd layout maps. This is true, but what you can do with a kbd-style layout is switch between different layouts within a single map file. In both kbd-style and xkb-style layouts, each key can produce up to four characters: one when you press it alone, one when you press it with shift, one when you press it with alt-gr, and one when you press it with shift and alt-gr. What you can also do in a kbd-style layout is define a key or key combination that effectively does for alt-gr what caps lock does for shift: toggles it. So if you write a kbd layout which defines alt-gr mappings for a lot of the keys, and has a key combo for toggling altr-gr, you’ve effectively got a switched layout. The layout I usually use to illustrate it is Russian: if you load the ‘ru’ kbd layout and start typing with the letter keys, you’ll notice it seems just like a US layout. If you then hit the left ctrl and shift keys together, the letter keys ‘switch’: they now output Cyrillic characters. That’s the alt-gr toggle. In a Russian layout, the letter keys output Latin characters when pressed alone, Cyrillic when pressed with alt-gr. This is how Russians expect their keyboards to behave.

Now in xkb, you can achieve basically the same result, but by a different method. A typical Russian xkb configuration would be to have one of the Russian layouts and US English both enabled, and define a key combination to switch between them. xkb can have as many layouts enabled as you like, and can designate a key combination that switches between layouts. So the user experience is much the same: you can type Latin characters, then hit a key combination and switch to typing Cyrillic. But the implementation mechanism is different.

Russian’s one example of this, but there are actually dozens of layouts of this type, where users expect them to be switched, so they can input both Latin and ‘native’ character sets. It’s not like a UK English, or French, or German, or whatever, layout, which is an alternative arrangement of keys but mostly inputs the same set of characters.

So the big problem we hadn’t really accounted for was simply this ‘switching problem’. No kbd layout which has been tool-generated from an xkb layout will have a set of alt-gr characters and a switch combo defined, because that’s just not how xkb layouts are designed to work: you’re supposed to switch between xkb layouts, not within them. So any kbd layout produced from an xkb layout whose users would expect to load it in combination with a Latin-capable layout and switch between them becomes a booby trap, because if you load it, you cannot input Latin characters at all. This is obviously not good.

Originally, the expectation was that everyone would use an xkb-converted layout, and so the ‘old’ kbd layouts were moved to a ‘legacy’ subdirectory and taken out of the loadkeys path: you could only load them with an explicit path. Really, we were just keeping them around in case we’d screwed something up. Which, of course, we had. Obviously that couldn’t really work, though. (We also set up symlinks for all the old kbd names which pointed to a similar xkb layout; this was intended to handle people upgrading from older releases. Of course, this meant anyone with a ‘switched’ layout who upgraded to F20 suddenly lost their switching, which I imagine they weren’t happy about…)

So in the last week, after thinking this through, we’ve come up with a set of changes to kbd and systemd. As I’d originally intended, we changed localed’s logic, so instead of always trying to find a ‘matching’ kbd layout for the selected xkb layout by using its table, if a kbd layout of the same name as the selected xkb layout exists, it will use that. But we kept the table of ‘corresponding’ layouts around, and the logic: it’s called only if a layout of the exact same name isn’t found. We changed up the kbd package so the legacy layouts are back in the loadkeys path, but after the xkb ones, and dropped the symlinks (the bug report is about a problem I didn’t actually mention in this post – console layout loading frequently didn’t seem to work at all. We think these changes incidentally fix that bug too). And I came up with a kinda-clever, kinda-gross hack: now, after it generates the full set of xkb-derived kbd layouts, the kbd package finds all the layouts which have no mapping for the character ‘A’ (an upper-case Latin ‘a’) and deletes them (this is a trivial zgrep | xargs rm pipeline). This is acting as a proxy for ‘cannot input Latin characters’ – there are 400+ xkb layouts and 150+ of them cannot input Latin characters, so trying to keep track of them in a ‘curated’ fashion seems inappropriate. Doing it this way at first felt like a hack, but to be honest, now it seems like the most sensible approach. I did do some manual sanity checks with layouts we know to be either capable or not capable of Latin input.

Now if you followed all that, award yourself a gold star, and you should be able to figure out the eventual result. If not, here’s how it works:

If you pick an xkb layout which is capable of inputting Latin characters, then the kbd package will contain a kbd layout generated from that xkb layout with the same name. localed will use that layout as your console layout, and you’ll have precisely the same layout in X and at the console.

If you pick an xkb layout which is not capable of inputting Latin characters, the kbd package will not contain a kbd layout generated from it, because you wouldn’t want to use that (you wouldn’t be able to type any Latin characters). localed will notice this, and use its table to try and find a matching ‘legacy’ kbd layout. If it finds one – for instance, if you picked Russian – it will use that, and you’ll have a proper, native, switchable layout. If it can’t, you’ll just wind up with the default kernel layout, which is, inevitably, US English. And at least you’ll be able to type Latin characters, which is the most important thing to be able to do at a console.

It’s a saner result than a ‘native’ layout which is functionally useless. People who upgrade should also wind up with something sane – they may wind up with a converted or a legacy layout (depending on the names), but they shouldn’t wind up with anything that doesn’t work usably.

One other change we put into place between Fedora 18 and Fedora 20 was to use langtable in the installer to try and be better about picking an ‘appropriate’ X layout for your language by default. langtable is a fairly new project aiming to store this kind of ‘if X, then Y’ information for i18n stuff: it’s intended to be a source of information like ‘what keyboard layouts are people who pick Language X, Variant Y most likely to want to use?’ It has a list of keyboard layouts, languages, territories, and timezones, and all sorts of mappings between them. It’s a work in progress, but it already contains a lot of the knowledge that system-config-keyboard had and localed inherited.

So the ultimate upshot of this: if we lined everything up right, after various teething issues in Fedora 18 and Fedora 19, Fedora 20 should be getting pretty good at this keyboard layout stuff. Most users should have a sensible default layout (or pair of layouts – when we know your ‘native’ xkb layout can’t input Latin characters, we automatically configure it alongside US English, with a layout switch combo) selected in the installer after they pick their language and locale. If you want to pick a different one in the installer, you can – you can set any number of layouts you like, and customize the layout switch key combo. And we should do a decent job of selecting the best available console layout to ‘match’ whatever xkb layout you set to be at the top of your list in the installer.

All these changes are in kbd-1.15.5-11.fc20 and systemd-208-6.fc20. They are not included in Final TC2, but will be in Final TC3. Testing with these packages would be greatly appreciated, to see if there’s anything we still didn’t account for. Well, except for changing console layouts post-install – we know there’s still a problem with that.

Around the same timeframe, GNOME has been similarly improving and refining its input handling. While there’ve been various transition pains and teething issues at all kinds of levels over the last few releases, I feel like we’re kind of reaching a point where we’re getting back to the level of correctness we had prior to Fedora 18, but with much more flexibility: you’re now not limited to a fairly arbitrary and bit-rotting subset of keyboard layouts and tied into the Fedora-specific system-config-keyboard (and even older system-setup-keyboard, which systemd killed). You have the whole set of xkb layouts to choose from, and at the GNOME level, with 3.8 and 3.10, we have really pretty awesome support for more complex input methods via ibus. There is much less Fedora-specific infrastructure around, and we’re at least more converged between xkb and kbd.

I’m now very much looking forward to the introduction of a new thing called kmscon to Fedora. kmscon is a user-space replacement for the kernel console, intended to replace the kernel console plus all the userspace bits that go with it (gettys, kbd and all the rest of it). kmscon actually uses xkb – via libxkbcommon, with no X dependency, of course! – for keyboard input. So in the Glorious Future when we can switch to kmscon for our consoles, we really will be able to forget all about this kbd vs. xkb mess, and have a single system-wide keyboard input framework, data and configuration. How I look forward to it…

15 Responses

  1. Bill Berry
    Bill Berry January 23, 2014 at 3:35 am | | Reply

    I have a problem with Fedora 20 on my laptop. It defaults to US English. I need it to be UK English. As far as I remember, the installer did not pause long enough for me to find the UK English option during install, and defaulted to US English. Now, when I type in my email address, it comes out as william.berry3″ntlworld.com. The UK currency symbol is NOT # , which is what comes out when I press that symbol on my keyboard.

    The only way I can change to the UK layout is by using system-config-keyboard. @loadkeys uk@ or @loadkeys gb@ does not work. (The @ symbol is the result of pressing shift-2, which is ” on a UK keyboard). However, s-c-k does not preserve the configuration between reboots. In my view, this is a bug.

    Thank you for explaining your efforts to co-ordinate kbd and xkbd. That seems to me the sane thing to do. However, it would be helpful if a switch in keyboard layout was preserved between reboots.

    This is probably not the place to ask for help. However, I believe it would be helpful if you were to post on the main Fedora site something to help ordinary users like me to deal with this problem. I had to a lot of searching around to find out that system-config-keybord even existed! I had previously used loadkeys, which, up to now (but not with F20) worked.

  2. Bill Berry
    Bill Berry February 1, 2014 at 3:06 am | | Reply

    Thank you so much for all your help. However, something odd is happening. etc/X11/xorg.conf.d/00-keyboard.conf tells me that keyboard is gb. Gnome settings tells me that language and region is gb. (I have removed us). In a terminal found by ctrl-alt f2 the keymap is gb. In LibreOffice the keymap is gb. In Gnome-terminal the layout is us. In Gnome-terminal running “loadkeys gb” and “loadkeys uk” both return “Couldn’t get a file descriptor referring to the console”. system-config-keyboard is the only thing that works, but the setting is not saved – I get the same problem every reboot.

    I had not originally intended to ask for help here. However, your original post was about unifying everything, to make one mechanism for regional settings. The fact that I am getting these problems may highlight something that is yet to be resolved.

  3. Bill Berry
    Bill Berry February 3, 2014 at 9:21 am | | Reply

    I forgot to say, in response to your comment about the installer and choosing keyboard layout, that I accept what you say. I did not take a note of everything that happened during install.

    Start again. Log out, then log back in to Cinnamon. Everything in Cinnamon now reverts to us keymap. However, ctl-alt-f2 for a vt, and the vt uses gb keymap.

    /etc/X11/xorg.conf.d/00-keyboard.conf
    # Read and parsed by systemd-localed. It’s probably wise not to edit this file
    # manually too freely.
    Section “InputClass”
    Identifier “system-keyboard”
    MatchIsKeyboard “on”
    Option “XkbLayout” “gb”
    Option “XkbModel” “pc105″
    EndSection

    I looked at your other post (https://www.happyassassin.net/2013/05/26/more-debugging/) and ran setxkbmap -print:

    xkb_keymap {
    xkb_keycodes { include “evdev+aliases(qwerty)” };
    xkb_types { include “complete” };
    xkb_compat { include “complete” };
    xkb_symbols { include “pc+gb+us:2+inet(evdev)” };
    xkb_geometry { include “pc(pc105)” };
    };

    After that, when I go back to gnome-terminal, I get a gb keyboard layout! (also in gedit – gb layout). I am confused – “man setxkbmap” indicates that -print only tells you what the current situation is – it doesn’t set anything.

    Shut down – restart. US keymap in gnome-terminal, gb keymap in vt. Run “setxkbmap” in vt -> “Cannot open display “Default Display”. (This is not altogether unexpected). In gnome-terminal “setxkbmap” (just that – no parameters) gives no output in response, but switches to gb keymap in gnome-terminal and gedit!

    /var/log/Xorg.0.log includes the lines

    [ 20.779] (II) config/udev: Adding input device AT Translated Set 2 keyboard
    (/dev/input/event3)
    [ 20.779] (**) AT Translated Set 2 keyboard: Applying InputClass “evdev keybo
    ard catchall”
    [ 20.779] (**) AT Translated Set 2 keyboard: Applying InputClass “system-keyb
    oard”
    [ 20.779] (II) Using input driver ‘evdev’ for ‘AT Translated Set 2 keyboard’
    [ 20.779] (**) AT Translated Set 2 keyboard: always reports core events
    [ 20.779] (**) evdev: AT Translated Set 2 keyboard: Device: “/dev/input/event
    3″
    [ 20.779] (–) evdev: AT Translated Set 2 keyboard: Vendor 0×1 Product 0×1
    [ 20.779] (–) evdev: AT Translated Set 2 keyboard: Found keys
    [ 20.780] (II) evdev: AT Translated Set 2 keyboard: Configuring as keyboard
    [ 20.780] (**) Option “config_info” “udev:/sys/devices/platform/i8042/serio0/input/input3/event3″
    [ 20.780] (II) XINPUT: Adding extended input device “AT Translated Set 2 keyboard” (type: KEYBOARD, id 11)
    [ 20.780] (**) Option “xkb_rules” “evdev”
    [ 20.780] (**) Option “xkb_model” “pc105″
    [ 20.780] (**) Option “xkb_layout” “gb”

    I know very little about what happens at boot, but I suspect that something should be happening that isn’t. I suppose I could just put setxkbmap in rc.local. That would solve the problem for me, but it wouldn’t tell you why I need it. I am assuming that the fact this is a laptop is not relevant?

    Is this blog the right place for this conversation? If it helps you to get closer to your goal, I suppose it is.

  4. Bill Berry
    Bill Berry February 5, 2014 at 5:30 am | | Reply

    I didn’t have dconf-editor, so I installed it. I created a new user “test” which behaves in exactly the same way. I have tried KDE, Gnome, LXDE desktop, but not Sugar, MATE or e17. (What a wonderful array of choices!). sudo grouplist tells me that ‘GNOME Desktop’ is available, but not installed (!). Actually, it is. I thought that perhaps some vital part of Cinnamon was not installed (I didn’t have dconf-editor). “sudo yum groupinstall Cinnamon” returned “No packages in any requested group available to install or update”.

    Gnome is the only DE I have tried that uses gb keyboard – the others default to us.

    I do not know what I am looking for in dconf-editor. org.freedesktop.ibus.general.xkblayoutconfig shows default as mi (Australia)! I have no idea why. In this key, there is no gb layout. “uk” is in Eastern Europe (Ukraine?). There is an “en” in the “Western Europe” list. So now we have different two-letter country codes in different places – uk (Ukraine, or United Kingdom?), gb (Great Britain), and en (England? The Scots and the Welsh will not like that!). Scrub that – further search shows that whatever I select here shows as “default”, but the “Set as default” button is greyed out.

    org.cinnamon.desktop.input-sources shows “Current 0″. “Sources [('xkb','gb'), ('xkb,gb+extd'), ('xkb, gb+intl')]

    org.cinnamon.system.locale shows en_GB.utf8. “Default” shows ” (two single quotes with nothing between, I suppose). System.locale is the same.

    sudo dconf-editor shows different values: org.cinnamon.desktop.input-sources is now completely blank. org.gnome.libgnomekbd.keyboard is also blank.

    I run system-config-language. It shows “English (Great Britain) highlighted. If I press on the button “System Defaults” it returns a box that says “Do you really want to change system language to default [en_US]?”

    /etc/locale.conf has LANG=”en_GB.UTF-8″. Should it also have something else? With this is place, why is the default according to system-config-language en_US? Is there another setting somewhere else?

    Can you make any sense out of this?

  5. Bill Berry
    Bill Berry February 5, 2014 at 11:15 am | | Reply

    No – just 00-keyboard.conf.
    I found this thread: https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=1028207 but didn’t understand most of it.
    /etc/locale.conf had LANG=”en_GB-UTF8″. That seems to be an error, so I changed it to LANG=”en_GB.utf8″. It didn’t make any difference.
    /etc/vconsole.conf had KEYMAP=uk. I hope that means United Kingdom, but as it might mean Ukraine, I changed it to gb. No difference.

    dracut -f, then reboot. No difference.

    I don’t have a /lib/kbd/keymaps/i386 folder.
    “ls /lib/kbd/keymaps
    legacy xkb”
    “cat /proc/cmdline
    BOOT_IMAGE=/boot/vmlinuz-3.12.9-301.fc20.x86_64 root=UUID=6384bded-1446-4b2e-98df-96269a5a40ac ro vconsole.font=latarcyrheb-sun16 rhgb quiet LANG=en_GB.UTF-8″

    en_GB.UTF-8? Is that correct? Should it not be en_GB.utf8? (locale -a shows it as en_GB.utf8) How do I change it if it is wrong? As it is wrong, that may explain why it defaults to en_US?

  6. Bill Berry
    Bill Berry February 6, 2014 at 10:25 am | | Reply

    I found the answer – it IS ibus. There is a small icon on the right of the lower bar,which looks a bit like a keyboard with a tail (connecting cable). Click on it, and it gave me 2 options: US or US.

    Right-click for a menu, and choose Preferences. The window that comes up is titled “IBus Preferences”. Select “Input Methods” tab, then “Select an input method”. I found a number of English options, including “English – English (UK, Extended WinKeys)”. Select that, click “Add”. I also removed US, but it was still there (as second and third choices) on reboot. However, the UK layout has priority, and is the one I have when I reboot.

    Problem solved – but I still don’t know why I have IBus if I don’t need it. I have other issues to sort out, but they have nothing to do with the keyboard. Now that I have a solution (even if not ideal), I shall stick here.

    Thank you for what you are doing to try to simplify keyboard configuration. I hope that this additional complication may help you in troubleshooting other’s problems, and possibly simplifying further.

  7. Antron
    Antron March 16, 2014 at 6:25 pm | | Reply

    Hi, my apologiese to chime here. I have a similar problem but needing ibus. I do use a Japanese keyboard in both romanji mode and kana/ kanji mode. The keyboard selected in Region & Language settings ins “Japanese (Kana/ Kanji)”.. which probably should be be the kkc ibus method. Anyway, after selecting this everything works fine until I reboot. After reboot, I get a US keyboard layout and hence can’t use the japanese keyboard options (i.e. Kana/ Zen..) to, among others, switch between Romanji and Japanese input anymore. I tried playing with the “Use custom keymap” option in the IM preference. It offers US & Japanese. But selecting either or neither, still will default me to the US keyboard layout after reboot. How can I make it stick?

You can comment without reCAPTCHA by using an OpenID as the URL, or logging in with an OpenID or an old site account.

Leave a Reply