Further sysadmin adventures: where's my FreeIPA badge?

So, hey, I have a working FreeIPA deployment for happyassassin.net.

After I put up my last post about how to do unified authentication for the various services I run through happyassassin.net, I got like five tweets and a couple of comments telling me to use FreeIPA. It's the obvious choice!, they said. It's easy!, they said. It only takes half an hour!, some jackass said.

To these people, I blow a gigantic raspberry.

But first, I figured I was smarter than them anyway, and didn't need it. The services I'm mostly interested in having unified authentication for are mail - which for me means dovecot and postfix - and owncloud. I already had postfix configured to auth via dovecot's SASL (which I'd totally forgotten about), so they were already in lockstep, and owncloud has an IMAP authentication option.

So it's easy as pie, right? Turn on IMAP authentication in owncloud and proceed directly to Go.

One problem: owncloud's IMAP authentication does not freaking well work. You will not convince me otherwise. I've read every goddamn Google result for 'owncloud imap', carefully negotiated all the gotchas documented therein, and it still doesn't goddamn work. Nothing logs anything useful anywhere. All I have to go on unless I feel like breaking out wireshark (here's a clue: I DON'T EVER FEEL LIKE BREAKING OUT WIRESHARK) is a 500 error from the server, and a 500 error from an HTTP server is roughly equivalent to "because I don't FEEL like it, that's why."

So. That was a bit of a dead end. Having cursed for a bit and set fire to some things, I decided, screw it, I can set up FreeIPA. It's a bit like using a large atomic bomb to crack a nut, but it's all shiny and I can probably convince people that it is somehow work-related and it'll, I don't know, save me about ten seconds of sshing into boxes and running 'passwd' when I want to change my passwords, or something. And hey, it's only going to take me half an hour anyway, right?

So, cue that raspberry.

I don't know how half hours work in CERTAIN PEOPLE's worlds, but in mine, it takes me half an hour to get through the damn introduction to the FreeIPA installation guide, never mind the rest. I've been up past 4am every day this week so far fiddling with it. But hey, now I actually have it working, and it is pretty shiny.

I don't even remember all the hurdles I leapt on the way any more, but here are some of them. One, DNS. You kind of want to let FreeIPA do your DNS for you, you see. If you're going to have a 'real' deployment of it on a domain with hundreds or thousands of machines and services, which will change quite a lot, you really want to let FreeIPA do your DNS for you. But I don't want to let FreeIPA do my DNS for me, because of REASONS.

Well, okay, because it's just not a very good fit for my setup, right? I'm running a small home network with a bunch of systems which won't ever be part of the FreeIPA domain, and there are issues with how I have my VPN access set up, and it just...didn't look like it was going to work out very well at all to have my network's DNS run out of the FreeIPA server. It makes a lot more sense to run it out of my router, which is what I'm doing. You can set up a subdomain with the hosts that are actually in the FreeIPA domain and just let FreeIPA manage the DNS for that, but ehhh, that would've been even more finicky in some ways. And the practical drawbacks to running DNS outside of FreeIPA once you get it working don't really apply to me: I don't need my boxes to be able to change their hostnames and push that info to the DNS server, and I don't need to keep moving services around between boxes and have the SRV records updated automatically when that happens, my network just isn't that big; I don't use SRV auto-discovery at all.

So. External DNS it was going to be. Setting up FreeIPA to work with external DNS is kind of a pain in the ass, though. Well, if you know a lot about FreeIPA and a lot about DNS I'm sure it's a walk in the freaking park, but for monkeys, no. That one took me most of one evening, including all the time I spent learning what the hell an SRV record is.

To save you the trouble, an SRV record is pretty much just a DNS record which says 'hey, on happyassassin.net, the LDAP server lives HERE'. Or something like that. You specify a service - which is something like LDAP, or Kerberos, or IMAP, or something like that - and list the name of the server that provides that service, the port and protocol, and optionally a couple of numeric parameters that are used for prioritizing multiple servers that provide the same service on big, grown-up domains (used for load balancing and fallback). In 'stuff I'm kinda glad I learned as a side benefit of this whole project', this is actually how mail auto-discovery works (well, it's the way it ought to work, I think Microsoft has their own ridiculous way of doing it too or something): that's RFC 6186 (on which the Evolution maintainer, Matthew Barnes, has some interesting thoughts, but I digress).

Anyhoo. If you're going to run FreeIPA with an external DNS server, you basically need to stick a whole whack of these SRV records in your DNS configuration. If you let FreeIPA handle DNS for you it does all this for you during server deployment, of course. When you run the server deployment script, FreeIPA will spit out an example configuration file in bind format that you can use. Unfortunately, the DNS server on my router's firmware is DNSmasq, so I had to figure out how to transpose the bind config lines to DNSmasq format, which took an hour or so before I grokked it. DNSMasq's man page and example config are invaluable here. And the format is:


srv-host= is the directive for dnsmasq that says 'add this arbitrary SRV record', _kerberos is the name of the service (always has a _ before it in SRV-speak for some reason), _udp is the protocol (ditto), id.happyassassin.net is the server, 88 is the port, and 0 and 100 are those numerical values I mentioned earlier, not really important unless you have multiple servers.

This record also illustrates one of the bear traps I fell into during this process: note the FQDN, id.happyassassin.net. According to the docs I read on the SRV format, just the hostname 'id' is perfectly valid so long as your DNS server will further resolve that to an IP, which mine does. If I set that bit of the line to be just 'id' and query the server with dig, the response I get is sensible. But for FreeIPA purposes, well, it's gotta be an FQDN. That simple. You want FreeIPA to work, make damn sure the server names in your SRV records use FQDNs. Public service announcement right there, folks.

edit: The FreeIPA devs would like me to point out that this is really a Kerberos requirement, not a FreeIPA requirement. Fine, folks, consider it pointed out. I wasn't trying to suggest that FreeIPA was being ridiculous here, just giving appropriate emphasis to the topic. To the monkey who never set up either before, the point is moot: Kerberos is just one of those scary bits that I'm counting on FreeIPA to deal with for me, and save me the trouble of knowing how the heck it works.

On that topic: actually, if you want FreeIPA to work, just generally make sure everything that can possibly be an FQDN is an FQDN. As part of this project I have learned that hostnames are far more psychopathically complicated than I ever wanted to know, but here is the very short version: for every machine that's going to be part of your FreeIPA domain, run 'hostnamectl set-hostname (fqdn)', and check that just plain 'hostname' returns the FQDN for that machine. This:

[adamw@id ~]$ hostname id [adamw@id ~]$ hostname -f id.happyassassin.net

is not good enough for FreeIPA (edit: fine, Kerberos). No. It needs this:

[adamw@id ~]$ hostname id.happyassassin.net [adamw@id ~]$ hostname -f id.happyassassin.net

So just give it that, and don't worry about it too much. If you are trying to be clever and get your hostnames set by DHCP, but when you do that, 'hostname' only gives a shortname, then give up on your clever DHCP thing and just whack the hostname with a hammer. Really. It's easier this way. (Please don't ask me why there is a 'hostname' and a 'hostname -f' because it will make me cry.)

Oh, and also, make sure every machine has a /etc/hosts line like this (for its own IP address and hostname):

192.168.1.XX machine.your.domain machine

make sure the FQDN comes before the shortname. You can even leave the shortname out. But don't have the shortname before the FQDN.

Another bear trap: it's very nice of the FreeIPA server install script to spit out an example BIND config when you run it with its own DNS configuration disabled, but it rather spoils the effect by going on to run the client install script automatically immediately after the server install script. Here's the problem: the client install script won't work properly unless your DNS records are in order. So inevitably it'll fail, because you haven't stuck all the DNS records in your DNS server's config yet. Sigh.

Somewhere around here I found that the freeipa client deployment script doesn't always clean up after itself perfectly when it fails. Bear trap: if you run the client install script once, and it fails, and you fix whatever was causing it to fail, and you run it again, and now it doesn't goddamn well auto-discover the IPA server even though it goddamn discovered it JUST FINE the first time and WHY?, the answer is 'rm -f /etc/ipa/ca.crt'. Bug #1011396. You're welcome. You might also get to a point where it gets so far as to actually register the system with the FreeIPA server but then fails later. If you do, then to do a complete setup once you fix whatever was causing the failure, you'll need to pass --force-join .

Once I finally worked out all THOSE bear traps, I got into a pretty good swing. I had a test client box which I used to get familar with the web UI and do a few test policies and things, and it was all working out nicely. So I was about ready to make the jump to actually registering my real desktop systems as clients and using FreeIPA in anger, to manage my HUGE RANGE of login accounts (which is...er...two, plus a 'facebook' account I use to sandbox facebook in a really janky way). Only one thing: sudo.

If you're going to centralize your auth and privilege policies you probably want it to include sudo, right? Bit pointless otherwise. When you find the obviously available instructions for enabling sudo with FreeIPA, you may, like me, be liable to run off screaming. It's not quite as complex as it looks at first glance, but it is still kinda complex. More significantly, AIUI, that approach has the rather significant drawback that it won't work if the system is offline. (Most bits of a typical FreeIPA setup do, thanks to sssd, which does credential caching; you can assume anything in your FreeIPA setup which runs through sssd will work just fine offline, I think).

Fortunately, there is another approach which is both simpler to implement and works offline, so what you should do is just use that. It's documented very concisely here: that is, indeed, all you need to do, on each machine in the domain. You will also want to add this line to /etc/rc.d/rc.local:

nisdomainname your.domain

or else group-based policies won't work, I think.

So once I found that out, I got on a nice roll of enrolling most of my machines in the domain and setting up a few basic policies. For each machine I'd tidy up so each existing local user matched the UID and GID for that user in the FreeIPA config, enrol it as a client, add the host to the appropriate host group so my policies took effect, set up the sudo stuff, check everything was working, and then delete the local user accounts. It was all working nicely all along the way.

Then I thought 'hey, I'm basically nearly done. I guess I should shut down the FreeIPA server VM and take a backup copy of it at this point.'

So I did. Then I booted up the FreeIPA server again. And found...no FreeIPA server at all. Just a failed ipa.service for no particularly obvious reason, in the logs. Not much to get my teeth into at all.

So I debugged that for four hours this evening, and let me tell you, that wasn't a fun one. It was complicated by the slightly weird way FreeIPA initializes itself: most of the relevant systemd units are disabled, and ipa.service actually starts them up itself. I'm guessing Lennart isn't a fan. I thought the fact that they were disabled was the bug and wasted hours fiddling about trying to enable them 'correctly', which wasn't the problem at all. No, it was this GIANT bear trap:

Sometimes, it seems, when you boot a FreeIPA server machine, the directive in /etc/tmpfiles.d/dirsrv-YOUR-DOMAIN.conf which tells systemd to create a directory called /var/lock/dirsrv/slapd-YOUR-DOMAIN...just doesn't work. That causes 389 Directory Server (FreeIPA's LDAP server) to fail to start.

edit: Thanks to viking-ice and nkinder, I find that this was recently re-reported, and the new bug reports have been far more fruitful. This is getting fixed, in multiple places (not just 389-ds has the problem, it turns out):

FreeIPA #996716 pki-core #996847 389-ds #1008306

Once I finally found that ticket, fixing it was easy: just manually run 'systemd-tmpfiles --create', restart ipa.service, and Bob's your uncle.

Well, he would be, except for this:

[adamw@adam ~]$ ssh id System is booting up. Connection closed by UNKNOWN

Ladies and gentlemen, what the freaking freak?

Maybe some of you ran into that error before. I don't know. But judging by Google, a lot of you are like me, and never have: Googling turns up some results, but not many, and they're mostly Arch users stumbling around and kinda fixing it but not really knowing what they did. I found one reference which clued me in to exactly what was going on, though.

If you ever see that message, what it means is that the file /var/run/nologin exists. This file seems to be as old as the hills, but I'd never come across it before. The idea is that it's created early in system startup and deleted at the end, then again created early in system shutdown, and while it exists, the system will deny any kind of login to any user but root. In a modern-day Fedora-y system, this is implemented by systemd (creates and destroys the file) and PAM (denies you access if it exists). The contents of the file are what is displayed when you are denied login; that's the "System is booting up." message in my output: systemd creates the file with the content "System is booting up."

So here's what happened: that file is actually created during early boot by systemd-tmpfiles, on a systemd system. It's then deleted at the end of boot by systemd-user-sessions.service (which more or less exists solely to do this). But if you have a problem with systemd-tmpfiles and run it manually to correct that problem...hey, you just created /var/run/nologin, and ain't nothing gonna delete it. You just locked yourself out. Congratulations.

The dumb fix for that is 'rm -f /var/run/nologin' after you run systemd-tmpfiles --create, but the smart fix is just to use a better systemd-tmpfiles command:

systemd-tmpfiles --create /etc/tmpfiles.d/dirsrv-YOUR-DOMAIN.conf

that tells tmpfiles to only run the directives from that specific config file, which is the only one we need, and avoids the whole /var/run/nologin problem, as that file is listed in a different tmpfiles config file.

So for now I just disabled ipa.service and have an icky /etc/rc.d/rc.local which runs systemd-tmpfiles then starts ipa.service, and that's working well enough for now.

And...whew, that brings me pretty much up to RIGHT NOW. I have a FreeIPA server which works, and survives a reboot, and all my systems but one are members of the domain, with no local user accounts any more, using the FreeIPA server for authentication and such. Oh, yeah, forgot to mention, one rather useful feature of FreeIPA is that you can store ssh pubkeys for each user, for 'authorized_keys' purposes: instead of having to create ~/.ssh/authorized_keys on every goddamn machine you ever configure, if you use FreeIPA, authorized_keys for any given user will just be read from the IPA server, so for any machine that's a member of the domain it'll "just work". Nice.

Just one thing: I haven't enrolled my mail server yet. Which is the original goal of the whole bloody exercise. But that's because it'll be the most complex bit - as postfix and dovecot are using the system user database for auth at the moment, I'll have to FreeIPA-ify those services as well, when I register it. So I was saving it for last. owncloud is using its own user db ATM, so I could happily register www in the FreeIPA domain and wipe local user accounts without breaking owncloud, but for mail, I'll have to work through the guides for setting up postfix and dovecot to use the FreeIPA server. But once I've done that, I should be home freeeee. And then, with luck and a following wind, I can pretty much grant anyone access to the main happyassassin.net services just by creating them a user on the server and sticking them in a particular group. Everything else should happen magically. Neato.


Stijn Hoop wrote on 2013-09-27 12:12:
Very interesting read, maybe it'll be worth my time to implement this at home as well. About the hostname bit: I think it is a bug that FreeIPA wants the kernel hostname as an FQDN. I understand they need to resolve stuff correctly, so having an FQDN in /etc/hosts and thus having 'hostname -f' return a resolvable IP adress always is preferable to relying on DHCP. But on a modern dynamicly connected system setting the kernel hostname to an FQDN is rather bogus...
Vincent Danen wrote on 2013-09-27 15:14:
A lot of that you can thank kerberos for. Proper DNS is _essential_ for FreeIPA. Now, I've not used FreeIPA itself (I use IPA on CentOS 6), but it's basically the same thing. The versions are pretty close (3.2 on Fedora 19 vs 3.0 on CentOS 6). Seems like you had a lot of troubles though. I didn't find setting it up nearly this difficult. One quick note... you really really REALLY want IPA/FreeIPA to run on its own machine/vm. Did I say you really want this? I meant it. I've got a dedicated vm that _only_ does IPA. It also doesn't handle DNS (but I have another server that does BIND which probably made it easier.. I didn't want IPA to handle DNS either for, as you, say, REASONS). For others reading, this document might be helpful: http://linsec.ca/Using_FreeIPA_for_User_Authentication I also suspect it wasn't so difficult for me as I've been using LDAP for auth on my network for a decade. I really like IPA, much better than the homebrew LDAP setup that I'd been using since the Mandriva days.
adamw wrote on 2013-09-27 16:47:
vdanen: I do have it running on its own VM, yeah. That's what 'id.happyassassin.net' is: the ID server. I didn't want to call it ipa.happyassassin.net because I'm not like all the other sheeple! I did use your blog as one of my references along with the FreeIPA guide for F18, indeed, sorry for not mentioning it. "I also suspect it wasn’t so difficult for me as I’ve been using LDAP for auth on my network for a decade." Yes, that would probably help. And if you used Kerberos before you probably know about the FQDN thing, and if you knew about THAT, you're less likely to have failures in ipa-client-install and hence less likely to run into the problems you get when you have failures in ipa-client-install. And the stuff with the tmpfiles appears to be new in F19, so if you set it up in F18 time frame, you dodged that one, and it'll probably be fixed by the time you upgrade your server.
Vincent Danen wrote on 2013-09-28 05:33:
Yeah, that sounds about right. I've setup Fedora 19 clients and they weren't a problem, but it's the server that would get you. And yes.. the most important thing about kerberos is DNS. The second most important thing is ntp. Glad I didn't have to fight this particular battle though!
Martin Kosek wrote on 2013-10-14 11:34:
Nice article, sorry for all the hoops you had to jump through when adopting FreeIPA. However, this is sometimes the way of evolution of a software, through pain and tears... But, good news is that you made it in the end, bugs and ideas for improvements were filed to our Bugzilla, some were fixed already, some are scheduled to be fixed. In case you would craft some nice FreeIPA HOWTO when integrating with 3rd party applications (mail server), please feel free to add an article or a link to our community wiki: http://www.freeipa.org/page/HowTos BTW, speaking of FreeIPA badge, I think you earned it - if you send me your mail address I can send you at least some FreeIPA stickers :)
Pavel Březina wrote on 2013-10-15 13:12:
Hi, this was fun to read :-) You may be interested in trying to make sudo rules available offline too... it is not auto-configured by FreeIPA yet but it is doable with recent SSSD (1.9+) and sudo (1.8.6+) versions. You will find more information in 'man sssd-sudo'.
adamw wrote on 2013-10-15 14:58:
Pavel: the references I linked above make sudo rules work fine offline - I've been running offline for the last three weeks as I've been in Europe, and sudo works fine...
Moshe Linzer wrote on 2014-01-27 18:02:
Thanks man, you saved my life with that FQDN info. I had figured out most of it, but the part I was missing was listing the FQDN first in the /etc/hosts file. Once I did that, ipa-client installed like a charm! Now on to the next challenge!
Anthony Affee wrote on 2015-01-09 19:24:
Wow what a great real article for once lol and it explained why you did things. Thanks for this. I am about to tackle FreeIPA or IPA for Centos 6 in the near future since our Company has grown and the days of manually applying changes and such to our soon to be 150 servers (0 desktops) will be a pain to manage without this. I dont want to use DNS either since our company uses RackSpace for pretty much our entire server infrastructure, but reading this I might try using DNS that is built in lol. For user management, since I dont have experience with Kerberos or LDAP, do you have to have user's accounts on the ipa server or can they fully stay on the individual servers but just authenticate via IPA? Also, I will dont use FQDN at all to name my servers. Should I just use a basic domain name and self-signed cert or do you think I should get a wildcard SSL cert from a Known CA? All of my servers are behind a firewall and only use internal IP Traffic, except for Web servers of course. Thank you.
adamw wrote on 2015-01-09 19:32:

"For user management, since I dont have experience with Kerberos or LDAP, do you have to have user’s accounts on the ipa server or can they fully stay on the individual servers but just authenticate via IPA?"

I'm not sure what exactly you mean by the accounts being 'on' the IPA server. For you to authenticate against the FreeIPA server, the server has to be the canonical reference for what accounts actually exist and what the appropriate authentication tokens are. It can't really work any other way. But if you mean stuff like the contents of /home/user, FreeIPA has nothing to do with that - that stuff is still stored on the individual machines, or however else you already arrange things.

You can still have local accounts on machines that are members of a FreeIPA domain, but the domain knows nothing about them. The machine itself handles authentication for such accounts.

"Also, I will dont use FQDN at all to name my servers. Should I just use a basic domain name and self-signed cert or do you think I should get a wildcard SSL cert from a Known CA?"

For FreeIPA you have to give all servers and clients proper fully-qualified hostnames. This is nothing to do with SSL, though.

The FreeIPA server automatically acts as a CA and issues certificates to client systems automatically for various purposes; it also provides a handy interface for issuing certificates for other purposes. There's no particular reason to buy a cert from a public CA for an in-house FreeIPA deployment that I'm aware of.

John wrote on 2016-03-22 05:30:
I have some raspberry pi's that I want to have IPA-SERVER control. Do I need to install IPA-CLIENT on those in order to manage them? I have not been able to install IPA-CLIENT on the raspberry pi's and I am not sure if I need it. If I need it for the raspberry pi's also, then IPA-SERVER is probably not for me. I only have one laptop. It would have paid off though if raspberry's could be included in the domain.