Sysadmin adventures: local DNS and more!

It's 4am, Adam's still up, and we're not in a validation crunch...so that must mean he's got his amateur sysadmin hat on!

My latest 'little' project: setting up local DNS resolution for my network, and rationalizing a few hostnames and IP addresses so internal and external hostnames match and there aren't ugly gaps in my IP ranges. All done and (fingers crossed) working, now. This machine is now www.happyassassin.net to me as well as you; formerly it was webserver.localdomain on the inside, which was just silly. ;) And I only have loopback entries in my static hosts files now. Much rejoicing, etc etc. In the end the only really tricky bit was getting hostnames set by DHCP, which is a bit of lily-gilding really, but what the hell, I wanted to make it work. Still can't do it on my desktop, NM doesn't seem to be co-operating, but it's working on all the servers at least, which still use network.service:

[root@www ~]# hostname www [root@www ~]# hostname -f www.happyassassin.net

whee!

This is just a prelude to the REAL fun times, tomorrow or whenever I get a few spare hours, when I'm planning to set up an LDAP server and convert all my services to use it for authentication. I may set up happyassassin accounts (which, currently, means mail and owncloud, pretty much) for some of my family, so I'd probably better have a more sane auth setup than 'everything uses its own auth database'...

...although now I think about it, maybe just pam_userdb would do the job sufficiently. Hmm. Have to think a bit.

ANYHOO, the above also reminded me to trumpet my latest triumph: after finally fixing CalDAV a few weeks back, I managed to fix CardDAV too. Somewhere I found log output which gave me the idea that the problem might be the UIDs of the contacts in my owncloud contact list; I think I'd populated it by syncing things across from Google in some crazy way, and when I downloaded a few as .vcfs and looked at them, the UIDs for the contacts were really messy, with all kinds of weird characters in there, % signs and lord knows what else. So I just wiped the whole contact list and re-imported the latest state from Google, this time by logging into contacts.google.com, downloading the entire address book as a .vcf, and uploading that to Owncloud. This got me a rather cleaner set of contacts in Owncloud, and now syncing with CardDAV works! From all my clients!

So finally - until something breaks again - I have perfect CalDAV and CardDAV sync from multiple Linux and Android clients against my own server. I've wiped my Google contacts and calendar again. Take that, tentacle. Except of course the tentacle has ten backups of it, but sigh...

Oh, yeah, and just for an encore I set owncloud to use an NFS share from my NAS as the storage space for user files, so now I have more than, like, 6GB of space in my 'cloud' (each of my server VMs itself has pretty limited disk space). Tweak, tweak, tweak...

XMPP (Jabber): Surprisingly Better Than It Used To Be

tl;dr version: Jabber (now known as XMPP) is way better than it used to be. It's super simple to set up an account, set up a decent modern Android or Linux client, and do text, voice and video(!!!) chat, and send files, and even do desktop remoting. And even host your own server if you like. So I'm back on the service: add me as adamw AT happyassassin DOT net (self-hosting FTW!) if you like.

So if you're anything like me, you may have used/tried Jabber for a bit back in the day, found it a bit over-complicated or awkward or buggy, not really had much use for it, and then stopped thinking about it for a long time.

Well, I was on vacation and wanted to meet up with Bochecha. The proprietary-but-popular-and-easy messaging service of choice with my friends is WhatsApp, so I was mostly using that for messaging to avoid paying SMS fees. Turns out, though, there's no WhatsApp for the Palm(!) phone bochecha runs. Stymied!

Look, I said, we're geeks, we can figure this out, and fairly quickly realized, hey, Jabber...er...sorry...it's called XMPP now...would be the geeky way to do it. So we did. bochecha sent me his ID, which is on the talkr.im server. I found none of my old accounts still worked, so I thought hey, I'll use what he's using.

First Modern Day XMPP surprise: modern Jabber providers are good! At least, talkr.im is pretty no-nonsense. (I think they're the same people who write ejabberd). You go to their perfectly decent site, enter a user ID and a password, and in about ten seconds you have an account. Well, that was painless.

Ah, but that's only half the battle, I thought. Now I have to fight with my XMPP client. Surely that'll suck as much as I remember.

Second Modern Day XMPP Surprise: they stopped making the setup so damn unnecessarily complex!

A bit of Googling suggested that Xabber is a nice open source Android client, and guess what, it actually is. I installed that, told it my ID and password and nothing else, and it actually successfully connected to the server. On the first try. I nearly fell off the bus.

Pushing my luck to its limits, I added bochecha's account to my contact list, and wrote a message to him. And sent it. And he sent one back to me. And it worked!

So...yeah, XMPP is still this incredibly open-ended protocol with all sorts of neat capabilities and features and extensions and whatknot, but you can also just use it to 'text' someone with about as simple a setup as is possible without tying your account to your phone number, and it all actually works and doesn't fall over or break in weird ways. Offline message delivery works. Typing notification works. Statuses work. This is surely not news to anyone who stuck with things, but when you last tried this stuff in like 2002 or something, it comes as a welcome and novel experience.

So this evening I decided to push my luck yet further, with continuing amazingly positive results.

I set myself up an XMPP server. In about ten minutes. It worked, second try (after I figured out there was some permissions problem with the TLS key, but that's pretty small beer). yum install prosody, poke at the config file for three minutes, set up port forwarding and firewalls for ports 5222 and 5269, and...it worked. I can send messages from my talkr account to my newly-created account right here at happyassassin.net.

I can also, and I really did nearly fall out of my freaking chair when this worked first time, sign into the talkr.im account on my laptop and the happyassassin.net account on my desktop and do a video chat between them. Using Empathy. And it worked, first time. And the sound worked. And it didn't fall over mysteriously half way through, or fail when I tried again. I imagine you can run into codec issues trying to talk to people using non-Linux-y clients, but holy moley, just working right out of the box between Fedora 19 and Fedora 20 is a hell of a lot better than I remember just about all my historic video calling attempts being. (It doesn't seem to work in F18, for me; something crashes as soon as you try to initiate or answer a voice/video call). There doesn't appear to be a stable Android client with support for the video/voice stuff (which is called Jingle) yet, but apparently the Android port of Jitsi which is currently available as unstable, unsupported nightlies can do it; I'll try that in a bit.

So yeah. If you're like me and you haven't tried XMPP for a long time you might want to give it another shot. It is actually kind of awesome now. Two minutes creating an account and adding it to empathy and you can video chat (and, of course, audio chat and plain old boring normal chat) with anyone else who spends two minutes doing the same thing. If you're a personal server nerd like me you can also set up your own federated server in like ten minutes, but there's not much reason to unless you think all public servers might be evil or you just want to have your Jabber ID match your personally-hosted email ID. Which is pretty damn cool. Mine does, now: I'm adamw AT happyassassin DOT net for XMPP as well as email. ;) Next stop, SIP?

edit: just tested, and you can also do remote desktop and file sharing. And, again, they both work. It's like the future!

Airline Security Theatre, Part #5,659,100

The topic of airline security theatre is one that has been harped on before, but there's always more to come, apparently.

Checking in for my Air Canada flight home from vacation, I noticed a third question had been added to the stuff about whether some nefarious villain could have slipped something into one's luggage, which runs along these lines:

"Does your carry-on luggage contain anything sharp or pointed, or anything which could be adapted to cause an injury to another person?"

headdesk

The only possible truthful answer is, of course, "yes". For everyone in the world. Just about anything can be adapted to cause an injury to another person. I can knock you over the head with a bag of mixed nuts. I could choke you to death with a piece of tissue paper if you'd hold still for long enough.

And yes, of bleeding course my carry-on contains something pointed. It's called a 'pen'. I carry it for the purpose of filling out forms asking me ludicrous questions about whether there's anything sharp in my carry-on.

Look, one very vaguely understands the point - it's the "box cutter question" - but it's still utterly ridiculous. I'm still not entirely sure of the answer to the oft-made point "do they actually expect anyone to say 'yes' to these questions?" - my best guess is that they can use the "no" for some kind of legal maneuver if you subsequently do go postal with your pencil or whatever - but even assuming there is some kind of reason for asking them, this question is a fundamentally idiotic one. Enough already, airlines / regulatory agencies of the world.

The effect of technology on labour markets

This article should be required reading for any newspaper columnist who writes one of those silly, superficial 'omg technology is TAKING ALL THE JOBS' articles. Actually, required reading for anyone. It looks at the precise impact of current trends in technology on various types of employment; every time I read one of the dumb thinkpieces my irritated reaction is very much along the lines of what the NYT article says, but it's much better researched and articulated. Right on the money.

Happy happy joy joy: CalDAV works again

I am so happy right now I could actually bounce: with many thanks to the caldav-sync author Marten Gajda, I finally figured out what was wrong with my OwnCloud instance's CalDAV support.

I very briefly had a working CalDAV setup before - I could sync my desktop, laptop and phone and it all worked. It was great. Then I tried OC 5.0 and it kinda stopped working right. Then I dropped down to OC 4.x again and my PCs mostly still seemed to work, but I could no longer set up sync with an Android client. At all. Also, adding the OC account via GNOME's 'Online Accounts' didn't add its CalDAV stuff to Evolution, though I could do it manually.

After lots of back and forth with Marten we found out a newly created account on my server worked fine - it was just my account that didn't. Grr.

Finally, a wireshark capture of an attempt to register via caldav-sync as me bore fruit: somehow, the server was sending the wrong component type when telling caldav-sync what sort of stuff it supports. It should send something like "VEVENT,VTODO", but it was sending the string "Default calendar", which makes no sense at all.

After a bit of poking I found this is set as a property of each calendar in the OwnCloud database, in the table oc_calendar_calendars as the "components" column. A simple:

UPDATE oc_calendar_calendars SET components="VEVENT,VTODO,VJOURNAL";

was all it took to fix it, and suddenly my Android devices can get my calendars again, setting up the OC account via GNOME makes its calendars instantly pop up in Evolution, birds sing in the trees, and all is right with the world.

I also noticed that both my Android devices and Evolution can actually sync tasks with the OC server via CalDAV too, so I'm using that. I have my todo list in Evo on my PCs and as a neat widget on one of the home screens of my Android devices. I can categorize todo items and check 'em off from anywhere, and it's awesome. I was using mytinytodo before, but this is way better, as I don't have to fiddle with signing into a web interface on my phone. Ahh, glorious.

Now, if only it'll all hold together for longer this time...

Flock 2013, and re-attaching the tentacle

First things first: Flock 2013 was excellent, fantastic conference.

You may have noticed I started out tweetin' and bloggin' up a storm, and then disappeared for a while. This is because, after attending some useful and interesting (and sometimes even both!) presentations for a couple of days, I discovered the glory and magnificence that is Badges.

What is Badges? Badges is the way, the truth, and the light! Well, it's a pretty awesome award system for Fedora, at least, with killer artwork and powered by some super-shiny technology. An earlier, blasphemous version of myself may have pooh-poohed it - perhaps on these very pages! - but once you see it in action, it's such a neat little thing. And the silly little badge system encourages people to contribute to Fedora in all sorts of ways. It's really an awesome setup.

So what I mostly did for the last two days of Flock was, well, hack on badges. I bugged Ralph Bean and David Gay and Luke Macken and Ian Weller and anyone else who didn't stand still long enough about badges. I suggested a bunch of badges. I earned a bunch of badges. And most usefully, I more or less customized up this entire site for the Badges folks.

Badges has been pretty successful and lots of people were starting to suggest ideas, and the badges team were getting a bit snowed under. They were just throwing all the suggestions in a completely non-customized trac instance. So I tried to set up a bit better process for them, and put a nice pretty front end on it for users and people who want to contribute to badges. It's where you can go and suggest an idea for a badge, or try to help flesh out existing badge ideas with names, descriptions, artwork ideas, and actual artwork and badge definitions if you're able to do those things. There is (what I hope is) a simple but efficient setup to triage the status of badge suggestions and a whole wad of queries I worked up so you can find badges that need definitions, badges that need art, badges that need ideas, pretty much anything. So far it seems to be working out okay, and I'm glad I was actually able to contribute something!

I also filled more or less an entire notebook on the plane home with notes about issues that are going to crop up if Badges continues to be successful in the long term: I'm going to draft those up into something useful for the badges mailing list just as soon as I can find the time.

Hell, I even managed to write up some badge definitions myself, including Python queries(!!!), with lots of patient hand-holding by Ralph and Ian. It's pretty awesome that a few of the badges already out there and in production were suggested and written by me and a couple of others, and the team of folks doing artwork has similarly expanded.

Aside from all the Badges fun, we managed to put together a super productive session on the last day. Someone - I think viking-ice - pointed out that we could simply run a 'live QA meeting' (that's a video link) as a hackfest, with most of the QA folks there, plus ARM and cloud people too, with IRC running for those not at Flock. So we did that, and it worked out really pretty great. We nailed down more or less exactly what we need process-wise to handle ARM and Cloud as first-class citizens for Fedora 20 (which they totally are going to be!), and I've been working to get all that put into production in the last few days.

I also told quite a few people at Flock that I like to imagine most of the people there (and pretty much everywhere else) walking around with a giant tentacle inserted into a major orifice and connected back to either Google or Apple HQ, depending. Makes life much more interesting. I also had a Peak+ on pre-order and was kinda interested to try out Firefox OS.

Well, I don't any more. I found myself at a loose end the other day and decided to give the tentacle a trial run, and you know what, I know why everyone's addicted to the damn tentacle. The tentacle brings such tentacle-y goodness. Google Now is actually really freaking useful. I can tell it to remind me about things and it does! It's like some kind of demon magic. I figured out how to stick a shortcut on my home screen that immediately gives me transit directions home from wherever the hell I happen to be, saving the annoyance of opening maps and opening the directions screen and telling it where I want to go for the umpteenth time.

Damnit, tentacle.

The other thing is that the Peak+ is pretty spectrum-impaired - it only runs HSPA on a couple of frequencies, and no LTE.

So I did a ton of research (quickly re-discovering the annoying side of Android use - all the goddamn hassle of figuring out how the hell to do anything more than buy a subsidised phone and use it locked to your carrier) and eventually figured out that what I really want is a Sony Xperia ZL C6506. Yeah, not a Moto X or a Nexus 4 or an HTC One or a Galaxy S4 or even the generally-preferred Xperia Z, I want that one specifically. For two reasons: 1) it's relatively cheap (goes for under $400 for what's basically a phone on the same level as the Z and S4 and One) and 2) it has I think the best frequency set of any phone in existence. It's pentaband HSPA and quad-band LTE, which basically means you can pretty much use it frickin' anywhere and get high-speed data. (The 'C6506' part is important: the C6502 has AWS HSPA but no LTE, and the C6503 has LTE but no AWS HSPA. And even the C6506's bootloader unlockability varies depending on who you buy it from. Yeeeeesh.)

The Nexus 4, and T-Mobile versions of a few other phones, are pentaband HSPA, but usually don't have LTE (at least officially). You can hack LTE into the Nexus 4, but you lose A-GPS if you do. And the ZL is definitely faster than the N4.

So, I bought one! And, so far, I love it. Except for one problem (well, two, but one big one): I can't get the damn thing unlocked, yet. Either SIM unlocked or bootloader unlocked. Bootloader unlock is a no-go until someone figures out an exploit, but I'm still waiting to hear on SIM lock. I can live without an unlocked bootloader, as the stock firmware is surprisingly non-terrible; once you set up your loader and apps it's actually a pretty nice experience and genuinely improves on Android in a few ways. But I really need it SIM unlocked, as I'm damn well not paying roaming rates to use it in the US or Hong Kong or Europe. So if I can't get it SIM unlocked I'm going to have to unload it and buy another one. Siiiigh. I do wish all the manufacturers and carriers would just quit with all this freaking locking palaver.

So, yeah. Looks like the tentacle is going to be firmly ensconced at least for a while. If Mozilla can come up with something that's got most of Google's shiny I'm still up for switching, but current Firefox OS and Peak+ seem to be a bit too far behind :( I'm still resisting Google+ for as long as I can, though...

Flock 2013 (and stuff)

Well hi there, strangers. I'm sitting in Robyn's keynote at Flock 2013, so obviously I need to do something other than listen to what happened to her this one time at band camp (yep, really)! Also, I need to write a blog post so Fedora Badges will pick it up. Fedora Badges is the awesome new gamification thing for Fedora which I told everyone who'd listen I was way too cool to get sucked into, so of course as soon as it went up I started refreshing the leaderboard every three seconds...

Flock is looking like it'll be fun so far - all the usual suspects and more are here, there are going to be some good talks, and Charleston seems like a nice town. We're actually IN town as well, as opposed to a Businessman Hotel in the wasteland between the airport and the university, so that's great. I'm giving talk on Saturday about submitting updates the right way (for some reason - never drink with the talk submission page open), which I'm pretty sure will be awesome when I get around to writing it. You should totally come. If you're not at Flock, there's a live stream of the main auditorium, and I think there may be some form of streaming or IRC live reporting of other talks: Mo has two posts with info for those who want to follow the conference from afar.

In the 'down time' between Fedora 19 release and Fedora 20 branching the QA elves have been working on various bits and pieces. We (by which I mean smart coders who are not me, like Tim) migrated AutoQA to run on Fedora 18 so it can stagger along for another half a year while we work on Taskbot. Our GSoC student Branislav has been kicking ass working on Gooey Karma, a GUI tool for sending Bodhi feedback on Fedora updates - check it out if you didn't yet, it's great (though still being worked on). We're also working on various updates and additions to the release criteria and release validation test cases to try and make sure we're actually running tests that cover all of the criteria, that everything is up to date with recent changes to anaconda, initial-setup and so on, and extend our test coverage to cover new functionality in the installer and the addition of ARM as a primary architecture and cloud images as prominent downloads.

I also had an ARM day of my own last week, finally getting a Fedora (19) installation onto the hard disk of my Trimslice and updating my XO 1.75 to the latest upstream build. Once you get it running, Fedora ARM really is mostly just Fedora, but dealing with the bootloader is - as rwmj pointed out - lots of fun...

I've been running Rawhide (so, F20) on my desktop since almost the day after F19 release, and trying to do my bit to do rebuilds and stuff to keep it usable. Kevin Fenzi has been writing a good series of posts on running Rawhide - my experience mostly matches his, although I run GNOME and we're going through the 3.9 unstable series at the moment, so there are a few more exciting bugs in that to deal with from time to time. It's definitely day-to-day usable, though.

If you're at Flock, find me and say hi! I'll try and update a few times, but as usual I'll probably just drink instead. You know how it goes. Will has brought moonshine: this can only end well.

Special Fedora 19 Test Day for FreeIPA Active Directory Trust improvements tomorrow 2013-07-25!

The main Fedora 19 Test Day cycle wrapped up a few weeks back, but we've had a special request from the FreeIPA team to run a test day for some improvements to Fedora 19's FreeIPA implementation - so we're doing it!

Tomorrow, Wednesday 2013-07-25, we'll be testing a couple of new features that the team is hoping to add to Fedora 19's FreeIPA. If you're using Active Directory via FreeIPA on Fedora 19 you may well be interested in coming along and helping out with the testing. All the relevant information and instructions are on the wiki page, and we'll be in #fedora-test-day on Freenode IRC for discussion and any help you need in testing or debugging. If you're not sure what IRC is or how to use it, we have instructions here, and you can also simply click here to join the chat through a Web front end.

Fedora 19/20 logfile explosions

PSA: if you're running Fedora 19 or 20, I highly recommend you stop what you're doing right now and do this instead. There are a couple of unfortunate bugs in F19/F20 right now which may well be screwing the hell out of your log files.

systemd-journald bug combines with new rsyslog to eat your CPU time, make /var/log/messages huge, break journalctl and generally spread woe and misery SELinux denial happening every three seconds to GNOME users

Chapter One

1) Run top and check if 'rsyslogd' is sucking a bunch of your CPU time 2) If so, immediately stop rsyslog.service, do 'yum downgrade rsyslog', and restart it 2b) If not, breathe a sigh of relief, and make a note not to update to rsyslog 7.4.0 until this blows over. But still read the following chapters

EDIT 2.5) There's now a systemd update which should both prevent any further buggy journal entries being written, and cope with reading existing buggy entries better: systemd-204-8.fc19. After updating to that systemd, the infinite loop in buggy journals should no longer occur. If you prune your journal as described in the rest of this post, and update to systemd 204-8, you should then be able to update back to rsyslog 7.4.0 without further problems. Once we get systemd 204-8 pushed stable, new F19 installs should no longer ever be affected by this problem in any way, so please upkarma it.

3) If you got bitten, /var/log/messages is now probably infeasibly huge (mine was 8GB). The easiest thing to do is probably just nuke it. If you have valuable logs, they can probably be extracted from journalctl.

...

ON WHICH TOPIC

Chapter Two

If you got hit by chapter one, you're now in for some joyous fun. If your system logs are of no particular important to you, you can probably just go to /var/log/journal and blow .journal away, and you should now be more or less happy (though read Chapter Three too). The files in /var/log/journal are the Fabled Systemd-Journald Binary Logfiles themselves: you wipe one, you lose all the logs in it.

If your system logs are important to you, you get to have some fun. Run 'journalctl --no-pager' and wait a while. It'll likely start looping around some very small period of time, endlessly. ctrl-c it, go to /var/log/journal , do 'ls -ltr' and spot the file(s) with a very close timestamp to where the loop happened. Move those files somewhere else (if you want to retain maximum data, you can move them one at a time, doing 'journalctl --no-pager' after each move until you hit the exact combo of files you need to move to unblock it. You may need to rinse and repeat: I had five different loops in mine.

You may now be able to update rsyslog to 7.4.0 again and be okay, but personally, I'm leaving it on the old build till things quiet down a bit.

Chapter Three

You may well still have a giant assload of SELinux alerts about bug #973849 in your logs - "SELinux is preventing /usr/libexec/accounts-daemon from 'read' accesses on the directory /var/log." accounts-daemon tries this read about every three seconds, spamming the logs each time it fails. So first order of business, install selinux-policy-3.12.1-52.fc19 to fix it. Then you can clean up your logs, if you like.

I just gave up and wiped all the really large files in /var/log/journal - if you do an 'ls -lSr' in that directory you should see that most files are a few MB at most, but a few recent ones are tens or hundreds of MB in size. I killed all of those. You can leave 'em, but they eat space and make journalctl really slow. Your choice.

Chapter Four

While doing all this you might notice a bunch of rather old files with the extension "journal~" in /var/log/journal . It seems that journald doesn't rotate these (they're journal files that were uncleanly closed on shutdown). I manually deleted all the really old ones I had.

Chapter Five

If you have gnome-shell-extension-fedmsg installed and you're not really using it, you might want to consider removing it, because it spams the hell out of the journal: see bug #974429. Just something else I noticed on my log sanitization quest.

I am now running a sweepstakes on the first slashdot "systemd SUCKS!" post to cite this little mess as justification for the 'binary logs are awful!' meme...