The great package format debate: why there's no need for distributions to use the same package format
As I mentioned, I spent the weekend at LinuxFest Northwest, a great conference I've been to three times now.
Bryan Lunduke does a recurring talk there which is called "Why Linux Sucks" or something like it. It's a great talk, where Bryan gives his tongue-in-cheek opinion on what's holding back desktop (and, more recently, mobile) Linux adoption. There's a video of this year's talk.
Bryan's awesome, and we get along great. I love his talks (in the last couple of years he's started following the "Why Linux Sucks" talk with a "Why Linux Rocks" talk which uses almost the exact same slide deck, which is an awesome idea). But I think he's wrong on one point he brings up. It's hard to do a good job of explaining why concisely during the talk - you can watch me doing a bad job of it in the video! - because it's a complex issue that's hard to distil. So I thought I'd write it down instead.
The problem Bryan identifies affects third parties providing Linux applications directly to users: Bryan trying to provide his games to users of different distributions, or Google trying to provide Chrome, or Mozilla trying to provide Firefox, and so on and so forth.
If you're one of those third parties, you can put your application in a package, or you can put it in a tarball.
If you put it in a package and provide a repository, your users can take advantage of the features of package management. Your app will show up in their package management applications, it will be updated when they do a system update, and a few other things. But there's a problem: different distributions use different package formats. So if you just do one package (and repository), you won't be covering all your potential users. If you do RPM and DEB you'll cover 90+% of them, but there are still a few who'll be left out.
If you put it in a tarball, you can just make one, and give it to users of all distributions. They'll all be able to extract your app and run it. But it won't show up in their package manager. You'll have to tell them to download a new tarball, or write your own update mechanism.
Bryan wants for third parties to be able to get all the neat stuff that comes with using a packaging format, without the inconvenience of doing multiple builds.
If distributions all used the same packaging format, would Bryan's goal be achieved? Absolutely. Is it a worthwhile goal? Sure. I don't think it'd change the world, but we agree it'd be a benefit.
So Bryan's right? I'm an idiot? Well...not so fast there!
Switching package formats for a distribution is a huge pile of work. I can explain why if anyone needs me to, but for now I'm going to assume you're willing to take it on trust.
If all distributions used the same package format, that wouldn't help them directly at all. A lot of people believe that if, say, Fedora, OpenSUSE and Ubuntu all used the same package format, you could just mix and match packages between them, and each distro wouldn't have to package everything separately.
This is not true, we can't ever make it true, and we don't want it to be true. I'll try to explain why as quickly as I can: distributions are very different, and they all are strongly committed to using shared resources.
When you get Chrome from Google, you get a static build: it uses very little of the shared resources the distribution provides, stuff that's present and compatible on virtually any distro.
When you download a package from your distro vendor, it's not like that. Distros use dynamic builds: distro packages use all the shared resources they can. Just take it on trust that the benefits are huge and distributions will never start using static linking outside of the cases where it's unavoidable (ask me for details if you want). Distros differ in terms of the shared resources they include, and this isn't just being ornery, it's a key part of what differentiates distributions from each other, and why we have them at all. Fedora 18 has openssl 1.0.1e. Ubuntu has 1.0.1c. RHEL 6 has 1.0.0. Debian stable has 0.9.8o. This is because Fedora is bleeding edge, Ubuntu is a bit more conservative, RHEL 6 is a lot more conservative, and Debian stable is even more so. We couldn't all get together and 'agree to compromise on 1.0.0'. Then none of those distributions except RHEL would be serving the needs of its users.
Multiply that by the thousands of packages in a distribution. You can't take a Fedora 18 package and install it on RHEL 6, even though they are both RPM distributions, because the package expects stuff from Fedora 18 that just isn't in RHEL 6.
So we've got a cost/benefit. The benefit? It's there. The cost? It's huge, and it falls on groups that don't directly benefit at all. This sucks. But fortunately, we don't need to pay that cost!
We don't need all the distributions to agree on a common package format for their packages so third parties can provide applications with most of the benefits of package management. All we need is for there to be a package management framework that third parties can rely on to be present on all distributions. It doesn't matter at all whether the distribution packages use it or not.
That's a much easier problem to solve. All we'd need to do is agree to all provide support for one existing package format out of the box. (Sidebar: LSB already actually tries this. It requires RPM.) That's one big bunfight at a conference and maybe like a week or so of work. No big deal. If Bryan pushed for this instead of saying 'all distributions should use the same package format!', I'd have no disagreement with him.
You can stop reading here if you like! But if you're thinking 'hey, waitaminute'...
To pre-emptively address one objection: what if a third party wants to provide a package that depends on something in the core distribution?
Bryan's idea would go a bit further to addressing that than mine would. But it's not so important, Bryan's idea still doesn't totally solve it, and you can modify mine to solve it quite easily.
It's not so important because most significant third party providers just want to make their package as independent as possible. They don't want to worry about making sure their app works with all the possible versions of the shared resources they want to use that are shipped by the various distributions. They usually just link almost everything statically. Here are the requirements of Google's Chrome RPM, for instance:
libatk-1.0.so.0()(64bit) libc.so.6(GLIBC_2.11)(64bit) libcurl.so.4()(64bit) libgconf-2.so.4()(64bit) libnss3.so(NSS_3.12.3)(64bit) libbz2.so.1()(64bit) libXss.so.1()(64bit) libXcomposite.so.1()(64bit) libXfixes.so.3()(64bit) wget xdg-utils zlib
Honestly, you really don't need to specify any of those. If someone's running a Linux distribution with a desktop (which they are if they want to install Chrome), it's about 99 to 1 they have all of that installed.
Bryan's idea doesn't fully address this requirement because distributions don't all agree on what different packages should be called, or how library dependencies should be written, etc etc etc. It's slightly more likely that we could all agree on that than that we could all switch to one package format, but it's still a hell of a long shot. Even if we all used the same package format, you couldn't just write "Requires: foo" and be done. One of the distros probably calls it "libfoo". You'd still have problems.
And you could pretty easily achieve this with my idea, too. Better, we don't have to do it all at once. We could just get the format in place first. If that was a success, then 'I want to be able to specify some dependencies!' is an obvious enhancement request, and now we're building on an existing idea, not just throwing impractical proposals around.
Maybe we can't get all the distros to agree on the names of absolutely all their packages, but we could get them all to agree to have their GTK+ 3.0 package provide gtk+-3.0 under the shared packaging system. The names could still differ under the native packaging systems - it wouldn't matter. But it's plausible to see us agreeing on a set of commonly-required core components, and doing the work to have the distro packages express those provides in the shared package manager as well as their native package manager, using an agreed nomenclature. And it could be incremental - we could just start by doing 50 packages, or 10, or just one; however many we could agree on. And then build it out as we went along. And none of this would in any way disturb the functioning of the distro's native package system and repositories.