A note about SSL/TLS trusted certificate stores, and platforms (OpenSSL and GnuTLS)

Pop quiz: where is OpenSSL's default store of trusted CA certificate files?

  1. /etc/ssl/certs
  2. /etc/pki/tls/certs/ca-bundle.crt
  3. /etc/ssl/certs/ca-bundle.crt
  4. /etc/pki/tls/certs/ca-bundle.trust.crt
  5. /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
  6. /System/Library/OpenSSL
  7. Some other goddamn place

If your answer was 8. It's a trick question, well done, take this gold star for knowledge and/or test-taking skills. Depending on the platform you're on, any of the above could exist, and sensible choices are 1, 2, 4, 6 (OS X, at least according to this page, which gives some other fun if rather old choices), or 7, depending mostly on what OS / distribution you're running on.

OpenSSL

If you're writing software that does SSL/TLS certification validation using OpenSSL - well, commiserations. But also, please don't assume that any of the above locations exists, and certainly don't hard code one as the default. Usually, what you should do is try and use your SSL library's default store; if that fails, you can fall back on trying a few default locations. You should also provide a configuration option for the user to specify a store location and it should handle two different types of location.

For OpenSSL (and derivatives like LibreSSL), a store of trusted CA certificates can be a single file containing one or more concatenated certificates in PEM format, or a directory containing individual certificate files in PEM format, where each file is named in a specific format according to its hash value (these directories are usually produced by running the c_rehash command on a directory full of certificate files with more human-readable names, which produces symlinks in the expected format; p11-kit's trust extract / p11-kit extract command also has some support for doing this).

Debian, Red Hat / Fedora, and OpenSUSE have systems which produces a canonical trust store from the certificates found in various locations - the idea being to allow for flexible packaging of the distribution's own default trusted certificates, and for the system administrator to add to or override that list in such a way that software will respect their choices.

In Debian, the system ultimately populates the /etc/ssl/certs directory with certificate files and runs c_rehash on it. It also produces a bundle file, /etc/ssl/certs/ca-certificates.crt. I'm not sure which is considered 'preferred' for Debian purposes, if either is - but on Debian and derivatives, you can rely on /etc/ssl/certs being usable as a hashed directory, and /etc/ssl/certs/ca-certificates.crt as a bundle file.

In RHEL 5 and higher and all versions of Fedora you care about, RHEL/Fedora-derived distributions, and some others which are not derived from RHEL or Fedora but use its system (e.g. Arch, since 2014-09 or so) or follow its default locations, the system creates a bundle file. A newer and more capable system is used since Fedora 19 and RHEL 7 than was used in earlier releases, but both systems ultimately provide the bundle in OpenSSL's expected format at /etc/pki/tls/certs/ca-bundle.crt, so that is a safe default location for all Fedora releases and, I believe, RHEL 5+.

In OpenSUSE, I believe - see their system - the canonical locations are under /var/lib/ca-certificates (the canonical bundle file is produced at /var/lib/ca-certificates/ca-bundle.pem), and a hashed /etc/ssl/certs directory exists for compatibility with things that expect that (Debian) layout. As this post does, OpenSUSE explicitly recommends relying on the SSL library's default paths: "Your system openSSL knows how to read that, don't hardcode the path! Call SSL_CTX_set_default_verify_paths() instead."

Note that Fedora and RHEL provide a /etc/ssl/certs directory as an attempt at Debian compatibility, but it actually doesn't work very well for that at all - it just provides a bundle file /etc/ssl/certs/ca-bundle.crt (which doesn't match Debian's bundle name), and does not make any attempt to make the directory usable as a hashed directory at all. Basically, don't use RHEL/Fedora's /etc/ssl/certs, it's a trap. Don't use /etc/ssl/certs/ca-bundle.crt, even though it does probably exist on most current Fedora/RHEL installs; it's not a canonical location for Fedora/RHEL and it's unlikely to exist on anything else, and it may possibly go away on Fedora in future. Just don't use it.

So, if you wanted to handle trust store locations yourself, you could probably cover all or most Linux distros by checking for /etc/pki/tls/certs/ca-bundle.crt and /etc/ssl/certs in that order and using the first found, in the appropriate way (the first as a bundle file, the second as a directory). But you still wouldn't be covering non-platform builds of OpenSSL, or OS X, or Windows.

So really, what you should do - like I said - is first of all, try letting OpenSSL handle it. If you're using OpenSSL directly, the function you want is SSL_CTX_set_default_verify_paths(). If a default trust store was specified at the time the OpenSSL build your app winds up using was done, it'll get used. Distributions should always make sure this is properly handled, and it ought to work on OS X and Windows as well. Hilariously enough, the function isn't officially documented; this thread has some discussion of it. If you're using some wrapper it probably ought to either do this for you somehow, or expose that function for you to use in some way (see e.g. SSLContext.load_default_certs() in Python, though you really ought to use something like Requests for Python if at all possible).

If you're worried about cases where it somehow doesn't work, you can fall back to checking the Fedora/RHEL and Debian default locations. Knowing when to fall back might be a bit tricky; I forget the details, but I believe SSL_CTX_set_default_verify_paths() can fail, but it can also 'succeed' but result in an empty set of trusted certificates. You can probably check for that case. Exactly how hard you want to look for a default trust store, and what your app should do if it can't find one, is to an extent dependent on what your app does; it may make sense to bail out and warn the user, or go ahead without certificate validation (i.e. insecurely) and warn the user, or do something else, it's pretty context-dependent. Just think about it carefully.

Whatever that case, the other thing you should probably do (as I mentioned above) is allow for user configuration. Even with a distro that allows for modification of the default trust store, there may be a case where a user wants/needs to use a different trust store for your app. Maybe, given the nature of your app and how it's deployed, the user wants it to trust only their own site CA, for instance.

If you're allowing user configuration, you really ought to allow the trust store to be in either format. This is very easy to handle. If you're using OpenSSL directly, you use the SSL_CTX_load_verify_locations() function. The first argument to this function is always the SSL context. If you're loading a bundle file as the trust store, it goes as the second argument, and the third is NULL. If you're loading a hashed directory as the trust store, it goes as the third argument, and the second is NULL. OpenSSL wrappers for other languages usually expose this fairly directly. All you need to do is test whether the provided location is a file or a directory, and load it appropriately (and handle the case where it's neither appropriately - usually you're going to want to throw some kind of error; I wouldn't recommend falling back to the system default trust store, as if the user is going to the trouble of specifying a location, they're quite strongly implying that's not what they want).

An aside on OPENSSLDIR

If you poke around enough random forum / mailing list /Stack Overflow discussions of this stuff, you'll find the occasional assertion that someone (usually Red Hat) isn't following the 'OpenSSL defaults', and making everyone's life harder - e.g. here. So far as I can tell, this isn't accurate.

For the sake of thoroughness I looked into how SSL_CTX_set_default_verify_paths() actually works. It goes through a fairly complex code path, but basically what it winds up doing is trying to load certificates from 'the default file' and then from 'the default directory'. The default locations are defined in cryptlib.h as OPENSSLDIR "/certs" (default directory) and OPENSSLDIR "/cert.pem" (default file).

So, what's OPENSSLDIR? Well, you configure it at build time with the --openssldir to the Configure script. If neither that parameter nor --prefix is passed - which would be the closest thing OpenSSL has to a 'default' - it becomes /usr/local/ssl. AFAIK no distro actually uses /usr/local/ssl as its OPENSSLDIR. As far as I can see, no standard location has ever been defined by a commonly-adopted specification: there's nothing in any version of the FHS or LSB about OpenSSL locations. Therefore there really is no 'default' OPENSSLDIR location; distributions have been pretty much picking it out of a hat since the year dot.

What's that, reader? You want more specific historical trivia? Fine! I can but oblige.

From Debian's openssl changelog, it looks like /etc/ssl came into existence one snowy night (indulge me, here) in 1999:

 -- Christoph Martin <christoph.martin@uni-mainz.de>  Wed, 31 Mar 1999 15:54:26 +0200

ssleay (0.9.0b-2) unstable; urgency=low

  * Include message about the (not)usage of RSAREF (#24409)
  * Move configfiles from /usr/lib/ssl to /etc/ssl (#26406)

We can also track the invention of Red Hat's /etc/pki/tls precisely, to this bug report, from 2004 - I'm indebted to Ben Kahn for the reference.

Both references also indicate the previous location: /usr/share/ssl on Red Hat, /usr/lib/ssl on Debian. Indeed, the very first entry in the Fedora openssl changelog confirms this:

* Tue Oct 26 1999 Bernhard Rosenkränzer <bero@redhat.de>
- inital packaging
- changes from base:
  - Move /usr/local/ssl to /usr/share/ssl for FHS compliance
  - handle RPM_OPT_FLAGS

So Debian and Red Hat disagreed about where OPENSSLDIR should be at least since 1999 - from the first day Red Hat had a package for it. Debian's openssl package dates back to 1997 (as ssleay), so you could make an argument that Red Hat should've matched Debian's location (they'd already moved to /etc/ssl at the time RH's openssl package showed up), and our 16-years-later lives would have been a lot simpler. But they didn't and so here I am, picking through stone age spec files. BAD BAD Red Hat of 1999. You displease the monkey.

Bonus random Google reference: this book, published in 2003, indicates that SUSE was using /usr/share/ssl/certs as its trust directory at the time - so presumably using /usr/share/ssl, the same OPENSSLDIR as Red Hat, but shipping its system trusted certs in /usr/share/ssl/certs rather than as a /usr/share/ssl/cert.pem bundle.

Even more historical trivia: what the hell is the deal with Red Hat's certs/ dir anyway?

So doing all this digging made me start wondering...what the hell is the deal with OPENSSLDIR/certs on Red Hat anyway?

To recap - it's existed at least since openssl-0.9.5a-14, which is the oldest NEVR you can distinguish in the Fedora git repo. That's from before the ca-bundle.crt file was added to the package. It's not entirely clear why the directory was created in the first place; you can't see if it was there before Makefile.certificate or if it was created to put the Makefile in, and there's no comment or any context indicating whether it was created with regard to OpenSSL's expectation for such a directory as the 'default CA trust directory'. (I think there's an offline copy of the old CVS repos somewhere which may still let someone with access distinguish the first few commits to the package, but I'm not sure).

At the time, the only thing installed to the new /usr/share/ssl/certs was Makefile.certificate, installed as /usr/share/ssl/certs/Makefile, which still exists today. At the time it could create key pairs, CSRs, and self-signed certificates.

The ssl.conf file that was shipped in the mod_ssl package did not, at the time, expect to find the server certificate and key in /usr/share/ssl; it was configured to look in mod_ssl's own config dirs, under /etc/httpd/conf/.

I took a troll through the openssl 0.9.5a source, and near as I can tell, even back then, all it ever expected to find in OPENSSLDIR/certs was c_rehash-style individual certificate files. In fact, the whole SSL_CTX_set_default_verify_paths() complex was pretty much the same in 0.9.5a as it is today.

So far as I can tell, the first thing RH ever put in OPENSSLDIR/certs - Makefile.certificate - couldn't generate anything OpenSSL would expect to find there; it doesn't create files with the hash-style names, or call c_rehash. In fact, now I look at it closely, it was originally set up in its most simple invocations to output to the locations expected by mod_ssl: make genkey would create /etc/httpd/conf/ssl.key/server.key. So why the file was placed in OPENSSLDIR/certs in the first place seems obscure.

The Makefile seems to have been placed there in early 2000:

* Wed Mar  1 2000 Florian La Roche <Florian.LaRoche@redhat.de>
- Bero told me to move the Makefile into this package

I'd guess that before that, the directory existed but was empty.

So, my tentative conclusion is that no-one at RH ever really thought about what the directory was supposed to be for; they saw a directory called certs and thought, hey, that looks like as good a place as any to stick this Makefile. If Bero or Florian are still around and want to disagree with me, I'm right here :)

Finally, where the hell does ca-bundle.crt come from?

It turns up in the openssl package, in OPENSSLDIR/certs, in late 2000:

* Wed Oct 25 2000 Nalin Dahyabhai <nalin@redhat.com>
- add a ca-bundle file for packages like Samba to reference for CA certificates

and for a while I thought RH had simply invented the file out of whole cloth, but that turns out not to be the case. We nicked it from mod_ssl (official site is down as of 2018-09, linking to Wayback Machine), the oldest (I think?) SSL module for Apache (which is now part of the Apache codebase). According to the changelog in old mod_ssl tarballs, it invented the ca-bundle.crt file in 1998:

  Changes with mod_ssl 2.1b4 (08-Sep-1998 to 17-Sep-1998)

      3. A ssl.crt/ca-bundle.crt is now installed (but not enabled!) which
         contains all 33 CA root certificates of known public CAs.  They were
         extracted from Netscape Communicator 4.06 with my certbundle stuff.

I've had ever so much fun poking through old Red Hat Linux packages. In 7.0, we only shipped the mod_ssl copy, at /etc/httpd/conf/ssl.crt/ca-bundle.crt, but in 7.1, we added the file to openssl but did not drop it from mod_ssl, so both packages had copies. OpenSSL's copy had a Red Hat certificate appended to it; mod_ssl's did not. We didn't drop the mod_ssl copy until 8.0, by the looks of things.

It seems reasonable to infer that the file was copied (then moved) into the openssl package to make it available to things other than Apache without the need to install Apache (indeed there's an ancient samba bug indicating this). The filename was preserved from mod_ssl. But so far as the path goes, my guess is again, it was probably just put into OPENSSLDIR/certs because it looked like the right kinda name at the time. (Later, of course, the ca-certificates package was separated from the openssl package, and OPENSSLDIR changed from /usr/share/ssl to /etc/pki/tls, but OPENSSLDIR/certs/ca-bundle.crt has been there all along, just following these changes).

Well, maybe no-one else cares, but I feel weirdly satisfied at having tracked that down!

So far as other SSL libraries goes...

NSS

NSS handles things fairly differently - it stores certs in a database, not directly as files. NSS stuff seems to usually work properly more or less 'out of the box', so I've less experience with messing with it.

GnuTLS

edit: I just spent like two hours researching GnuTLS. I demand gratitude.

Current (very current) GnuTLS can use a PEM bundle, a PKCS #11 module via p11-kit, or a directory as a trust store. It does not handle directories in the same way as OpenSSL; it simply throws every file it finds in the directory at its 'load from file' code, so it'll presumably read in both individual certificates and bundle files found in the directory in the case of a hybrid directory like Debian's /etc/ssl/certs.

You use gnutls_x509_trust_list_add_trust_file() to load either a cert file/bundle or (a bit confusingly) a PKCS #11 URL into the trusted cert list. gnutls_x509_trust_list_add_trust_dir() is for directories.

GnuTLS introduced a 'system default trust store' concept on 2012-05-08 - specifically, with this commit. The first release with the feature was 3.0.20. At that point it could only be a PKCS #11 URL or a file; no directories allowed. The ability to use a directory was added with this commit on 2014-07-21, the first release with directory support was 3.3.6.

Since the default trust store feature was added, if a specific default trust store location was not passed at compile time, it has checked a few locations and used the first one that matched if any. At first it only checked for /etc/ssl/certs/ca-certificates.crt and /etc/pki/tls/cert.pem; on 2012-07-19 it was updated to also check for /etc/ssl/cert.pem and /usr/local/share/certs/ca-root-nss.crt (I'll have to look into that one...); and on 2013-06-01 it was updated to also look for /etc/ssl/ca-bundle.pem.

If you have a sufficiently new GnuTLS that was compiled with an appropriate parameter (or whose compile time location guessing worked), the function gnutls_certificate_set_x509_system_trust() tells GnuTLS to load the certificates from the default trust store. gnutls-cli will try this by default, which is a useful way to check if it's working on a given platform; run gnutls-cli google.com and look at the top of the output, see if there's a count of loaded certificates, or a warning message. When loading a directory it sets the GNUTLS_TL_NO_DUPLICATES flag, so presumably it won't actually end up wasting effort storing duplicates.

If you parse all that crap out and look at when it landed in various distros...well, you can be pretty sure that there's no default trust store in absolutely any GnuTLS build older than 3.0.20, for a start. That's a freebie. (It doesn't look like any of the LTS / enterprise distros backports the feature to 2.x). Debian and Ubuntu are still shipping gnutls 2.x in current releases, alongside 3.x; I have not been able to figure out if they're shipping any packages built against it yet (still trying to figure out the magic commands). RHEL 6, nope. SLES 11, nope. The first Debian which included 3.x at all is Jessie; the first Ubuntu I'm not totally sure, but it was at least in 14.04. The first OpenSUSE with a sufficiently new build was 12.2, I believe.

But even if the build is new enough it either needs to have had a correct parameter passed at build time, or the location guessing needs to have worked (that's not only a question of whether the distro provides one of the guessed locations; the package that provides it must have been installed when the package was built).

Fedora 21, OpenSUSE 13.2, Ubuntu 14.10, and Debian since 3.2.4-2 (which I think is in sid and wheezy at least) build with an explicit trust store location passed to configure, so it seems pretty safe to assume that the default trust store function will work on those versions and newer of those distributions.

For some, it will work with earlier releases thanks to the 'guessing' functionality, but you really have to check individually to make sure whether this is the case. I've tested Fedora 20, and it does work there (probably using the guessed location /etc/pki/tls/cert.pem). I haven't tested older Ubuntu or OpenSUSE builds yet, or RHEL 7.

So for gnutls, I'd say the advice is broadly the same as OpenSSL, but it's rather less likely that you can rely on the 'system default store' stuff working. It doesn't handle directories the same as OpenSSL, but most directories that works as an OpenSSL store ought to work as a GnuTLS store, so you should be able to try loading the same 'common' locations with the appropriate functions. You'll probably want to set the GNUTLS_TL_NO_DUPLICATES flag to avoid unnecessary duplication, especially when using directories. Note that GnuTLS cannot handle OpenSSL's 'trusted certificate' format - the one with BEGIN TRUSTED CERTIFICATE - so neither bundles nor directories containing certificates of this type will work.

So, tl;dr: please don't assume any given location of trusted certs can be relied upon - just because it exists on your dev platform, it doesn't exist for all your users. Go with the library's defaults, if possible. Otherwise, at least check the canonical locations for Fedora/RHEL (/etc/pki/tls/certs/ca-bundle.crt as a bundle file) and Debian (/etc/ssl/certs as a hashed directory, which will also work on OpenSUSE), not just one or the other. And allow a user specified location, and handle it being either a bundle file or a hashed directory.

(note: I'm not a security professional, just a QA monkey who keeps running into issues with this stuff; I think the above is all broadly correct, but corrections from distro and/or upstream SSL specialists are of course welcome!)

Comments

Bhaskar Chowdhury wrote on 2015-01-14 02:51:
Fascinating read Adam! kudo for the digging :)
Alex Biddle wrote on 2015-11-04 09:27:
Good post, I agree, kudos for the digging!
Oliver Schonrock wrote on 2015-12-01 18:18:
Hi, very nice article. If you have any energy left I have a special challenge for you. See my post here: https://lists.freebsd.org/pipermail/freebsd-questions/2015-November/269278.html I cannot for the devil in me, make this work. Is it a broken binary (seems very unlikely), or is there some more path magic happening.
Matěj Cepl wrote on 2016-03-30 14:46:
Except that on March 31, 1999 it was between 4° and 14°C so not much snowy night (unless, we are talking Stephen Leacock’s “Gertrude the Governess or Simple Seventeen” (“It was a wild and stormy night on the West Coast of Scotland. This, however, is immaterial to the present story, as the scene is not laid in the West of Scotland.”).
Matěj Cepl wrote on 2016-03-30 14:47:
Sorry forgot URL http://is.gd/ex9Rpw
adamw wrote on 2016-04-08 19:28:

I SAID INDULGE ME GODDAMNIT

Keating wrote on 2016-04-28 17:31:
What a marvelous, and informative read. Gratitude is gifted, and indulgence granted! Keating
A. Jesse Jiryu Davis wrote on 2016-07-30 01:19:
This is such a saintly effort, Heaven holds rewards in store for someone who put in the time to do OpenSSL code archeology like this.
adamw wrote on 2016-08-16 20:10:

Thanks a lot! I hope it makes up for all my other sins. ;)

Joachim Nilsson wrote on 2016-08-16 19:14:
This is like the single best source for information on this mess I've seen! Big thanks mate! Helps me fix https://github.com/troglobit/inadyn certificate validation. Cheers
adamw wrote on 2016-08-16 20:09:

you're welcome, glad you found it useful :)

SkyRaT wrote on 2017-02-25 17:37:
This is the only (visible) article out there putting all the info together, which I was googling and researching the whole afternoon. Nice job!
pozdrawiam wrote on 2017-10-24 12:09:
they are there: open("/usr/lib/ssl/cert.pem", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)