[oe] SSL Certificate Path for Python

[Ash Charles]

Hi

To follow up with one more piece of the puzzle: c_rehash.

With the suggested install_append in-place, openssl can find
certificates but they aren't indexed/hashed so aren't usable---the
c_rehash utility (from the openssl-misc package) needs to be called.
It looks like this would normally be done by the
update-ca-certificates script as part of the do_install step for the
ca-certificates package but it has been disabled by the
"0001-update-ca-certificates-remove-c-rehash.patch".  By disabling
this patch, c_rehash (albeit the host's version) runs and openssl is
able to verify correctly.

It looks like this patch has a long history [1,2, 3] from before 2009
when it was part of meta-shr (I think?) but I can't find the original
reason.  Anyone familiar with this?

Second: am I doing something wrong in my usage of OpenSSl? I started
debugging because the smart package manager couldn't use a package
repository at a https:// endpoint---a perhaps unusual use-case.  As
I've gone down the debugging rabbit hole though, it seems like SSL
basically shouldn't be working for any application relying on openssl
which makes me think either a) others are running into this or, more
likely, b) I'm doing something silly.

[1] https://git.congatec.com/yocto/meta-openembedded/blob/89500c583e0f1dc1b4ffdf72914e08e505e427e0/meta-oe/recipes-support/ca-certificates/ca-certificates_20090814+nmu2.bb
[2] http://lists.openembedded.org/pipermail/openembedded-devel/2011-April/077006.html
[3] https://github.com/openembedded/oe-core/blob/master/meta/recipes-support/ca-certificates/ca-certificates/0001-update-ca-certificates-remove-c-rehash.patch

--Ash

On Tue, Jul 7, 2015 at 5:37 PM, Ash Charles <ashcharles at gmail.com> wrote:
> Hi,
>
> I did a little more digging---the issue is not so much related to
> python as it is a difference in configuration between the
> ca-certificates and openssl packages.
>
> SSL certificates provided by the ca-certificates package are installed
> to /usr/share/ca-certificates/mozilla and then symlinked to the
> standard /etc/ssl/certs (controlled by ETCCERTSDIR) as a post-install.
>
> OpenSSL gets built expecting that certificates will be stored at
> /usr/lib/ssl/certs (check the 'openssldir' compiled in using 'openssl
> version -d')---a standard but empty directory.  This means we get
> verification failures for anything using openssl that doesn't
> explicitly provide a set of valid certificates.  In my case, this is
> breaking the Smart package manager when using a repository at a
> 'https' URL (the certificate is valid).
>
> Debian solves this by symlinking the whole /usr/lib/ssl to /etc/ssl/
> as a postinstall step on the openssl package.  I added this in a
> similar fashion in a bbappend for testing:
> do_install_append() {
>     rm -r ${D}${libdir}/ssl/certs
>     ln -sf /etc/ssl/certs ${D}${libdir}/ssl/certs
> }
>
> Does this seem like a reasonable approach?  Another approach could be
> setting the SSL_CERT_DIR to /etc/ssl/certs in .profile (or similar).
>
> --Ash
>
> On Mon, Jul 6, 2015 at 5:16 PM, Ash Charles <ashcharles at gmail.com> wrote:
>> Hi,
>>
>> With the move from python 2.7.3 (dizzy) to 2.7.9 (fido), Python
>> actually validates SSL-transport for https URLs [1].  Python, by
>> default (i.e. no environment variable SSL_CERT_DIR set), looks for
>> certificates at '/usr/lib/ssl/certs'.  I tested this in a Python
>> shell:
>> import ssl
>> ssl.get_default_verify_paths()
>>
>> The ca-certificates recipe seems to be installing certificates to
>> ${D}${sysconfdir}/ssl/certs or /usr/share/ca-certificates/mozilla/
>> instead.  I think that Python will need a way to find the system's
>> certificates.  I can create a patch to do this but this seems to
>> couple the configuration in the ca-certificates and python recipes.
>>
>> Has anyone stumbled across this issue?  Is there a standard way of
>> looking up where a system is storing its certificates?
>>
>> [1] https://www.python.org/dev/peps/pep-0476/
>>
>> Thanks for any insights---I'm learning much more about SSL
>> certificates than I expected today ;-).
>>
>> --Ash