Build packages in poudriere without OpenSSL in the base jail

Goals: Use poudriere to build clean packages which are guaranteed to not depend on openssl from base (by forcing it to be missing).  Provide details on what I encountered and how I worked around it to save other people time if they wish to continue on my path.  Encourage discussion about why these steps are needed in the first place and what can/will be done to properly solve some. 

This procedure is not meant as a full guide to using poudriere or a full guide to using libressl in ports.  It mentions many steps towards setting up poudriere but doesn't make an effort to entirely integrate with any poudriere set up you may already have.  I'm leaving that up to you.  This procedure is also not about creating a running system without openssl in base, but just a build jail for packages.

This procedure is mostly for people who already know how to setup poudriere and for people who already know how to make ports use libressl, but wish to ensure poudriere packages are not compiled against openssl from base.  It will leverage existing knowledge of poudriere and experience patching local ports for libressl compat where needed.  See Bernard's excellent work at https://wiki.freebsd.org/LibreSSL

This procedure is based on FreeBSD 10.1-RELEASE source code as a common reference point for compile options and patches, although newer builds can be used with less workarounds (noted below where known).

I cannot guarantee you will get perfect results with the ports you decide to package with this procedure.  I had to patch a few myself, and avoid some others like the EXAMPLES option of ldns, or ldns as a dep of openssh-portable.  The primary purpose is to make sure all built packages (except pkg itself) do not depend on openssl from base.  The secondary purpose is to expose some (but not all) ports that ideally need fixes to honor WITH_OPENSSL_PORT=yes properly.  The fact that a port compiles is not good enough because I've found packages resulting in things like this:

root@cacti:~ # lsof | grep libssl
httpd   22748    root  txt     VREG     186,2270822501             393456 249397 /usr/local/lib/libssl.so.30.0.0
httpd   22748    root  txt     VREG     186,2270822501             438864 224016 /usr/lib/libssl.so.7
httpd   22749     www  txt     VREG     186,2270822501             393456 249397 /usr/local/lib/libssl.so.30.0.0
httpd   22749     www  txt     VREG     186,2270822501             438864 224016 /usr/lib/libssl.so.7

Worse yet, my Apache 2.2 would crash at start.

Some of this is surely caused by net-mgmt/net-snmp trying to use openssl from base, then getting pulled in to apache through php.

This procedure should also be applicable to using openssl from ports, although I worked with libressl here.  Just remove OPENSSL_PORT= security/libressl from 10amd64-nossl-make.conf.

I am using the result for production packages on my systems.  Some of the steps may have to be manually redone like compiling pkg when a new version comes out.  Some of the other alterations I hope can be fixed in ports or the source tree so less workarounds are required.




Install poudriere if it isn't:

pkg install poudriere

Now we need to build a custom poudriere jail without openssl in base.  It is not yet as simple as WITHOUT_OPENSSL=yes, but as FreeBSD improves compilation options, this will get easier in the future.  I started with WITHOUT_OPENSSL=yes and made several iterations finding out which components to disable and how.  Most of them do not affect package building at all, such as iSCSI support in the package building jail.  Compiling libfetch without SSL support DOES affect poudriere somewhat since it uses fetch from inside the jail and won't be able to reach any https-only mirrors.  Fortunately many distfiles are still available without SSL.  If I run into a https-only distfile, I might just fetch it manually from outside the jail, but eventually fetch will need a workaround or a replacement in this setup.


Put these in /usr/local/etc/poudriere.d/10amd64-nossl-src.conf
to enable compile options for our build jail in poudriere:

# WITHOUT_OPENSSL=yes automatically causes a number of
# buildworld components to compile without SSL support
# or skip compiling at all.
WITHOUT_OPENSSL=yes
#
# Disable other things that will not presently compile
# with WITHOUT_OPENSSL=yes. The need to do this in
# addition to WITHOUT_OPENSSL=yes can be considered a bug.
WITHOUT_LDNS=yes
WITHOUT_LDNS_UTILS=yes
WITHOUT_PKGBOOTSTRAP=yes
WITHOUT_SVNLITE=yes
# 10.1-RELEASE needs this, newer builds don't such as r276751
WITHOUT_BSNMP=yes

# libfetch mostly supports WITHOUT_OPENSSL=yes but
# there is a small bug encountered when compiling it.
# This is handled as part of buildworld-nossl.diff.
# See https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=191951

# We still want to compile libfetch because pkg
# requires it and fetch uses it. Our resulting
# fetch and libfetch will not support SSL, but
# maybe poudriere can use something else instead
# of fetch for getting distfiles from https:// sites.

# When the below options exist, hopefully they are
# automatically conditional on WITHOUT_OPENSSL=yes.
# If not, that is a bug.

# WITHOUT_BSDINSTALL exists in 10-STABLE as of
# 20150213 r278713 but this is after 10.1-RELEASE.
# We'll patch usr.sbin/Makefile instead for now.
#
# WITHOUT_CTLD=yes does not exist yet in any version
# as of 20150324. We'll patch usr.sbin/Makefile instead
# for now.
#
# WITHOUT_ISCSI exists in 10-STABLE as of
# 20150211 r278555 but this is after 10.1-RELEASE.
# We'll patch usr.sbin/Makefile instead for now.

Some of the above options are not sufficient to complete buildworld. Below we'll use /root/buildworld-nossl.diff to patch a bug in libfetch when built with WITHOUT_OPENSSL=yes, and manually disable bsdinstall, ctld, iscsi as noted above. See https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=191951 for the libfetch bug:



Fetch some patches we'll need:

cd /root
fetch http://www.egr.msu.edu/~mcdouga9/nossl-patches/buildworld-nossl.diff
fetch http://www.egr.msu.edu/~mcdouga9/nossl-patches/patch-emulators-virtualbox-libressl
fetch http://www.egr.msu.edu/~mcdouga9/nossl-patches/patch-lang-python27-libressl

In /usr/local/etc/poudriere.conf I've only changed:
ZPOOL=rpool

This is to match the name of my zpool.



Build a new poudriere jail for 10.1-RELEASE from source while applying buildworld-nossl.diff:

poudriere jail -c -j 10amd64-nossl -v release/10.1.0 -a amd64 -m svn -P /root/buildworld-nossl.diff

Put this content in /usr/local/etc/poudriere.d/10amd64-nossl-make.conf
to set up some port changes to fix some port compiles in poudriere:

WITH_OPENSSL_PORT=yes
OPENSSL_PORT= security/libressl
# ldns examples subdir does not link with openssl properly
dns_ldns_UNSET=EXAMPLES
ftp_curl_SET=GSSAPI_NONE
ftp_curl_UNSET=GSSAPI_BASE TLS_SRP
#
.if ${.CURDIR:M*/lang/python27}
EXTRA_PATCHES+= /distfiles/custom/patch-lang-python27-libressl
.endif
#
.if ${.CURDIR:M*/emulators/virtualbox-ose}
USE_OPENSSL=    yes
CONFIGURE_ARGS+= --with-openssl-dir=/usr/local
EXTRA_PATCHES+= /distfiles/custom/patch-emulators-virtualbox-libressl
.endif
#
.if ${.CURDIR:M*/security/p5-Crypt-SSLeay}
CONFIGURE_ENV=  OPENSSL_INCLUDE="/usr/local/include"
.endif
#
.if ${.CURDIR:M*/security/ca_root_nss}
USE_OPENSSL=    yes
.endif

Prepare storage for some custom port patches:

ln -s /usr/local/poudriere/ports/default /usr/ports
mkdir -p /usr/local/poudriere/ports/default/distfiles/custom
cp /root/patch-emulators-virtualbox-libressl /usr/local/poudriere/ports/default/distfiles/custom/
cp /root/patch-lang-python27-libressl /usr/local/poudriere/ports/default/distfiles/custom/
rm /usr/local/poudriere/ports/default/net-mgmt/net-snmp/files/patch-agent_Makefile.in

Note about net-snmp patch delete:
/usr/ports/net-mgmt/net-snmp/files/patch-agent_Makefile.in causes snmp to link against libpkg apparently so snmp can report package information.  Since our libpkg is compiled against openssl which is not present in the jail, linking against libpkg will fail.  This may be complicated to fix properly, however since I do not use snmp in this manner, I found it was pretty trivial to avoid linking with libpkg.



Temporarily copy openssl in so pkg can compile.  It is not ready to link against libressl and it would be complicated when no other ports are installed.  It dynamically links against openssl for the 'pkg' command which is okay for systems that install our package of pkg because those systems will have openssl in base, but it also statically links against openssl and libfetch for 'pkg-static' which poudriere uses for compiles.

cp /usr/bin/openssl /usr/local/poudriere/jails/10amd64-nossl/usr/bin/
cp -Rp /usr/local/include/openssl/ /usr/local/poudriere/jails/10amd64-nossl/usr/include/openssl/
cp /usr/lib/libssl* /usr/local/poudriere/jails/10amd64-nossl/usr/lib/
cp /usr/lib/libcrypto* /usr/local/poudriere/jails/10amd64-nossl/usr/lib/
zfs destroy rpool/poudriere/jails/10amd64-nossl@clean
zfs snap rpool/poudriere/jails/10amd64-nossl@clean

Compile pkg only:

poudriere bulk -j 10amd64-nossl ports-mgmt/pkg

Delete temporary openssl:

rm /usr/local/poudriere/jails/10amd64-nossl/usr/bin/openssl
rm -r /usr/local/poudriere/jails/10amd64-nossl/usr/include/openssl/
rm /usr/local/poudriere/jails/10amd64-nossl/usr/lib/libcrypto*
rm /usr/local/poudriere/jails/10amd64-nossl/usr/lib/libssl*
zfs destroy rpool/poudriere/jails/10amd64-nossl@clean
zfs snap rpool/poudriere/jails/10amd64-nossl@clean

Now compile some sample packages:

poudriere bulk -j 10amd64-nossl emulators/virtualbox-ose ftp/curl net-mgmt/php56-snmp \
net/p5-SOAP-Lite www/apache24 mail/postfix security/openssh-portable

I've compiled over 1400 packages successfully using the methods above plus other patches just for libressl.

Additional notes:
x11/xscreensaver would not compile because 'bc' was missing from base. 'bc' is a wrapper for 'dc' which was also not built because it uses openssl for a math library. It may be possible to modify 'dc' to use a simpler math library, or make/find a 'bc' workalike from ports and make xscreensaver depend on it, even if it needs libressl, or find a different way for xscreensaver to do math. I wasn't actually using xscreensaver for anything, I'm more likely to use xlockmore which does build.

devel/android-tools-adb would not compile... I didn't look at why since I don't use it. May or may not be simple to fix.

dns/ldns has a subdirectory of examples with it's own Makefile, but it does not seem to look for openssl in /usr/local/. I spent some time trying to fix this but lost interest in favor of making everything else a usable solution.

ports-mgmt/pkg links against openssl from base which is understandable, non-trivial to extract, and unlikely to switch to an ssl library from ports. Because of this, you'll have to copy openssl back into your build root any time pkg needs to be built, build pkg, then remove openssl again. Perhaps it could switch to a bundled ssl library or equivalent replacement? Fortunately pkg version bumps are not extremely frequent and you can't forget to fix it because the compile will fail which will remind you.

Switching to OpenSSL from ports in FreeBSD

The intended audience is people who build their own ports or packages.  Switching ports to use openssl from ports is pretty straightforward.  These steps were summarized from my efforts several months ago.

Put WITH_OPENSSL_PORT=yes in the appropriate make.conf, then make sure all affected ports are recompiled.  I'm pretty sure poudriere bulk would take care of that.  (See end about libressl notes).

Nothing from the base system will compile against libraries from ports, at least without making ugly hacks.  I wanted openssh to use openssl (or libressl) from ports, so I had to switch my systems to openssh-portable:

Install openssh from my custom pkg repo:
pkg install openssh-portable

Copy existing SSH host keys:

cp /etc/ssh/*host* /usr/local/etc/ssh/

View config differences and manually resolve any issues:

diff -u /etc/ssh/sshd_config /usr/local/etc/ssh/sshd_config

Switch to openssh server from ports:

service sshd stop
sysrc sshd_enable=NO
sysrc openssh_enable=YES
service openssh start

If you are doing this remotely your SSH connection will not be dropped, but leave it open and test using a new connection before you disconnect the first.

In our setup, some systems depend on Kerberos which introduced a number of other issues I had to solve but I'll skip that unless someone requests it.  Dealing with Kerberos was at least half of my effort.

Since I'm now using openssh, openssl, and kerberos from ports and I do my own buildworlds, I desired to use WITHOUT_OPENSSH=yes, WITHOUT_KERBEROS=yes, WITHOUT_KERBEROS_SUPPORT=yes, and WITHOUT_OPENSSL=yes in src.conf to avoid having two installs of each.  Unfortunately, I can't use WITHOUT_OPENSSL=yes because it will break buildworld.  The source tree isn't prepared to cleanly skip SSL support in a number of places such as Kerberos and fetch.  That is open to experimentation if you want a custom build.  You'll have to put effort into replacing or compensating non-building portions of the source tree.

WITHOUT_OPENSSH=yes works pretty well and if you do your own buildworlds, you'll want to get rid of the old binaries so they aren't sitting insecure, unupdated, and possibly broken in the future.  make delete-old and delete-old-libs appeared to work but left behind a now-broken scp binary in my path. Since I do a delete-old as part of my standard upgrade procedure, I knew I had to fix OptionalObsoleteFiles.inc before upgrading my systems.  I determined missing files from OptionalObsoleteFiles.inc and filed a patch:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193980
I just sent a reminder to encourage someone to commit it.

When initially switching over or upgrading the openssl port, it is important to remember to restart enough daemons that still have the libraries open so they use new copies.  These commands are helpful:

lsof | grep libcrypto
lsof | grep libssl

Make sure lsof is up to date enough for your system so it produces accurate results.  (pkg install -f lsof).  For example, sometimes lsof compiled on 10.0 may not work on 10.1.  You'll probably see some false positives from shells, ntpd, etc because some binaries appear to inherit libraries from their parent process.  Processes such as Apache, postfix, openssh definitely need to be restarted.  If you are doing the initial switch from base to ports openssl, the paths of the binaries in lsof output will make it clear which are being used. However, in the future any time the openssl port is upgraded (replaced), active filehandles in lsof dissapear since the old files were deleted.  If you want to avoid rebooting, run lsof before upgrade so you know which processes to restart afterwards. The best way is to reboot after upgrading the openssl port/package.

At this stage, you've done most of the groundwork for switching to libressl if you prefer.  Just add OPENSSL_PORT=security/libressl to your make.conf (alongside WITH_OPENSSL_PORT=yes) and make sure your packages get rebuilt.  In the past you had to override OPENSSL_SHLIBVER but I think ports has been fixed to handle that automatically.