For my day-to-day development work I currently have four separate physical servers, one old x86_64 server for file storage, two new x86_64 servers and one new aarch64 server. Even with a fast fibre internet connection, downloading the never ending stream of Fedora RPM updates takes non-negligible time. I also have cause to install distro chroots on a reasonably frequent basis for testing various things related to containers & virtualization, which involves yet more RPM downloads. So I decided it was time to investigate the setup of a local caching proxy for Fedora YUM repositories. I could have figured this out myself, but I fortunately knew that Matthew Booth had already setup exactly the kind of system I wanted, and he shared the necessary config steps that are outlined below.
The general idea is that we will reconfigure the YUM repository location on each machine needing updates to point to a local apache server, instead of the Fedora mirror manager metalink locations. This apache server will be setup using mod_proxy to rewrite requests to point to the offsite upstream download location, but will also be told to use a local squid server to access the remote site, thereby caching the downloads.
Apache setup
Apache needs to be installed, if not already present:
# dnf install httpd
A new drop-in config file addition for apache is created with two mod_proxy directives. The ProxyPass directive tells apache that any requests for http://<our-ip>/fedora/* should be translated into requests to the remote site http://dl.fedoraproject.org/pub/fedora/linux/*. The ProxyRemote directive tells apache that it should not make direct connections to the remote site, but instead use the local proxy server running on port 3128. IOW, requests that would go to dl.fedoraproject.org will instead get sent to the local squid server.
# cat > /etc/httpd/conf.d/yumcache.conf <<EOF
ProxyPass /fedora/ http://dl.fedoraproject.org/pub/fedora/linux/
ProxyPass /fedora-secondary/ http://dl.fedoraproject.org/pub/fedora-secondary/
ProxyRemote * http://localhost:3128/
EOF
The ‘fedora-secondary’ ProxyPass is just there for my aarch64 machine – not required if you are x86_64 only
The out of the box SELinux configuration prevents apache from making network requests, so it is necessary to toggle a SELinux boolean flag before starting apache
# setsebool httpd_can_network_relay=1
With that done, we can start apache and set it to run on future boots too
# systemctl start httpd.service
# systemctl enable httpd.service
Squid setup
Squid needs to be installed, if not already present:
# dnf install squid
The out of the box configuration for squid needs a few small tweaks to optimize it for YUM repo mirroring. The default cache replacement policy purges the least recently used objects from the cache. This is not ideal for YUM repositories – if the YUM update needs 100 RPMS downloading and only 95 of the fit in cache, by the time the last package is downloaded we’ll be pushing the first package out of cache again, which means the next machine will have cache miss. The LFUDA policy keeps popular objects in the cache regardless of size and optimizes the byte hit rate at expense of object hit rate. Some RPMS can be really rather large, so the default maximum object size of 4 MB is totally inadequate, increasing it to 8 GB is probably overkill but will ensure we always attempt to cache any RPM regardless of its size. The cache_dir directive is there to tell squid to use threads for accessing objects to give greater concurrency. The last two directives are critical telling squid not to cache the repomd.xml files whose contents change frequently – without this you’ll often YUM trying to fetch outdated repo data files which no longer exist
# cat >> /etc/squid/squid.conf <<EOF
cache_replacement_policy heap LFUDA
maximum_object_size 8192 MB
cache_dir aufs /var/spool/squid 16000 16 256 max-size=8589934592
acl repomd url_regex /repomd\.xml$
cache deny repomd
EOF
With that configured, squid can be started and set to run on future boots
# systemctl start squid.service
# systemctl enable squid.service
Firewall setup
If a firewall is present on the cache machine, it is necessary to allow remote access to apache. This can be enabled with a simple firewall-cmd instruction
# firewall-cmd --add-service=http --permanent
Client setup
With the cache server setup of the way, all that remains is to update the Fedora YUM config files on each client machine to point to the local server. There is a convenient tool called ‘fedrepos’ which can do this, avoiding the need to open an editor and change the files manually.
# dnf install fedrepos
# fedrepos baseurl http://yumcache.mydomain/fedora --no-metalink
NB on the aarch64 machine, we need to point to fedora-secondary instead
# fedrepos baseurl http://yumcache.mydomain/fedora-secondary --no-metalink
Replace ‘yumcache.mydomain’ with the hostname or IP address of the server running the apache+squid cache of course. If the cache is working as expected you should see YUM achieve 100 MB/s download speed when it gets a cache hit.
Last year I migrated my website off Blogger to a WordPress installation hosted on my Debian server. Historically my website has only been exposed over plain old HTTP, which was fine since the Blogger publishing UI was running HTTPS. With the migration to WordPress install though, the publishing UI is now running on my own webserver and thus the lack of HTTPS on my server becomes a reasonably serious problem. Most people’s first approach to fixing this would be to just generate a self-signed certificate and deploy that for their server, but I rather wanted to have a x509 certificate that would be immediately trusted by any visiting browser.
Getting free x509 certificates from StartSSL
The problem is that the x509 certificate authority system is a bit of a protection racket with recurring fees that just cannot justify the level of integrity they provide. There is one exception to the norm though, StartSSL offer some basic x509 certificates at zero cost. In particular you can get Class1 web server certificates and personal client identity certificates. The web server certificates are restricted in that you can only include 2 domain names in them, your basic domain name & the same domain name with ‘www.’ prefixed. If you want wildcard domains, or multiple different domain names in a single certificate you’ll have to go for their pay-for offerings. For many people, including myself, this limitation will not be a problem.
StartSSL have a nice self-service web UI for generating the various certificates. The first step is to generate a personal client identity certificate, which the rest of their administrative control panel relies on for authentication. After generation is complete, it automatically gets installed into firefox’s certificate database. You are wisely reminded to export the database to a pkcs12 file and back it up somewhere securely. If you loose this personal client certificate, you will be unable to access their control panel for managing your web server certificates. The validation they do prior to issuing the client certificate is pretty minimal, but fully automated, in so much as they send a message to the email address you provide with a secret URL you need to click on. This “proves” that the email address is yours, so you can’t request certificates for someone else’s email address, unless you can hack their email accounts…
Generating certificates for web servers is not all that much more complicated. There are two ways to go about it though, either you can fill in their interactive web form & let their site generate the private key, or you can generate a private key offline and just provide them with a CSR (Certificate Signing Request). I tried todo the former first of all, but for some reason it didn’t work – it got stuck generating the private key, so I switched to generating a CSR instead. The validation they do prior to issuing a certificate for a web server is also automated. This time they do a whois lookup on the domain name you provide, and send a message with a secret URL to the admin, technical & owner email addresses in the whois record. This “proves” that the domain is yours, so you can’t requests certificates for someone else’s domain name, unless you can hack their whois data or admin/tech/owner email accounts…
Setting up Apache to enable SSL
The next step is to configure apache to enable SSL for the website as a whole. There are four files that need to be installed to provide the certificates to mod_ssl
- ssl-cert-berrange.com.pem – this is the actual certificate StartSSL issued for my website, against StartSSL’s Class1 root certificate
- ssl-cert-berrange.com.key – this is the private key I generated and used with my CSR
- ssl-ca-start.com.pem – this is the master StartSSL CA certificate
- ssl-ca-chain-start.com-class1-server.pem – this is the chain of trust between your website’s certificate and StartSSL’s master CA certificate, via their Class1 root certificate
On my Debian Lenny host, they were installed to the following locations
- /etc/ssl/certs/ssl-cert-berrange.com.pem
- /etc/ssl/private/ssl-cert-berrange.com.key
- /etc/ssl/certs/ssl-ca-chain-start.com-class1-server.pem
- /etc/ssl/certs/ssl-ca-start.com.pem
The only other bit I needed todo was to setup a new virtual host in the apache config file, listening on port 443
<VirtualHost *:443>
ServerName www.berrange.com
ServerAlias berrange.com
DocumentRoot /var/www/berrange.com
ErrorLog /var/log/apache2/berrange.com/error_log
CustomLog /var/log/apache2/berrange.com/access_log combined
SSLEngine on
SSLCertificateFile /etc/ssl/certs/ssl-cert-berrange.com.pem
SSLCertificateKeyFile /etc/ssl/private/ssl-cert-berrange.com.key
SSLCertificateChainFile /etc/ssl/certs/ssl-ca-chain-start.com-class1-server.pem
SSLCACertificateFile /etc/ssl/certs/ssl-ca-start.com.pem
</VirtualHost>
After restarting Apache, I am now able to connect to https://berrange.com/ and that my browser trusts the site with no exceptions required.
Setting up Apache to require SSL client cert for WordPress admin pages
The next phase is to mandate use of a client certificate when accessing any of the WordPress administration pages. Should there be any future security flaws in the WordPress admin UI, this will block any would be attackers since they will not have the requisite client SSL certificate. Mnadating use of client certificates is done with the “SSLVerifyClient require” directive in Apache. This allows the client to present any client certificate that is signed by the CA configured earlier – that is potentially any user of StartSSL. My intention is to restrict access exclusively to the certificate that I was issued. This requires specification of some match rules against various fields in the certificate. First lets see the Apache virtual host configuration additions:
<Location /wp-admin>
SSLVerifyClient require
SSLVerifyDepth 3
SSLRequire %{SSL_CLIENT_I_DN_C} eq "IL" and \
%{SSL_CLIENT_I_DN_O} eq "StartCom Ltd." and \
%{SSL_CLIENT_I_DN_OU} eq "Secure Digital Certificate Signing" and \
%{SSL_CLIENT_I_DN_CN} eq "StartCom Class 1 Primary Intermediate Client CA" and \
%{SSL_CLIENT_S_DN_CN} eq "dan@berrange.com" and \
%{SSL_CLIENT_S_DN_Email} eq "dan@berrange.com"
</Location>
The first 4 match rules here are saying that the client certificate must have been issued by the StartSSL Class1 client CA, while the last 2 matches are saying that the client certificate must contain my email address. The security thus relies on StartSSL not issuing anyone else a certificate using my email address. The whole lot appears inside a location match against ‘/wp-admin’ which is the URL prefix all the WordPress administration pages have. The entire block must also be duplicated using a location match against ‘/wp-login.php’ to protect the user login page too.
<Location /wp-login.php>
SSLVerifyClient require
SSLVerifyDepth 3
SSLRequire %{SSL_CLIENT_I_DN_C} eq "IL" and \
%{SSL_CLIENT_I_DN_O} eq "StartCom Ltd." and \
%{SSL_CLIENT_I_DN_OU} eq "Secure Digital Certificate Signing" and \
%{SSL_CLIENT_I_DN_CN} eq "StartCom Class 1 Primary Intermediate Client CA" and \
%{SSL_CLIENT_S_DN_CN} eq "dan@berrange.com" and \
%{SSL_CLIENT_S_DN_Email} eq "dan@berrange.com"
</Location>
Preventing access to the WordPress admin pages via non-HTTPS connections.
Finally, to ensure the login & admin pages cannot be accessed over plain HTTP, it is necessary to alter the virtual host config for port 80, to include
RewriteEngine On
RewriteRule ^(/wp-admin/.*) https://www.berrange.com$1 [L,R=permanent]
RewriteRule ^(/wp-login.php.*) https://www.berrange.com$1 [L,R=permanent]
To be honest, I should just put a redirect on ‘/’ to prevent any use of the plain HTTP site at all, but I want to test how well my tiny virtual server copes with the load before enabling HTTPs for everything.
Hopefully this blog post has demonstrated that setting up your personal webserver with certificates that any browser will trust, is both easy and cheap (free), so there is no reason to use self-signed certificates unless you need multiple domain names / wildcard addresses in your certificates and you’re unwilling to pay money for them.