Today I’m diving into Mutual TLS to securely expose my homelab services! TLS is already ubiquitous in the modern era, providing strong symmetric encryption, perfect forward secrecy, and a public chain of trust to authenticate the server. But, it also has a lesser known ability to authenticate the client. By creating our own certificate authority to issue certs to clients, we can securely authenticate them to the server, preventing other users from even hitting our web app and probing it for vulnerabilities.

This is a simpler solution than using a VPN to ’expose’ your services, as long as the app is already relying on TLS (which includes more protocols than just HTTPS). There’s less user friction in installing a .p12 cert than setting up a VPN client, which could be important if you are sharing your services with friends and family.

Contents⌗

Video⌗

Video Thumbnail

Create a Simple Root CA⌗

This is section will setup a small 2-layer (root + leaf) authority, instead of the usual 3-layer (root + intermediate + leaf), for testing only. You should probably use the Smallstep CA for anything scalable.

I’m also using an ECDSA key chain, more to see how the process compares to RSA. So, the root will use scep384. I chose scep384 due to its arguably overkill security level, roughly equivalent to ~7000 bit RSA.

#Generate private key using scep384:
openssl ecparam -name secp384r1 -genkey -out root.key

#Sign the root certificate
#Pathlen:0 means there can be only one more cert below this CA (no more CAs)
#Make sure you update the subj name with your own names
#C=US is also the country, it's optional
#O= is the organization, also optional
#CN= is the Common Name and it's required
#I also set validity to 4 years, make sure you watch for expiration (manually)
openssl req -new -key root.key -x509 -nodes -days 1461 -out root.pem -subj "/C=US/O=apalrd.net/CN=test" -addext "basicConstraints=critical,CA:TRUE,pathlen:0"

#Now you can view it (for fun)
openssl x509 -in root.pem -text -noout

Sign a User Cert using the Root CA⌗

Now we can use openssl to generate and sign a user cert using the root key. I chose scep256 for this since it has a smaller ‘blast radius’ than the root and scep256 still provides roughly equivalent security to rsa 3072.

#Generate scep256 key for this client
openssl ecparam -name prime256v1 -genkey -out adventure@apalrd.net.key

#Generate a CSR (certificate signing request) for my new key
#again, C and O are optional, CN is the Common Name of the cert
openssl req -new -key adventure@apalrd.net.key -out adventure@apalrd.net.csr -subj "/C=US/O=apalrd.net/CN=adventure@apalrd.net" -addext "extendedKeyUsage = clientAuth"

#Sign the CSR using the root
#Sign it allowing for server and client auth as the key usage
openssl x509 -req -in adventure@apalrd.net.csr -CA root.pem -CAkey root.key -CAcreateserial -out adventure@apalrd.net.crt -days 365 -sha256 -copy_extensions=copyall

#Now you can view it (for fun)
openssl x509 -in adventure@apalrd.net.crt -text -noout

#Now let's package it into a P12 archive so you can send it to your favorite client device
#You *must* enter a password here or some OSes will not accept the P12
#The password just encrypts the P12 file itself
openssl pkcs12 -export -out adventure@apalrd.net.p12 -in adventure@apalrd.net.crt -inkey adventure@apalrd.net.key

Important note, I know it’s tempting to try to use Bernsein’s curves (ed25519 especially), but they aren’t approved by the CA + Browser Forum for use in public CAs, and aren’t implemented in any major web browser. So, I’ve chosen scep256 (which OpenSSL confusingly calls prime256) and scep384 for my CA. Of course, RSA is always an option as well.

Sign a User Cert using Smallstep⌗

If you’ve already setup a Smallstep CA from my previous TLS videos, you can use that instead of OpenSSL:

#Sign cert (run as root on the CA)
#laptop.crt/laptop.key are the key files
#I signed this one for a long ass time
step ca certificate adventure@apalrd.net laptop.crt laptop.key --not-after=2160h 

#Bundle into p12 and include intermediate cert we are using
#The P12 file can be imported into any OS
step certificate p12 laptop.p12 laptop.crt laptop.key --ca /etc/step/certs/intermediate_ca.crt

Setup Caddy for Client Auth⌗

Here’s the Caddyfile I used (Debian packs a start page with Caddy):

# The Caddyfile is an easy way to configure your Caddy web server.

#TLS email (global section)
#Please stop using my email for your Let's Encrypt certs
#I am sick of getting your renewal notices
{
    email mail@example.net
}

#Test website
test1.apalrd.net {
    #Caddy example lives here
    root * /usr/share/caddy
    file_server

    #mTLS verify client
    tls {
        client_auth {
            #Default is none
            #there are other options here if you want it to be optional
            #i.e. to bypass a signin page when using mTLS
            mode require_and_verify
            trust_pool file {
                #Can be specified multiple times for multiple roots
                pem_file /etc/caddy/root.pem
            }
        }
    }
}

Alternatively, you can make a snippet in the config to make it easier - this Caddyfile has the exact same behavior:

# The Caddyfile is an easy way to configure your Caddy web server.

#TLS email (global section)
#Please stop using my email for your Let's Encrypt certs
#I am sick of getting your renewal notices
{
    email mail@example.net
}

#Snippet for TLS client auth
(tls_client) {
    #mTLS verify client
    tls {
        client_auth {
            #Default is none
            #there are other options here if you want it to be optional
            #i.e. to bypass a signin page when using mTLS
            mode require_and_verify
            trust_pool file {
                #Can be specified multiple times for multiple roots
                pem_file /etc/caddy/root.pem
            }
        }
    }
}

#Test website
test1.apalrd.net {
    #Caddy example lives here
    root * /usr/share/caddy
    file_server

    #Reuse the snipper from earlier
    import tls_client
}

Setup Nginx for Client Auth⌗

Here’s a full nginx configuration (/etc/nginx/nginx.conf) with client auth. The default Debian config is quite .. verbose .. so I’ve cut out a lot of things which I didn’t need, but you might have needed them. Anyway, it’s framework. Nginx on Debian also subscribes to the whole sites-available and sites-enabled thing, which I find unnecessary.

user www-data;
worker_processes auto;
worker_cpu_affinity auto;
pid /run/nginx.pid;
error_log /var/log/nginx/error.log;
include /etc/nginx/modules-enabled/*.conf;

events {
    worker_connections 768;
    # multi_accept on;
}

http {

    ##
    # Basic Settings
    ##

    sendfile on;
    tcp_nopush on;
    types_hash_max_size 2048;
    server_tokens off; # Recommended practice is to turn this off

    # server_names_hash_bucket_size 64;
    # server_name_in_redirect off;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    ##
    # SSL Settings
    ##

    ssl_protocols TLSv1.2 TLSv1.3; # Dropping SSLv3 (POODLE), TLS 1.0, 1.1
    ssl_prefer_server_ciphers off; # Don't force server cipher order.

    ##
    # Logging Settings
    ##

    access_log /var/log/nginx/access.log;

    ##
    # Gzip Settings
    ##

    gzip on;


    ##
    # Virtual Host Configs
    ##

    # Redirect all HTTP to HTTPS
    server {
        listen 80 default_server;
        server_name _;
        return 301 https://$host$request_uri;
    }

    # Default (TLS) server configuration
    #
    server {
        # TLS (Server) configuration
        listen 443 ssl default_server;
        listen [::]:443 ssl default_server;

        # This is where Certbot puts them in standalone mode
        ssl_certificate     /etc/letsencrypt/live/test-nginx.apalrd.net/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/test-nginx.apalrd.net/privkey.pem;

        # TLS (Client) configuration
        # Possible values are 'on' or 'optional' (or 'off', the default)
        # If you use 'on', nginx will return 400 Bad Request if it fails
        # If you like 400, then use 'on'
        # If you'd rather drop, then use 'optional' and drop below (return 444)
        ssl_verify_client optional;
        # Path to the client CA
        ssl_client_certificate /etc/nginx/root.pem;
        # Number of steps we allow from the root
        ssl_verify_depth 2;

        # Drop connections which fail verification (optional)
        if ($ssl_client_verify != "SUCCESS") { return 444; }

        # Example website
        root /var/www/html;

        # Add index.php to the list if you are using PHP
        index index.html index.htm index.nginx-debian.html;

        server_name _;

        location / {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                try_files $uri $uri/ =404;
        }
    }
}