Setup Certificate Authority

Introduction

We all know certificates are critical for securing our self-hosted applications. They are also important for securing the communication between various nodes of a clustered application such as Consul, Nomad, and others. This will guide you through creating your own Certificate Authority and issuing a wildcard certificate for your internally managed DNS domain.

Environment

For the sake of this guide, assume there are three cluster nodes with addressing such as:

node1 - 10.0.3.101
node2 - 10.0.3.102
node3 - 10.0.3.103

Performing actions as root

The ideal security model dictates that interactively operating as root is incorrect, and that operations should run as a user, using ‘sudo’ to elevate permissions where necessary. Unfortunately, almost everything that needs to be done here will require ‘sudo’, so it will be faster to just become root and run everything as root:

sudo su -

Perform the following steps on each node until instructed otherwise

Determining the plan

You’re going to need to decide how you want to handle DNS domains for your internal network. Here are some options:

• Option 1 – Own a public (registered) domain, hosted somewhere such as Cloudflare, and use that domain for all your services both externally accessible and internal. Use Let’s Encrypt for certificate generation. If this is what you want to do, then consider that you are placing your dependency on an external entity for your TLS security. If you want to be truly free from the cloud and self host, this would seem to violate that intent. Perhaps another option is a better choice. However, this method is less complicated. Not by much though.
• Option 2 – Own a public (registered) domain, hosted somewhere such as Cloudflare, and use that domain for all your services both externally accessible and internal. Use a self-managed certificate authority for certificate generation. This alleviates the dependency on any external service to provide certificates, you’re self sufficient. However, you need to load your root certificates on all your devices. This isn’t that hard actually. However, no one will trust any externally accessible services (think Plex). Maybe not a good option either.
• Option 3 – Do not use a public (registered) domain at all, and do not host any externally accessible services. Use a private (un-registered) domain for all internal services. Use a self-managed certificate authority for certificate generation. This alleviates the dependency on any external service to provide certificates, you’re self sufficient. However, you can’t really make anything externally available without VPN.
• Option 4 – Own a public (registered) domain and a private (un-registered) domain. The public domain would be hosted somewhere such as Cloudflare, and you would use that domain for all your externally accessible services. Use the private domain, a sub-domain of the public domain or even a completely different domain, for all internal services. Use Let’s Encrypt to generate certificates for the public domain and use a self-managed certificate authority for for the private domain. This relieves the dependencies on external services for internal domains, and if the external certificate provider becomes unavailable all you lose is the trust of the external services. You still need to load your internal root certificates on your devices, but again that isn’t hard. This what I recommend and what I will demonstrate.

Identify your domains

In these examples, the public domain of ‘digitaldann.net’ will be used for externally accessible services. The private domain will be ‘home.digitaldann.net’, a subdomain of ‘digitaldann.com’, for internal-only services. The private domain can be something completely different, and not even a real domain or TLD such as ‘dann.iot’.

Create the internal certificate authority

Change your working directory to your GlusterFS mount, which in this example is ‘vagabond, and create the directory for the CA files’:

cd /mnt/vagabond

mkdir ca

cd ca

Get this ansible script, which will make this process quicker and easier:

wget https://raw.githubusercontent.com/digital-dann/nomad-cluster/main/create-ca.yaml

Edit the ‘create-ca.yaml’:

nano create-ca.yaml

And update the ‘Set Parameters’ section:

- name: Set Parameters
  set_fact:
    private_domain: 'home.digitaldann.net'
    cluster_ips:
      - '10.0.3.101'
      - '10.0.3.102'
      - '10.0.3.103'

The ‘private_domain’ is the domain discussed above… the private domain.

The ‘cluster_ips’ should include all the addresses of your cluster nodes.

Once you have updated the ‘create-ca.yaml’ file, you should execute it by running:

ansible-playbook create-ca.yaml

This playbook will do the following:

• create a ‘/certs’ sub directory
• Create a script to renew the generated wildcard certificate
• Create the necessary configuration files for the certificate authority
• Execute the script to generate the wildcard certificate

When the playbook is complete, you should see the new Root CA (root.crt/root.key) in the ‘/certs’ sub directory, along with the Intermediate CA (int.crt/int.key), and the Wildcard certificate (wildcard.crt/wildcard.key) which will provide TLS for all your services running on your cluster nodes. There is also a ‘wildcard.pem’ which is the Wildcard certificate with both of the CA certificates, used to provide the complete trust chain to services such as Traefik or Nginx.

Installing the Root CA on the cluster

You’ve generated a certificate authority, but your cluster nodes don’t trust it. We can make them trust it by importing the Root CA into the system trust chain (you only need the Root CA). You can do this by running these commands on each node:

cp /mnt/vagabond/ca/certs/root.crt /usr/local/share/ca-certificates/root.crt

update-ca-certificates

And that’s it!

How do I install the Root CA certificate on another device / operating system?

scp dann@10.0.3.101:/mnt/vagabond/ca/certs/root.crt .

certutil -addstore "Root" root.crt

scp dann@10.0.3.101:/mnt/vagabond/ca/certs/root.crt .

sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain root.crt