Setup a Docker Registry

Introduction

If you’ve used Docker before, you’ve used a registry (whether you know it or not). Docker registries are where Docker images are stored and downloaded from. There are several well known Docker registries on the Internet, but our goal here is to be as self-sufficient as possible. Thus, we will host our own Docker registry. Any container images we want to use will be first loaded the local registry (pulled from an Internet registry) and then Nomad will load the images from the local registry. This will speed up re-downloading of images as tasks are deployed repeatedly. It will also prevent failures should internet access be lost or Internet registries become unavailable.

Environment

For the sake of this guide, assume there are three cluster nodes with addressing such as:

node1 - 10.0.3.101
node2 - 10.0.3.102
node3 - 10.0.3.103

Performing actions as root

The ideal security model dictates that interactively operating as root is incorrect, and that operations should run as a user, using ‘sudo’ to elevate permissions where necessary. Unfortunately, almost everything that needs to be done here will require ‘sudo’, so it will be faster to just become root and run everything as root:

sudo su -

Perform the following steps on one node until instructed otherwise

Pull the registry image

You will start by laying out the structure of the container data on the GlusterFS mount. Execute the following on one node (remember, GlusterFS will replicate these changes to all nodes):

cd /mnt/vagabond

mkdir docker && cd docker

mkdir _build && cd _build

mkdir registry && cd registry

At the time of the writing of this, ‘2.8.3’ was the latest tagged version of the Docker registry. You can see what the current tags are here: https://hub.docker.com/_/registry/tags

mkdir 2.8.3 && cd 2.8.3

Now you will create a script that will pull the Docker registry image, save it it to a local archive and then pull it into your local Docker library from the saved archive:

cat << EOF > build.sh
#!/bin/bash
export DOCKER_IMAGE=registry
export DOCKER_TAG=2.8.3
docker pull \${DOCKER_IMAGE}:\${DOCKER_TAG}
docker tag \${DOCKER_IMAGE}:\${DOCKER_TAG} local/\${DOCKER_IMAGE}:\${DOCKER_TAG}
docker image rm \${DOCKER_IMAGE}:\${DOCKER_TAG}
docker image save -o \${DOCKER_IMAGE}_\${DOCKER_TAG}.tar local/\${DOCKER_IMAGE}:\${DOCKER_TAG}
gzip \${DOCKER_IMAGE}_\${DOCKER_TAG}.tar
EOF

chmod u+x build.sh

cat << EOF > load.sh
#!/bin/bash
docker image load -i registry_2.8.3.tar.gz
EOF

chmod u+x load.sh

./build.sh

Perform the following steps on each node until instructed otherwise

Install the registry

The Docker registry will not run from Nomad, it is what one would consider a foundational service. Like Consul and Nomad, each node will run it’s own instance of the registry.

Load the container image for the registry service:

cd /mnt/vagabond/docker/_build/registry/2.8.3

./load.sh

Create the ‘docker-compose.yml’, assuming that there is no existing file:

cd /docker

cat << EOF > /docker/docker-compose.yml
version: '3'
services:
  registry:
    image: local/registry:2.8.3
    container_name: registry
    ports:
      - 169.254.254.254:8443:443
    volumes:
      - /docker/registry:/etc/docker/registry:ro
      - /mnt/vagabond/docker/registry/data:/var/lib/registry
    restart: unless-stopped
EOF

Create the configuration file for the registry service:

mkdir /docker/registry

Make sure you substitute your own domain name to reflect your internal domain.

cat << EOF > /docker/registry/config.yml
version: 0.1
log:
  fields:
    service: registry
storage:
  delete:
    enabled: true
  cache:
    blobdescriptor: inmemory
  filesystem:
    rootdirectory: /var/lib/registry
http:
  addr: 0.0.0.0:443
  tls:
    certificate: /etc/docker/registry/wildcard.pem
    key: /etc/docker/registry/wildcard.key
  headers:
    X-Content-Type-Options: [nosniff]
    Access-Control-Allow-Origin: ['https://registry.home.digitaldann.net']
    Access-Control-Allow-Credentials: [true]
    Access-Control-Allow-Headers: ['Authorization', 'Accept', 'Cache-Control']
    Access-Control-Allow-Methods: ['HEAD', 'GET', 'OPTIONS', 'DELETE']
    Access-Control-Expose-Headers: ['Docker-Content-Digest']
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3
EOF

Copy the wildcard certificates you issued:

cp /mnt/vagabond/ca/certs/wildcard.pem /docker/registry/wildcard.pem

cp /mnt/vagabond/ca/certs/wildcard.key /docker/registry/wildcard.key

Now start the registry service:

docker compose up -d registry

Make sure the registry resolves locally (again, make sure to substitute the domain name):

echo "169.254.254.254 registry.home.digitaldann.net" >> /etc/hosts

Some Notes

The registry service listens on 169.254.254.254, which is a virtual network interface that was created by the Ubuntu configuration playbook. This interface is useful so that containers can reach services specific to the local machine, and the interface address is the same across all nodes. Trying to reach the loopback interface (127.0.0.1) on the host from within a container will fail since each container also has a loopback adapter. If you want the registry to be accessible from outside the nodes, remove the IP address prefix on the port mapping in the compose file.

Also, note that the registry service is NOT running on the standard port 443 for HTTPS, but on an alternate port of 8443. This is to avoid conflict with reverse proxied applications on the standard port 443.

Conclusion

That’s all! Move on to the next part.