Hot on the heels of yesterday’s deploying the application, I realised that deploying a new version causes some downtime. Even more downtime if the deploy fails or the application isn’t working for some reason.

I was rolling out a new version like this:

docker pull my-app
docker compose up -d

Docker compose then notices that my-app has changed. It turns off my-app and then restarts it, recreating with the new image. Even if this is very quick, it still leads to nginx not being able to route requests for that period.

This calls for blue-green deployments; we run two copies of the application (a BLUE and a GREEN), we attempt to roll out the new update to BLUE and check if it’s healthy, then tell nginx to send traffic to there for a bit, and then do the same roll out to GREEN, and send nginx back to GREEN. Whilst GREEN is down and being deployed, BLUE handles all the connections and requests.

Abdullah Obaid has a great tutorial that I followed, and might be suitable for you. However, my needs were a little different:

  1. I have other services in my docker-compose which I needed to share. (No need to run two Postgres containers.)
  2. I didn’t want to use systemctl to inject an environment variable. I want all my configuration to be in my ~/my-app directory, not hidden away elsewhere.
  3. I have a different idea about how the health check and “switch” script should work.

Splitting services

My docker-compose.yml originally contained registry, web, database services.

I kind of want the registry and database to stick around and be shared between the two. It feels like I’ve wrangled this into working and I’d be interested to hear further thoughts. Here’s what I did.

  1. Keep docker-compose.yml
    1. But remove web service
    2. Specify a project name to avoid relying on Docker’s autogenerated names: name: my_app. This will be used to scope our services, which we want to reference later on.
  2. Create a docker-compose.blue.yml
    1. name: my_app_blue
  3. Create a docker-compose.green.yml
    1. name: my_app_green
    2. Give web a different exposed port

docker-compose.yml defines a volume which I need access to in BLUE and GREEN. Same with the network too.

# in docker-compose.yml

volumes:
  registry_data:
  
networks:
  external_network:
  internal_network:
    internal: true
  
# in docker-compose.blue.yml

volumes:
  my_app_registry_data:
    external: true
    
networks:
  my_app_external_network:
    external: true
  my_app_internal_network:
    external: true
    internal: true

This is a little brittle for a couple of reasons. First, docker-compose.yml has to be loaded first to create those resources. Second, I’m relying on magically getting the name of the resources correctly. It’s the main reason why setting the name was so important.

This does work, however!

Then we can start up each in order (though, the order only matters once).

docker compose up --file docker-compose.yml -d
docker compose up --file docker-compose.blue.yml -d
docker compose up --file docker-compose.green.yml -d

docker ps will show you all of your services working. You can even rails c into your BLUE to change something in the database, then pop over to GREEN to see it correctly reflected.

Directing traffic with nginx

You’ll have an upstream defined in your nginx.conf somewhere. You’ll remove the existing upstream and define two new ones: one for BLUE and one for GREEN.

upstream my-app-blue {
  server 127.0.0.1:3845 fail_timeout=0;
}

upstream my-app-green {
  server 127.0.0.1:3844 fail_timeout=0;
}

We need to tell nginx which one to use though, and to do that we’ll make a new file, nginx-my-app-upstream.conf. The full contents of which should be:

set $my_app_upstream "my-app-green";

And then we include that file in our server directive:

server {
  include /root/my-app/nginx-my-app-upstream.conf;
  server_name my-app.shane.computer;

  location / {
    proxy_pass http://$my_app_upstream;

    # Preserve client headers for Rails
    proxy_set_header Host              $host;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
  }
}

Now, whenever you run nginx -s reload, it’ll check our extra config file and set the variable to the upstream we want to serve.

So now, all we need is to smartly write to nginx-my-app-upstream.conf.

Toggling between upstreams

I’ve written a deploy-latest script (and chmod +x deploy-latest‘d it) whose job it is to:

  1. Pull the latest tag again. Note that this doesn’t change GREEN or BLUE.
  2. Start BLUE with that new tag
  3. Check if it’s working, using our already working healthcheck (comes for free with Rails)
  4. If yes, toggle the upstream
  5. Start GREEN with the new tag
  6. If it’s working, toggle nginx back
  7. Optionally, you can shut down BLUE at this point
#!/usr/bin/env bash
set -euo pipefail

docker pull my_app.shane.computer:5500/my_app:latest
docker compose --file docker-compose.blue.yml up -d

wait_for_healthy() {
  local container=$1
  local max_attempts=${2:-15}
  local attempt=1

  while [ $attempt -le $max_attempts ]; do
    if docker ps --filter "name=$container" --format ' ' | grep -q "(healthy)"; then
      echo "$container is healthy!"
      return 0
    fi
    echo "[$container] attempt $attempt/$max_attempts: not healthy yet..."
    attempt=$(( attempt + 1 ))
    sleep 2
  done

  echo "[$container] did not become healthy in time"
  return 1
}

wait_for_healthy my_app_blue-web-1
echo "set \$my_app_upstream \"my_app_blue\";" > nginx-my-app-upstream.conf
nginx -s reload

docker compose --file docker-compose.green.yml up -d
wait_for_healthy my_app-green-web-1
echo "set \$my_app_upstream \"my_app_green\";" > nginx-my-app-upstream.conf
nginx -s reload

Deploy flow

  1. On your development machine, build and push your new image.
  2. On your production machine, run ./deploy-latest.

Done!

Some people will tell you to look into Kubernetes or Nomad to help orchestrate and deploy with a more robust blue-green methodology, but if this works, then it works! Bonus: you understand every line of it making debugging easier in the future.

Enjoy.