Upgrade the prod server

# Connect to the composition management server
ssh compose0.exoscale.infra.camptocamp.org

# a bunch of envvars with the list of all the servers
source /opt/c2corg-docker/server-list.env

# one directory for each production server, self-descriptive names:
ls /opt/c2corg-docker/composition/production/

# choose one
cd /opt/c2corg-docker/composition/production/api0/

Notice the hidden ./.env in this directory. https://docs.docker.com/compose/environment-variables/#/the-env-file This envvar file teaches docker-compose to connect to a remote docker instance. Each per-server directory has it's own .env file.

You can now edit the docker-compose.yml file. In this example, you'll want to change the image version to a more recent one which travis made available on dockerhub (https://hub.docker.com/r/c2corg/v6_api/tags/).

# check which image flavour is used, update file accordingly:
grep image: docker-compose.yml

# Set container in mainternance mode (will not get new requests for Haproxy)
docker-compose exec api touch maintenance_mode.txt

# download the new image
docker-compose pull

# shutdown the running container(s)
docker-compose down

# re-create it with the new image version (you'll notice 2 monitoring
# containers get stopped/started too)
docker-compose up -d

# make sure everything is working properly
docker-compose ps
docker-compose logs -f

# don't forget to save your changes
git add . && git commit

Now you can clean obsolete docker images using spotify/docker-gc

Do not use this on postgresql, redis or discourse servers, only for kleenex servers

docker -H $DOCKER_HOST run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /etc:/etc:ro spotify/docker-gc

If everything goes fine, do the same for other services.

Here is an example of what you have to do to update all containers:

source /opt/c2corg-docker/server-list.env

cd /opt/c2corg-docker/composition/production
for instance in api0 api1 api2 ui0 ui1 images0 images1
do
  echo "Updating $instance"
  cd $instance
  docker-compose exec $instance touch maintenance_mode.txt
  docker-compose pull
  docker-compose down
  docker-compose up -d
  docker -H $DOCKER_HOST run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /etc:/etc:ro spotify/docker-gc
  cd ..
done

Congrats, you've done a rolling upgrade of the service with zero downtime !

BTW, please don't clone this repository elsewhere. It's full of credentials, and it's backuped on the c2c.org backup server.

You can check haproxy's status here: http://www.camptocamp.org:8008/stats You should notice one backend going offline if you refresh the page between docker-compose "down" and "up".

Also, I setup 2 basic kibana dashboards, you'll probably want to keep track of the first few days:

https://c2corgv6.logs.camptocamp.net/kibana/#/dashboard/elasticsearch/Python%20stacktraces https://c2corgv6.logs.camptocamp.net/kibana/#/dashboard/elasticsearch/HTTP%2050x