Upgrade the prod server

# Connect to the composition management server
ssh compose0.exoscale.infra.camptocamp.org

# a bunch of envvars with the list of all the servers
source /opt/c2corg-docker/server-list.env

# one directory for each production server, self-descriptive names:
ls /opt/c2corg-docker/composition/production/

# choose one
cd /opt/c2corg-docker/composition/production/api0/

Notice the hidden ./.env in this directory. https://docs.docker.com/compose/environment-variables/#/the-env-file This envvar file teaches docker-compose to connect to a remote docker instance. Each per-server directory has it's own .env file.

You can now edit the docker-compose.yml file. In this example, you'll want to change the image version to a more recent one which travis made available on dockerhub (https://hub.docker.com/r/c2corg/v6_api/tags/).

# check which image flavour is used, update file accordingly:
grep image: docker-compose.yml

# Set container in mainternance mode (will not get new requests for Haproxy)
docker-compose exec api touch maintenance_mode.txt

# download the new image
docker-compose pull

# shutdown the running container(s)
docker-compose down

If the database model have been modified, you also have to run the alembic migration on api0 :

docker-compose run --rm api .build/venv/bin/alembic upgrade head

Now you can restart the container service container:

# re-create it with the new image version (you'll notice 2 monitoring
# containers get stopped/started too)
docker-compose up -d

# make sure everything is working properly
docker-compose ps
docker-compose logs -f

# don't forget to save your changes
git add . && git commit

Now you can clean obsolete docker images using spotify/docker-gc

Do not use this on postgresql, redis or discourse servers, only for kleenex servers

docker -H $DOCKER_HOST run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /etc:/etc:ro spotify/docker-gc

If everything goes fine, do the same for other services.

Here is an example of what you have to do in case you want to update all containers to new version, without database migration:

source /opt/c2corg-docker/server-list.env

cd /opt/c2corg-docker/composition/production
for instance in api0 api1 api2 ui0 ui1 images0 images1
do
  echo "Updating $instance"
  cd $instance
  docker-compose exec $instance touch maintenance_mode.txt
  docker-compose pull
  docker-compose down
  docker-compose up -d
  docker -H $DOCKER_HOST run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /etc:/etc:ro spotify/docker-gc
  cd ..
done

Congrats, you've done a rolling upgrade of the service with zero downtime !

BTW, please don't clone this repository elsewhere. It's full of credentials, and it's backuped on the c2c.org backup server.

You can check haproxy's status here: http://www.camptocamp.org:8008/stats You should notice one backend going offline if you refresh the page between docker-compose "down" and "up".

Also, I setup 2 basic kibana dashboards, you'll probably want to keep track of the first few days:

https://c2corgv6.logs.camptocamp.net/kibana/#/dashboard/elasticsearch/Python%20stacktraces https://c2corgv6.logs.camptocamp.net/kibana/#/dashboard/elasticsearch/HTTP%2050x