Upgrade the prod server

Most common use case how-to

If you simply need to update API version, without any data model update. First of all, open http://www.camptocamp.org:8008/stats to follow that everything goes well. Then :

# Connect to the composition management server
ssh <user_id>@compose0.exoscale.infra.camptocamp.org

# IMPORTANT : set-up env variables
source /opt/c2corg-docker/server-list.env

# Got to first API instance
cd /opt/c2corg-docker/composition/production/api0/

# check which image flavour is used, and update it :
grep image: docker-compose.yml
vi docker-compose.yml

# Set container in mainternance mode (will not get new requests for Haproxy)
docker-compose exec api touch maintenance_mode.txt

# download the new image
docker-compose pull

# shutdown the running container, and restart it
docker-compose stop && docker-compose rm -f
docker-compose up -d

# make sure everything is working properly
docker-compose ps

# Perfect ! Let's do the same for api1

cd ../api1
docker-compose exec api touch maintenance_mode.txt
docker-compose pull
docker-compose stop && docker-compose rm -f
docker-compose up -d
docker-compose ps

# and for api2

cd ../api2
docker-compose exec api touch maintenance_mode.txt
docker-compose pull
docker-compose stop && docker-compose rm -f
docker-compose up -d
docker-compose ps

A last eye on http://www.camptocamp.org:8008/stats, and time to have a beer 🍺

Full how-to

# Connect to the composition management server
ssh <user_id>@compose0.exoscale.infra.camptocamp.org

# a bunch of envvars with the list of all the servers
source /opt/c2corg-docker/server-list.env

# one directory for each production server, self-descriptive names:
ls /opt/c2corg-docker/composition/production/

# choose one
cd /opt/c2corg-docker/composition/production/api0/

Notice the hidden ./.env in this directory. https://docs.docker.com/compose/environment-variables/#/the-env-file This envvar file teaches docker-compose to connect to a remote docker instance. Each per-server directory has it's own .env file.

You can now edit the docker-compose.yml file. In this example, you'll want to change the image version to a more recent one which travis made available on dockerhub (https://hub.docker.com/r/c2corg/v6_api/tags/).

# check which image flavour is used, update file accordingly:
grep image: docker-compose.yml

# Set container in mainternance mode (will not get new requests for Haproxy)
docker-compose exec api touch maintenance_mode.txt

# download the new image
docker-compose pull

# shutdown the running container(s)
# docker-compose down
docker-compose stop && docker-compose rm -f

If the database model have been modified, you also have to run the alembic migration on api0. You may want to backup th database first: https://github.com/c2corg/v6_api/wiki/Backup-the-prod-database

docker-compose run --rm api .build/venv/bin/alembic upgrade head

Now you can restart the container service container:

# re-create it with the new image version (you'll notice 2 monitoring
# containers get stopped/started too)
docker-compose up -d

# make sure everything is working properly
docker-compose ps
docker-compose logs -f

# don't forget to save your changes
git add . && git commit

Now you can clean obsolete docker images using spotify/docker-gc

Do not use this on postgresql, redis or discourse servers, only for kleenex servers

docker -H $DOCKER_HOST run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /etc:/etc:ro spotify/docker-gc

If everything goes fine, do the same for other services.

Here is an example of what you have to do in case you want to update all containers to new version, without database migration:

source /opt/c2corg-docker/server-list.env

cd /opt/c2corg-docker/composition/production
for instance in api0 api1 api2 ui0 ui1 images0 images1
do
  echo "Updating $instance"
  cd $instance
  docker-compose exec $instance touch maintenance_mode.txt
  docker-compose pull
  docker-compose down
  docker-compose up -d
  docker -H $DOCKER_HOST run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /etc:/etc:ro spotify/docker-gc
  cd ..
done

Congrats, you've done a rolling upgrade of the service with zero downtime !

BTW, please don't clone this repository elsewhere. It's full of credentials, and it's backuped on the c2c.org backup server.

You can check haproxy's status here: http://www.camptocamp.org:8008/stats You should notice one backend going offline if you refresh the page between docker-compose "down" and "up".

Also, I setup 2 basic kibana dashboards, you'll probably want to keep track of the first few days:

https://c2corgv6.logs.camptocamp.net/kibana/#/dashboard/elasticsearch/Python%20stacktraces https://c2corgv6.logs.camptocamp.net/kibana/#/dashboard/elasticsearch/HTTP%2050x