Skip to content

Commit

Permalink
docker: update to compose V2
Browse files Browse the repository at this point in the history
  • Loading branch information
PascalEgn committed Aug 5, 2024
1 parent 882cf07 commit 189ced5
Show file tree
Hide file tree
Showing 8 changed files with 50 additions and 50 deletions.
32 changes: 16 additions & 16 deletions .github/workflows/backend-tests-on-docker.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ jobs:
key: docker-images-cache-${{ hashFiles('.github/workflows/backend-tests-on-docker', 'services.yml', 'docker-compose.test.yml', 'Dockerfile.dev', 'poetry.lock', 'setup.py', 'setup.cfg') }}
- name: Build images
if: steps.cache.outputs.cache-hit != 'true'
run: docker-compose -f services.yml build --force-rm base
run: docker compose -f services.yml build --force-rm base
- name: Install lrzip
if: steps.cache.outputs.cache-hit != 'true'
run: |
Expand Down Expand Up @@ -59,13 +59,13 @@ jobs:
uses: ./.github/actions/load-image-from-cache
- name: Prepare docker
run: |
docker-compose -f docker-compose.test.yml pull --parallel test-database test-indexer test-rabbitmq test-redis
docker-compose -f docker-compose.test.yml build --parallel workflows test-web
docker-compose -f docker-compose.test.yml up -d test-database test-indexer test-rabbitmq test-redis test-web
docker compose -f docker-compose.test.yml pull --parallel test-database test-indexer test-rabbitmq test-redis
docker compose -f docker-compose.test.yml build --parallel workflows test-web
docker compose -f docker-compose.test.yml up -d test-database test-indexer test-rabbitmq test-redis test-web
sh docker/wait-for-services.sh
- name: Run tests
run: |
docker-compose -f docker-compose.test.yml run --rm workflows py.test tests/integration/workflows \
docker compose -f docker-compose.test.yml run --rm workflows py.test tests/integration/workflows \
--ignore tests/integration/workflows/test_audit.py \
--ignore tests/integration/workflows/test_hep_approval.py \
--ignore tests/integration/workflows/test_submissions_views.py \
Expand Down Expand Up @@ -95,13 +95,13 @@ jobs:
uses: ./.github/actions/load-image-from-cache
- name: Prepare docker
run: |
docker-compose -f docker-compose.test.yml pull --parallel test-database test-indexer test-rabbitmq test-redis
docker-compose -f docker-compose.test.yml build --parallel workflows test-web
docker-compose -f docker-compose.test.yml up -d test-database test-indexer test-rabbitmq test-redis test-web
docker compose -f docker-compose.test.yml pull --parallel test-database test-indexer test-rabbitmq test-redis
docker compose -f docker-compose.test.yml build --parallel workflows test-web
docker compose -f docker-compose.test.yml up -d test-database test-indexer test-rabbitmq test-redis test-web
sh docker/wait-for-services.sh
- name: Run tests
run: |
docker-compose -f docker-compose.test.yml run --rm workflows py.test \
docker compose -f docker-compose.test.yml run --rm workflows py.test \
tests/integration/workflows/test_views.py \
tests/integration/workflows \
--ignore tests/integration/workflows/test_arxiv_merge.py \
Expand Down Expand Up @@ -133,11 +133,11 @@ jobs:
uses: ./.github/actions/load-image-from-cache
- name: Prepare docker
run: |
docker-compose -f docker-compose.test.yml pull --parallel test-database test-indexer test-redis
docker-compose -f docker-compose.test.yml up -d test-database test-indexer test-redis
docker compose -f docker-compose.test.yml pull --parallel test-database test-indexer test-redis
docker compose -f docker-compose.test.yml up -d test-database test-indexer test-redis
sh docker/wait-for-services.sh
- name: Run tests
run: docker-compose -f docker-compose.test.yml run --rm integration
run: docker compose -f docker-compose.test.yml run --rm integration

Integration-async:
runs-on: ubuntu-latest
Expand All @@ -158,11 +158,11 @@ jobs:
uses: ./.github/actions/load-image-from-cache
- name: Prepare docker
run: |
docker-compose -f docker-compose.test.yml pull --parallel test-database test-indexer test-redis test-rabbitmq
docker-compose -f docker-compose.test.yml up -d test-database test-indexer test-redis test-rabbitmq
docker compose -f docker-compose.test.yml pull --parallel test-database test-indexer test-redis test-rabbitmq
docker compose -f docker-compose.test.yml up -d test-database test-indexer test-redis test-rabbitmq
sh docker/wait-for-services.sh
- name: Run tests
run: docker-compose -f docker-compose.test.yml run --rm integration_async
run: docker compose -f docker-compose.test.yml run --rm integration_async

Unit:
needs: Build
Expand All @@ -183,7 +183,7 @@ jobs:
uses: ./.github/actions/load-image-from-cache
- name: Run tests
run: |
docker-compose -f docker-compose.test.yml run --rm unit
docker compose -f docker-compose.test.yml run --rm unit
Release:
if: ${{ github.event_name == 'push' || github.event_name == 'release'}}
Expand Down
2 changes: 1 addition & 1 deletion INSTALL.rst
Original file line number Diff line number Diff line change
Expand Up @@ -215,7 +215,7 @@ Then create the database and database tables if you haven't already done so:

.. code-block:: console
(inspirehep)$ docker-compose up database indexer redis rabbitmq
(inspirehep)$ docker compose up database indexer redis rabbitmq
Everything else: Honcho
Expand Down
4 changes: 2 additions & 2 deletions docker-compose.test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -326,8 +326,8 @@ services:
- APP_LEGACY_PID_PROVIDER="http://inspirehep.net//batchuploader/allocaterecord"
- APP_CRAWLER_HOST_URL=http://test-scrapyd:6800

# This is needed as docker-compose does not wait for the entry services to
# be actually healthy when doing `docker-compose run e2e`, but only for
# This is needed as docker compose does not wait for the entry services to
# be actually healthy when doing `docker compose run e2e`, but only for
# derived dependent services.
e2e-deps-placeholder:
extends:
Expand Down
4 changes: 2 additions & 2 deletions docker/wait-for-services.sh
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,11 @@ check_ready() {
fi
done
}
_db_check(){ docker-compose -f docker-compose.test.yml exec --user postgres test-database bash -c "pg_isready" &>/dev/null; }
_db_check(){ docker compose -f docker-compose.test.yml exec --user postgres test-database bash -c "pg_isready" &>/dev/null; }
check_ready "postgres" _db_check

_es_check(){ curl -sL -w "%{http_code}\\n" "http://localhost:9200/" -o /dev/null | grep '200' &> /dev/null; }
check_ready "Elasticsearch" _es_check

_redis_check(){ docker-compose -f docker-compose.test.yml exec test-redis bash -c 'redis-cli ping' | grep 'PONG' &> /dev/null; }
_redis_check(){ docker compose -f docker-compose.test.yml exec test-redis bash -c 'redis-cli ping' | grep 'PONG' &> /dev/null; }
check_ready "redis" _redis_check
38 changes: 19 additions & 19 deletions docs/docker.rst
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ Get the latest Docker appropriate to your operationg system, by visiting `Docker
*Get Docker* section.

.. note:: If you are using Mac, please build a simple box with ``docker-engine`` above ``1.10`` and
``docker-compose`` above ``1.6.0``.
``docker compose V2``.

Make sure you can run docker without ``sudo``.

Expand All @@ -48,7 +48,7 @@ Make sure you can run docker without ``sudo``.

- ``sudo usermod -a -G docker $USER``

Get the latest `docker-compose
Get the latest `docker compose
<https://docs.docker.com/compose/>`_:

.. code-block:: console
Expand All @@ -68,7 +68,7 @@ By default the virtualenv and everything else will be kept on ``/tmp`` and they

.. Note::

From now on all the docker-compose commands must be run at the root of the
From now on all the docker compose commands must be run at the root of the
inspire-next repository, you can get a local copy with:

.. code-block:: console
Expand All @@ -78,8 +78,8 @@ By default the virtualenv and everything else will be kept on ``/tmp`` and they
.. code-block:: console
$ docker-compose pull
$ docker-compose -f docker-compose.deps.yml run --rm pip
$ docker compose pull
$ docker compose -f docker-compose.deps.yml run --rm pip
.. note:: If you have trouble with internet connection inside docker probably you are facing known
DNS issue. Please follow `this solution
Expand All @@ -88,19 +88,19 @@ By default the virtualenv and everything else will be kept on ``/tmp`` and they

.. code-block:: console
$ docker-compose -f docker-compose.deps.yml run --rm assets
$ docker compose -f docker-compose.deps.yml run --rm assets
- Run the service locally

.. code-block:: console
$ docker-compose up
$ docker compose up
- Populate database

.. code-block:: console
$ docker-compose run --rm web scripts/recreate_records
$ docker compose run --rm web scripts/recreate_records
Once you have the database populated with the tables and demo records, you can
Expand All @@ -112,10 +112,10 @@ go to `localhost:5000 <http://localhost:5000>`_

.. Note::

The tests use a different set of containers than the default ``docker-compose
The tests use a different set of containers than the default ``docker compose
up``, so if you run both at the same time you might start having ram/load
issues, if so, you can stop all the containers started by ``docker-compose
up`` with ``docker-compose kill -f``
issues, if so, you can stop all the containers started by ``docker compose
up`` with ``docker compose kill -f``

You can choose one of the following tests types:

Expand All @@ -126,8 +126,8 @@ You can choose one of the following tests types:

.. code-block:: console
$ docker-compose -f docker-compose.test.yml run --rm <tests type>
$ docker-compose -f docker-compose.test.yml down
$ docker compose -f docker-compose.test.yml run --rm <tests type>
$ docker compose -f docker-compose.test.yml down
.. tip:: - cleanup all the containers:

Expand All @@ -148,38 +148,38 @@ Extra useful tips

.. code-block:: console
$ docker-compose run --rm web inspirehep shell
$ docker compose run --rm web inspirehep shell
- Run *virtualenv* bash shell for running scripts manually (e.g. recreating records or `building documentation`_)

.. _building documentation: http://inspirehep.readthedocs.io/en/latest/building_the_docs.html

.. code-block:: console
$ docker-compose run --rm web bash
$ docker compose run --rm web bash
- Reload code in a worker

.. code-block:: console
$ docker-compose restart worker
$ docker compose restart worker
- Quick and safe reindex

.. code-block:: console
$ docker-compose restart worker && docker-compose run --rm web scripts/recreate_records
$ docker compose restart worker && docker compose run --rm web scripts/recreate_records
- Recreate all static assets. Will download all dependencies from npm and copy all static
files to ``${DOCKER_DATA}/tmp/virtualenv/var/inspirehep-instance/static``.

.. code-block:: console
$ docker-compose -f docker-compose.deps.yml run --rm assets
$ docker compose -f docker-compose.deps.yml run --rm assets
- Monitor the output from all the services (elasticsearch, web, celery workers, database, flower, rabbitmq, scrapyd, redis)
via the following command:

.. code-block:: console
$ docker-compose up
$ docker compose up
10 changes: 5 additions & 5 deletions docs/e2e_tutorial.rst
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ Let us now run this "test" and see what happens:

.. code-block:: bash
docker-compose -f docker-compose.test.yml run --rm e2e pytest tests/e2e/test_arxiv_in_hp.py
docker compose -f docker-compose.test.yml run --rm e2e pytest tests/e2e/test_arxiv_in_hp.py
Proxy Web UI
++++++++++++
Expand Down Expand Up @@ -254,7 +254,7 @@ In order to start the web container (don't forget the ``.local`` at the end!):

.. code-block:: bash
docker-compose -f docker-compose.test.yml up test-web-e2e.local
docker compose -f docker-compose.test.yml up test-web-e2e.local
For any other container, change the ``test-web-e2e.local`` to the suitable name; other containers
don't end in ``.local``, this is needed only for inspire-next node as it has to be a domain name.
Expand All @@ -266,15 +266,15 @@ To view the logs of a container:

.. code-block:: bash
docker-compose -f docker-compose.test.yml logs test-worker-e2e
docker compose -f docker-compose.test.yml logs test-worker-e2e
In order to run a shell in an already running container (e.g. to investigate errors):

.. code-block:: bash
# E.g. for INSPIRE
docker-compose -f docker-compose.test.yml exec test-web-e2e.local bash
docker compose -f docker-compose.test.yml exec test-web-e2e.local bash
# For MITM-Proxy we use `ash`, as it runs on Alpine Linux base, which doesn't ship with `bash`
docker-compose -f docker-compose.test.yml exec mitm-proxy ash
docker compose -f docker-compose.test.yml exec mitm-proxy ash
6 changes: 3 additions & 3 deletions docs/harvesting.rst
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ using docker:

.. code-block:: console
docker-compose -f docker-compose.deps.yml run --rm scrapyd-deploy
docker compose -f docker-compose.deps.yml run --rm scrapyd-deploy
The simplest way to get records into your system is to harvest from arXiv.org using OAI-PMH.

Expand All @@ -86,7 +86,7 @@ See `the diagram in hepcrawl documentation <https://pythonhosted.org/hepcrawl/in
to see what happens behind the scenes.

Single records like this (if you are running docker, you first will need to open bash and get into
the virtual environment in one of the workers, e.g. ``docker-compose run --rm web bash``, read the
the virtual environment in one of the workers, e.g. ``docker compose run --rm web bash``, read the
:ref:`other_sources` section if you aren't using docker):

.. code-block:: bash
Expand Down Expand Up @@ -162,7 +162,7 @@ If you have not previously deployed your spiders, you will have to do it like so

.. code-block:: console
docker-compose -f docker-compose.deps.yml run --rm scrapyd-deploy
docker compose -f docker-compose.deps.yml run --rm scrapyd-deploy
Afterwards you can schedule a harvest from the CLI or shell:

Expand Down
4 changes: 2 additions & 2 deletions docs/postgresql.rst
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ The first step is run the web container, in order to start our database.

.. code-block:: bash
$ docker-compose run --rm web
$ docker compose run --rm web
3. Connect to the PostgresSQL Database
Expand All @@ -48,7 +48,7 @@ When all the containers are up you have to open a new console and run the follow

.. code-block:: bash
$ docker-compose exec database psql -U inspirehep
$ docker compose exec database psql -U inspirehep
.. code-block:: sql
Expand Down

0 comments on commit 189ced5

Please sign in to comment.