Skip to content

User_Using your own compute workers

Jump to bottom
Isabelle Guyon edited this page Oct 27, 2017 · 65 revisions

In these instructions, we guide you step-by-step to create your own "compute worker" (a server to which submissions of challenge participants are sent to be executed).

Contents

Setting up a server as compute worker

To set up a server as compute worker, we provide instructions for creating Ubuntu Virtual machines (VM), but you may also use your own machine:

Then ssh into your machine and run the following commands:

  • Get docker (quick and dirty way; for a full clean installation see Docker Ubuntu instructions):

    $ curl https://get.docker.com | sudo sh

  • Add yourself to the docker group (so you don't nee to run as root):

    $ sudo usermod -aG docker ubuntu

  • Log out of your server and log back in. Then check your installation:

    $ docker run hello-world

  • Run this command to be able to run many containers at once:

    $ sudo apt-get install docker-compose

  • Clone the Codalab Competitions docker image:

    $ git clone https://github.com/codalab/codalab-competitions.git

  • Create and edit the configuration file .env:

    $ cd codalab-competitions

    $ cp .env_sample .env

    $ vim .env

Note: vim is an editor coming with Ubuntu. If you rather use emacs, type first sudo apt-get install emacs.

  • Uncomment in .env the line BROKER_URL= and paste your worker queue key:

    BROKER_URL=pyamqp://6ad9ac58-88e3-4a22-9be7-6ed5126ef388:40546f9d-0f2e-4413-a6b2-a1d86cad2b30@localhost/37324ab2-ee78-4e8d-a6ee-6089a159d253

To get your key, see next section. KNOWN BUG: Sometimes the server IP address or URL is replaced by localhost, like in the example above. If you get this, substitute localhost by the IP address or URL of your server.

  • Get your worker to start computing (it will start listening to the queue via BROKER_URL and pick up jobs):

    $ docker-compose up -d worker_compute

  • Make a submission to your competition and check logs with to confirm it is working:

    $ docker-compose logs -f

  • Should you need to change the queue that your compute worker listens to, edit again .env, then run:

    $ docker-compose down worker_compute

    $ docker-compose up -d worker_compute

  • Should you need to stop your compute worker and re-start it, type:

    $ docker-compose stop worker_compute

    $ docker-compose start worker_compute

Hooking up a compute worker to a queue

Go to "Worker Queue Management":

Select "Add Queue":

Give the queue a name. Select whether you want is public or private (public queues can be used by everybody). For private queues, select which users you want to share them with. You can edit these settings later.

Get the queue key you needed in the previous section. Put in the .env configuration file as per instructions.

Assigning a queue to a competition

Go to your competition page and click "Edit":

Select the queue that you have created:

Clone this wiki locally