Skip to content

Home

Jump to bottom
Isabelle Guyon edited this page Aug 3, 2021 · 293 revisions

User manual

CodaLab Competitions is a powerful open source framework for running competitions that involve result or code submission. You can either participate in an existing competition or host a new competition. Most competitions hosted on Codalab are machine learning (data science) competitions, but Codalab is NOT limited to this application domain. It can accommodate any problem for which a solution can be provided in the form of a zip archive containing a number of files to be evaluated quantitatively by a scoring program (provided by the organizers). The scoring program must return a numeric score, which is displayed on a leaderboard where the performances of participants are compared.

Release notes v1.5:

  • Participants: can join competition-specific teams.
  • Organizers: can enable/monitor competition-specific teams; can run competitions in their own docker; can launch their own compute workers (to process participants' submissions) without needing to run their own Codalab instance; can provide an ingestion program to receive/parse participants' submissions; can dump their competition bundle (to save changes made with the editor); can run their own instance in "single competition mode".
  • Developers: can run Codalab in dockers.
  • Default docker: if not special docker is specified, submissions are run in codalab/codalab-legacy the default docker, which can be found on Github.

Robot policy: Codalab does not forbid the use of robots (bots) to access the website, provided that it is not done with malicious intentions to disturb the normal use and jam the system. A user who abuses their rights by knowingly or recklessly using jamming submissions will have his account revoked.

1) Participants

2) Organizers

2.1) Organizers using the public Codalab instance

The public instance of Codalab may be used to organize research or education competitions within the limits of our hosting capabilities. For code submission competitions, please contact us: you may have to supplement Codalab with additional computer workers.

2.2) Organizers providing custom "compute workers"

If your competition needs extra computational power, you can easily add to the backend of the public Codalab instance your own server(s) to process participant submissions.

2.3) Organizers who want to run their own instance

This section explains how to run your own Codalab instance (front-end and back-end). This allows you to customize Codalab and have full control and full privacy. We prepared for you a re-usable AWS virtual machine image (called AMI), already pre-configured, thus easier to install. You can also install Codalab from scratch on your own server. Before you venture into doing this, check "Customizing your website" in section 2.1 and the previous section, which may suffice to customize Codalab for your needs.

a) RUNNING CODALAB INSTANCE

b) Set up data storage

c) EXTRAS:

3) Developers and administrators

Developers should first go through section 2.3.

4) How you can contribute

5) How to contact us

6) Self-debugging

7) Codalab v2

Clone this wiki locally