Home

User manual

CodaLab Competitions is a powerful open source framework for running competitions that involve result or code submission. You can either participate in an existing competition or host a new competition. Most competitions hosted on Codalab are machine learning (data science) competitions, but Codalab is NOT limited to this application domain. It can accommodate any problem for which a solution can be provided in the form of a zip archive containing a number of files to be evaluated quantitatively by a scoring program (provided by the organizers). The scoring program must return a numeric score, which is displayed on a leaderboard where the performances of participants are compared.

To see Codalab Competition's in action, visit codalab.lisn.fr.

Table of contents

1. Participants
2. Organizers
3. Developers and administrators
4. How you can contribute
5. How to contact us

1. Participants

• User Settings
• List of Current Competitions
• Participating in a Competition
• Joining a Team
• Frequently Asked Questions (FAQ)

2. Organizers

2.1. Organizers using the public Codalab instance

The public instance of Codalab may be used to organize research or education competitions within the limits of our hosting capabilities. For code submission competitions, please contact us: you may have to supplement Codalab with additional computer workers.

• How to create your first competition on Codalab (Quick hands-on guide)
• What does it take to organize a competition? (ChaLearn tips)
• Codalab competition roadmap. For advanced users desiring more flexibility, Codalab competition bundles allow arbitrary number of phases and metrics, use of customized ingestion and scoring programs, and more...
  • Sample Competitions
  • Building a Competition Bundle
  • Building an Ingestion Program
  • Building a Scoring Program
  • All competition features in the YAML configuration file and the editor.
  • Running a Competition.
  • Configuring the leaderboard
• Enabling...
  1. Large datasets and hot changes of data and code during competitions.
  2. Forum
  3. Teams
  4. Leaderboard features and public submissions.
  5. Automatic code forward: The last submission of one phase becomes the first one of the next phase.
  6. Dockers: Run your competition in a specific docker.
  7. Audit trail
  8. Coopetitions
  9. Chagrade Features
• Contacting your participants
• Accessing Codalab from your own website with Codalab REST API

2.2. Organizers providing custom "compute workers"

If your competition needs extra computational power, you can easily add to the backend of the public Codalab instance your own server(s) to process participant submissions.

• Understanding the Codalab architecture
• Using your own compute workers
• Protocol for testing queues
• Administering compute workers using Fabric

2.3. Organizers who want to run their own instance

This section explains how to run your own Codalab instance (front-end and back-end). This allows you to customize Codalab and have full control and full privacy. We prepared for you a re-usable AWS virtual machine image (called AMI), already pre-configured, thus easier to install. You can also install Codalab from scratch on your own server. Before you venture into doing this, check "Customizing your website" in section 2.1 and the previous section, which may suffice to customize Codalab for your needs.

a) RUNNING CODALAB INSTANCE

• Use a pre-setup image:

  • AWS Image AMI with Docker/Codalab, Ubuntu
  • Google Cloud Image with Docker/Codalab, Ubuntu

• (OR, if you do not want to use our AMI) Configure CodaLab from scratch (harder) -- works on Azure, Google Cloud Compute, etc. anywhere you can spin up an Ubuntu/Linux box

b) Set up data storage

c) EXTRAS:

• Enable SSL
• E-mail notification
• Configure for single competition (disable hosting other competitions)
• Minio Storage (self-hosting storage)
• Putting Codalab into Maintenance mode
• Instance Admin: Setting up Scheduled VM Backups on Google Cloud
• Instance Admin: Restoring from VM Backups on Google Cloud
• Instance Admin: Creating GPU Workers on Google Cloud
• Instance Admin: Setting up Static IPs on Google Cloud

3. Developers and administrators

Developers should first go through section 2.3.

• Administrator procedures
• Developer Guidelines
• Instance debugging & testing
• Executing jobs
• Configure CodaLab to work with Visual Studio
• Configure settings
• Groups and Permissions
• Database migration
• Code checker
• Developer reference manual: work in progress, will be ready for v2.
• Production Server Settings (disable default worker, backup database)
• Disaster recovery instructions [Production server]

4) How you can contribute

• Use CodaLab Competitions by either participating in a competition or hosting a new competition.
• Find a bug? Got a feature request? Submit a GitHub issue. Follow the Process for Bugs and Issues.
• Submit pull requests on GitHub to implement new features or fix bugs. Follow the Developer Guidelines.
• Improve this documentation. Follow these instructions to add images.
• Let others know about CodaLab!

5) How to contact us

• The preferred way is via posting a GitHub issue. See instructions.
• In case of emergency, contact us directly.
• If you wish to get in touch with the community, you can use the Google Groups