Skip to content

Example_yellow_world

Jump to bottom
Joseph Rollinson edited this page Oct 26, 2017 · 22 revisions

Hello World competition

This is a very simple example of competition with result submission.

The competition bundle contains the following files:

reference_data.zip	    The solution to the problem
scoring_program.zip	    The program evaluating the solution
logo.jpg                    The logo
competition.yaml            The YAML configuration file
data.html	            HTML documentation pages	
evaluation.html		
overview.html	
terms_and_conditions.html

The YAML file is the configuration file. It consist in a number of attribute: value pairs. For a full list of attributes, see the Codalab competition YAML definition language.

  1. First section: general settings and HTML pages
title: Example Hello World Competition
description: An example competition where submissions should output "Hello World!"
image: logo.jpg
has_registration: True
html:
    overview: overview.html
    evaluation: evaluation.html
    terms: terms_and_conditions.html
    data: data.html
  1. Section section: phases (only 1 phase in this example)
phases:
    1:
        phasenumber: 1
        label: "First phase"
        start_date: 2013-06-30
        max_submissions: 100
        scoring_program: scoring_program.zip
        reference_data: reference_data.zip
  1. Third section: leaderboard configuration
leaderboard:
    leaderboards:
        RESULTS: &RESULTS
            label: Results
            rank: 1
    columns:
        correct:
            leaderboard: *RESULTS
            label: correct
            rank: 1
            numeric_format: 1

In this case, the scoring program writes a file scores.txt containing:

correct:1

for the sample submission.

Building an evaluation program that works with CodaLab

This example uses python.

evaluate.py - is an example that checks that the submission data matches the truth data, which is "Hello World!"

metadata - this is a file that lists the contents of the program.zip bundle for the CodaLab system.

Once these pieces are assembled they are packages as program.zip which CodaLab can then use to evaluate the submissions for a competition.

Clone this wiki locally