Problem_Format_Verification.mw

'''NB! This page has not been kept up to date. See [https://github.com/Kattis/problemtools/ reference implementation] for actual behavour'''

== Determination of Time Limit ==

The execution time limit for the problem is determined as follows. First, all the provided <tt>accepted/</tt> solutions are run. Let ''t<sub>max</sub>'' be the maximum running time of these solutions on any of the input files.  The time limit is then set to be ''t<sub>lim</sub>'' = &lceil;''t<sub>max</sub> &middot; M''&rceil; where ''M'' is the value of the time multiplier parameter from the [[Problem_Format#limits|limits configuration of problem.yaml]].

Furthermore, it is required that all of the provided time limit exceeded submissions run for at least ''t<sub>lim</sub> &middot; S'' seconds, where S is the value of the time safety margin parameter from the [[Problem_Format#limits|limits configuration]].

== Verification ==

Solutions or validators in languages that are not supported by the CCS
should be ignored and a warning to that effect shown.

Verification Checks (in order)
# Check files (all files present as required + check problem.yaml)
# Check compile (check that all programs compile)
# Check input (run input validators)
# Check solutions (run all solutions check that they get the expected verdicts)

 Warn if:
       there are no *.in in data/sample/

 Error if:
       there is no problem.yaml
       there is no problem statement (i.e., a problem*.tex file)
       any value in problem.yaml is invalid
       there are no *.in in data/secret/
       there are .in files without corresponding .ans files in data/*/
       there are .ans files without corresponding .in files in data/*/
       there are no solutions in submissions/accepted/
       there are no validators in input_format_validators/
       validator begins with "custom" and there are no validators in output_validators/
       there are validators in output_validators/ and validator does not begin with "custom"
       any validator (input format or output) does not compile

 For each *.in in in data/*/:
       For each validator in input_format_validators/:
               If the validator does not accept the input file: Error!

 For each solution in test_submissions/accepted/:
       For each *.in in data/*/:
               Run the solution on the input
               For the built-in validator if corrector is "diff" or each validator in output_validators/:
                       If the validator does not accept the output of the solution: Error!
 Let t be the longest time any of the solutions ran on any of the inputs.

 For each solution in submissions/time_limit_exceeded/:
       For each *.in in data/*/:
               Run the solution on the input for at least t * time_limit_safety_margin seconds.
 Let t_slow be the shortest time any of the solutions ran on any of the inputs.

 If t_slow is less than t * time_limit_safety_margin: Error!

 For each solution in submissions/wrong_answer/:
       For each *.in in data/*/:
               Run the solution on the input
               For the built-in validator if corrector = "diff" or each validator in output_validators/:
                       If the validator accepts the output of the solution: Error!

 For each solution in submissions/run_time_error/:
       For each *.in in data/*/:
               Run the solution on the input
               If the solution is not judged Run-Time Error: Error!