Docker submission testing

Docker Submission Testing Guide

Welcome to the Docker Submission Testing guide. This utility allows developers to run tests within Docker containers to ensure code meets certain criteria before allowing pushes to the server. This way the teacher is certain that projects have a basic functionality before getting reviewed. The guide details two methods to utilize this utility effectively.

Prerequisites

To begin using the Docker submission testing utility, you must have the Docker image installed that is necessary for the test. This can be done using the following command:

SubmissionTestModel.addDocker("fedora:latest");

Using the Docker Submission Testing Utility

Below are two structured methods to use the Docker submission testing utility, each catering to different testing needs and setups.

Method 1: Console Log Output and Push Permission

This method focuses on direct console log outputs and conditional push permissions based on the results obtained from the Docker container.

Variables and Setup

• Image: fedora (Docker container image)
• Input: /shared/input/helloworld.sh containing the script submissioned by the student:

  echo 'Hello World'

• Script:

  String script = "output=$(bash /shared/input/helloworld.sh); if [[ \"$output\" == 'Hello World' ]]; then echo 'Test one is successful'; 
  echo 'PUSH ALLOWED' > /shared/output/testOutput; else echo 'PUSH DENIED'; > /shared/output/testOutput; fi"

Executing the container

1. Create and Start a Container

   • Initialize the SubmissionTestModel with the fedora Docker image:

     SubmissionTestModel stm = new SubmissionTestModel("fedora");

2. Prepare the Script

   • Define a script to execute within the Docker container that checks if helloworld.sh outputs "Hello World".

     String[] script = {"bash", "-c", script};

3. Run the Submission and Handle Output

   • Execute the script in the container and capture the results:

     SubmissionTestModel.TestOutput to = stm.runSubmission(script);

   • Key fields in TestOutput:
     • TestOutput.allowed: Boolean indicating if a push is allowed (true if "PUSH ALLOWED" is found in /output/testOutput).
     • TestOutput.logs: Console logs from the Docker run, useful for diagnostics and insights into the push process.

Method 2: Using Templates for Test Outputs

This method uses a templated approach to manage and evaluate outputs, making it ideal for structured and scalable testing frameworks.

Variables and Setup

• Image: fedora (Docker container image)

• Input: /shared/input/helloworld.py containing a python script submissioned by the student:

  print('Hello world')

• Template Definition: The test output expectations in a template

  @ConsoleLogHelloWorld
  >description="Test if the python application returns 'Hello World'"
  >required
  Hello world"
  

• Script:

  String script = "python3 /shared/input/helloworld.py > /shared/doutput/ConsoleLogHelloWorld

Executing the container

1. Create and Start a Container

   • Initialize the SubmissionTestModel with the fedora Docker image:

     SubmissionTestModel stm = new SubmissionTestModel("fedora");

2. Prepare the Script

   • Define a script to execute within the Docker container that checks if helloworld.sh outputs "Hello World".

    String[] script = {"bash", "-c", script};

3. Run the Submission and Handle Output

   • Execute the script in the container and capture the results:

     SubmissionTestModel.TemplateResults tr = stm.runSubmissionWithTemplate(script, template);

   • Key fields in TemplateResults:
     • TemplateResults.results: A list with information on all the tests, in TemplateResult variables:
     • TemplateResults key fields:
       • Required: A Boolean stating if the test is required for it to pass
       • Correct: A string with the correct answer.
       • Result: A string with the answer by executing the project.

Template Options

• required: Indicates the test must pass for the push to be allowed.

• optional: Allows the test to fail while still permitting the push.

• description: Provides a clear description of what the test verifies.