Outcomes of testing

In brief

There are several different outcomes in the hs-test-web library.

1. Wrong answer
2. Exception
3. Error
4. Unexpected error
5. Compilation error

Wrong answer

This outcome appears when the user's program has successfully completed, but the test case returns wrong(message) (or throws WrongAnswer). In this case, the user is also shown feedback.

If the user gets the wrong answer with the feedback Your program didn't output a number! in test 1, the output will look like the following:

Wrong answer in test #1  

Your program didn't output a number!

And in the IDE it will look like this:

If the user writes this program below (also incorrect):

Example of wrong answer feedback in test 2:

Wrong answer in test #2  

You should output a sum of numbers

So, you may notice that the library automatically calculates how many tests have been passed and which test has not been passed on the account and displays the information about the number of the failed test together with feedback.

Exception

This outcome appears when the student program is terminated with an exception. In this case, the student is shown the test number and the stack trace of the exception.

The user will get the exception outcome.

Error

This outcome occurs when a user has clearly done something wrong, but it has not led to an exception. This outcome is shown in the following cases:

1. The browser or its page has been closed during the tests execution:

Error in test #1
    
Error: Protocol error (Runtime.callFunctionOn): Session closed. Most likely the page has been closed.
    at __puppeteer_evaluation_script__:2:11
    ...

Unexpected error

This outcome appears when the student's program has completed successfully, but an exception occurred during checking. In this case, it is the fault of the person who wrote the tests and the tests should be corrected. The feedback It will look like the following:

Unexpected error during testing
    
We have recorded this bug and will fix it soon.
    
Testing library version 2.1.0
Node.js version 12.0.0
Puppeteer version 10.2.0
Jest version 24.9.0
    
Error: No tests found
    Stack trace:
    ...

With a bunch of useful information attached to the report.

Compilation error

This outcome appears when a React project can't be builded by webpack because of missing some needed files or syntax errors in the user code:

Compilation error
    
EntryModuleNotFoundError: Entry module not found: Error: Can't resolve 'C:\Users\...' in 'C:\Users\Admin\Desktop\hs-test-web'
    
ModuleNotFoundError: Module not found: Error: Can't resolve '/src/index.js' in 'C:\Users\lolip\Desktop\hs-test-web'