Skip to content
This repository has been archived by the owner on Jul 12, 2024. It is now read-only.

Home

Jump to bottom
Ziteng Jiao edited this page Jun 14, 2021 · 9 revisions

Welcome to the 2020-LPCVC-Referee wiki!

Referee Workflow

Server File Structure

Home Directory
│
├─── referee [MUST BE CALLED referee]
│    ├── source_code [This Repository]
│    ├── queue [Location of Queue] 
│    └── referee.environment [Location to store enivronment variables]
│
├─── website_directory 
│    ├─── results 
│    └─── submissions 
│
├── test_data [Location of Data Source -> Ground Truths etc..]

Content that goes inside referee.environment file

WORKING_DIRECTORY="Working Directory aka Location of referee directory (as shown above)"
LPCVC_SUBMISSION_DIR="Location of Submissions"
LPCVC_TEST_VIDEOS="video1.mp4 video2.mp4 etc.."
LPCVC_TEST_DATA_DIR="Location of your Data Bank (sources, groundtruths, correct)"
LPCVC_PI_SCRIPTS_DIR="{Source Code Location}/scripts/pi_scripts"
LPCVC_PI_USER="username@[local ipv4 address]"
LPCVC_PI_PASSWORD="Password to Raspberry Pi"
LPCVC_PI_TEST_DIR="Directory to Run Contestant Solution"
LPCVC_SITE="Location of 'website_directory'. Required for 'results' directory"
METER_CMD="Python Command with {} as parameters to run script from Meter to Raspberry Pi"
METER_CSV="Output from the meter."

All environment variables are used inside lpcvc/LPCVC2021/lpcvc_queue directory. I recommend looking through there, in addition, posting issues if you notice a major flaw.

Required Hardware and Software

System Setup

Setting up the Yokogawa WT310 Power Meter

The power meter must be connected so that it can record the power consumption of the Pi.

Setting up the Linux Server

Create a virtual environment for this repository and install the requirements from requirements.txt. Create a folder to store the submission files and update the SUBMISSION_DIR variable of the lpcvc.py file in this repository. Create an SSH key for the Linux Server using the ssh-keygen utility.

Setting up the Windows Computer

After installing Microsoft Visual Studio and Python 3, open the HELPS WT310 Controlling Program solution and build the solution. Update the locations of the Python binary and the Debug folder in the METER_CMD and METER_CSV variables of the lpcvc.py file in this repository. Create another SSH key for the computer using the ssh-keygen.exe utility. Add the Linux Server key to the authorized_keys file in the user/.ssh folder using the ssh-add.exe utility.

Setting up the Raspberry Pi

Run the following commands on the Raspberry Pi.

sudo dnf update
sudo dnf install git cmake ninja-build clang unzip distcc python3-virtualenv python3-devel python3-opencv python-pillow gstreamer1-plugins-base gstreamer1-plugins-good
sudo reboot

Create a virtual environment for the solutions with the --site-packages option. Create an empty folder for the testing of the submissions. Update the PI_TEST_DIR variable of the lpcvc.py file in this repository. Create yet another SSH key for the Linux Server using the ssh-keygen utility. Add all three keys to the authorized_keys file in the ~/.ssh folder using the ssh-add utility. DO NOT add the Pi's key to the other two machines. Doing so is a security risk.

GitHub Actions [Optional]

The team uses GitHub Actions for automatic deployment to the server. This is done by creating a ssh channel through GitHub, which as a result, run a command on the computer to automatically install. The username, password, and port is encrypted, and no PR or Fork can trigger a run on the computer.

Setting up GitHub Action

GitHub Action Location

This page is located under Settings, then Secrets.

You click on New repository secret, and enter the HOST (url to computer), username, password, and port (usually 22).

~/scripts/referee.sh

This script can be anything. For this purpose, it will run scripts inside the scripts directory in this repository.

Using the Referee (2020)

Starting the Referee Queue

systemctl start lpcvc.service --user

Seeing the Logs Messages

systemctl status lpcvc.service --user

Manually Enqueuing Submissions

touch queue/test.pyz

Make sure inside the "LPCVC_SUBMISSION_DIR", that there is a file called test.pyz that meets the requirements of running the solution.

Stopping the Referee Queue

systemctl stop lpcvc.service --user

Levenstein Distance Script (LDCalc.py)

Usage

Requirements can be found in requirements_Levenstein.txt

Import methods from LDCalc

import LDCalc.py *

distanceCalc("realA.txt", "A.txt")

Distance Calc method will take the ground truth answer text file as the first parameter and the generated answers text file as the second parameter and return the final score of the solution as a float.

Method

The LDCalc.py script will take the answers given per question and evaluate the Levenstein distance between each word in the ground truth answers to each question and every word in the respective answers to each question from the generated answers. Then the Levenstein distance score for each word will be divided by word length to find the score to length ratio for each word. Lastly, the score to length ratios for every word in the answers will be averaged to give the final accuracy score of the generated answers. The final accuracy score will be an error float that will range from 0 to 1, 0 being a perfect set of generated answers, and 1 being a completely inaccurate set of generated answers.

Answer Format

Answers must include all questions given and must also follow the format below:

MULTIPURPOSE: 137 TRAINED PERSONNEL ONLY; 137: TRAINED PERSONNEL ONLY MULTIPURPOSE; 132: MEN; MEN: 132

Questions are specified by the following colon, and answers are separated by semi-colons. All answers and questions are on one line in the .txt file.

Sample Output

Example Ground Truth Answer:

10: Emergency Manager; 12: two

Example Generated Answer:

10: Emergency Emergency; 12: to

Example Output from LDCalc.py:

The error of the solution is: 0.444444

Clone this wiki locally