Guide for adding new tasks

[malteos]

1. Check whether your task is already present in the framework.
2. Create a new GitHub Issue for your task: https://github.com/OpenGPTX/lm-evaluation-harness/issues
3. Fetch recent changes from GitHub git fetch origin.
4. Create a new branch based on the master branch (git checkout origin/master), Name it according to the task that you will contribute. (git checkout -b <your_name>).
5. Try to reuse code from existing similar tasks in the framework. Otherwise, make use of existing datasets and metrics by HuggingFace. For more information about creating new tasks in our framework, see the task guide
6. Verify that your scores align with the scores reported in the literature
7. Run the pre-commit hooks (see below)
8. Commit and push only the task-relevant changes (lm_eval/tasks/__init__.py and lm_eval/tasks/<your task>.py) to a remote branch (git push origin <your branch>), create a pull request.
9. Close the issue for your task and, if necessary, delete the old branch in your fork.

Run tests before commit

# dependecy pre-commit
pip install pre-commit>=2.2.0

# install hooks
pre-commit install

# run hooks
pre-commit run --show-diff-on-failure --color=always --all-files