Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create dataset loader for Okapi m-TruthfulQA #477

Open
SamuelCahyawijaya opened this issue Mar 3, 2024 · 2 comments
Open

Create dataset loader for Okapi m-TruthfulQA #477

SamuelCahyawijaya opened this issue Mar 3, 2024 · 2 comments
Assignees

Comments

@SamuelCahyawijaya
Copy link
Collaborator

Dataloader name: okapi_m_truthfulqa/okapi_m_truthfulqa.py
DataCatalogue: http://seacrowd.github.io/seacrowd-catalogue/card.html?okapi_m_truthfulqa

Dataset okapi_m_truthfulqa
Description m-TruthfulQA is a multi-lingual version of TruthfulQA, a benchmark to measure whether a language model is truthful in generating answers to questions. The benchmark comprises 817 questions that span 38 categories, including health, law, finance and politics.
Subsets -
Languages ind, vie
Tasks Question Answering
License Creative Commons Attribution Non Commercial 4.0 (cc-by-nc-4.0)
Homepage http://nlp.uoregon.edu/download/okapi-eval/datasets/
HF URL https://huggingface.co/datasets/jon-tow/okapi_truthfulqa
Paper URL https://arxiv.org/abs/2307.16039
@SamuelCahyawijaya SamuelCahyawijaya converted this from a draft issue Mar 3, 2024
@tellarin
Copy link
Collaborator

tellarin commented Mar 3, 2024

#self-assign

@tellarin
Copy link
Collaborator

Back working on this. Sorry for the delay.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: No status
Development

No branches or pull requests

2 participants