New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

XStannce Topic Classification #43

Open

malteos opened this issue Nov 4, 2022 · 1 comment

Assignees

Labels

task

malteos commented Nov 4, 2022

Use xstance benchmark for topic classification instead of stance detection.

See https://github.com/OpenGPTX/lm-evaluation-harness/blob/master/lm_eval/tasks/x_stance.py

malteos assigned aishwaryaanegundi

malteos added the task label

Collaborator

aishwaryaanegundi commented Nov 21, 2022 •

edited

Loading

Evaluation of topic classification task on XStance dataset

HF dataset: https://huggingface.co/datasets/x_stance
Task: xstance_tc
Metrics: Classification Accuracy (acc, precision, recall, f1)

Zero-shot Results

From literature

No works found on topic classification for XStance dataset

From eval harness

Model: gpt2

With only question in prompt

Task	Version	Metric	Value		Stderr
xstance_tc	0	acc	0.1
		precision	0.0303
		recall	0.0909
		f1	0.0455

With both question and comment included in the prompt

Task	Version	Metric	Value		Stderr
xstance_tc	0	acc	0.1
		precision	0.0227
		recall	0.0909
		f1	0.0364

Comments

XStance dataset consists of question and comment. The performance is slightly better with just question included in the prompt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment