Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add "Language Modeling to Unpaired Preference" in Utilities for converting dataset types #2436

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

AMindToThink
Copy link

This link adds a section to "Dataset Formats" for converting Language Modeling data to Unpaired Preference Data.

My motivation is that I wanted to apply reinforcement learning techniques like BCO to the WMDP dataset: a dataset of for "unlearning" harmful information.

I used the snippet I added to the docs to make an unpaired preference version of the WMDP cyber dataset. Others may want to make similar datasets.

This PR fixes a typo or improves the docs.

@qgallouedec
Copy link
Member

qgallouedec commented Dec 4, 2024

Thanks @AMindToThink for the addition.

Strictly speaking, you can't convert a language-modelling dataset (only text column) to a prompt-completion dataset, because you'd have to be able to extract the prompt from it. I'm afraid that adding this part to the documentation will create confusion.

The workaround you use is to have an empty prompt column. Which is a bit strange.

How about instead making the algorithms natively support this new type in which you only have a text column and a label colone?

@qgallouedec qgallouedec added the 😴 stale No update from the author, will be closed soon label Dec 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
😴 stale No update from the author, will be closed soon
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants