Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Loading Dataset returns error from huggingface #2

Open
catherinelee274 opened this issue May 12, 2024 · 0 comments
Open

Loading Dataset returns error from huggingface #2

catherinelee274 opened this issue May 12, 2024 · 0 comments

Comments

@catherinelee274
Copy link

running this:

from datasets import load_dataset
dataset = load_dataset("alexshengzhili/mllm-dpo",split='train[0:1]',trust_remote_code=True)

returns this error:

ArrowInvalid Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/datasets/packaged_modules/json/json.py in _generate_tables(self, files)
121 try:
--> 122 pa_table = paj.read_json(
123 io.BytesIO(batch), read_options=paj.ReadOptions(block_size=block_size)

17 frames
ArrowInvalid: JSON parse error: Column() changed from object to array in row 0

During handling of the above exception, another exception occurred:

ArrowTypeError Traceback (most recent call last)
ArrowTypeError: Expected bytes, got a 'int' object

The above exception was the direct cause of the following exception:

DatasetGenerationError Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/datasets/builder.py in _prepare_split_single(self, gen_kwargs, fpath, file_format, max_shard_size, job_id)
2036 if isinstance(e, DatasetGenerationError):
2037 raise
-> 2038 raise DatasetGenerationError("An error occurred while generating the dataset") from e
2039
2040 yield job_id, True, (total_num_examples, total_num_bytes, writer._features, num_shards, shard_lengths)

DatasetGenerationError: An error occurred while generating the dataset

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant