Fix test fetcher (doctest) + Idefics2's doc example
#30274
Conversation
| @@ -507,7 +507,7 @@ def get_all_doctest_files() -> List[str]: | |||
| # change to use "/" as path separator | |||
| test_files_to_run = ["/".join(Path(x).parts) for x in test_files_to_run] | |||
| # don't run doctest for files in `src/transformers/models/deprecated` | |||
| test_files_to_run = [x for x in test_files_to_run if "models/deprecated" not in test_files_to_run] | |||
| test_files_to_run = [x for x in test_files_to_run if "models/deprecated" not in x] | |||
|
|
||
| >>> # Generate | ||
| >>> generated_ids = model.generate(**inputs, bad_words_ids=BAD_WORDS_IDS, max_new_tokens=500) | ||
| >>> generated_ids = model.generate(**inputs, bad_words_ids=BAD_WORDS_IDS, max_new_tokens=20) |
There was a problem hiding this comment.
it's very slow, so let's just use 20
| >>> generated_texts = processor.batch_decode(generated_ids, skip_special_tokens=True) | ||
|
|
||
| >>> print(generated_texts) | ||
| ['In this image, we can see the city of New York, and more specifically the Statue of Liberty. In this image, we can see the city of New York, and more specifically the Statue of Liberty.\n\n', 'In which city is that bridge located?\n\nThe bridge is located in the city of Pittsburgh, Pennsylvania.\n\n\nThe bridge is'] |
There was a problem hiding this comment.
The results are not very good. But we don't have it previously, so I don't know if we consider it bad.
There was a problem hiding this comment.
I'm surprised by these generations - they also don't look like the outputs when I was integrating the model 🤔
There was a problem hiding this comment.
maybe you were using other type of GPUs. CI is using T4.
There was a problem hiding this comment.
Quite possibly - I was using NVIDIA A10G. I find it quite concerning the generations can be so much worse between hardware though, especially given this isn't a tiny model
There was a problem hiding this comment.
I can check it tomorrow. But I will merge first 🙏
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
| >>> generated_texts = processor.batch_decode(generated_ids, skip_special_tokens=True) | ||
|
|
||
| >>> print(generated_texts) | ||
| ['In this image, we can see the city of New York, and more specifically the Statue of Liberty. In this image, we can see the city of New York, and more specifically the Statue of Liberty.\n\n', 'In which city is that bridge located?\n\nThe bridge is located in the city of Pittsburgh, Pennsylvania.\n\n\nThe bridge is'] |
There was a problem hiding this comment.
I'm surprised by these generations - they also don't look like the outputs when I was integrating the model 🤔
) fix Co-authored-by: ydshieh <[email protected]>
fix Co-authored-by: ydshieh <[email protected]>
fix Co-authored-by: ydshieh <[email protected]>
fix Co-authored-by: ydshieh <[email protected]>
What does this PR do?
Idefics2's doc example: not able to fit into GPU even with fp16. So I changed it.