#1340 fix top_n_tokens > 0

[greg-us]

What does this PR do?

Fix issue #1340

I found that the error 'Request failed during generation: Server error: Argument for field generate.v2.Generation.top_tokens is not iterable' seems caused by the partial update of parameters used to instanciate the Generation class in causal_lm.py, flash_causal_lm.py, idefics_causal_lm.py, and seq2seq_lm.py.

top_tokens parameter should be a list of Tokens to match the repeated Tokens expected by proto.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[drbh]

Hi @greg-us thanks for the contribution! It appears this solution may not work when top_n_tokens is 0 or undefined because the [top_tokens] may be [None]

ie this request

{
    "inputs": "What color is the sky?",
    "parameters": {
        "max_new_tokens": 20
    }
}

during generation

[
  Generation(
    request_id=3, prefill_tokens=None, tokens=Tokens(token_ids=[13], 
    logprobs=[-0.45703125], texts=['\n'], is_special=[False]
  ), 
  generated_text=None, top_tokens=[None])]
]

I've tested this with NousResearch/Llama-2-7b-chat-hf and gpt2 and this works when top_n_tokens is greater than 0

{
    "inputs": "What color is the sky?",
    "parameters": {
        "max_new_tokens": 20,
        "top_n_tokens": 2
    }
}

would it be possible to address these issues? thank you!

[greg-us]

Hi @drbh,
Thank you for your feedback, I overlooked a test here.

The issue has been resolved. Now, top_n_token can be either undefined, 0, or greater than 0.

Using the same query:

{
    "inputs": "What color is the sky?",
    "parameters": {
        "max_new_tokens": 20
    }
}

I now have an answer:

{
    "generated_text": "\n\nThe sky is not a single color, but rather a complex mixture of colors that change depending"
}

[OlivierDehaene]

I think top tokens is broken at a deeper level because of speculative decoding. @Narsil can you have a look?

[Narsil]

Indeed it is. I fixed everything here: #1497

To be fair top_n_tokens + Speculate was broken, each individually seemed fine.

[drbh]

closing due to this being fixed in another PR #1497

Thanks everyone!