v1.3.x release: HTTP 424 when requesting top_n_tokens > 0

[CaveSven]

System Info

Official docker container, v1.3.x

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

Steps to reproduce:

1. Launch docker container
2. Navigate to swagger api
3. Send vanilla example request for POST /generate (top_n_tokens > 0) and observe HTTP 424

{
  "error": "Request failed during generation: Server error: Argument for field generate.v2.Generation.top_tokens is not iterable",
  "error_type": "generation"
}

Expected behavior

HTTP 200

[shivanandmn]

i am still facing this issue, top_n_tokens = None is works fine, but if you any int value it throw error

[Soumendraprasad]

@CaveSven , @shivanandmn did you find any way to solve it . I am also getting this error . Let docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:1.3 --model-id $model , it is the command . Where to make changes .

[shivanandmn]

while calling the api just add top_n_tokens=None

[SvenRohrTNG]

Hi @Narsil,
I think the offending change is the newly added "repeated" keyword in the generate.proto, see 9ecfa16#r135971436
If I remove it, everything is as expected.

[SvenRohrTNG]

I would be happy to contribute a fix, but after looking through the code I wonder why 9ecfa16#diff-b1fa6727513f9184386a8371f182d51c9ccd6d131d61be6218170a6e2444146aR527 is laid out to handle multiple tokens in one single generation: as far as I can see, the python backend always just sends a single token per generation (e.g. see 9ecfa16#diff-f16065c0ba908e84102217a2c5d97bf724feed593985d7cbed77f6778a12a772R705)

[greg-us]

I think I found a way to solve this problem.

In PR1308, proto generate.v2 was introduced. Token details are replaced by the Tokens message, and TopTokens is replaced by the repeated Tokens message.

The outputs of server models evolved accordingly, but only partially. Even if a single token is generated, a list is returned in a Tokens class, whereas top_tokens didn't adapt, as seen here.

I have proposed a simple fix, returning a list of top_tokens from generation and thus a repeated Token message for protobuf.