[DataCollatorForCompletionOnlyLM] Warn on identical eos_token_id and pad_token_id

[MustSave]

What does this PR do?

This PR displays a warning message when the values of pad_token_id and eos_token_id are identical. This is to prevent unexpected behavior during multi-turn training.

After the multi-turn data training with DataCollatorForCompletionOnlyLM, I encountered an issue where the model continued generating outputs even after the assistant's turn had been completed. This issue was due to the equivalence of the tokenizer's eos token and pad token by default in some model, resulting in the eos token not being properly trained.

For instance, in the torch_call() function within data_collator.py, the pad_token_id is converted to ignore_id(-100).

if self.mlm:
    batch["input_ids"], batch["labels"] = self.torch_mask_tokens(
        batch["input_ids"], special_tokens_mask=special_tokens_mask
    )
else:
    labels = batch["input_ids"].clone()
    if self.tokenizer.pad_token_id is not None:
        labels[labels == self.tokenizer.pad_token_id] = -100
    batch["labels"] = labels

If the multi-turn data is formatted as shown below, and eos_token_id and pad_token_id are identical, the eos_token would not be properly trained. This could lead to a scenario where the model continuously generates both user and assistant turns without recognizing the end of sequence (eos) token.

<s>### User: What is up?
### Assistant: Hello! How can I help you today?</s>
### User: Goodbye
### Assistant: Goodbye! If you have any more questions in the future, don't hesitate to ask.</s>

Suggestion

In case of Vicuna, they set pad_token to unk_token. Would it be a solution?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[younesbelkada]

Makes a lot of sense , thank you!
Related: #976