[SFTTrainer] Support logging response in wandb

[vwxyzjn]

What does this PR do?

This PR attempts to log some sample and reference responses in wandb, which gives concrete examples for our inspection and also makes training more informative. Basically, it's going to log something as follows:

Some datasets such as timdettmers/openassistant-guanaco do not really have a query/response structure, so I basically give the model the first half of the token in the dataset and let it generate the remaining tokens; I also added the second half of the token as the reference response to test out the SFT policy.

Question:

I wasn't quite sure how to get the batching working with dataset and generate, though... The dataloader complains

ValueError: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length. Perhaps your features (`remaining_input_ids` in this case) have excessive nesting (inputs type `list` where type `int` is expected).

if I try to use a batch_size > 1 for the dataloader.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

[vwxyzjn]

update: after chatting with @lvwerra, we think maybe the change in SFTTrainer is unnecessary. Will checkout how can we further improve it.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

[JamesSand]

Hi, I have met the same problem. Does anyone solve that?

I have tried to use the demo code provided in the readme, but it still does not work....