Question on DPO's concatenated_forward

[DaehanKim]

Hi!

It seems like chosen_input_ids and rejected_input_ids (and other inputs as well) are concatenated and fed into the behavior model to get concatenated logits : [chosen_logits | rejected_logits]

Doesn't it give a slight bias of looking at chosen_input_ids when computing rejected_logits? I wonder whether the impact of this on DPO performance is negligible compared to its efficiency trade-off.

Any comments would be appreciated! Thanks.

[lvwerra]

tagging @kashif here :)

[raghavgarg97]

@DaehanKim i think there may be slight confusion here..It is only concatenating inputs at batch level to avoid multiple passes..the calculation for chosen and rejected logits is independent.

[DaehanKim]

@raghavgarg97
I thought it was sequence level concat. Thanks for the correction!