SimPO (better preference optimization for NLP and without reference model)

[samedii]

Hi! Has anyone else done any experiments with this for diffusion? I've tried a bunch of settings (based on DPO) and have been unable to get results on the same level as DPO so far.

Here's the official for NLP:
https://github.com/princeton-nlp/SimPO

Here's mine for diffusion:

DPO

            beta_dpo = 5000  # 2500 for SD15, 5000 for SDXL
            loss = (
                -F.logsigmoid(beta_dpo * (reference_preference_loss - preference_loss))
                .mean()
            )

SimPO

            loss = (
                -F.logsigmoid(beta * (gamma / beta - preference_loss)).mean()
            )