Bug in loss?

[rachtibat]

Hey,

thanks for the nice work!
I've noticed that you subtract in

AtMan/atman-magma/atman_magma/logit_parsing.py

Line 73 in 847f090

delta_cross_entropies_matrix[i] = (default_loss - loss)

the ground truth loss from the suppressed loss.

However, in your paper you do it the other way around as far as I understand: subtracting the suppressed loss from the ground truth. In my experiments, the latter also makes more sense because the suppressed loss should be bigger than the ground truth loss if you flip the correct token and if you want to assign a positive importance value to it.

Is this a bug?

Thanks!