Model architecture with --model="ScaleShiftMACE"

[wcwitt]

Assuming most other defaults are used, models trained with --model="ScaleShiftMACE" appear to be different in at least one important, nonobvious way from models trained with --model="MACE".

Specifically, --model="ScaleShiftMACE" yields an architecture where both layer interactions are RealAgnosticResidualInteractionBlock, whereas the default MACE uses RealAgnosticInteractionBlock in the first layer and RealAgnosticResidualInteractionBlock in the second.

Perhaps the most significant consequence is that the linear weights applied after the first pooling are element-dependent for the default architecture but not element-dependent by default for --model="ScaleShiftMACE.

Hopefully I have understood everything correctly - @ilyes319 can confirm. Perhaps not technically a bug, but probably needs documentation. My impression is that some users are still in the habit of setting --model=ScaleShiftMACE, and they may not be getting what they expect.

[birdyLinch]

Thanks for bringing this up! Upon checking the main branch: here for --model="MACE" the first interaction block would be fixed as RealAgnosticInteractionBlock while here it would be reading the argument --interaction_first from the config and was settled with RealAgnosticResidualInteractionBlock as default here in argparser. Shall we change the default to RealAgnosticInteractionBlock, or is it required for backward compatibility @ilyes319 ?