Why doesn't the model input include attention_mask?

[li3cmz]

DialoGPT/LSP_train.py

Line 281 in fa0c0c5

loss, ppl = model(input_ids, position_ids, token_ids, label_ids)

Since it is a LMHeadModel, the 1^th-n^th tokens are used to predict the (n+1)^th token during training, so why not introduce attention_mask for masking the (n+2)^th-(n+m)^th tokens. Without attention_mask, there maybe an inconsistency between the training and the testing scene. Is it possible to add attention_mask during training to make the testing better?

[chujiezheng]

Because GPT is a uni-directional language model. It does not need attention mask.

[lmrojasb]

DialoGPT/data_loader.py

Why is the response concatenated to the input_ids for both the train and validation datasets? Would not this create over-fitted models? Would it be possible to somehow mask the response ids?