Stage 2 training issues

[Fixed-Sidet]

Hello,

first, thanks for your outstanding work!

I am curently trying to train UniAD on the Carla dataset.
Due to ressource limitations, I reduced the size of the training dataset to 150 scenes, while keeping the distribution of towns and weather conditions equal.

However, when training stage2 I run into the following problem:

The loss goes up and then drops down again after a while.
This can be seen most clearly in the past_traj loss:

it is however also present in the motion.ade loss as well as others.
Removing the past_traj loss from the training does not solve the issue.

Maybe you can give me an insight in what the problem is here.

Thanks!

[jiaxiaosong1002]

@Fixed-Sidet It might be due to the sudden disappear of some vehicles in CARLA Thinklab-SJTU/Bench2Drive#31, while the TrackFormer could not handle it well.