my loss getting bigger and bigger when I train

[fangshengbaihua]

Hi,I have a question. When I train, my loss is getting bigger and bigger, and I don't know what's causing it

Namespace(addaptadj=True, adjdata='data/sensor_graph/adj_mx.pkl', adjtype='doubletransition', aptonly=False, batch_size=64, data='data/METR-LA', device='cuda:0', dropout=0.3, epochs=100, expid=1, gcn_bool=True, in_dim=2, learning_rate=0.001, nhid=32, num_nodes=207, print_every=50, randomadj=True, save='./garage/metr', seq_length=12, weight_decay=0.0001)
start training...
Iter: 000, Train Loss: 11.5344, Train MAPE: 0.3075, Train RMSE: 13.9952
Iter: 050, Train Loss: 377.5500, Train MAPE: 7.0916, Train RMSE: 630.7411
Iter: 100, Train Loss: 352.2685, Train MAPE: 6.7022, Train RMSE: 619.5583
Iter: 150, Train Loss: 376.2829, Train MAPE: 7.7274, Train RMSE: 651.6628
Iter: 200, Train Loss: 415.7277, Train MAPE: 8.6364, Train RMSE: 692.0403
Iter: 250, Train Loss: 427.4500, Train MAPE: 8.3187, Train RMSE: 706.2341
Iter: 300, Train Loss: 403.1333, Train MAPE: 7.4553, Train RMSE: 686.9592
Iter: 350, Train Loss: 421.0964, Train MAPE: 8.3744, Train RMSE: 704.3835
Epoch: 001, Inference Time: 4.2194 secs
Epoch: 001, Train Loss: 388.0823, Train MAPE: 7.5629, Train RMSE: 660.2726, Valid Loss: 409.1663, Valid MAPE: 8.0573, Valid RMSE: 683.0002, Training Time: 1063.4692/epoch
Iter: 000, Train Loss: 383.0825, Train MAPE: 7.3515, Train RMSE: 672.6279
Iter: 050, Train Loss: 436.3956, Train MAPE: 7.9620, Train RMSE: 721.0291
Iter: 100, Train Loss: 423.3630, Train MAPE: 7.6969, Train RMSE: 715.8647
Iter: 150, Train Loss: 414.0448, Train MAPE: 9.1630, Train RMSE: 713.6893
Iter: 200, Train Loss: 396.0155, Train MAPE: 7.4647, Train RMSE: 699.6281
Iter: 250, Train Loss: 433.5888, Train MAPE: 8.1940, Train RMSE: 735.1317
Iter: 300, Train Loss: 445.8810, Train MAPE: 8.4676, Train RMSE: 747.1703
Iter: 350, Train Loss: 430.3223, Train MAPE: 8.1599, Train RMSE: 734.9741
Epoch: 002, Inference Time: 3.6677 secs
Epoch: 002, Train Loss: 423.8275, Train MAPE: 8.3188, Train RMSE: 721.4823, Valid Loss: 434.8144, Valid MAPE: 8.5525, Valid RMSE: 726.0127, Training Time: 122.6467/epoch
Iter: 000, Train Loss: 429.0443, Train MAPE: 7.9226, Train RMSE: 733.3237
Iter: 050, Train Loss: 417.4486, Train MAPE: 8.1126, Train RMSE: 723.7980
Iter: 100, Train Loss: 428.7359, Train MAPE: 8.0147, Train RMSE: 734.3259
Iter: 150, Train Loss: 443.9696, Train MAPE: 8.9476, Train RMSE: 747.8458
Iter: 200, Train Loss: 431.4466, Train MAPE: 8.3825, Train RMSE: 737.4367
Iter: 250, Train Loss: 418.8755, Train MAPE: 7.8954, Train RMSE: 726.3869
Iter: 300, Train Loss: 438.4908, Train MAPE: 8.5767, Train RMSE: 744.3538
Iter: 350, Train Loss: 421.3578, Train MAPE: 8.0956, Train RMSE: 728.9158
Epoch: 003, Inference Time: 4.0353 secs
Epoch: 003, Train Loss: 433.9845, Train MAPE: 8.4935, Train RMSE: 739.3757, Valid Loss: 437.7613, Valid MAPE: 8.6092, Valid RMSE: 730.9322, Training Time: 120.3957/epoch

[xxsenior]

I got the same problem. Do you solve that? Please share your experience.