Training example

[Hinode]

HI [Jacob Bieker], can you show us an example to prepared training data set and train the model? And I recently tried to implement the model and got "segmentation fault" after certain training epoch. Do you experience any memory leakage? Thanks.

[jacobbieker]

Hi, I don't have the training data ready yet, #13 is tracking on me fixing that so that the paper can be reproduced, or at least tried to be reproduced. As I am still working on getting the training data prepped, I haven't actually tried training it for a lot of iterations. Could you maybe submit a bug report with the code you are using that results in the segmentation fault? I do know these models are quite large and can mean running out of memory when trying to train them. I'd recommend shrinking the model parameters to make it more easily fit on whatever device you are using.

[Hinode]

Thanks for the quick reply. I don't know if it is a bug in the MetNet model or my input dimension is problematic since there is no example. I also did shrink the model sizes (parameters) but still got the segmentation fault at a later epoch. I train the model on Nvidia Tesla P100 16 GB GPU. I actually have four identical GPUs. But I don't know how to parallel the code. Can you give me a little bit of hint to do so? Again, thank you so much for your response.

[jacobbieker]

Okay, I've added some examples on passing through an input to MetNet and MetNet-2 to the readme, so hopefully that helps? You could try parallelizing it with PyTorch Lightning, which helps abstract a lot of that work away and makes it a bit easier.