What is NVM, why it takes so long to get data from it?

[DzAvril]

Below is the profile result of restnet_50 running on DLA profiled by TensorRT. If using --int8, total latency is about 1.7ms, but with –fp16 it takes 17ms. The only difference is with –fp16 enabled, it has a process called fc1000 to nvm. And fc1000i s the last node of the model. So what is nvm, and why it takes so long to copy data from nvm?

[03/01/2023-11:45:20] [I]                          Layer   Time (ms)   Avg. Time (ms)   Time %
[03/01/2023-11:45:20] [I]                    data to nvm       15.52           0.0862      0.5
[03/01/2023-11:45:20] [I]  {ForeignNode[conv1...fc1000]}       54.09           0.3005      1.7
[03/01/2023-11:45:20] [I]                fc1000 from nvm     3116.62          17.3145     97.8
[03/01/2023-11:45:20] [I]               data copy finish        0.49           0.0027      0.0
[03/01/2023-11:45:20] [I]             fc1000 copy finish        0.80           0.0044      0.0
[03/01/2023-11:45:20] [I]                          Total     3187.51          17.7084    100.0```

[rcheruku09]

This is due to an issue in the reporting with older TensorRT versions where the DLA runtime is reported under "fc1000 from nvm". This has been resolved in versions after TensorRT 8.5.