DLA operator functionality is exposed through the TensorRT builder, which internally links to DLA SW libraries (see DLA Workflow). While some ONNX operators or functions may already be available in DLA SW, TensorRT may not expose them yet. See below for the support matrix of ONNX operators & functions on Orin DLA. If you are interested in a specific DLA operator that is not supported through TensorRT yet, feel free to raise a GitHub Issue and/or inform your NVIDIA representative (in particular for NVIDIA DRIVE customers).
See General Restrictions that apply to all operations below. Many of those ops are supported on Xavier DLA as well, see Layer Support and Restrictions.
TensorRT 8.6 supports operators up to Opset 17. Latest information of ONNX operators can be found here.
Note that the scripts in op_reconstruction/ are intended as a recipe for how ops currently not supported on DLA can be decomposed into supported ops. Depending on your setup, you may choose to perform such op reconstructions in the ONNX domain post-training (as done here) or during the training process (for example in TensorFlow or PyTorch). The case of "Native" in the DLA SW support column and "Reconstruction" in the TensorRT support column indicates that an op can be supported through TensorRT by decomposing it into other DLA ops already supported by TensorRT.
Below Operator Support Matrix requires the following minimum system config (the OS by default gets shipped with the DLA SW and TensorRT versions to its right):
| Hardware platform | OS | DLA SW version | TensorRT version |
|---|---|---|---|
| DRIVE Orin (Automotive) | DRIVE OS 6.0.6.0 | DLA 3.12.0 | TensorRT 8.5.10 |
| Jetson Orin (Embedded) | JetPack 5.1.1 | DLA 3.12.1 | TensorRT 8.5.2 |
| DRIVE Orin (Automotive) | DRIVE OS 6.0.7.0 | DLA 3.13.0 | TensorRT 8.6.10 |
| DRIVE Orin (Automotive) | DRIVE OS 6.0.8.0 | DLA 3.14.0 | TensorRT 8.6.11 |
If you are interested in a specific DLA operator to be enabled in TensorRT, feel free to raise a GitHub Issue and/or inform your NVIDIA representative (in particular for NVIDIA DRIVE customers).
| Operator/Function | Orin DLA support through TensorRT | Orin DLA support through DLA SW | Restrictions |
|---|---|---|---|
| Abs | Native | Native | See Unary layer in Layer Support and Restrictions |
| Acos | See RFE | Native (as of DLA 3.14.0) | |
| Acosh | See RFE | Native (as of DLA 3.14.0) | |
| Add | Native | Native | See Scale layer and ElementWise Layer in Layer Support and Restrictions |
| And | Reconstruction | Reconstruction | See op_reconstruction/And.py |
| Asin | See RFE | Native (as of DLA 3.14.0) | |
| Asinh | See RFE | Native (as of DLA 3.14.0) | |
| Atan | See RFE | Native (as of DLA 3.14.0) | |
| Atanh | See RFE | Native (as of DLA 3.14.0) | |
| AveragePool | Native | Native | See Pooling layer in Layer Support and Restrictions |
| BatchNormalization | Native | Native | See Scale layer in Layer Support and Restrictions |
| Ceil | See RFE | Native (as of DLA 3.14.0) | |
| Celu | See RFE | Native (as of DLA 3.14.0) | |
| Clip | Native | Native | See Activation layer in Layer Support and Restrictions |
| Concat | Native | Native | See Concatenation layer in Layer Support and Restrictions |
| Constant | Native | See RFE | TensorRT performs constant folding into supported DLA ops. You may need to allow GPU fallback to trigger the folding but the final operator would only run on DLA. |
| ConstantOfShape | Can be inferred at build time | Can be inferred at build time | |
| Conv | Native | Native | See Convolution and Fully Connected layers in Layer Support and Restrictions |
| ConvTranspose | Native | Native | See Deconvolution layer in Layer Support and Restrictions |
| Cos | See RFE | Native (as of DLA 3.14.0) | |
| Cosh | See RFE | Native (as of DLA 3.14.0) | |
| CumSum | Reconstruction | Reconstruction | With axis=1, it can be expressed through a 1x1 Conv |
| DepthToSpace | Reconstruction | Native (as of DLA 3.14.0) | See op_reconstruction/DepthToSpace.py |
| DequantizeLinear | Reconstruction | Reconstruction | Can be collapsed to INT8 scaling factor, switching from explicit to implicit quantization |
| Div | See RFE | Native (as of DLA 3.14.0) | |
| Elu | See RFE | Native (as of DLA 3.14.0) | |
| Equal | Native | Native | See Equal operation in Layer Support and Restrictions |
| Erf | See RFE | Native (as of DLA 3.14.0) | |
| Exp | See RFE | Native (as of DLA 3.14.0) | |
| Expand | See RFE | Native (as of DLA 3.14.0) | |
| Flatten | See RFE | See RFE | |
| Floor | See RFE | Native (as of DLA 3.14.0) | |
| Gather | Reconstruction | Reconstruction | See op_reconstruction/Gather.py |
| GatherElements | Reconstruction | Reconstruction | See op_reconstruction/Gather.py which works similarly |
| GatherND | Reconstruction | Reconstruction | See op_reconstruction/Gather.py which works similarly |
| Gemm | Native | See RFE | The second input must be a constant, also see Convolution and Fully Connected layers in Layer Support and Restrictions. TensorRT can translate this to a 1x1 Conv internally. |
| GlobalAveragePool | See RFE | Native | |
| GlobalLpPool | See RFE | Native (as of DLA 3.14.0) | |
| GlobalMaxPool | See RFE | Native | |
| Greater | See RFE | Native (as of DLA 3.14.0) | |
| GreaterOrEqual | See RFE | Native (as of DLA 3.14.0) | |
| GroupNormalization | See RFE | Reconstruction (as of DLA 3.14.0) | Can be expressed with ReduceMean, Sub, Pow, Add, Sqrt, Div, Mul |
| GRU | Reconstruction | Reconstruction | See op_reconstruction/GRU.py |
| HardSigmoid | See RFE | Native (as of DLA 3.14.0) | |
| HardSwish | See RFE | Native (as of DLA 3.14.0) | |
| Identity | Reconstruction | Native (as of DLA 3.14.0) | Can be expressed with identity BatchNormalization (TensorRT Scale layer) for example |
| InstanceNormalization | See RFE | Reconstruction (as of DLA 3.14.0) | Can be expressed with ReduceMean, Sub, Pow, Add, Sqrt, Div, Mul |
| LayerNormalization | See RFE | Reconstruction (as of DLA 3.14.0) | Can be expressed with ReduceMean, Sub, Pow, Add, Sqrt, Div, Mul |
| LeakyRelu | Native | Native | See Activation layer in Layer Support and Restrictions |
| Less | See RFE | Native (as of DLA 3.14.0) | |
| LessOrEqual | See RFE | Native (as of DLA 3.14.0) | |
| Log | See RFE | Native (as of DLA 3.14.0) | |
| LogSoftmax | See RFE | Reconstruction (as of DLA 3.14.0) | Can be expressed with Log and Softmax |
| LpNormalization | See RFE | Native (as of DLA 3.14.0) | |
| LpPool | See RFE | Native (as of DLA 3.14.0) | |
| LRN | Native | Native | See LRN (Local Response Normalization) layer in Layer Support and Restrictions |
| LSTM | Reconstruction | Reconstruction | Can be unrolled similarly to op_reconstruction/GRU.py |
| MatMul | Native | See RFE | The second input must be a constant, also see Convolution and Fully Connected layers in Layer Support and Restrictions. TensorRT can translate this to a 1x1 Conv internally. |
| Max | Native | Native | See ElementWise layer in Layer Support and Restrictions |
| MaxPool | Native | Native | See Pooling layer in Layer Support and Restrictions |
| Mean | See RFE | Native (as of DLA 3.14.0) | |
| Min | See RFE | Native | |
| Mish | See RFE | Native (as of DLA 3.14.0) | |
| Mod | See RFE | Native (as of DLA 3.14.0) | |
| MeanVarianceNormalization | See RFE | Reconstruction (as of DLA 3.14.0) | Can be expressed with ReduceMean, Sub, Pow, Add, Sqrt, Div, Mul |
| Mul | Native | Native | See Scale layer and ElementWise layer in Layer Support and Restrictions |
| Neg | Reconstruction | Reconstruction | See op_reconstruction/Neg.py |
| Not | Reconstruction | Reconstruction | See op_reconstruction/Not.py |
| Or | Reconstruction | Reconstruction | See op_reconstruction/Or.py |
| Pad | See RFE | Native (as of DLA 3.14.0) | |
| Pow | See RFE | Native (as of DLA 3.14.0) | |
| PRelu | Native | Native | See Parametric ReLU layer in Layer Support and Restrictions |
| QuantizeLinear | Reconstruction | Reconstruction | Can be collapsed to INT8 scaling factor, switching from explicit to implicit quantization |
| Reciprocal | See RFE | Native (as of DLA 3.14.0) | |
| ReduceMax | Reconstruction | Native | Can be reconstructied by decomposing into several MaxPool nodes |
| ReduceMean | See RFE | Native | Can be reconstructied by decomposing into several AveragePool or Conv nodes |
| ReduceMin | See RFE | Native | |
| ReduceLogSum | See RFE | Reconstruction (as of DLA 3.14.0) | Can be expressed with Log, ReduceSum and Exp |
| ReduceLogSumExp | See RFE | Reconstruction (as of DLA 3.14.0) | Can be expressed with ReduceSum and Log |
| ReduceL1 | See RFE | Native (as of DLA 3.14.0) | |
| ReduceL2 | See RFE | Native (as of DLA 3.14.0) | |
| ReduceSum | Reconstruction | Native (as of DLA 3.14.0) | Can be reconstructied by decomposing into Conv nodes or several AveragePool nodes while re-scaling with for example BatchNormalization (TensorRT Scale layer) |
| ReduceSumSquare | See RFE | Native (as of DLA 3.14.0) | |
| Relu | Native | Native | See Activation layer in Layer Support and Restrictions |
| Reshape | Native | Native | See Shuffle layer in Layer Support and Restrictions |
| Resize | Native | Native | See Resize layer in Layer Support and Restrictions |
| RNN | Reconstruction | Reconstruction | Can be unrolled similarly to op_reconstruction/GRU.py |
| Round | See RFE | Native (as of DLA 3.14.0) | |
| Scatter | Reconstruction | Reconstruction | See op_reconstruction/ScatterElements.py |
| ScatterElements | Reconstruction | Reconstruction | See op_reconstruction/ScatterElements.py which works similarly |
| Selu | See RFE | Native (as of DLA 3.14.0) | |
| Shape | Can be inferred at build time | Can be inferred at build time | |
| Shrink | See RFE | Native (as of DLA 3.14.0) | |
| Sigmoid | Native | Native | See Activation layer in Layer Support and Restrictions |
| Sign | See RFE | Native (as of DLA 3.14.0) | |
| Sin | See RFE | Native (as of DLA 3.14.0) | |
| Sinh | See RFE | Native (as of DLA 3.14.0) | |
| Slice | Native | Native | See Slice layer in Layer Support and Restrictions |
| Softmax | Native | Native | See SoftMax layer in Layer Support and Restrictions |
| Softplus | See RFE | Native (as of DLA 3.14.0) | |
| Softsign | See RFE | Native (as of DLA 3.14.0) | |
| SpaceToDepth | Reconstruction | Native (as of DLA 3.14.0) | See op_reconstruction/SpaceToDepth.py |
| Split | Native | Native | Translated to Slice inside TensorRT's ONNX parser. See Slice layer in Layer Support and Restrictions |
| Sqrt | See RFE | Native (as of DLA 3.14.0) | |
| Squeeze | See RFE | See RFE | |
| Sub | Native | Native | See ElementWise layer in Layer Support and Restrictions |
| Sum | Native | Native | See ElementWise layer in Layer Support and Restrictions |
| Tan | See RFE | Native (as of DLA 3.14.0) | |
| Tanh | Native | Native | See Activation layer in Layer Support and Restrictions |
| ThresholdedRelu | See RFE | Native (as of DLA 3.14.0) | |
| Transpose | Native | Native | See Shuffle layer in Layer Support and Restrictions |
| Tile | See RFE | Native (as of DLA 3.14.0) | |
| Unsqueeze | See RFE | See RFE | |
| Upsample | Native | Native | See Resize layer in Layer Support and Restrictions |
| Where | Reconstruction | Reconstruction | See op_reconstruction/Where.py |
| Xor | Reconstruction | Reconstruction | See op_reconstruction/Xor.py |