TensorRT Optimization

[nityanandmathur]

Is there any TensorRT optimised inference available?

[78Alpha]

No

[UmerrAhsan]

StyleTTS2 consists of several models and components that work together to generate audio. To optimize it using TensorRT, you first need to convert each model separately from PyTorch to ONNX.

Once converted, you can either:

• Run the ONNX models using ONNX Runtime with the TensorRT execution provider, or
• Convert the ONNX models directly into TensorRT format and perform inference using TensorRT's Python or C++ API.

From my experience with ablation studies, the decoder is the most resource-intensive component in StyleTTS2. If you aim for partial optimization, converting just the decoder from PyTorch to ONNX and running it in TensorRT can provide significant speed improvements. Alternatively, converting all models to ONNX and running them in ONNX Runtime with the TensorRT execution provider will also yield noticeable performance gains. This approach is feasible and I have tested it.

[nityanandmathur]

HI @UmerrAhsan. Could you please share the overall latencies of your ONNX model?

[UmerrAhsan]

Hi @nityanandmathur. I have ran the decoder model and predictor.text encoder model in tensorrt. It decreases my latency by over 50%. Also I have cached the style vectors from diffusion and style encoder before. After that, a single short sentence runs in under 100ms.