iommu smmu dma and performance

Performance Tip

set cma=0 and iommu.passthrough=1 when testing device performance, unless device cannot work or there is security concern.

• Disable CMA: set cma=0 in kernel command line. Before 5.10 (see the per-numa-cma patch), there is only one default CMA reserved. This may affect performance in numa system if DMA is mapped from CMA.
• Bypass SMMU for DMA translation(set iommu.passthrough=1), when performance is more important than security. Both host kernel driver and user space driver (VFIO) can work when setting iommu.passthrough=1. VFIO is used in DPDK, SPDK and KVM device assignment.
• If for any reason we cannot set iommu.passthrough=1, enable iommu lazy mode (set iommu.strict=0).
• When security is a concern, enable iommu strict mode.

Details

https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html

        iommu.passthrough=
                        [ARM64, X86] Configure DMA to bypass the IOMMU by default.
                        Format: { "0" | "1" }
                        0 - Use IOMMU translation for DMA.
                        1 - Bypass the IOMMU for DMA.
                        unset - Use value of CONFIG_IOMMU_DEFAULT_PASSTHROUGH.

        iommu.strict=   [ARM64] Configure TLB invalidation behaviour
                        Format: { "0" | "1" }
                        0 - Lazy mode.
                          Request that DMA unmap operations use deferred
                          invalidation of hardware TLBs, for increased
                          throughput at the cost of reduced device isolation.
                          Will fall back to strict mode if not supported by
                          the relevant IOMMU driver.
                        1 - Strict mode (default).
                          DMA unmap operations invalidate IOMMU hardware TLBs
                          synchronously.

        cma=nn[MG]@[start[MG][-end[MG]]]
                        [KNL,CMA]
                        Sets the size of kernel global memory area for
                        contiguous memory allocations and optionally the
                        placement constraint by the physical address range of
                        memory allocations. A value of 0 disables CMA
                        altogether. For more information, see
                        kernel/dma/contiguous.c

        cma_pernuma=nn[MG]
                        [ARM64,KNL,CMA]
                        Sets the size of kernel per-numa memory area for
                        contiguous memory allocations. A value of 0 disables
                        per-numa CMA altogether. And If this option is not
                        specificed, the default value is 0.
                        With per-numa CMA enabled, DMA users on node nid will
                        first try to allocate buffer from the pernuma area
                        which is located in node nid, if the allocation fails,
                        they will fallback to the global default memory area.