diff --git a/configs/AM62AX/AM62AX_linux_toc.txt b/configs/AM62AX/AM62AX_linux_toc.txt index 960ea78ec..8f95c955b 100644 --- a/configs/AM62AX/AM62AX_linux_toc.txt +++ b/configs/AM62AX/AM62AX_linux_toc.txt @@ -103,7 +103,6 @@ linux/Foundational_Components_IPC62ax linux/Foundational_Components_Edge_AI linux/Foundational_Components_OPTEE linux/Foundational_Components_ATF -linux/Foundational_Components_Multimedia_Other_Tools linux/Foundational_Components_Multimedia_wave5 linux/How_to_Guides linux/How_to_Guides_Developer_Notes diff --git a/configs/AM62PX/AM62PX_linux_toc.txt b/configs/AM62PX/AM62PX_linux_toc.txt index 9cfe583fe..13cbe337b 100644 --- a/configs/AM62PX/AM62PX_linux_toc.txt +++ b/configs/AM62PX/AM62PX_linux_toc.txt @@ -108,7 +108,6 @@ linux/Foundational_Components_IPC62px linux/Foundational_Components_Machine_Learning linux/Foundational_Components_OPTEE linux/Foundational_Components_ATF -linux/Foundational_Components_Multimedia_Other_Tools linux/Foundational_Components_Multimedia_wave5 linux/Foundational_Components_Virtualization linux/Foundational_Components/Virtualization/Docker diff --git a/configs/AM67/AM67_linux_toc.txt b/configs/AM67/AM67_linux_toc.txt index cc7fe9d88..a2de72672 100644 --- a/configs/AM67/AM67_linux_toc.txt +++ b/configs/AM67/AM67_linux_toc.txt @@ -84,7 +84,6 @@ linux/Foundational_Components_Virtualization linux/Foundational_Components/Virtualization/Docker linux/Foundational_Components_OPTEE linux/Foundational_Components_ATF -linux/Foundational_Components_Multimedia_Other_Tools linux/Foundational_Components_Multimedia_wave5 linux/Foundational_Components_Graphics diff --git a/configs/AM68/AM68_linux_toc.txt b/configs/AM68/AM68_linux_toc.txt index 8d7fb4999..9b2d286e3 100644 --- a/configs/AM68/AM68_linux_toc.txt +++ b/configs/AM68/AM68_linux_toc.txt @@ -89,7 +89,6 @@ linux/Foundational_Components_Virtualization linux/Foundational_Components/Virtualization/Docker linux/Foundational_Components_OPTEE linux/Foundational_Components_ATF -linux/Foundational_Components_Multimedia_Other_Tools linux/Foundational_Components_Multimedia_wave5 linux/Foundational_Components_Graphics diff --git a/configs/AM69/AM69_linux_toc.txt b/configs/AM69/AM69_linux_toc.txt index 2d8f7f494..0411e52b7 100644 --- a/configs/AM69/AM69_linux_toc.txt +++ b/configs/AM69/AM69_linux_toc.txt @@ -89,7 +89,6 @@ linux/Foundational_Components_Virtualization linux/Foundational_Components/Virtualization/Docker linux/Foundational_Components_OPTEE linux/Foundational_Components_ATF -linux/Foundational_Components_Multimedia_Other_Tools linux/Foundational_Components_Multimedia_wave5 linux/Foundational_Components_Graphics diff --git a/configs/J721S2/J721S2_linux_toc.txt b/configs/J721S2/J721S2_linux_toc.txt index 226d0ef0f..c36c205e3 100644 --- a/configs/J721S2/J721S2_linux_toc.txt +++ b/configs/J721S2/J721S2_linux_toc.txt @@ -92,7 +92,6 @@ linux/Foundational_Components_Virtualization linux/Foundational_Components/Virtualization/Docker linux/Foundational_Components_OPTEE linux/Foundational_Components_ATF -linux/Foundational_Components_Multimedia_Other_Tools linux/Foundational_Components_Multimedia_wave5 linux/Foundational_Components_Graphics diff --git a/configs/J722S/J722S_linux_toc.txt b/configs/J722S/J722S_linux_toc.txt index 0f73d400f..fd644cf4f 100644 --- a/configs/J722S/J722S_linux_toc.txt +++ b/configs/J722S/J722S_linux_toc.txt @@ -87,7 +87,6 @@ linux/Foundational_Components_Virtualization linux/Foundational_Components/Virtualization/Docker linux/Foundational_Components_OPTEE linux/Foundational_Components_ATF -linux/Foundational_Components_Multimedia_Other_Tools linux/Foundational_Components_Multimedia_wave5 linux/Foundational_Components_Graphics diff --git a/configs/J742S2/J742S2_linux_toc.txt b/configs/J742S2/J742S2_linux_toc.txt index d3ee7d93b..20817e1c3 100644 --- a/configs/J742S2/J742S2_linux_toc.txt +++ b/configs/J742S2/J742S2_linux_toc.txt @@ -91,7 +91,6 @@ linux/Foundational_Components_Virtualization linux/Foundational_Components/Virtualization/Docker linux/Foundational_Components_OPTEE linux/Foundational_Components_ATF -linux/Foundational_Components_Multimedia_Other_Tools linux/Foundational_Components_Multimedia_wave5 linux/Foundational_Components_Graphics diff --git a/configs/J784S4/J784S4_linux_toc.txt b/configs/J784S4/J784S4_linux_toc.txt index 6fb506838..701e446d9 100644 --- a/configs/J784S4/J784S4_linux_toc.txt +++ b/configs/J784S4/J784S4_linux_toc.txt @@ -91,7 +91,6 @@ linux/Foundational_Components_Virtualization linux/Foundational_Components/Virtualization/Docker linux/Foundational_Components_OPTEE linux/Foundational_Components_ATF -linux/Foundational_Components_Multimedia_Other_Tools linux/Foundational_Components_Multimedia_wave5 linux/Foundational_Components_Graphics diff --git a/source/linux/Foundational_Components.rst b/source/linux/Foundational_Components.rst index 17e4f3391..86e340a70 100644 --- a/source/linux/Foundational_Components.rst +++ b/source/linux/Foundational_Components.rst @@ -29,7 +29,6 @@ Foundational Components Foundational_Components_Graphics Foundational_Components_Multimedia_IVAHD Foundational_Components_Multimedia_D5520_VXE384 - Foundational_Components_Multimedia_Other_Tools Foundational_Components_Multimedia_wave5 Foundational_Components_OpenCL Foundational_Components_OpenCV diff --git a/source/linux/Foundational_Components_Multimedia_Other_Tools.rst b/source/linux/Foundational_Components_Multimedia_Other_Tools.rst deleted file mode 100644 index 970018e52..000000000 --- a/source/linux/Foundational_Components_Multimedia_Other_Tools.rst +++ /dev/null @@ -1,224 +0,0 @@ -############################# -FFmpeg Plugins for Multimedia -############################# - -:command:`ffmpeg` is an open-source multimedia framework. This is useful for decoding, encoding and transcoding videos. This is a command line tool. - -The library comes with :command:`ffplay` and :command:`ffprobe`. - -:command:`ffplay` is a very simple and portable media player using the :command:`ffmpeg` libraries and the SDL library. It is mostly used as a testbed for the various :command:`ffmpeg` APIs. - -:command:`ffprobe` gathers information from multimedia streams and prints it in human and machine-readable fashion. - -For example, it can be used to check the format of the container used and the format and type of each media stream contained in it. - -To enable :command:`ffplay`, the following needs to be added to :file:`local.conf`: - -.. code-block:: text - - PACKAGECONFIG:append:pn-ffmpeg = " sdl2" - -More information about the :command:`ffmpeg` tools can be found here: - - https://ffmpeg.org/ffmpeg.html - - https://ffmpeg.org/ffplay.html - - https://ffmpeg.org/ffprobe.html - -************ -Useful Flags -************ - -ffmpeg -====== - -Help Flags ----------- -``-h`` or ``-?`` or ``-help`` - Topic show help -``-codecs`` - Displays all codecs supported -``-decoders`` - Displays all decoders supported -``-formats`` - Displays all available formats -``-pix_fmts`` - Displays all available pixel formats - -``-codec:v`` - Specifies the stream decoder. For decoding, we only ``hevc_v4l2m2m`` and ``h264_v4l2m2m``. :v stands for video - -Stream Specific Flags ---------------------- -``-re`` - Reads input at native frame rate -``-i [filename]`` - Input stream -``-fps_mode passthrough`` - Allows for frames to pass even if out of sync. Needed since there are no timestamps in decoded h264/5 streams that use the hardware GPU to decode. Must be placed in between input and output stream -``-pix_fmt`` - Specifies the pixel format for the output stream. NV12 is recommended for wave5 -``-f`` - Forces format of output stream -``-frames`` - Set the number of frames to record - -ffplay -====== -``-framerate`` - Specifies framerate of display stream -``-video_size`` - Specifies video size of video. It may be the string of the form or *widthxheigh*. For example, 2k or 2048x1080 are equivalent. -``-pixel_format`` - Specifies pixel format of display stream -``-f`` - Forces format of display stream -``-fs`` - Start in fullscreen mode -``-autoexit`` - Exits ffplay after stream is finished - -ffprobe -======= -``-show_data`` - Show payload data, as a hexadecimal and ASCII dump. Coupled with -show_packets, it will dump the packets’ data. Coupled with -show_streams, it will dump the codec extradata - The dump is printed as the "data" field. It may contain newlines -``-show_packets`` - Show information about each packet contained in the input multimedia stream - The information for each single packet is printed within a dedicated section with name "PACKET" -``-show_frames`` - Show information about each frame and subtitle contained in the input multimedia stream - The information for each single frame is printed within a dedicated section with name "FRAME" or "SUBTITLE" -``-show_format`` - Show information about the container format of the input multimedia stream - All the container format information is printed within a section with name "FORMAT" - -**************** -Example Commands -**************** - -To convert a HEVC file to a raw video with pixel format NV12 named :file:`foo.yuv` with software acceleration, the following can be used: - -.. code-block:: console - - ffmpeg -i foo.265 -f rawvideo -pix_fmt nv12 foo.yuv - -To convert a H264 file to a raw video named :file:`bar.yuv` with hardware acceleration and each frame passed, the following can be used: - -.. code-block:: console - - ffmpeg -codec:v h264_v4l2m2m -i bar.h264 -f rawvideo -fps_mode passthrough bar.yuv - -To play a H264 video and output to the display at fullscreen and exit once the video is done playing, the following can be used: - -.. code-block:: console - - ffplay -fs -autoexit foobar.264 - -To pipe the output from ffmpeg to ffplay with hardware acceleration, the following can be used: - -.. code-block:: console - - ffmpeg -re -codec:v hevc_v4l2m2m -i input.h265 -fps_mode passthrough -f rawvideo - | ffplay -framerate 30 -video_size 1920x1080 -f rawvideo -autoexit - - -To print out the packet's data and payload data in hexadecimal from a given multimedia stream named :file:`foo.h265`, the following can be used: - -.. code-block:: console - - ffprobe -show_data -show_packets foo.h265 - -To print information about each frame and/or subtitle in the given multimedia stream named :file:`bar.h264`, the following can be used: - -.. code-block:: console - - ffprobe -show_frames bar.h264 - -########################## -MPV Plugins for Multimedia -########################## -mpv is a media player based on MPlayer and mplayer2. It supports a wide variety of video file formats, audio and video codecs, and subtitle types. -Special input URL types are available to read input from a variety of sources other than disk files. -Depending on platform, a variety of different video and audio output methods are supported. - -This is a command line tool that can also use keyboard shortcuts. - -https://mpv.io/manual/stable/ - -:command:`yt-dlp` can be combined with mpv to play videos from the web. (YouTube, TikTok, etc.) - -To install :command:`yt-dlp` run the following command - -.. code-block:: console - - python3 -m pip install yt-dlp - -************ -Useful Flags -************ - -``--vo=[driver]`` - -Specify the video output backend to be used. See `VIDEO OUTPUT DRIVERS `_ for details and descriptions of available drivers. - -``--hwdec=[api1,api2,...|no|auto|auto-safe|auto-copy]`` - -Specify the hardware video decoding API that should be used if possible. Whether hardware decoding is actually done depends on the video codec. If hardware decoding is not possible, mpv will fall back on software decoding. -Hardware decoding is not enabled by default, to keep the out-of-the-box configuration as reliable as possible. However, when using modern hardware, hardware video decoding should work correctly, offering reduced CPU usage, and possibly lower power consumption. - -``--gpu-context=[sys]`` - -The value auto (the default) selects the GPU context. You can also pass help to get a complete list of compiled in backends (sorted by autoprobe order). - -``--fs`` - -Fullscreen playback. - -``--autofit=<[W[xH]]>`` - -Set the initial window size to a maximum size specified by WxH, without changing the window's aspect ratio. The size is measured in pixels, or if a number is followed by a percentage sign (%), in percents of the screen size. -This option never changes the aspect ratio of the window. If the aspect ratio mismatches, the window's size is reduced until it fits into the specified size. - -``--no-correct-pts`` - -Switches mpv to a mode where video timing is determined using a fixed framerate value (either using the --container-fps-override option, or using file information). Sometimes, files with very broken timestamps can be played somewhat well in this mode. - -``--fps=[framerate]`` - -Set framerate of output stream. Can be combined with --no-correct-pts for streams with incorrect/no timestamps - -**************** -Example Commands -**************** - -To play a H264 video to display at 30 frames per seconds, the following can be used: - -.. code-block:: console - - mpv --no-correct-pts --fps=30 foo.h264 - -To play a HEVC video with full/assisted hardware acceleration at 30 frames per second, the following can be used: - -.. code-block:: console - - mpv --vo=gpu --hwdec=auto --no-correct-pts --fps=30 bar.h265 - -To play a mp4 video to display in fullscreen and set the gpu context to wayland, the following can be used: - -.. code-block:: console - - mpv --vo=gpu --gpu-context=wayland --fs foobar.mp4 - -To download a video from a link and pipe the video to mpv with no correct points at 30 frames per seconds, the following can be used: - -.. code-block:: console - - yt-dlp -o - | mpv --no-correct-pts --fps=30 - - -Tp download a video from a link at 1080p and play the video with mpv, the following can be used: - -.. code-block:: console - - yt-dlp -f 270 -o - | mpv - - -``-f 232`` can be used to get a 720p video. - -In instances where running yt-dlp doesn't work because of a protocol issue, updating to the newest version of yt-dlp or changing -the format/quality by running ``-f 270`` or ``-f 232`` for example, or playing other videos might solve the issue. \ No newline at end of file diff --git a/source/linux/Foundational_Components_Multimedia_wave5.rst b/source/linux/Foundational_Components_Multimedia_wave5.rst index 8bbbb00d5..ea48c7495 100644 --- a/source/linux/Foundational_Components_Multimedia_wave5.rst +++ b/source/linux/Foundational_Components_Multimedia_wave5.rst @@ -10,8 +10,7 @@ Multimedia Video Codec Introduction ************ -The Encoder/Decoder (VENC/VDEC) is a stateful encoder/decoder. It is found on the |__PART_FAMILY_DEVICE_NAMES__| SoC. -Combined H.264 and H.265 encoder/decoder used in the Texas Instruments |__PART_FAMILY_DEVICE_NAMES__| SoC. +The Wave5 Codec IP is a combined H.264 and H.265 stateful encoder/decoder. It is found on the Texas Instruments |__PART_FAMILY_DEVICE_NAMES__| SoC. Hardware capabilities: - Maximum resolution: 8192x8192 (It can handle this resolution, but not @@ -33,17 +32,89 @@ Decoder : - Capable of decoding H.265 Main and Main Still Picture Profile @ L5.1 High tier. - Capable of decoding H.264 Baseline/Constrained Baseline/Main/High Profiles @ L5.2. -| -**The V4L2 compliance tests report the following controls as available:** +********************* +Software Architecture +********************* + +Software Stack of Accelerated Codec Encoding/Decoding +===================================================== + +As shown in the figures below, the software stack of the accelerated +encoding and decoding has two parts: + +.. ifconfig:: CONFIG_part_variant in ('AM62AX', 'AM62PX', 'J722S', 'AM67') + + - A V4L2 (Video4Linux version 2) software driver running on Linux on the A53 MPU subsystem + - The firmware running on the DECODER and ENCODER + +.. ifconfig:: CONFIG_part_variant in ('J721S2', 'J784S4', 'J742S2') + + - A V4L2 (Video4Linux version 2) software driver running on Linux on the A72 MPU subsystem + - The firmware running on the DECODER and ENCODER + +The driver communicates with the firmware running on the ENCODER/DECODER +through its own IPC (inter-processor communication). + +.. ifconfig:: CONFIG_part_variant in ('AM62AX', 'AM62PX', 'J722S', 'AM67') + + For the DECODER, at the highest level in the MPU subsystem on the A53, + there is a Linux user space application which is based on GStreamer. GStreamer + is an open source framework that simplifies the development of multimedia + applications. The GStreamer library loads and interfaces with the GStreamer + plugin (V4L2 plugin), which handles all the details specific to the use of + the hardware accelerator. Specifically, the GStreamer plugin interfaces + with the V4L2 decoder kernel driver interface. + +.. ifconfig:: CONFIG_part_variant in ('J721S2', 'J784S4', 'J742S2') + + For the DECODER, at the highest level in the MPU subsystem on the A72, + there is a Linux user space application which is based on GStreamer. GStreamer + is an open source framework that simplifies the development of multimedia + applications. The GStreamer library loads and interfaces with the GStreamer + plugin (V4L2 plugin), which handles all the details specific to the use of + the hardware accelerator. Specifically, the GStreamer plugin interfaces + with the V4L2 decoder kernel driver interface. + +.. figure:: ../images/MM_Wave5_SW_overview.png + :alt: codec software stack + + CODEC Software Stack + + +Linux Kernel Drivers +==================== + +.. rubric:: TI-Provided V4L2 Drivers for Multimedia + :name: ti-provided-v4l2-drivers-for-multimedia + +Video4Linux version 2 (V4L2) is an open source framework that +provides a media interface to all Linux-based applications. V4L2 is +a collection of device drivers and an API for supporting realtime +video capture and video memory-to-memory operations on Linux systems. + +Video encode and decode using the ENCODER and DECODER hardware, respectively, +are enabled as V4L2 drivers. The V4L2 is integrated with the ENCODER and +DECODER drivers by a thin layer that implements the V4L2 node ioctls +and translates the V4L2 data structures to those understood by the +ENCODER/DECODER. + +The V4L2 drivers that are integrated with ENCODER and DECODER are fully conformant +with the upstream standards. This can be tested as follows: + +.. code-block:: console + + v4l2-compliance -d0 .. code-block:: text + Compliance test for wave5-dec device /dev/video0: + Driver Info: - Driver name : vpu-dec - Card type : vpu-dec - Bus info : platform:vpu-dec - Driver version : 5.14.0 + Driver name : wave5-dec + Card type : wave5-dec + Bus info : platform:4210000.video-codec + Driver version : 6.6.44 Capabilities : 0x84204000 Video Memory-to-Memory Multiplanar Streaming @@ -54,17 +125,21 @@ Decoder : Streaming Extended Pix Format Detected Stateful Decoder + Required ioctls: test VIDIOC_QUERYCAP: OK + test invalid ioctls: OK + Allow for multiple opens: test second /dev/video0 open: OK test VIDIOC_QUERYCAP: OK test VIDIOC_G/S_PRIORITY: OK test for unlimited opens: OK - test invalid ioctls: OK + Debug ioctls: test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported) test VIDIOC_LOG_STATUS: OK (Not Supported) + Input ioctls: test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) @@ -73,6 +148,7 @@ Decoder : test VIDIOC_G/S/ENUMINPUT: OK (Not Supported) test VIDIOC_G/S_AUDIO: OK (Not Supported) Inputs: 0 Audio Inputs: 0 Tuners: 0 + Output ioctls: test VIDIOC_G/S_MODULATOR: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) @@ -80,11 +156,13 @@ Decoder : test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported) test VIDIOC_G/S_AUDOUT: OK (Not Supported) Outputs: 0 Audio Outputs: 0 Modulators: 0 + Input/Output configuration ioctls: test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported) test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported) test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported) test VIDIOC_G/S_EDID: OK (Not Supported) + Control ioctls: test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK test VIDIOC_QUERYCTRL: OK @@ -92,7 +170,8 @@ Decoder : test VIDIOC_G/S/TRY_EXT_CTRLS: OK test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK test VIDIOC_G/S_JPEGCOMP: OK (Not Supported) - Standard Controls: 2 Private Controls: 1 + Standard Controls: 2 Private Controls: 0 + Format ioctls: test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK test VIDIOC_G/S_PARM: OK (Not Supported) @@ -104,20 +183,21 @@ Decoder : test Cropping: OK (Not Supported) test Composing: OK test Scaling: OK (Not Supported) + Codec ioctls: test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported) test VIDIOC_G_ENC_INDEX: OK (Not Supported) test VIDIOC_(TRY_)DECODER_CMD: OK + Buffer ioctls: + warn: v4l2-test-buffers.cpp(693): VIDIOC_CREATE_BUFS not supported + warn: v4l2-test-buffers.cpp(693): VIDIOC_CREATE_BUFS not supported test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK + test CREATE_BUFS maximum buffers: OK test VIDIOC_EXPBUF: OK test Requests: OK (Not Supported) -V4L2 compliance tests report can be generated by following command: - -.. code-block:: console - - v4l2-compliance -d0 + Total for wave5-dec device /dev/video0: 46, Succeeded: 46, Failed: 0, Warnings: 2 Similarly for the encoder, V4L2 compliance tests report can be generated by following command: @@ -126,72 +206,97 @@ Similarly for the encoder, V4L2 compliance tests report can be generated by foll v4l2-compliance -d1 +.. code-block:: text -********************* -Software Architecture -********************* - -Software Stack of Accelerated Codec Encoding/Decoding -===================================================== - -As shown in the figures below, the software stack of the accelerated -encoding and decoding has two parts: - -.. ifconfig:: CONFIG_part_variant in ('AM62AX') - - - A V4L2 (Video4Linux version 2) software driver running on Linux on the A53 MPU subsystem - - The firmware running on the DECODER and ENCODER - -.. ifconfig:: CONFIG_part_variant in ('J721S2') + Compliance test for wave5-enc device /dev/video1: - - A V4L2 (Video4Linux version 2) software driver running on Linux on the A72 MPU subsystem - - The firmware running on the DECODER and ENCODER + Driver Info: + Driver name : wave5-enc + Card type : wave5-enc + Bus info : platform:4210000.video-codec + Driver version : 6.6.44 + Capabilities : 0x84204000 + Video Memory-to-Memory Multiplanar + Streaming + Extended Pix Format + Device Capabilities + Device Caps : 0x04204000 + Video Memory-to-Memory Multiplanar + Streaming + Extended Pix Format + Detected Stateful Encoder -The driver communicates with the firmware running on the ENCODER/DECODER -through its own IPC (inter-processor communication). + Required ioctls: + test VIDIOC_QUERYCAP: OK + test invalid ioctls: OK -.. ifconfig:: CONFIG_part_variant in ('AM62AX') + Allow for multiple opens: + test second /dev/video1 open: OK + test VIDIOC_QUERYCAP: OK + test VIDIOC_G/S_PRIORITY: OK + test for unlimited opens: OK - For the DECODER, at the highest level in the MPU subsystem on the A53, - there is a Linux user space application which is based on GStreamer. GStreamer - is an open source framework that simplifies the development of multimedia - applications. The GStreamer library loads and interfaces with the GStreamer - plugin (V4L2 plugin), which handles all the details specific to the use of - the hardware accelerator. Specifically, the GStreamer plugin interfaces - with the V4L2 decoder kernel driver interface. + Debug ioctls: + test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported) + test VIDIOC_LOG_STATUS: OK (Not Supported) -.. ifconfig:: CONFIG_part_variant in ('J721S2') + Input ioctls: + test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported) + test VIDIOC_G/S_FREQUENCY: OK (Not Supported) + test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported) + test VIDIOC_ENUMAUDIO: OK (Not Supported) + test VIDIOC_G/S/ENUMINPUT: OK (Not Supported) + test VIDIOC_G/S_AUDIO: OK (Not Supported) + Inputs: 0 Audio Inputs: 0 Tuners: 0 - For the DECODER, at the highest level in the MPU subsystem on the A72, - there is a Linux user space application which is based on GStreamer. GStreamer - is an open source framework that simplifies the development of multimedia - applications. The GStreamer library loads and interfaces with the GStreamer - plugin (V4L2 plugin), which handles all the details specific to the use of - the hardware accelerator. Specifically, the GStreamer plugin interfaces - with the V4L2 decoder kernel driver interface. + Output ioctls: + test VIDIOC_G/S_MODULATOR: OK (Not Supported) + test VIDIOC_G/S_FREQUENCY: OK (Not Supported) + test VIDIOC_ENUMAUDOUT: OK (Not Supported) + test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported) + test VIDIOC_G/S_AUDOUT: OK (Not Supported) + Outputs: 0 Audio Outputs: 0 Modulators: 0 -.. figure:: ../images/MM_Wave5_SW_overview.png - :alt: codec software stack + Input/Output configuration ioctls: + test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported) + test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported) + test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported) + test VIDIOC_G/S_EDID: OK (Not Supported) - CODEC Software Stack + Control ioctls: + test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK + test VIDIOC_QUERYCTRL: OK + test VIDIOC_G/S_CTRL: OK + test VIDIOC_G/S/TRY_EXT_CTRLS: OK + test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK + test VIDIOC_G/S_JPEGCOMP: OK (Not Supported) + Standard Controls: 47 Private Controls: 0 + Format ioctls: + test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK + test VIDIOC_G/S_PARM: OK + test VIDIOC_G_FBUF: OK (Not Supported) + test VIDIOC_G_FMT: OK + test VIDIOC_TRY_FMT: OK + test VIDIOC_S_FMT: OK + test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported) + test Cropping: OK + test Composing: OK (Not Supported) + test Scaling: OK (Not Supported) -Linux Kernel Drivers -==================== + Codec ioctls: + test VIDIOC_(TRY_)ENCODER_CMD: OK + test VIDIOC_G_ENC_INDEX: OK (Not Supported) + test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported) -.. rubric:: TI-Provided V4L2 Drivers for Multimedia - :name: ti-provided-v4l2-drivers-for-multimedia + Buffer ioctls: + test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK + test CREATE_BUFS maximum buffers: OK + test VIDIOC_EXPBUF: OK + test Requests: OK (Not Supported) -Video4Linux version 2 (V4L2) is an open source framework that -provides a media interface to all Linux-based applications. V4L2 is -a collection of device drivers and an API for supporting realtime -video capture and video memory-to-memory operations on Linux systems. + Total for wave5-enc device /dev/video1: 46, Succeeded: 46, Failed: 0, Warnings: 0 -Video encode and decode using the ENCODER and DECODER hardware, respectively, -are enabled as V4L2 drivers. The V4L2 is integrated with the ENCODER and -DECODER drivers by a thin layer that implements the V4L2 node ioctls -and translates the V4L2 data structures to those understood by the -ENCODER/DECODER. GStreamer Plugins for Multimedia ================================ @@ -215,7 +320,7 @@ executed on hardware-accelerated IPs like wave5 (DECODER and ENCODER). GStreamer is a multimedia framework based on data flow paradigm. It allows easy plugin registration just by deploying new shared objects to the -/usr/lib/gstreamer-1.0 folder. The shared libraries in this folder are +:file:`/usr/lib/gstreamer-1.0` folder. The shared libraries in this folder are scanned for reserved data structures identifying capabilities of individual plugins. Individual processing nodes can be interconnected as a pipeline at run-time, creating complex topologies. Node interfacing @@ -228,7 +333,7 @@ demuxers, codecs, and filters). New GStreamer features are continuously being added, and the core libraries are actively supported by participants in the GStreamer community. Additional information about the GStreamer framework is available on the GStreamer project site: -http://gstreamer.freedesktop.org/. +https://gstreamer.freedesktop.org/ .. rubric:: Hardware-Accelerated GStreamer Plugins :name: video-decode-gstreamer-plugins @@ -261,12 +366,41 @@ formats: - V4L2_PIX_FMT_H264 - V4L2_PIX_FMT_HEVC +Wave5 encoder/decoder can work on raw data in the following formats: + + - V4L2_PIX_FMT_YUV420 + - V4L2_PIX_FMT_NV12 + - V4L2_PIX_FMT_NV21 + - V4L2_PIX_FMT_YUV420M + - V4L2_PIX_FMT_NV12M + - V4L2_PIX_FMT_NV21M + - V4L2_PIX_FMT_YUV422P + - V4L2_PIX_FMT_NV16 + - V4L2_PIX_FMT_NV61 + - V4L2_PIX_FMT_YUV422M + - V4L2_PIX_FMT_NV16M + - V4L2_PIX_FMT_NV61M + +Encoder also has support for a few packed YUV422 formats. These formats only +apply to the input to encoder - decoder has no support for them. The formats include: + + - V4L2_PIX_FMT_YUYV + - V4L2_PIX_FMT_YVYU + - V4L2_PIX_FMT_UYVY + +.. note:: + + Raw data means input for the encoder and output for the decoder. For example, when + encoding, your incoming camera feed can be in any of the previously mentioned formats + and work with the encoder without any color conversion elements. Decoder can take an + NV12 H264/H265 encoded stream and output raw data to display in any of the formats + mentioned above assuming the display has support. ******************************** Encoder and Decoder Capabilities ******************************** -.. ifconfig:: CONFIG_part_variant in ('J721S2') +.. ifconfig:: CONFIG_part_variant in ('J721S2', 'AM62PX', 'J722S') The Max Capability of the Encoder/Decoder is 4K60fps equivalent load. @@ -311,8 +445,6 @@ The external controls supported by Encoder and Decoder can be seen using below c Decoder: v4l2-ctl -d 0 -l - - ******************* GStreamer Pipelines ******************* @@ -335,13 +467,13 @@ Decode from raw file: .. code-block:: console - target # gst-launch-1.0 filesrc location=/ ! h264parse ! queue ! v4l2h264dec ! filesink location=/ + target # gst-launch-1.0 filesrc location=/ ! h264parse ! queue ! v4l2h264dec ! video/x-raw,format='(string)'I420 ! videocodectestsink location=/ - H.265: .. code-block:: console - target # gst-launch-1.0 filesrc location=/ ! h265parse ! queue ! v4l2h265dec ! filesink location=/ + target # gst-launch-1.0 filesrc location=/ ! h265parse ! queue ! v4l2h265dec ! video/x-raw,format='(string)'I420 ! videocodectestsink location=/ Video only file playback: @@ -382,39 +514,253 @@ Video Streaming use-case: target # gst-launch-1.0 filesrc location=sample_1072.yuv blocksize=3087360 ! rawvideoparse width=1920 height=1072 framerate=30/1 format=nv12 colorimetry=bt709 ! v4l2h264enc ! h264parse ! fakesink +***************************** +FFmpeg Plugins for Multimedia +***************************** + +:command:`ffmpeg` is an open-source multimedia framework. This is useful for decoding, encoding and transcoding videos. This is a command line tool. + +The library comes with :command:`ffplay` and :command:`ffprobe`. + +:command:`ffplay` is a very simple and portable media player using the :command:`ffmpeg` libraries and the SDL library. It is mostly used as a testbed for the various :command:`ffmpeg` APIs. + +:command:`ffprobe` gathers information from multimedia streams and prints it in human and machine-readable fashion. + +For example, it can be used to check the format of the container used and the format and type of each media stream contained in it. + +To enable :command:`ffplay`, the following needs to be added to :file:`local.conf`: + +.. code-block:: text + + PACKAGECONFIG:append:pn-ffmpeg = " sdl2" + +More information about the :command:`ffmpeg` tools can be found here: + - https://ffmpeg.org/ffmpeg.html + - https://ffmpeg.org/ffplay.html + - https://ffmpeg.org/ffprobe.html + +ffmpeg +====== + +Help Flags +---------- +``-h`` or ``-?`` or ``-help`` + Topic show help +``-codecs`` + Displays all codecs supported +``-decoders`` + Displays all decoders supported +``-formats`` + Displays all available formats +``-pix_fmts`` + Displays all available pixel formats + +``-codec:v`` + Specifies the stream decoder. For decoding, we only ``hevc_v4l2m2m`` and ``h264_v4l2m2m``. :v stands for video + +Stream Specific Flags +--------------------- +``-re`` + Reads input at native frame rate +``-i [filename]`` + Input stream +``-fps_mode passthrough`` + Allows for frames to pass even if out of sync. Needed since there are no timestamps in decoded h264/5 streams that use the hardware GPU to decode. Must be placed in between input and output stream +``-pix_fmt`` + Specifies the pixel format for the output stream. NV12 is recommended for wave5 +``-f`` + Forces format of output stream +``-frames`` + Set the number of frames to record + +ffplay +====== +``-framerate`` + Specifies framerate of display stream +``-video_size`` + Specifies video size of video. It may be the string of the form or *widthxheigh*. For example, 2k or 2048x1080 are equivalent. +``-pixel_format`` + Specifies pixel format of display stream +``-f`` + Forces format of display stream +``-fs`` + Start in fullscreen mode +``-autoexit`` + Exits ffplay after stream is finished + +ffprobe +======= +``-show_data`` + Show payload data, as a hexadecimal and ASCII dump. Coupled with -show_packets, it will dump the packets’ data. Coupled with -show_streams, it will dump the codec extradata + The dump is printed as the "data" field. It may contain newlines +``-show_packets`` + Show information about each packet contained in the input multimedia stream + The information for each single packet is printed within a dedicated section with name "PACKET" +``-show_frames`` + Show information about each frame and subtitle contained in the input multimedia stream + The information for each single frame is printed within a dedicated section with name "FRAME" or "SUBTITLE" +``-show_format`` + Show information about the container format of the input multimedia stream + All the container format information is printed within a section with name "FORMAT" -****************** -Memory Requirement -****************** -.. ifconfig:: CONFIG_part_variant in ('J721S2', 'AM62AX') +Example Commands +================ + +To convert a HEVC file to a raw video with pixel format NV12 named :file:`foo.yuv` with software acceleration, the following can be used: + +.. code-block:: console + + ffmpeg -i foo.265 -f rawvideo -pix_fmt nv12 foo.yuv + +To convert a H264 file to a raw video named :file:`bar.yuv` with hardware acceleration and each frame passed, the following can be used: + +.. code-block:: console + + ffmpeg -codec:v h264_v4l2m2m -i bar.h264 -f rawvideo -fps_mode passthrough bar.yuv + +To play a H264 video and output to the display at fullscreen and exit once the video is done playing, the following can be used: + +.. code-block:: console + + ffplay -fs -autoexit foobar.264 - The following calculations are taken for 1080p single channel 30fps stream using vmstat. +To pipe the output from ffmpeg to ffplay with hardware acceleration, the following can be used: - - Encoder - #. v4l2h264enc : 31.78 MB - #. v4l2h265enc : 31.90 MB - - Decoder - #. v4l2h264dec : 51.47 MB - #. v4l2h265dec : 39.59 MB +.. code-block:: console + + ffmpeg -re -codec:v hevc_v4l2m2m -i input.h265 -fps_mode passthrough -f rawvideo - | ffplay -framerate 30 -video_size 1920x1080 -f rawvideo -autoexit - + +To print out the packet's data and payload data in hexadecimal from a given multimedia stream named :file:`foo.h265`, the following can be used: + +.. code-block:: console + + ffprobe -show_data -show_packets foo.h265 + +To print information about each frame and/or subtitle in the given multimedia stream named :file:`bar.h264`, the following can be used: + +.. code-block:: console + + ffprobe -show_frames bar.h264 + +************************** +MPV Plugins for Multimedia +************************** + +mpv is a media player based on MPlayer and mplayer2. It supports a wide variety of video file formats, audio and video codecs, and subtitle types. +Special input URL types are available to read input from a variety of sources other than disk files. +Depending on platform, a variety of different video and audio output methods are supported. + +This is a command line tool that can also use keyboard shortcuts. + +https://mpv.io/manual/stable/ + +:command:`yt-dlp` can be combined with mpv to play videos from the web. (YouTube, TikTok, etc.) + +To install :command:`yt-dlp` run the following command + +.. code-block:: console + + python3 -m pip install yt-dlp + + +Useful Flags +============ + +``--vo=[driver]`` + +Specify the video output backend to be used. See `VIDEO OUTPUT DRIVERS `_ for details and descriptions of available drivers. + +``--hwdec=[api1,api2,...|no|auto|auto-safe|auto-copy]`` + +Specify the hardware video decoding API that should be used if possible. Whether hardware decoding is actually done depends on the video codec. If hardware decoding is not possible, mpv will fall back on software decoding. +Hardware decoding is not enabled by default, to keep the out-of-the-box configuration as reliable as possible. However, when using modern hardware, hardware video decoding should work correctly, offering reduced CPU usage, and possibly lower power consumption. + +``--gpu-context=[sys]`` + +The value auto (the default) selects the GPU context. You can also pass help to get a complete list of compiled in backends (sorted by autoprobe order). + +``--fs`` + +Fullscreen playback. + +``--autofit=<[W[xH]]>`` + +Set the initial window size to a maximum size specified by WxH, without changing the window's aspect ratio. The size is measured in pixels, or if a number is followed by a percentage sign (%), in percents of the screen size. +This option never changes the aspect ratio of the window. If the aspect ratio mismatches, the window's size is reduced until it fits into the specified size. + +``--no-correct-pts`` + +Switches mpv to a mode where video timing is determined using a fixed framerate value (either using the --container-fps-override option, or using file information). Sometimes, files with very broken timestamps can be played somewhat well in this mode. + +``--fps=[framerate]`` + +Set framerate of output stream. Can be combined with --no-correct-pts for streams with incorrect/no timestamps + + +Example Commands +================ - .. note:: +To play a H264 video to display at 30 frames per seconds, the following can be used: - The Actual Memory foot print may vary depending on the input stream. +.. code-block:: console + + mpv --no-correct-pts --fps=30 foo.h264 + +To play a HEVC video with full/assisted hardware acceleration at 30 frames per second, the following can be used: + +.. code-block:: console -.. ifconfig:: CONFIG_part_variant not in ('J721S2', 'AM62AX') + mpv --vo=gpu --hwdec=auto --no-correct-pts --fps=30 bar.h265 - Please refer to the Performance guide. +To play a mp4 video to display in fullscreen and set the gpu context to wayland, the following can be used: + +.. code-block:: console + + mpv --vo=gpu --gpu-context=wayland --fs foobar.mp4 + +To download a video from a link and pipe the video to mpv with no correct points at 30 frames per seconds, the following can be used: + +.. code-block:: console + + yt-dlp -o - | mpv --no-correct-pts --fps=30 - + +Tp download a video from a link at 1080p and play the video with mpv, the following can be used: + +.. code-block:: console + + yt-dlp -f 270 -o - | mpv - + +``-f 232`` can be used to get a 720p video. + +In instances where running yt-dlp doesn't work because of a protocol issue, updating to the newest version of yt-dlp or changing +the format/quality by running ``-f 270`` or ``-f 232`` for example, or playing other videos might solve the issue. + + +****************** +Memory Requirement +****************** + +The following calculations are taken for 1080p single channel 30fps stream using vmstat. + + - Encoder + #. v4l2h264enc : 31.78 MB + #. v4l2h265enc : 31.90 MB + - Decoder + #. v4l2h264dec : 51.47 MB + #. v4l2h265dec : 39.59 MB + +.. note:: + + The Actual Memory foot print may vary depending on the input stream. ******************* Performance metrics ******************* -.. ifconfig:: CONFIG_part_variant in ('J721S2', 'AM62AX') - - The following figures illustrate the theoretical latency of the IP with - regards to different resolutions. +The following figures illustrate the theoretical latency of the IP with +regards to different resolutions. +---------------+------------+---------+ | CodecH265/264 | Resolution | Latency | @@ -428,10 +774,6 @@ Performance metrics | | 480p | 1.2 ms | +---------------+------------+---------+ -.. ifconfig:: CONFIG_part_variant not in ('J721S2', 'AM62AX') - - Please refer to the Performance guide. - ********************************************************** Calculation of Performance metrics using native driver API @@ -446,7 +788,9 @@ Please refer the below source code. static void wave5_vpu_dec_finish_decode(struct vpu_instance *inst) { ... - dev_dbg(inst->dev->dev, "frame_cycle %8d\n", dec_output_info.frame_cycle); + dev_dbg(inst->dev->dev, "%s: frame_cycle %8u (payload %lu)\n", + __func__, dec_info.frame_cycle, + vb2_get_plane_payload(&disp_buf->vb2_buf, 0)); ... } @@ -456,7 +800,8 @@ Please refer the below source code. static void wave5_vpu_enc_finish_encode(struct vpu_instance *inst) { ... - dev_dbg(inst->dev->dev, "frame_cycle %8d\n", enc_output_info.frame_cycle); + dev_dbg(inst->dev->dev, "%s: frame_cycle %8u\n", + __func__, enc_output_info.frame_cycle); ... } @@ -533,7 +878,7 @@ Measuring decoder latency in client pipeline: .. code-block:: console - target # GST_TRACERS="latency(flags=pipeline+element)" GST_DEBUG_FILE="/run/latency_client.txt" gst-launch-1.0 -v udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtpjitterbuffer latency=50 ! rtph264depay ! h264parse ! v4l2h264dec capture-io-mode=dmabuf ! queue ! fpsdisplaysink text-overlay=false name=fpssink video-sink="kmssink driver-name=tidss sync=true show-preroll-frame=false" sync=true -v > /run/client.txt 2>&1& + target # GST_TRACERS="latency(flags=pipeline+element)" GST_DEBUG_FILE="/run/latency_client.txt" gst-launch-1.0 -v udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtpjitterbuffer latency=50 ! rtph264depay ! h264parse ! v4l2h264dec capture-io-mode=dmabuf ! kmssink Instantaneous decoder latency (ns): @@ -630,27 +975,43 @@ Buffer import on decoder can be tested by setting the ``capture-io-mode`` to ``5 Configuration of CMA Size ************************* -The CMA size can be increased or decreased depending on the requirement and the memory map usage by other components. +Wave5 uses common CMA region to read and write buffers to, since the IP does not posses an on-chip MMU. .. ifconfig:: CONFIG_part_variant in ('J721S2') - The macro that specifies the CMA size is CONFIG_CMA_SIZE_MBYTES present in the file :file:`arch/arm64/configs/tisdk_j721s2-evm_defconfig` in the linux directory of sdk. The default value is 896MB. + The CMA size can be adjusted by opening the board dtsi file :file:`arch/arm64/boot/dts/ti/k3-j721s2-som-p0.dtsi`, + finding the "linux,cma" node and changing the size attribute. The default value is 896MB. .. ifconfig:: CONFIG_part_variant in ('AM62AX') - The macro that specifies the CMA size is CONFIG_CMA_SIZE_MBYTES present in the file :file:`arch/arm64/configs/tisdk_am62axx-evm_defconfig` in the linux directory of sdk. The default value is 576MB. + The CMA size can be adjusted by opening the board dts file :file:`arch/arm64/boot/dts/ti/k3-am62a7-sk.dts`, + finding the "linux,cma" node and changing the size attribute. The default value is 576MB. -.. ifconfig:: CONFIG_part_variant in ('J784S4','J742S2') +.. ifconfig:: CONFIG_part_variant in ('J784S4', 'J742S2') + + The CMA size can be adjusted by opening the board dts file :file:`arch/arm64/boot/dts/ti/k3-j784s4-evm.dts`, + finding the "linux,cma" node and changing the size attribute. The default value is 1792MB. - The macro that specifies the CMA size is CONFIG_CMA_SIZE_MBYTES present in the file :file:`arch/arm64/configs/tisdk_am62axx-evm_defconfig` in the linux directory of sdk. The default value is 1792MB. +.. ifconfig:: CONFIG_part_variant in ('J722S') + The CMA size can be adjusted by opening the board dts file :file:`arch/arm64/boot/dts/ti/k3-j722s-evm.dts`, + finding the "linux,cma" node and changing the size attribute. The default value is 896MB. + +.. ifconfig:: CONFIG_part_variant in ('AM62PX') + + The CMA size can be adjusted by opening the board dts file :file:`arch/arm64/boot/dts/ti/k3-am62p5-sk.dts`, + finding the "linux,cma" node and changing the size attribute. The default value is 576MB. The value can be increased according to the availability of space in DDR memory map. -Also to change cma without re-compilation, one can stop at u-boot prompt during bootup and update cma as below and then boot: +CMA allocation can also be changed without re-compilation. One can stop at u-boot prompt during boot and update the CMA parameter as shown below: .. code-block:: console target # setenv args_all $args_all cma=1000M target # boot -:doc:`Other Multimedia Tools <./Foundational_Components_Multimedia_Other_Tools>` \ No newline at end of file +.. note:: + + Allocating at u-boot tends to place the CMA in the lower 32 bit region of memory. This is fine if the Wave5 is being + used in isolated pipelines where the only goal is to encode/decode. If pipelines that incorporate other hardware elements + are being used, then the correct method would be to allocate at device tree level for compatibility.