drivers: video: video_mcux_smartdma: add SMARTDMA video driver #72827

danieldegrasse · 2024-05-15T22:55:58Z

Add SMARTDMA video driver. This driver uses the SMARTDMA engine as a
parallel camera interface, which can read QVGA frames from a camera
device. Due to SRAM constraints, the video driver divides the camera
stream into multiple horizontal video buffers as it streams them back to
an application.

Note that this PR also contains two additional major changes:

an addition to the video API to enable streaming partial frames within one video buffer
a rework of the SMARTDMA driver interface to better support custom firmware use cases like the camera in this PR

~~This PR depends on #72826 for camera support~~

PR has been tested on the following boards:

frdm_mcxn947//cpu0: samples/subsys/video/capture
mimxrt595_evk//cm33: samples/subsys/display/lvgl (using G1120B0MIPI shield)

zephyrbot · 2024-05-15T23:05:18Z

The following west manifest projects have been modified in this Pull Request:

Name	Old Revision	New Revision	Diff

Note: This message is automatically posted and updated by the Manifest GitHub Action.

josuah · 2024-05-20T07:03:40Z

an addition to the video API to enable streaming partial frames within one video buffer

Some news about it there, with a proposed API: #66994 (comment)

Introduce driver for ov7670 camera, supporting QCIF,QVGA,CIF, and VGA resolution in YUV and RGB mode. Support was verified on the FRDM-MCXN947, using the SmartDMA camera engine, which is enabled in the following PR: zephyrproject-rtos#72827 Signed-off-by: Daniel DeGrasse <[email protected]>

josuah · 2024-05-23T13:01:06Z

here (need to click on "load diff") is a new API that seems underway:

	if (drv_data->fifo_length == 0) {
		vbuf->flags = VIDEO_BUF_EOF;
	} else {
		vbuf->flags = VIDEO_BUF_FRAG;
		k_work_submit_to_queue(&ac_work_q, &drv_data->buf_work);
	}

danieldegrasse · 2024-05-24T15:15:19Z

here (need to click on "load diff") is a new API that seems underway:

No issues with this API, I suppose we would like to make this PR dependent on that API merging? I could also cherry pick the commits adding the API from that PR (@ArduCAM would you be ok with that approach?)

ArduCAM · 2024-05-25T02:46:21Z

here (need to click on "load diff") is a new API that seems underway:

No issues with this API, I suppose we would like to make this PR dependent on that API merging? I could also cherry pick the commits adding the API from that PR (@ArduCAM would you be ok with that approach?)

Yes, have marked resolved

Introduce driver for ov7670 camera, supporting QCIF,QVGA,CIF, and VGA resolution in YUV and RGB mode. Support was verified on the FRDM-MCXN947, using the SmartDMA camera engine, which is enabled in the following PR: zephyrproject-rtos#72827 Signed-off-by: Daniel DeGrasse <[email protected]>

Introduce driver for ov7670 camera, supporting QCIF,QVGA,CIF, and VGA resolution in YUV and RGB mode. Support was verified on the FRDM-MCXN947, using the SmartDMA camera engine, which is enabled in the following PR: #72827 Signed-off-by: Daniel DeGrasse <[email protected]>

Introduce driver for ov7670 camera, supporting QCIF,QVGA,CIF, and VGA resolution in YUV and RGB mode. Support was verified on the FRDM-MCXN947, using the SmartDMA camera engine, which is enabled in the following PR: zephyrproject-rtos/zephyr#72827 (cherry picked from commit 3ce3ed3) Original-Signed-off-by: Daniel DeGrasse <[email protected]> GitOrigin-RevId: 3ce3ed3 Change-Id: I20f10c984e4a0c2a6c805fde3625c0b6310f4fe4 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/zephyr/+/5607905 Tested-by: ChromeOS Prod (Robot) <[email protected]> Reviewed-by: Fabio Baltieri <[email protected]> Commit-Queue: Fabio Baltieri <[email protected]>

loicpoulain

Looks good, is it ok to merge @danieldegrasse ?

josuah

It is true that many video processing cores have a line buffer, or a 5-line buffers (i.e. small kernels) rather than an XYZ bytes buffers.

A line-based min/max size sounds good.

danieldegrasse · 2024-10-11T21:32:12Z

Looks good, is it ok to merge @danieldegrasse ?

Yes, provided you are ok with the API changes. Let me rebase to fix conflicts.

Add min and max line count to the video_caps structure- these fields can be used by applications to determine the size of video buffer they need to allocate for a video endpoint The min and max line count fields are designed to enable supporting endpoints that may produce or consume partial frames within each video buffer, and may support arbitrarily sized video buffers. Signed-off-by: Daniel DeGrasse <[email protected]>

Update existing video drivers to handle the min/max line count field within the video_caps structure. All drivers work with full frames currently, so use the special LINE_COUNT_HEIGHT value to indicate this. Signed-off-by: Daniel DeGrasse <[email protected]>

Update video samples to use min/max line count capabilities when allocating video buffers. Signed-off-by: Daniel DeGrasse <[email protected]>

Add line_offset field to the video_buffer structure. This field indicates the offset (in horizontal lines) within a frame that a video buffer starts at. This is useful for video devices that produce or consume video buffers that constitute a partial frame. Signed-off-by: Daniel DeGrasse <[email protected]>

Since all video drivers in tree use a full frame, their video buffers will always start at a line_offset of 0 within the frame. Signed-off-by: Daniel DeGrasse <[email protected]>

Handle line_offset field within video capture sample. Other samples do not support partial framebuffers and therefore can ignore this field. Signed-off-by: Daniel DeGrasse <[email protected]>

The SMARTDMA is a programmable DMA engine, and supports custom firmware in order to run complex DMA operations. Update the driver to increase the flexibility users have when configuring the SMARTDMA with custom firmware, and remove the RT500 display firmware specific definitions and functionality from the driver. This display setup is now handled from the MIPI DSI driver, since the firmware used for this case is specific to the MIPI DSI IP. This change also requires an update to the RT500 devicetree, as the register definition for the SMARTDMA has changed, so the base address must as well. Signed-off-by: Daniel DeGrasse <[email protected]>

Improve resolution init code for OV7670 driver, to properly program the full set of registers needed to realize a given output resolution. The settings for these registers are based on those used in the MCUX SDK driver, which are derived from the resolution register settings given in the OV7670 programming guide, with a different configuration for the CLKSRC and DBLV registers used for the input clock. Signed-off-by: Daniel DeGrasse <[email protected]>

Add SMARTDMA video driver. This driver uses the SMARTDMA engine as a parallel camera interface, which can read QVGA frames from a camera device. Due to SRAM constraints, the video driver divides the camera stream into multiple horizontal video buffers as it streams them back to an application. Signed-off-by: Daniel DeGrasse <[email protected]>

Add definition for SMARTDMA device to the MCXN94x devicetree. Signed-off-by: Daniel DeGrasse <[email protected]>

Add support for LPI2C7. This peripheral is used for interfacing with cameras connected to the J9 header. Signed-off-by: Daniel DeGrasse <[email protected]>

Add support for SMARTDMA camera engine, and a OV7670 parallel camera definition for the frdm_mcnx947 board. This support has been tested with the video capture sample. Signed-off-by: Daniel DeGrasse <[email protected]>

Add support for using the SMARTDMA engine on the FRDM-MCXN947 board with the video capture sample. Signed-off-by: Daniel DeGrasse <[email protected]>

Do not apply format setting unless needed. Also, correct the check for the RGB565 format setting- the zephyr display API treats RGB565 and BGR565 as big endian, so the format needed here is BGR565. Signed-off-by: Daniel DeGrasse <[email protected]>

josuah

Thank you for adjusting all the other drivers too!

danieldegrasse force-pushed the feature/sdma-camera branch from 96d24ea to ecdd67f Compare May 15, 2024 23:04

zephyrbot added manifest manifest-hal_nxp DNM This PR should not be merged (Do Not Merge) labels May 15, 2024

danieldegrasse force-pushed the feature/sdma-camera branch from ecdd67f to 7755b6a Compare May 16, 2024 14:41

josuah mentioned this pull request May 17, 2024

drivers: video: API enhancement #72959

Open

danieldegrasse mentioned this pull request May 20, 2024

drivers: video: ov7670: introduce driver for ov7670 camera #72826

Merged

danieldegrasse mentioned this pull request Jun 1, 2024

drivers: video: Add support for arducam mega #66994

Open

danieldegrasse force-pushed the feature/sdma-camera branch from 7755b6a to 5028998 Compare June 3, 2024 17:35

danieldegrasse force-pushed the feature/sdma-camera branch from 5028998 to bb59e5e Compare June 7, 2024 18:58

danieldegrasse marked this pull request as ready for review June 7, 2024 18:58

danieldegrasse requested review from mmahadevan108, dleach02, erwango, ABOSTM and FRASTM as code owners June 7, 2024 18:58

zephyrbot added platform: NXP Drivers NXP Semiconductors, drivers area: MIPI-DSI platform: STM32 ST Micro STM32 area: DMA Direct Memory Access labels Jun 7, 2024

danieldegrasse force-pushed the feature/sdma-camera branch 4 times, most recently from 9bf9e5d to 765cc42 Compare October 8, 2024 20:54

loicpoulain previously approved these changes Oct 11, 2024

View reviewed changes

josuah previously approved these changes Oct 11, 2024

View reviewed changes

danieldegrasse dismissed stale reviews from josuah and loicpoulain via d230daa October 11, 2024 21:34

danieldegrasse force-pushed the feature/sdma-camera branch from 765cc42 to d230daa Compare October 11, 2024 21:34

danieldegrasse added 14 commits October 11, 2024 17:34

samples: drivers: video: update video samples for min/max line count

a3457f1

Update video samples to use min/max line count capabilities when allocating video buffers. Signed-off-by: Daniel DeGrasse <[email protected]>

drivers: video: handle line_offset field

8d91670

Since all video drivers in tree use a full frame, their video buffers will always start at a line_offset of 0 within the frame. Signed-off-by: Daniel DeGrasse <[email protected]>

samples: drivers: video: handle line_offset field

b0fd716

Handle line_offset field within video capture sample. Other samples do not support partial framebuffers and therefore can ignore this field. Signed-off-by: Daniel DeGrasse <[email protected]>

dts: arm: nxp: nxp_mcxn94x: add definition for SMARTDMA device

73b0ba9

Add definition for SMARTDMA device to the MCXN94x devicetree. Signed-off-by: Daniel DeGrasse <[email protected]>

boards: nxp: frdm_mcn947: add support for LPI2C7

6f38321

Add support for LPI2C7. This peripheral is used for interfacing with cameras connected to the J9 header. Signed-off-by: Daniel DeGrasse <[email protected]>

boards: nxp: frdm_mcnx947: add support for SMARTDMA camera engine

788b5e3

Add support for SMARTDMA camera engine, and a OV7670 parallel camera definition for the frdm_mcnx947 board. This support has been tested with the video capture sample. Signed-off-by: Daniel DeGrasse <[email protected]>

samples: video: capture: add support for SMARTDMA

40fa8f9

Add support for using the SMARTDMA engine on the FRDM-MCXN947 board with the video capture sample. Signed-off-by: Daniel DeGrasse <[email protected]>

danieldegrasse force-pushed the feature/sdma-camera branch from d230daa to 7763056 Compare October 11, 2024 22:35

josuah approved these changes Oct 12, 2024

View reviewed changes

decsny requested a review from loicpoulain October 12, 2024 20:10

dleach02 approved these changes Oct 14, 2024

View reviewed changes

nashif merged commit 20cce99 into zephyrproject-rtos:main Oct 15, 2024
26 checks passed

josuah mentioned this pull request Nov 5, 2024

doc: release: 4.0: Add video driver and video API updates #80945

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

drivers: video: video_mcux_smartdma: add SMARTDMA video driver #72827

drivers: video: video_mcux_smartdma: add SMARTDMA video driver #72827

danieldegrasse commented May 15, 2024 •

edited

Loading

zephyrbot commented May 15, 2024 •

edited

Loading

josuah commented May 20, 2024

josuah commented May 23, 2024

danieldegrasse commented May 24, 2024

ArduCAM commented May 25, 2024

loicpoulain left a comment

josuah left a comment •

edited

Loading

danieldegrasse commented Oct 11, 2024

josuah left a comment

drivers: video: video_mcux_smartdma: add SMARTDMA video driver #72827

drivers: video: video_mcux_smartdma: add SMARTDMA video driver #72827

Conversation

danieldegrasse commented May 15, 2024 • edited Loading

zephyrbot commented May 15, 2024 • edited Loading

josuah commented May 20, 2024

josuah commented May 23, 2024

danieldegrasse commented May 24, 2024

ArduCAM commented May 25, 2024

loicpoulain left a comment

Choose a reason for hiding this comment

josuah left a comment • edited Loading

Choose a reason for hiding this comment

danieldegrasse commented Oct 11, 2024

josuah left a comment

Choose a reason for hiding this comment

danieldegrasse commented May 15, 2024 •

edited

Loading

zephyrbot commented May 15, 2024 •

edited

Loading

josuah left a comment •

edited

Loading