Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve NXP CSI and MIPI_CSI2Rx drivers #76475

Conversation

ngphibang
Copy link
Contributor

@ngphibang ngphibang commented Jul 30, 2024

This PR improves the NXP's csi and mipi_csi2_rx drivers including

Fixes #80304

This PR depends on #74415 and #72254, also relates to #76144

@josuah josuah added priority: low Low impact/importance bug platform: NXP NXP area: Video Video subsystem platform: NXP Drivers NXP Semiconductors, drivers labels Aug 3, 2024
@ngphibang ngphibang marked this pull request as ready for review August 20, 2024 09:33
@ngphibang ngphibang force-pushed the improve_nxp_csi_mipicsi2rx_drivers branch from 2f6a738 to ece0adc Compare August 21, 2024 15:24
@zephyrbot zephyrbot added the area: Shields Shields (add-on boards) label Aug 21, 2024
@ngphibang ngphibang force-pushed the improve_nxp_csi_mipicsi2rx_drivers branch 2 times, most recently from 6aee5b6 to 23af338 Compare August 21, 2024 19:39
@trunghieulenxp trunghieulenxp force-pushed the improve_nxp_csi_mipicsi2rx_drivers branch from 23af338 to dcaf23c Compare August 28, 2024 09:48
@trunghieulenxp
Copy link
Contributor

trunghieulenxp commented Aug 28, 2024

Rebase because the PR #73009 is merged.

@trunghieulenxp trunghieulenxp force-pushed the improve_nxp_csi_mipicsi2rx_drivers branch from dcaf23c to 406703f Compare August 29, 2024 14:50
ngphibang and others added 14 commits October 24, 2024 12:36
Switch to use the new video interfaces bindings

Signed-off-by: Phi Bang Nguyen <[email protected]>
Switch to use the new video interfaces bindings

Signed-off-by: Phi Bang Nguyen <[email protected]>
The ov5640 camera driver now supports both MIPI CSI2 (DPHY) and DVP
modes. It is in MIPI CSI2 mode in this overlay. Add bus-type property
for this. In this mode, data-lanes property is required as well.

Signed-off-by: Phi Bang Nguyen <[email protected]>
Get number of data lanes from device tree instead of hard-coding it.

Signed-off-by: Phi Bang Nguyen <[email protected]>
The peer remote device "sensor_dev" can be retrieved from the
remote-endpoint-label. Direct reference via phandle is not needed.

Signed-off-by: Phi Bang Nguyen <[email protected]>
Add set_ctrl callback to propagate controls to the sensor.

Signed-off-by: Farah Fliss <[email protected]>
Signed-off-by: Phi Bang Nguyen <[email protected]>
Instead of fixing csi2rx clock frequencies, set them according to the
pixel rate got from the camera sensor.

Signed-off-by: Trung Hieu Le <[email protected]>
Signed-off-by: Phi Bang Nguyen <[email protected]>
Add support for changing frame rate.

Signed-off-by: Trung Hieu Le <[email protected]>
Signed-off-by: Phi Bang Nguyen <[email protected]>
As getting bytes per pixel of a pixel format is a very common operation,
add an utility function for it instead of repeating the same codes in
different drivers.

Signed-off-by: Phi Bang Nguyen <[email protected]>
Add support for changing frame rate

Signed-off-by: Trung Hieu Le <[email protected]>
Signed-off-by: Phi Bang Nguyen <[email protected]>
Remove the obsolete comment about the init order of the CSI and the
camera sensors (e.g. mt9m114) which is not true anymore.

Signed-off-by: Phi Bang Nguyen <[email protected]>
The peer remote device "source_dev" can be retrieved from the
remote-endpoint-label. Direct reference via phandle is not needed.

Signed-off-by: Phi Bang Nguyen <[email protected]>
The CSI needs to be initialized BEFORE the camera sensor to provide
clock to the camera sensor. It is now possible to do so as direct
reference to the sensor via phandle is now removed. There will be
no check failure on the init order anymore when compiling.

Signed-off-by: Phi Bang Nguyen <[email protected]>
The CSI is an NXP IP and the driver has been very much changed
by NXP, so add NXP copyright to it.

Signed-off-by: Phi Bang Nguyen <[email protected]>
@ngphibang ngphibang force-pushed the improve_nxp_csi_mipicsi2rx_drivers branch from c89858b to d07370c Compare October 24, 2024 10:42
@ngphibang
Copy link
Contributor Author

@danieldegrasse Regarding the commit "drivers: video: csi: Fix start / stop issue", your points are almost right. To summerize:

  • After stopping, users are expected to re-enqueue the buffers before restart streaming
  • However, when stopping, the SDK CSI driver puts all the active buffers into the input (empty) queue with expect that user can start again right away as can be seen here
  • While the Zephyr CSI driver puts all the buffers in the input queue into the output queue as can be seen here
  • So, to fix the issue with these conflict behaviors, I think we need to call CSI_STOP() and disable interrupts directly without passing by CSI_TransferStop() as you said. The input and output queues in the Zephyr CSI driver need to be clean too.

Given the strict timeline approaching and to facilitate the review, we removed this commit from the PR and wait for next time to submit it.

@danieldegrasse
Copy link
Collaborator

@danieldegrasse Regarding the commit "drivers: video: csi: Fix start / stop issue", your points are almost right. To summerize:

* After stopping, users are expected to re-enqueue the buffers before restart streaming

* However, when stopping, the SDK CSI driver puts all the active buffers into the input (empty) queue with expect that user can start again right away as can be seen [here](https://github.com/zephyrproject-rtos/hal_nxp/blob/99570ee6fa32a9ab413b8ee9c0226cda26f6a2ae/mcux/mcux-sdk/drivers/csi/fsl_csi.c#L705)

* While the Zephyr CSI driver puts all the buffers in the input queue into the output queue as can be seen [here](https://github.com/zephyrproject-rtos/zephyr/blob/aefda52e42b599592dcfbcbd359aa8dd6e908994/drivers/video/video_mcux_csi.c#L271)

* So, to fix the issue with these conflict behaviors, I think we need to call CSI_STOP() and disable interrupts directly without passing by CSI_TransferStop() as you said. The input and output queues in the Zephyr CSI driver need to be clean too.

Given the strict timeline approaching and to facilitate the review, we removed this commit from the PR and wait for next time to submit it.

Reading the video driver documentation, I think the behavior for video_mcux_csi_flush is correct- if the user opts to cancel buffer processing, then queued buffers are supposed to be moved through the queue without being processed (like the implementation does)

There isn't clear documentation on how video_stream_stop is supposed to function regarding input buffers. We could either:

  • discard all buffers in the input queue, and require user to re-queue them
  • keep buffers queued in the input queue, and when the stream is restarted use those buffers

@loicpoulain do you know which behavior is correct?

I do not think we should flush or modify the output buffer queue when the stream is stopped.

@danieldegrasse danieldegrasse dismissed their stale review October 24, 2024 14:59

Request addressed

@ngphibang
Copy link
Contributor Author

ngphibang commented Oct 24, 2024

keep buffers queued in the input queue, and when the stream is restarted use those buffers

=> The CSI needs at least 2 buffers to be able to start. Remaining buffers in the input queue may be insufficient. That's why users need to re-queue buffer (option 1).

I do not think we should flush or modify the output buffer queue when the stream is stopped.

=> If users re-queue buffers (of course they will not allocate new buffers but re-use the ones in the buffer pool). We need to clean the output queue unless buffers will be present on both input and output queues.

@ngphibang ngphibang requested a review from josuah October 24, 2024 15:12
@danieldegrasse
Copy link
Collaborator

=> The CSI needs at least 2 buffers to be able to start. Remaining buffers in the input queue may be insufficient. That's why users need to re-queue buffer (option 1).

Sorry, I think I might have been unclear. My understanding is that user might interact with the video API like so:

  • Queue buffer A
  • Queue buffer B
  • Queue buffer C
  • start CSI stream
  • dequeue processed buffer A
  • stop CSI stream

At this point, if the CSI has processed buffer B we should expect that buffer to be in the output queue, and if the user tries to dequeue a video buffer they will get buffer B.

Video buffer C is where I am not sure. I think that the buffer should remain in the input queue, so that if the user then does something like so:

  • Queue buffer A
  • Queue buffer B
  • dequeue buffer

They would receive buffer C. However, I'm not sure if this is correct, or if we should be clearing the input queue so that the user now would receive buffer A when they run the dequeue operation.

Does this help clarify my question here?

@dleach02 dleach02 added this to the v4.0.0 milestone Oct 24, 2024
@henrikbrixandersen henrikbrixandersen merged commit 5e41249 into zephyrproject-rtos:main Oct 25, 2024
25 checks passed
@ngphibang ngphibang deleted the improve_nxp_csi_mipicsi2rx_drivers branch October 25, 2024 07:31
@ngphibang
Copy link
Contributor Author

ngphibang commented Oct 25, 2024

Video buffer C is where I am not sure. I think that the buffer should remain in the input queue, so that if the user then does something like so:

Queue buffer A
Queue buffer B
dequeue buffer
They would receive buffer C. However, I'm not sure if this is correct, or if we should be clearing the input queue so that the user now would receive buffer A when they run the dequeue operation.

Yes, buffer C is in the input queue. But, sorry I don't really get your idea behind when raising these points (?) So, let me summerize what's going on in the application:

  • allocate an array of 2 buffers (most of the cases the min_buffer_count = 2)
  • enqueue 2 buffers (step X)
  • start stream
  • In an infinite loop, doing:
    => dequeue a buffer
    => do something with the buffer
    => re-enqueue the buffer

At this point, user can "stop stream" and this can happen at any step in the loop. So, the questions are:

  • What should the (CSI) driver is supposed to do when stopping so that user can restart again ?
  • What should user be expected to do to restart ?

My points are

  • A restart should be treated as a "new" start. That means users will enqueue the 2 buffers (jump to step X) and call "start stream". For the question 2, users do not need to know which buffers are in the input or output queue and do not need to dequeue all the buffers remaining in the output queue.
  • Currently, at the stopping step, when flushing, the Zephyr CSI driver moves all the buffers in the input queue into the output queue. A (little) advantage of this is users can get the remaining processed buffer (usually just 1 buffer, the last frame). But the drawback is that users need to dequeue all the buffers in the output queue to empty this queue before enqueuing them again into the input queue (while he/she may not do this). That's why I said that we need to modify this to just clean up the input and output queue, so that users can re-enqueue the buffers without any problems.

A typical example of capturing application can be found here in Linux. They don't really do a restart but we can suppose that to restart, just call start_capturing() again. And you can see the start_capturing() function including the enqueuing buffer steps as I mentionned above.

        open_device();
        init_device();
        start_capturing();
        mainloop();
        stop_capturing();
=> can restart by calling start_capturing(); here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: Clock Control area: Shields Shields (add-on boards) area: Video Video subsystem platform: NXP Drivers NXP Semiconductors, drivers platform: NXP NXP platform: STM32 ST Micro STM32
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Video capture sample does not work on NXP i.MX RT10xx (RT1060-EVK, RT1064-EVK)
8 participants