Intel MIPI Camera can't capture the picture

[seankingyang]

Bug Description

-----------[ Webcam still image capture test for Intel_MIPI_Camera ]------------                                                                                                                                                              
ID: com.canonical.certification::camera/still_video0                                                                                                                                                                                          
Category: com.canonical.plainbox::camera                                                                                                                                                                                                      
Purpose:

This test will check that the Intel_MIPI_Camera works

Steps:

1. Click on Test to display a still image from the camera for ten seconds.

Pick an action
    => press ENTER to continue
  c => add a comment
  s => skip this job
  q => save the session and quit
[csq]: 
... 8< -------------------------------------------------------------------------
Error: Internal data stream error.
Format: NV12 (Y/UV 4:2:0)
Resolutions: 1280x720

Failed to capture image with fswebcam, using gstreamer
Error: Internal data stream error.

Intel mipi camera can't capture the 640x480 (the script's default https://github.com/canonical/checkbox/blob/main/providers/base/bin/camera_test.py#L718-L726) which is not support on Intel MIPI camer.
And I try the gst cmd:

ubuntu@ubuntu:~$ gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=10 ! "video/x-raw,width=640,height=480" ! jpegenc ! filesink location="640_480.jpeg"
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Additional debug info:
../libs/gst/base/gstbasesrc.c(3127): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)
Execution ended after 0:00:00.000824281
Setting pipeline to NULL ...
Freeing pipeline ...

ubuntu@ubuntu:~$ ls -al | grep jpeg
-rw-rw-r--  1 ubuntu ubuntu        0 Oct  9 13:53 640_480.jpeg

It will generate the empty size file.

Intel MIPI camera will use the v4l2-relayd.service to transfer the fixed format and width/hight (1280x720).

$ systemctl status v4l2-relayd.service 
● v4l2-relayd.service - v4l2-relay daemon service
     Loaded: loaded (/lib/systemd/system/v4l2-relayd.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2024-09-25 22:44:39 CST; 1 week 6 days ago
   Main PID: 1203 (v4l2-relayd)
      Tasks: 4 (limit: 37880)
     Memory: 1.2G
        CPU: 39.423s
     CGroup: /system.slice/v4l2-relayd.service
             └─1203 /usr/bin/v4l2-relayd -i "icamerasrc buffer-count=7" -o "appsrc name=appsrc caps=video/x-raw,format=NV12,width=1280,height=720,framerate=30/1 ! videoconvert ! v4l2sink name=v4l2sink device=/dev/video0"

Try the same gst cmd with the 1280x720:

ubuntu@ubuntu:~$ gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=10 ! "video/x-raw,width=1280,height=720" ! jpegenc ! filesink location="1280_720.jpeg"                                                                                 
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Redistribute latency...
Got EOS from element "pipeline0".
Execution ended after 0:00:01.013901459
Setting pipeline to NULL ...
Freeing pipeline ...
ubuntu@ubuntu:~$ ls -al | grep 1280_720.jpeg 
-rw-rw-r--  1 ubuntu ubuntu   236634 Oct  9 13:56 1280_720.jpeg

Although it can capture the capture the picture, but it is black.

Try the longer framebuffer and replace multiplefilesink:

ubuntu@ubuntu:~$ gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=60 ! "video/x-raw,width=1280,height=720" ! jpegenc ! multifilesink location="1280_720_multifilesink_buffer60.jpeg"                                                     

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Redistribute latency...
Got EOS from element "pipeline0".
Execution ended after 0:00:02.690528436
Setting pipeline to NULL ...
Freeing pipeline ...

ubuntu@ubuntu:~$ ls -al | grep 1280_720_multifilesink_buffer60.jpeg
-rw-rw-r--  1 ubuntu ubuntu   122818 Oct  9 14:01 1280_720_multifilesink_buffer60.jpeg

Try not set the Width and height:

ubuntu@ubuntu:~$ gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=60 ! jpegenc ! multifilesink location="No_set_WH_multifilesink_buffer60.jpeg"                                                                                          
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Redistribute latency...
Got EOS from element "pipeline0".
Execution ended after 0:00:02.737873971
Setting pipeline to NULL ...
Freeing pipeline ...

ubuntu@ubuntu:~$ ls -al grep No_set_WH_multifilesink_buffer60.jpeg
ls: cannot access 'grep': No such file or directory
-rw-rw-r-- 1 ubuntu ubuntu 120208 Oct  9 14:03 No_set_WH_multifilesink_buffer60.jpeg

---

If we can't get the supported_formats_info, we should not set the capability (ie. format, width, height ...etc).
Intel MIPI camera need longer time to initial and open the video, so the number-buffer need to set larger.

---

Note:
Intel MIPI camera can use fswebcam capture the picture if we set the LD_PRELOAD.
But still can't set the resolutiion. (same reason with above).

Not set the LD_PRELOAD

ubuntu@ubuntu:~$ fswebcam -v -d v4l2:/dev/video0 -S 60  Not_set_LD_preload.jpeg                                                                                                                                                               
--- Opening v4l2:/dev/video0...                            
>>> Using 'v4l2' source module.                            
/dev/video0 opened.                                                                                                                                                                                                                           
src_v4l2_get_capability,87: /dev/video0 information:                                                                   
src_v4l2_get_capability,88: cap.driver: "v4l2 loopback"                                                                
src_v4l2_get_capability,89: cap.card: "Intel MIPI Camera"                                                              
src_v4l2_get_capability,90: cap.bus_info: "platform:v4l2loopback-000"                                                                                                                                                                         
src_v4l2_get_capability,91: cap.capabilities=0x85200001                                                                
src_v4l2_get_capability,92: - VIDEO_CAPTURE                                                                            
src_v4l2_get_capability,101: - READWRITE                                                                               
src_v4l2_get_capability,103: - STREAMING                                                                               
No input was specified, using the first.                                                                               
src_v4l2_set_input,181: /dev/video0: Input 0 information:                                                              
src_v4l2_set_input,182: name = "loopback"                                                                              
src_v4l2_set_input,183: type = 00000002                                                                                
src_v4l2_set_input,185: - CAMERA                           
src_v4l2_set_input,186: audioset = 00000000                                                                            
src_v4l2_set_input,187: tuner = 00000000                                                                               
src_v4l2_set_input,188: status = 00000000                                                                              
src_v4l2_set_pix_format,520: Device offers the following V4L2 pixel formats:                                                                                                                                                                  
src_v4l2_set_pix_format,533: 0: [0x3231564E] 'NV12' (Y/UV 4:2:0)                                                       
Unable to find a compatible palette format.

Set the LD_PRELOAD

ubuntu@ubuntu:~$ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libv4l/v4l2convert.so fswebcam -v -d v4l2:/dev/video0 -S 60  SET_LD_PRELOAD.jpeg
--- Opening v4l2:/dev/video0...
>>> Using 'v4l2' source module.
/dev/video0 opened.
src_v4l2_get_capability,87: /dev/video0 information:
src_v4l2_get_capability,88: cap.driver: "v4l2 loopback"
src_v4l2_get_capability,89: cap.card: "Intel MIPI Camera"
src_v4l2_get_capability,90: cap.bus_info: "platform:v4l2loopback-000"
src_v4l2_get_capability,91: cap.capabilities=0x85200001
src_v4l2_get_capability,92: - VIDEO_CAPTURE
src_v4l2_get_capability,101: - READWRITE
src_v4l2_get_capability,103: - STREAMING
No input was specified, using the first.
src_v4l2_set_input,181: /dev/video0: Input 0 information:
src_v4l2_set_input,182: name = "loopback"
src_v4l2_set_input,183: type = 00000002
src_v4l2_set_input,185: - CAMERA
src_v4l2_set_input,186: audioset = 00000000
src_v4l2_set_input,187: tuner = 00000000
src_v4l2_set_input,188: status = 00000000
src_v4l2_set_pix_format,520: Device offers the following V4L2 pixel formats:
src_v4l2_set_pix_format,533: 0: [0x33424752] 'RGB3' (RGB3)
src_v4l2_set_pix_format,533: 1: [0x33524742] 'BGR3' (BGR3)
src_v4l2_set_pix_format,533: 2: [0x32315559] 'YU12' (YU12)
src_v4l2_set_pix_format,533: 3: [0x32315659] 'YV12' (YV12)
Using palette RGB24
Adjusting resolution from 384x288 to 1280x720.
src_v4l2_set_mmap,672: mmap information:
src_v4l2_set_mmap,673: frames=2
src_v4l2_set_mmap,722: 0 length=2764800
src_v4l2_set_mmap,722: 1 length=2764800
--- Capturing frame...
Skipping 60 frames...
Capturing 1 frames...
Captured 61 frames in 2.72 seconds. (22 fps)
--- Processing captured image...
Writing JPEG image to 'SET_LD_PRELOAD.jpeg'.
ubuntu@ubuntu:~$ ls -al ~/SET_LD_PRELOAD.jpeg 
-rw-rw-r-- 1 ubuntu ubuntu 95343 Oct  9 14:26 /home/ubuntu/SET_LD_PRELOAD.jpeg

To Reproduce

1. checkbox-cli run com.canonical.certification::camera-manual

Environment

• OS: 22.04
• checkbox: deb 4.2.0.dev99

Relevant log output

No response

Additional context

No response

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CHECKBOX-1610.

This message was autogenerated

[tomli380576]

Since the new version of camera_test.py no longer uses a string pipeline we can actually just insert a valve element before filesink with drop=True at the beginning and set drop=False after a certain period of time. We can combine this with jpegenc snapshot=True

...some pipeline ! valve drop=True ! jpegenc snapshot=True ! filesink location=filename.jpeg

snapshot=True is a nice property of jpegenc that automatically sends an EOS signal when the first buffer reaches the encoder. As soon as the valve is opened, jpegenc will get the 1st buffer, let filesink write it, and stop the pipeline.

The timeout step can be done with the following:

GLib.timeout_add_seconds(n_seconds, lambda: valve.set_property("drop", False))

I'll make a PR that implements this.

(technically you can still do this with a string pipeline by using <Gst.Pipeline>.get_by_name("element_name_here") after Gst.parse_launch(string_pipeline) but I didn't know that existed when this bug was opened 😿 )