[SUGGESTION] Provide an option to capture_ methods that guarantee a frame after the request is made

[jlprojects]

This is related to the discussion in issue #787 but is more of a feature request. It's probably relevant for projects requiring synchronising the camera to outside events or machine movement. I found myself with a problem where a photo has to be taken after some motors have stopped moving.

No matter what settings of buffer_count and queue=False sometimes an old frame that was taken while the motors were running is being supplied even when the request is made after the motors stop, leading to blurred images. Although this happened to some extent on the Pi4, it seems more frequent with the Pi5 and/or recent libcamera/picamera2 updates. Possibly related to the greater speed of the system?

With information from #787, my workaround is to store the time of the capture request using time.monotonic_ns() which is compared to SensorTimeStamp in the metadata of the request. If SensorTimeStamp is earlier, then another capture_request is made. Some partial code from my camera control object - the do_capture method is called when the motors stop:

    def do_capture(self):
        self.capture_timestamp = time.monotonic_ns()
        self.picam2.capture_request(signal_function=self.raw_captured)

    def raw_captured(self, job):
        r = self.picam2.wait(job)
        img_buf = r.make_array(name='raw')
        m = r.get_metadata()
        r.release()
        if m['SensorTimestamp'] < self.capture_timestamp:
            # Libcamera has delivered a frame prior to capture request
            self.picam2.capture_request(signal_function=self.raw_captured)
        else:
            # start processing img_buf
            ...

Although the above seems to do the job, it would be nice to incorporate an option into the capture_ methods so that picamera2 can be told to always return a frame at or after the time of the request, and saves having this hacky workaround code in the signal_function.

[davidplowman]

Hi, thanks for that, I think that's an interesting idea. Are you thinking of maybe a timestamp argument for capture_request(), so that you could do something like

request = picam2.capture_request(timestamp=time.monotonic_ns())

and you'd be guaranteed to get a frame where the 'SensorTimestamp' is >= the timestamp passed in?

Just as a note, even that doesn't seem fully correct, you probably want to subtract the exposure time from the SensorTimestamp. Otherwise some of (even all) the pixels have started being exposed before the time in question. Presumably you'd prefer it that nothing gets exposed beforehand?

[jlprojects]

Thanks, David.
It's preferable if nothing is exposed before the capture request. I was thinking about something along the lines of a simple flag like:

request = picam2.capture_request(from_now=True)

the capture would simply skip frames with a timestamp earlier than the system time of the call to capture_request.
The analogy is with a stills camera shutter - you press the button and the shutter fires. You wouldn't expect an image from before you pressed the button - which is what appears to be happening due to some latency in the camera system. 😄
In my use case the motor controller signals the pi via a gpio pin that the motors have stopped, so it is safe to grab an image.

For info, it's not happening every time, and appears to need to skip only one frame if it's caught. When I put something like the following into my capture code:

sensor_timestamp =  m['SensorTimestamp']
print(f'Frame before capture - diff: {self.capture_timestamp-sensor_timestamp}')

I get results like:

Frame before capture - diff: 59891897
Frame before capture - diff: 18285219
Frame before capture - diff: 27011932
Frame before capture - diff: 16928185
Frame before capture - diff: 12854275
Frame before capture - diff: 25012663
Frame before capture - diff: 16019137
Frame before capture - diff: 30893324
Frame before capture - diff: 6676741

A range of 6 to 60mS. In this case the shutter speed is 10495, but not factored in the above check.

[davidplowman]

I quite like the idea of being able to specify a timestamp, I could imagine some code being informed that some event happened "a short while ago", and you might want to use a timestamp that you're given. Or if it's not a single capture, and you're trying to pipeline your captures and synchronise them with other (asynchronous) events.

Maybe one could have a "flush" parameter, where flush=True means use the current time (the system will fetch it), or flush=my_timestamp if you want a different time (or flush=None for the existing behaviour, which is obviously the default).

I'm still thinking that you really want "SensorTimestamp - ExposureTime >= timestamp", though - would you agree?

[jlprojects]

The flush=True option seems fine by me as it would appear to achieve the desired effect. I'm trying to simplify things for my particular usercase and simplify my terrible Python code. 😄
The comparision with current time should be from when the frame capture started, so yes SensorTimestamp - ExposureTime >= timestamp is valid if SensorTimestamp is the time when the frame capture has ended.

[davidplowman]

The flush=True option seems fine by me as it would appear to achieve the desired effect. I'm trying to simplify things for my particular usercase and simplify my terrible Python code. 😄 The comparision with current time should be from when the frame capture started, so yes SensorTimestamp - ExposureTime >= timestamp is valid if SensorTimestamp is the time when the frame capture has ended.

The SensorTimestamp is the time when the capture of the first pixel has happened. These are (mostly) rolling shutter sensors, so the rest of the frame is either still being exposed, or possibly hasn't even started being exposed yet!

But anyway, here's a first attempt at something: #851

[jlprojects]

Thanks, David! That was a quick implementation! 👍
As far as I can tell, this is perfect. I just did a quick and dirty test (by copying picamera2 from #851 and putting it into /usr/lib/python3/dist-packages/picamera2/ (backed up the original, first).
It appears to be working for my code at least. With the line self.picam2.capture_request(signal_function=self.raw_captured, flush=True) , the frame check in self.raw_captured isn't being triggered.
With luck this feature will be included in the official release, soon.

I suppose it would also make sense to make the flush parameter available to the other capture_ methods - before requiring capture_request to get the metadata, I was using capture_array(name='raw') to get just the image data. 😄

[juanmf]

I was looking for this feature exactly. Thanks.
but also was using image = picam2.capture_array("main")
How do I obtain image in the format capture_array("main") returned it from request in request = self.picam2.capture_request(flush=True) ?

Edit:: think I found out (need to test.)

    result = request.result()
    # Access the image data from the result
    image0 = result["main"]

---

Does this new feature help apply config setigns as well?

From Section 5.1 of manual:

with picam2.controls as controls:
  controls.ExposureTime = 10000
  controls.AnalogueGain = 1.0

In this final case we note the use of the with construct. Although you would normally get by without it (just set the
picam2.controls directly), that would not absolutely guarantee that both controls would be applied on the same frame.
You could technically find the analogue gain being set on the frame after the exposure time.
In all cases, the same rules apply as to whether the controls take effect immediately or incur several frames of delay.

These "several frames of delay" are not present when flush=True, right?

So in the following snippet time.sleep(0.05) is redundant?

                with self.picam2.controls as ctl:
                    ctl.ExposureTime = exposure
                time.sleep(0.05)

                tprint(f"Setting exposure to {exposure}: ")
                request = self.picam2.capture_request(flush=True)
                result = request.result()
                # Access the image data from the result
                image0 = result["main"]

[davidplowman]

I haven't added this feature to the other capture methods as the parameter lists are getting quite cluttered already. I took the view that folks who were sufficiently advanced to care about these details would be able to do

request = picam2.capture_request(flush=True)
array = request.make_array('main')
request.flush()

but I'm always open to persuasion on this kind of thing.

One thing to note is that this doesn't really help in capturing a frame when the exposure has changed. It takes several frames for the exposure to change even after you've sent the command to change it, so the only way to know that it has happened is to sit in a loop and watch the image metadata (at which point the sleep would be redundant).