sensor crop, frame resolution, packing

[ballen4705]

I have a question about the relationship between sensor crop, frame resolution, and packing/alignment.

My camera is a RP HQ, which offers four modes. The mode I want is picam2.sensor_modes[2], for which the relevant entries are:

'crop_limits': (0, 0, 4056, 3040)
'size': (2028, 1520)

The relevant part of my config is:

raw  = {"size":(2028,1520)}
main = {"size":(2028,1520), "format":"YUV420"}

Although I do not use raw directly, I specify the raw size to ensure that main contains the full uncropped 4056x3040 field of view (FOV).

The motivation for my question is that I'd like to have an efficient packing/alignment for the data. Appendix A of the "Picamera2 Library" says that YUV420 is best aligned on 64-pixel boundaries. Indeed, picam2.align_configuration(config) suggests 1984 x 1520 pixels for main, corresponding to

raw  = {"size":(2028,1520)}
main = {"size":(1984,1520), "format":"YUV420"}

My issue: this has the wrong aspect ratio; about 1984 x 1487 pixels would be correct.

My question: how does libcamera/picamera2 cope with this? Does it:

• crop the image, dropping 22 columns of pixels on the far left side of the sensor and 22 columns on the far right side?
• interpolate/scale to produce a 1984 x 1520 pixel image where the full FOV is mapped onto that rectangle, but the aspect ratio is no longer correct?
• fit the full field of view into a 1984x1487 subset of the 1984x1520 output, putting zeros or 255 or random numbers into the final 33 rows?
• do something else?

To pose the question differently, suppose that I set

raw  = {"size":(2028,1520)}
main = {"size":(2048,1536)}

which aligns main on 64-pixel boundaries and is very close to the sensor's aspect ratio (almost exactly 2048x1535). Is the full FOV mapped onto the 2048x1536 array? Or is the final row of the image set to nonsense values?

I'm not sure if this matters for photography, but it is important for image analysis.

Cheers,
Bruce

[davidplowman]

To answer these kinds of questions, it might be helpful, after configuring Picamera2 (with picam2.configure(config)) to check picam2.camera_controls['ScalerCrop'][2]. This will give you the rectangle it has chosen to crop from the sensor array.

For the example above, where the main stream is 1984x1520 in size, picam2.camera_controls['ScalerCrop][2] reports (44, 0, 3968, 3040). So it's cropping out the largest part of the sensor that it can to provide the aspect ratio that was requested (without "squishing" anything in only one dimension).

For YUV420 output you can't have an image that is 1487 pixels high, but if you ask for a 1984x1488 output the scaler crop reports (2, 0, 4052, 3040), so you have, to all intents and purposes, the full FoV.

Generally there shouldn't be too much reason to align configurations unless you have downstream software that can't deal with arrays that are not fully contiguous (i.e. may have padding at the ends of rows), though others may have more practical experience of this.