[HOW-TO] Optimizing PiCam v3 Autofocus for a High-Altitude Balloon launch

[darksidelemm]

Describe what it is that you want to accomplish
Figure out how to make the autofocus behave better on a High-Altitude Balloon flight.
We've been launching PiCameras of various types on high altitude balloon payloads for many years. The fixed/adjustable focus PiCam2 and PiCamHQ cameras work pretty well, though the PiCam2 image quality leaves a little to be desired, and the PiCamHQ can be quite heavy if you put a good lens on it. The PiCam v3 looks to be a nice option... however there are issues.

Normally we want to set the focus to near-infinity. I usually adjust the focus using a distant house down my street. On the PiCam v3 we can set the lens position to 0.0, which on-ground looks fine.
However, on our first flight with a PiCam v3 in this configuration, we found that the actual lens position appears to drift with temperature, resulting in fuzzy images as the camera got cold, then better again as it got warmer at the top of the flight (>30km altitude).
See here and here for examples of the fuzzy images.

So, onto trying out Autofocus.

The typical image payload configuration has the camera pointing roughly at the horizon. I say roughly, as the payloads swing and spin under the balloon, so the field of view is always moving around. A PiCamv3 had been successfully used on a balloon launch in the UK with good results, but in that case there was a 'target' (Babbage the teddybear!) in the centre of the frame, so the autofocus probably had an easier job.

We bit the bullet and Andrew KE5GDB tried out a PiCam v3 in continuous autofocus mode without a focus target on a launch a few weeks ago. Our observations:

• Autofocus seemed to work fairly well just after launch.
• On ascent, the images started to get blurrier, with the best results seeming to be when the camera was angled slightly downwards, which makes me think the default autofocus zone (roughly in the centre of the frame I think?) is not ideal.
• At the 'top' of the fight, good images were produced. There may still be temperature effects involved here.
• Some example images from this flight are available here.

So, some questions:

• I could set an AfWindow so that focusing is performed using the lower-centre part of the frame. Is the 'reference' for the AfWindow rectangles the top-left of the frame? e.g. if I wanted roughly the lower middle of a full-res PiCamV3 frame, would (1152, 1296, 2304, 1036) be appropriate?
• Are there any other autofocus optimisations that might help with the payload movement issue? Range constraints? Speed?
• I'd like to query and send down the lens position via our telemetry link (~1Hz updates) to investigate the behaviour of the autofocus during a flight - to see if it's hunting, or if the average focus position is drifting with temperature - is querying this via get_camera_metadata() while images are not being saved an appropriate way of doing this?
• If there are still temperature issues, is there any way to 'pre-adjust' the lens to try and correct for these effects? (probably with some reduction in how close the lens will be able to focus)

I get that this use-case is pretty out-of-spec for the camera, but it would be great if we could figure out a way to make use of them in a flight.

Describe alternatives you've considered

• Fixed lens position at infinity: Doesn't work, as the focus mechanism appears to drift with temperature. We tried this on a previous flight, resulting in lots of very blurry images as the payload ascended through the tropopause (the coldest part of the ascent).
• Trying to heat the camera to avoid temperature drift: Probably not impossible, but difficult and increases the electrical complexity of the payload.

Additional context
While the entire code is a bit long, I'm essentially doing this:

# Set focus, white balance, exposure settings
self.cam.set_controls(
            {'AwbMode': self.whitebalance,
            'AeMeteringMode': controls.AeMeteringModeEnum.Matrix,
            'NoiseReductionMode': controls.draft.NoiseReductionModeEnum.Off}
            )
self.cam.set_controls({"AfMode": controls.AfModeEnum.Continuous})
# Set AfWindows
self.cam.set_controls({"AfWindows": [self.af_window_rectangle]})

# Start the camera (self.cam is the picam2 object)
self.cam.start()

# Go into a long running loop, containing:

    # Capture 5 images in a row
    self.cam.capture_file(...)
    self.cam.capture_file(...)
    self.cam.capture_file(...)
    self.cam.capture_file(...)
    self.cam.capture_file(...)
    # Some analysis to pick the sharpest image here (biggest JPEG file size), then some resizing, takes ~30-60 seconds
    # Image is then put into a queue to be transmitted via our radio downlink.

   # Additional 10 second delay here, though the other processing delays mean it can be up to a minute

[davidplowman]

Hi, l'll try and answer a few of your questions, and then perhaps pass on some of the others!

1. Yes, using self.cam.capture_metadata() would be the way to capture information about the camera system. There should be quite a lot in there, including the LensPosition, and also the AfState which will tell you whether the lens is currently scanning, or whether it's in a "focused" or "failed to focus" state.

2. The AfWindow you gave - (1152, 1296, 2304, 1036) - looks good to me, taking the middle half horizontally, and just under half of the image vertically, starting half way down.

3. These lens mechanisms are known to behave differently according to both temperature and orientation (and probably other things too). It is possible to mitigate some of these effects in the camera tuning file - perhaps I can ask @njhollinghurst to comment on what changes might be helpful? Thanks!

[njhollinghurst]

I can't think of anything very helpful. The tuning file can be edited for a static offset - which might help in cases where a negative lens position would otherwise be required (since negative values are not allowed). It might also make AF more stable by reducing the loop gain. But this kind of imagery is challenging for PDAF as it doesn't contain much in the way of horizontal deltas (vertical gratings).

[darksidelemm]

On the question of offsets - is the 0.0 'infinity focus' position at the end of the lens travel range? Or can it go past this?

What I'm still pretty unsure about is if the thermal issue we encountered in the fixed-lens-position flight was the infinity focus position ending up being at some 'negative travel' location.
Doing a flight with autofocus enabled, and logging/transmitting the lens position throughout the flight might shed some light on how this focus point 'moves' with temperature.

[njhollinghurst]

Clipping at 0.0 is a software limit only. In fact, the standard tuning has a tiny bias so 0.0 is slightly beyond infinity in most cameras, to allow for variation.

If necessary, this margin can be increased by lowering the mapping from dioptres to hardware lens driver settings (values between 0 and 1023, where 512 means "zero drive current" and 1023 pushes the lens out for nearest focus; the usable range when horizontal and at room temperature is about 300-960 before the lens hits its end-stops). This can be done either in Python, or by copying and editing the camera tuning file.

I also don't know the mechanism of the thermal effect: whether it causes an absolute shift in lens position or merely scales the current (the difference from 512) or something else...

Is this the standard or the wide-angle lens?

[darksidelemm]

Standard lens.

[njhollinghurst]

Well I don't know if it will help, but the following code snippet would offset everything by 3 dioptres, so that to focus on infinity you would now need to set lens position to +3.0

from picamera2 import Picamera2, Preview
from libcamera import controls

OFFSET_DIOPTRES = -3.0

tuning = Picamera2.load_tuning_file("imx708.json")
map = Picamera2.find_tuning_algo(tuning, "rpi.af")["map"]
print(map)
offset_hw = OFFSET_DIOPTRES * (map[3]-map[1])/(map[2]-map[0])
for i in range(1, len(map), 2):
    map[i] += offset_hw
print(Picamera2.find_tuning_algo(tuning, "rpi.af")["map"])

picam = Picamera2(0, tuning=tuning)

In short it's lowering the dioptres->driver mapping by 96 by reducing map[1] and map[3] by that amount. The hack enables more lens travel in the "beyond infinity" direction. (Assuming that's the correct direction of offset, which isn't clear.)

If you can discover the actual range of useful settings, set map[1] to start at the smallest such value. You could then set find_tuning_algo(tuning, "rpi.af")["ranges"]["normal"]["max"] to the spread of useful settings (in Dioptres), which will restrict the search range that Autofocus will sweep through; it doesn't restrict manual lens positions.

[darksidelemm]

I can certainly give this a go!
Related question - If I use horizontal and vertical flipping, is the AfWindow applied before or after the flip is performed?

[njhollinghurst]

That is a good question. It ought to apply to the flipped image. I will have to do some more tests to check it happens consistently.

[darksidelemm]

Thanks! I did some tests myself, but without much conclusive results as I don't think my 'test scene' was working all that well.

[njhollinghurst]

I have just done some tests and I think it's correct. Both Phase and Contrast statistics align with the image as received (after any flips, which are implemented inside the sensor).

[It won't necessarily work the same on every sensor, e.g. those without hardware "flip" capability, but we don't currently support PDAF on any others.]

[darksidelemm]

OK, this is really good to know!

I'm hoping to do a launch sometime in November using a PiCam v3 and autofocus. My thinking right now is I will use:

• AfWindows set as above (approx lower middle of frame)
• Offset the lens by maybe 1 dioptre (still need to test this).

We will also now be sending the lens position in our downlinked telemetry, so we'll have this data at 1 Hz time resolution - should be interesting to see if it drifts during flight!

I did get some 'fixed focus' IMX708 lenses from ArduCam to try.. unfortunately these really are fixed focus, the lens is glued in place, with the focus set too close to be useful...

[darksidelemm]

I've now integrated your lens offset example into my codebase, with the offset being provided as a command-line parameter.

For a lens offset of -1, I end up with the map values pre/post of:
[0.0, 445, 15.0, 925]
[0.0, 413.0, 15.0, 893.0]

It seems to be working, as my 'test scene' has moved from a lens position value of about 0.8 to 1.8-2.0. More testing to be done...

[darksidelemm]

Today we performed a high-altitude balloon launch with the PiCam v3, using autofocus and a lens offset of -1.
Lens position was transmitted live down to the ground all throughout the flight, allowing us to see the behaviour of the autofocus.

Unfortunately it seemed that the offset of -1 was not enough, as we still ended up with the lens position clamping to 0 as we ascended into colder temperatures. Near the top of the flight we got a few better photos. We saw similar on the last flight with a PiCam v3, suggesting it's heating up again as we get to much higher altitudes.

I have a lot of data from the flight, though scattered in various places that I'm trying to bring together.
For now:

• Dropbox folder containing all the images captured: https://www.dropbox.com/scl/fo/fttr6yq6ma3ajipsixh11/AAyuIe1m1xdk7daBXDq_wAw?rlkey=i7c56rkbjtfagrhsclgv8grj6&dl=0
• CSV file with the lens position for each image: https://gist.github.com/darksidelemm/b15cf80dac41b231c9842964ce1a8ec1
• Grafana dashboard showing the lens position (and a lot of other stuff) reported from the payload throughout the flight: https://grafana.v2.sondehub.org/d/ddzty5w2yqt4wc/wenet?var-Payload=VK5ARG-Wenet&from=1733009735249&to=1733017797611&orgId=1 (scroll down to find the lens position data)
• Grafana dashboard from another payload on the flight showing external temperature: https://grafana.v2.sondehub.org/d/HJgOZLq7k/basic?var-Payload=HORUS-V2&from=1733009732000&to=1733017614027&orgId=1

Since it does look like the lens position was ending up at 0, I'm wondering if the offset needs to be increased further, or perhaps needs to be adjusted dynamically throughout the flight?
What offset is possible?

[njhollinghurst]

Thank you for all the data. It is hard to test this stuff on the ground -- we do have a temperature controlled chamber but it's too small for camera experiments, especially those requiring focus at infinity.

It sounds like the offset is helping, at least. Yes it is possible to increase it. The range of "raw" lens driver values (after the mapping has been applied) is 0-1023; this controls an electrical current rather than an absolute mechanical position. Under normal conditions there are hard stops at about 350 and 950 (i.e. settings outside this range have no further effect) but it's hard to say how that generalizes to other conditions.

[darksidelemm]

Thanks for the info - so in this case the -3 dioptre offset (-96 raw units) is really as far as we can go before hitting the mechanical limits of the lens. I guess this gives us something to test on the next flight - if that doesn't work, then there isnt anything else we can do.

As for temperature testing, even lens position vs temperature data for a closer target would still probably provide valuable information, at least to confirm what direction the lens is 'shifting' with temperature!

[darksidelemm]

Related question - in the camera metadata we see the field 'FocusFoM'. Is this an output from the autofocus algorithms?
Would this give us some understanding of how well the autofocus is working throughout the flight?
From some basic testing on the bench, I've seen this number go as high as 22000 when the camera is focused correctly on a scene, and down to ~1500 when i put a postit note over the lens so it has nothing to focus on.

[ke5gdb]

I ran the Pi Cam v3 Wide in a temperature chamber for several cycles between 25C and -60C with a target placed on the chamber wall. The target was approximately 70cm from the camera. It's not infinity, but it is the best I can do with the hardware available. An offset of -5 dioptres was applied (thus making the map [0, 285, 15, 765]).

Temperature profile:

1. Ramp from 25C to -60C -- 60 minutes
2. Soak at -60C -- 60 minutes
3. Ramp from -60C to 25C -- 60 minutes
4. Soak at 25C -- 10 minutes
5. Jump to step 1

This is the data plotted over several cycles:

The minimum LensPosition recorded was 1.268 dioptres, and the maximum was 4.01. (Reminder: offset -5 dioptres). The FocusFoM parameter was hovering between 11000 and 13000. The onboard sensor temperature sensor clipped quite easily at -20C. It was typically reading ~20 degrees above ambient.

Test pattern taped to chamber wall:

Is there a way to convert the LensPosition value to a raw lens driver value? My napkin math didn't have great correlation when I held everything constant and changed the offset value.

[njhollinghurst]

Wow. Thank you!

It's a piecewise linear mapping with (typically) 2 points. [d0,h0,d1,h1] ought to mean a forward mapping of:
h0 + (dioptres - d0) * (h1 - h0) / (d1 - d0) or
lambda = (dioptres - d0) / (d1 - d0); hw_setting = h0 * (1 - lambda) + h1 * lambda

and a reverse mapping of
d0 + (hw_setting - h0) * (d1 - d0) / (d1 - h0) or
lambda = (hw_setting - h0) / (h1 - h0); dioptres = d0 * (1 - lambda) + d1 * lambda

So here I reckon 4.0 => 413; 1.0 => 317.

I'm a bit confused about the scale for "sensor temp". Did it go up to 45C? Is this the camera's own temperature sensor?

[njhollinghurst]

Related question - in the camera metadata we see the field 'FocusFoM'. Is this an output from the autofocus algorithms?

We have two focus metrics: this one is based on image contrast (CDAF) which is generally quite stable and reliable for an unchanging scene, but it has a flat top and doesn't tell AF in which direction to move.

The other one (PDAF) is directional but flakier. PDAF is not currently included in the metadata. We mostly use PDAF, though when it's working, the two measures tend to agree quite well.

[nzottmann]

I ran the Pi Cam v3 Wide in a temperature chamber for several cycles between 25C and -60C with a target placed on the chamber wall. The target was approximately 70cm from the camera. It's not infinity, but it is the best I can do with the hardware available.

To test infinity focus in a space constricted environment, you can use a close up lens. You can get those often in set like (+1, +2, +4, +10) from the well known online marketplaces with diameters ranging between 40mm to 80mm. With a +10 lens an object 100mm (1000mm/10) away from the camera is in focus, when the camera is adjusted to infinity.

When I tried this, the theoretical value of 100mm did not match exactly, it was about 10-20mm off. But you can adjust the exact distance with a live preview and watching the FocusFoM value, adjusting the distance to match the value to real infinity focusing.

The diameter of the lens does not really matter as every available lens is big enough for the Pi Camera to look through. Just build yourself some fixture to adapt it in front of the lens.

[darksidelemm]

We have two focus metrics: this one is based on image contrast (CDAF) which is generally quite stable and reliable for an unchanging scene, but it has a flat top and doesn't tell AF in which direction to move.

Ok - the context here is that at the moment when we capture imagery to downlink, we capture a block of 5 images with a second gap between each, then pick the 'best' image out of that block. The current approach to picking the 'best' image is to pick the one that has the highest JPEG-compressed file size. Bigger file = More detail.
We do this selection as the payload often spins and swings under the balloon, which means that the field of view changes a lot between ground, sky (black), and something in between. With the selection process, it helps us avoid sending images that are entirely black.

I was thinking of adding an option to select the image based on the FocusFoM metric. Take a sequence of images, and downlink the one with the highest FocusFoM.

[njhollinghurst]

Focus FoM is based on image gradients (after a bit of noise removal) and should correlate pretty well with the file-size approach and ought to reject the all-black images just as well. It only looks at a region of interest which IIRC defaults to the middle two cells of a 4x3-cell grid.

I've just remembered that focus offset can vary with tilt as well as with temperature! (I'm a little mystified why smartphone cameras don't seem to suffer from that; perhaps they use an accelerometer...)

[njhollinghurst]

To test infinity focus in a space constricted environment, you can use a close up lens.

nzottmann That's a good idea. Though relative to the focal length of this camera, 70cm is "not very far" from infinity and may still tell us something relevant about the temperature coefficient.

ke5gdb's reported lens positions were surprisingly low, even at room temperature - we may have to revisit our default tuning.

Now I am wondering whether to add a temperature coefficient to the AF algorithm in future...

[ke5gdb]

@njhollinghurst correct, "sensor temp" is what comes back in the image metadata. It did hit 45C when chamber ambient reached 25C.

I just purchased a close up lens kit. It should be here by the end of the day. I'll also swing by Microcenter today and buy a new camera. The one I performed the testing with has already flown once and may have some FOD/dirt in or near the lens. I don't think this is an issue, but I'd like to eliminate all variables.

Let me know if there are any specific test conditions or additional data you would like to see.

[njhollinghurst]

Thanks. In the new year I will try to repeat this (and also with the standard V3 lens). If you're happy to do further tests, it would be great if you could also try ambient temperatures above 25C (maybe up to 40C) and for two different focus positions (say, ~infinity and ~20cm).

We may be able to add temperature compensation to the focus algorithm (using the reported sensor temperature). I don't yet know if it should be purely additive or if it should also depend on focus position. Any changes will be made here.

[ke5gdb]

Give me a few days, and I'll see what I can do. I'll be out of office starting this Friday and won't be back until the new year, so hopefully there is chamber availability before then.

@nzottmann with regards to the close up lenses: is there an ideal distance that I should have between the close up lens and the front element on the Pi Camera?

[nzottmann]

@ke5gdb In my experience the exact distance between the close up lens and the camera module did not matter. I mounted it with a distance of about 10mm.

You will have to adjust the distance to the object to be sharp when focused to infinity anyways, as the theoretical value of 100mm is only approximately correct.
This may also be due to the lens not being exact +10 dioptries, as it was quite cheap.

[njhollinghurst]

I imagine a much weaker lens, such as [EDIT] +2 dioptres, is called for here.