How to adjust Camera Zoom using the SDK APIs?

[rayw-dronesense]

I can arm the aircraft, take off, and get the video feed using the camera.videoStreamInfo API. However, I am having some trouble finding an API for Zooming in/out of the camera - could you please help with this? Thanks!

The Android API appears to be based on https://github.com/mavlink/MAVSDK-Proto/tree/main/protos

[julianoes]

Zoom is missing, see open PR: mavlink/MAVSDK#2094
I'll be adding zoom next week, hopefully.

What camera are you using?

[rayw-dronesense]

We are trying to add support for a drone that advertises its cameras as Greenseer EO and Greenseer IR (it's a commercial product built using MAVLink). We can get the video feed from it using the existing API as stated earlier, as well as take photos and record videos, using the respective APIs - at least with the "EO" visual camera sensor.

We also have not yet found a way to successfully switch over to the IR camera. (selectCamera(..) doesn't seem to be working, but maybe I'm just doing it wrong)

[julianoes]

The question is how the IR camera is exposed. We might have to add support for mavlink/mavlink#2079, if the camera supports that (it's very new).

[rayw-dronesense]

Hi Julian,

Thanks a lot for the information.

It looks like (to my own sleuthing) that the IR camera is exposed the same way the visual normal camera is exposed, using the mavlinkcamera definition XMLs on the sdcard, here's a snippet of one

    <parameters>
        <!-- control = 0 tells us this should not create an automatic UI control -->
        <parameter name="CAM_MODE" type="uint8" default="1" control="0">
            <description>Camera Mode</description>
            <!-- This tells us when this parameter changes, these parameters must be updated (requested)-->
            <options>
                <option name="Photo" value="0">
                    <!-- This tells us when Camera Mode is set to Photo mode, the following parameters should be ignored (hidden from UI or disabled)-->
                    <exclusions>
                        <exclude>CAM_PHOTOFMT</exclude>
                        <exclude>CAM_PHOTOQUAL</exclude>
                        <exclude>CAM_VIDFMT</exclude>
                        <exclude>CAM_COLORMODE</exclude>
                        <exclude>CAM_ISO</exclude>
                        <exclude>CAM_WBMODE</exclude>
                        <exclude>CAM_EXPMODE</exclude>
                        <exclude>CAM_SHUTTERSPD</exclude>
                        <exclude>CAM_ISO</exclude>
                        <exclude>CAM_EV</exclude>
                        <exclude>CAM_VIDRES</exclude>
                    </exclusions>
                </option>
                <option name="Video" value="1">
                    <!-- Conversely, when Camera Mode is set to Photo mode, the following parameters should be ignored (hidden from UI or disabled)-->
                    <exclusions>
                        <exclude>CAM_PHOTOFMT</exclude>
                        <exclude>CAM_PHOTOQUAL</exclude>
                        <exclude>CAM_VIDFMT</exclude>
                        <exclude>CAM_COLORMODE</exclude>
                        <exclude>CAM_ISO</exclude>
                        <exclude>CAM_WBMODE</exclude>
                        <exclude>CAM_EXPMODE</exclude>
                        <exclude>CAM_SHUTTERSPD</exclude>
                        <exclude>CAM_ISO</exclude>
                        <exclude>CAM_EV</exclude>
                        <exclude>CAM_VIDRES</exclude>
                    </exclusions>
                </option>
            </options>
        </parameter>
        <parameter name="CAM_COLORPAL" type="uint32" default="0">
            <description>Color Palette</description>
            <options>
                ...
            </options>
        </parameter>
        <parameter name="CAM_AGCMODE" type="uint32" default="1">
            <description>AGC Mode</description>
            <options>
                ...
            </options>
        </parameter>

Thanks again,

Ray

[julianoes]

Oh, interesting, so it's actually implemented as a separate "camera". That should indeed work as well. I don't think we have recently tested two separate cameras in MAVSDK, so it might be broken. Do you know if both cameras send mavlink from different component IDs?

One thing about selectCamera. You should do that right on init, and then not switch. That way you would essentially create the camera plugin twice. However, I now realize that that might not be feasible in the language wrappers.

@JonasVautherin do you know?

[rayw-dronesense]

Oh, interesting, so it's actually implemented as a separate "camera". That should indeed work as well. I don't think we have recently tested two separate cameras in MAVSDK, so it might be broken. Do you know if both cameras send mavlink from different component IDs?

I believe it does emit the cameras as different component IDs, yes.

One thing about selectCamera. You should do that right on init, and then not switch

Hmm...but the use case is - customers will want to switch cameras at-will, at any point in time

[julianoes]

Right, well this switchCamera doesn't switch it for the user. It just sets the plugin to use one or the other.

[rayw-dronesense]

Thanks - is there anything we can do in the meantime, to get this functionality in our Android app? What would be your recommendation? (for both ability to zoom and switch cameras on the fly)

@julianoes @JonasVautherin

[JonasVautherin]

I think it is a bit hard with the language wrappers, because with gRPC (if I remember correctly) it does not seem possible to treat a service like an object and instantiate it multiple times. So Camera is a service that is instantiated once, and in this case we would want it twice.

Using gRPC, my idea a few years ago was to have one gRPC server per component, and have them listed through the core plugin. Then the user (or a helper) could instantiate a corresponding object to each server separately (e.g. a camera1 and a camera2 that would connect to two different ports).

Then my preference went to actually remove gRPC and use FFI, so that the language bindings would be closer to the C++ version.

But unfortunately I never got resources to work on either of those. The gRPC solution could potentially be less work, but still it's not completely trivial 😕.

[julianoes]

@JonasVautherin one not so crazy change would be to add an argument camera_id to each camera API call. I would think that should work.

@rayw-dronesense if you want to move this issue up on my (long) todo list, you can consider github sponsoring.

[rayw-dronesense]

@JonasVautherin @julianoes Thanks for getting back to me. Perhaps we wouldn't need to instantiate twice - if the one instance can switch over to completely operating on the newly selected camera.

I was able to use the selectCamera call to switch the stream, finally. Specifically, had to stopVideoStreaming, do the selectCamera(1) call, and then startVideoStreaming. This successfully switched the live stream to start using the IR camera. I'm assuming it worked because it just so happens this drone is set up to broadcast its video on the same port, regardless of camera selection.

But, both the 'current' and 'possible' camera settings do not reflect the IR camera parameters - only of the visual camera with ID of 0 after the selectCamera call succeeded. Knowing what we know now, would this be something feasible to do?

[julianoes]

But, both the 'current' and 'possible' camera settings do not reflect the IR camera parameters - only of the visual camera with ID of 0 after the selectCamera call succeeded. Knowing what we know now, would this be something feasible to do?

I have to check whether we properly assign this data.