Questions on conventions

[cfneuhaus]

Hi,

I've got some questions regarding conventions used in Symforce:

1. For Lie groups, Symforce uses a convention where
   retract(a,vec) = compose(a, from_tangent(vec))
   however, some people (notably [1] for example)
   use a convention where
   retract(a,vec) = compose(from_tangent(vec), a)
   To change it, is it enough to change ops/lie_group_ops.py or are there other parts dependent on this convention?
   How would you suggest to perform this adaption, and are there any plans to have this configurable at some point?

(By the way, do you think this is simply a matter of preference, or do you see any advantages in Symforce's choice?)

2. It is an interesting use-case to use Symforce-generated factors in Ceres-Solver. However, Ceres-Solver does not currently support tangent-space derivatives, as discussed here [2]. Is it enough to wrap a function: f(q: geo:Rot3) -> Whatever into
   into

def f_ceres(q_xyzw: geo:V4) -> Whatever:
   return f(geo.Rot3.from_storage([q_xyzw[0],q_xyzw[1].q_xyzw[2],q_xyzw[3]));

in order to obtain ambient derivatives, or what is your proposed workaround for this issue?

3. Symforce uses quaternions with an xyzw storage order. How would I switch to wxyz for compatibility with other parts of a user's program?
   Simply changing quaternion.py does not seem enough, for example rot3.storage_D_tangent is certainly dependent on the quaternion storage, right?
   (I am not sure whether it is actually important to change storage order, because Lie Group derivatives are independent of the order... and when interfacing with Ceres, the wrapper function (see 2.) could deal with the swap...)

Best
Frank

[1] Bloesch, Michael, et al. "A primer on the differential calculus of 3d orientations." arXiv preprint arXiv:1606.05285 (2016). https://arxiv.org/abs/1606.05285
[2] ceres-solver/ceres-solver#615

[aaron-skydio]

1. The references we tend to use do right-retract (e.g. [1]), but we don't have a particular reason to think this is superior to a left-retract convention, just a matter of preference. If you wanted to switch things to left-retract, you'd have to modify the implementation of retract in symforce/ops/interfaces/lie_group.py, as well as the implementations for different types in symforce/ops/impl (although I think the only non-container types in there are vector types where it doesn't matter since they're abelian anyway). You'd want to do the same for local_coordinates, since it's generally assumed that local_coordinates and retract are compatible throughout the code. Additionally, if you're computing any tangent-space jacobians, you'll have to modify those implementations in symforce/jacobian_helpers.py. There may be other places I'm missing. We don't plan to support configuring SymForce to switch between right-retract and left-retract for everything, although it might make sense to add a left_retract method to LieGroupOps as a thing you can call.
2. Yes wrapping like that should be sufficient. You could also just not use Codegen.with_linearization or Codegen.with_jacobians, and instead return the jacobian from your wrapper directly, which wouldn't require changing the inputs to storage vectors. For instance (assuming f returns a vector like f(q: sf.Rot3) -> sf.V3):

def f_ceres(q: sf.Rot3) -> T.Tuple[sf.V3, sf.Matrix]:
    result = f(q)
    return result, result.jacobian(q, tangent_space=False)

3. If you do really need to switch the storage order, I think a wrapper function like you had makes the most sense. We're generally very careful to not depend on the storage order of types across the codebase unless necessary; if you did really want to change the quaternion storage order in particular you should just need to change quaternion.py and Rot3.storage_D_tangent on the symbolic side. On the C++ side though we do quite a bit of interop with Eigen::Quaternion so you'd also likely have to fix things in rot3.h and rot3.cc here to keep all of that working. This shouldn't be necessary though and isn't something we plan to make configurable.