Overlays are cool (would be interesting to see your keymap using them!).

They were quite a complex feature, and did not see enough use to pull their weight, so we removed them in xkbcommon. If you provide a patch, we'll certainly consider it, though we can't guarantee we'll accept it.

I’ll need at least some hints about where in the codebase to put the code.

xkbcommon is directly derived from xkbcomp and the XKB code in the xserver. So you can basically see how overlays are implemented there. I have never investigated them much myself.

One issue you'll eventually hit, as you found out, is that overlays use "controls" -- these are separate parts of the XKB state, like depressed/latched/locked modifeirs/layouts. Activating an overlay toggles a bit in the controls state. The problem is that in xkbcommon this state component went entirely unused, so was removed from the API (e.g. the xkb_state_component enum and state serialization functions). I suppose you can ignore this problem to start out with.

I have a draft PR, see #514. I do not see much issue with controls, but rather find it difficult to not mess with keycode redirections, or rather: not make it complicated to use for compositors. It requires extreme rigor to redirect keycode at this level.

However, permanent overlays could be completely internal (just copy the target key) and thus make it easy to permute keys in XKB.