Topic prefix stripping in server-to-server connections

[smurfix]

Hi,

While FlashMQ supports prefixing topics on messages it sends and receives, it doesn't seem to be able to remove a prefix from incoming or outgoing topics, at least as far as flashmq.conf manpage is concerned.

This would afford transparently tunneling arbitrary messages through a shared server.

Mosquitto can do it ;-)

[halfgaar]

I can see why you would want it. I'd have to spend some time looking at how complex it would be to make, and what the performance implications are (I tend to avoid adding overly heavy features).

[smurfix]

I hear you. A "strip the first N components of the incoming / outgoing message" feature would work for most if not all users, if that's easier than a comparison; I didn't yet encounter any such link that needed more complex arrangements.

In any case, on the positive side, at least this feature has no performance impact if it's not used.

[marcelrv]

This topic mapping would indeed be very appreciated.
@smurfix you indicate topic prefixing when sending, how did you accomplish that as that would already do most of the job in my case, but did not recognise how to do that from the conf file nor from the bridge code

[smurfix]

I currently accomplish topic prefixing with RabbitMQ, but I'd rather switch to something faster + more extensible (FlashMQ has this nice plugin architecture …) if at all possible.

[halfgaar]

I did some checking for feasibility. One question pops to mind: if there is authentication in place, do you expect that to happen before or after the mangling? Let's say we're talking about messages on topic a/b/c:

• In incoming publish: do you want the 'write' permission to be checked on prefix/a/b/c or a/b/c?
• In delivering a publish to a subscriber, same question: do you want that 'read' permission to be checked on prefix/a/b/c or a/b/c?

There are some architectural difficulties if you would want the ACL check to happen with the prefix. Logically, I'd say without the prefix makes most sense.

[smurfix]

As long as you're consistent whether the checks are done on the paths as seen by the client or as seen by the server (= every other client that doesn't have mangling applied to it) I can write an ACL to match.

Thus, my answer is "whatever happens to be easier to implement".

[wiebeytec]

OK, then I'll give it a go. Prefix stripping on receive is easy. Adding a prefix on send is a bit more difficult, but if we can do it at the very last step, meaning after authentication on the original topic string, it's probably doable.

Do note that it will incur some extra overhead, because there are certain optimizations it can't do then.

Probably the next feature you want is multiple bridge connections with load balancing. It's on my list...

[halfgaar]

I created a branch with a test version: https://github.com/halfgaar/FlashMQ/tree/PREVIEW-bridge-prefixes. If compiling is too hard, I can probably give you a .deb package.

The bridge config sections supports a local_prefix and remote_prefix option. It should act the same as Mosquitto's, in terms of how it subscribes and what it adds and removes.

The prefix options are defined per bridge, not per pattern. There are some conceptual problems with having them per pattern. I'm not sure how Mosquitto does it...

The ACL checking turned out to be the following:

• Remote prefix is removed and local prefixes is added before ACL write checks. So, the ACL write checking is done after the mangling.
• The read ACL checks (delivery to subscribers) is done before mangling. So, the ACL is done on the path as it appears locally. Then, after ACL, the local prefix is removed and the remote prefix added.

[halfgaar]

Actually, I was a bit premature. There are various issues with that branch still.

[smurfix]

Thanks. Looking forward to testing it once you've got the issues ironed out.

Building a .deb is not a problem.

[smurfix]

OK, then I'll give it a go.

👍🏼

Probably the next feature you want is multiple bridge connections with load balancing. It's on my list...

Heh. Not really. My volume is high but not that high.

What I do want is redundancy, i.e. connect my MQTT servers in an arbitrary mesh and lose no messages (*) if any one of them hangs / fails / whatever. I'm currently using my own protocol on top of MQTT for this and I'd love to be able to bypass the whole thing.

(*) except exact duplicates of course; when a sensor tells me five times that it's 18° out there I don't care how many of those arrive – as long as the number is greater than zero.

[halfgaar]

As an update; I was doing some prep work for this change, which kind of opened up can of worms of things that needed fixing. So, I'm working on a new release, coming in a few days I hope, and then I can continue with this.

[smurfix]

Thanks for the update. Not a problem.

[halfgaar]

I think I'm close to giving you a .deb to test. For what OS+version do you want it?

[smurfix]

Cool.
Standard Debian stable should be fine.

[halfgaar]

You can download a test version for Bookwork here. It's a release build, so optimized and no symbols. At this point, I don't expect test scenarios I need debuggers for, just whether it behaves correctly. I can always make a debug build if required.

Install with dpkg -i.

This comment still applies as to how it works.

The more scenarios tested, the better. Like bridges with persistent sessions (non-clean start and non-0 expiry), severed connections, queued messages, using aliases, etc.

[halfgaar]

I'm eager to any test results, both positive and negative alike :)

[smurfix]

Yeah and I'm eager for some "free" time in which to test the setup I have in mind …

I'll try to shovel some hours free next week.

[halfgaar]

Has the shoveling worked ;)?

[smurfix]

On 28.05.24 08:27, Wiebe Cazemier wrote: Has the shoveling worked ;)?

Still shoveling, unfortunately. Best laid plans and all that *grumble*.

…

-- -- regards -- -- Matthias Urlichs

[halfgaar]

I created new test builds, based off 1.15.2:

• https://www.flashmq.org/wp-content/uploads/testing/flashmq_1.15.2-1719506565+jammy_amd64.deb
• https://www.flashmq.org/wp-content/uploads/testing/flashmq_1.15.2-1719592043+bookworm_amd64.deb

There were some architectural changes in master, so this branch was adapted to that.

[smurfix]

Could you rebase this to the subscription-id branch? That'd let me set up a mostly-integrated test environment which I hopefully can (lightly) torture over the holidays.

[halfgaar]

It has some merge conflicts, and more with the subscription IDs. I can, but I'd rather first finish the subscription IDs and have that in master.

[smurfix]

Thanks. No hurry.

[halfgaar]

I tried to rebase, but now the tests fail. I don't know why yet.

I also saw some likely mistakes. So, this still needs some work.

[halfgaar]

I have a new test branch:

https://github.com/halfgaar/FlashMQ/tree/WIP-bridge-prefixes

The prefix is not applied automatically to the bridge's subscribe and publish directives. It felt too magic to do so, plus it would mean the prefixes are applied to all subscribe and publish directives, which is not good.

Let me know how it works out.