Buffering topics

[skinkie]

After reading #77 again another use case last week came to our attention. We have some vendors that are unable to keep up, they are basically asking "Can you buffer the output (further)?"

I could obviously write a client that would implement a timeout queue so it would republish a topic at a specific rate, but guarantee that the topic is republished at least once per minute. But maybe from server perspective this behaviour would be something that more people would be interested in.

[bigsmoke]

@skinkie, I am not sure what you are describing here, but it sounds like it might be related to the [client_max_write_buffer_size](https://www.flashmq.org/man/flashmq.conf.5#client_max_write_buffer_size). What is client_max_write_buffer_size` currently set to for these client configurations?

[skinkie]

No this is something else. The updates on topics are too fast for them, so imagine a sensor that operates at 10Hz, the client only wants to have 1Hz updates.

[bigsmoke]

Off the bat, my own inclination would be to advice in favor of your suggestion to do the buffering in a client that republishes at the desired rate, but perhaps @halfgaar will know of a way to “cheaply” implement this in FlashMQ. And by “cheap” I mean: in a way that won't bog down anything when the feature is disabled, nor introduce other foreseeable difficulties in terms of maintaining the feature in the context of other expected or planned feature.

One of the core architectural tenets that @halfgaar followed in the design of FlashMQ is to do one thing right rather than include a whole bunch of features.

[halfgaar]

I'm afraid I'm going to have to reject this request. This would make some sense for a datagram based protocol, but for TCP you don't have control over how your chunks are sent or received anyway. When you send one chunk, it may very well be received as 10. This is easily seen in the field when you have many topics and the client and server are far apart from each other.

It will also be too dependent on the client's implementation whether this will have any effect.

There are various other methods to reduce the publish rate for clients, either by aggregating or dropping publishes, using a plugin or perhaps a bridged client, but this will have to be custom made.

Perhaps MQTT5 shared subscriptions is also an option for you?

[skinkie]

@halfgaar I think you are too deep into the network layer. In this case a playlist is received, with recorded play times. For some actions this is relatively fast, so the integral playlist is published over and over. One of the subscribees asked to reduce the publishing rate (hence the 10Hz to 1Hz example). What I am looking for is a way to define either a value with a connected client, or special topic that basically gives the last value of the topic after at most N seconds of publication, but at least every N seconds.

And as mentioned, creating a custom client will work, and maybe "timed subscriptions" would be rather expensive to maintain within the server, but I am sure that this is not something that would have a single use case.