Propagate queue pausing across multiple application instances

[keyan]

Currently Verk doesn't support distributed deployments. If you are running multiple application instances then pausing a queue in one instance will not pause the queue in the other N instances of Verk.

What if instead of writing queue pause/running status to ets we write to Redis, then check Redis during queue manager startup and at some interval to know if the queue status should be changed? Is a multiple instance Verk deployment a common use-case?

[edgurgel]

Currently Verk doesn't support distributed deployments. If you are running multiple application instances then pausing a queue in one instance will not pause the queue in the other N instances of Verk.

Yes! Our pause for now is per instance.

What if instead of writing queue pause/running status to ets we write to Redis, then check Redis during queue manager startup and at some interval to know if the queue status should be changed? Is a multiple instance Verk deployment a common use-case?

Yes, multiple instance deployment is a really common use-case.

Using Redis is an option, yes. I really wanted to have a way to make Verk instances aware that they are part of a cluster. My hope was to use Partisan but I'm still playing with it.

Do you want to take a stab implementing the status on Redis? Then we if decide to move to something like Partisan we can always refactor it etc.

One thing that we need to think about is that the status will become persistent across restarts. It's probably not an issue but something that we need to document so it's not a surprise to anyone.

What do you think?

Thanks again for creating relevant issues 👍