A pub-sub architecture with remote capabilities for Python

mpubsub implements a publish-subscribe messaging architecture in Python that can be used by local and remote processes. Remote publishers or subscribers can join any time to exchange messages.

Features:

• Extremally fast and compact pub-sub object — the entire local pub-sub module has about 100 lines, including docstring documentation for all public methods.
• All batteries included — mpubsub only uses Python’s standard library modules. No external dependencies needed.
• Remote connections — subscribers are transparently notified of topics published on remote pub-sub objects. Messages published locally are forwarded to remote pub-subs.
• Local topic support — messages can be flagged for local delivery only.
• OOP interface — you can have multiple independent pub-sub objects on the same program.
• Remote authentication — mpubsub uses the same authentication mechanism used by the (multiprocessing module)[https://docs.python.org/3.7/library/multiprocessing.html] to authenticate remote connections.

mpubsub requires Python 3.7 or above.

Installation

pip install mpubsub

Architecture

mpubsub consists of three classes: PubSub, NetPubSub, and Broker.

The PubSub class is an efficient implementation of the publish-subscribe messaging pattern for same-process publishers and subscribers. This class alone can be used to implement pub-sub in an application when no remote capabilities are needed.

The NetPubSub is a networking-capable version of PubSub. A process can connect a NetPubSub to a mpubsub message broker so that messages published locally are forwarded to all other pub-subs connected to the same broker. Likewise, all subscribers of the local pub-sub object receive messages published to the broker.

The Broker class implements the mpubsub message broker. The broker coordinates message passing between remote publishers and subscribers.

The PubSub class

The PubSub class provides an interface for publishers and subscribers to interact with the messaging system. Publishers can post messages to a topic, and subscribers receive messages for the subscribed topic.

Topics

Topics are Python tuples, and those tuple-topics are handled hierarchically. For example, if the topic (‘foo’, ‘bar’) is published, then subscribers are notified in the following order:

1. All (‘foo’, ‘bar’) subscribers
2. All (‘foo’,) subscribers — NB: a one-element tuple
3. All catch-all subscribers

For convenience, you can subscribe to, and publish string topics. Internally, those are transformed to a single tuple topic (in other words, publishing or subscribing to 'foo' is the same as publishing or subscribing('foo',)).

When subscribing, the empty tuple is the catch-all topic. Subscribers to this topic will get all messages published on the PubSub object.

When publishing, the empty tuple is the broadcast topic. Messages published with the broadcast topic are sent to all subscribers in the pub-sub.

You can pass None instead of () to mean the catch-all/broadcast topic. In all cases, subscribers callbacks always get tuples as their first parameter.

Note: mpubsub does not validate topic types for performance reasons. Due to implementation details, other pickable, hashable objects may “work” as topic values, but this is not supported. It is up to the callers to ensure that only tuples of strings are used as topics (sole strings and None are fine, since they’re translated to tuples following the rules outlined above).

Messages

Topics can carry additional key=value data, called the message. Those are mapped directly to keyword arguments when a subscriber is called.

mpubsub will issue warnings if a subscriber does not support a keyword argument present in a message.

The NetPubSub Class

The NetPubSub is a subclass of PubSub that adds remote capabilities to it. Beside all PubSub methods, it also provides methods to connect and disconnect the pub-sub to a broker.

The Broker Class

The Broker class acts as the intermediary between publishers and subscribers in remote pub-subs. It listens for incoming connections from NetPubSub objects, and coordinate message forwarding between them.

Notice that the broker run asynchronously in relation to remote NetPubSub object. You can start a broker using multiprocessing or as a separate Python script. Using threads is not supported.

Examples

Local pub-sub example

First import the class and create a PubSub object:

>>> from mpubsub import PubSub
>>>
>>> pubsub = PubSub()

Create two functions that just print their parameters. Those will be our example subscribers:

>>> def subscriber_1(topic, message):
...     print(f'Received by subscriber 1: {message}')
>>>

>>> def subscriber_2(topic, message):
...     print(f'Received by subscriber 2: {message}')
>>>

Add the subscribers to the pub-sub object:

>>> pubsub.add_subscriber('hello', subscriber_1)
>>> pubsub.add_subscriber('hello', subscriber_2)

Now publish some data:

>>> pubsub.publish('hello', message='foo')
Received by subscriber 1: foo
Received by subscriber 2: foo

To unsubscribe, just call remove_subscriber() with the same parameters:

>>> pubsub.remove_subscriber('hello', subscriber_2)
>>> pubsub.publish('hello', message='foo')
Received by subscriber 1: foo

To unsubscribe all subscribers, call clear_subscribers():

>>> pubsub.clear_subscribers()
>>> pubsub.publish('hello', message='foo')

Remote pub-sub example

See demo.py for an example of how to use mpubsub to implement remote pub-subs.

Remote pub-subs addresses

mpubsub connects to remote pub-subs using multiprocessing Connection objects. Address used by NetPubSub and Broker classes have the same format and semantincs as described in the Python’s documentation.

Standalone broker

You can start a standalone broker with the following command:

python -m mpubsub.broker $HOME/.mpubsub.pckl

The broker’s address and authkey are pickled as a tuple (address, authkey), and saved to the file provided in the command line. Clients can read this file to unpickle the values and connect to the broker. Run python -m mpubsub.broker --help for usage information.

Security

• mpubsub uses multiprocessing’s authentication keys to authenticate connections. If you pass None to the broker, multiprocessing.current_process().authkey will be used. The mpubsub broker does not support unauthenticated connections. See the relevant documentation for more details about Python’s multiprocessing authentication mechanism.

• The authentication used by multiprocessing protects only against unauthorized connections. Message contents are transmitted in plain text, and because of that anyone on the network(s) where connection packets travel can see the contents of published topic and messages. Is advisable to use remote connections only on trusted networks.

• All data received from authenticated connections are unpickled automatically. This opens the possibility of remote code execution, therefore you should connect pub-subs only to brokers you trust. See the relevant warning in the Python’s documentation.

Questions? Critics? Suggestions?

Visit https://github.com/flaviovs/mpubsub