cooper

Master Branch:

Develop Branch:

A simple and novel approach to the Actor pattern in C++

The actor pattern is a means of concurrent programming in which active "actor" objects run concurrently although each runs in its own single thread of control. The actors keep their internal data hidden from the others and only communicate through messages to each other.

Languages that support the actor pattern natively, such as Elixir and Erlang, usually do so by pattern matching data messages that are passed between the actors. C++ does not support the powerful pattern matching paradims of these other languages, and therefore some efforts to bring the Actor pattern to C++ have turned rather unwieldy.

The classical C++ object-oriented means of passing messages within a single process uses functon calls to the public interface of a C++ class, with overloading to match different parameter sets.

The 'cooper' library attempts to bring that simplicity to the Actor pattern in C++, by also mixing in the Actor client/server model of the Elixir/Erlang OTP behaviour of the "GenServer".

In this model, a C++ actor class creates a public "client" interface of standard class methods that can be called by any thread or other actor in the application. The public methods then schedule the operations to run, in sequential order, on the actor thread by queueing matched methods from the private "server" API.

Like the Elixir GenServer, the C++ clients send messages to the sever thread by using the "call()" and "cast()" operations of the actor base class. The call() performs a synchronous operation in which the caller thread/actor is blocked until the operation completes in the internal actor thread. A cast() sends an asynchronous request to the server thread.

Using in your CMake application

With the library installed, you can use it in a C++ application built with CMake. It creates targets for Cooper::cooper-shared and/or Cooper::cooper-static for the shared and static versions of the library, respectively, if each of the versions were built. It also creates a target Cooper::cooper for whichever version for the library was built, if only one was built, and sets it to the shared library if both were built. So use it if the application doesn't care about the version, or would prefer the shared library, if available.

An application CMake can look like this:

# CMakeLists.txt
#
# CMake build file for a 'cooper' C++ application.
#

cmake_minimum_required(VERSION 3.12)
project(MyProj VERSION 0.1.0 LANGUAGES CXX)

# --- Dependencies ---

set(THREADS_PREFER_PTHREAD_FLAG ON)
find_package(Threads REQUIRED)

find_package(cooper REQUIRED)

# --- Application ---

add_executable(my_app my_app.cpp)
target_link_libraries(my_app Cooper::cooper Threads::Threads)

You can also use it from a submodule in your source tree, to build the library along with your own code. Simply replace the find_package() with an add_subdirectory(), like:

add_subdirectory(cooper)

The rest of the CMake file is the same. The submodule will also alias the targets with the namespace, so you still set the link library as Cooper::cooper, etc.

Example

This is an example of an in-memory key/value store that can be shared between the threads of an application.

A classic approach might pair a std::map with a mutex to protect it from simultaneous access from multiple threads. Even better, a reader/writer lock would allow multiple reader threads to access it simultaneously.

This example hides the map inside an actor. It has a few advantages:

• The class maintains full control of access to the map.
• An errant thread can't lock the map indefinitely and starve out other threads.
• The internal actor doesn't need to use locks since the actor thread is the only one to ever directly access the map.
• There's a degree of fairness since requests from all the different client threads are queued sequentially, in the order received.
• Writer threads don't need to block waiting for access to the map. The set(key,val) method can run asynchronously, but...
• Even though writes are asynchronous from the point of view of the client thread, there's still a deterministic outcome for the reads, since all operations are queued sequentially. A get(key) will always return the last value set - there's no race condition due to the asynchronous behavior of the writes.

class shared_keyval : public cooper::actor
{
    /** The data store */
    std::map<string, string> kv_;

    // ----- The server API -----

    // These always run in the context of the single actor 
    // thread and therefore can't interrupt each other - so
    // no locking is necessary.

    // Set the value in the key/value store.
    void handle_set(const string& key, const string& val) {
        assert(on_actor_thread());
        kv_[key] = val;
    }

    // Get the value from the key/val store.
    // Returns the value if found, otherwise it return nullopt.
    optional<string> handle_get(const string& key) {
        assert(on_actor_thread());
        auto p = kv_.find(key);
        if (p != kv_.end())
            return { p->second };

        return {};
    }

public:
    /**
     * Create an empty key/value store.
     */
    shared_keyval() {}

    // ----- The client API -----

    // These run in the context of the calling (client) thread
    // and therefor are not allowed to touch the private data.

    /**
     * Sets a value in the key/value store.
     * This is an asynchronous operation.
     */
    void set(const string& key, const string& val) {
        cast(&shared_keyval::handle_set, this, key, val);
    }

    /**
     * Retrieves a value from the key/value store.
     * This blocks the caller until the value is retrieved.
     */
    optional<string> get(const string& key) {
        return call(&shared_keyval::handle_get, this, key);
    }

    /**
     * Wait for all pending operations to complete.
     */
    void flush() { call([]{}); }
};

The public members of the class create the API that can be called by, and also run in the context of the client thread. In this case, it is the get(key) and set(key, value) calls. These public functions, by convention, are not allowed to touch the member data in the class. Instead they each send a message to the internal actor thread to run a closure with a matching call signature to the public API, handle_get(key) and handle_set(key, val), respectively. These private methods are expected to run sequentially in the context of the actor thread for the object.

The private handle_...() methods are the only ones that can touch the internal data for the object. Since they are guaranteed to be run in the context of a single actor thread, they do not need to perform any locking on the data. They are free to manipulate the data however necessary without worry of thread contention or race conditions.

The Actor has two primary ways to queue closures for the internal thread, cast() and call(). The first, cast() is an asynchronous operation. It puts the closure into the queue for the internal thread, and then return immediately. It doesn't return a value.

The second way is the call() operation. This queues the closure, and then blocks the client thread until the operation runs to completion inside the actor's thread. It then returns the value from the operation (i.e., the return value from the private call in the closure). So, for example, the get(key) will queue up a call to handle_get(key), wait for it to run, and then return the value from handle_get(key) back to the client that called it originally.

And exceptions thrown by the internal actor thread will also get passed back to the client thread.

Conventions

There are several conventions that are helpful (and possibly essential) to follow:

• The public API is callable by external code and runs in the context of the calling thread. This should be considered the "Client API".
• The private methods in the class (or a subset of them) should run in the context of the actor thread. This should be considered the "Server API".
• The public client methods should never touch the internal data in the object (neither write nor read).
• The private server methods can manipulate the data freely without using locks since they run one after the other, always in the context of the actor thread.
• It is helpful to match calls between the Client and Server API's, and simply have the client call the matching server method.
• Even if you don't touch the data directly, you should never do a read/modify/write operation from the client API. It wouldn't be guaranteed to run atomically from the perspective of the other clients. Rather, that should be moved into a server call and then all clients would percieve it as being atomic. As a rule of thumb: If you do more than one cast() or call() operation in a client method, you may be doing something wrong!
• The server methods should try to run as quickly as possible and return. Each object has a single execution context, and a blocked call will prevent any other operations from running.
• A server call should never block waiting for another client operation, since the blocked actor thread will not be able to run the other calls and deadlock will occur.
• Server calls that are assumed to be running in the actor thread context should probably test that that is the case - at least during the develop and debug cycles. A good idea is to have them assert that they are actually running on the correct actor thread:

assert(on_actor_thread());

• Actors are also a good way to share resources, such as sockets, serial ports, database connections, etc.
• The cast() operation is very helpful to keep a client from blocking on an opperation, but an errant client can overload an actor with a lot of asynchronous (cast) operations. Sometimes, even if a client method does not require a return value, it might be helpful to code it as a call() to apply back-pressure to the client.