I agree that we should target 2 to begin with. We will still need python multiprocessing over here to run actors and learners seperately right?

As for the structure and fitting it into the rest of the library I was thinking of having DistributedOnPolicyTrainer and DistributedOffPolicyTrainer which will act as the main process and spawn the multile actors while maintaining and updating the central weights. In this case, the agent would only need to implement update_params (to be called in the main process) and select_action (to be called for each actor). The trajectories and weights would be transported through reverb.

I am holding off on #233 since a reverb buffer wrapper would heavily depend on the structure we go with. Plus it is not really useful in the non-distributed case.

Stale issue message