Potential memory leak in training PPO agents

[ThomasChen98]

Hi,

I followed the tutorial setting up the docker container and ran train.py with all default hyper parameters on tictactoe. Here is my command
sudo docker-compose exec app python3 train.py -r -e tictactoe
I did not use parallelization and I notice that the RAM is growing linearly w.r.t. the training steps. Roughly about 700 MB memory increase after 0.2M steps. After training for ~20M steps my computer's 32GB RAM will be fully occupied.

Has anyone encountered similar issue and is there a way to resolve this?