Sending human created episodes to trainer to speed up model training

[Palamabron]

Hi!, I'm currently working on incorporating my episodes to replay buffer to improve critic training, but I'm facing some major challenges in the process. :( So at least I can give an idea of recording human-made episodes at the beginning of critics' training to make training more time-efficient.

[yannbouteiller]

Hi, this is a cool project but I can see the main technical challenge here: you have to act as an ActorModule otherwise your demonstrations won't really make sense for training one in your real-time environment.

This is doable, but you will have to decouple the game from your gamepad/keyboard input. In other words, you need to make sure whatever input you send have no effect in the game, and are instead intercepted by your program, which then translates them into actions for the step() function of your environment, which in turn will control the game.

You also need to be thoughtful about what action corresponds to what observation: don't forget that, in the real-time setting, your neural network will need one time-step to compute an action (this will be natural if you use something like the tmrl environment, which is rtgym-based, to collect observation and send actions: the action sent to step() is then simply computed from the observation retrieved from the previous step(), like in the usual RL setting).

[Palamabron]

I've already captured input from Xbox controller and change them to actions order. I've also change step function to get action from the controller instead of from the model. However, as this functionality is quite difficult with uncertain ending results, I am switching to writing IQN (TQC is quite terrible in Trackmania due to its' different perpuse or my code is poorly written).
I'll propably write to you on discord (to keep discussions section more sturdy) in a few weeks with more insight where I have problems with human created episodes and my findings on adding new functionalities so far.