Questions about the files mentioned in this project

[BlackPeonyFish]

Dear author, I am a graduate student, and I want to use parallel genetic algorithm to train my racing AI, so I read your article. I have to say that your article is very hardcore. As a novice, I can only barely understand your "get started" part and implement the running of the test file. But then, you asked me to read the Tutorial: TMRL as a Python library part, which made me confused. Please forgive my stupidity, but you tried to teach me to write a py file, but even if I learned how to write this file, which folder should I put it in, and what file should I run? In addition, you also provided a large number of script files in the directory, such as custom, tools, etc. Where should these files be placed, and which file should I run? I have no answer to these questions. I hope you can provide more detailed explanations for beginners like me, thank you very much! !

[yannbouteiller]

Hi @BlackPeonyFish :)

The TMRL as a Python library tutorial is written for RL professionals, it is clearly not a good fit for beginners. Implementing custom TMRL pipelines is a bit involved and there is no real beginner-friendly way of doing this I'm afraid.

What you can do instead is to simply use tmrl.get_environment(). This will give you access to the Gymnasium environment for TrackMania 2020 as configured in config.json. You can then use this environment as described here. Collect a training episode, pause the environment with env.wait(), train your model using your genetic algorithm, and repeat.

A solution that is simpler than the Python tutorial if you want to implement a TMRL pipeline in TrackMania rather than using the environment manually as described before is to read and adapt this script. But if you are not super familiar with Python, I recommend using tmrl.get_environment() and coding your training pipeline from scratch instead, as this will be much more incremental and intuitive.

[BlackPeonyFish]

Thank you very much for your guidance. I really don't need to build my own pipeline, and I don't even need to modify the parameters of the neural network. I just want to find a platform that allows my own genetic algorithm code to train the weights and biases of the neural network, which is exactly what I want.

However, I hope you can take a look at whether my understanding is correct: first, I should modify the parameters in the config.json file according to my own requirements; then, I should run the script file you wrote in the "TrackMania Gymnasium environment" chapter of the tutorial; then I can collect my data on the "Weights and Biases" website (I have some doubts about this. You mentioned in the tutorial that in order to save the training data, you need to run python -m tmrl --trainer --wandb. I am not sure whether running this script can save the training data); wait until enough data is collected, use env.wait(), and then put the data into my own genetic algorithm, and the genetic algorithm will get the weights and biases of the next generation; then loop (here I have another question, that is, how can I give the new weights and new biases calculated by the genetic algorithm to the next training neural network?)

Thank you for your time!

[yannbouteiller]

The Weights and Biases website is only a website for logging performance metrics online during training. It has nothing to do with the training algorithm per-se and you should not worry about it for now.

The best (but hard) course of actions to achieve what you want is to adapt the competition tutorial script that I referenced in my previous post. More specifically, you want to adapt the train() function of the SACTrainingAgent class.

However, this will only easily work with off-policy 1-step backup approaches such as Q-learning. Because you want to implement a genetic algorithm, I suppose you instead need an on-policy pipeline that gives you access to the entire latest episode, which is possible but tricky to implement in TMRL (because you would need to implement a custom Memory object and use the not-so-well-documented synchronization mechanics). This is why I recommend you don't do this as long as you are not very familiar with TMRL.

Instead, you should implement your training pipeline manually from scratch, and use TMRL only for the TrackMania Gymnasium environment to interact with the game. This environment is rather straightforward to use once you understand how to use it.

You will have to code your own model (neural net) and your own way of training it from scratch, but believe me, unless you want to implement a very advanced, distributed training pipeline, this will be much easier than implementing an on-policy algorithm in TMRL.