F1tenth Racing Using Soft Double Q-Learning

Train your SQN agent to learn a policy to win the race!

ROS

Execute learned policy in ROS with https://github.com/pnorouzi/rl-path-racing

F1tenth_gym environment

The environment here is modified from https://github.com/f1tenth/f1tenth_gym

Installation for f1tenth-spinningup (Native)

The environment officially supports Python3. You'll need several dependencies to run this environment:

Eigen and protobuf dependencies:

$ sudo apt-get install -y libzmq3-dev build-essential autoconf libtool libeigen3-dev
$ sudo cp -r /usr/include/eigen3/Eigen /usr/include

Protobuf:

$ git clone https://github.com/protocolbuffers/protobuf.git
$ cd protobuf
$ git checkout tags/v3.8.0
$ git submodule update --init --recursive
$ ./autogen.sh
$ ./configure
$ make -j8
$ make install
$ ldconfig
$ make clean
$ cd ..
$ rm -r protobuf

Python packages for f1tenth_gym and spinningup:

$ pip3 install --trusted-host pypi.python.org -r requirements.txt

To install the simulation environment natively, clone this repo (it is already cloned, follow the installation steps).

$ git clone https://github.com/f1tenth/f1tenth_gym

Then install the env via the following steps:

$ cd f1tenth_gym
$ mkdir build
$ cd build
$ cmake ..
$ make
$ cp sim_requests_pb2.py ../gym/
$ cd ..
$ pip3 install --user -e gym/

Finally install spinning up:

$ cd f1tenth-spinningup/
$ pip install -e .

Training:

Simple opponent

$ python3 spinup/scripts/train_script.py

or a more difficult opponent that uses TTC-PP SQN policy

$ python3 spinup/scripts/train_advsersarial.py

Testing:

$ python3 spinup/scripts/test_script.py --path data/sqn/$(directory)

or

$ python3 spinup/scripts/test_adversarial.py --path data/advsqn/$(directory)

Plotting logs:

see https://spinningup.openai.com/en/latest/user/plotting.html for plotting options

$ python3 -m spinup.run plot data/sqn/$(directory)/

Status: Maintenance (expect bug fixes and minor updates)

Welcome to Spinning Up in Deep RL!

This is an educational resource produced by OpenAI that makes it easier to learn about deep reinforcement learning (deep RL).

For the unfamiliar: reinforcement learning (RL) is a machine learning approach for teaching agents how to solve tasks by trial and error. Deep RL refers to the combination of RL with deep learning.

This module contains a variety of helpful resources, including:

• a short introduction to RL terminology, kinds of algorithms, and basic theory,
• an essay about how to grow into an RL research role,
• a curated list of important papers organized by topic,
• a well-documented code repo of short, standalone implementations of key algorithms,
• and a few exercises to serve as warm-ups.

Get started at spinningup.openai.com!

Citing Spinning Up

If you reference or use Spinning Up in your research, please cite:

@article{SpinningUp2018,
    author = {Achiam, Joshua},
    title = {{Spinning Up in Deep Reinforcement Learning}},
    year = {2018}
}