This code learns reward functions from human preferences in various tasks by actively generating batches of scenarios and querying a human expert.
Companion code to CoRL 2018 paper:
E Bıyık, D Sadigh. "Batch Active Preference-Based Learning of Reward Functions". Conference on Robot Learning (CoRL), Zurich, Switzerland, Oct. 2018.
You need to have the following libraries with Python3:
Throughout this demo,
- [task_name] should be selected as one of the following: Driver, LunarLander, MountainCar, Swimmer, Tosser
- [method] should be selected as one of the following: nonbatch, greedy, medoids, boundary_medoids, successive_elimination, random For the details and positive integer parameters K, N, M, b, B; we refer to the publication. You should run the codes in the following order:
This is the preprocessing step, so you need to run it only once (subsequent runs will overwrite for each task). It is not interactive and necessary only if you will use batch active preference-based learning. For non-batch version and random querying, you can skip this step.
You simply run
python input_sampler.py [task_name] KFor quick (but highly suboptimal) results, we recommend K=1000. In the article, we used K=500000.
This is where the actual algorithms work. You can simply run
python run.py [task_name] [method] N M bb is required only for batch active learning methods. We fixed B=20b. To change that simply go to demos.py and modify 11th line. Note: N must be divisible by b. After each query or batch, the user will be showed the w-vector learned up to that point. To understand what those values correspond to, one can check the 'Tasks' section of the publication.
This is just for demonstration purposes. run_optimizer.py starts with 3 parameter values. You can simply modify them to see optimized behavior for different tasks and different w-vectors.