Training vs inference performance disparity

[Rajivrocks-Ltd]

Hello. I've trained the SAC exmple agent on my own track. I have been training for 10-15 hours on a RTX 2070 (17 Epochs). It it's not the best, but it finishes the track consistently. But now when I load the pickled model weights with the command "py -m tmrl --test". The agent barely does anything. it drives forward and than just crawls to a halt. I don't get timestep errors in the console. The config.json file has the same "run_name" as the saved model file.

When I run the same pickled model weights on my laptop I do get the timestep errors, but the car does drive further, but still doesn't come close to finishing.

What do you think the issue is? Is it just that more time training is necessary even though during training the agent is consistently finishing the track?

[yannbouteiller]

When you do python -m tmrl --test, the weights that are loaded are those whose name corresponds to the "RUN_NAME" defined in config.json. Are you sure this is loading your model and not the default pretrained weights?

If it completes the track consistently during training, there is no reason for it not to complete the track consistently at test time.

[Rajivrocks-Ltd]

Is it important to have the correct reward.pkl file when you run the --test command? I asusme so, because when i ran the pre-trained weights on the tmrl-test track it also was pretty bad. It tried to take the first corner and just hit the wall and resets

This is, I think, because I have my reward.pkl file for my own custom track in the folder. But since my track does use that .pkl file I don't understand why it's not working :( I'll see tomorrow when i ask a teammate to run it as well

[yannbouteiller]

The reward.pkl has to be the right one, yes, but it doesn't impact the performance of the trained policy. The reason why you need the correct reward function for testing your trained policy is that, without it, tmrl won't be able to recognize that your car is driving in the correct direction and it will trigger the episode termination signal, resulting in your car respawning repeatedly.

[Rajivrocks-Ltd]

Strangely, using the pre-trained weights actually works really well on my track. But my track is just two consecutive corners (left and right turn) and a small straight bit with a finish. But I guess the pre-trained model is just very good.

Also, weights from the same SAC agent on the tmrl-test track that I trained a few days ago work pretty well as well. I think something might be wrong with the model saving. That's the only thing I can come up with

[yannbouteiller]

It was trained on the tmrl-train track with vanilla Soft Actor-Critic for something like 20h, using a remote V100 GPU and, I think, at least 2 PC as rollout workers.

Real-time RL is quite hard, that's why non-roboticists avoid it :P

[yannbouteiller]

For your model saving pipeline, something seems off there indeed.

If you are using a single PC for training, testing should be at least as good as what you get during training (since you don't have the CPU burden of the Trainer during testing).

[Rajivrocks-Ltd]

I reset my example file and now it's just stock, as in the repo. The only thing I changed is this. I am going to retrain from scratch and see what happens

` def save(self, path):
"""
JSON-serialize a detached copy of the ActorModule and save it in path.

    IMPORTANT: FOR THE COMPETITION, WE ONLY ACCEPT JSON AND PYTHON FILES.
    IN PARTICULAR, WE *DO NOT* ACCEPT PICKLE FILES (such as output by torch.save()...).

    All your submitted files must be human-readable, for everyone's safety.
    Indeed, untrusted pickle files are an open door for hackers.

    Args:
        path: pathlib.Path: path to where the object will be stored.
    """
    # with open(path, 'w') as json_file:
    #     json.dump(self.state_dict(), json_file, cls=TorchJSONEncoder)
    torch.save(self.state_dict(), path)

def load(self, path, device):
    """
    Load the parameters of your trained ActorModule from a JSON file.

    Adapt this method to your submission so that we can load your trained ActorModule.

    Args:
        path: pathlib.Path: full path of the JSON file
        device: str: device on which the ActorModule should live (e.g., "cpu")

    Returns:
        The loaded ActorModule instance
    """
    self.device = device
    # with open(path, 'r') as json_file:
    #     state_dict = json.load(json_file, cls=TorchJSONDecoder)
    # self.load_state_dict(state_dict)
    self.load_state_dict(torch.load(path, map_location=self.device))
    self.to_device(device)
    return self`

[yannbouteiller]

I get the same issue as you do when I try to train with the competition script: the agent does not learn at all (it just takes random actions when running train episodes and stays still when running test episodes - staying still during test episodes just means that the model outputs a <0 gaussian mean on the gas command, i.e., it didn't learn anything). When you trained a policy successfully, was that using this script, or was that using python -m tmrl --server / python -m tmrl --trainer / python -m tmrl --worker please? And if using this script, was that using the JSON serializer/deserializer? (I think nobody including me has ever actually used this competition tutorial script, I just tested it quickly and believed it was working since it seemed to save/load/transfer properly, but maybe the JSON serialization/deserialization messes up the model, or there is another bug in this script.)

EDIT: or it may just be the choice of hyperparameters of training_agent_cls that is stupid in this script, because I see that these are outdated and don't make sense compared to the default hyperparameters in config.json. The SAC implementation that we use here is more polished than the example provided in the competition tutorial, and the reward scale of the TrackMania environment has changed recently, probably rendering the hyperparameter values chosen in the competition script entirely absurd.

[Rajivrocks-Ltd]

Hey, so I figured out what was going on, really obvious to be honest now that I think of it. I trained my agent using "py -m <myfilename.py> --server/trainer/worker" but I ran the --test command like so "py -m tmrl --test" but what I should've done was "py <myfilename.py> --test". I guess it loads the reward.pkl file from the tmrl package instead of my custom reward.pkl file of my custom track. Does this make sense to you?

I though that running "py -m tmrl --test" would use all my files (as in model weights, rewards etc), but I guess it uses some pre-defined stuff in the package.

[yannbouteiller]

Well, it uses TmrlData/reward/reward.pkl and TmrlData/weights/run_name.tmod.

Unless you renamed the reward.pkl file, it got erased when you recorded your custom reward. Also, the reward file is only meaningful during training, it has no impact during testing except truncating/terminating episodes.

[Rajivrocks-Ltd]

I figured, but changing the reward and running --test really changes the behaviour of the agent. Also, strange that py -m tmrl --test didn't work than if it worked like that. Because it doesn't work with that command but it does with "py -m <myfilename.py> --test". But I won't question it at this point. I am glad it just works now

[yannbouteiller]

I just pushed Release 0.6.6, which fixes the issues you have encountered. The default hyperparameters of the competition script were outdated, the new default should work much better with the current TrackMania environment. Also, I have commented the wandb logging call, you need to uncomment it (and comment the run() call instead) to continue using that.