Habitat baselines PPOAgent fails when using pre-trained weights

[mikimena01]

Habitat-Lab: master

Habitat-Sim: master

Habitat is under active development, and we advise users to restrict themselves to stable releases of Habitat-Lab and Habitat-Sim. The bug you are about to report may already be fixed in the latest version.

Master branch contains 'bleeding edge' code, but we do appreciate bug reports for it!

When using PPOAgent with default_config and pre-trained weights there is a mismatch between dimensions in the visual encoder output and in the visual_fc input. I have tried to use different weights and configurations. When trying with the training command suggested in the README it works.

File "top_down_test.py", line 345, in example_top_down_map_measure
action = agent.act(observations)
File "/habitat-lab/habitat-baselines/habitat_baselines/agents/ppo_agents.py", line 129, in act
action_data = self.actor_critic.act(
File "/habitat-lab/habitat-baselines/habitat_baselines/rl/ppo/policy.py", line 332, in act
features, rnn_hidden_states, _ = self.net(
File "/.conda/envs/habitat/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/habitat-lab/habitat-baselines/habitat_baselines/rl/ddppo/policy/resnet_policy.py", line 647, in forward
visual_feats = self.visual_fc(visual_feats)
File "/.conda/envs/habitat/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/.conda/envs/habitat/lib/python3.9/site-packages/torch/nn/modules/container.py", line 217, in forward
input = module(input)
File "/.conda/envs/habitat/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/.conda/envs/habitat/lib/python3.9/site-packages/torch/nn/modules/linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x10240 and 2048x512)