ERROR: failed to destroy actor 290 : unable to destroy actor: not found

[aleallievi]

I am working on adapting the baseline LbC agent by @bradyz here to work with the latest versions of CARLA (9.10.1) and ScenarioRunner. I keep getting errors that seem to point towards an improper clean_up at the end of each test route. For instance, when running test_route_00 I get the following output:

> Registering the route statistics
ERROR: failed to destroy actor 290 : unable to destroy actor: not found

And sometimes I see the following:

ERROR: Invalid session: no stream available with id 788529155

Eventually the simulator crashes with the following error (although potentially unrelated to the clean_up issue above:

terminating with uncaught exception of type clmdep_msgpack::v1::type_error: std::bad_cast
Signal 6 caught.
Malloc Size=65538 LargeMemoryPoolOffset=65554 
Malloc Size=65535 LargeMemoryPoolOffset=131119 
Malloc Size=122688 LargeMemoryPoolOffset=253824 
Aborted (core dumped)

Any ideas where wires may be crossing with the implementation?

Thanks!

[glopezdiest]

Hey @aleallievi. About your errors:

ERROR: failed to destroy actor 290 : unable to destroy actor: not found

This error is actually a warning. This is caused by the script trying to eliminate the same actor multiple times. It doesn't really create any problems though.

ERROR: Invalid session: no stream available with id 788529155

This one is indeed problematic. Could you explain in further detail how are you running the simulation?

Also, about the malloc size error. I'm confused here. Is this error related to the Invalid session one? or is it independent?

[aleallievi]

Hi @glopezdiest
Thanks for taking up the issue. Let me address each point separately, since as you mentioned they are likely to not be linked.

ERROR: failed to destroy actor 290 : unable to destroy actor: not found
This error is actually a warning. This is caused by the script trying to eliminate the same actor multiple times. It doesn't really create any problems though.

I did some more debugging inside the leaderboard_evaluator.py, and this seems to be caused by scenario.remove_all_actors() function using the data in scenario.other_actors, which still contains actors that are no longer in the world (can be checked by querying both CarlaDataProvider.get_world().get_actors() and CarlaDataProvider._carla_actor_pool). However this may not be a big issue as you state.

ERROR: Invalid session: no stream available with id 788529155
This one is indeed problematic. Could you explain in further detail how are you running the simulation?

This error does not occur nearly as often as the others, and it could be a result of improper clean-up, or wires crossing with the traffic manager.
Here's the details of my set-up: I am on Ubuntu 18.10, on a Lambda Labs GPU workstation 2019 (4 x NVIDIA RTX 2080 Ti). I am using Python's subprocess to spawn and run multiple instances of the simulator in parallel. I took care that each server-client rtc and traffic manager ports are a unique match. The actor uses the baseline learning-by-cheating model by @bradyz . Which leads me to the last error.

terminating with uncaught exception of type clmdep_msgpack::v1::type_error: std::bad_cast
Signal 6 caught.
Malloc Size=65538 LargeMemoryPoolOffset=65554 
Malloc Size=65535 LargeMemoryPoolOffset=131119 
Malloc Size=122688 LargeMemoryPoolOffset=253824 
Aborted (core dumped)

This is the real error that's been bugging me. There seems to be no consistent pattern to the simulator crashes. I have, at most, got to simulating 5 consecutive routes before at least one of the servers crashes throwing this error. But sometimes it happens even during the very first run. The interesting thing is that the error happens even if I launch just a single instance of the simulator through subprocess, which makes me think it could be an incompatibility with subprocess.Popen.

Thanks in advance for your help

[glopezdiest]

Okay, we'll investigate, see if it can be reproduced

[duydeja-vu]

I seen same error in Carla server "ERROR: Invalid session: no stream available with id 9" when run carla leaderboard in local machine for 1 day (one carla server in docker, and only one client connect to this server). Someone else have more comment on this? Please guide me how to avoid it. This is very critical situation because we need to run carla leaderboard in many days to collect data and evaluate our AD software, so that carla server need stable to accomplish this purpose.