GUI sporadic crashes when updating plugins

[chapulina]

Environment

• OS Version: Ubuntu Bionic
• Source or binary build? Source (main, future Edifice)

Description

• Expected behavior: Gazebo runs indefinitely without crashing
• Actual behavior: After running for a while, Gazebo crashes

While testing the Edifice demo world, I noticed that the simulation crashes from time to time. Getting a backtrace, it pointed to the EntitiesByComponents call inside JointPositionController:

https://github.com/ignitionrobotics/ign-gazebo/blob/14d967d94884da881a3a2701b09a9240d11e0a75/src/gui/plugins/joint_position_controller/JointPositionController.cc#L212-L213

Initially I thought that the periodic updates introduced in #497 may introduce a race condition where 2 threads try to access the ECM at the same time. But it looks like that's properly protected by a mutex, so I think there's something else going on.

I can't really parse the backlog up to #11, which is where it enters stl_set.

Steps to reproduce

1. ign gazebo -v 4 "https://fuel.ignitionrobotics.org/1.0/OpenRobotics/worlds/Edifice demo"
2. Leave it running for several minutes until it eventually crashes

Output

Backtrace

#0  0x00007fda9e6ff2d7 in tcache_get (tc_idx=1) at malloc.c:2952                                                                            
#1  0x00007fda9e6ff2d7 in __GI___libc_malloc (bytes=40) at malloc.c:3060                                                                    
#2  0x00007fda9acb4298 in operator new(unsigned long) () at /usr/lib/x86_64-linux-gnu/libstdc++.so.6                                        
#3  0x00007fda4b04da86 in __gnu_cxx::new_allocator<std::_Rb_tree_node<unsigned long> >::allocate(unsigned long, void const*) (this=<optimize
d out>, __n=1) at /usr/include/c++/8/ext/new_allocator.h:99                                                                                 
#4  0x00007fda4b04da86 in std::allocator_traits<std::allocator<std::_Rb_tree_node<unsigned long> > >::allocate(std::allocator<std::_Rb_tree_
node<unsigned long> >&, unsigned long) (__a=..., __n=1) at /usr/include/c++/8/bits/alloc_traits.h:436                                       
#5  0x00007fda4b04da86 in std::_Rb_tree<unsigned long, unsigned long, std::_Identity<unsigned long>, std::less<unsigned long>, std::allocato
r<unsigned long> >::_M_get_node() (this=<optimized out>) at /usr/include/c++/8/bits/stl_tree.h:588                                          
#6  0x00007fda4b04da86 in std::_Rb_tree<unsigned long, unsigned long, std::_Identity<unsigned long>, std::less<unsigned long>, std::allocato
r<unsigned long> >::_M_create_node<unsigned long const&>(unsigned long const&) (this=<optimized out>)                                       
    at /usr/include/c++/8/bits/stl_tree.h:642                                                                                               
#7  0x00007fda4b04da86 in std::_Rb_tree<unsigned long, unsigned long, std::_Identity<unsigned long>, std::less<unsigned long>, std::allocato
r<unsigned long> >::_Alloc_node::operator()<unsigned long const&>(unsigned long const&) const (this=<synthetic pointer>, __arg=@0x7fda7586d6
88: 3297509811873971798) at /usr/include/c++/8/bits/stl_tree.h:556                                                                          
#8  0x00007fda4b04da86 in std::_Rb_tree<unsigned long, unsigned long, std::_Identity<unsigned long>, std::less<unsigned long>, std::allocato
r<unsigned long> >::_M_insert_<unsigned long const&, std::_Rb_tree<unsigned long, unsigned long, std::_Identity<unsigned long>, std::less<un
signed long>, std::allocator<unsigned long> >::_Alloc_node>(std::_Rb_tree_node_base*, std::_Rb_tree_node_base*, unsigned long const&, std::_
Rb_tree<unsigned long, unsigned long, std::_Identity<unsigned long>, std::less<unsigned long>, std::allocator<unsigned long> >::_Alloc_node&
) (__node_gen=<synthetic pointer>..., __v=@0x7fda7586d688: 3297509811873971798, __p=0x7fda6c06df62, __x=<optimized out>, this=0x7fda7586d560
) at /usr/include/c++/8/bits/stl_tree.h:1776                                                                                                
#9  0x00007fda4b04da86 in std::_Rb_tree<unsigned long, unsigned long, std::_Identity<unsigned long>, std::less<unsigned long>, std::allocato
r<unsigned long> >::_M_insert_unique_<unsigned long const&, std::_Rb_tree<unsigned long, unsigned long, std::_Identity<unsigned long>, std::
less<unsigned long>, std::allocator<unsigned long> >::_Alloc_node>(std::_Rb_tree_const_iterator<unsigned long>, unsigned long const&, std::_
Rb_tree<unsigned long, unsigned long, std::_Identity<unsigned long>, std::less<unsigned long>, std::allocator<unsigned long> >::_Alloc_node&
) (__node_gen=<synthetic pointer>..., __v=@0x7fda7586d688: 3297509811873971798, __position=..., this=0x7fda7586d560)                        
    at /usr/include/c++/8/bits/stl_tree.h:2229                                                                                              
#10 0x00007fda4b04da86 in std::_Rb_tree<unsigned long, unsigned long, std::_Identity<unsigned long>, std::less<unsigned long>, std::allocato
r<unsigned long> >::_M_insert_unique<unsigned long const*>(unsigned long const*, unsigned long const*) (__last=0x7fda7586d690, __first=0x7fd
a7586d688, this=0x7fda7586d560) at /usr/include/c++/8/bits/stl_tree.h:2475                                                                  
#11 0x00007fda4b04da86 in std::set<unsigned long, std::less<unsigned long>, std::allocator<unsigned long> >::set(std::initializer_list<unsig
ned long>, std::less<unsigned long> const&, std::allocator<unsigned long> const&) (__a=..., __comp=..., __l=..., this=0x7fda7586d560)       
    at /usr/include/c++/8/bits/stl_set.h:247                                                                                                
#12 0x00007fda4b04da86 in ignition::gazebo::v5::EntityComponentManager::FindView<ignition::gazebo::v5::components::Component<std::add_lvalue
_reference<void>, ignition::gazebo::v5::components::JointTag, ignition::gazebo::v5::serializers::DefaultSerializer<std::add_lvalue_reference

<void> > >, ignition::gazebo::v5::components::Component<unsigned long, ignition::gazebo::v5::components::ParentEntityTag, ignition::gazebo::v5::serializers::DefaultSerializer<unsigned long> > >() const (this=this@entry=0x55ad49ec3b00)                                              
    at /home/chapulina/ws_edifice/src/ign-gazebo/include/ignition/gazebo/detail/EntityComponentManager.hh:522                               #13 0x00007fda4b04df0f in ignition::gazebo::v5::EntityComponentManager::EntitiesByComponents<ignition::gazebo::v5::components::Component<std
::add_lvalue_reference<void>, ignition::gazebo::v5::components::JointTag, ignition::gazebo::v5::serializers::DefaultSerializer<std::add_lval
ue_reference<void> > >, ignition::gazebo::v5::components::Component<unsigned long, ignition::gazebo::v5::components::ParentEntityTag, igniti
---Type <return> to continue, or q <return> to quit---                                                                                      
on::gazebo::v5::serializers::DefaultSerializer<unsigned long> > >(ignition::gazebo::v5::components::Component<std::add_lvalue_reference<void
>, ignition::gazebo::v5::components::JointTag, ignition::gazebo::v5::serializers::DefaultSerializer<std::add_lvalue_reference<void> > > cons
t&, ignition::gazebo::v5::components::Component<unsigned long, ignition::gazebo::v5::components::ParentEntityTag, ignition::gazebo::v5::seri
alizers::DefaultSerializer<unsigned long> > const&) const (this=this@entry=0x55ad49ec3b00)                                                  
    at /home/chapulina/ws_edifice/src/ign-gazebo/include/ignition/gazebo/detail/EntityComponentManager.hh:229                               
#14 0x00007fda4b041d0c in ignition::gazebo::gui::JointPositionController::Update(ignition::gazebo::v5::UpdateInfo const&, ignition::gazebo::
v5::EntityComponentManager&) (this=0x55ad4b3163a0, _ecm=...)                                                                        
    at /home/chapulina/ws_edifice/src/ign-gazebo/include/ignition/gazebo/components/Component.hh:392                                        
#15 0x00007fda9c189af3 in ignition::gazebo::v5::GuiRunner::Implementation::UpdatePlugins() (this=0x55ad49ec3b00)                            
    at /home/chapulina/ws_edifice/src/ign-gazebo/src/gui/GuiRunner.cc:208                                                                   
#16 0x00007fda9c189d03 in ignition::gazebo::v5::GuiRunner::OnState(ignition::msgs::SerializedStepMap const&) (this=0x55ad49eb68f0, _msg=...)
 at /home/chapulina/ws_edifice/src/ign-gazebo/src/gui/GuiRunner.cc:197                                                                      
#17 0x00007fda9c1a0ff5 in std::function<void (ignition::msgs::SerializedStepMap const&, ignition::transport::v10::MessageInfo const&)>::oper
ator()(ignition::msgs::SerializedStepMap const&, ignition::transport::v10::MessageInfo const&) const (__args#1=..., __args#0=..., this=0x7fd
a6c0dcb00) at /usr/include/c++/8/bits/std_function.h:682                                                                                    
#18 0x00007fda9c1a0ff5 in ignition::transport::v10::SubscriptionHandler<ignition::msgs::SerializedStepMap>::RunLocalCallback(google::protobu
f::Message const&, ignition::transport::v10::MessageInfo const&) (this=0x7fda6c0dcaa0, _msg=..., _info=...)                                 
    at /home/chapulina/ws_edifice/install/include/ignition/transport10/ignition/transport/SubscriptionHandler.hh:220