Improve backtrace generation using Backward #63
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Addisu Z. Taddese <[email protected]>
github-actions
bot
added
🌱 garden
chapulina
reviewed
Oct 11, 2021
Signed-off-by: Addisu Z. Taddese <[email protected]>
Signed-off-by: Addisu Z. Taddese <[email protected]>
Signed-off-by: Addisu Z. Taddese <[email protected]>
mjcarroll
reviewed
Oct 12, 2021
Signed-off-by: Addisu Z. Taddese <[email protected]>
Signed-off-by: Addisu Z. Taddese <[email protected]>
chapulina
reviewed
Oct 12, 2021
There was a problem hiding this comment.
We recently removed a bunch of stuff from ign-tools because it didn't have any C++ code (gazebo-tooling/release-tools#472), but now it will have. Do we need to revert any parts of these PRs?
- Remove all cmake code related to tests since it is not used #54
- Remove all cmake code related to compilations #55
Related, we also made changes on how downstream libraries find ign-tools, for example gazebosim/gz-msgs#166
CC @j-rivero
I added a raise(SIGSEGV) somewhere in ign-gazebo to demonstrate:
My go-to is this 😅
ign gazebo rolling_shapes.sdf -z 10000000000000 -r
It feels nicer to see it crash now:
backtrace
ODE INTERNAL ERROR 1: assertion "aabbBound >= dMinIntExact && aabbBound < dMaxIntExact" failed in collide() [collision_space.cpp:460]
Stack trace (most recent call last):
#31 Object "/usr/lib/x86_64-linux-gnu/libruby-2.7.so.2.7", at 0x7fb4935d1daa, in rb_vm_invoke_bmethod
#30 Object "/usr/lib/x86_64-linux-gnu/libruby-2.7.so.2.7", at 0x7fb4935c5025, in rb_dtrace_setup #29 Object "/usr/lib/x86_64-linux-gnu/libruby-2.7.so.2.7", at 0x7fb4935231be, in rb_exec_async_signal_safe
#28 Object "/usr/lib/x86_64-linux-gnu/libruby-2.7.so.2.7", at 0x7fb4934635d2, in rb_protect
#27 Object "/usr/lib/x86_64-linux-gnu/libruby-2.7.so.2.7", at 0x7fb4935dc9f0, in rb_yield
#26 Object "/usr/lib/x86_64-linux-gnu/libruby-2.7.so.2.7", at 0x7fb4935cf86f, in rb_vm_exec
#25 Object "/usr/lib/x86_64-linux-gnu/libruby-2.7.so.2.7", at 0x7fb4935c9130, in rb_check_funcall
#24 Object "/usr/lib/x86_64-linux-gnu/libruby-2.7.so.2.7", at 0x7fb4935b8405, in rb_f_notimplement
#23 Object "/usr/lib/x86_64-linux-gnu/libruby-2.7.so.2.7", at 0x7fb4935d1daa, in rb_vm_invoke_bmethod #22 Object "/usr/lib/x86_64-linux-gnu/libruby-2.7.so.2.7", at 0x7fb4935c5025, in rb_dtrace_setup #21 Object "/usr/lib/x86_64-linux-gnu/ruby/2.7.0/fiddle.so", at 0x7fb48fa24714, in Init_fiddle #20 Object "/usr/lib/x86_64-linux-gnu/libruby-2.7.so.2.7", at 0x7fb4935966d9, in rb_nogvl #19 Object "/usr/lib/x86_64-linux-gnu/ruby/2.7.0/fiddle.so", at 0x7fb48fa248fb, in Init_fiddle #18 Object "/usr/lib/x86_64-linux-gnu/libffi.so.7", at 0x7fb48fa19409, in ffi_closure_free
#17 Object "/usr/lib/x86_64-linux-gnu/libffi.so.7", at 0x7fb48fa19ff4, in ffi_prep_go_closure
#16 Source "/home/chapulina/dev_focal/ws_citadel/src/ign-gazebo/src/ign.cc", line 360, in runServer [0x7fb48edfe556]
357: ignition::gazebo::Server server(serverConfig);
358:
359: // Run the server
> 360: server.Run(true, _iterations, _run == 0);
361:
362: igndbg << "Shutting down ign-gazebo-server" << std::endl; 363: return 0; #15 Source "/home/chapulina/dev_focal/ws_citadel/src/ign-gazebo/src/Server.cc", line 233, in ignition::gazebo::v3::Server::Run(bool, unsigned long, bool) [0x7fb48ea11fdb] 230: }
231:
232: if (_blocking)
> 233: return this->dataPtr->Run(_iterations);
234:
235: // Make sure two threads are not created
236: std::unique_lock<std::mutex> lock(this->dataPtr->runMutex);
#14 Source "/home/chapulina/dev_focal/ws_citadel/src/ign-gazebo/src/ServerPrivate.cc", line 165, in ignition::gazebo::v3::ServerPrivate::R
un(unsigned long, std::optional<std::condition_variable*>) [0x7fb48ea1ad8b]
162: // simulation runner, and we can avoid using the thread pool.
163: if (this->simRunners.size() == 1) 164: {
> 165: result = this->simRunners[0]->Run(_iterations);
166: }
167: else
168: {
#13 Source "/home/chapulina/dev_focal/ws_citadel/src/ign-gazebo/src/SimulationRunner.cc", line 767, in ignition::gazebo::v3::SimulationRunner::Run(unsigned long) [0x7fb48ea2851a]
764: }
765: else
766: {
> 767: this->Step(this->currentInfo);
768: }
769:
770: // Handle Server::RunOnce(false) in which a single paused run is executed
#12 Source "/home/chapulina/dev_focal/ws_citadel/src/ign-gazebo/src/SimulationRunner.cc", line 802, in ignition::gazebo::v3::SimulationRunner::Step(ignition::gazebo::v3::UpdateInfo const&) [0x7fb48ea27cfa]
799: this->ProcessSystemQueue();
800:
801: // Update all the systems.
> 802: this->UpdateSystems();
803:
804: if (!this->Paused() && this->pendingSimIterations > 0)
805: {
#11 Source "/home/chapulina/dev_focal/ws_citadel/src/ign-gazebo/src/SimulationRunner.cc", line 553, in ignition::gazebo::v3::SimulationRunner::UpdateSystems() [0x7fb48ea20811]
550: {
551: IGN_PROFILE("Update");
552: for (auto& system : this->systemsUpdate)
> 553: system->Update(this->currentInfo, this->entityCompMgr);
554: }
555:
556: {
#10 Source "/home/chapulina/dev_focal/ws_citadel/src/ign-gazebo/src/systems/physics/Physics.cc", line 586, in ignition::gazebo::v3::systems::Physics::Update(ignition::gazebo::v3::UpdateInfo const&, ignition::gazebo::v3::EntityComponentManager&) [0x7fb4781fdefc]
583: // Only step if not paused.
584: if (!_info.paused)
585: {
> 586: this->dataPtr->Step(_info.dt);
587: }
588: this->dataPtr->UpdateSim(_ecm);
#9 | Source "/home/chapulina/dev_focal/ws_citadel/src/ign-gazebo/src/systems/physics/Physics.cc", line 1767, in ignition::gazebo::v3::systems::PhysicsPrivate::Step(std::chrono::duration<long, std::ratio<1l, 1000000000l> > const&)
| 1765: for (const auto &world : this->entityWorldMap.Map())
| 1766: {
| >1767: world.second->Step(output, state, input);
| 1768: }
| 1769: }
Source "/home/chapulina/dev_focal/ws_citadel/install/include/ignition/physics2/ignition/physics/ForwardStep.hh", line 162, in ignition::physics::ForwardStep::World<ignition::physics::FeaturePolicy<double, 3ul>, ignition::gazebo::v3::systems::PhysicsPrivate::MinimumFeatureList>::Step(ignition::physics::SpecifyData<ignition::physics::RequireData<ignition::physics::WorldPoses>, ignition::physics::ExpectData<ignition::physics::Contacts, ignition::physics::JointPositions> >&, ignition::physics::CompositeData&, ignition::physics::ExpectData<ignition::physics::ApplyExternalForceTorques, ignition::physics::ApplyGeneralizedForces, ignition::physics::VelocityControlCommands, ignition::physics::ServoControlCommands> const&) [0x7fb4781f5a64]
159: {
160: public: void Step(Output &_h, State &_x, const Input &_u)
161: {
> 162: this->template Interface<ForwardStep>()->
163: WorldForwardStep(this->identity, _h, _x, _u);
164: }
165: };
#8 Source "/home/chapulina/dev_focal/ws_citadel/src/ign-physics/dartsim/src/SimulationFeatures.cc", line 54, in ignition::physics::dartsim::SimulationFeatures::WorldForwardStep(ignition::physics::Identity const&, ignition::physics::SpecifyData<ignition::physics::RequireData<ignition::physics::WorldPoses>, ignition::physics::ExpectData<ignition::physics::Contacts, ignition::physics::JointPositions> >&, ignition::physics::CompositeData&, ignition::physics::ExpectData<ignition::physics::ApplyExternalForceTorques, ignition::physics::ApplyGeneralizedForces, ignition::physics::VelocityControlCommands, ignition::physics::ServoControlCommands> const&) [0x7fb45c69b02d]
51: }
52:
53: // TODO(MXG): Parse input
> 54: world->step();
55: // TODO(MXG): Fill in output and state
56: }
#7 Object "/usr/lib/x86_64-linux-gnu/libdart.so.6", at 0x7fb45c433d20, in dart::simulation::World::step(bool)
#6 Object "/usr/lib/x86_64-linux-gnu/libdart.so.6", at 0x7fb45c4187b5, in dart::constraint::ConstraintSolver::solve()
#5 Object "/usr/lib/x86_64-linux-gnu/libdart.so.6", at 0x7fb45c41727d, in dart::constraint::ConstraintSolver::updateConstraints()
#4 Object "/usr/lib/x86_64-linux-gnu/libdart-collision-ode.so.6", at 0x7fb46800fb6b, in dart::collision::OdeCollisionDetector::collide(dart::collision::CollisionGroup*, dart::collision::CollisionOption const&, dart::collision::CollisionResult*)
#3 Object "/usr/lib/x86_64-linux-gnu/libode.so.8", at 0x7fb45b22c456, in dxHashSpace::collide(void*, void (*)(void*, dxGeom*, dxGeom*))
#2 Object "/usr/lib/x86_64-linux-gnu/libode.so.8", at 0x7fb45b234c57, in dDebug
#1 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7fb4931ea858, in abort
#0 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7fb49320b18b, in gsignal
Aborted (Signal sent by tkill() 287825 1001)
Signed-off-by: Addisu Z. Taddese <[email protected]>
|
I found an issue with pretty printed stack traces while I was running ign-gazebo. The stack trace only showed the bottom frame only So I've disabled it for now. It'll still show a useful backtrace, just not the code snippets. eg. |
Aw those were nice. But it's better to get the full trace than super pretty trace |
mjcarroll
approved these changes
Oct 14, 2021
j-rivero
reviewed
Oct 14, 2021
Also fixes how the STACK_DETAILS_AUTO_DETECT cmake variable is set to ensure auto detection is disabled. Signed-off-by: Addisu Z. Taddese <[email protected]>
chapulina
approved these changes
Oct 15, 2021
|
I think this change is breaking CI jobs that install ign-tools with colcon: |
scpeters
reviewed
Oct 15, 2021
| @@ -267,9 +281,23 @@ if ARGV.include?('--versions') | |||
| exit(0) | |||
| end | |||
|
|
|||
| # Start Backward before loading plugins | |||
| backward_library_name = if (/darwin/ =~ RUBY_PLATFORM).nil? | |||
| '$<TARGET_FILE_NAME:backward>' | |||
There was a problem hiding this comment.
this generator expression means we now need to generate the ign file instead of just configuring it from ign.in
this is causing windows CI failures
There was a problem hiding this comment.
There was a problem hiding this comment.
never mind, you are doing generation, but we need an extra subfolder. We had to do this in gazebosim/gz-transport#216; I'll submit a fix
|
It also looks like this introduced a Windows warning downstream: https://build.osrfoundation.org/job/ign_msgs-pr-win/7/consoleFull |
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
🦟 Bug fix
Fixes #4
Summary
As described in #4, the backtrace currently generated includes a lot of information related to ruby that is not useful for diagnosing a crash. This PR vendors
Backwardand creates a small shared library that initializes it to handle the following signalsThis library gets loaded before
ignloads commands from other libraries so that it catches any signal generated by our C++ libraries. I believe @mjcarroll had added Backward to ign-gazebo before, but when we moved over from using standalone binaries to usingign, that functionality was lost.I added a
raise(SIGSEGV)somewhere in ign-gazebo to demonstrate:Before this PR
With Backward (this PR)
Checklist
codecheckpassed (See contributing)Note to maintainers: Remember to use Squash-Merge