🧑‍🌾 TestServiceIntrospection.service_introspection_enable_disable_events failing in Rolling, Jazzy and Iron Connext

[Crola1702]

Bug report

Required Info:

• Operating System:
  • Ubuntu Noble
• Installation type:
  • Source
• Version or commit hash:
  • Rolling
• DDS implementation:
  • RTI Connext
• Client library (if applicable):
  • Rclcpp

Steps to reproduce issue

1. Run a build in Rci__nightly-connext_ubuntu_noble_amd64
2. See test regression fail

Additional information

Reference build:

• projectroot.test.rclcpp.test_service_introspection
• rclcpp.TestServiceIntrospection.service_introspection_enable_disable_events
• rclcpp.TestServiceIntrospection.service_introspection_enable_disable_event_content

Log output:

[ RUN      ] TestServiceIntrospection.service_introspection_enable_disable_events
RTI Connext DDS Non-commercial license is for academic, research, evaluation and personal use only. USE FOR COMMERCIAL PURPOSES IS PROHIBITED. See RTI_LICENSE.TXT for terms. Download free tools at rti.com/ncl. License issued to Non-Commercial User [email protected] For non-production use only.
Expires on 00-jan-00 See www.rti.com for more information.
/tmp/ws/src/ros2/rclcpp/rclcpp/test/rclcpp/test_service_introspection.cpp:201: Failure
Expected equality of these values:
  events.size()
    Which is: 1
  2U
    Which is: 2

/tmp/ws/src/ros2/rclcpp/rclcpp/test/rclcpp/test_service_introspection.cpp:218: Failure
Expected equality of these values:
  events.size()
    Which is: 3
  4U
    Which is: 4

[  FAILED  ] TestServiceIntrospection.service_introspection_enable_disable_events (3309 ms)
[ RUN      ] TestServiceIntrospection.service_introspection_enable_disable_event_content
RTI Connext DDS Non-commercial license is for academic, research, evaluation and personal use only. USE FOR COMMERCIAL PURPOSES IS PROHIBITED. See RTI_LICENSE.TXT for terms. Download free tools at rti.com/ncl. License issued to Non-Commercial User [email protected] For non-production use only.
Expires on 00-jan-00 See www.rti.com for more information.
/tmp/ws/src/ros2/rclcpp/rclcpp/test/rclcpp/test_service_introspection.cpp:340: Failure
Expected equality of these values:
  events.size()
    Which is: 3
  4U
    Which is: 4

[  FAILED  ] TestServiceIntrospection.service_introspection_enable_disable_event_content (1307 ms)

It's a flaky with a high percentage failure rate:

job_name	last_fail	first_fail	build_count	failure_count	failure_percentage
Ici__nightly-connext_ubuntu_jammy_amd64	2024-08-28	2024-08-14	6	4	66.67
Rci__nightly-connext_ubuntu_noble_amd64	2024-08-27	2024-08-14	12	11	91.67
Jci__nightly-connext_ubuntu_noble_amd64	2024-08-26	2024-08-13	6	5	83.33

[fujitatomoya]

i cannot reproduce this issue with my local environment.

root@tomoyafujita:~/ros2_ws/colcon_ws# RMW_IMPLEMENTATION=rmw_connextdds
root@tomoyafujita:~/ros2_ws/colcon_ws# colcon test --event-handlers console_direct+ --retest-until-fail 100 --packages-select rclcpp --ctest-args -R test_service_introspection
...
The following tests passed:
        test_service_introspection

100% tests passed, 0 tests failed out of 1

Label Time Summary:
gmock     =   1.24 sec*proc (1 test)
mimick    =   1.24 sec*proc (1 test)

Total Test time (real) = 125.27 sec
Finished <<< rclcpp [2min 6s]

Summary: 1 package finished [2min 7s]

according to the log information in the report, client events are okay as expected, but server side missing one event.
and we can see at least one service event is received from service server, that means publisher and topic are created okay.
it would be worth to try to extend the timeout from 1000msec to 3000msec, but i am not sure if that actually fixes the problem.

root cause of this issue seems like almost same with #2588 (comment), that is dropping the messages (some messages are missing, but some are okay) using same publisher object.

[fujitatomoya]

@fgallegosalido this test also fails with rmw_connextdds only, can you take a look at my comment on #2651? i was trying to address this unstable problem, but it does not work as expected.

[fgallegosalido]

I will look into it and let you know what I find.

[fujitatomoya]

@fgallegosalido just checking, any updates on this?

[fgallegosalido]

@fujitatomoya sorry, not much was done. I was able to reproduce the issue and started working on it, but couldn't find the root cause, so I moved to do other things. I will resume the work on it as soon as possible

[fujitatomoya]

@fgallegosalido no worries, thanks for working on this. i will revisit #2651 to see if i can find some clue on this.