Add LOT implementation and benchmarks

README.md

@@ -0,0 +1,149 @@
+# LOT - Low Overhead Transport
+
+## General description
+
+This repository contains the C++ implementation of the shared memory mechanism described in the paper _Smart Pointers and Shared Memory Synchronisation for Efficient Inter-process Communication in ROS on an Autonomous Vehicle_ (IROS 2021) along with detailed build and deployment instructions.
+
+The implementation consists of a modified version of the `ros_comm` package from ROS Kinetic (v1.12.14) along with a [suite of benchmarks](./clients/benchmark).
+
+The code can be built either directly on the host machine _or_ within a Docker container via [the provided Docker script](./clients/benchmark/docker/Dockerfile). The former assumes the installation of ROS 1.12.14, Boost 1.58.0 and python2 on the host machine. The latter only requires Docker installation on the host machine. See [the Docker website](https://docs.docker.com/get-docker/) for details of setting up Docker on the host machine.
+
+The benchmarks - either the entire suite or individual tests - can be executed following the procedure described below. Currently, four protocols are supported: TCP, UDP, SHM and TZC.
+
+When executed from within the Docker containers, the benchmarks results are generated under `${HOME}/lot` folder. Otherwise, environment variable `lot_HOME` dictates the results folder.
+
+### Notes on provided code
+
+We apply the following bug fixes on top of ROS 1.12.14 in addition to our own code:
+ - _roslaunch_ [bug fix](https://github.com/ros/ros_comm/pull/1115).
+ - replace `boost::condition_variable` variable with `std::condition_variable` in [callback_queue.cpp](./clients/roscpp/src/libros/callback_queue.cpp).
+
+### Third party code
+
+[Boost.Process 1_65_0](https://www.boost.org/doc/libs/1_65_0/doc/html/process.html) has been imported into the source tree to support [unit tests](./test/test_roscpp/test/test_shm.cpp). We note that [Boost 1.65.0](https://github.com/boostorg/boost/tree/boost-1.65.0) is released under the [Boost Software License](http://www.boost.org/LICENSE_1_0.txt).
+
+For convenience of running the benchmarks, we have also included [TZC](https://github.com/qboticslabs/tzc_transport) in this repository. TZC was developed by a group researchers afilliated to Tsinghua University, China and University of Maryland, USA and is described more fully in the _TZC: Efficient Inter-Process Communication for Robotics Middleware with Partial Serialization_. Its authors have released it under the [BSD](https://github.com/qboticslabs/tzc_transport/blob/master/package.xml) license.
+
+## Cloning the repository and switching to the target branch
+
+Note that we clone the repository into `$HOME/ros_comm/src/ros_comm` in this step. We assume this directory in subsequent steps. If you prefer to use a different working directory, the subsequent steps will need to be appropriately modified.
+
+```
+git clone [email protected]:fiveai/ros_comm.git $HOME/ros_comm/src/ros_comm
+cd $HOME/ros_comm/src/ros_comm
+git checkout lot
+```
+
+## Building and running the benchmarks natively
+
+The following steps build the code directly on the host machine.
+
+### Compiling the entire workspace
+
+```
+cd $HOME/ros_comm
+source /opt/ros/kinetic/setup.sh  # this assumes ROS kinetic 1.12.14 is already installed
+catkin_make_isolated
+  --source ./src/  \
+  --build $HOME/ros_comm/build-release \
+  --devel $HOME/ros_comm/devel-release \
+  --install-space $HOME/ros_comm/install-release  \
+  --install \
+  --cmake-args -DCMAKE_BUILD_TYPE=Release
+```
+
+### Executing the benchmarks specified in launch.xml overriding some of the parameters
+
+```
+cd $HOME/ros_comm
+source ./devel-release/setup.sh
+roslaunch --screen -v  benchmark launch.xml   \
+  use_case:=5p1s_same_host sub_stats_file_path:=$HOME \
+  transport:=shm pub_queue_size:=200 pub_image_count:=200
+```
+
+## Building and running benchmarks within Docker containers
+
+### Building the Docker image
+
+```
+cd ~/ros_comm/src/ros_comm/clients/benchmark/docker
+export DOCKER_BUILDKIT=1
+docker build --ssh default --tag lot .
+```
+
+### Executing the benchmark suite, deploying the ROS nodes in separate docker images
+
+Assuming the Docker container built as above, with a tag of `lot`:
+
+```
+cd ~/ros_comm/src/ros_comm/clients/benchmark/docker
+docker-compose up
+
+# on a different terminal attach to node1; note it must be node1!
+docker attach node1
+```
+
+And then, from within the container, our benchmarks can be executed using commands of the form:
+
+```
+python2 /ros_comm/src/ros_comm/clients/benchmark/execute.py --tcp=no --shm=yes --use_case=5p1s_separate_docker
+```
+
+The next sections provides examples of such commands.
+
+### Benchmark execution command examples
+
+1. Execute the benchmarks suite for 1p5s using TZC protocol, in separate Docker containers, enforcing the subscribers start up order, allocating 16GB of shared memory to be used by the publishers, and telling the subscriber to wait 15 seconds before starting publishing messages, `TCP_NODELAY` enabled.
+
+```
+python2 /ros_comm/src/ros_comm/clients/benchmark/execute.py      \
+        --tcp=no --shm=no --tzc=yes --udp=no                     \
+        --use_case=1p5s_separate_docker --no_pool=yes --pool=no  \
+        --extra_params sub_enable_synch_startup:=true            \
+                       pub_extra_delay_ms:=15000                 \
+                       shm_size_mega_bytes:=16000
+```
+
+2. Execute the benchmarks suite for 5p1s using TZC protocol, in separate Docker containers, with each publisher waiting for the subscriber to establish connection and with the subscriber start up delayed by 15secs, `TCP_NODELAY` enabled.
+
+```
+python2 /ros_comm/src/ros_comm/clients/benchmark/execute.py      \
+        --tcp=no --shm=no --tzc=yes --udp=no                     \
+        --use_case=5p1s_separate_docker --no_pool=yes --pool=no  \
+        --extra_params  pub_wait_for_subscribers:=true           \
+                        sub_extra_delay_ms:=15000
+```
+
+3. Execute the benchmarks suite for 5p1s using TZC protocol, in separate Docker containers, with each publisher waiting for the subscriber to establish connection and with the publishers start up order enforced, `TCP_NODELAY` enabled, overriding the default results path.
+
+```
+python2 /ros_comm/src/ros_comm/clients/benchmark/execute.py      \
+        --tcp=no --shm=no --tzc=yes --udp=no                     \
+        --use_case=5p1s_separate_docker --no_pool=yes --pool=no  \
+        --extra_params  pub_wait_for_subscribers:=true           \
+                        pub_enable_synch_startup:=true           \
+                        sub_stats_file_path:=/path/to/results
+```
+
+4. Execute the benchmarks suite for 5p1s using SHM protocol, in separate Docker containers, with each publisher waiting for the subscriber to establish connection and with the publishers start up order enforced, and image pools disabled.
+
+```
+python2 /ros_comm/src/ros_comm/clients/benchmark/execute.py      \
+        --tcp=no --shm=yes --tzc=no --udp=no                     \
+        --use_case=5p1s_separate_docker --no_pool=yes --pool=no  \
+        --extra_params  pub_wait_for_subscribers:=true           \
+                        pub_enable_synch_startup:=true
+```
+
+5. Execute _one single test_ using TCP protocol, in same docker container, with image pool disabled.
+
+```
+roslaunch --screen -v  benchmark launch.xml     \
+        pub_image_count:=200                    \
+        use_case:=1p5s_same_docker              \
+        transport:=tcp                          \
+        image_width_pixels:=512                 \
+        image_height_pixels:=512                \
+        pub_pool_size:=0
+```

clients/benchmark/CMakeLists.txt

@@ -0,0 +1,82 @@
+# ***************************************************************************************************
+# * Copyright Five AI 2021.
+# * All rights reserved.
+# ***************************************************************************************************
+
+cmake_minimum_required(VERSION 2.8.3)
+project(benchmark)
+
+add_compile_options(-std=c++11)
+
+find_package(catkin REQUIRED COMPONENTS
+  roscpp
+  rospy
+  std_msgs
+)
+
+find_package(Boost REQUIRED COMPONENTS system)
+
+## The catkin_package macro generates cmake config files for your package
+## Declare things to be passed to dependent projects
+## INCLUDE_DIRS: uncomment this if your package contains header files
+## LIBRARIES: libraries you create in this project that dependent projects also need
+## CATKIN_DEPENDS: catkin_packages dependent projects also need
+## DEPENDS: system dependencies of this project that dependent projects also need
+catkin_package(
+#  INCLUDE_DIRS include
+#  LIBRARIES benchmark
+#  CATKIN_DEPENDS roscpp rospy std_msgs
+#  DEPENDS system_lib
+)
+
+include_directories(
+  include
+  ${catkin_INCLUDE_DIRS}
+)
+
+add_library(${PROJECT_NAME}_util
+    STATIC
+    src/Util.cpp
+)
+
+
+add_executable(${PROJECT_NAME}_publisher src/Publisher.cpp)
+add_executable(${PROJECT_NAME}_subscriber src/Subscriber.cpp)
+add_executable(${PROJECT_NAME}_monitor src/Monitor.cpp)
+
+add_custom_target(${PROJECT_NAME}_all)
+add_dependencies(${PROJECT_NAME}_all
+    ${PROJECT_NAME}_publisher
+    ${PROJECT_NAME}_subscriber
+    ${PROJECT_NAME}_monitor
+)
+
+target_link_libraries(${PROJECT_NAME}_publisher
+    ${catkin_LIBRARIES}
+    ${PROJECT_NAME}_util
+    rt
+)
+
+target_link_libraries(${PROJECT_NAME}_subscriber
+    ${PROJECT_NAME}_util
+    ${catkin_LIBRARIES}
+    rt
+)
+
+target_link_libraries(${PROJECT_NAME}_monitor
+    ${PROJECT_NAME}_util
+    ${catkin_LIBRARIES}
+)
+
+install(
+  TARGETS 
+    ${PROJECT_NAME}_publisher
+    ${PROJECT_NAME}_subscriber
+    ${PROJECT_NAME}_monitor
+  RUNTIME DESTINATION ${CATKIN_PACKAGE_BIN_DESTINATION}
+)
+
+ install(FILES
+    launch/launch.xml
+  DESTINATION ${CATKIN_PACKAGE_SHARE_DESTINATION}
+)

clients/benchmark/docker/Dockerfile

@@ -0,0 +1,71 @@
+# syntax=docker/dockerfile:experimental
+
+# WARNING: The top line must go first in the file and is required to
+#          allow cloning the git repository through ssh. More details
+#          can be found at: https://medium.com/@tonistiigi/build-secrets-and-ssh-forwarding-in-docker-18-09-ae8161d066
+# To build this docker image run:
+#    1. export DOCKER_BUILDKIT=1
+#    2. docker build --ssh default --tag XXX .
+
+# ***************************************************************************************************
+# * Copyright Five AI 2021.
+# * All rights reserved.
+# ***************************************************************************************************
+
+FROM osrf/ros:kinetic-desktop-full-xenial
+
+RUN apt-get update && apt-get install -y \
+            openssh-client \
+            nano \
+            htop \
+            openssh-server \
+            iputils-ping    \
+            net-tools
+
+RUN mkdir -p -m 0600 ~/.ssh && \ssh-keyscan github.com >> ~/.ssh/known_hosts
+RUN mkdir /var/run/sshd
+RUN echo 'root:abcde' | chpasswd
+RUN sed -i 's/PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config
+
+# SSH login fix. Otherwise user is kicked off after login
+RUN sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd
+ENV NOTVISIBLE "in users profile"
+
+RUN echo "export VISIBLE=now" >> /etc/profile
+RUN echo "/usr/sbin/sshd" >> ~/.bashrc
+RUN echo "export ROSLAUNCH_SSH_UNKNOWN=1" >> ~/.bashrc
+RUN echo "source /ros_comm/devel-release/setup.sh" >> ~/.bashrc
+RUN echo "mkdir -p \${LOT_HOME}" >> ~/.bashrc
+# This line is need to workaround a roslaunch bug where multiple instances of roscore are launched 
+RUN echo "export ROS_MASTER_URI=http://\$(uname -n):11311" >> ~/.bashrc
+RUN echo "nano /ros_comm/src/ros_comm/clients/benchmark/launch/launch.xml" >> ~/.bash_history
+RUN echo "catkin_make_isolated  \
+--source ./src/  \
+--build /ros_comm/build-release \
+--devel /projects/ros_comm/devel-release \
+--install-space /ros_comm/install-release  \
+--pkg=benchmark --install --cmake-args -DCMAKE_BUILD_TYPE=Release && \
+roslaunch --screen -v  benchmark launch.xml pub_image_count:=2000 \
+sub_stats_file_path:=/root/tmp use_case:=1p1s_same_host" >> ~/.bash_history
+
+RUN echo "python2 /ros_comm/src/ros_comm/clients/benchmark/execute.py --use_case=1p1s_same_host --extra_params=''" >> ~/.bash_history
+
+
+RUN --mount=type=ssh git clone [email protected]:fiveai/ros_comm.git /ros_comm/src/ros_comm && \
+    cd /ros_comm/src/ros_comm     && \
+    git pull                      && \
+    git checkout lot              && \
+    cd /ros_comm
+
+RUN  /bin/bash -c "source /opt/ros/kinetic/setup.sh   &&        \
+    catkin_make_isolated                                        \
+    --source        /ros_comm/src/                       \
+    --build         /ros_comm/build-release              \
+    --devel         /ros_comm/devel-release              \
+    --install-space /ros_comm/install-release            \
+    --install                                                   \
+    --cmake-args -DCMAKE_BUILD_TYPE=Release"
+
+WORKDIR /ros_comm
+
+EXPOSE 22

clients/benchmark/docker/docker-compose.yaml

@@ -0,0 +1,48 @@
+# ***************************************************************************************************
+# * Copyright Five AI 2021.
+# * All rights reserved.
+# ***************************************************************************************************
+
+version: "3.7"
+
+x-base_node: &base_node
+    image: lot
+    ipc: host
+    volumes: 
+        - "${HOME}:${HOME}"
+    stdin_open: true # docker run -i
+    tty: true        # docker run -t
+    environment:
+        - LOT_HOME=${HOME}/lot
+
+services:
+    # define the nodes as copies of the base node
+    node_1:
+        << : *base_node
+        container_name: node1
+        hostname: node1
+
+    node_2:
+        << : *base_node
+        container_name: node2
+        hostname: node2
+
+    node_3:
+        << : *base_node
+        container_name: node3
+        hostname: node3
+
+    node_4:
+        << : *base_node
+        container_name: node4
+        hostname: node4
+
+    node_5:
+        << : *base_node
+        container_name: node5
+        hostname: node5
+
+    node_6:
+        << : *base_node
+        container_name: node6
+        hostname: node6