Skip to content

Commit

Permalink
# This is a combination of 49 commits.
Browse files Browse the repository at this point in the history 
# This is the 1st commit message:

Add LOT implementation and benchmarks

# The commit message #2 will be skipped:

# add benchmarks

# The commit message #3 will be skipped:

# add TZC and SHM header files

# The commit message #4 will be skipped:

# backport Boost 1.65.0

# The commit message #5 will be skipped:

# port ros_comms/test folder

# The commit message #6 will be skipped:

# fixup! port ros_comms/tools folder

# The commit message #7 will be skipped:

# port ros_comms/utilities folder

# The commit message #8 will be skipped:

# remove metrics.cpp/hpp

# The commit message #9 will be skipped:

# enable C++11 in rosbag package

# The commit message #10 will be skipped:

# defined missing macro ROSCPP_START_STOP_STREAM_DEBUG

# The commit message #11 will be skipped:

# remove feature file

# The commit message #12 will be skipped:

# backporting roslaunch fix - see ros/ros_comm#1115

# The commit message #13 will be skipped:

# relocate and update README file

# The commit message #14 will be skipped:

# remove references to Five repos in the Docker file

# The commit message #15 will be skipped:

# add copyright to various files

# The commit message #16 will be skipped:

# fix typo in README file + minor cosmetic fixes

# The commit message #17 will be skipped:

# make links relative

# The commit message #18 will be skipped:

# fix broken link

# The commit message #19 will be skipped:

# add copyright to Dockerfile

# The commit message #20 will be skipped:

# add explanatory comments regarding the usage of "experimental" Docker keyword

# The commit message #21 will be skipped:

# remove reference to Five registry from docker-compose file

# The commit message #22 will be skipped:

# some more copyright fixes + remove commented code

# The commit message #23 will be skipped:

# trim the collection of marcos

# The commit message #24 will be skipped:

# remove unsued code from String.h

# The commit message #25 will be skipped:

# removed unused type

# The commit message #26 will be skipped:

# several cosmetic fixes

# The commit message #27 will be skipped:

# use static_cast and replace shm alias by shaed memory

# The commit message #28 will be skipped:

# fix several typos

# The commit message #29 will be skipped:

# remove redundant comments

# The commit message #30 will be skipped:

# remove commented code + some extra blank lines

# The commit message #31 will be skipped:

# revert unnecessary formatting change to ROS source file

# The commit message #32 will be skipped:

# enhance comment in shm_puller.cpp

# The commit message #33 will be skipped:

# remove "platform" intermediate namespace

# The commit message #34 will be skipped:

# update error message

# The commit message #35 will be skipped:

# remove confusing comment

# The commit message #36 will be skipped:

# now benchmark results folder is specified via SHM_ON_ROS_HOME env variable in both docker-compose and launch.xml files
#
# create SHM_ON_ROS_HOME folder upon user log in in the container
#
# set SHM_ON_ROS_HOME in the docker-compose file
#
# escape $ in docker file

# The commit message #37 will be skipped:

# update README file

# The commit message #38 will be skipped:

# generate MD5 sums for the newly introduced SHM ROS messages

# The commit message #39 will be skipped:

# remove ofending comment from roscpp test cmake file

# The commit message #40 will be skipped:

# add comments to various source files

# The commit message #41 will be skipped:

# placing SHM messages in fiveai::shm_msgs to reduce code verbosity

# The commit message #42 will be skipped:

# remove fiveai sufix

# The commit message #43 will be skipped:

# now update README file

# The commit message #44 will be skipped:

# updates to docker and README files after renaming branch to fiveshm

# The commit message #45 will be skipped:

# enhance comment clarity

# The commit message #46 will be skipped:

# update github repo to point to fiveai

# The commit message #47 will be skipped:

# Readme tweaks and remove SHM_ON_ROS_HOME references.

# The commit message #48 will be skipped:

# Address review comments.

# The commit message #49 will be skipped:

# Wrapped code in appropriate markdown.
  • Loading branch information
Costin Iordache committed Jul 16, 2021
1 parent fb6a6bd commit e7423d7
Show file tree
Hide file tree
Showing 220 changed files with 24,063 additions and 70 deletions.
138 changes: 138 additions & 0 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
# FiveSHM

## General description

This repository contains the C++ implementation of the shared memory mechanism described in the paper _Smart Pointers and Shared Memory Synchronisation for Efficient Inter-process Communication in ROS on an Autonomous Vehicle_ (IROS 2021) along with detailed build and deployment instructions.

The implementation consists of a modified version of the `ros_comm` package from ROS Kinetic (v1.12.14) along with a [suite of benchmarks](./clients/benchmark).

The code can be built either directly on the host machine _or_ within a Docker container via [the provided Docker script](./clients/benchmark/docker/Dockerfile). The former assumes the installation of ROS 1.12.14, Boost 1.58.0 and python2 on the host machine. The latter only requires Docker installation on the host machine. See [the Docker website](https://docs.docker.com/get-docker/) for details of setting up Docker on the host machine.

The benchmarks - either the entire suite or individual tests - can be executed following the procedure described below. Currently, four protocols are supported: TCP, UDP, SHM and TZC.

When executed from within the Docker containers, the benchmarks results are generated under `${HOME}/fiveshm` folder. Otherwise, environment variable `FIVESHM_HOME` dictates the results folder.

### Notes on provided code

We apply the following bug fixes on top of ROS 1.12.14 in addition to our own code:
- _roslaunch_ [bug fix](https://github.com/ros/ros_comm/pull/1115).
- replace `boost::condition_variable` variable with `std::condition_variable` in [callback_queue.cpp](./clients/roscpp/src/libros/callback_queue.cpp).

### Third party code

[Boost.Process 1_65_0](https://www.boost.org/doc/libs/1_65_0/doc/html/process.html) has been imported into the source tree to support [unit tests](./test/test_roscpp/test/test_shm.cpp). We note that Boost is released under the [Boost Software License](http://www.boost.org/LICENSE_1_0.txt).

For convenience of running the benchmarks, we have also included TZC in this repository. TZC was developed by a group researchers afilliated to Tsinghua University, China and University of Maryland, USA and is described more fully in the _TZC: Efficient Inter-Process Communication for Robotics Middleware with Partial Serialization_. Its authors have released it under the [BSD](https://github.com/qboticslabs/tzc_transport/blob/master/package.xml) license.

## Cloning the repository and switching to the target branch

Note that we clone the repository into `$HOME/ros_comm/src/ros_comm` in this step. We assume this directory in subsequent steps. If you prefer to use a different working directory, the subsequent steps will need to be appropriately modified.

```
git clone [email protected]:fiveai/ros_comm.git $HOME/ros_comm/src/ros_comm
cd $HOME/ros_comm/src/ros_comm
git checkout fiveshm
```

## Building and running the benchmarks natively

The following steps build the code directly on the host machine.

### Compiling the entire workspace

```
cd $HOME/ros_comm
source /opt/ros/kinetic/setup.sh # this assumes ROS kinetic 1.12.14 is already installed
catkin_make_isolated
--source ./src/ \
--build $HOME/ros_comm/build-release \
--devel $HOME/ros_comm/devel-release \
--install-space $HOME/ros_comm/install-release \
--install \
--cmake-args -DCMAKE_BUILD_TYPE=Release
```

### Executing the benchmarks specified in launch.xml overriding some of the parameters

```
cd $HOME/ros_comm
source ./devel-release/setup.sh
roslaunch --screen -v benchmark launch.xml \
use_case:=5p1s_same_host sub_stats_file_path:=$HOME \
transport:=shm pub_queue_size:=200 pub_image_count:=200
```

## Building and running benchmarks within Docker containers

### Building the Docker image

```
cd ~/ros_comm/src/ros_comm/clients/benchmark/docker
export DOCKER_BUILDKIT=1
docker build --ssh default --tag fiveshm .
```

### Executing the benchmark suite, deploying the ROS nodes in separate docker images

Assuming the Docker container built as above, with a tag of `fiveshm`:

```
cd ~/ros_comm/src/ros_comm/clients/benchmark/docker
docker-compose up
# on a different terminal attach to node1; note it must be node1!
docker attach node1
```

And then, from within the container, our benchmarks can be executed using commands of the form:

```
python2 /ros_comm/src/ros_comm/clients/benchmark/execute.py --tcp=no --shm=yes --use_case=5p1s_separate_docker
```

The next sections provides examples of such commands.

### Benchmark execution command examples

Execute the benchmarks for 1p5s using TZC protocol, in separate Docker containers, enforcing the subscribers start up order, allocating 16GB of shared memory to be used by the publishers, and telling the subscriber to wait 15 seconds before starting publishing messages, `TCP_NODELAY` enabled.
```
python2 /ros_comm/src/ros_comm/clients/benchmark/execute.py \
--tcp=no --shm=no --tzc=yes --udp=no \
--use_case=1p5s_separate_docker --no_pool=yes --pool=no \
--extra_params sub_enable_synch_startup:=true \
pub_extra_delay_ms:=15000 \
sub_stats_file_path:=/path/to/results \
shm_size_mega_bytes:=
```


Execute the benchmarks for 5p1s using TZC protocol, in separate Docker containers, with each publisher waiting for the subscriber to establish connection and with the subscriber start up delayed by 15secs, `TCP_NODELAY` enabled.
```
python2 /ros_comm/src/ros_comm/clients/benchmark/execute.py \
--tcp=no --shm=no --tzc=yes --udp=no \
--use_case=5p1s_separate_docker --no_pool=yes --pool=no \
--extra_params pub_wait_for_subscribers:=true \
sub_extra_delay_ms:=15000 \
sub_stats_file_path:=/path/to/results
```

Execute the benchmarks for 5p1s using TZC protocol, in separate Docker containers, with each publisher waiting for the subscriber to establish connection and with the publishers start up order enforced, `TCP_NODELAY` enabled.
```
python2 /ros_comm/src/ros_comm/clients/benchmark/execute.py \
--tcp=no --shm=no --tzc=yes --udp=no \
--use_case=5p1s_separate_docker --no_pool=yes --pool=no \
--extra_params pub_wait_for_subscribers:=true \
pub_enable_synch_startup:=true \
sub_stats_file_path:=/path/to/results
```


Execute the benchmarks for 5p1s using SHM protocol, in separate Docker containers, with each publisher waiting for the subscriber to establish connection and with the publishers start up order enforced, and image pools disabled.
```
python2 /ros_comm/src/ros_comm/clients/benchmark/execute.py \
--tcp=no --shm=yes --tzc=no --udp=no \
--use_case=5p1s_separate_docker --no_pool=yes --pool=no \
--extra_params pub_wait_for_subscribers:=true \
pub_enable_synch_startup:=true \
sub_stats_file_path:=/path/to/results
```
82 changes: 82 additions & 0 deletions clients/benchmark/CMakeLists.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
# ***************************************************************************************************
# * Copyright Five AI 2021.
# * All rights reserved.
# ***************************************************************************************************

cmake_minimum_required(VERSION 2.8.3)
project(benchmark)

add_compile_options(-std=c++11)

find_package(catkin REQUIRED COMPONENTS
roscpp
rospy
std_msgs
)

find_package(Boost REQUIRED COMPONENTS system)

## The catkin_package macro generates cmake config files for your package
## Declare things to be passed to dependent projects
## INCLUDE_DIRS: uncomment this if your package contains header files
## LIBRARIES: libraries you create in this project that dependent projects also need
## CATKIN_DEPENDS: catkin_packages dependent projects also need
## DEPENDS: system dependencies of this project that dependent projects also need
catkin_package(
# INCLUDE_DIRS include
# LIBRARIES benchmark
# CATKIN_DEPENDS roscpp rospy std_msgs
# DEPENDS system_lib
)

include_directories(
include
${catkin_INCLUDE_DIRS}
)

add_library(${PROJECT_NAME}_util
STATIC
src/Util.cpp
)


add_executable(${PROJECT_NAME}_publisher src/Publisher.cpp)
add_executable(${PROJECT_NAME}_subscriber src/Subscriber.cpp)
add_executable(${PROJECT_NAME}_monitor src/Monitor.cpp)

add_custom_target(${PROJECT_NAME}_all)
add_dependencies(${PROJECT_NAME}_all
${PROJECT_NAME}_publisher
${PROJECT_NAME}_subscriber
${PROJECT_NAME}_monitor
)

target_link_libraries(${PROJECT_NAME}_publisher
${catkin_LIBRARIES}
${PROJECT_NAME}_util
rt
)

target_link_libraries(${PROJECT_NAME}_subscriber
${PROJECT_NAME}_util
${catkin_LIBRARIES}
rt
)

target_link_libraries(${PROJECT_NAME}_monitor
${PROJECT_NAME}_util
${catkin_LIBRARIES}
)

install(
TARGETS
${PROJECT_NAME}_publisher
${PROJECT_NAME}_subscriber
${PROJECT_NAME}_monitor
RUNTIME DESTINATION ${CATKIN_PACKAGE_BIN_DESTINATION}
)

install(FILES
launch/launch.xml
DESTINATION ${CATKIN_PACKAGE_SHARE_DESTINATION}
)
71 changes: 71 additions & 0 deletions clients/benchmark/docker/Dockerfile
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
# syntax=docker/dockerfile:experimental

# WARNING: The top line must go first in the file and is required to
# allow cloning the git repository through ssh. More details
# can be found at: https://medium.com/@tonistiigi/build-secrets-and-ssh-forwarding-in-docker-18-09-ae8161d066
# To build this docker image run:
# 1. export DOCKER_BUILDKIT=1
# 2. docker build --ssh default --tag XXX .

# ***************************************************************************************************
# * Copyright Five AI 2021.
# * All rights reserved.
# ***************************************************************************************************

FROM osrf/ros:kinetic-desktop-full-xenial

RUN apt-get update && apt-get install -y \
openssh-client \
nano \
htop \
openssh-server \
iputils-ping \
net-tools

RUN mkdir -p -m 0600 ~/.ssh && \ssh-keyscan github.com >> ~/.ssh/known_hosts
RUN mkdir /var/run/sshd
RUN echo 'root:abcde' | chpasswd
RUN sed -i 's/PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config

# SSH login fix. Otherwise user is kicked off after login
RUN sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd
ENV NOTVISIBLE "in users profile"

RUN echo "export VISIBLE=now" >> /etc/profile
RUN echo "/usr/sbin/sshd" >> ~/.bashrc
RUN echo "export ROSLAUNCH_SSH_UNKNOWN=1" >> ~/.bashrc
RUN echo "source /ros_comm/devel-release/setup.sh" >> ~/.bashrc
RUN echo "mkdir -p \${FIVESHM_HOME}" >> ~/.bashrc
# This line is need to workaround a roslaunch bug where multiple instances of roscore are launched
RUN echo "export ROS_MASTER_URI=http://\$(uname -n):11311" >> ~/.bashrc
RUN echo "nano /ros_comm/src/ros_comm/clients/benchmark/launch/launch.xml" >> ~/.bash_history
RUN echo "catkin_make_isolated \
--source ./src/ \
--build /ros_comm/build-release \
--devel /projects/ros_comm/devel-release \
--install-space /ros_comm/install-release \
--pkg=benchmark --install --cmake-args -DCMAKE_BUILD_TYPE=Release && \
roslaunch --screen -v benchmark launch.xml pub_image_count:=2000 \
sub_stats_file_path:=/root/tmp use_case:=1p1s_same_host" >> ~/.bash_history

RUN echo "python2 /ros_comm/src/ros_comm/clients/benchmark/execute.py --use_case=1p1s_same_host --extra_params=''" >> ~/.bash_history


RUN --mount=type=ssh git clone [email protected]:fiveai/ros_comm.git /ros_comm/src/ros_comm && \
cd /ros_comm/src/ros_comm && \
git pull && \
git checkout fiveshm && \
cd /ros_comm

RUN /bin/bash -c "source /opt/ros/kinetic/setup.sh && \
catkin_make_isolated \
--source /ros_comm/src/ \
--build /ros_comm/build-release \
--devel /ros_comm/devel-release \
--install-space /ros_comm/install-release \
--install \
--cmake-args -DCMAKE_BUILD_TYPE=Release"

WORKDIR /ros_comm

EXPOSE 22
48 changes: 48 additions & 0 deletions clients/benchmark/docker/docker-compose.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# ***************************************************************************************************
# * Copyright Five AI 2021.
# * All rights reserved.
# ***************************************************************************************************

version: "3.7"

x-base_node: &base_node
image: fiveshm
ipc: host
volumes:
- "${HOME}:${HOME}"
stdin_open: true # docker run -i
tty: true # docker run -t
environment:
- FIVESHM_HOME=${HOME}/fiveshm

services:
# define the nodes as copies of the base node
node_1:
<< : *base_node
container_name: node1
hostname: node1

node_2:
<< : *base_node
container_name: node2
hostname: node2

node_3:
<< : *base_node
container_name: node3
hostname: node3

node_4:
<< : *base_node
container_name: node4
hostname: node4

node_5:
<< : *base_node
container_name: node5
hostname: node5

node_6:
<< : *base_node
container_name: node6
hostname: node6
Loading

0 comments on commit e7423d7

Please sign in to comment.