Merge pull request #290 from UHHRobotics22-23/doc/top_level

Extending the top level documentation
UHHRobotics22-23 · Oct 12, 2023 · 4aeeae9 · 4aeeae9
2 parents 8240b5c + 50ba392
commit 4aeeae9
Show file tree

Hide file tree

Showing 8 changed files with 192 additions and 13 deletions.
diff --git a/README.md b/README.md
@@ -4,12 +4,30 @@
 [![ROS Version Noetic](https://img.shields.io/badge/ROS%20Version-Noetic-%2388b55a)](http://wiki.ros.org/noetic)
 
 This repository contains the codebase for a [Marimba](https://en.wikipedia.org/wiki/Marimba) playing robot developed during the TAMS Master Project 2022/2023.
-It is currently still "work in progress".
 
 <p align="center">
   <img width="30%" src="https://github.com/UHHRobotics22-23/marimbabot/assets/15075613/277ce391-edd8-4c7a-8142-e50c12e855a2" alt="marimba playing robot" />
 </p>
 
+## Project Overview
+
+The project is separated into multiple different modules.
+Each module contains localized documentation.
+
+| Module                                                                                | Brief Description                                                                                   |
+|---------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------|
+| [marimbabot_audio](marimbabot_audio/)                                                 | Contains the audio analysis related code pieces                                                     |
+| [marimbabot_behavior](marimbabot_behavior/)                                           | Implements the behaviour of the robot which interfaces with most other systems                      |
+| [marimbabot_bringup](marimbabot_bringup/)                                             | The main bringup of the system used to for starting the project                                     |
+| [marimbabot_description](marimbabot_description/)                                     | Contains the descriptions of the physical compontents of the robot for control and simulation       |
+| [marimbabot_hardware](marimbabot_hardware/)                                           | The ros_control driver for the servo based hardware.                                                |
+| [marimbabot_msgs](marimbabot_msgs/)                                                   | The shared ros message and action definitions                                                       |
+| [marimbabot_planning](marimbabot_planning/)                                           | Contains the code for taking a input of notes and generate a trajectory for the robot               |
+| [marimbabot_simulation](marimbabot_simulation/)                                       | Facilitates the simulation of the robot and marimba                                                 |
+| [marimbabot_speech](marimbabot_speech/)                                               | Text to speech and speech to text and eventual command                                              |
+| [marimbabot_ur5_flex_double_moveit_config](marimbabot_ur5_flex_double_moveit_config/) | MoveIT config for the UR5 with the marimba and the two mallet holder.                               |
+| [marimbabot_ur5_moveit_config](marimbabot_ur5_moveit_config/)                         | MoveIT config for the UR5 and the marimba                                                           |
+| [marimbabot_vision](marimbabot_vision/)                                               | Implements the vision model for the note recognition and facilitates the related dataset generation |
 ## Setup
 
 The robot uses Ubuntu 20.04 and ROS noetic.
@@ -121,19 +139,52 @@ cd catkin_ws
 source devel/setup.bash
 ```
 
-In order to run the whole project on the real robot, launch the bringup package that brings up the launch file for each package:
+#### Prerequisites and Configuration
+Additionally to the UR5, the following devices have to be connected and configured before launching the project:
+1. Logitech StreamCam (packages marimbabot_vision and marimbabot_speech)
+2. Scarlett 2i2 USB Audio Interface (package marimbabot_audio)
+
+#### Logitech StreamCam (required for packages marimbabot_vision and marimbabot_speech):
+Change the parameter <i>device</i> of the node <i>audio_capture</i> in the launch file 
 
 ```bash
-roslaunch marimbabot_bringup marimbabot.launch
+marimbabot_speech/launch/command_recognition.launch
+```
+
+and modify the <i>device_id</i> parameter in the configuration file:
+
+```bash
+marimbabot_vision/config/cv_camera.yaml
+```
+
+#### Scarlett 2i2 USB Audio Interface (required for package marimbabot_audio):
+
+Adjust the <i>device</i> parameter for the <i>note_audio_capture</i> node in the launch file:
+
+```bash
+marimbabot_audio/launch/audio_feedback.launch
 ```
 
+#### Launch the whole project
+In order to run the whole project on the real robot, one has to run two launch files. First, the launch file that sets up the robot and its hardware:
+
+```bash
+roslaunch marimbabot_bringup marimbabot_ur5_bringup.launch
+```
+
+Second, the launch file that brings up the launch file for each package:
+
+
+```bash
+roslaunch marimbabot_bringup marimbabot.launch
+```
 
 #### Note for development: Add the main launch files to the bringup if they are created.
 
-To run the UR5 setup with the MoveIt Demo Mode run
+To run the UR5 setup with the MoveIt Demo Mode and two mallets run
 
 ```bash
-roslaunch marimbabot_ur5_moveit_config demo.launch
+roslaunch marimbabot_ur5_flex_double_moveit_config demo.launch
 ```
 
 you should be able to test simple planning things with it.
@@ -178,7 +229,15 @@ git push
 
 Now you can [create a pull request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request#creating-the-pull-request) for your branch and merge it (after it is approved) into the `main` branch using the GitHub website.
 
-## Project Overview
+## Additional Resources
+
+### MoveIT config
+The project contains two MoveIT configuration definitions ([marimbabot_ur5_moveit_config](marimbabot_ur5_moveit_config/) and [marimbabot_ur5_flex_double_moveit_config](marimbabot_ur5_flex_double_moveit_config/)).
+
+The MoveIT configs were generated using the [MoveIT Setup Assistant](http://docs.ros.org/en/kinetic/api/moveit_tutorials/html/doc/setup_assistant/setup_assistant_tutorial.html).
+When the description of the robot is changed and contains new controlable joints.
+
+Generally the main additional work required for the MoveIT config setup is the collision matrix setup, where non-colliding elements have to be removed from the collision checks.
 
 ### ROS Node Diagram
 ```mermaid

diff --git a/marimbabot_bringup/README.md b/marimbabot_bringup/README.md
@@ -0,0 +1,46 @@
+# TAMS Master Project 2022/2023 - Bringup
+
+This package contains the launch files for the bringup of the robot.
+The launch process is described in the top level [README](../README.md).
+
+## Launch Files
+
+### marimbabot_ur5_bringup.launch
+This file is already configured to work with our setup.
+However, the following parameters and code parts may be of interest. Also, for further information you may want to visit [hardware_design](https://github.com/UHHRobotics22-23/hardware_design).
+
+When using a wifi connection to the mallet, one can set the ip address and port here:
+
+```xml
+<group if="$(eval mallet_holder_type=='flex_double')">
+            <!--> ... <-->
+            <group if="$(arg mallet_joint_wifi)">
+                <node name="mallet_hardware_control_node" pkg="marimbabot_hardware" type="mallet_hardware_control_node_wifi" output="screen">
+                    <!-- Setting device parameter -->
+                    <param name="address" value="192.168.42.1"/>
+                    <param name="port" value="8888"/>
+                </node>
+            </group>
+            <!--> ... <-->
+</group>
+
+```
+
+The robot ip address can be set here:
+```xml
+<include file="$(find ur_robot_driver)/launch/ur_control.launch">
+    <!-- Other arguments -->
+    <arg name="robot_ip" value="192.168.1.12"/>
+    <!-- Other arguments -->
+</include>
+
+```
+
+### marimbabot_bringup.launch
+Launches the bringup of the robot. It launches the following packages:
+
+- marimbabot_vision
+- marimbabot_speech
+- marimbabot_behavior
+- marimbabot_audio
+- marimbabot_planning
diff --git a/marimbabot_msgs/README.md b/marimbabot_msgs/README.md
@@ -0,0 +1,64 @@
+# TAMS Master Project 2022/2023 - Msgs
+
+This package contains all the custom messages and actions used in the project.
+
+## Actions
+
+### [HitSequence.action](action/HitSequence.action)
+The HitSequence action is used to request the execution of a sequence of hits. It is defined as follows:
+
+#### Goal Definition:
+
+- **marimbabot_msgs/HitSequenceElement[] hit_sequence_elements**:  An array of HitSequenceElement messages that specify the hits to be played.
+#### Result Definition:
+
+- **time first_note_hit_time**: The absolute time when the first note of the sequence was hit.
+- **bool success**: Indicates whether the goal was successfully achieved.
+- **uint16 error_code**:
+Provides an error code in case of failure
+
+- - **uint16 SUCCESS = 0**: Successful completion.
+- - **uint16 PLANNING_FAILED = 1**: Indicates that the planning process failed.
+- - **uint16 EXECUTION_FAILED = 2**: Indicates that the execution of the action failed.
+
+#### Feedback:
+
+- **bool playing**: Indicates whether the robot is currently playing, meaning that the goal is still active.
+
+
+### [LilypondAudio.action](action/LilypondAudio.action)
+The LilypondAudio action is used to request the playing of an audio piece generated from a Lilypond string. It is defined as follows:
+
+#### Goal Definition:
+
+- **std_msgs/String lilypond_string**: A string containing Lilypond notation to be interpreted and played.
+#### Result Definition:
+
+- **bool success**: Indicates whether the audio is assumed to be played successfully.
+#### Feedback:
+
+- **bool in_progress**: Indicates whether the audio is currently being played.
+
+
+## Messages
+
+The following messages are defined in this package. For further information on the individual messages, please refer to the individual message files.
+
+### [Command.msg](msg/Command.msg)
+This message is used to send commands from the speech node to the behavior node.
+Have a look at [README](../marimbabot_speech/README.md#5-command-examples) for more information on the command syntax and examples.
+
+### [CQTStamped.msg](msg/CQTStamped.msg)
+This message contains information of the Constant-Q Transform(CQT). See [README](../marimbabot_audio/README.md#4-pipeline-of-music-note-detection) for more information.
+### [HitSequence.msg](msg/HitSequence.msg)
+This message contains an array of HitSequenceElement messages.
+
+### [HitSequenceElement.msg](msg/HitSequenceElement.msg)
+This message contains the information single element of a HitSequence message. It includes tone information, the start time of the note, the duration of the note and the loudness of the note.
+
+### [NoteOnset.msg](msg/NoteOnset.msg)
+Used for publish the detect music note.
+### [SequenceMatchResult.msg](msg/SequenceMatchResult.msg)
+For published the final evaluation of the robot performance
+### [Speech.msg](msg/Speech.msg)
+For published the transcribed test
diff --git a/marimbabot_msgs/msg/CQTStamped.msg b/marimbabot_msgs/msg/CQTStamped.msg
@@ -7,8 +7,7 @@ int32 number_of_semitones
 string min_note
 
 # time between two consecutive windows (rows)
-# TODO: in samples of the signal to avoid severe float artifacts
 duration hop_length
 
 # 2d vector with rows over time and step width specified by number_of_semitones
-float32[] data
+float32[] data
diff --git a/marimbabot_msgs/msg/Command.msg b/marimbabot_msgs/msg/Command.msg
@@ -1,6 +1,11 @@
 std_msgs/Header header
+# Behavior is the behavior to execute, e.g. 'play'.
 string behavior
-string action
-string parameter
 
+# Action is the action to execute, e.g. 'decrease speed'. May be empty.
+string action
 
+# Parameter is the parameter associated with the action, e.g. '20' when using the action 'decrease speed'.
+# This would decrease the speed by 20bpm.
+# May be empty, then behavior node will use default setup.
+string parameter
diff --git a/marimbabot_msgs/msg/HitSequence.msg b/marimbabot_msgs/msg/HitSequence.msg
@@ -1,3 +1,6 @@
 std_msgs/Header header
+# identifier that is used by the audio package to identify the sequence 
 int32 sequence_id
+
+# list of hit sequence elements
 marimbabot_msgs/HitSequenceElement[] hit_sequence_elements
diff --git a/marimbabot_msgs/msg/NoteOnset.msg b/marimbabot_msgs/msg/NoteOnset.msg
@@ -1,5 +1,6 @@
 std_msgs/Header header
 
+# the detected note
 string note
 
 # confidence in note classification *not confidence in overall detection*
@@ -9,6 +10,4 @@ float32 confidence
 float32 duration
 
 # loudness of the onset
-float32 loudness
-
-
+float32 loudness
diff --git a/marimbabot_msgs/msg/Speech.msg b/marimbabot_msgs/msg/Speech.msg
@@ -1,3 +1,7 @@
 std_msgs/Header header
+
+# the text that was recognized
 string text
+
+# the probability of no speech
 float32 no_speech_prob