From 4e0423863258c2ecf6ac38a2b8d3f1e820d9e116 Mon Sep 17 00:00:00 2001 From: ateRstones <8556589+ateRstones@users.noreply.github.com> Date: Wed, 11 Oct 2023 20:14:43 +0200 Subject: [PATCH 01/10] Extending the top level documentation to include the sections --- README.md | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 369f9e95..fda6d757 100644 --- a/README.md +++ b/README.md @@ -10,6 +10,25 @@ It is currently still "work in progress". marimba playing robot

+## Project Overview + +The project is separated into multiple different modules. +Each module contains localized documentation. + +| Module | Brief Description | +|---------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------| +| [marimbabot_audio](marimbabot_audio/) | Contains the audio analysis related code pieces | +| [marimbabot_behavior](marimbabot_behavior/) | Implements the behaviour of the robot which interfaces with most other systems | +| [marimbabot_bringup](marimbabot_bringup/) | The main bringup of the system used to for starting the project | +| [marimbabot_description](marimbabot_description/) | Contains the descriptions of the physical compontents of the robot for control and simulation | +| [marimbabot_hardware](marimbabot_hardware/) | The ros_control driver for the servo based hardware. | +| [marimbabot_msgs](marimbabot_msgs/) | The shared ros message and action definitions | +| [marimbabot_planning](marimbabot_planning/) | Contains the code for taking a input of notes and generate a trajectory for the robot | +| [marimbabot_simulation](marimbabot_simulation/) | Facilitates the simulation of the robot and marimba | +| [marimbabot_speech](marimbabot_speech/) | Text to speech and speech to text and eventual command | +| [marimbabot_ur5_flex_double_moveit_config](marimbabot_ur5_flex_double_moveit_config/) | MoveIT config for the UR5 with the marimba and the two mallet holder. | +| [marimbabot_ur5_moveit_config](marimbabot_ur5_moveit_config/) | MoveIT config for the UR5 and the marimba | +| [marimbabot_vision](marimbabot_vision/) | Implements the vision model for the note recognition and facilitates the related dataset generation | ## Setup The robot uses Ubuntu 20.04 and ROS noetic. @@ -178,7 +197,15 @@ git push Now you can [create a pull request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request#creating-the-pull-request) for your branch and merge it (after it is approved) into the `main` branch using the GitHub website. -## Project Overview +## Additional Resources + +### MoveIT config +The project contains two MoveIT configuration definitions ([marimbabot_ur5_moveit_config](marimbabot_ur5_moveit_config/) and [marimbabot_ur5_flex_double_moveit_config](marimbabot_ur5_flex_double_moveit_config/)). + +The MoveIT configs were generated using the [MoveIT Setup Assistant](http://docs.ros.org/en/kinetic/api/moveit_tutorials/html/doc/setup_assistant/setup_assistant_tutorial.html). +When the description of the robot is changed and contains new controlable joints. + +Generally the main additional work required for the MoveIT config setup is the collision matrix setup, where non-colliding elements have to be removed from the collision checks. ### ROS Node Diagram ```mermaid From 02019536a625c32626fa6ab9dc2f7c0b6600e337 Mon Sep 17 00:00:00 2001 From: Juphex <20795522+Juphex@users.noreply.github.com> Date: Wed, 11 Oct 2023 21:54:31 +0200 Subject: [PATCH 02/10] documentation on two mallet launching --- README.md | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index fda6d757..bbafb83a 100644 --- a/README.md +++ b/README.md @@ -140,7 +140,14 @@ cd catkin_ws source devel/setup.bash ``` -In order to run the whole project on the real robot, launch the bringup package that brings up the launch file for each package: +In order to run the whole project on the real robot, one has to run two launch files. First, the launch file that sets up the robot and its hardware: + +```bash +roslaunch marimbabot_bringup marimbabot_ur5_bringup.launch +``` + +Second, the launch file that brings up the launch file for each package: + ```bash roslaunch marimbabot_bringup marimbabot.launch @@ -149,10 +156,10 @@ roslaunch marimbabot_bringup marimbabot.launch #### Note for development: Add the main launch files to the bringup if they are created. -To run the UR5 setup with the MoveIt Demo Mode run +To run the UR5 setup with the MoveIt Demo Mode and two mallets run ```bash -roslaunch marimbabot_ur5_moveit_config demo.launch +roslaunch marimbabot_ur5_flex_double_moveit_config demo.launch ``` you should be able to test simple planning things with it. From 128378691a6cae65826aae89030999e804631ddc Mon Sep 17 00:00:00 2001 From: Juphex <20795522+Juphex@users.noreply.github.com> Date: Wed, 11 Oct 2023 22:12:00 +0200 Subject: [PATCH 03/10] add documentation on the camera and external microphone --- README.md | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index bbafb83a..07c3aada 100644 --- a/README.md +++ b/README.md @@ -140,6 +140,33 @@ cd catkin_ws source devel/setup.bash ``` +#### Prerequisites and Configuration +Additionally to the UR5, the following devices have to be connected and configured before launching the project: +1. Logitech StreamCam (packages marimbabot_vision and marimbabot_speech) +2. Scarlett 2i2 USB Audio Interface (package marimbabot_audio) + +#### Logitech StreamCam (required for packages marimbabot_vision and marimbabot_speech): +Change the parameter device of the node audio_capture in the launch file + +```bash +marimbabot_speech/launch/command_recognition.launch +``` + +and modify the device_id parameter in the configuration file: + +```bash +marimbabot_vision/config/cv_camera.yaml +``` + +#### Scarlett 2i2 USB Audio Interface (required for package marimbabot_audio): + +Adjust the device parameter for the note_audio_capture node in the launch file: + +```bash +marimbabot_audio/launch/audio_feedback.launch +``` + +#### Launch the whole project In order to run the whole project on the real robot, one has to run two launch files. First, the launch file that sets up the robot and its hardware: ```bash @@ -153,7 +180,6 @@ Second, the launch file that brings up the launch file for each package: roslaunch marimbabot_bringup marimbabot.launch ``` - #### Note for development: Add the main launch files to the bringup if they are created. To run the UR5 setup with the MoveIt Demo Mode and two mallets run From 92b57a490ee29cd7c127d3bcfc8aff0b60b814f6 Mon Sep 17 00:00:00 2001 From: Juphex <20795522+Juphex@users.noreply.github.com> Date: Wed, 11 Oct 2023 22:30:43 +0200 Subject: [PATCH 04/10] add information on the ips --- marimbabot_bringup/README | 46 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) create mode 100644 marimbabot_bringup/README diff --git a/marimbabot_bringup/README b/marimbabot_bringup/README new file mode 100644 index 00000000..d77c67a5 --- /dev/null +++ b/marimbabot_bringup/README @@ -0,0 +1,46 @@ +# TAMS Master Project 2022/2023 - Bringup + +This package contains the launch files for the bringup of the robot. +The launch process is described in the top level [README](../README.md). + +## Launch Files + +### marimbabot_ur5_bringup.launch +This file is already configured to work with our setup. +However, the following parameters and code parts may be of interest. + +When using a wifi connection to the mallet, one can set the ip address and port here: + +```xml + + ... <--> + + + + + + + + ... <--> + + +``` + +The robot ip address can be set here: +```xml + + + + + + +``` + +### marimbabot_bringup.launch +Launches the bringup of the robot. It launches the following packages: + +- marimbabot_vision +- marimbabot_speech +- marimbabot_behavior +- marimbabot_audio +- marimbabot_planning \ No newline at end of file From 0c4010f9646a5797d9275dabf17f755ecb8a7208 Mon Sep 17 00:00:00 2001 From: Juphex <20795522+Juphex@users.noreply.github.com> Date: Wed, 11 Oct 2023 23:09:48 +0200 Subject: [PATCH 05/10] initial documentation --- marimbabot_msgs/README.md | 62 +++++++++++++++++++++++++++++ marimbabot_msgs/msg/CQTStamped.msg | 2 +- marimbabot_msgs/msg/Command.msg | 9 ++++- marimbabot_msgs/msg/HitSequence.msg | 3 ++ marimbabot_msgs/msg/NoteOnset.msg | 5 +-- marimbabot_msgs/msg/Speech.msg | 4 ++ 6 files changed, 79 insertions(+), 6 deletions(-) create mode 100644 marimbabot_msgs/README.md diff --git a/marimbabot_msgs/README.md b/marimbabot_msgs/README.md new file mode 100644 index 00000000..a9027cbb --- /dev/null +++ b/marimbabot_msgs/README.md @@ -0,0 +1,62 @@ +# TAMS Master Project 2022/2023 - Msgs + +This package contains all the custom messages and actions used in the project. + +## Actions + +### [HitSequence.action](action/HitSequence.action) +The HitSequence action is used to request the execution of a sequence of hits. It is defined as follows: + +#### Goal Definition: + +- **marimbabot_msgs/HitSequenceElement[] hit_sequence_elements**: An array of HitSequenceElement messages that specify the hits to be played. +#### Result Definition: + +- **time first_note_hit_time**: The absolute time when the first note of the sequence was hit. +- **bool success**: Indicates whether the goal was successfully achieved. +- **uint16 error_code**: +Provides an error code in case of failure + +- - **uint16 SUCCESS = 0**: Successful completion. +- - **uint16 PLANNING_FAILED = 1**: Indicates that the planning process failed. +- - **uint16 EXECUTION_FAILED = 2**: Indicates that the execution of the action failed. + +#### Feedback: + +- **bool playing**: Indicates whether the robot is currently playing, meaning that the goal is still active. + + +### [LilypondAudio.action](action/LilypondAudio.action) +The LilypondAudio action is used to request the playing of an audio piece generated from a Lilypond string. It is defined as follows: + +#### Goal Definition: + +- **std_msgs/String lilypond_string**: A string containing Lilypond notation to be interpreted and played. +#### Result Definition: + +- **bool success**: Indicates whether the audio is assumed to be played successfully. +#### Feedback: + +- **bool in_progress**: Indicates whether the audio is currently being played. + + +## Messages + +The following messages are defined in this package. For further information on the individual messages, please refer to the individual message files. + +### [Command.msg](msg/Command.msg) +This message is used to send commands from the speech node to the behavior node. +Have a look at [README](../marimbabot_speech/README.md#5-command-examples) for more information on the command syntax and examples. + +### [CQTStamped.msg](msg/CQTStamped.msg) +This message contains information of the Constant-Q Transform(CQT). See [README](../marimbabot_audio/README.md#4-pipeline-of-music-note-detection) for more information. +### [HitSequence.msg](msg/HitSequence.msg) +This message contains an array of HitSequenceElement messages. + +### [HitSequenceElement.msg](msg/HitSequenceElement.msg) +This message contains the information single element of a HitSequence message. It includes tone information, the start time of the note, the duration of the note and the loudness of the note. + +### [NoteOnset.msg](msg/NoteOnset.msg) + +### [SequenceMatchResult.msg](msg/SequenceMatchResult.msg) +### [Speech.msg](msg/Speech.msg) \ No newline at end of file diff --git a/marimbabot_msgs/msg/CQTStamped.msg b/marimbabot_msgs/msg/CQTStamped.msg index b4c758e3..ed50ad92 100644 --- a/marimbabot_msgs/msg/CQTStamped.msg +++ b/marimbabot_msgs/msg/CQTStamped.msg @@ -11,4 +11,4 @@ string min_note duration hop_length # 2d vector with rows over time and step width specified by number_of_semitones -float32[] data +float32[] data \ No newline at end of file diff --git a/marimbabot_msgs/msg/Command.msg b/marimbabot_msgs/msg/Command.msg index 3f9b55af..49652d1a 100644 --- a/marimbabot_msgs/msg/Command.msg +++ b/marimbabot_msgs/msg/Command.msg @@ -1,6 +1,11 @@ std_msgs/Header header +# Behavior is the behavior to execute, e.g. 'play'. string behavior -string action -string parameter +# Action is the action to execute, e.g. 'decrease speed'. May be empty. +string action +# Parameter is the parameter associated with the action, e.g. '20' when using the action 'decrease speed'. +# This would decrease the speed by 20bpm. +# May be empty. +string parameter \ No newline at end of file diff --git a/marimbabot_msgs/msg/HitSequence.msg b/marimbabot_msgs/msg/HitSequence.msg index 2076bc81..e308e2f5 100644 --- a/marimbabot_msgs/msg/HitSequence.msg +++ b/marimbabot_msgs/msg/HitSequence.msg @@ -1,3 +1,6 @@ std_msgs/Header header +# identifier that is used by the audio package to identify the sequence int32 sequence_id + +# list of hit sequence elements marimbabot_msgs/HitSequenceElement[] hit_sequence_elements \ No newline at end of file diff --git a/marimbabot_msgs/msg/NoteOnset.msg b/marimbabot_msgs/msg/NoteOnset.msg index 70e0909e..eb47afaf 100644 --- a/marimbabot_msgs/msg/NoteOnset.msg +++ b/marimbabot_msgs/msg/NoteOnset.msg @@ -1,5 +1,6 @@ std_msgs/Header header +# the detected note string note # confidence in note classification *not confidence in overall detection* @@ -9,6 +10,4 @@ float32 confidence float32 duration # loudness of the onset -float32 loudness - - +float32 loudness \ No newline at end of file diff --git a/marimbabot_msgs/msg/Speech.msg b/marimbabot_msgs/msg/Speech.msg index 8fdf5274..aba3e0cb 100644 --- a/marimbabot_msgs/msg/Speech.msg +++ b/marimbabot_msgs/msg/Speech.msg @@ -1,3 +1,7 @@ std_msgs/Header header + +# the text that was recognized string text + +# the probability of no speech float32 no_speech_prob \ No newline at end of file From ae8f435c58912ebcc948ce634e87b027f6833542 Mon Sep 17 00:00:00 2001 From: Christian Salamut <20795522+Juphex@users.noreply.github.com> Date: Thu, 12 Oct 2023 00:08:52 +0200 Subject: [PATCH 06/10] Docs/bringup (#291) * add source doc --- marimbabot_bringup/README.md | 46 ++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) create mode 100644 marimbabot_bringup/README.md diff --git a/marimbabot_bringup/README.md b/marimbabot_bringup/README.md new file mode 100644 index 00000000..d77c67a5 --- /dev/null +++ b/marimbabot_bringup/README.md @@ -0,0 +1,46 @@ +# TAMS Master Project 2022/2023 - Bringup + +This package contains the launch files for the bringup of the robot. +The launch process is described in the top level [README](../README.md). + +## Launch Files + +### marimbabot_ur5_bringup.launch +This file is already configured to work with our setup. +However, the following parameters and code parts may be of interest. + +When using a wifi connection to the mallet, one can set the ip address and port here: + +```xml + + ... <--> + + + + + + + + ... <--> + + +``` + +The robot ip address can be set here: +```xml + + + + + + +``` + +### marimbabot_bringup.launch +Launches the bringup of the robot. It launches the following packages: + +- marimbabot_vision +- marimbabot_speech +- marimbabot_behavior +- marimbabot_audio +- marimbabot_planning \ No newline at end of file From 02e550acbb98815f40351929dbb842f6034d7fa8 Mon Sep 17 00:00:00 2001 From: Yunlong Wang <92165787+yunlong-wang-cn@users.noreply.github.com> Date: Thu, 12 Oct 2023 00:16:52 +0200 Subject: [PATCH 07/10] update doc --- marimbabot_msgs/README.md | 6 ++++-- marimbabot_msgs/msg/CQTStamped.msg | 1 - marimbabot_msgs/msg/Command.msg | 2 +- 3 files changed, 5 insertions(+), 4 deletions(-) diff --git a/marimbabot_msgs/README.md b/marimbabot_msgs/README.md index a9027cbb..46ebcb9d 100644 --- a/marimbabot_msgs/README.md +++ b/marimbabot_msgs/README.md @@ -57,6 +57,8 @@ This message contains an array of HitSequenceElement messages. This message contains the information single element of a HitSequence message. It includes tone information, the start time of the note, the duration of the note and the loudness of the note. ### [NoteOnset.msg](msg/NoteOnset.msg) - +Used for publish the detect music note. ### [SequenceMatchResult.msg](msg/SequenceMatchResult.msg) -### [Speech.msg](msg/Speech.msg) \ No newline at end of file +For published the final evaluation of the robot performance +### [Speech.msg](msg/Speech.msg) +For published the transcribed test \ No newline at end of file diff --git a/marimbabot_msgs/msg/CQTStamped.msg b/marimbabot_msgs/msg/CQTStamped.msg index ed50ad92..942997b7 100644 --- a/marimbabot_msgs/msg/CQTStamped.msg +++ b/marimbabot_msgs/msg/CQTStamped.msg @@ -7,7 +7,6 @@ int32 number_of_semitones string min_note # time between two consecutive windows (rows) -# TODO: in samples of the signal to avoid severe float artifacts duration hop_length # 2d vector with rows over time and step width specified by number_of_semitones diff --git a/marimbabot_msgs/msg/Command.msg b/marimbabot_msgs/msg/Command.msg index 49652d1a..9816bbc4 100644 --- a/marimbabot_msgs/msg/Command.msg +++ b/marimbabot_msgs/msg/Command.msg @@ -7,5 +7,5 @@ string action # Parameter is the parameter associated with the action, e.g. '20' when using the action 'decrease speed'. # This would decrease the speed by 20bpm. -# May be empty. +# May be empty, then behavior node will use default setup. string parameter \ No newline at end of file From 1706ba0d11bd9380b0cdc019b946f00636fd0b38 Mon Sep 17 00:00:00 2001 From: Christian Salamut <20795522+Juphex@users.noreply.github.com> Date: Thu, 12 Oct 2023 13:16:34 +0200 Subject: [PATCH 08/10] rm work in progress sentence --- README.md | 1 - 1 file changed, 1 deletion(-) diff --git a/README.md b/README.md index 07c3aada..9ba95fa3 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,6 @@ [![ROS Version Noetic](https://img.shields.io/badge/ROS%20Version-Noetic-%2388b55a)](http://wiki.ros.org/noetic) This repository contains the codebase for a [Marimba](https://en.wikipedia.org/wiki/Marimba) playing robot developed during the TAMS Master Project 2022/2023. -It is currently still "work in progress".

marimba playing robot From 2b57ec9b2b4da0527bb1bdbaf9a57e6f379e0eaf Mon Sep 17 00:00:00 2001 From: Christian Salamut <20795522+Juphex@users.noreply.github.com> Date: Thu, 12 Oct 2023 13:17:20 +0200 Subject: [PATCH 09/10] rm duplicate README --- marimbabot_bringup/README | 46 --------------------------------------- 1 file changed, 46 deletions(-) delete mode 100644 marimbabot_bringup/README diff --git a/marimbabot_bringup/README b/marimbabot_bringup/README deleted file mode 100644 index d77c67a5..00000000 --- a/marimbabot_bringup/README +++ /dev/null @@ -1,46 +0,0 @@ -# TAMS Master Project 2022/2023 - Bringup - -This package contains the launch files for the bringup of the robot. -The launch process is described in the top level [README](../README.md). - -## Launch Files - -### marimbabot_ur5_bringup.launch -This file is already configured to work with our setup. -However, the following parameters and code parts may be of interest. - -When using a wifi connection to the mallet, one can set the ip address and port here: - -```xml - - ... <--> - - - - - - - - ... <--> - - -``` - -The robot ip address can be set here: -```xml - - - - - - -``` - -### marimbabot_bringup.launch -Launches the bringup of the robot. It launches the following packages: - -- marimbabot_vision -- marimbabot_speech -- marimbabot_behavior -- marimbabot_audio -- marimbabot_planning \ No newline at end of file From 50ba39292d400dda4326c1256e26d249c103e5b1 Mon Sep 17 00:00:00 2001 From: Christian Salamut <20795522+Juphex@users.noreply.github.com> Date: Thu, 12 Oct 2023 13:27:08 +0200 Subject: [PATCH 10/10] add reference to hardware design --- marimbabot_bringup/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/marimbabot_bringup/README.md b/marimbabot_bringup/README.md index d77c67a5..69845b87 100644 --- a/marimbabot_bringup/README.md +++ b/marimbabot_bringup/README.md @@ -7,7 +7,7 @@ The launch process is described in the top level [README](../README.md). ### marimbabot_ur5_bringup.launch This file is already configured to work with our setup. -However, the following parameters and code parts may be of interest. +However, the following parameters and code parts may be of interest. Also, for further information you may want to visit [hardware_design](https://github.com/UHHRobotics22-23/hardware_design). When using a wifi connection to the mallet, one can set the ip address and port here: