zenoh-plugin-dds: interoperability with other DDS implementations?

[lanner17]

Hi,
Does zenoh-plugin-dds support other DDS implementations apart from CycloneDDS ? From a line ("no matter the DDS implementation as soon as it implements the standard DDSI protocol") in the README it seems to be the case. I just want to confirm.
Also, can zenoh-plugin-dds run on Windows10 ? (or has it been tested on Windows) ?

Thank you

[JEnoch]

Hi @lanner17 ,

Yes, zenoh-plugin-dds internally uses CycloneDDS which can interoperate with any other compliant DDS implementation.
Interoperability tests are regularly performed between the major DDS implementations.

No, we didn't yet test it on Windows.

N.B.: as discussed below, zenoh-plugin-dds might not support non-standard extensions of some DDS implementations - for instance Connext's BATCH QosPolicy.

[lanner17]

Hi @JEnoch, Ok, that sounds good. Thanks.

[lanner17]

Hi
I have a zenoh-plugin-dds setup to read from an existing DDS application (not Cyclone) and using the REST interface I get for one of the topics

curl http://localhost:8000/@/service/**/participant/**/VFDInfo

[
  {
    "key": "/@/service/4E3F58229FD94E969533F7F6F533BBD7/dds/participant/a0d4064b05da444800000001000001c1/writer/a0d4064b05da4448000000018002d602/VFDInfo",
    "value": {
      "key": "a0d4064b05da4448000000018002d602",
      "keyless": false,
      "participant_key": "a0d4064b05da444800000001000001c1",
      "qos": {
        "deadline": {
          "period": 9223372036854776000
        },
        "destination_order": {
          "kind": "BY_RECEPTION_TIMESTAMP"
        },
        "durability": {
          "kind": "VOLATILE"
        },
        "durability_service": {
          "history_depth": 1,
          "history_kind": "KEEP_LAST",
          "max_instances": -1,
          "max_samples": -1,
          "max_samples_per_instance": -1,
          "service_cleanup_delay": 0
        },
        "history": {
          "depth": 1,
          "kind": "KEEP_LAST"
        },
        "latency_budget": {
          "duration": 0
        },
        "liveliness": {
          "kind": "AUTOMATIC",
          "lease_duration": 9223372036854776000
        },
        "ownership": {
          "kind": "SHARED"
        },
        "partitions": [],
        "reliability": {
          "kind": "BEST_EFFORT",
          "max_blocking_time": 100000000
        }
      },
      "routes": {
        "*": {
          "Routed": "/VFDInfo"
        }
      },
      "topic_name": "VFDInfo",
      "type_name": "jVFDInfo"
    },
    "encoding": "application/json",
    "time": "None"
  }
]

My question is how do one read the samples/instances ? It seems obvious to try something like curl http://localhost:8000/VFDInfo, but it does not return anything. I realize the data has to be de-serialized.

Thanks

[p-avital]

Hi,

The REST API distinguishes queries and subscriptions by treating GET requests with header Accept: text/event-stream as subscriptions, and other requests as queries.

I'm guessing you wanted to subscribe to obtain samples, so you should try that (just add `-H "Accept: text/event-stream" to your curl command).

I hope that helps :)

[JEnoch]

Actually, just a curl -H "Accept: text/event-stream" won't work.
This header is managed by the REST API as part of the implementation of a HTML Server-sent events feature.
So you have to use the EventSource Web API (for instance in JavaScript) to subscribe to zenoh-plugin-dds publications on /VFDInfo.

We have an example of such usage with a ROS2 + zenoh-plugin-dds example. But you can easily adapt it to your DDS use case.
See: https://github.com/eclipse-zenoh/zenoh-demos/blob/debc4a1bbd85de865721ea2d82242446b4fe094c/ROS2/zenoh-rest-teleop/ros2-teleop.html

In this example, all DDS types are re-defined as JS classes, with CDR encode()/decode() functions - see Twist and Log classes.
The subscription via EventSource is managed in the subscribeToRosout() function.
You can adapt this code to your use case by:

• defining your jVFDInfo class in JavaScript with a decode(cdrReader) function that will decode the CDR payload
• creating the EventSource as such: event_source = new EventSource("http://localhost:8000/VFDInfo");
• call jVFDInfo.decode(reader) instead of Log.decode(reader)

[JEnoch]

Another (maybe simplest) solution if you don't want to subscribe, but to retrieve the latest DDS publication via a simple curl, is to configure a Zenoh memory storage on /VFDInfo within the zenoh router. This storage will store the latest DDS publication routed by the bridge on /VFDInfo, and will reply to curl http://localhost:8000/VFDInfo commands made on the REST API.
Still, the reply you get will be a JSON with "value" field containing the DDS data encoded as CDR+base64. It would need to be decoded in a similar way than inJavaScript.

[lanner17]

Hi @JEnoch ,
I followed your first suggestion (using the EventSource API and Javascript) - for me it seemed simpler :). However the event source listener never fire. The console output that is passed to the listener does not get printed. As a first step I would expect that. The following were verified:

1. DDS data is present on the domain and the topic is published periodically (every 250ms)
2. The DDS bridge shows the routes for the topics from DDS to zenoh.
3. The event_source shows its state as ready (true).
4. The URL for the event source points directly to the dds bridge: http://192.168.0.66:8000/VFDInfo

It almost seems to me the even_source does not "get" the data from the bridge. Will the bridge "push" the data ? I do not know what else to check.

Thanks

[JEnoch]

What you can do is to open your browser's console.
You probably get some error message within. Let me know which ones.

If not, activate the debug logs on the DDS bridge (via RUST_LOG=debug environment variable), and check if you well get such logs:

[2022-09-01T15:05:10Z INFO  zplugin_dds] New route: DDS Reader 954810018f6c742d9f59935f00000307 on 'VFDInfo' => zenoh 'VFDInfo' with type 'jVFDInfo'

then:

[2022-09-01T15:05:11Z DEBUG zplugin_rest] Subscribe to VFDInfo for SSE stream (task 46)
[2022-09-01T15:05:11Z DEBUG zenoh::net::routing::pubsub] Register client subscription
[2022-09-01T15:05:11Z DEBUG zenoh::net::routing::pubsub] Register client subscription VFDInfo
[2022-09-01T15:05:11Z DEBUG zenoh::net::routing::pubsub] Register subscription VFDInfo for Face{0, BA8BA662B8BE4F7385190AB638E83D58}

[lanner17]

I looked at the browser console and there are no error messages. I just get

Subscribe to EventSource: http://192.168.0.66:8000/fb20/AnalogInput when the page is loaded or refreshed.

Note that I changed to a different topic. AnalogInput is much simpler to decode than VFDInfo for purposes of this exploration.

With RUST_LOG=debug:
The various log messages related to AnalogInput are:

[2022-09-01T18:04:27Z DEBUG zplugin_dds::dds_mgt] Discovered DDS publication a0d4064b05da85e00000000180008402 from Participant a0d4064b05da85e000000001000001c1 on AnalogInput with type jIOReal (keyless: false)

[2022-09-01T18:04:27Z DEBUG zplugin_dds] Discovered DDS Writer on AnalogInput: a0d4064b05da85e00000000180008402
[2022-09-01T18:04:27Z DEBUG zenoh::net::routing::resource] Register resource /fb20/AnalogInput
[2022-09-01T18:04:27Z DEBUG zplugin_dds::dds_mgt] Discovered DDS subscription 8b1e10012559a80fd7a9fedf00000b07 from Participant 8b1e10012559a80fd7a9fedf000001c1 on AnalogInput with type jIOReal (keyless: false)
[2022-09-01T18:04:27Z DEBUG zplugin_dds::dds_mgt] Ignoring discovery of 8b1e10012559a80fd7a9fedf00000b07 (AnalogInput) from local participant
[2022-09-01T18:04:27Z INFO  zplugin_dds] New route: DDS 'AnalogInput' => zenoh '/fb20/AnalogInput' with type 'jIOReal'

Later when the page is loaded in the browser:

[2022-09-01T18:04:42Z DEBUG zenoh_ext::group] Sending Keep Alive for: AE350C19EB644D58A1AA8C0E651F68E4
[2022-09-01T18:04:43Z INFO  tide::log::middleware] <-- Request received
[2022-09-01T18:04:43Z INFO  tide::log::middleware] --> Response sent
[2022-09-01T18:04:43Z DEBUG zplugin_rest] Subscribe to /fb20/AnalogInput for SSE stream (task 9127)
[2022-09-01T18:04:43Z DEBUG zenoh::net::routing::pubsub] Register client subscription
[2022-09-01T18:04:43Z DEBUG zenoh::net::routing::pubsub] Register client subscription /fb20/AnalogInput
[2022-09-01T18:04:43Z DEBUG zenoh::net::routing::pubsub] Register subscription /fb20/AnalogInput for Face{0, AE350C19EB644D58A1AA8C0E651F68E4}
[2022-09-01T18:04:43Z DEBUG zenoh::net::routing::pubsub] Register peer subscription /fb20/AnalogInput (peer: AE350C19EB644D58A1AA8C0E651F68E4)
[2022-09-01T18:04:45Z DEBUG zenoh_ext::group] Sending Keep Alive for: AE350C19EB644D58A1AA8C0E651F68E4

So, it seems the route was created and the bridge acknowledged the subscription. It is just strange that it does not trigger the event handler in the browser

[JEnoch]

That's indeed strange!
You can try several things to analyse this issue:

1. activate the trace log level to see if some data are indeed routed by the bridge:
   setting RUST_LOG=zplugin=trace you'll get only the logs from the plugins. If the bridge well receives and routes publications on this topic, you should see such logs:

   [2022-09-02T06:39:01Z TRACE zplugin_dds::dds_mgt] Route data from DDS AnalogInput to zenoh key=/fb20/AnalogInput
   

2. If the publication rate on this topic is intensive, try to lower it (e.g. 1Hz)
3. Try with another browser (we tested only Safari and Chrome/Chromium-based browsers)

[lanner17]

Ok, that provided another clue. With RUST_LOG=zplugin=trace I noticed that the data of topics with QoS's Reliability=RELIABLE and durability=TRANSIENT_LOCAL get routed and topics Reliability=BEST_EFFORT do not get routed. AnalogInput has Reliability=BEST_EFFORT QoS.

[JEnoch]

Considering you well get this log:
New route: DDS 'AnalogInput' => zenoh '/fb20/AnalogInput' with type 'jIOReal'
It means the bridge correctly created a DDS Reader to serve the route, with the exact same QoS than the discovered Writer.
But if you don't get any log of routed data, it probably means this Reader doesn't receive any publication.

I suggest you analyse further in 2 possible ways:

1. If your DDS implementation provides some tooling allowing to explore the running DDS Readers/Writers and their QoS, you could check if there is indeed a Reader created by the bridge, and if its QoS are matching the Writer QoS.
2. To check if it could be an interop issue between your DDS implementation and CycloneDDS, you can try to create a CycloneDDS application that declares a DDS Reader on your topic, with the same QoS than the Writer.

[lanner17]

As far as your option 1 is concerned: The tool of my DDS implementation shows the reader for the bridge. There are no QoS mismatches, the tool can subscribe to AnalogInput data (so it is published). However the tool also shows the data type of readers and writers and it seems the bridge reader cannot determine the data types (The data types are on the wire). That may be where it is incompatible.
Option 2 will take some more time to investigate.
Thank you for the help so far.

[JEnoch]

In standard DDS (i.e. implementing the standard DCPS protocol), the data type is just a string (jIOReal in your case) sent in the SEDP message. The bridge doesn't need to know what's the structure of this type, as it just forward the serialized data, without deserializing it.
When creating the DDS Reader that matches the discovered DDS Writer, the bridge uses the same type string (jIOReal) to declare the Reader.

What do you mean by "The data types are on the wire" ?
Does your DDS implementation require an extra type structure declaration on some proprietary topic ?
BTW what's this DDS implementation ?

[lanner17]

Ok, it makes sense that the serialized data can just be forwarded. Based on this, my comment "The data types are on the wire" is moot.
My implementation is Connext. The types can, optionally, be published as a separate topic. A subscriber can use this to de-serialize the topic that match the type.

[lanner17]

I changed the Javascript code to subscribe to a topic with Reliable QoS and the data got printed in the browser console. So it definitely seems it is only the BEST_EFFORT data does not get routed yet.

[lanner17]

I just remembered/realized that Connext has a QoS setting to batch data. So, if you have a topic that has small data samples, you can set a QoS and the middleware would batch the samples together. This makes for more efficient transfer on the wire - sending one batch of data instead of small packets. This must be a propriety feature that CycloneDDS do not have implemented. Once this Qos is set, it is transparent so it is easy to forget about it.
I turned off batching and AnalogInput is now routed by the bridge and received in the browser.
Thanks for the help.

[JEnoch]

Glad you found a way to make it work!
Still, I think it's weird that Connext does batching in a way that CycloneDDS fails to receive it. As far as I remember, the DCPS protocol specifies standard ways to perform batching of publications.
What's this specific QoS name ?

@eboasson: you probably have more insights than me of what could cause this issue. To summarize:

• a Connext BEST_EFFORT Writer configured with some specific QoS to batch data
• a CycloneDDS v0.7 Reader (in zenoh-bridge-dds) with exact same QoS (standard ones) fails to receive the Writer's publications

[eboasson]

As far as I remember, the DCPS protocol specifies standard ways to perform batching of publications.

It does, and Cyclone DDS uses it. Unfortunately, RTI uses a special "batched" encoding that is not part of the spec — and to add insult to injury, they use message identifiers that are part of the spec-reserved name space instead of using a vendor-specific identifier. I suspect that this is what you're dealing with.

It is easy to check if you have WireShark, that proprietary encoding of RTI looks like this:

[JEnoch]

Thanks for your confirmation Erik. That's clear now that CycloneDDS (and therefore zenoh-bridge-dds) cannot receive non-standard packets from Connext, in case this proprietary batch QoS is set.

[lanner17]

@JEnoch, @eboasson Thanks for the clarification. For my current purpose, it is possible to work around the issue.

[kuonangzhe]

Hi guys, thanks for fruitful discussion above. So here's one more curious question. When we say Zenoh is compatible with common ddsi implementation, is shared memory included? For example, using fastdds and cyclonedds as two applications (both with shared memory enabled in node) on one host, and does Zenoh be able to bridge those shared memory messages?

[JEnoch]

Hi @kuonangzhe,
the DDSI specification specifies the interoperability only for UDP/IP transport (see §9). There is no specification for interoperability over shared memory, so each DDS vendor implemented it's own solution.
FastDDS has a proprietary implementation, while Eclipse CycloneDDS relies on Eclipse Iceoryx. Therefore, FastDDS and CycloneDDS can't communicate via shared memory.
The Zenoh bridge for DDS depends on CycloneDDS for DDS communication. Thus, it can't either communicate with FastDDS via shared memory.