Add UBX-MON-COMMS

[ARK3r]

I think UBX-MON-COMMS is working and I'm getting a message like this:

header:
  stamp:
    sec: 1726088901
    nanosec: 268752991
  frame_id: gnss
version: 0
n_ports: 5
tx_errors: 16
reserved0: 0
prot_ids:
- 0
- 1
- 5
- 6
ports:
- port_id: 256
  tx_pending: 0
  tx_bytes: 793237
  tx_usage: 0
  tx_peak_usage: 10
  rx_pending: 0
  rx_bytes: 0
  rx_usage: 0
  rx_peak_usage: 0
  overrun_errs: 0
  msgs:
  - 0
  - 0
  - 0
  - 0
  skipped: 0
- port_id: 0
  tx_pending: 0
  tx_bytes: 0
  tx_usage: 0
  tx_peak_usage: 2
  rx_pending: 0
  rx_bytes: 3231614
  rx_usage: 0
  rx_peak_usage: 0
  overrun_errs: 0
  msgs:
  - 0
  - 0
  - 0
  - 0
  skipped: 19775
- port_id: 0
  tx_pending: 0
  tx_bytes: 0
  tx_usage: 0
  tx_peak_usage: 0
  rx_pending: 0
  rx_bytes: 65
  rx_usage: 0
  rx_peak_usage: 3
  overrun_errs: 0
  msgs:
  - 0
  - 0
  - 0
  - 0
  skipped: 1296
- port_id: 1024
  tx_pending: 0
  tx_bytes: 36
  tx_usage: 0
  tx_peak_usage: 0
  rx_pending: 0
  rx_bytes: 0
  rx_usage: 0
  rx_peak_usage: 0
  overrun_errs: 0
  msgs:
  - 0
  - 0
  - 0
  - 0
  skipped: 4585706
- port_id: 257
  tx_pending: 0
  tx_bytes: 2628010
  tx_usage: 1
  tx_peak_usage: 7
  rx_pending: 0
  rx_bytes: 9398
  rx_usage: 0
  rx_peak_usage: 0
  overrun_errs: 0
  msgs:
  - 0
  - 0
  - 0
  - 0
  skipped: 0

but honestly, I can't figure out if the values make any sense since I don't know exactly what each field means

[hortovanyi]

@ARK3r its one of those messages that could be used to debug a configuration.

Some of the field descriptions are a bit poor in the manual.

I noticed you didn't add the out frame? That's used to potential poll for the data as opposed to having it sent based on a frequency update.

One issue is the cpu usage on the device. If everything is turned on, it can't cope and limits output anyway.

Which firmware are you using? ... I upgraded to the latest this week.

Is it possible for you to confirm the changes I made for the spartn messages?

[ARK3r]

@hortovanyi Thanks for your work on the the rxm-spartn and rxm-spartnkey.

I'm just confused about how I would provide the keys. This is a good example of how a json file gets parsed, and this shows which data bits are used and here they are put together as a UBX msg to be sent.

Regarding spartn message this is what I'm getting:

header:
  stamp:
    sec: 1726089657
    nanosec: 634434780
  frame_id: gnss
version: 1
msg_used: 1
sub_type: 0
msg_type: 1
---
header:
  stamp:
    sec: 1726089657
    nanosec: 641821699
  frame_id: gnss
version: 1
msg_used: 1
sub_type: 0
msg_type: 0
---
header:
  stamp:
    sec: 1726089657
    nanosec: 651982608
  frame_id: gnss
version: 1
msg_used: 1
sub_type: 1
msg_type: 0
---
header:
  stamp:
    sec: 1726089657
    nanosec: 661819118
  frame_id: gnss
version: 1
msg_used: 1
sub_type: 2
msg_type: 0
---
header:
  stamp:
    sec: 1726089657
    nanosec: 671816052
  frame_id: gnss
version: 1
msg_used: 1
sub_type: 0
msg_type: 1

which I assume means spartn messages are being recieved but not integrated, as I have not yet provided the key.

[ARK3r]

I'll check the out frame right now and get back to you.

I upgraded the firmware last week as well to: ZED-F9P L1/L2 HPG 1.50 firmware 25-Jul-2024

[hortovanyi]

@ARK3r thanks 😊 ... ohh I see, so we need a separate node to get the keys and then need to publish a msg for ublox_dgnss_node to subscribe to, to then send the spartnkeys to the device??

[ARK3r]

I pushed the out frame debug info to my branch, not sure how I can reopen this pr or if I should make another one.

I did make a super hacky script recently, let me check if spartn messages are being decrypted properly using this method, and then we can integrate it in somehow

super hacky script:

from sys import argv
from time import sleep
import os

from serial import Serial
from pyubx2 import UBX_PROTOCOL, UBXMessage, UBXReader, POLL, val2bytes, SET, U1, U2, U4

from datetime import datetime, timedelta

from spartn_json_config import SpartnJsonConfig

GPSEPOCH0 = datetime(1980, 1, 6)  # for Wno and Tow calculations
RESERVED0 = b"\x00\x00"
RESERVED1 = b"\x00"

def date2wnotow(dat: datetime) -> tuple:
    """
    Get GPS Week number (Wno) and Time of Week (Tow)
    for given datetime.

    GPS Epoch 0 = 6th Jan 1980

    :param datetime dat: calendar date
    :return: tuple of (Wno, Tow)
    :rtype: tuple
    """

    wno = int((dat - GPSEPOCH0).days / 7)
    tow = ((dat.weekday() + 1) % 7) * 86400
    return wno, tow

def send_keys():
    json_file = os.path.join(os.path.expanduser("~"), "keys.json")

    spartn_json_config = SpartnJsonConfig(json_file)

    spartn_key1 = spartn_json_config.current_key
    spartn_key2 = spartn_json_config.next_key

    print(spartn_key1)

    version = val2bytes(1, U1)
    numKeys = val2bytes(2, U1)
    key1 = bytes.fromhex(spartn_key1[0])
    keylen1 = val2bytes(len(key1), U1)
    wno1, tow1 = date2wnotow(spartn_key1[1])
    wno1b = val2bytes(wno1, U2)
    tow1b = val2bytes(tow1, U4)
    key2 = bytes.fromhex(spartn_key2[0])
    keylen2 = val2bytes(len(key2), U1)
    wno2, tow2 = date2wnotow(spartn_key2[1])
    wno2b = val2bytes(wno2, U2)
    tow2b = val2bytes(tow2, U4)

    payload = (
        version
        + numKeys
        + RESERVED0
        + RESERVED1
        + keylen1
        + wno1b
        + tow1b
        + RESERVED1
        + keylen2
        + wno2b
        + tow2b
        + key1
        + key2
    )
    msg = UBXMessage("RXM", "RXM-SPARTN-KEY", SET, payload=payload)

    return msg

def main(**kwargs):
    """
    Main routine.
    """


    port = kwargs.get("port", "/dev/ttyACM0")
    baudrate = int(kwargs.get("baudrate", 38400))
    timeout = float(kwargs.get("timeout", 0.1))

    with Serial(port, baudrate, timeout=timeout) as stream:
        ubr = UBXReader(stream, protfilter=UBX_PROTOCOL)

        print("\nStarting. Press Ctrl-C to terminate...")

        try:
            while True:
                # Polling RXM-SPARTN-KEY message type
                print("Polling RXM-SPARTN-KEY message type...")
                msg = UBXMessage("RXM", "RXM-SPARTN-KEY", POLL)
                stream.write(msg.serialize())
                
                # Check if data is available and read it
                if stream.in_waiting:
                    try:
                        raw_data, parsed_data = ubr.read()
                        if parsed_data and parsed_data.identity == "RXM-SPARTN-KEY":
                            print("has keys" if parsed_data.numKeys > 0 else "no keys found")
                            if parsed_data.numKeys > 0:
                                break
                            else:
                                keys_msg = send_keys()
                                stream.write(keys_msg.serialize())
                            # print(parsed_data.numKeys)  # Directly print parsed data
                            # print(parsed_data.identity)
                    except Exception as err:
                        print(f"\n\nSomething went wrong {err}\n\n")

                # Sleep briefly to avoid busy-waiting and control polling rate
                sleep(1)
        
            print(parsed_data)

        except KeyboardInterrupt:  # capture Ctrl-C
            print("\n\nTerminated by user.")

        print("\nProcessing complete")


if __name__ == "__main__":

    main(**dict(arg.split("=") for arg in argv[1:]))

[ARK3r]

The only thing I wanted to point out is that this is params I ended up using:

  params = [
        {'FRAME_ID': "gnss"},
        # {'DEVICE_SERIAL_STRING': '/dev/ttyUbloxGPS'},
        {'CFG_USBOUTPROT_NMEA': False},
        {'CFG_RATE_MEAS': 10},
        {'CFG_RATE_NAV': 100},
        {'CFG_MSGOUT_UBX_NAV_HPPOSLLH_USB': 1},
        {'CFG_MSGOUT_UBX_NAV_STATUS_USB': 1},
        {'CFG_MSGOUT_UBX_NAV_COV_USB': 1},
        {'CFG_MSGOUT_UBX_NAV_SAT_USB': 1},
        {'CFG_MSGOUT_UBX_NAV_RELPOSNED_USB': 1},
        {'CFG_SPARTN_USE_SOURCE': 1},
        {'CFG_UART2INPROT_SPARTN': True},
        {'CFG_MSGOUT_UBX_RXM_RTCM_USB': 1},
        {'CFG_MSGOUT_UBX_RXM_SPARTN_USB': 1},
        {'CFG_MSGOUT_UBX_RXM_COR_USB': 1},
        {'CFG_MSGOUT_UBX_MON_COMMS_USB': 1},
        # {'CFG_MSGOUT_UBX_MON_COMMS_USB': 1},
        
    ]

because there was some sort of an error saying CFG_UART2INPROT_SPARTN expects boolean and not integer

[hortovanyi]

@ARK3r have a look at ubx_esf_meas.hpp it constructs a poll frame with msg data to send to the device ... the ubx_rxm_spartnkey I just realised is an in/out message so similar scenario to ubx_esf_meas

[hortovanyi]

i've tried to keep all the code besides the launch files as c++ ... we could set up a ros2 service to receive the keys?? (and then publish them) or otherwise are you suggesting to pass the json file name as a parameter (it not specified dont do anything or if there at a later state after initialisation send to the device once off)??

Am not sure if some of the other implementations are reading the keys straight from the internet? and then sending to the device?

I'll leave this decision up to you at present. Just need to make sure that if people are using NTRIP instead, that the node still works the same

[ARK3r]

This is what the structure of the json file that I got from thingstream looks like (with the values redacted):

- MQTT:
        - Connectivity:
                - Name: **********
                - ClientID: **********
                - Protocol: **********
                - ServerURI: **********
                - ClientCredentials:
                        - Type: **********
                        - Key: **********
                        - Cert: **********
                        - RootCA: **********
                - CleanSession: True
                - ConnectionTimeout: **********
                - KeepAliveInterval: **********
                - SSL:
                        - Mode: **********
                        - Protocol: **********
        - Subscriptions:
                - Key:
                        - QoS: **********
                        - KeyTopics: **********
                - AssistNow:
                        - QoS: **********
                        - AssistNowTopics: **********
                - Data:
                        - QoS: **********
                        - DataTopics: **********
        - dynamickeys:
                - current:
                        - start: **********
                        - duration: **********
                        - value: **********
                - next:
                        - start: **********
                        - duration: **********
                        - value: **********

This structure suggests there must be more ways other than just providing the dynamic keys and using those to decrypt the spartn messages, however, at the moment I have only experimented with providing the keys and the f9p has achieved "fixed" state by connection to the d9s.

I think a service that accepts these keys with a custom message actually sounds good, and it offloads the work of processing the json file, or if someone creates their own way of passing the keys to the device connected to the module, I think a service would be the most versatile solution.

I spent some time looking at the UBX-ESF-MEAS message and I'm afraid it looks way too complicated for me to figure out at the moment. Is it possible that you implement this service? If not, I can work on it sometime next week.

[hortovanyi]

@ARK3r need to create a new Full Payload to send messages to the device ... it basically is taking the ros2 msg and creating a UBX binary payload to send (ie poll) out. look at load_from_msg method

ublox_dgnss/ublox_dgnss_node/include/ublox_dgnss_node/ubx/esf/ubx_esf_meas.hpp

Line 136 in 0c37a48

class ESFMeasFullPayload : UBXPayload

I suspect from what I'm seeing in this chat, is that a new node will be needed as a MQTT client as there are subscription and dynamic keys ... the ubx message allows for dynamic key sizes. I had to think twice about what I coded up for all of that. Extrapolating from the keys binary to the ubx poll payload, while tricky is not impossible - its a bit different from the other messages so takes some time.

I dont know the frequency of change in any of the those keys ?? at first I had thought they were relatively static but looking at your last comment makes me think not.

Have you created a MQTT client in c++ before for a ros2 node? I haven't and am not sure if the std ros2 build farm libraries include the client??? Normally this stuff when I try it, works and just compiles. I suspect the mqtt clients are standard libraries in linux distributions.

There doesnt seem to be anything here that suggests its not worthwhile doing, however I cant test it all so probably best for you to code it when you can. Can help you if you get stuck

[hortovanyi]

@ARK3r any progress? if stuck let me know and I'll help best I can

[ARK3r]

I have been pulled to other things at the moment, once I have to get back on this I'll reach back out.

[hortovanyi]

@ARK3r ok. I might set up an Issue for this and move the conversation there