Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MNT/ENH: Add a raw CCSDS packet generator #107

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

greglucas
Copy link
Collaborator

@greglucas greglucas commented Oct 10, 2024

This refactors out the reading of a byte-stream into a separate generator function that can yield raw CCSDSPackets with only the raw bytes. The definitions and parsing logic can use this generator to yield packets and then parse whatever they need to after the fact. This enables someone to use this generator without a packet definition and to investigate what is in the packet stream without needing to know more detailed information about how individual packets are laid out internally.

closes #81
closes #104

Checklist

  • Changes are fully implemented without dangling issues or TODO items
  • Deprecated/superseded code is removed or marked with deprecation warning
  • Current dependencies have been properly specified and old dependencies removed
  • New code/functionality has accompanying tests and any old tests have been updated to match any new assumptions
  • The changelog.md has been updated

@greglucas greglucas requested a review from medley56 as a code owner October 10, 2024 17:58
@medley56 medley56 force-pushed the packet-parsing-refactor branch from aa806eb to ffd7316 Compare October 11, 2024 03:20
@medley56
Copy link
Owner

Nice! I like this. My current thinking is that I prefer this to your alternative idea because I want to avoid changing the user API unnecessarily and I think the use cases for examining packets as binary data only are mostly going to use an additional abstraction layer rather than users calling the packet_generator directly. I think we should leave it public in case someone decides they want to use it in a custom way but I think packet_generator is mostly going to be used through CLI tools.

@greglucas
Copy link
Collaborator Author

I think we could support backwards compatibility with the definition.packet_generator() if we want and use that to redirect to the primary packet generator location with a deprecation if we want too.

        definition = None if ccsds_headers_only else self
        for packet in packets.packet_generator(binary_data,
                                               definition=definition,
                                               buffer_read_size_bytes=buffer_read_size_bytes,
                                               show_progress=show_progress,
                                               skip_header_bytes=skip_header_bytes):
            yield packet

The benefit I see is that there is only one generator that someone would have to know about and they could bring their own definition(s) to it. I'm envisioning that we have multiple XTCE definitions hanging around on a mission and don't want to potentially have to know which definition I need to use, so create a list and iterate through it. (Is there a way to combine XTCE definitions internally for us to consider as well?)

definitions = [XtcePacketDefinition(file) for file in list_of_files]

Currently I think we'd have to do this:

for definition in definitions:
    generator = getattr(definition, "packet_generator")
    for packet in generator(data):
        # Do something

With a single entry-point we could write it like this:

for definition in definitions:
    for packet in packets.packet_generator(data, definition=definition):
        # Do something

Thinking towards the future, we could even potentially support passing in multiple definitions with this approach and doing the iteration for a user:

for packet in packets.packet_generator(data, definition=definitions):
    # Do something

@greglucas
Copy link
Collaborator Author

I pushed two new commits. One somewhat unrelated but was easier to clean it up here by removing the **parse_kwargs that appear unused now. I can move that to a separate PR if desired. The final commit moves all of the packet parsing logic into the packets.py file now, but leaves the definition.packet_generator() around and just yields from the main generator passing through the proper arguments. Let me know your thoughts on those and I can break them off into separate PRs if you'd prefer to keep this just on adding the new functionality for now.

@greglucas greglucas force-pushed the packet-parsing-refactor branch from 21ee13c to 5cd86d2 Compare December 15, 2024 14:41
Comment on lines +445 to +446
n_bytes_data = _extract_bits(header_bytes, 32, 16) + 1
n_bytes_packet = RawPacketData.HEADER_LENGTH_BYTES + n_bytes_data
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that we aren't parsing all header items anymore here. We only parse the length of the data field so we know how long the packet is. We then create the raw bytes object that has cached properties that can do the lookup later when accessed.

return _extract_bits(self, 0, 3)

@cached_property
def type(self) -> int:
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

type is a built-in function in Python, so do we care that this is called RawPacketData.type? Should we prefix it with something like packet_type instead? To me that is redundant, but happy to change if others want a different naming.

- Add header properties to the RawPacketData class that can be used
  to look up the values directly from the bytes object.
- Add a create_ccsds_packet function that takes in integers for
  the header items and packs them for users. Especially useful for
  creating test data packets.
…arsing

Create a separate CCSDS packet generator that is dedicated to handling the input
data and yielding individual packets. The parsing functions then utilize
this generator to handle the actual data parsing.
These are unused now that the packet itself can be used to get
the previously parsed values.
@greglucas greglucas force-pushed the packet-parsing-refactor branch from 5cd86d2 to 4d4af20 Compare December 15, 2024 14:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants