Introduce explicit object model for device streams #342

glopesdev · 2024-03-20T15:09:53Z

As discussed in #334 this PR introduces a new API and object model for describing device stream schemas. Below we introduce the proposed API design, and provide some examples of usage. The proposed design is fully backwards compatible, so it can be used interchangeably with the previous approach of using functions for stream definition.

Detailed Design

The main change introduced by the new API is to replace functions and static classes with two base classes: Stream and StreamGroup.

Stream

The Stream class represents a single device stream, and defines the __iter__ method which is used to update the schema dictionary by automatically using the name of the class as dictionary key. This is the main advantage over the previous approach, where functions returned dictionaries with a single arbitrary key, which was both repetitive and error prone since there was no validation on the relationship between the name of the key and the name of the function.

StreamGroup

The StreamGroup class represents a group of device streams to allow more succinct declarations of device schemas when using multiple instances of the same device. The class also implements the __iter__ method, which allows us to remove the special case for classes which was present in the implementation of compositeStream.

We are adding a variable argument list in the constructor to make it easier to both declare nested classes as streams, or reuse other existing stream classes in the composition.

Usage Examples

To illustrate how to use the new API, we contrast it side-by-side with example definitions taken from the current architecture.

Simple Streams

These are single stream schemas where each chunk is named by a pattern composed by the device name, followed by a standard suffix and file extension.

old API

def position(pattern):
    """Position tracking data for the specified camera."""
    return {"Position": _reader.Position(f"{pattern}_200_*")}

new API

class Position(Stream):
    """Position tracking data for the specified camera."""
    def __init__(self, pattern):
        super().__init__(_reader.Position(f"{pattern}_200_*"))

Stream Groups

These are reusable logical groups of schemas applied to a single device. They are meant to be used as macros or templates for a composition of streams which is usually repeated across a large number of devices in a specific experiment.

old API

def environment(pattern):
    """Metadata for environment mode and subjects."""
    return _device.compositeStream(pattern, environment_state, subject_state)

new API

class Environment(StreamGroup):
    """Metadata for environment mode and subjects."""
    def __init__(self, pattern):
        super().__init__(pattern, EnvironmentState, SubjectState)

Stream Group Classes

The contents of reusable stream groups can also be specified by defining nested stream classes.

old API

class TaskLogic:
    @staticmethod
    def trial_initiation(pattern):
        return {"TrialInitiation": _reader.Harp(f"{pattern}_1_*", columns=["trial_type"])}

    @staticmethod
    def response(pattern):
        return {"Response": _reader.Harp(f"{pattern}_2_*", columns=["wall_id", "poke_id"])}

new API

class TaskLogic(StreamGroup):
    def __init__(self, path):
        super().__init__(path)

    class TrialInitiation(Stream):
        def __init__(self, pattern):
            super().__init__(_reader.Harp(f"{pattern}_1_*", columns=["trial_type"]))

    class Response(Stream):
        def __init__(self, pattern):
            super().__init__(_reader.Harp(f"{pattern}_2_*", columns=["wall_id", "poke_id"]))

Summary by CodeRabbit

Release Notes:

New Feature: Introduced Stream, StreamGroup, and Device classes for enhanced data stream management.
Refactor: Updated the Device class to handle multiple data streams and create streams based on provided arguments.
Deprecation: Certain usage patterns have been deprecated with warnings introduced for these cases. Users are advised to update their code accordingly.
Change: Altered signatures of exported functions and global data structures, which may affect external interface and behavior. Please review your integration if you're using these interfaces.

Please note that these changes involve significant modifications to the code structure and functionality.

github-actions

CodeRabbit

Commits

Files that changed from the base of the PR and between 616155b and e85986b commits.

Files selected (2)

aeon/io/device.py (2)
aeon/io/streams.py (1)

Review comments generated (6)

Review: 5
LGTM: 1

Tips

Chat with CodeRabbit Bot (`@coderabbitai`)

Reply on review comments left by this bot to ask follow-up questions. A review comment is a comment on a diff or a file.
Invite the bot into a review comment chain by tagging @coderabbitai in a reply.

Code suggestions

The bot may make code suggestions, but please review them carefully before committing since the line number ranges may be misaligned.
You can edit the comment made by the bot and manually tweak the suggestion if it is slightly off.

Pausing incremental reviews

Add @coderabbitai: ignore anywhere in the PR description to pause further reviews from the bot.

aeon/io/streams.py

github-actions

CodeRabbit

Commits

Files that changed from the base of the PR and between e85986b and 1e499c7 commits.

Files selected (1)

aeon/io/streams.py (1)

Review comments generated (5)

Review: 2
LGTM: 3

Tips

Chat with CodeRabbit Bot (`@coderabbitai`)

Reply on review comments left by this bot to ask follow-up questions. A review comment is a comment on a diff or a file.
Invite the bot into a review comment chain by tagging @coderabbitai in a reply.

Code suggestions

The bot may make code suggestions, but please review them carefully before committing since the line number ranges may be misaligned.
You can edit the comment made by the bot and manually tweak the suggestion if it is slightly off.

Pausing incremental reviews

Add @coderabbitai: ignore anywhere in the PR description to pause further reviews from the bot.

aeon/io/streams.py

github-actions

CodeRabbit

Commits

Files that changed from the base of the PR and between 1e499c7 and 31721cc commits.

Files selected (1)

aeon/io/streams.py (1)

Review comments generated (5)

Review: 2
LGTM: 3

Tips

Chat with CodeRabbit Bot (`@coderabbitai`)

Reply on review comments left by this bot to ask follow-up questions. A review comment is a comment on a diff or a file.
Invite the bot into a review comment chain by tagging @coderabbitai in a reply.

Code suggestions

The bot may make code suggestions, but please review them carefully before committing since the line number ranges may be misaligned.
You can edit the comment made by the bot and manually tweak the suggestion if it is slightly off.

Pausing incremental reviews

Add @coderabbitai: ignore anywhere in the PR description to pause further reviews from the bot.

aeon/io/streams.py

github-actions

CodeRabbit

Commits

Files that changed from the base of the PR and between 31721cc and dc847ab commits.

Files selected (1)

aeon/io/streams.py (1)

Review comments generated (5)

Review: 2
LGTM: 3

Tips

Chat with CodeRabbit Bot (`@coderabbitai`)

Reply on review comments left by this bot to ask follow-up questions. A review comment is a comment on a diff or a file.
Invite the bot into a review comment chain by tagging @coderabbitai in a reply.

Code suggestions

The bot may make code suggestions, but please review them carefully before committing since the line number ranges may be misaligned.
You can edit the comment made by the bot and manually tweak the suggestion if it is slightly off.

Pausing incremental reviews

Add @coderabbitai: ignore anywhere in the PR description to pause further reviews from the bot.

aeon/io/streams.py

github-actions

CodeRabbit

Commits

Files that changed from the base of the PR and between 616155b and 31721cc commits.

Files selected (2)

aeon/io/device.py (2)
aeon/io/streams.py (1)

Review comments generated (7)

Review: 4
LGTM: 3

Tips

Chat with CodeRabbit Bot (`@coderabbitai`)

Reply on review comments left by this bot to ask follow-up questions. A review comment is a comment on a diff or a file.
Invite the bot into a review comment chain by tagging @coderabbitai in a reply.

Code suggestions

The bot may make code suggestions, but please review them carefully before committing since the line number ranges may be misaligned.
You can edit the comment made by the bot and manually tweak the suggestion if it is slightly off.

Pausing incremental reviews

Add @coderabbitai: ignore anywhere in the PR description to pause further reviews from the bot.

aeon/io/device.py

aeon/io/streams.py

github-actions

CodeRabbit

Commits

Files that changed from the base of the PR and between 31721cc and 8f4a90b commits.

Files selected (1)

aeon/io/streams.py (1)

Review comments generated (3)

Review: 2
LGTM: 1

Tips

Chat with CodeRabbit Bot (`@coderabbitai`)

Reply on review comments left by this bot to ask follow-up questions. A review comment is a comment on a diff or a file.
Invite the bot into a review comment chain by tagging @coderabbitai in a reply.

Code suggestions

The bot may make code suggestions, but please review them carefully before committing since the line number ranges may be misaligned.
You can edit the comment made by the bot and manually tweak the suggestion if it is slightly off.

Pausing incremental reviews

Add @coderabbitai: ignore anywhere in the PR description to pause further reviews from the bot.

aeon/io/streams.py

github-actions

CodeRabbit

Commits

Files that changed from the base of the PR and between 8f4a90b and b5627a6 commits.

Files selected (2)

aeon/io/device.py (2)
aeon/io/streams.py (1)

Files not summarized due to errors (2)

aeon/io/streams.py (nothing obtained from openai)
aeon/io/device.py (nothing obtained from openai)

Files not reviewed due to errors (2)

aeon/io/streams.py (no response)
aeon/io/device.py (no response)

Review comments generated (0)

Review: 0
LGTM: 0

Tips

Chat with CodeRabbit Bot (`@coderabbitai`)

Reply on review comments left by this bot to ask follow-up questions. A review comment is a comment on a diff or a file.
Invite the bot into a review comment chain by tagging @coderabbitai in a reply.

Code suggestions

The bot may make code suggestions, but please review them carefully before committing since the line number ranges may be misaligned.
You can edit the comment made by the bot and manually tweak the suggestion if it is slightly off.

Pausing incremental reviews

Add @coderabbitai: ignore anywhere in the PR description to pause further reviews from the bot.

github-actions

CodeRabbit

Commits

Files that changed from the base of the PR and between b5627a6 and e5e75f0 commits.

Files selected (1)

aeon/io/streams.py (1)

Files not summarized due to errors (1)

aeon/io/streams.py (nothing obtained from openai)

Files not reviewed due to errors (1)

aeon/io/streams.py (no response)

Review comments generated (0)

Review: 0
LGTM: 0

Tips

Chat with CodeRabbit Bot (`@coderabbitai`)

Reply on review comments left by this bot to ask follow-up questions. A review comment is a comment on a diff or a file.
Invite the bot into a review comment chain by tagging @coderabbitai in a reply.

Code suggestions

The bot may make code suggestions, but please review them carefully before committing since the line number ranges may be misaligned.
You can edit the comment made by the bot and manually tweak the suggestion if it is slightly off.

Pausing incremental reviews

Add @coderabbitai: ignore anywhere in the PR description to pause further reviews from the bot.

github-actions · 2024-03-20T21:57:05Z

CodeRabbit

Uplevel your code reviews with CodeRabbit Pro

CodeRabbit Pro

If you like this project, please support us by purchasing the Pro version. The Pro version has advanced context, superior noise reduction and several proprietary improvements compared to the open source version. Moreover, CodeRabbit Pro is free for open source projects.

github-actions

CodeRabbit

Commits

Files that changed from the base of the PR and between 616155b and 9220b1e commits.

Files selected (3)

aeon/io/device.py (2)
aeon/schema/streams.py (1)
tests/io/test_api.py (1)

Files not summarized due to errors (3)

tests/io/test_api.py (nothing obtained from openai)
aeon/io/device.py (nothing obtained from openai)
aeon/schema/streams.py (nothing obtained from openai)

Files not reviewed due to errors (3)

aeon/io/device.py (no response)
aeon/schema/streams.py (no response)
tests/io/test_api.py (no response)

Review comments generated (0)

Review: 0
LGTM: 0

Tips

Chat with CodeRabbit Bot (`@coderabbitai`)

Reply on review comments left by this bot to ask follow-up questions. A review comment is a comment on a diff or a file.
Invite the bot into a review comment chain by tagging @coderabbitai in a reply.

Code suggestions

The bot may make code suggestions, but please review them carefully before committing since the line number ranges may be misaligned.
You can edit the comment made by the bot and manually tweak the suggestion if it is slightly off.

Pausing incremental reviews

Add @coderabbitai: ignore anywhere in the PR description to pause further reviews from the bot.

github-actions

CodeRabbit

Commits

Files that changed from the base of the PR and between 9220b1e and 258900f commits.

Files selected (4)

aeon/schema/core.py (1)
aeon/schema/dataset.py (1)
aeon/schema/foraging.py (2)
aeon/schema/octagon.py (1)

Files not summarized due to errors (4)

aeon/schema/octagon.py (diff tokens exceeds limit)
aeon/schema/core.py (nothing obtained from openai)
aeon/schema/dataset.py (nothing obtained from openai)
aeon/schema/foraging.py (nothing obtained from openai)

Files not reviewed due to errors (4)

aeon/schema/dataset.py (no response)
aeon/schema/core.py (no response)
aeon/schema/octagon.py (no response)
aeon/schema/foraging.py (no response)

Review comments generated (0)

Review: 0
LGTM: 0

Tips

Chat with CodeRabbit Bot (`@coderabbitai`)

Reply on review comments left by this bot to ask follow-up questions. A review comment is a comment on a diff or a file.
Invite the bot into a review comment chain by tagging @coderabbitai in a reply.

Code suggestions

The bot may make code suggestions, but please review them carefully before committing since the line number ranges may be misaligned.
You can edit the comment made by the bot and manually tweak the suggestion if it is slightly off.

Pausing incremental reviews

Add @coderabbitai: ignore anywhere in the PR description to pause further reviews from the bot.

glopesdev added 3 commits March 20, 2024 12:29

Add object model for new stream io infrastructure

19e925e

Support legacy stream group classes

a9eac91

Ensure deprecation message

e85986b

glopesdev requested review from jkbhagatio, lochhh and ttngu207 March 20, 2024 15:09