Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement stream specs #11685

Merged
merged 5 commits into from
Jan 11, 2024
Merged

Implement stream specs #11685

merged 5 commits into from
Jan 11, 2024

Conversation

samsondav
Copy link
Collaborator

No description provided.

Copy link
Contributor

github-actions bot commented Jan 4, 2024

I see that you haven't updated any CHANGELOG files. Would it make sense to do so?

Copy link
Contributor

github-actions bot commented Jan 4, 2024

I see that you haven't updated any README files. Would it make sense to do so?

@samsondav samsondav requested a review from a team as a code owner January 4, 2024 18:48
@samsondav samsondav marked this pull request as draft January 4, 2024 20:33
@samsondav samsondav force-pushed the MERC-2876-implement-stream-specs branch 4 times, most recently from b8a89d4 to b25acde Compare January 8, 2024 19:43
OffchainReporting: 1,
Stream: 1,
VRF: 1,
Webhook: 1,
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are simply sorted to make future additions easier.

core/services/streams/stream_registry.go Outdated Show resolved Hide resolved
initialize := s.spec.Pipeline == nil
s.RUnlock()
if initialize {
pipeline, err := s.spec.ParsePipeline()
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One of the benefits of using these stream specs is that we can avoid re-parsing the dot dag on every run and instead keep the parsed pipeline in memory.

This will probably result in some speedups especially if we are executing thousands of pipelines per second.

@bolekk
Copy link
Contributor

bolekk commented Jan 9, 2024

Is any new job type automatically supported by CLO?

@samsondav samsondav force-pushed the MERC-2876-implement-stream-specs branch 4 times, most recently from 7b95603 to d21f8ae Compare January 9, 2024 17:20
@samsondav samsondav marked this pull request as ready for review January 9, 2024 17:20
@samsondav samsondav requested review from a team and jmank88 as code owners January 9, 2024 17:20
@samsondav samsondav force-pushed the MERC-2876-implement-stream-specs branch from 9a47018 to 7227a9b Compare January 9, 2024 17:52
k, err := app.KeyStore.P2P().GetAll()
require.NoError(t, err)
require.Len(t, k, 1)
var pks []vrfkey.KeyV2
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to change this to keep the linter happy

@samsondav samsondav force-pushed the MERC-2876-implement-stream-specs branch from 7227a9b to 88791c6 Compare January 9, 2024 18:06
@jkongie
Copy link
Contributor

jkongie commented Jan 9, 2024

Is any new job type automatically supported by CLO?

Running new job types is supported automatically however each new job type will need some changes for the core node and CLO to register that they will accept jobs of that type. This is so we know how to assign the correct node for the a given job type.

@samsondav samsondav changed the title implement stream specs Implement stream specs Jan 10, 2024
@samsondav samsondav force-pushed the MERC-2876-implement-stream-specs branch from 88791c6 to 40173fb Compare January 11, 2024 15:22
@samsondav samsondav enabled auto-merge January 11, 2024 15:24
@cl-sonarqube-production
Copy link

@samsondav samsondav requested a review from jmank88 January 11, 2024 15:59
@samsondav samsondav added this pull request to the merge queue Jan 11, 2024
Merged via the queue into develop with commit dbc0f91 Jan 11, 2024
80 checks passed
@samsondav samsondav deleted the MERC-2876-implement-stream-specs branch January 11, 2024 17:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants