Create EventSource and EventSink handles #1267

cardenaso11 · 2024-01-22T17:24:37Z

CHANGELOG updated or not needed
Documentation updated or not needed
Haddocks updated or not needed
No new TODOs introduced or explained herafter

ch1bo · 2024-01-23T07:54:52Z

@cardenaso11 Hey it's great to see you started work on implementing ADR29!

I'm a bit surprised though that you would update the Persistence handle instead of adding EventSource and EventSink(s) in the central HydraNode handle as designed?

Happy to have an alignment chat on discord or so about this?

ch1bo

We discussed the approach and also drafted the call sites here: 4a281ce

hydra-node/src/Hydra/Node.hs

hydra-node/src/Hydra/Persistence.hs

ch1bo

Wasn't sure if this is ready to review already, but left some comments nontheless.

I think you would benefit a lot from not changing the HeadLogic and API Server as this results in a lot of noise and is a too big of a step (see my "Must" comments)

hydra-node/src/Hydra/Persistence.hs

hydra-node/src/Hydra/API/Server.hs

hydra-node/src/Hydra/HeadLogic/Outcome.hs

hydra-node/src/Hydra/HeadLogic.hs

hydra-node/src/Hydra/Network/Reliability.hs

hydra-node/src/Hydra/Node.hs

hydra-node/src/Hydra/Persistence.hs

hydra-node/test/Hydra/HeadLogicSpec.hs

hydra-node/test/Hydra/PersistenceSpec.hs

…entSink.\n\nIt's not clear yet if we can have the statechange ID in the StateChanged type as a substitute for tracking the current stateChangeID in the node\nBut for now it works to get the node's stateChangedID into the disk-persistence eventSink's putEvent', where we can compare it against the last persisted stateChange\nThis works for the particular disk-based eventSink where we *do have* exactly-once, but in ex kinesis poc or redis or something, it can be the key for at-least-once dict lint

…to minimize first PR

lint

We can define an event source and event sink from the PersistenceIncremental.

…irectly

Having a dedicated module allows easier import in forks of hydra and hopefully serves as a better extension points (with less breakage).

We kept the EventSource/EventSink very abstract to make implementations not realy on the internas of the actual data type used. However, an event source / sink will need at least identify individual events to tell them apart, e.g. to deduplicate them in memory.

Current idea: Keep notes where extensions are possible so we don't change things by accident breaking external/custom code of users.

We do not want to update this module (yet) with a different means for persistence.

By making the step-wise construction of HydraNode monadic (although not specifically needed) we can simplify call-sites in the NodeSpec.

This is unfortunately very "white-boxy" and the testing code needs to know that the generated events must be using parameters consistent with the environment. The alternative would be to move checkHeadState outside of hydrate, but then 'WetHydraNode's can exist without this check happening.

ch1bo · 2024-03-13T10:35:01Z

Superseded by #1351

cardenaso11 changed the title ~~Persistence (WIP)~~ Persistence Updates (WIP) Jan 22, 2024

ch1bo assigned ch1bo and cardenaso11 Jan 23, 2024

ch1bo reviewed Jan 31, 2024

View reviewed changes

hydra-node/src/Hydra/Node.hs Outdated Show resolved Hide resolved

hydra-node/src/Hydra/Persistence.hs Outdated Show resolved Hide resolved

hydra-node/src/Hydra/Persistence.hs Show resolved Hide resolved

ch1bo removed their assignment Feb 2, 2024

cardenaso11 force-pushed the SB-1352-persistence-types branch from f23b790 to fe72281 Compare February 15, 2024 17:14

cardenaso11 marked this pull request as ready for review February 21, 2024 09:48

cardenaso11 force-pushed the SB-1352-persistence-types branch from 4208eaf to 067edec Compare February 21, 2024 10:07

ch1bo self-requested a review February 21, 2024 10:35

ch1bo requested changes Feb 21, 2024

View reviewed changes

ch1bo mentioned this pull request Feb 26, 2024

Streaming Plugins #1325

Closed

3 tasks

ch1bo force-pushed the SB-1352-persistence-types branch 6 times, most recently from 95ea20a to 599b405 Compare March 4, 2024 17:54

ch1bo marked this pull request as draft March 4, 2024 17:55

ch1bo changed the title ~~Persistence Updates (WIP)~~ Create EventSource and EventSink handles Mar 4, 2024

ch1bo force-pushed the SB-1352-persistence-types branch 2 times, most recently from e379909 to c92df27 Compare March 4, 2024 18:34

cardenaso11 added 8 commits March 11, 2024 16:18

initial work (WIP)

22ead15

WIP checkpoint

4182e5b

checkpoint apiserver

2585923

remove inaccurate fixme

b0576d0

checkpoint: lib build

94d2995

WIP: going back and forth on how to manage state in tests

a0133ae

eventid

16890a2

cardenaso11 and others added 13 commits March 11, 2024 17:06

remove newpersistence data type (in exchange for type synonym) stuff …

e4fe10e

…to minimize first PR

checkpoint for testing

d692b7d

some tests

2b22fe2

checkpoint: fix all hydra-node tests besides golden

b72c15d

lint

Use eventPairFromPersistenceIncremental

e0be84a

Use the original PersistenceIncremental in withAPIServer

f8be89f

Only use eventPairFromPersistenceIncremental

3679cf5

We can define an event source and event sink from the PersistenceIncremental.

remove NewPersistenceIncremental type, use eventSource + eventSinks d…

11da78f

…irectly

Move EventSource and EventSink definitions into Hydra.Events

02f3c46

Having a dedicated module allows easier import in forks of hydra and hopefully serves as a better extension points (with less breakage).

Draft extension points in Hydra.Node.Run

0e59d98

Current idea: Keep notes where extensions are possible so we don't change things by accident breaking external/custom code of users.

Rename getEvents' -> getEvents and putEvent' -> putEvent

f0ce297

Restore usage of persistenceIncremental in Hydra.Network.Reliability

8a00c06

We do not want to update this module (yet) with a different means for persistence.

ch1bo force-pushed the SB-1352-persistence-types branch from c92df27 to 8a00c06 Compare March 11, 2024 16:08

Draft a new way to wire HydraNode in Node.Run

dbd3be3

ch1bo changed the base branch from master to rename-events March 11, 2024 17:49

ch1bo added 5 commits March 11, 2024 19:55

Add node-level tests for event source / sink handling

a058ff8

Unify Hydra.Node function signatures and refine NodeSpec

68e9ab4

By making the step-wise construction of HydraNode monadic (although not specifically needed) we can simplify call-sites in the NodeSpec.

Merge loadStateEventSource into hydrate function

e5c2eec

Move Environment into dedicated module

46d83d4

ch1bo force-pushed the SB-1352-persistence-types branch 2 times, most recently from 2a1d836 to 644d232 Compare March 12, 2024 10:21

ch1bo self-assigned this Mar 12, 2024

Add more tests on event ids being strictly monotonic

fa7d970

ch1bo force-pushed the SB-1352-persistence-types branch from 644d232 to fa7d970 Compare March 12, 2024 11:13

v0d1ch deleted the branch cardano-scaling:rename-events March 12, 2024 16:03

v0d1ch closed this Mar 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create EventSource and EventSink handles #1267

Create EventSource and EventSink handles #1267

cardenaso11 commented Jan 22, 2024

ch1bo commented Jan 23, 2024 •

edited

Loading

ch1bo left a comment

ch1bo left a comment

ch1bo commented Mar 13, 2024

Create EventSource and EventSink handles #1267

Create EventSource and EventSink handles #1267

Conversation

cardenaso11 commented Jan 22, 2024

ch1bo commented Jan 23, 2024 • edited Loading

ch1bo left a comment

Choose a reason for hiding this comment

ch1bo left a comment

Choose a reason for hiding this comment

ch1bo commented Mar 13, 2024

ch1bo commented Jan 23, 2024 •

edited

Loading