Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Emit when the CDC event was executed #930

Merged
merged 9 commits into from
Sep 26, 2024
Merged

Emit when the CDC event was executed #930

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
259f7cb
Expose execution time.
Tang8330 Sep 26, 2024
d5c2c88
Clean up.
Tang8330 Sep 26, 2024
2a6b204
Clean up.
Tang8330 Sep 26, 2024
ad40320
Update comment.
Tang8330 Sep 26, 2024
4043af0
Remove extra line.
Tang8330 Sep 26, 2024
3bcc60b
Merge branch 'master' into ingestion-lag
Tang8330 Sep 26, 2024
f5d0e6e
PR Feedback.
Tang8330 Sep 26, 2024
96184fb
Update.
Tang8330 Sep 26, 2024
0a42142
Remove comment.
Tang8330 Sep 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 22 additions & 6 deletions models/event/event.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ import (
"github.com/artie-labs/transfer/lib/kafkalib"
"github.com/artie-labs/transfer/lib/optimization"
"github.com/artie-labs/transfer/lib/stringutil"
"github.com/artie-labs/transfer/lib/telemetry/metrics/base"
"github.com/artie-labs/transfer/lib/typing"
"github.com/artie-labs/transfer/lib/typing/columns"
"github.com/artie-labs/transfer/models"
Expand All @@ -28,10 +29,11 @@ type Event struct {

OptionalSchema map[string]typing.KindDetails
Columns *columns.Columns
ExecutionTime time.Time // When the SQL command was executed
Deleted bool

mode config.Mode
// When the database event was executed
executionTime time.Time
mode config.Mode
}

func hashData(data map[string]any, tc kafkalib.TopicConfig) map[string]any {
Expand Down Expand Up @@ -86,16 +88,30 @@ func ToMemoryEvent(event cdc.Event, pkMap map[string]any, tc kafkalib.TopicConfi
return Event{}, err
}

return Event{
_event := Event{
executionTime: event.GetExecutionTime(),
mode: cfgMode,
Table: tblName,
PrimaryKeyMap: pkMap,
ExecutionTime: event.GetExecutionTime(),
OptionalSchema: optionalSchema,
Columns: cols,
Data: hashData(evtData, tc),
Deleted: event.DeletePayload(),
}, nil
}

return _event, nil
}

// EmitExecutionTimeLag - This will check against the current time and the event execution time and emit the lag.
func (e *Event) EmitExecutionTimeLag(metricsClient base.Client, mode config.Mode) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the mode need to be passed in, or can we use e.mode?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good flag. We should be able to just use e.mode!

metricsClient.GaugeWithSample(
"row.execution_time_lag",
float64(time.Since(e.executionTime).Milliseconds()),
map[string]string{
"mode": mode.String(),
"table": e.Table,
},
0.5)
}

func (e *Event) Validate() error {
Expand Down Expand Up @@ -247,7 +263,7 @@ func (e *Event) Save(cfg config.Config, inMemDB *models.DatabaseData, tc kafkali
td.PartitionsToLastMessage[message.Partition()] = append(td.PartitionsToLastMessage[message.Partition()], message)
}

td.LatestCDCTs = e.ExecutionTime
td.LatestCDCTs = e.executionTime
flush, flushReason := td.ShouldFlush(cfg)
return flush, flushReason, nil
}
3 changes: 3 additions & 0 deletions processes/consumer/process.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ func (p processArgs) process(ctx context.Context, cfg config.Config, inMemDB *mo
tags["what"] = "to_mem_event_err"
return "", fmt.Errorf("cannot convert to memory event: %w", err)
}

// Table name is only available after event has been cast
tags["table"] = evt.Table
if topicConfig.tc.ShouldSkip(_event.Operation()) {
Expand All @@ -72,6 +73,8 @@ func (p processArgs) process(ctx context.Context, cfg config.Config, inMemDB *mo
return evt.Table, nil
}

// Emit execution time lag for non-skipped events.
evt.EmitExecutionTimeLag(metricsClient, cfg.Mode)
shouldFlush, flushReason, err := evt.Save(cfg, inMemDB, topicConfig.tc, p.Msg)
if err != nil {
tags["what"] = "save_fail"
Expand Down