Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Restructure deltacat metrics to allow for decorators #241

Closed
wants to merge 3 commits into from
Closed

Restructure deltacat metrics to allow for decorators #241

wants to merge 3 commits into from

Conversation

yankevn
Copy link
Collaborator

@yankevn yankevn commented Oct 24, 2023 •
edited
Loading

This change restructures deltacat's metrics code to allow for metrics emission via decorator functions. To accomplish this, the MetricsConfigSingleton class is introduced to make the metrics_config object globally available while enforcing immutability as much as possible.

Testing:

  • pytest
  • Tested on personal stack and verified metrics were created

@yankevn yankevn force-pushed the metrics_config_singleton branch from 0b83015 to a8f3b61 Compare October 25, 2023 20:03
@yankevn yankevn requested a review from raghumdani October 25, 2023 20:06
@yankevn yankevn marked this pull request as ready for review October 25, 2023 20:23
raghumdani
raghumdani reviewed Oct 25, 2023
View reviewed changes
Copy link
Collaborator

@raghumdani raghumdani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not a full review as I have only taken a glance. I have left comments for the main things I saw in the implementation that we may need to discuss.

deltacat/compute/compactor_v2/model/merge_result.py Outdated Show resolved Hide resolved
deltacat/compute/compactor_v2/steps/merge.py Show resolved Hide resolved
deltacat/compute/compactor/compaction_session.py Show resolved Hide resolved
deltacat/compute/compactor/steps/materialize.py Show resolved Hide resolved
deltacat/utils/metrics.py Outdated Show resolved Hide resolved
@yankevn
Copy link
Collaborator Author

yankevn commented Oct 30, 2023 •
edited
Loading

Performance test IM results (3 runs each):
With changes: [271369, 271709, 260248]
Without changes: [278824, 283523, 277561]

If anything this seems to show that the changes are more efficient for whatever reason.

@yankevn yankevn force-pushed the metrics_config_singleton branch 2 times, most recently from 63232fb to 64bc7c8 Compare November 1, 2023 22:50
@yankevn yankevn force-pushed the metrics_config_singleton branch from 64bc7c8 to 7bf31f8 Compare November 28, 2023 19:44
@yankevn yankevn force-pushed the metrics_config_singleton branch from 7bf31f8 to 82c9059 Compare January 5, 2024 23:20
@raghumdani
Copy link
Collaborator

Resolved by #294

@raghumdani raghumdani closed this May 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@raghumdani raghumdani raghumdani left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@yankevn @raghumdani