optimize bucket level monitor to resolve alias to query only those time-series indices that contain docs within timeframe of range query filter in search input #1701

eirsep · 2024-10-18T01:41:23Z

Description

optimize bucket level monitor to resolve alias to query only those time-series indices that contain docs within timeframe of range query filter in search input

Related Issues

#1710

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

alerting/src/main/kotlin/org/opensearch/alerting/InputService.kt

AWSHurneyt

Changes LGTM. Approving for when the separate PRs to fix the build error is merged.

sbcd90 · 2024-10-19T08:51:17Z

alerting/src/test/kotlin/org/opensearch/alerting/MonitorRunnerServiceIT.kt

@@ -1190,6 +1191,133 @@ class MonitorRunnerServiceIT : AlertingRestTestCase() {
        assertEquals("Incorrect search result", 2, buckets.size)
    }

+    fun `test execute bucket-level monitor with alias optimization - indices not skipped`() {


can we add a test case with partial index consideration e.g. after the current monitor run, 5 new docs are added to index1, index1 is rolled over to index2, 5 new docs added to index2, index2 is rolled over to index3, 5 new docs added to index3, index3 is rolled over to index4, 5 new docs added to index4 & now bucket-level monitor run happens. So, this run considers, index1, index2, index3 & index4.

added 2 test scenarios

scenario one
12 indices in alias
1st index skipped as creation time out of time range of query contains 3 docs (3 new buckets in response)
2nd index NOT SKIPPED with creation time out of time range of query contains 3 docs (3 new buckets in response)
10 indices contains 2 docs (2 new buckets in response)
Response has only 5 buckets

scenario 2
12 indices in alias
1st index NOT SKIPPED as creation time in time range of query contains 3 docs (3 new buckets in response)
2nd index NOT SKIPPED with creation time in time range of query contains 3 docs (3 new buckets in response)
10 indices contains 2 docs (2 new buckets in response)
Response has only 8 buckets

…me-series indices that contain docs within timeframe of range query filter in search input Signed-off-by: Surya Sashank Nistala <[email protected]>

Signed-off-by: Surya Sashank Nistala <[email protected]>

…ces being skipped and not skipped Signed-off-by: Surya Sashank Nistala <[email protected]>

Signed-off-by: Surya Sashank Nistala <[email protected]>

…me-series indices that contain docs within timeframe of range query filter in search input (#1701) * optimize bucket level monitor to resolve alias to query only those time-series indices that contain docs within timeframe of range query filter in search input Signed-off-by: Surya Sashank Nistala <[email protected]> * add tests to verify alias based optimziation scenarios Signed-off-by: Surya Sashank Nistala <[email protected]> * fix tests to verify aggregation query on alias optimziation with indices being skipped and not skipped Signed-off-by: Surya Sashank Nistala <[email protected]> * null check addded Signed-off-by: Surya Sashank Nistala <[email protected]> --------- Signed-off-by: Surya Sashank Nistala <[email protected]> (cherry picked from commit e3c9fc2) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…me-series indices that contain docs within timeframe of range query filter in search input (#1701) (#1712) * optimize bucket level monitor to resolve alias to query only those time-series indices that contain docs within timeframe of range query filter in search input * add tests to verify alias based optimziation scenarios * fix tests to verify aggregation query on alias optimziation with indices being skipped and not skipped * null check addded --------- (cherry picked from commit e3c9fc2) Signed-off-by: Surya Sashank Nistala <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…me-series indices that contain docs within timeframe of range query filter in search input (#1701) (#1713) * optimize bucket level monitor to resolve alias to query only those time-series indices that contain docs within timeframe of range query filter in search input * add tests to verify alias based optimziation scenarios * fix tests to verify aggregation query on alias optimziation with indices being skipped and not skipped * null check addded --------- (cherry picked from commit e3c9fc2) Signed-off-by: Surya Sashank Nistala <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…me-series indices that contain docs within timeframe of range query filter in search input (#1701) (#1711) * optimize bucket level monitor to resolve alias to query only those time-series indices that contain docs within timeframe of range query filter in search input * add tests to verify alias based optimziation scenarios * fix tests to verify aggregation query on alias optimziation with indices being skipped and not skipped * null check addded --------- (cherry picked from commit e3c9fc2) Signed-off-by: Surya Sashank Nistala <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

eirsep requested review from lezzago, AWSHurneyt, sbcd90, getsaurabh02, praveensameneni, bowenlan-amzn, rishabhmaurya, engechas, riysaxen-amzn, jowg-amazon, amsiglan and goyamegh as code owners October 18, 2024 01:41

eirsep changed the title ~~optimize bucket level monitor to resolve alias to query only those ti…~~ optimize bucket level monitor to resolve alias to query only those time-series indices that contain docs within timeframe of range query filter in search input Oct 18, 2024

AWSHurneyt reviewed Oct 18, 2024

View reviewed changes

alerting/src/main/kotlin/org/opensearch/alerting/InputService.kt Show resolved Hide resolved

AWSHurneyt approved these changes Oct 18, 2024

View reviewed changes

sbcd90 reviewed Oct 19, 2024

View reviewed changes

eirsep added 3 commits October 21, 2024 03:16

optimize bucket level monitor to resolve alias to query only those ti…

92549fc

…me-series indices that contain docs within timeframe of range query filter in search input Signed-off-by: Surya Sashank Nistala <[email protected]>

add tests to verify alias based optimziation scenarios

8bcba11

Signed-off-by: Surya Sashank Nistala <[email protected]>

fix tests to verify aggregation query on alias optimziation with indi…

4578bc8

…ces being skipped and not skipped Signed-off-by: Surya Sashank Nistala <[email protected]>

eirsep force-pushed the agg_optimization branch from 4de98fc to 4578bc8 Compare October 21, 2024 10:23

sbcd90 approved these changes Oct 21, 2024

View reviewed changes

null check addded

b8b37f0

Signed-off-by: Surya Sashank Nistala <[email protected]>

AWSHurneyt approved these changes Oct 21, 2024

View reviewed changes

eirsep merged commit e3c9fc2 into opensearch-project:main Oct 21, 2024
12 of 14 checks passed

eirsep added backport 2.x backport 2.15 backport 2.17 labels Oct 21, 2024

opensearch-trigger-bot bot mentioned this pull request Oct 21, 2024

[Backport 2.15] optimize bucket level monitor to resolve alias to query only those time-series indices that contain docs within timeframe of range query filter in search input #1711

Merged

opensearch-trigger-bot bot mentioned this pull request Oct 21, 2024

[Backport 2.x] optimize bucket level monitor to resolve alias to query only those time-series indices that contain docs within timeframe of range query filter in search input #1712

Merged

opensearch-trigger-bot bot mentioned this pull request Oct 21, 2024

[Backport 2.17] optimize bucket level monitor to resolve alias to query only those time-series indices that contain docs within timeframe of range query filter in search input #1713

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

optimize bucket level monitor to resolve alias to query only those time-series indices that contain docs within timeframe of range query filter in search input #1701

optimize bucket level monitor to resolve alias to query only those time-series indices that contain docs within timeframe of range query filter in search input #1701

eirsep commented Oct 18, 2024 •

edited

Loading

AWSHurneyt left a comment

sbcd90 Oct 19, 2024

eirsep Oct 21, 2024

optimize bucket level monitor to resolve alias to query only those time-series indices that contain docs within timeframe of range query filter in search input #1701

optimize bucket level monitor to resolve alias to query only those time-series indices that contain docs within timeframe of range query filter in search input #1701

Conversation

eirsep commented Oct 18, 2024 • edited Loading

Description

Related Issues

AWSHurneyt left a comment

Choose a reason for hiding this comment

sbcd90 Oct 19, 2024

Choose a reason for hiding this comment

eirsep Oct 21, 2024

Choose a reason for hiding this comment

eirsep commented Oct 18, 2024 •

edited

Loading