Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support fine-grained provisioning; handle more edge cases on search flow #268

Merged

Conversation

ohltyler
Copy link
Member

@ohltyler ohltyler commented Aug 6, 2024

Description

This PR improves the search flow by allowing fine-grained provisioning (creating a search pipeline and attaching to an existing index) instead of the limitation from before, forcing full index deletion and re-creation. Now data is persisted and can be searched on in the search flow end-to-end. Additionally, cleans up some edge cases and improves the search flow experience overall, making it fully functional end-to-end now.

More details:

  • adds reprovision parameter on the update workflow API call to support fine-grained provisioning (ref: Adds reprovision API to support updating search pipelines, ingest pipelines index settings flow-framework#804)
  • persists several other fields in the UI config, including 1/ ingest pipeline name, 2/ search pipeline name, and 3/ search index name. The pipeline names are created initially & persisted so we have consistent template node IDs during subsequent updates (before, we were re-generating every time, in which case node IDs did not match, which caused fine-grained provisioning to fail, as it perceived this as node deletions, since the node IDs from the existing template were no longer present). The search index name is a way to persist a user-chosen index (if they skip ingest) and persist it in the UI. This may be expanded upon in the future if we want to pull existing UI configs into an augmented template for users as an option.
  • blocks 'Run query' if there is no search pipeline config (the request and response processor configs are empty). This prevents several different edge cases and confusing user experiences otherwise
  • adds fine-grained validation: only validate the relevant form fields depending on users being in the context of ingest or search.
  • removed bunch of TODO comments :)

Testing:
Tested following scenarios:

  • create ingest resources, add on search resources
  • skip ingest resources, add on search resources
  • skip ingest resources, add on search resources, add ingest resources, update search resources

Demo video, showing fine-grained provisioning happening on the search side. We provision an ingest pipeline & index, then on search, configure and provision a search pipeline separately, but keeping the existing ingest resources. Notice the search call made after running, and how data is still present. Lastly, shows that selecting different indices (if ingest is disabled) is persisted in the UI.

screen-capture.21.webm

Issues Resolved

Makes progress on #23
Resolves #267

Check List

  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Copy link
Member

@owaiskazi19 owaiskazi19 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LG overall with few styling comments

@ohltyler ohltyler merged commit 1e4127c into opensearch-project:main Aug 7, 2024
6 checks passed
@ohltyler ohltyler deleted the fine-grained-provisioning branch August 7, 2024 01:34
opensearch-trigger-bot bot pushed a commit that referenced this pull request Aug 7, 2024
…low (#268)

Signed-off-by: Tyler Ohlsen <[email protected]>
(cherry picked from commit 1e4127c)
ohltyler added a commit that referenced this pull request Aug 7, 2024
…low (#268) (#269)

Signed-off-by: Tyler Ohlsen <[email protected]>
(cherry picked from commit 1e4127c)

Co-authored-by: Tyler Ohlsen <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Integrate with the fine-grained provisioning API
2 participants