Compatibility with segment replication

[dreamer-89]

Summary

With 2.9.0 release, there are lot of enhancements going in for segment replication[1][2] feature (went GA in 2.7.0), we need to ensure different plugins are compatible with current state of this feature. Previously, we ran tests on plugin repos to verify this compatibility but want plugin owners to be aware of these changes so that required updates (if any) can be made. With 2.10.0 release, remote store feature is going GA which internally uses SEGMENT replication strategy only i.e. it enforces all indices to use SEGMENT replication strategy. So, it is important to validate plugins are compatible with segment replication feature.

What changed

1. Refresh policy behavior

1. RefreshPolicy.IMMEDIATE will only refresh primary shards but not replica shards immediately. Instead post refresh, primary will start a round of segment replication to update the replica shard copies leading to eventual consistency.
2. RefreshPolicy.WAIT_UNTIL ensures the indexing operation is searchable in your cluster i.e. RAW (Read after write guarantee). With segment replication, this guarantee is not promised due to delay in replica shared updates from asynchronous background refreshes.

2. Refresh lag on replicas

With segment replication, there is inherent delay in documents to be searchable on replica shard copies. This is due to the fact that replica shard copies over data (segment) files from primary. Thus, compared to document replication, there will be on average increase in amount of time the replica shards are consistent with primaries.

3. System/hidden indices support

With opensearch-project/OpenSearch#8200, system and hidden indices are now supported with SEGMENT replication strategy. We need to ensure there are no bottlenecks which prevents system/hidden indices with segment replication.

Next steps

With segment replication strong reads are not guaranteed. Thus, if the plugin needs strong reads guarantees specially as alternative to change in behavior of refresh policy and lag on replicas (point 1 and 2 above), we need to update search requests to target primary shard only. With opensearch-project/OpenSearch#7375, core now supports primary shards only based search. Please follow documentation for examples and details

Open questions

In case of any questions or issues, please post it in core issue

Reference

[1] Design

[2] Documentation

[dreamer-89]

Request owners to add v2.9.0 label on this issue. @dai-chen

[dreamer-89]

Hi Plugin Owners,
Gentle reminder to look into this issue as code freeze date for 2.9.0 release is near i.e. July 11th.

[dreamer-89]

Gentle reminder for plugin owners as we are past code freeze date for 2.9.0 release

[ps48]

@dreamer-89 For Observability plugin, users create objects in the observability-dashboards plugin and may want to view immediately in the UI. Thus, we are adding the search preferences to _primary. Usually, there number of objects created is low in volume and small in size. We don't see heavy load on primary due to our requests.

[dreamer-89]

@dreamer-89 For Observability plugin, users create objects in the observability-dashboards plugin and may want to view immediately in the UI. Thus, we are adding the search preferences to _primary. Usually, there number of objects created is low in volume and small in size. We don't see heavy load on primary due to our requests.

Thanks @ps48 for the update. _primary preference based search may work for search queries if you need strong reads.

In case, your plugin performs realtime reads via get/mget APIs, I wanted to update that core now supports realtime reads for segment replication enabled indices with opensearch-project/OpenSearch#8536. So, if your plugin performs search via get/mget APIs only, then there should be no change needed. Please check opensearch-project/OpenSearch#8536 for more details.

[joshuali925]

i see in opensearch-project/sql#2013 you mentioned

Existing OpenSearchDataSourceMetadataStorage.java uses match_all & term based search queries which does not provide strong reads and thus as part of opensearch-project/sql#1801 there was no need to perform _primary based routing

In observability, there are get and search APIs, the search API also uses term based search queries

observability/src/main/kotlin/org/opensearch/observability/index/ObservabilityIndex.kt

Line 277 in 4345041

fun getAllObservabilityObjects(

, i believe the same statement applies here. closing

[dreamer-89]

Thanks @joshuali925 for looking into this.

Caution: Please do verify that your plugin does not need strong reads. I am asking as it is still possible to have strong reads with queries you mentioned above by using IMMEDIATE refresh policy during indexing operations. I did a quick search in your repository but did not find references of IMMEDIATE usage. So, I suspect there is no action needed but I will leave it upto you to validate.

[joshuali925]

@dreamer-89 thanks for the note, we do prefer strong reads if possible for better user experience.

I see targeting _primary was recommended in the original issue, WAIT_FOR is used by dashboards, and IMMEDIATE is recommended here. The observability and reporting plugin only stores metadata created by the user, which should imply low overhead. would you provide some guidance regarding IMMEDIATE, WAIT_FOR, and primary shard only search? i understand the distinction but i'm not sure what's recommended for our use case

[dreamer-89]

Thanks @joshuali925 for the update.

we do prefer strong reads if possible for better user experience.

I suggest you to confirm whether your use case needs strong reads or not. This is important before we perform any action to support strong reads. One way to achieve this is via manual testing and see if you see any discrepancy.

The observability and reporting plugin only stores metadata created by the user, which should imply low overhead. would you provide some guidance regarding IMMEDIATE, WAIT_FOR, and primary shard only search? i understand the distinction but i'm not sure what's recommended for our use case

As with SEGMENT replication, IMMEDIATE, WAIT_FOR refresh policies do not guarantee replica shard refreshes (and thus strong reads). The only options to get strong reads is via get/mget APIs or using _primary preference on search queries. The later puts extra burden on primary shards as all queries lands on it. But, I suggest you to first identify if strong reads are really needed or not for your use case.

CC @mch2 @anasalkouz

[joshuali925]

Not sure if i understood properly, the two statements seems a bit contradicting. Could you elaborate it a bit? e.g. if _primary is the only way for strong reads, why was IMMEDIATE brought up?

it is still possible to have strong reads with queries you mentioned above by using IMMEDIATE refresh policy during indexing operations

The only options to get strong reads is via get/mget APIs or using _primary preference on search queries

---

The reason I think we need strong read is because users will want to see the objects they created immediately after creation. For example observability allows users to add sample notebooks, it will be confusing if they clicked "Add sample notebooks" and the new sample ones doesn't show up in the notebooks list, which theoretically could happen.

Our use case is very similar to dashboards saved objects, would be better if we can align to their behavior. @kavilla do you know how is OSD saved objects handling strong reads?

[kavilla]

To avoid choosing, have we considered making it configurable? Like _msearch and _search is configurable right now. Might not good in certain situations as it some stuff require immediate access, I also don't believe the saved objects are being updated all the time will need to verify if adding sample notebooks reinvokes it but I'm pretty sure it doesn't need to re get the visualizations and dashboards every time sometimes the slower option might be acceptable.

Still need to really decide.

[dreamer-89]

Thanks @joshuali925 for the comment and seeking clarification.

Some context on strong read guarantees from core. There are two mechanism by which you can get strong reads as mentioned below.

1. During indexing operation. This is done by refreshing all shard copies inside the indexing operation. This results in any subsequent data search query to return the latest data. This is done today via IMMEDIATE, WAIT_FOR refresh policies. Segment replication does not support this as mentioned in issue description.
2. During data search request. get/mget APIs and using _primary preference parameter ensures strong reads. This is handled automatically on core and thus with SEGMENT replication only mechanism to get strong reads.

@kavilla : I do not have much background on dashboards. Does your last comment suggest that there is no change needed ?

[joshuali925]

Thanks. I think we would want to align with OSD behavior, since observability objects should act in the same way as OSD saved objects

[joshuali925]

i did some manual testing in observability, didn't see inconsistent behavior. will close the issue for now