Fixing _list/shards API for closed indices

[gargharsh3134]

Description

By default, the Broadcast requests use IndicesOptions.strictExpandOpenAndForbidClosed(). For _list/shards API, concrete indices from strategy are being passed to IndicesStats Action with the default indicesOptions.
Details around IndexNameExpressionResolver which are causing issues with the list/shards API:

1. Concrete indices if specified in request with indices options as IndicesOptions.strictExpandOpenAndForbidClosed(), it would result in an error (index_closed_exception).
2. Concrete index even if is hidden, will get evaluated and fetched. Basically, if wildcard is to be expanded the IndexOption will ignore hidden indices, but if concretely specified it will get included. Similar is the behaviour with existing _cat/indices API, which by default (curl localhost:9200/_cat/indices), omits the hidden indices but if one of them is explicitly queried, it will return the stats (curl localhost:9200/_cat/indices/hidden-index).

_list/shards API throws index_closed_exception whenever the cluster has closed indices. This change is to resolve and ignore the closed indices from getting explicitly passed to IndicesStatsAction.

Functional Testing

On a local cluster with 2 indices:
open-test-index -> Open index with 1P and 1R.
new-test-idx -> closed index with 1P and 2 R.

gkharsh@bcd07443f159 OpenSearch % curl localhost:9200/_cat/indices
green open  open-test-index rvZzxo5BSnK3TFlmlPQMkw 1 1 0 0 416b 208b
green close new-test-idx    eHm4npFbTQ-dfU8abSLJoQ 1 2 

Existing _cat/shards behaviour:

Unfiltered default query outputs closed shards without stats.

gkharsh@bcd07443f159 OpenSearch % curl "localhost:9200/_cat/shards"                                           
open-test-index 0 p STARTED 0 208b 127.0.0.1 data1
open-test-index 0 r STARTED 0 208b 127.0.0.1 data2
new-test-idx    0 p STARTED        127.0.0.1 data1
new-test-idx    0 r STARTED        127.0.0.1 data2
new-test-idx    0 r STARTED        127.0.0.1 data3

Explicitly querying for a concrete closed index, throws error.

gkharsh@bcd07443f159 OpenSearch % curl "localhost:9200/_cat/shards/new-test-idx"   
{"error":{"root_cause":[{"type":"index_closed_exception","reason":"closed","index":"new-test-idx","index_uuid":"eHm4npFbTQ-dfU8abSLJoQ"}],"type":"index_closed_exception","reason":"closed","index":"new-test-idx","index_uuid":"eHm4npFbTQ-dfU8abSLJoQ"},"status":400}%    

Wildcards only match open indices and response is empty if only closed indices match the expression

gkharsh@bcd07443f159 OpenSearch % curl "localhost:9200/_cat/shards/new-test-idx*"
gkharsh@bcd07443f159 OpenSearch % 

Fixed _list/shards behaviour:

Unfiltered default query will output closed shards without stats.

gkharsh@bcd07443f159 OpenSearch % curl "localhost:9200/_list/shards"             
new-test-idx    0 p STARTED        127.0.0.1 data1
new-test-idx    0 r STARTED        127.0.0.1 data2
new-test-idx    0 r STARTED        127.0.0.1 data3
open-test-index 0 p STARTED 0 208b 127.0.0.1 data1
open-test-index 0 r STARTED 0 208b 127.0.0.1 data2
next_token null

Explicitly querying for a concrete closed index, will output closed shards without stats instead of throwing error.

gkharsh@bcd07443f159 OpenSearch % curl "localhost:9200/_list/shards/new-test-idx" 
new-test-idx 0 p STARTED   127.0.0.1 data1
new-test-idx 0 r STARTED   127.0.0.1 data2
new-test-idx 0 r STARTED   127.0.0.1 data3
next_token null

Wildcards continue to match only open indices and response is empty if only closed indices match the expression

gkharsh@bcd07443f159 OpenSearch % curl "localhost:9200/_list/shards/new-test-idx*"
next_token null

Related Issues

Resolves #16626

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for c62e5f2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 691803c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 4663f2b: SUCCESS

[codecov]

Codecov Report

Attention: Patch coverage is 20.83333% with 19 lines in your changes missing coverage. Please review.

Project coverage is 72.15%. Comparing base (c82cd2e) to head (77c7c9f).
Report is 7 commits behind head on main.

Files with missing lines	Patch %	Lines
...admin/cluster/shards/TransportCatShardsAction.java	0.00%	11 Missing ⚠️
...ction/admin/indices/stats/IndicesStatsRequest.java	50.00%	2 Missing and 2 partials ⚠️
...min/indices/stats/TransportIndicesStatsAction.java	20.00%	3 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #16606      +/-   ##
============================================
- Coverage     72.51%   72.15%   -0.36%     
+ Complexity    65562    65246     -316     
============================================
  Files          5318     5318              
  Lines        303945   303964      +19     
  Branches      43976    43980       +4     
============================================
- Hits         220413   219338    -1075     
- Misses        65798    66666     +868     
- Partials      17734    17960     +226     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

✅ Gradle check result for 06a1118: SUCCESS

[rajiv-kv]

Can you briefly explain why the behaviour of _cat/shards and _list/shards is different in PR description ?

[github-actions]

❌ Gradle check result for 692eeb6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❕ Gradle check result for 503683f: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

❕ Gradle check result for 537fb2d: UNSTABLE

• TEST FAILURES:

      2 org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

✅ Gradle check result for 40b2641: SUCCESS

[github-actions]

✅ Gradle check result for 10d0701: SUCCESS

[github-actions]

✅ Gradle check result for 77c7c9f: SUCCESS