[BUG] ConcurrentModificationException seen in logs during the startup of a brand new node in a docker container

[andrross]

What is the bug?
ConcurrentModificationException seen in logs during the startup of a brand new node in a docker container:

[2024-11-08T16:27:31,961][WARN ][o.o.c.s.ClusterApplierService] [c2e68ef6aa29] failed to notify ClusterStateListener
java.util.ConcurrentModificationException: null
	at java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1095) ~[?:?]
	at java.base/java.util.ArrayList$Itr.next(ArrayList.java:1049) ~[?:?]
	at org.opensearch.securityanalytics.indexmanagment.DetectorIndexManagementService.clusterChanged(DetectorIndexManagementService.java:271) ~[?:?]
	at org.opensearch.cluster.service.ClusterApplierService.callClusterStateListener(ClusterApplierService.java:655) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.cluster.service.ClusterApplierService.callClusterStateListeners(ClusterApplierService.java:641) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:599) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:503) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:205) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:946) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:283) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:246) [opensearch-2.18.0.jar:2.18.0]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]

How can one reproduce the bug?
I simply ran:

docker run -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" -e 'OPENSEARCH_INITIAL_ADMIN_PASSWORD=<password>' opensearchproject/opensearch:2.18.0

and saw the exception in the stdout logging during startup. I had not yet even interacted with the server. However, I haven't been able to reproduce it so I suspect it is a race condition.

Looking at the code in question, the cluster state apply method (clusterChanged()) is iterating over an ArrayList making the assumption that the list will not be modified by another thread. This exception suggests that assumption is wrong.

[dblock]

[Catch All Triage - 1, 2, 3]