Explore how to use the ISM features from another plugin

[AlexRuiz7]

Description

The initialization plugin needs to ensure that the index templates, indices and ISM policies are created during startup.

In the Spike, we demonstrated how to create an index using custom mappings and settings, but not how to do the same with the Index Templates or the ISM policies. This issue aims to close this gap.

We need to explore how to create index templates (using the Index Management plugin), as a way to overcome the problems stated above. The strategy of creating single indices with mappings and settings only works for that particular index, and won't apply to rotated stream indices.

OpenSearch Playground has 2 composable index templates. Searching the name .opensearch-sap-detectors-queries-index-template, I found out there is an API at opensearch.admin.indices.template.put.

1. playground.opensearch.org
2. DetectorMonitorConfig.java#L22
3. RuleTopicIndices.java#L16
4. PutComposableIndexTemplateAction.java#L61

Originally posted by @AlexRuiz7 in #11 (comment)

Tasks

• Explore how to create index templates programmatically
• Explore how to create composable index templates programmatically
• Explore how to create ISM policies programmatically
• Explore how to create stuff above from JSON files (same format as using the REST API)
• Add functionality to the plugin

[AlexRuiz7]

I managed to create a very simple index template on startup.

public void putTemplate(ActionListener<AcknowledgedResponse> actionListener) {
    String indexTemplate = "wazuh";
    PutIndexTemplateRequest putRequest = new PutIndexTemplateRequest()
            .name(indexTemplate)
            .patterns(List.of("wazuh-*"));
    try {
        client.admin().indices().putTemplate(putRequest, actionListener);

    } catch (Exception e) {
        String errorMessage = new MessageFormat(
                "failed to create index template [{0}]",
                Locale.ROOT
        ).format(indexTemplate);
        log.error(errorMessage, e);
        throw new IllegalStateException(errorMessage, e);
    }
}

[2024-07-11T13:31:50,192][INFO ][o.w.s.WazuhIndexerSetupPlugin] [integTest-0] template created

We need no improve this approach by reading the index templates from the file system.

[AlexRuiz7]

About composable index templates (index templates v2), OpenSearch playground has an index template which is created programmatically by a plugin.

• https://github.com/opensearch-project/security-analytics/blob/main/src/main/java/org/opensearch/securityanalytics/util/RuleTopicIndices.java#L69
• https://github.com/search?q=org%3Aopensearch-project+%22new+ComposableIndexTemplate%22&type=code
• https://github.com/opensearch-project/OpenSearch/blob/451606535752a73be80d5203ae417e7d57fc5cef/server/src/internalClusterTest/java/org/opensearch/indices/template/ComposableTemplateIT.java#L69
• https://github.com/opensearch-project/OpenSearch/blob/451606535752a73be80d5203ae417e7d57fc5cef/server/src/main/java/org/opensearch/cluster/metadata/MetadataIndexTemplateService.java#L694

[AlexRuiz7]

Moved to the Feature Complete stage.