Workflow for automatic generation of index templates on changes

[AlexRuiz7]

Description

We include many new indices in Wazuh 5 and as responsible for their maintenance and generation, we have generated some tools to automate the process, however, the invocation of these tools is manual and a bit tedious when we change many indices at one. Also, the tools need to be set up before using them, as described in this README.

It would be great to have the index templates re-generated automatically using a GitHub Workflow that triggers on PRs that modify the definitions files.

A possible solution would be to have a Docker image with the requirements to build the templates.

Functional requirements

• The index templates are generated when their definition changes.

Implementation restrictions

• Use a GitHub Workflow that triggers automatically on PRs that modify these files.
• Consider using our Quay.io registry to store the image.

Plan

• Generate Docker image to generate the index templates.
• Generate a workflow that detects changes to files under ECS and generated the corresponding index template, using the Docker image created for that.

[QU3B1M]

Created initial version of the Docker image to generate the ECS index templates, it uses a modified version of the generate.sh tool, and currently its only saving the resulting templates inside the container itself.

Building the image

 docker build -f docker/ecs/images/Dockerfile . -t ecs-generator
[+] Building 1.5s (11/11) FINISHED                                                                                                                                                  docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                                                                                                                0.0s
 => => transferring dockerfile: 633B                                                                                                                                                                0.0s
 => [internal] load metadata for docker.io/library/python:3.10                                                                                                                                      0.7s
 => [internal] load .dockerignore                                                                                                                                                                   0.0s
 => => transferring context: 2B                                                                                                                                                                     0.0s
 => [1/6] FROM docker.io/library/python:3.10@sha256:3ba2e48b887586835af6a0c35fc6fc6086fb4881e963082330ab0a35f3f42c16                                                                                0.0s
 => => resolve docker.io/library/python:3.10@sha256:3ba2e48b887586835af6a0c35fc6fc6086fb4881e963082330ab0a35f3f42c16                                                                                0.0s
 => [internal] load build context                                                                                                                                                                   0.1s
 => => transferring context: 231.38kB                                                                                                                                                               0.1s
 => CACHED [2/6] RUN apt-get update &&     apt-get upgrade -y &&     apt-get install  -y git jq &&     apt-get clean &&     rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* &&     git clone https:/  0.0s
 => CACHED [3/6] ADD docker/ecs/images/generate.sh /ecs/generate.sh                                                                                                                                 0.0s
 => [4/6] ADD ecs/ /source/ecs                                                                                                                                                                      0.1s
 => [5/6] RUN chmod +x /ecs/generate.sh                                                                                                                                                             0.1s
 => [6/6] WORKDIR /ecs                                                                                                                                                                              0.0s
 => exporting to image                                                                                                                                                                              0.5s
 => => exporting layers                                                                                                                                                                             0.5s
 => => exporting manifest sha256:b19ca762ecf63c7e57d7767e9f6cdb0d94ac8ae3e22b40a91e3d0595a1771e0b                                                                                                   0.0s
 => => exporting config sha256:b1b8201f80d570a2d6bb5ee4d7425cd5f5bdbdce995009d6bdb5f48fc6f97b8c                                                                                                     0.0s
 => => exporting attestation manifest sha256:5ccac2093aae0f9cf4ab65f00cb5616d9b587d790ac46a32d7ca4382dbd65762                                                                                       0.0s
 => => exporting manifest list sha256:2bfd5a1b52a921fb238685989a7f83a345dd266f6143bb700837943348548da1                                                                                              0.0s
 => => naming to docker.io/library/ecs-generator:latest                                                                                                                                             0.0s

View build details: docker-desktop://dashboard/build/desktop-linux/desktop-linux/z6646fy4lug9e87big311tenp

Using the tool

docker run --rm -e ECS_MODULE=agent ecs-generator              
Loading schemas from git ref v8.11.0
Running generator. ECS version 8.11.0
Loading user defined schemas: ['/source/ecs/agent/fields/custom/']
Replacing unsupported types in generated mappings
Deleting the "tags" field from the index template
Removing multi-fields from the index template
Mappings saved to /source/ecs/agent/mappings/v8.11.0

---

Next steps:

• Use a volume as ECS definitions source
• Implement Docker Compose and a script to handle the tool execution
• Add a GHA Workflow to generate the index templates and push them on the wazuh-indexer-plugin repo

[QU3B1M]

Implemented Docker Compose to define the required environment variables (ECS_MODULE) and the volume where the ECS definitions are stored

services:
  ecs-mapping-generator:
    image: wazuh-ecs-generator
    container_name: wazuh-ecs-generator
    build:
      context: ./../..
      dockerfile: ${REPO_PATH:-.}/docker/ecs/images/Dockerfile
    volumes:
      - ${REPO_PATH:-.}/ecs:/source/ecs
    environment:
      - ECS_MODULE=${ECS_MODULE:-default_module}

Also a bash util was developed to handle the interaction with this Docker Compose environment, making the process easier. It only requires one command

bash docker/ecs/mapping-generator.sh run <ECS_MODULE>

This script can be launch from any point inside of the repository's directories.

Example execution

bash docker/ecs/mapping-generator.sh run alerts     
[+] Running 1/0
 ✔ Container wazuh-ecs-generator  Created                                                                                                                                                           0.0s 
Attaching to wazuh-ecs-generator
wazuh-ecs-generator  | Loading schemas from git ref v8.11.0
wazuh-ecs-generator  | Running generator. ECS version 8.11.0
wazuh-ecs-generator  | Loading user defined schemas: ['/source/ecs/alerts/fields/custom/']
wazuh-ecs-generator  | Replacing unsupported types in generated mappings
wazuh-ecs-generator  | Deleting the "tags" field from the index template
wazuh-ecs-generator  | Removing multi-fields from the index template
wazuh-ecs-generator  | Mappings saved to /source/ecs/alerts/mappings/v8.11.0
wazuh-ecs-generator exited with code 0
[+] Stopping 1/0
 ✔ Container wazuh-ecs-generator  Stopped   

[QU3B1M]

Generated new GitHub Actions Workflow that executes the utility and creates a PR on wazuh-indexer-plugin with modified template. Currently the GHA is not working due to permissions.

[update-ecs-template-alerts 95b56f3] Update ECS template for module alerts
 1 file changed, 1 insertion(+), 1 deletion(-)
remote: Permission to wazuh/wazuh-indexer-plugins.git denied to github-actions[bot].
fatal: unable to access 'https://github.com/wazuh/wazuh-indexer-plugins/': The requested URL returned error: 403
Error: Process completed with exit code 128.