Update indices with agents information (#544)

* Migrate 525 to 2.17.1 * Adding custom agent.host custom field definitions to remaining indices * Add custom fields to index templates * Fix host custom schema * Fix host custom schema in networks template * Fix host custom schema in ports template * Fix host field in states-vulnerabilities * Include specific agent fields in alerts index subset * Add agent and host fields to states-fim * Add host fields to alerts top level * Add agent fields to states-inventory-hardware * Add agent fields to states-inventory-hardware * Add agent fields to states-inventory-hotfixes * Add agent fields to states-inventory-packages * Add agent fields to states-inventory-ports * Add agent fields to states-inventory-processes * Add agent fields to states-inventory-system * Add all-in-one script --------- Co-authored-by: Alex Ruiz <[email protected]>
wazuh · Nov 13, 2024 · 9e0edb3 · 9e0edb3
1 parent 2c3c1e6
commit 9e0edb3
Show file tree

Hide file tree

Showing 55 changed files with 504 additions and 94 deletions.
diff --git a/ecs/README.md b/ecs/README.md
@@ -45,24 +45,24 @@ files to generate the mappings. These are the inputs for the ECS generator.
       * INDEXER_SRC: Path to the wazuh-indexer repository
       * MODULE: Module to generate mappings for
       * --upload <URL>: Upload generated index template to the OpenSearch cluster. Defaults to https://localhost:9200
-    Example: generate.sh v8.11.0 ~/wazuh-indexer vulnerability-detector --upload https://indexer:9200
+    Example: generate.sh v8.11.0 ~/wazuh-indexer states-vulnerabilities --upload https://indexer:9200
     ```
 
 3. Use the `generate.sh` script to generate the mappings for a module. The script takes 3 arguments,
 plus 2 optional arguments to upload the mappings to the `wazuh-indexer`. Both, composable and legacy mappings
-are generated. For example, to generate the mappings for the `vulnerability-detector` module using the
+are generated. For example, to generate the mappings for the `states-vulnerabilities` module using the
     ECS version `v8.11.0` and assuming that path of this repository is `~/wazuh/wazuh-indexer`:
 
     ```bash
-    ./generate.sh v8.11.0 ~/wazuh/wazuh-indexer vulnerability-detector
+    ./generate.sh v8.11.0 ~/wazuh/wazuh-indexer states-vulnerabilities
     ```
 
     The tool will output the folder where they have been generated.
 
     ```console
     Loading schemas from git ref v8.11.0
     Running generator. ECS version 8.11.0
-    Mappings saved to ~/wazuh/wazuh-indexer/ecs/vulnerability-detector/mappings/v8.11.0
+    Mappings saved to ~/wazuh/wazuh-indexer/ecs/states-vulnerabilities/mappings/v8.11.0
     ```
 
 4. When you are done. Exit the virtual environment.
@@ -93,7 +93,7 @@ The script takes care of these changes automatically, generating the `opensearch
 You can either upload the index template using cURL or the UI (dev tools).
 
 ```bash
-curl -u admin:admin -k -X PUT "https://indexer:9200/_index_template/wazuh-vulnerability-detector" -H "Content-Type: application/json" -d @opensearch-template.json
+curl -u admin:admin -k -X PUT "https://indexer:9200/_index_template/wazuh-states-vulnerabilities" -H "Content-Type: application/json" -d @opensearch-template.json
 ```
 
 Notes:
@@ -117,7 +117,7 @@ are required.
 ### Event generator
 
 For testing purposes, the script `generate_events.py` can be used to generate events for a given module.
-Currently, it is only able to generate events for the `vulnerability-detector` module. To support other
+Currently, it is only able to generate events for the `states-vulnerabilities` module. To support other
 modules, please extend of refactor the script.
 
 The script prompts for the required parameters, so it can be launched without arguments:
@@ -137,3 +137,31 @@ The script uses log file. Check it out for debugging or additional information.
 - [ECS repository](https://github.com/elastic/ecs)
 - [ECS usage](https://github.com/elastic/ecs/blob/main/USAGE.md)
 - [ECS field reference](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html)
+
+### All-in-one script
+
+```bash
+#!/bin/bash
+
+indices=(
+    agent
+    alerts
+    command
+    states-fim
+    states-inventory-hardware
+    states-inventory-hotfixes
+    states-inventory-networks
+    states-inventory-packages
+    states-inventory-ports
+    states-inventory-processes
+    states-inventory-system
+    states-vulnerabilities
+)
+
+ECS="v8.11.0"
+WI_REPO_PATH=~/wazuh/wazuh-indexer
+
+for index in "${indices[@]}"; do
+    bash generate.sh $ECS $WI_REPO_PATH "$index"
+done
+```
diff --git a/ecs/agent/fields/custom/agent.yml b/ecs/agent/fields/custom/agent.yml
@@ -20,8 +20,13 @@
       level: custom
       description: >
         The last time the agent logged in.
-    - name: is_connected
-      type: boolean
+    - name: status
+      type: keyword
       level: custom
       description: >
         Agents' interpreted connection status depending on `agent.last_login`.
+      allowed_values:
+        - name: active
+          description: Active agent status
+        - name: disconnected
+          description: Disconnected agent status
diff --git a/ecs/agent/fields/custom/host.yml b/ecs/agent/fields/custom/host.yml
@@ -0,0 +1,6 @@
+---
+- name: host
+  reusable:
+    top_level: false
+    expected:
+      - agent
diff --git a/ecs/agent/fields/custom/os.yml b/ecs/agent/fields/custom/os.yml
@@ -0,0 +1,6 @@
+---
+- name: os
+  reusable:
+    top_level: false
+    expected:
+      - agent.host
diff --git a/ecs/agent/fields/custom/risk.yml b/ecs/agent/fields/custom/risk.yml
@@ -0,0 +1,6 @@
+---
+- name: risk
+  reusable:
+    top_level: false
+    expected:
+      - agent.host
diff --git a/ecs/agent/fields/subset.yml b/ecs/agent/fields/subset.yml
@@ -13,10 +13,6 @@ fields:
       groups: {}
       key: {}
       last_login: {}
-      is_connected: {}
-  host:
-    fields:
-      ip: {}
-      os:
-        fields:
-          full: {}
+      status: {}
+      host:
+        fields: "*"
diff --git a/ecs/alerts/fields/custom/agent.yml b/ecs/alerts/fields/custom/agent.yml
@@ -9,4 +9,24 @@
       type: keyword
       level: custom
       description: >
-        The groups the agent belongs to.
+        List of groups the agent belong to.
+    - name: key
+      type: keyword
+      level: custom
+      description: >
+        The registration key of the agent.
+    - name: last_login
+      type: date
+      level: custom
+      description: >
+        The agent's last login.
+    - name: status
+      type: keyword
+      level: custom
+      description: >
+        Agents' interpreted connection status depending on `agent.last_login`.
+      allowed_values:
+        - name: active
+          description: Active agent status
+        - name: disconnected
+          description: Disconnected agent status
diff --git a/ecs/alerts/fields/custom/host.yml b/ecs/alerts/fields/custom/host.yml
@@ -0,0 +1,6 @@
+---
+- name: host
+  reusable:
+    top_level: true
+    expected:
+      - { at: agent, as: host }
diff --git a/ecs/alerts/fields/custom/os.yml b/ecs/alerts/fields/custom/os.yml
@@ -0,0 +1,6 @@
+---
+- name: os
+  reusable:
+    top_level: false
+    expected:
+      - agent.host
diff --git a/ecs/alerts/fields/custom/risk.yml b/ecs/alerts/fields/custom/risk.yml
@@ -0,0 +1,6 @@
+---
+- name: risk
+  reusable:
+    top_level: false
+    expected:
+      - agent.host
diff --git a/ecs/alerts/fields/subset.yml b/ecs/alerts/fields/subset.yml
@@ -4,7 +4,14 @@ fields:
     base:
         fields: "*"
     agent:
-        fields: "*"
+        fields:
+            groups: {}
+            id: {}
+            name: {}
+            type: {}
+            version: {}
+            host:
+                fields: "*"
     as:
         fields: "*"
     client:

diff --git a/ecs/states-fim/fields/custom/host.yml b/ecs/states-fim/fields/custom/host.yml
@@ -0,0 +1,6 @@
+---
+- name: host
+  reusable:
+    top_level: true
+    expected:
+      - { at: agent, as: host }
diff --git a/ecs/states-fim/fields/custom/os.yml b/ecs/states-fim/fields/custom/os.yml
@@ -0,0 +1,6 @@
+---
+- name: os
+  reusable:
+    top_level: false
+    expected:
+      - agent.host
diff --git a/ecs/states-fim/fields/custom/risk.yml b/ecs/states-fim/fields/custom/risk.yml
@@ -0,0 +1,6 @@
+---
+- name: risk
+  reusable:
+    top_level: false
+    expected:
+      - agent.host
diff --git a/ecs/states-fim/fields/subset.yml b/ecs/states-fim/fields/subset.yml
@@ -6,8 +6,13 @@ fields:
       tags: []
   agent:
     fields:
-      id: {}
       groups: {}
+      id: {}
+      name: {}
+      type: {}
+      version: {}
+      host:
+        fields: "*"
   file:
     fields:
       attributes: {}
@@ -28,6 +33,8 @@ fields:
       type: {}
       uid: {}
       owner: {}
+  host:
+    fields: "*"
   registry:
     fields:
       key: {}

diff --git a/ecs/states-inventory-hardware/fields/custom/agent.yml b/ecs/states-inventory-hardware/fields/custom/agent.yml
@@ -3,9 +3,30 @@
   title: Wazuh Agents
   short: Wazuh Inc. custom fields.
   type: group
+  group: 2
   fields:
     - name: groups
       type: keyword
       level: custom
       description: >
         List of groups the agent belong to.
+    - name: key
+      type: keyword
+      level: custom
+      description: >
+        The registration key of the agent.
+    - name: last_login
+      type: date
+      level: custom
+      description: >
+        The agent's last login.
+    - name: status
+      type: keyword
+      level: custom
+      description: >
+        Agents' interpreted connection status depending on `agent.last_login`.
+      allowed_values:
+        - name: active
+          description: Active agent status
+        - name: disconnected
+          description: Disconnected agent status
diff --git a/ecs/states-inventory-hardware/fields/custom/host.yml b/ecs/states-inventory-hardware/fields/custom/host.yml
@@ -1,9 +1,9 @@
 ---
 - name: host
-  title: host
-  type: group
-  description: >
-    Host related data.
+  reusable:
+    top_level: true
+    expected:
+      - { at: agent, as: host }
   fields:
     - name: memory
       description: >

diff --git a/ecs/states-inventory-hardware/fields/custom/os.yml b/ecs/states-inventory-hardware/fields/custom/os.yml
@@ -0,0 +1,6 @@
+---
+- name: os
+  reusable:
+    top_level: false
+    expected:
+      - agent.host
diff --git a/ecs/states-inventory-hardware/fields/custom/risk.yml b/ecs/states-inventory-hardware/fields/custom/risk.yml
@@ -0,0 +1,6 @@
+---
+- name: risk
+  reusable:
+    top_level: false
+    expected:
+      - agent.host
diff --git a/ecs/states-inventory-hardware/fields/subset.yml b/ecs/states-inventory-hardware/fields/subset.yml
@@ -7,22 +7,15 @@ fields:
       "@timestamp": {}
   agent:
     fields:
-      id: {}
       groups: {}
+      id: {}
+      name: {}
+      type: {}
+      version: {}
+      host:
+        fields: "*"
   observer:
     fields:
       serial_number: {}
   host:
-    fields:
-      memory:
-        fields:
-          total: {}
-          free: {}
-          used:
-            fields:
-              percentage: {}
-      cpu:
-        fields:
-          name: {}
-          cores: {}
-          speed: {}
+    fields: "*"
diff --git a/ecs/states-inventory-hotfixes/fields/custom/agent.yml b/ecs/states-inventory-hotfixes/fields/custom/agent.yml
@@ -3,9 +3,30 @@
   title: Wazuh Agents
   short: Wazuh Inc. custom fields.
   type: group
+  group: 2
   fields:
     - name: groups
       type: keyword
       level: custom
       description: >
         List of groups the agent belong to.
+    - name: key
+      type: keyword
+      level: custom
+      description: >
+        The registration key of the agent.
+    - name: last_login
+      type: date
+      level: custom
+      description: >
+        The agent's last login.
+    - name: status
+      type: keyword
+      level: custom
+      description: >
+        Agents' interpreted connection status depending on `agent.last_login`.
+      allowed_values:
+        - name: active
+          description: Active agent status
+        - name: disconnected
+          description: Disconnected agent status
diff --git a/ecs/states-inventory-hotfixes/fields/custom/host.yml b/ecs/states-inventory-hotfixes/fields/custom/host.yml
@@ -0,0 +1,6 @@
+---
+- name: host
+  reusable:
+    top_level: true
+    expected:
+      - { at: agent, as: host }
diff --git a/ecs/states-inventory-hotfixes/fields/custom/os.yml b/ecs/states-inventory-hotfixes/fields/custom/os.yml
@@ -0,0 +1,6 @@
+---
+- name: os
+  reusable:
+    top_level: false
+    expected:
+      - agent.host
diff --git a/ecs/states-inventory-hotfixes/fields/custom/risk.yml b/ecs/states-inventory-hotfixes/fields/custom/risk.yml
@@ -0,0 +1,6 @@
+---
+- name: risk
+  reusable:
+    top_level: false
+    expected:
+      - agent.host