[BUG] process.command_line field mapping is not properly picked up by the dashboard

[f-galland]

Description

The process.command_line field is of wildcard type with multi-fields.
This seems to throw the dashboard off, showing unknown as the type:

We need to investigate and fix the issue.

[AlexRuiz7]

ECS process.command_line is defined as a multi-field. We are probably missing the removal of the multi-fields for this field in ecs/generate.sh.

[f-galland]

Removing the fields object from the template doesn't fix the issue. Changing the type to match_only_text doesn't seem to work either:

curl -sku admin:admin -XGET https://192.168.58.254:9200/_template/states-inventory-processes | jq '."states-inventory-processes".mappings.properties.process.properties.command_line'
{
  "type": "match_only_text"
}

[f-galland]

It seems this is not the only field we see this problem with:

$ curl -sku admin:admin -XGET https://192.168.58.254:9200/_template/states-inventory-processes | jq '."states-inventory-processes".mappings.properties.agent.properties.host.properties.os.properties.full'
{
  "ignore_above": 1024,
  "fields": {
    "text": {
      "type": "match_only_text"
    }
  },
  "type": "keyword"
}

$ curl -sku admin:admin -XGET https://192.168.58.254:9200/_template/states-inventory-processes | jq '."states-inventory-processes".mappings.properties.agent.properties.host.properties.os.properties.name'
{
  "ignore_above": 1024,
  "fields": {
    "text": {
      "type": "match_only_text"
    }
  },
  "type": "keyword"
}

[f-galland]

This still happens even after removing the fields field (and recreating indices and index patterns):

(env) fede@tyner:~/src/ecs (main *)
$ curl -sku admin:admin -XGET https://192.168.58.254:9200/_template/states-inventory-processes | jq '."states-inventory-processes".mappings.properties.agent.properties.host.properties.os.properties.full'
{
  "ignore_above": 1024,
  "type": "keyword"
}
(env) fede@tyner:~/src/ecs (main *)
$ curl -sku admin:admin -XGET https://192.168.58.254:9200/_template/states-inventory-processes | jq '."states-inventory-processes".mappings.properties.agent.properties.host.properties.os.properties.name'
{
  "ignore_above": 1024,
  "type": "keyword"
}
(env) fede@tyner:~/src/ecs (main *)
$ curl -sku admin:admin -XGET https://192.168.58.254:9200/_template/states-inventory-processes | jq '."states-inventory-processes".mappings.properties.process.properties.command_line'
{
  "type": "wildcard"
}

[f-galland]

It looks like the agent.host* unknown fields in the pattern were due to the fields object not being excluded in the generator for them. This fixed that part of the problem:

$ sed -n 22,30p generate.sh 
  jq 'del(
    .mappings.properties.host.properties.os.properties.full.fields,
    .mappings.properties.host.properties.os.properties.name.fields,
    .mappings.properties.agent.properties.host.properties.os.properties.full.fields,
    .mappings.properties.agent.properties.host.properties.os.properties.name.fields,
    .mappings.properties.vulnerability.properties.description.fields,
    .mappings.properties.process.properties.command_line.fields,
    .mappings.properties.process.properties.name.fields
  )' "$IN_FILE" > "$OUT_FILE"

With process.command_line, the issue seems to also be caused by the wildcard type. Changing to match_only_text doesn't help either (there is a reported issue in opensearch).

$ curl -sku admin:admin -XGET https://192.168.58.254:9200/_template/index-template-processes | jq '."index-template-processes".mappings.properties.process.properties.command_line'
{
  "type": "match_only_text"
}

Only switching to keyword helped in my tests:

(env) fede@tyner:~/src/ecs (main *)
$ curl -sku admin:admin -XGET https://192.168.58.254:9200/_template/index-template-processes | jq '."index-template-processes".mappings.properties.process.properties.command_line'
{
  "type": "keyword"
}