Add initial support to validate the mappings in system tests #2214

mrodm · 2024-11-08T11:57:11Z

Relates #2206

Add validation of the mappings when running system tests.

This process compares the mappings installed by Fleet (preview mappings) with the ones obtained after ingesting new documents into Elasticsearch.

This validation for the time being is behind an environment variable as a technical preview:

# Default value and same behavior as now (validation of fields)
ELASTIC_PACKAGE_FIELD_VALIDATION_TEST_METHOD=fields elastic-package test system -v

# Test just validation of mappings 
ELASTIC_PACKAGE_FIELD_VALIDATION_TEST_METHOD=mappings elastic-package test system -v

# Test both validation of mappings and fields
ELASTIC_PACKAGE_FIELD_VALIDATION_TEST_METHOD=all elastic-package test system -v

Assumptions considered while working on this PR: #2206 (comment)

As part of this PR:

CI pipeline has been updated to remove steps related to running system tests with Elastic Agent.
- It will be kept one step to ensure that the feature is working.
CI pipeline has been updated to add steps to run mapping validation in system tests.

Author's checklist

Update CI steps to remove old stack agents
Update CI to add new steps for validating fields using mappings
Add environment variable to validate that fields are documented with different methods
Add new environment variables to docs.
Remove debug/testing log statements
Test with Elastic stack 8.x
- Some packages are failing like prometheus failing since some fields depend on dynamic templates (not implemented yet)
Test with Elastic stack 7.x (f5 package in integrations repository)
Test with integration and input packages (sql_input test package in elastic-package repo).
Test with Elastic stacks with LogsDB enabled (synthetics).
- Tested in CI with ti_anomali_logsdb and auth0_logsdb

mrodm · 2024-11-12T18:24:15Z

/test

mrodm · 2024-11-13T15:00:22Z

internal/fields/validate.go

+
+	dataStreamName string
+
+	LocalSchema []FieldDefinition


New variable to hold just the Field Definitions from the local directory (package).

It is needed to check if a given mapping is related to a field definition with type array.

Why not using Schema?

mrodm · 2024-11-13T15:00:42Z

internal/fields/validate.go

@@ -295,6 +328,8 @@ func createValidatorForDirectoryAndPackageRoot(fieldsParentDir string, finder pa
 	}

 	v.Schema = append(fields, v.Schema...)
+
+	v.LocalSchema = fields


Added here for completeness, but it is not used in this validator.

Should we remove it from here?

If not used I would say to remove it.

mrodm · 2024-11-13T15:02:03Z

internal/fields/mappings.go

+		if !found {
+			return nil, errors.New("package root not found and dependency management is enabled")
+		}
+		fdm, v.Schema, err = initDependencyManagement(packageRoot, v.specVersion, v.enabledImportAllECSSchema)


v.Schema holds the Field Definitions from ECS
v.LocalSchema holds the Field Definitions found in the package (local directory).

Make use of the External value in the ECS fields loaded into the schema to be able to distinguish which fields come from ECS and which ones come from the fields directory from the package. This allows to perform different tests when validating mappings

mrodm · 2024-11-22T10:09:08Z

test integrations

elastic-vault-github-plugin-prod · 2024-11-22T10:14:28Z

Created or updated PR in integrations repository to test this version. Check elastic/integrations#11828

mrodm · 2024-11-22T12:05:15Z

Looking at the integrations PR testing with the new validation mappings (not all packages tested, there were some filtered for now):

there are some packages that fail because they use dynamic templates and they are not supported yet:
- airflow, cockroachdb, docker, etcd, istio, kibana, linux, system, prometheus, prometheus_input, ti_mandiant_advantage
there are packages that looks like they are missing some field definitions:
- auditd_manager: fields: auditd.data.a0, auditd.data.a1, auditd.data.a2, auditd.data.a3
- sql_input: fields: sql.metrics.active_time, sql.metrics.cntr_value...
  - There is a dynamic template that it is not created: https://github.com/elastic/integrations/blob/79ec43e48fdee7752f57fc66b480a5de34c5d45a/packages/sql_input/fields/input.yml#L23
box_events, claroty_ctd, mimecast, ti_anomali:
- failures like for threat.indicator.modified_at different types in ECS and in Elasticsearch
- failures depend on the stack version
cef fails for url.original
- type in ECS is wildcard and the one set in Elasticsearch is text
cisco_ise failures at pipeline tests, not related
gcp: failing a field with type nested that the preview does not contain properties.
- "ip_port_info":{"type":"nested","properties":{"ip_protocol":{"type":"keyword","ignore_above":1024},"port_range":{"type":"keyword","ignore_above":1024}}}
- "ip_port_info":{"type":"nested"}
ibmmq: process.pid float type vs long type in ECS.
m365_defender: missing definition for path process.parent (all its child exists, but that specific field in ECS does not)

cc @jsoriano

mrodm · 2024-11-22T13:10:16Z

gcp: failing a field with type nested that the preview does not contain properties.
- "ip_port_info":{"type":"nested","properties":{"ip_protocol":{"type":"keyword","ignore_above":1024},"port_range":{"type":"keyword","ignore_above":1024}}}
"ip_port_info":{"type":"nested"}

This failure has been skipped, as for packages with spec < 3.0.1 it was not checked when using the fields validation and gcp is defining 3.0.0:

elastic-package/internal/fields/validate.go

Line 1200 in 8cc126a

if v.specVersion.LessThan(semver3_0_1) {

mrodm · 2024-11-22T14:36:20Z

m365_defender: missing definition for path process.parent (all its child exists, but that specific field in ECS does not)

This field was an empty object (but inside another that neither was present in the preview):

  "process": {
    "properties": {
      "parent": {
        "type": "object"
      }
    }
  }

Maybe as a result of the ingest pipeline?

As process does not exist in the preview, it was missing to skip these kind of objects (empty ones) in this case too.

Added the skip logic for this scenario here: cdf7821

mrodm · 2024-11-22T14:43:38Z

test integrations

elastic-vault-github-plugin-prod · 2024-11-22T14:49:14Z

Created or updated PR in integrations repository to test this version. Check elastic/integrations#11828

mrodm · 2024-11-22T14:57:13Z

box_events, claroty_ctd, mimecast, ti_anomali:

failures like for threat.indicator.modified_at different types in ECS and in Elasticsearch

failures depend on the stack version

There is a difference in the ecs@mappings component template between 8.13.0 and 8.15.3 related to the dynamic template to detect dates causing these errors:

8.13.0 (it is missing *.indicator.last_seen, *.indicate.first_seen and *.indicator.modified_at)

        {
          "ecs_date": {
            "path_match": [
              "*.timestamp",
              "*_timestamp",
              "*.not_after",
              "*.not_before",
              "*.accessed",
              "created",
              "*.created",
              "*.installed",
              "*.creation_date",
              "*.ctime",
              "*.mtime",
              "ingested",
              "*.ingested",
              "*.start",
              "*.end"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "type": "date"
            }
          }
        },

8.15.3

        {
          "ecs_date": {
            "path_match": [
              "*.timestamp",
              "*_timestamp",
              "*.not_after",
              "*.not_before",
              "*.accessed",
              "created",
              "*.created",
              "*.installed",
              "*.creation_date",
              "*.ctime",
              "*.mtime",
              "ingested",
              "*.ingested",
              "*.start",
              "*.end",
              "*.indicator.first_seen",
              "*.indicator.last_seen",
              "*.indicator.modified_at",
              "*threat.enrichments.matched.occurred"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "type": "date"
            }
          }
        }

Not sure if it could be applied some kind of exception @jsoriano

elasticmachine · 2024-11-22T14:59:17Z

💚 Build Succeeded

Buildkite Build
Commit: cdf7821

History

💚 Build #4359 succeeded f4451d2
💚 Build #4355 succeeded e20d563
💚 Build #4352 succeeded 650b3ed
💚 Build #4351 succeeded a058dfb
💚 Build #4338 succeeded 5730b0e
💛 Build #4329 was flaky 21dca78

cc @mrodm

mrodm · 2024-11-22T15:15:14Z

For the cef package there is a similar error as in box_events and others:

8.6.1 dynamic templates (this mapping does not match url.original):

        {
          "_embedded_ecs-url_original_to_multifield": {
            "path_match": "*.url.original",
            "mapping": {
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              },
              "type": "wildcard"
            }
          }
        },

8.15.3

        {
          "ecs_path_match_wildcard_and_match_only_text": {
            "path_match": [
              "*.body.content",
              "*url.full",
              "*url.original"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              },
              "type": "wildcard"
            }
          }
        },

mrodm · 2024-11-22T15:38:34Z

In the second build from integrations, teleport started to fail but I could not reproduce it yet locally:
https://buildkite.com/elastic/integrations/builds/18608#_
but in the third attempt it run successfully. I don't know why there could be differences in the mappings.

In the first build, this package was not failing:
https://buildkite.com/elastic/integrations/builds/18601#_

mrodm · 2024-11-22T16:04:41Z

The error related to ibmmq it's about process.pid.
In this case, this field comes from the Ingest pipeline where it is set the type float (link).
This field is not defined in the package, and when checking with ECS it fails because ECS sets for this field long type (link).

This package fails with both 8.14.0 and 8.15.3 stack versions.

Mapping created for this field in this package during tests:

        "process": {
          "properties": {
            "pid": {
              "type": "float"
            },
            "title": {
              "type": "keyword",
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              }
            }
          }
        }

EDIT: It looks like that currently (with fields validation), this is accepted if the definition is long (ECS) and the value has type float:

elastic-package/internal/fields/validate.go

Lines 1234 to 1247 in 8cc126a

    
           case "float", "long", "double": 
        
           	switch val := val.(type) { 
        
           	case float64: 
        
           	case json.Number: 
        
           	case string: 
        
           		if !slices.Contains(v.stringNumberFields, key) { 
        
           			return invalidTypeError() 
        
           		} 
        
           		if _, err := strconv.ParseFloat(val, 64); err != nil { 
        
           			return invalidTypeError() 
        
           		} 
        
           	default: 
        
           		return invalidTypeError() 
        
           	}

mrodm · 2024-11-22T16:20:38Z

Triggered a new execution in integrations with the rest of packages:
https://buildkite.com/elastic/integrations/builds/18617

mrodm · 2024-11-25T09:38:14Z

In the latest integrations build, there are some failures in network_traffic package. These look related to the same issue #2214 (comment)

There are fields undefined under a group field, that I think thy are not taken into account since until spec 3.0.1 they are not checked.

Definition from ecs: https://github.com/elastic/ecs/blob/ce703ab38f0c128c0dfeeb2daf0893ff3021cdff/generated/ecs/ecs_flat.yml#L2204-L2222

EDIT: dns.answers in this case it's an array of objects: https://github.com/elastic/integrations/blob/ef28635e4ed1fd875b50861e7bd783af7ef68968/packages/network_traffic/data_stream/dns/sample_event.json#L27

Not sure how to detect this via the actual or preview mappings

mrodm added 9 commits November 7, 2024 16:49

Add validation using mappings

f8c82cf

Remove validation per setting/parameter

e211e34

Add environment variable

724329a

[CI] Keep just one step to test system tests with stack agents

a2a4c92

[CI] Add steps to test new validation method

4ee6da6

Add tests

14d7e2a

Add exception for empty objects - TBC

ff3d85d

Update label CI step

b5bed3a

Reuse same fields validator to sanitize docs

2ef29cf

mrodm self-assigned this Nov 8, 2024

mrodm added 14 commits November 8, 2024 15:31

Update field definition to type keyword

36d59fe

Update README test package nginx

6c7af00

Add skip for dynamic objects

4e12656

Add more checks about objects with dynamic true

a326b7f

Add docs for environment variables

8dedfc2

Add more tests

c36fc3b

Compare value constant_keyword just in one case

a48f52d

Restore array type

e94169c

Add validation for type array objects

ec0dd1c

[CI] Remove leftover in label step

7d7ad49

Revamp comments and examples

1949adf

Create private method for creating the mappings validator

ac612fd

Revamp comments

facee62

Include also the definition that contains multi_fields

e03dbab

mrodm added 2 commits November 13, 2024 10:55

Rename methods

1e05f97

Fix typo in README

20775d8

mrodm commented Nov 13, 2024

View reviewed changes

mrodm added 11 commits November 20, 2024 16:41

Add another test case for multi_fields

f7a594b

Remove number of members condition in isObject

109cb07

Enable validation of each parameter of a mapping

4f72eb2

Update transform used in comparison (diff)

b9078ad

Remove leftovers

a058dfb

Commented out one debug statement

1f0c180

Refactor: Move to a new function

06d5463

Updaste logger warn message

1faeab4

Add comments and rename functions

650b3ed

Add External ecs to all ECS fields imported

0f980fa

elastic-vault-github-plugin-prod bot mentioned this pull request Nov 22, 2024

Test elastic-package#2214 - DO NOT MERGE elastic/integrations#11828

Draft

mrodm requested a review from jsoriano November 22, 2024 12:19

Add functions as MappingValidator methods

f4451d2

Skip nested empty mappings

cdf7821

jsoriano mentioned this pull request Nov 26, 2024

Validate mapping ECS compliance in system test #2120

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add initial support to validate the mappings in system tests #2214

Add initial support to validate the mappings in system tests #2214

mrodm commented Nov 8, 2024 •

edited

Loading

mrodm commented Nov 12, 2024

mrodm Nov 13, 2024

jsoriano Nov 18, 2024

mrodm Nov 13, 2024

jsoriano Nov 19, 2024

mrodm Nov 13, 2024

mrodm commented Nov 22, 2024

elastic-vault-github-plugin-prod bot commented Nov 22, 2024

mrodm commented Nov 22, 2024 •

edited

Loading

mrodm commented Nov 22, 2024

mrodm commented Nov 22, 2024 •

edited

Loading

mrodm commented Nov 22, 2024

elastic-vault-github-plugin-prod bot commented Nov 22, 2024

mrodm commented Nov 22, 2024

elasticmachine commented Nov 22, 2024

mrodm commented Nov 22, 2024

mrodm commented Nov 22, 2024

mrodm commented Nov 22, 2024 •

edited

Loading

mrodm commented Nov 22, 2024

mrodm commented Nov 25, 2024 •

edited

Loading

Add initial support to validate the mappings in system tests #2214

Are you sure you want to change the base?

Add initial support to validate the mappings in system tests #2214

Conversation

mrodm commented Nov 8, 2024 • edited Loading

Author's checklist

mrodm commented Nov 12, 2024

mrodm Nov 13, 2024

Choose a reason for hiding this comment

jsoriano Nov 18, 2024

Choose a reason for hiding this comment

mrodm Nov 13, 2024

Choose a reason for hiding this comment

jsoriano Nov 19, 2024

Choose a reason for hiding this comment

mrodm Nov 13, 2024

Choose a reason for hiding this comment

mrodm commented Nov 22, 2024

elastic-vault-github-plugin-prod bot commented Nov 22, 2024

mrodm commented Nov 22, 2024 • edited Loading

mrodm commented Nov 22, 2024

mrodm commented Nov 22, 2024 • edited Loading

mrodm commented Nov 22, 2024

elastic-vault-github-plugin-prod bot commented Nov 22, 2024

mrodm commented Nov 22, 2024

elasticmachine commented Nov 22, 2024

💚 Build Succeeded

History

mrodm commented Nov 22, 2024

mrodm commented Nov 22, 2024

mrodm commented Nov 22, 2024 • edited Loading

mrodm commented Nov 22, 2024

mrodm commented Nov 25, 2024 • edited Loading

mrodm commented Nov 8, 2024 •

edited

Loading

mrodm commented Nov 22, 2024 •

edited

Loading

mrodm commented Nov 22, 2024 •

edited

Loading

mrodm commented Nov 22, 2024 •

edited

Loading

mrodm commented Nov 25, 2024 •

edited

Loading