Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add an option to populate event.created on all inputs. #15865

Open
jvalente-salemstate opened this issue Jan 28, 2024 · 1 comment
Open

Add an option to populate event.created on all inputs. #15865

jvalente-salemstate opened this issue Jan 28, 2024 · 1 comment

Comments

@jvalente-salemstate
Copy link

jvalente-salemstate commented Jan 28, 2024

The description of the field in the ECS documentation is


event.created contains the date/time when the event was first read by an agent, or by your pipeline.

This field is distinct from @timestamp in that @timestamp typically contain the time extracted from the original event.

From the couple of inputs I've checked there isn't an option to do this. The next best option is adding it within the first filter of the pipeline. However this isn't exactly what is intended for the field (though a few ms difference is trivial) and could, at least conceptually, obfuscate any delays between the input and the next filter adding the field.

What I am suggesting is adding an option to each input plugin that will populate [event][created] with the current time at ingestion. This aligns with the intended use of the field and improves its use for pipeline monitoring.

This probably should be an optional field that defaults to false to avoid issues for pipelines that have already added the field or do not want to use it. Possibly, the field name itself could also differ (or be a setting) depending on the input plugin/pipeline's ecs_compatibility settings.

There may also need to consideration of handling input that may already have it included (beats, agent, another logstash node) and whether to skip setting it or overwrite it. In terms of ecs it's probably better to skip that unless the pipeline's specific use treats this ingestion as the creation. Though, largely, I am guessing any of those cases just would not set the option if it was already set in the pipeline.

@jvalente-salemstate
Copy link
Author

jvalente-salemstate commented Jan 28, 2024

After opening this I see #15701 was also opened--it didn't come up in a search as it's about the @timestamp field but provides another example of where setting this field could be helpful, as well as an example of handling overwrites should logstash's ingestion be the desired timestamp or creation time.

They're slightly different issues but overlap in some use cases.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant