Initial S3 download

[mpucholblasco]

Description

S3logsbeat can read new elements added to S3 via SQS. However, old ones are not included to ElasticSearch.

Below is an option to implement this and can changes can be discussed.

Configuration

Add a new input type called s3 on yaml config file with these fields:

• buckets: array of string with format s3://bucket_name/bucket_path
• since: get only those objects with creation timestamp after or equal to this field. Ignore if this field is not set. Must be in ISO8601 format.
• to: get only those objects with creation timestamp before or equal to this field. Ignore if this field is not set. Must be in ISO8601 format.

Also must have these fields from sqs type:

• log_format
• key_regex_fields

Would be perfect if ignore_keys (from #3 ) is added to this new input type.

Add a new command called s3import to import these s3 logs. In this mode execution, sqs type present on yaml file are ignored and type s3 is only taken into account when this command is executed and ignored in other cases.

Execution

Execution could be something similar to:

• Command s3import is executed with a config file with types s3 present on it.
• All S3 objects present on buckets and paths present on field buckets that matches timestamps between since and to are imported to ElasticSearch.
• Once all S3 objects are imported, the execution finishes.