Skip to content

Latest commit

 

History

History
1541 lines (899 loc) · 55.3 KB

data-prepper.change-log-2.8.0.md

File metadata and controls

1541 lines (899 loc) · 55.3 KB
Raw

  • Adding 'single_word_only' option to obfuscate processor (#4476) (#4550)

    opensearch-trigger-bot[bot] - Thu, 16 May 2024 09:37:36 -0700

    EAD -> refs/heads/2.8, tag: refs/tags/2.8.0, refs/remotes/origin/2.8 Adding 'single_word_only' option to obfuscate processor Signed-off-by: Utkarsh Agarwal <[email protected]> (cherry picked from commit 6d48efba0d71ae0e0674b65ef33ce492b8b215ee) Co-authored-by: Utkarsh Agarwal <[email protected]>

  • Generated THIRD-PARTY file for cedaf87 (#4548)

    opensearch-trigger-bot[bot] - Wed, 15 May 2024 15:36:08 -0700

    Signed-off-by: GitHub <[email protected]> Co-authored-by: dlvenable <[email protected]>

  • Updates werkzeug to 3.0.3 in examples to fix CVE-2024-34069. Resolves #4515 (#4546) (#4547)

    opensearch-trigger-bot[bot] - Wed, 15 May 2024 15:26:50 -0700

    Signed-off-by: David Venable <[email protected]> (cherry picked from commit 906e8255f3440faa90bc2cde4cbd6ffa76ba004c) Co-authored-by: David Venable <[email protected]>

  • Change version for 2.8 release to be 2.8.0 (#4540)

    Krishna Kondaka - Wed, 15 May 2024 10:55:21 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Logging update and config validation (#4541) (#4542)

    opensearch-trigger-bot[bot] - Wed, 15 May 2024 10:54:55 -0700

    • Logging improvements for export and stream processing for DocumentDB source Signed-off-by: Dinu John <[email protected]>

    • Add validation for DocumentDB Collection Config Signed-off-by: Dinu John <[email protected]>

    Signed-off-by: Dinu John <[email protected]> (cherry picked from commit 94fa30d74d5857a19110eb04e959d990b755b48e) Co-authored-by: Dinu John <[email protected]>

  • Add aggregate metrics (#4531) (#4537)

    opensearch-trigger-bot[bot] - Tue, 14 May 2024 17:43:09 -0500

    Signed-off-by: Hai Yan <[email protected]> (cherry picked from commit 0495fae1ae0c6eb36a45e587de89932892e5cbbb) Co-authored-by: Hai Yan <[email protected]>

  • Adds an ndjson input codec. This reads JSON objects for ND-JSON and more lenient formats that do not have the newline. (#4533) (#4536)

    opensearch-trigger-bot[bot] - Tue, 14 May 2024 14:17:04 -0700

    Signed-off-by: David Venable <[email protected]> (cherry picked from commit 59e9fd6227117d3b309975256e87b1ab324d6e30) Co-authored-by: David Venable <[email protected]>

  • Address route and subpipeline for pipeline tranformation (#4528) (#4535)

    opensearch-trigger-bot[bot] - Tue, 14 May 2024 12:15:31 -0700

    Address route and subpipeline for pipeline tranformation Signed-off-by: srigovs <[email protected]> (cherry picked from commit e35b4eaf26af79d67834c0f183a846f501be4c16) Co-authored-by: Srikanth Govindarajan <[email protected]>

  • Fix an issue that exception messages are masked (#4416)

    Hai Yan - Mon, 13 May 2024 16:25:42 -0500

    Signed-off-by: Hai Yan <[email protected]>

  • Fix aggregate processor local mode (#4529)

    Krishna Kondaka - Mon, 13 May 2024 12:30:19 -0700

    • Fix aggregate processor local mode Signed-off-by: Krishna Kondaka <[email protected]>

    • Added comments and removed commented out code in test case Signed-off-by: Krishna Kondaka <[email protected]>

    • Fixed code for failing tests Signed-off-by: Krishna Kondaka <[email protected]>

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Key value processor enhancements (#4521)

    Krishna Kondaka - Mon, 13 May 2024 08:53:15 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Checkpoint records at an interval for TPS case when AckSet is enabled (#4526)

    Dinu John - Sat, 11 May 2024 08:27:29 -0500

    • Checkpoint records at an interval for TPS case when AckSet is enabled Signed-off-by: Dinu John <[email protected]>

    • Fix regular checkpoint internal with ack received Signed-off-by: Dinu John <[email protected]>

    Signed-off-by: Dinu John <[email protected]>

  • Fix write json basic test (#4527)

    Krishna Kondaka - Fri, 10 May 2024 14:13:33 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Complete AcknowledgementSet for checkpoint thread & fix BsonTimeStamp field conversion (#4525)

    Dinu John - Fri, 10 May 2024 10:10:45 -0500

    • Complete AcknowledgementSet for checkpoint thread when Ack is enabled Signed-off-by: Dinu John <[email protected]>

    • Use time in seconds for BsonTimeStamp field when converting to record Signed-off-by: Dinu John <[email protected]>

    Signed-off-by: Dinu John <[email protected]>

  • Write stream events that timeout to write to internal buffer in separate thread (#4524)

    Dinu John - Thu, 9 May 2024 19:51:23 -0500

    • Write stream events that timeout to write to internal buffer in separate thread Signed-off-by: Dinu John <[email protected]>

    • Fix Log message level and minor PR comments Signed-off-by: Dinu John <[email protected]>

    • Handle exception in Stream Acknowledgement Manager. Signed-off-by: Dinu John <[email protected]>

    • Extend the lease by default time Signed-off-by: Dinu John <[email protected]>

    Signed-off-by: Dinu John <[email protected]>

  • Modify typeof operator grammar and add support for ArrayList (#4523)

    Krishna Kondaka - Thu, 9 May 2024 13:54:36 -0700

    • Modify typeof operator grammar and add support for ArrayList Signed-off-by: Krishna Kondaka <[email protected]>

    • Fixed checkstyle error Signed-off-by: Krishna Kondaka <[email protected]>

    • Added test case for more code coverage Signed-off-by: Krishna Kondaka <[email protected]>

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Clear system property to disable s3 scan when stream worker exits (#4522)

    Taylor Gray - Thu, 9 May 2024 14:38:58 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Add logging change to documentdb source and s3 source with folder partitions (#4519)

    Taylor Gray - Thu, 9 May 2024 10:22:38 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Bump parquet version to 1.14.0. (#4520)

    Adi Suresh - Thu, 9 May 2024 09:50:48 -0500

    1.14.0 has critical bug fixes for parquet files written in OCSF 1.1 schema. Signed-off-by: Adi Suresh <[email protected]>

  • Write json processor (#4514)

    Krishna Kondaka - Wed, 8 May 2024 20:48:47 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Add bytesProcessed metric for stream (#4518)

    Hai Yan - Wed, 8 May 2024 21:37:23 -0500

    Signed-off-by: Hai Yan <[email protected]>

  • Add bucket owner validation support to s3 sink (#4504)

    Taylor Gray - Wed, 8 May 2024 13:28:35 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Exclude updateDescription field from Mongo/DocDB change stream (#4516)

    Dinu John - Wed, 8 May 2024 10:42:38 -0500

    Signed-off-by: Dinu John <[email protected]>

  • Fix document version for DocDB/Mongo Stream events (#4513)

    Dinu John - Tue, 7 May 2024 13:55:37 -0500

    Signed-off-by: Dinu John <[email protected]>

  • MAINT: inject external origination timestamp (#4507)

    Qi Chen - Tue, 7 May 2024 13:46:05 -0500

    • MAINT: inject external origination timestamp

    • MAINT: set External origination time in event handle Signed-off-by: George Chen <[email protected]>

  • Add metadata for primary key and primary key type and validations for Mongo/DocDb source (#4512)

    Dinu John - Tue, 7 May 2024 12:03:33 -0500

    • Add metadata for primary key and primary key type and validations for Mongo/DocDb source Signed-off-by: Dinu John <[email protected]>

    Signed-off-by: Dinu John <[email protected]>

  • Update docdb metrics (#4508)

    Hai Yan - Tue, 7 May 2024 10:30:40 -0500

    Signed-off-by: Hai Yan <[email protected]>

  • Fix depth field in template (#4509)

    Srikanth Govindarajan - Tue, 7 May 2024 06:53:51 -0500

    Signed-off-by: srigovs <[email protected]>

  • Add BsonType of primary key to metadata (#4506)

    Dinu John - Mon, 6 May 2024 18:19:11 -0500

    Signed-off-by: Dinu John <[email protected]>

  • Add support to export/full load MongoDB/DocumentDB collection with _id field of different data type (#4503)

    Dinu John - Mon, 6 May 2024 17:24:20 -0500

    • Add support to export/full load MongoDB/DocumentDB collection with _id field of different data type Signed-off-by: Dinu John <[email protected]>

    • Add internal type for sorting Signed-off-by: Dinu John <[email protected]>

    • Fix 'static' modifier out of order Signed-off-by: Dinu John <[email protected]>

    Signed-off-by: Dinu John <[email protected]>

  • Add 'typeof' operator to DataPrepper expressions (#4500)

    Krishna Kondaka - Mon, 6 May 2024 15:12:44 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Add new S3 source option and modify docdb template (#4492)

    Srikanth Govindarajan - Mon, 6 May 2024 11:18:21 -0500

    Signed-off-by: srigovs <[email protected]>

  • Move stopping of s3 scan from docdb leader scheduler to stream scheduler (#4498)

    Taylor Gray - Mon, 6 May 2024 11:01:52 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Initial work to support core data types in Data Prepper (#4496)

    David Venable - Fri, 3 May 2024 12:59:49 -0700

    Adds an enum to represent core data types in Data Prepper. This is initially created to help connect the convert_entry_type processor with upcoming work for evaluating type information as part of #4478. Signed-off-by: David Venable <[email protected]>

  • Support default mapping for Mongo/DocumentDB data types (#4499)

    Dinu John - Fri, 3 May 2024 13:55:00 -0500

    • Support default mapping for Mongo/DocumentDB data types Signed-off-by: Dinu John <[email protected]>

    • Fix test indentation Signed-off-by: Dinu John <[email protected]>

    • Use ParameterizedTest for Mongo/DocDB dataType mapping Signed-off-by: Dinu John <[email protected]>

    Signed-off-by: Dinu John <[email protected]>

  • Support Mongo/DocDB stream Delete and other Operation types (#4497)

    Dinu John - Thu, 2 May 2024 17:35:04 -0500

    • Support Mongo/DocDB stream Delete and other Operation types Signed-off-by: Dinu John <[email protected]>

    • Add reset checkpoint method to mongo/docdb partition checkpoint class Signed-off-by: Dinu John <[email protected]>

    Signed-off-by: Dinu John <[email protected]>

  • HTTP data chunking support for kafka buffer (#4475)

    Krishna Kondaka - Thu, 2 May 2024 14:51:46 -0700

    Signed-off-by: Kondaka <[email protected]>

  • Add system property to allow pausing and resuming s3 scan worker thread processing, optionally set this property to pause on documentdb leader (#4495)

    Taylor Gray - Thu, 2 May 2024 14:36:51 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • FIX: null certificate value should be valid in opensearch connection (#4494)

    Qi Chen - Thu, 2 May 2024 10:22:24 -0500

    • FIX: null certificate value should be valid Signed-off-by: George Chen <[email protected]>

  • Fix JacksonEvent to propagate ExternalOriginalTime if its set at the time of construction (#4489)

    Krishna Kondaka - Wed, 1 May 2024 17:06:53 -0700

    • Fix JacksonEvent to propagate ExternalOriginalTime if its set at the time of construction Signed-off-by: Krishna Kondaka <[email protected]>

    • Added test Signed-off-by: Krishna Kondaka <[email protected]>

    • Fixed code coverage failure by adding more tests Signed-off-by: Krishna Kondaka <[email protected]>

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Move MongoDB/DocDB stream checkpoint to separate thread (#4477)

    Dinu John - Tue, 30 Apr 2024 19:16:55 -0500

    • Move MongoDB/DocDB stream checkpoint to separate thread Signed-off-by: Dinu John <[email protected]>

    • Checkpoint DocDB/MongoDB stream processing periodically Signed-off-by: Dinu John <[email protected]>

    Signed-off-by: Dinu John <[email protected]>

  • Fix transient test failure for subpipelines (#4479)

    Srikanth Govindarajan - Tue, 30 Apr 2024 13:37:53 -0700

    Signed-off-by: srigovs <[email protected]>

  • Fix for S3PartitionCreatorScheduler ConcurrentModification Exception (#4473)

    Dinu John - Mon, 29 Apr 2024 12:41:24 -0500

    • Fix for S3PartitionCreatorScheduler ConcurrentModification Exception Signed-off-by: Dinu John <[email protected]>

    • Remove warn log message Signed-off-by: Dinu John <[email protected]>

    Signed-off-by: Dinu John <[email protected]>

  • Publish data-prepper-expression and data-prepper-logstash-configuration to Maven. (#4474)

    David Venable - Mon, 29 Apr 2024 10:37:28 -0700

    Signed-off-by: David Venable <[email protected]>

  • Fix DocumentDB source S3PathPrefix null or empty (#4472)

    Dinu John - Mon, 29 Apr 2024 11:26:40 -0500

    • Fix DocumentDB source S3PathPrefix null or empty Signed-off-by: Dinu John <[email protected]>

    Signed-off-by: Dinu John <[email protected]>

  • Modify documentdb template (#4469)

    Srikanth Govindarajan - Fri, 26 Apr 2024 13:19:50 -0500

    Signed-off-by: srigovs <[email protected]>

  • Changing logging level for config transformation and fixing rule (#4466)

    Srikanth Govindarajan - Fri, 26 Apr 2024 12:50:08 -0500

    Signed-off-by: srigovs <[email protected]>

  • Let data-prepper-core know if docdb has acknowledgments enabled (#4467)

    Taylor Gray - Fri, 26 Apr 2024 10:39:09 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Add folder-based partitioning for s3 scan source (#4455)

    Taylor Gray - Thu, 25 Apr 2024 23:45:55 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Pipeline Configuration Transformation (#4446)

    Srikanth Govindarajan - Wed, 24 Apr 2024 20:13:36 -0700

    templates and rules Signed-off-by: srigovs <[email protected]>

    Signed-off-by: srigovs <[email protected]>

  • MAINT: use authentication-for-basic-credentials (#4435)

    Qi Chen - Wed, 24 Apr 2024 14:39:25 -0500

    Signed-off-by: George Chen <[email protected]>

  • Adding Support for 'BigDecimal' for integer, long, double, string type. (#4395)

    Utkarsh-Aga - Wed, 24 Apr 2024 11:00:27 -0700

    Adding Support for 'BigDecimal' for the following 'Target type' - integer, long, double, string Signed-off-by: Utkarsh Agarwal <[email protected]> Co-authored-by: Utkarsh Agarwal <[email protected]>

  • MAINT: use authentication block to replace username and password in opensearch sink (#4438)

    Qi Chen - Wed, 24 Apr 2024 12:35:26 -0500

    Signed-off-by: George Chen <[email protected]>

  • Revert DocumentDB Source String array host to String host (#4457)

    Dinu John - Wed, 24 Apr 2024 12:02:01 -0500

    • Revert DocumentDB Source String array host to String host Signed-off-by: Dinu John <[email protected]>

  • MAINT: allow latest schema version if not specified in confluent schema (#4453)

    Qi Chen - Wed, 24 Apr 2024 11:13:54 -0500

    Signed-off-by: George Chen <[email protected]>

  • Update DocumentDB source pipeline config parameters (#4451)

    Dinu John - Wed, 24 Apr 2024 11:11:46 -0500

    • Update DocumentDB source pipeline config parameters Signed-off-by: Dinu John <[email protected]>

    Signed-off-by: Dinu John <[email protected]>

  • Added support for multiple workers in S3 Scan Source (#4439)

    Krishna Kondaka - Tue, 23 Apr 2024 13:56:14 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • MAINT: deprecate certificate_content with certificate_key (#4434)

    Qi Chen - Tue, 23 Apr 2024 10:11:55 -0500

    • MAINT: deprecate certificate_content with certificate_key Signed-off-by: George Chen <[email protected]>

  • Support for Event Json input and output codecs (#4436)

    Krishna Kondaka - Mon, 22 Apr 2024 16:22:17 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • ENH: docdb credential auto refreshment (#4399)

    Qi Chen - Mon, 22 Apr 2024 16:22:29 -0500

  • Use positive numbers in S3DlqWriterTest for consistent test success. (#4443)

    David Venable - Mon, 22 Apr 2024 08:27:15 -0700

    Signed-off-by: David Venable <[email protected]>

  • Maint/renaming kafka source plugin setting (#4429)

    Qi Chen - Fri, 19 Apr 2024 09:29:17 -0500

  • MAINT: allow either plain or plaintext under SASL (#4433)

    Qi Chen - Fri, 19 Apr 2024 09:28:20 -0500

    Signed-off-by: George Chen <[email protected]>

  • MAINT: deprecate pipeline_configurations with extension (#4428)

    Qi Chen - Fri, 19 Apr 2024 09:27:16 -0500

    • MAINT: deprecate-pipeline_configurations with extension Signed-off-by: George Chen <[email protected]>

  • Updates Ameria to 1.28.2. To support this change, also updates gRPC to 1.63.0, and Netty to 4.1.108. Fixes unit tests that were expecting a failure. Armeria fixed a bug so the old expectation was no longer valid. (#4440)

    David Venable - Thu, 18 Apr 2024 16:32:28 -0700

    Signed-off-by: David Venable <[email protected]>

  • Make s3 partition size configurable and add unit test for S3 partition creator classes (#4437)

    Dinu John - Thu, 18 Apr 2024 13:19:44 -0500

    • Make s3 partition size configurable and add unit test for S3 partition creator classes Signed-off-by: Dinu John <[email protected]>

    • Rename export partition size to export batch size Signed-off-by: Dinu John <[email protected]>

    Signed-off-by: Dinu John <[email protected]>

  • Change s3 sink client to async client (#4425)

    Taylor Gray - Thu, 18 Apr 2024 11:09:26 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Remove creating S3 prefix path partition upfront (#4432)

    Dinu John - Wed, 17 Apr 2024 14:59:30 -0500

    This will be done when S3 sink writes data with path prefix. Signed-off-by: Dinu John <[email protected]>

  • Gradle 8.7 (#4417)

    David Venable - Wed, 17 Apr 2024 11:02:05 -0700

    Signed-off-by: David Venable <[email protected]>

  • Rename DocDB/MongoDB config parameters (#4426)

    Dinu John - Tue, 16 Apr 2024 18:15:34 -0500

    • Rename DocDB/MongoDB config parameters Signed-off-by: Dinu John <[email protected]>

    • Update hostname to host in DocDB/MongDb config parameter Signed-off-by: Dinu John <[email protected]>

    Signed-off-by: Dinu John <[email protected]>

  • Encode MongoDB/DocumentDB username and password while constructing connection string (#4423)

    Dinu John - Tue, 16 Apr 2024 13:30:35 -0500

    Signed-off-by: Dinu John <[email protected]>

  • Moves the Maven publish Gradle configuration into a Gradle convention plugin. This splits the build logic for publication and allows different projects to determine whether they publish to Maven rather than having to track this with conditionals in the root Gradle project. (#4421)

    David Venable - Tue, 16 Apr 2024 10:57:25 -0700

    Signed-off-by: David Venable <[email protected]>

  • Fix acknowledgements in DynamoDB (#4419)

    Krishna Kondaka - Mon, 15 Apr 2024 16:45:58 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Fix bug in s3 sink dynamic bucket and catch invalid bucket message (#4413)

    Taylor Gray - Thu, 11 Apr 2024 18:48:14 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • AWS DocumentDB Stream/Export improvements with S3 partition (#4409)

    Dinu John - Thu, 11 Apr 2024 17:02:12 -0500

    • AWS DocumentDB Stream/Export improvements with S3 partition Signed-off-by: Dinu John <[email protected]>

    • Update S3 Folder partition with one global state Signed-off-by: Dinu John <[email protected]>

    Signed-off-by: Dinu John <[email protected]>

  • Add support for dynamic bucket and default bucket in S3 sink (#4402)

    Taylor Gray - Thu, 11 Apr 2024 16:51:19 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Do not write empty lists of DlqObject to the DLQ. Just ignore these requests in the S3DlqWriter class. (#4403)

    David Venable - Thu, 11 Apr 2024 12:14:20 -0700

    Signed-off-by: David Venable <[email protected]>

  • Validate the AWS account Id in the S3 source when configuring either the default_bucket_owner or the bucket_owners map. Implemented this by adding a new bean validation annotation @AwsAccountId in the aws-plugin-api. Resolves #4398 (#4400)

    David Venable - Thu, 11 Apr 2024 12:14:13 -0700

    Signed-off-by: David Venable <[email protected]>

  • Fix bug where using upsert or update without routing parameter caused NoSuchElementException (#4397)

    Taylor Gray - Thu, 11 Apr 2024 13:51:18 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Create new codec for each s3 group in s3 sink (#4410)

    Taylor Gray - Thu, 11 Apr 2024 11:18:34 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Create unit test report as html (#4384)

    Dinu John - Wed, 10 Apr 2024 15:48:21 -0500

    Signed-off-by: Dinu John <[email protected]>

  • Modified GRPC exception handler return BAD_REQUEST for certain internal errors (#4387)

    Krishna Kondaka - Tue, 9 Apr 2024 15:16:58 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Simple script to help generate the release notes file as a starting point (#4322)

    David Venable - Tue, 9 Apr 2024 12:43:46 -0700

    This PR adds a small script that I've used to create release notes recently. It can help us create release notes going forward. Signed-off-by: David Venable <[email protected]>

  • MAINT: float up exception (#4391)

    Qi Chen - Tue, 9 Apr 2024 14:24:25 -0500

    Signed-off-by: George Chen <[email protected]>

  • Updates the S3/file codecs to use the EventFactory over JacksonLog/JacksonEvent builder(). (#4355)

    David Venable - Tue, 9 Apr 2024 11:47:14 -0700

    Signed-off-by: David Venable <[email protected]>

  • Add developer guide/integration test section (#4330)

    Jürgen Walter - Fri, 5 Apr 2024 14:08:33 -0700

    • Add developer guide/integration test section Aligned with Kafka example

    https://github.com/opensearch-project/data-prepper/blob/51ee0df595aaad1b921c888394bf3e110ffc74e9/data-prepper-plugins/kafka-plugins/README.md#developer-guide

    Signed-off-by: Jürgen Walter <[email protected]>

    • Use latest OpenSearch for testing Version 2.12.0 onwards requires to use non-default admin password

    https://opensearch.org/blog/replacing-default-admin-credentials/ Signed-off-by: Jürgen Walter <[email protected]>

    Signed-off-by: Jürgen Walter <[email protected]>

  • Corrects the release date for 2.7.0 (#4349)

    David Venable - Fri, 5 Apr 2024 08:42:11 -0700

    Signed-off-by: David Venable <[email protected]>

  • Cache geolocation data within any given batch of Data Prepper events to avoid extra calls to the MaxMind GeoIP code. (#4357)

    David Venable - Fri, 5 Apr 2024 07:24:16 -0700

    Signed-off-by: David Venable <[email protected]>

  • Adds KarstenSchnitter as a maintainer to the Data Prepper project (#4389)

    David Venable - Fri, 5 Apr 2024 07:23:06 -0700

    Adds KarstenSchnitter as a maintainer to the Data Prepper project. Signed-off-by: David Venable <[email protected]>

  • Remove unexpected event handle message (#4388)

    Krishna Kondaka - Thu, 4 Apr 2024 15:09:03 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Add aggregate_threshold with maximum_size to s3 sink (#4385)

    Taylor Gray - Thu, 4 Apr 2024 16:54:05 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Add server connections metric to http and otel sources (#4393)

    Hai Yan - Thu, 4 Apr 2024 10:53:26 -0500

    Signed-off-by: Hai Yan <[email protected]>

  • Added 'long' as a target type for 'convert_entry_type' processor (#4359)

    Utkarsh-Aga - Thu, 4 Apr 2024 08:14:59 -0700

    Added 'long' as a target type for 'convert_entry_type' processor Signed-off-by: Utkarsh Agarwal <[email protected]>

    Signed-off-by: Utkarsh Agarwal <[email protected]> Co-authored-by: Utkarsh Agarwal <[email protected]>

  • Log the User-Agent when Data Prepper shuts down from the POST /shutdown request. (#4390)

    David Venable - Thu, 4 Apr 2024 07:12:33 -0700

    Signed-off-by: David Venable <[email protected]>

  • ENH: automatic credential refresh in kafka source (#4258)

    Qi Chen - Wed, 3 Apr 2024 14:49:20 -0500

    • ADD: basic credentials and dynamic refreshment Signed-off-by: George Chen <[email protected]>

  • Fix flaky PipelineConfigurationFileReaderTest (#4386)

    Taylor Gray - Wed, 3 Apr 2024 12:16:16 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Do not require field_split_characters is not empty for key_value processor (#4358)

    Taylor Gray - Wed, 3 Apr 2024 12:15:52 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Update Stream Ack Manager unit test and code refactor (#4383)

    Dinu John - Tue, 2 Apr 2024 15:17:16 -0500

    • Update Stream Ack Manager unit test Signed-off-by: Dinu John <[email protected]>

    • Move CheckpointStatus to stream sub package and make it package protected Signed-off-by: Dinu John <[email protected]>

    Signed-off-by: Dinu John <[email protected]>

  • Add AcknowledgementSet support to DocumentDB/MongoDB streams (#4379)

    Dinu John - Tue, 2 Apr 2024 13:20:59 -0500

    • Add AcknowledgementSet support to DocumentDB/MongoDB streams Signed-off-by: Dinu John <[email protected]>

    • Update StreamAcknowledgementManagerTest Signed-off-by: Dinu John <[email protected]>

    • Add BackgroundThreadFactory that adds thread name prefix for debugging Signed-off-by: Dinu John <[email protected]>

    • Update unit test Signed-off-by: Dinu John <[email protected]>

    Signed-off-by: Dinu John <[email protected]>

  • Add creation and aggregation of dynamic S3 groups based on events (#4346)

    Taylor Gray - Mon, 1 Apr 2024 17:43:12 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Aggregate Processor: local mode should work when there is no when condition (#4380)

    Krishna Kondaka - Mon, 1 Apr 2024 12:22:09 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Fix count aggregation exemplar data (#4341)

    Krishna Kondaka - Mon, 1 Apr 2024 12:12:39 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Signed-off-by: Krishna Kondaka <[email protected]>

    Co-authored-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Bump org.jetbrains.kotlin:kotlin-stdlib-common (#4363)

    dependabot[bot] - Mon, 1 Apr 2024 09:29:19 -0700

    Bumps org.jetbrains.kotlin:kotlin-stdlib-common from 1.9.22 to 1.9.23.

    updated-dependencies:

    • dependency-name: org.jetbrains.kotlin:kotlin-stdlib-common dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump software.amazon.awssdk:auth in /performance-test (#4362)

    dependabot[bot] - Mon, 1 Apr 2024 09:25:52 -0700

    Bumps software.amazon.awssdk:auth from 2.25.0 to 2.25.21.

    updated-dependencies:

    • dependency-name: software.amazon.awssdk:auth dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Adds integration tests for the geoip processor (#4353)

    David Venable - Thu, 28 Mar 2024 12:06:31 -0700

    Adds integration tests for the geoip processor. Signed-off-by: Asif Sohail Mohammed <[email protected]> Signed-off-by: David Venable <[email protected]> Co-authored-by: Asif Sohail Mohammed <[email protected]>

  • ENH: automatic credential refresher opensearch sink (#4283)

    Qi Chen - Thu, 28 Mar 2024 13:49:44 -0500

    ENH: automatic credential refresher opensearch sink Signed-off-by: George Chen <[email protected]>

  • Integration test to verify that the core HTTP server starts. (#4255)

    David Venable - Thu, 28 Mar 2024 09:19:08 -0700

    Signed-off-by: David Venable <[email protected]>

  • Fixes the S3 source tests when running together (#4280)

    David Venable - Thu, 28 Mar 2024 09:18:33 -0700

    When running all the S3 source tests together, there are empty objects. This adds a mocked counter to avoid a NPE. Signed-off-by: David Venable <[email protected]>

  • Add support for MongoDB/DocumentDB stream processing (#4338)

    Dinu John - Thu, 28 Mar 2024 11:05:36 -0500

    Signed-off-by: Dinu John <[email protected]>

  • Use the org.lz4:lz4-java project instead of the old net.jpountz.lz4 project. (#4347)

    David Venable - Wed, 27 Mar 2024 16:27:28 -0700

    Signed-off-by: David Venable <[email protected]>

  • Adds new metrics to help debug errors connecting to OpenSearch. Resolves #4343, #4344 (#4348)

    David Venable - Wed, 27 Mar 2024 15:51:00 -0700

    Signed-off-by: David Venable <[email protected]>

  • Fix the release build by only applying the Maven publish task to either projects with a name starting with data-prepper or any projects that are children of the data-prepper-plugins project. (#4337)

    David Venable - Tue, 26 Mar 2024 15:28:44 -0700

    Signed-off-by: David Venable <[email protected]>

  • Handle errors from OpenSearch by checking status field as well as error (#4335)

    David Venable - Tue, 26 Mar 2024 14:27:44 -0700

    Handle errors from OpenSearch by checking both the status field and the error body for each bulk response item. Signed-off-by: David Venable <[email protected]>

  • Removes the release flag from the release GitHub Action. It is no longer needed and it conflicts with the release task. (#4332)

    David Venable - Tue, 26 Mar 2024 12:14:51 -0700

    Signed-off-by: David Venable <[email protected]>

  • Revert HTTP data chunking changes for kafka buffer done in PR 4266 (#4329)

    Krishna Kondaka - Tue, 26 Mar 2024 11:14:08 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Adds the release notes for Data Prepper 2.7.0. (#4321)

    David Venable - Tue, 26 Mar 2024 10:44:24 -0700

    Signed-off-by: David Venable <[email protected]>

  • Fix Router performance issue (#4327)

    Krishna Kondaka - Mon, 25 Mar 2024 16:17:02 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Truncate Processor: Add support to truncate all fields in an event (#4317)

    Krishna Kondaka - Mon, 25 Mar 2024 11:09:01 -0700

    Truncate Processor: Add support to truncate all fields in an event Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Split up the Maven groupIds for core, plugins, and test to match package names. (#4324)

    David Venable - Mon, 25 Mar 2024 10:44:36 -0700

    Signed-off-by: David Venable <[email protected]>

  • Updates the next Data Prepper version to 2.8. (#4320)

    David Venable - Fri, 22 Mar 2024 14:05:51 -0700

    Signed-off-by: David Venable <[email protected]>