Skip to content

Stream Reactor 8.1.18

Latest
Latest
Compare
Choose a tag to compare
@github-actions github-actions released this 29 Nov 15:23
8.1.18
c272cd5
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.

🚀 New Features

Sequential Message Sending for Azure Service Bus

  • Introduced a new KCQL property: batch.enabled (default: true).
  • Users can now disable batching to send messages sequentially, addressing specific scenarios with large message sizes (e.g., >1 MB).
  • Why this matters: Batching improves performance but can fail for large messages. Sequential sending ensures reliability in such cases.
  • How to use: Configure batch.enabled=false in the KCQL mapping to enable sequential sending.

Post-Processing for Datalake Cloud Source Connectors

  • Added post-processing capabilities for AWS S3 and GCP Storage source connectors ( Azure Datalake Gen 2 support coming soon).
  • New KCQL properties:
    • post.process.action: Defines the action (DELETE or MOVE) to perform on source files after successful processing.
    • post.process.action.bucket: Specifies the target bucket for the MOVE action (required for MOVE).
    • post.process.action.prefix: Specifies a new prefix for the file’s location when using the MOVE action (required for MOVE).
  • Use cases:
    • Free up storage space by deleting files.
    • Archive or organize processed files by moving them to a new location.
  • Example 1 : Delete Files:
INSERT INTO `my-bucket`
SELECT * FROM `my-topic`
PROPERTIES ('post.process.action'=`DELETE`)
  • Example 2: Move files to an archive bucket:
INSERT INTO `my-bucket:archive/`
SELECT * FROM `my-topic`
PROPERTIES (
    'post.process.action'=`MOVE`,
    'post.process.action.bucket'=`archive-bucket`,
    'post.process.action.prefix'=`archive/`
)

🛠 Dependency Updates

Updated Azure Service Bus Dependencies

  • azure-core updated to version 1.54.1.
  • azure-messaging-servicebus updated to version 7.17.6.

These updates ensure compatibility with the latest Azure SDKs and improve stability and performance.

Upgrade Notes

  • Review the new KCQL properties and configurations for Azure Service Bus and Datalake connectors.
  • Ensure compatibility with the updated Azure Service Bus dependencies if you use custom extensions or integrations.

Thank you to all contributors! 🎉

Assets 19
Loading