Batch max size question

[Amertz08]

We are attempting to reduce the number of messages being processed by each lambda invocation. We have batching enabled but it doesn't appear that you can configure the message count. It looks like there is a hard coded value for the bytes size of the batch here

kafka-connect-lambda/src/main/java/com/nordstrom/kafka/connect/lambda/LambdaSinkTask.java

Line 26 in eec5306

private final static int maxBatchSizeBytes = (6 * 1024 * 1024) - 1;

.

Is it even possible?

[dylanmei]

6MB is the AWS Lambda payload max limit: https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html

Setting a smaller value through configuration, say aws.lambda.batch.max.bytes, seems like a reasonable addition.

[Amertz08]

Gotcha. What we're wondering is how to only send N messages to a lambda at a time. We are seeing like 7500 messages sent and the lambda is timing out. We'd like to only send a few at a time. This doesn't appear possible with this connector.

[dinegri]

You can control consumer batch size thought consumer.override.max.poll.records in the connector config and enable aws.lambda.batch.enabled, but before that you have to enable override policy in the worker config using connector.client.config.override.policy=All

[ross-paypay]

For ASYNC invocations, there is a different size limitation in AWS.

I am seeing issues when changing from SYNC to ASYNC mode.

I think that the default size should change based on which one of these is set