Skip to content

Commit

Permalink
Move to use of AWS SDK v3 and Correct history
Browse files Browse the repository at this point in the history
* Update to use aws-sdk v3 clients and commands

* Updated @aws-sdk/* to the latest version

* Update Readme.md, removed some comments from test

* Updated receiveMessage() and fixed sqsMessageSizeUtils.test.js

* Added simple examples of lambda functions

* Updated Readme.md

Feature/switch to aws sdk v3 (#12)

Feature/switch to aws sdk v3 fix (#13)

* Removed integration-test folder

* Updated jest paths for test matcher

Update remote refs
  • Loading branch information
konrad-dvla authored and rod-dvla committed Oct 27, 2023
1 parent ba7d3eb commit f2f634d
Show file tree
Hide file tree
Showing 10 changed files with 12,490 additions and 1,117 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
coverage
node_modules
.idea
82 changes: 43 additions & 39 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,22 @@
# SQS Extended Client

A library for managing large AWS SQS message payloads using S3. In particular it supports message payloads that exceed the 256KB SQS limit. It is largely a Javascript version of the [Amazon SQS Extended Client Library for Java](https://github.com/awslabs/amazon-sqs-java-extended-client-lib), although not an exact copy.
A library for managing large AWS SQS message payloads using S3. In particular it supports
message payloads that exceed the 256KB SQS limit. It is largely a Javascript version
of the [Amazon SQS Extended Client Library for Java](https://github.com/awslabs/amazon-sqs-java-extended-client-lib),
although not an exact copy.

This module is intended to be used within node18.x lambda runtime environment, as AWS will remove support for node 16.x in the distant future.

If this module will be used in a node16.x lambda runtime,
`@aws-sdk/client-s3` and `@aws-sdk/client-sqs` dependencies will have to be bundled into the consumer app.

## version 1.0.0

**Breaking change!**

- Module uses now only aws-sdk v3
- `SqsExtendedClient()` do not require passing aws sdk sqs and s3 clients as a parameters anymore, as it uses aws-sdk v3 internally.


## Install

Expand All @@ -12,44 +28,26 @@ npm install sqs-extended-client

## Usage

The SQS Extended Client wraps supplied SQS and S3 instances from the AWS SDK V2 and V3. In order to send messages a `bucketName` is required, which is the S3 bucket where the message payloads will be stored:

Using aws-sdk V2:
The SQS Extended Client uses SQS and S3 instances from the AWS SDK v3.
In order to send messages a `bucketName` is required, which is the S3 bucket where the message payloads will be stored:

```Javascript
const AWS = require('aws-sdk');
const SqsExtendedClient = require('sqs-extended-client');

const sqs = new AWS.SQS({ /* your SQS configuration */ });
const s3 = new AWS.S3({ /* your S3 configuration */ });

const sqsExtendedClient = new SqsExtendedClient(sqs, s3,
const sqsExtendedClient = new SqsExtendedClient(
{
bucketName: '/* your bucket name */' // required for send message
// required for send message
bucketName: '/* your bucket name */',
// other configuration options
sqsClientConfig: {},
s3ClientConfig: {}
}
);
```

Using aws-sdk V3:

```Javascript
const { SQS } = require('@aws-sdk/client-sqs');
const { S3 } = require('@aws-sdk/client-s3');
const SqsExtendedClient = require('sqs-extended-client');

const sqs = new SQS({ /* your SQS configuration */ });
const s3 = new S3({ /* your S3 configuration */ });
The SQS Extended Client is used exactly as an SQS instance from the AWS SDK.
It supports all the message level functions:

const sqsExtendedClient = new SqsExtendedClient(sqs, s3,
{
bucketName: '/* your bucket name */' // required for send message
// other configuration options
}
);
```

The SQS Extended Client is used exactly as an SQS instance from the AWS SDK. It supports all the message level functions, and both promise() (aws-sdk V2 only) and callbacks:
```Javascript
changeMessageVisibility()
changeMessageVisibilityBatch()
Expand All @@ -59,25 +57,24 @@ sendMessage()
sendMessageBatch()
receiveMessage()

// e.g. aws-sdk v2:
// e.g.
const response = await sqsExtendedClient.receiveMessage({
QueueUrl: queueUrl,
}).promise();

// e.g. aws-sdk v3:
const response = await sqsExtendedClient.receiveMessage({
QueueUrl: queueUrl,
})
});
```
For bucket level functions (e.g. createBucket) use the SDK S3 instance directly.

Note that for `sendMessageBatch()` only the size of each message is considered, not the overall batch size. For this reason it is recommended to either use `alwaysUseS3: true` or reduce the message size threshold proportionally to the maximum batch size (e.g. `messageSizeThreshold: 26214`) when sending batches.
Note that for `sendMessageBatch()` only the size of each message is considered, not the overall batch size.
For this reason it is recommended to either use `alwaysUseS3: true` or reduce the message size threshold
proportionally to the maximum batch size (e.g. `messageSizeThreshold: 26214`) when sending batches.

## Options

The SQS Extended Client supports the following options:

* `bucketName` - S3 bucket where message payloads are stored (required for sending messages)
* `sqsClientConfig` - optional SQS client options
* `s3ClientConfig` - optional S3 client options
* `alwaysUseS3` - flag indicating that messages payloads should always be stored in S3 regardless of size (default: `false`)
* `messageSizeThreshold` - maximum size in bytes for message payloads before they are stored in S3 (default: `262144`)
* `sendTransform` - see _Transforms_ section
Expand Down Expand Up @@ -110,7 +107,7 @@ const receiveTransform = (sqsMessage, s3Content) => ({
largeItem: s3Content,
});

const sqsExtendedClient = new SqsExtendedClient(sqs, s3,
const sqsExtendedClient = new SqsExtendedClient(
{
bucketName: '/* your bucket name */',
sendTransform,
Expand All @@ -121,14 +118,15 @@ const sqsExtendedClient = new SqsExtendedClient(sqs, s3,

## Middleware

If using [Middy](https://github.com/middyjs/middy) middleware with AWS Lambda then the SQS Extended Client provides a middleware implementation:
If using [Middy](https://github.com/middyjs/middy) middleware with AWS Lambda then the SQS Extended Client provides
a middleware implementation:

```Javascript
const middy = require('@middy/core');
const SqsExtendedClient = require('sqs-extended-client');

const handler = middy(/* Lambda event handler */)
.use(new SqsExtendedClient(sqs, s3).middleware());
.use(new SqsExtendedClient().middleware());
```

## Test
Expand All @@ -146,6 +144,12 @@ The system tests require AWS Localstack to be installed and started. The system
npm run system-test
```

## Examples

[sender.js](example/sender.js) - AWS Lambda function which send message.

[receiver.js](example/receiver.js) - AWS Lambda function which receives messages from sqs.

## License

The MIT License (MIT)
Expand Down
33 changes: 33 additions & 0 deletions example/receiver.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
/**
* Example of lambda function which receives messages from sqs with big content.
*/
const middy = require('@middy/core');
const SqsExtendedClient = require('sqs-extended-client');

const sqsClientConfig = { region: 'eu-west-2'};
const s3ClientConfig = { region: 'eu-west-2'};

const sqsExtendedClient = new SqsExtendedClient({
sqsClientConfig,
s3ClientConfig,
bucketName: '--BUCKET-NAME--'
});

const processRecord = async (data) => {
// Do whatever is required with every record from sqs.
}

const processEvent = async (event) => {
await Promise.all(
event.Records.map(async (record) => {

This comment has been minimized.

Copy link
@superern

superern Nov 10, 2023

change to .forEach.
based from what I experience It does not really wait an async function when using .map

This comment has been minimized.

Copy link
@konrad-dvla

konrad-dvla Nov 10, 2023

Author Contributor

Promise.All() expect iterable object. ForEach() returns undefined by design.
So, sorry i don't agree :)
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/all

return processRecord(record);
})
);
};

const handler = middy(processEvent);

// Ensure that middleware is called
handler.use(sqsExtendedClient.middleware());

module.exports = { handler };
36 changes: 36 additions & 0 deletions example/sender.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
/**
* Example of lambda function which send message with big content to SQS.
*/
const SqsExtendedClient = require('sqs-extended-client');

const DEFAULT_MESSAGE_SIZE_THRESHOLD = 262144;
const QUEUE_URL = '--YOUR-QUEUE-URL--'

const sqsClientConfig = { region: 'eu-west-2'};
const s3ClientConfig = { region: 'eu-west-2'};

const sqsExtendedClient = new SqsExtendedClient({
sqsClientConfig,
s3ClientConfig,
bucketName: '--BUCKET-NAME--',
messageSizeThreshold: DEFAULT_MESSAGE_SIZE_THRESHOLD
});

const handler = async (event, context ) => {

const content = '--BIG-CONTENT--';

const sqsResponse = await sqsExtendedClient.sendMessage({
QueueUrl: QUEUE_URL,
MessageBody: JSON.stringify(content),
});

return {
statusCode: 200,
body: {
MessageId: sqsResponse.MessageId,
}
}
}

module.exports = { handler }
Loading

0 comments on commit f2f634d

Please sign in to comment.