Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ESM] Add table for feature-specific coverage #1594

Merged
merged 3 commits into from
Dec 16, 2024
Merged

Conversation

gregfurman
Copy link
Contributor

Changes

Adds a new coverage table for the Lambda event-source mapping section of the docs.

The new table will (approximately) look like:

    SQS   Stream   Kafka ⭐️  
Parameter Description Standard FIFO Kinesis DynamoDB Amazon MSK Self-Managed
BatchSize Batching events by count. 🟡 🟢 🟢 🟢 🟢 🟢
Batch when ≥ 6 MB limit. 🟠 🟠 🟠 🟠 🟢 🟢
MaximumBatchingWindowInSeconds Batch by Time Window. 🟠 🟠 🟠 🟢 🟢
MaximumRetryAttempts Discard after N retries. 🟢 🟢
MaximumRecordAgeInSeconds Discard records older than time t. 🟢 🟢
Enabled Enabling/Disabling. 🟢 🟢 🟢 🟢 🟢 🟢
FilterCriteria Filter criteria object. 🟢 🟢 🟢 🟢 🟢 🟢
FunctionResponseTypes Enabling ReportBatchItemFailures. 🟢 🟢 🟢 🟢
BisectBatchOnFunctionError Bisect a batch on error and retry. 🟠 🟠
ScalingConfig The scaling configuration for the event source. 🟠 🟠
ParallelizationFactor Parallel batch processing by shard. 🟠 🟠
DestinationConfig.OnFailure SQS Failure Destination. 🟢 🟢 🟢 🟢
  SNS Failure Destination. 🟢 🟢 🟢 🟢
  S3 Failure Destination. 🟠 🟠 🟠 🟠
DestinationConfig.OnSuccess Success Destinations.
MetricsConfig CloudWatch metrics. 🟠 🟠 🟠 🟠 🟠 🟠
ProvisionedPollerConfig Control throughput via min-max limits. 🟠 🟠
StartingPosition Position to start reading from. 🟢 🟢 🟢 🟢
StartingPositionTimestamp Timestamp to start reading from. 🟢 🟢 🟢
TumblingWindowInSeconds Duration (seconds) of a processing window. 🟠 🟠
Topics ⭐️ Kafka topics to read from. 🟢 🟢

Legend

  • ⭐️ Only Available in Pro
  • 🟢 Fully Implemented
  • 🟡 Partially Implemented
  • 🟠 Not Implemented
  • ➖ Not Applicable (Not Supported by AWS)

Copy link

github-actions bot commented Dec 12, 2024

🎊 PR Preview has been successfully built and deployed to https://localstack-docs-preview-pr-1594.surge.sh 🎊

@gregfurman gregfurman force-pushed the add/esm/coverage-table branch from 866b40b to 77f24c9 Compare December 12, 2024 16:57
@gregfurman gregfurman self-assigned this Dec 12, 2024
@gregfurman gregfurman added the documentation Improvements or additions to documentation label Dec 12, 2024
@gregfurman gregfurman marked this pull request as ready for review December 12, 2024 16:59
- `TumblingWindowInSeconds`.
- [Self-Managed Apache Kafka](https://docs.aws.amazon.com/lambda/latest/dg/with-kafka.html)

### Coverage
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From @joe4dev:

@HarshCasper How do we best present such a feature model to the user?

We are trying to improve the transparency regarding the supported behavioral features of Lambda Event Source Mapping beyond the simple list of supported sources we currently have in the Lambda docs here. Should we just add the table below to the Lambda docs (similar than the API coverage docs, but for behavior)

Copy link
Member

@joe4dev joe4dev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing work @gregfurman bringing clarity into the ESM djungle 💯

Looking forward to further feedback from @HarshCasper et. al to refine and potentially generalize to other services and areas.

Hint: It's better to link the preview rather than hardcoding a potentially outdated table into the PR description https://localstack-docs-preview-pr-1594.surge.sh/user-guide/aws/lambda/#lambda-event-source-mappings

@@ -165,20 +165,51 @@ The product of 10 and 10 is 100%
[Lambda event source mappings](https://docs.aws.amazon.com/lambda/latest/dg/invocation-eventsourcemapping.html) allows you to connect Lambda functions to other AWS services.
The following event sources are supported in LocalStack:

- [Simple Queue Service (SQS)](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html)
- [DynamoDB](https://docs.aws.amazon.com/lambda/latest/dg/with-ddb.html)
- [Kinesis](https://docs.aws.amazon.com/lambda/latest/dg/with-kinesis.html)
- [Managed Streaming for Apache Kafka (MSK)](https://docs.aws.amazon.com/lambda/latest/dg/with-msk.html)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think we should add Self-Managed Kafka and the Pro image star ⭐ or is the table sufficient? It might be easier to see a quick list at first glance at the cost of some redundancy 🤷‍♂️

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done


### Coverage

The coverage table below shows feature-level coverage for all supported event sources for the latest version of LocalStack.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Helpful clarification 👍

Can we add a link to the API coverage and a sentence clarifying the difference between API coverage and behavior/feature coverage?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a sentance for this!

|--------------------------------|-------------------------------------------------|:--------:|:----:|:---------:|:----------:|:----------:|:------------:|
| **Parameter** | **Description** | **Standard** | **FIFO** | **Kinesis** | **DynamoDB** | **Amazon MSK** | **Self-Managed** |
| BatchSize | Batching events by count. | 🟡 [^1] | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
| ➖ | Batch when ≥ 6 MB limit. | 🟠 | 🟠 | 🟠 | 🟠 | 🟢 | 🟢 |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: remove ➖

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replaced with *Not Configurable*

| MaximumRetryAttempts | Discard after N retries. | ➖ | ➖ | 🟢 | 🟢 | ➖ | ➖ |
| MaximumRecordAgeInSeconds | Discard records older than time `t`. | ➖ | ➖ | 🟢 | 🟢 | ➖ | ➖ |
| Enabled | Enabling/Disabling. | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
| FilterCriteria | Filter criteria object. | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make sense to distinguish between Filter engine, filter validations, and event parity (as discussed in Notion)? The current description "Filter criteria object." sounds quite API like

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For posterity: we added a footnote showing certain Metadata fields are unsupported as well as a link to the filter criteria docs on AWS.

| ParallelizationFactor | Parallel batch processing by shard. | ➖ | ➖ | 🟠 | 🟠 | ➖ | ➖ |
| DestinationConfig.OnFailure | SQS Failure Destination. | ➖ | ➖ | 🟢 | 🟢 | 🟢 | 🟢 |
| | SNS Failure Destination. | ➖ | ➖ | 🟢 | 🟢 | 🟢 | 🟢 |
| | S3 Failure Destination. | ➖ | ➖ | 🟠 | 🟠 | 🟠 | 🟠 |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great solution 👏

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🫡

Copy link
Member

@joe4dev joe4dev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing 🤩

Let's merge it and get some feedback 👂

@gregfurman gregfurman merged commit 82e4896 into main Dec 16, 2024
5 checks passed
@gregfurman gregfurman deleted the add/esm/coverage-table branch December 16, 2024 13:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants