Skip to content

v0.0.3 - Splits and Spills!

Pre-release
Pre-release
Compare
Choose a tag to compare
@dacort dacort released this 04 Oct 17:57
· 3 commits to main since this release
v0.0.3
54c7362
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

Hello folks! 👋

This is the first version of the Athena Federation Python SDK that is ready for real-world use cases.

Please note! This is still a pre-release, so if you run into any issues or questions, please ask first in the Discussions.

Overview

This release introduces support for Splits and automatic Spill-to-S3 functionality.

  • Splits can be used to parallelize queries to your Lambda function. Any properties you return in the splits method will be passed to the record method as a split parameter. You can then use your properties to filter the data source being queried.
  • Spill-to-S3 is used in the event that the size of the data being returned exceeds 6MB, the maximum response for a Lambda function. This is automatically handled in the ReadRecordsRequest portion of the AthenaLambdaHandler.

Usage

An example data source is available in the example/ subdirectory. It hard-codes sample values, but illustrates how to create a new data source as well as connect it to a Lambda handler in Python.

Instructions on creating a Lambda function can be found in the README.

Happy coding! 🧑‍💻

Assets 2
Loading