Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sink: Amazon S3 #654

Merged
merged 6 commits into from
Nov 28, 2024
Merged

Sink: Amazon S3 #654

merged 6 commits into from
Nov 28, 2024

Conversation

gwaramadze
Copy link
Contributor

Add FileSink with pluggable destinations

This PR introduces a new design for the FileSink that separates storage concerns from data serialization. The main changes include:

Key Changes

  • Refactored FileSink to use a pluggable destination system
  • Created a base Destination class to define the storage interface
  • Added two concrete implementations:
    • LocalDestination for writing to local filesystem
    • S3Destination for writing to Amazon S3 buckets
  • Split documentation into separate pages for local and S3 storage

Benefits

  • Cleaner separation of concerns between data serialization and storage
  • Easier to add new storage backends
  • More flexible configuration options for each destination type
  • Better error handling with specific exception types for S3

Implementation Details

  • FileSink now accepts a destination parameter that defaults to LocalDestination
  • Each destination implements common interface methods: write(), set_directory(), etc.
  • S3 implementation includes proper AWS credential handling and bucket validation
  • Local implementation supports both append and create modes for compatible formats

Documentation

  • Created separate documentation pages for local and S3 storage options
  • Updated examples to show both local and S3 configurations

@gwaramadze gwaramadze added the connector Issues updating Sinks or Sources label Nov 27, 2024
@gwaramadze gwaramadze merged commit 04f2438 into main Nov 28, 2024
3 checks passed
@gwaramadze gwaramadze deleted the feature/s3-sink branch November 28, 2024 12:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
connector Issues updating Sinks or Sources
Projects
Development

Successfully merging this pull request may close these issues.

2 participants