Skip to content

Commit

Permalink
Adds loaders target-hdfs (automattic) (#1536)
Browse files Browse the repository at this point in the history
Co-authored-by: Edgar Ramírez Mondragón <[email protected]>
  • Loading branch information
pnadolny13 and edgarrmondragon authored Oct 3, 2023
1 parent 85b85d5 commit d2c47a3
Show file tree
Hide file tree
Showing 3 changed files with 61 additions and 0 deletions.
1 change: 1 addition & 0 deletions _data/default_variants.yml
Original file line number Diff line number Diff line change
Expand Up @@ -605,6 +605,7 @@ loaders:
target-elasticsearch: dtmirizzi
target-gcs: datateer
target-gsheet: singer-io
target-hdfs: automattic
target-hubspot: meltanolabs
target-iomete: iomete
target-jsonl: andyh1203
Expand Down
60 changes: 60 additions & 0 deletions _data/meltano/loaders/target-hdfs/automattic.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
capabilities: []
description: Hadoop File System - Parquet
domain_url: https://arrow.apache.org/docs/python/generated/pyarrow.fs.HadoopFileSystem.html
executable: target-hdfs
keywords: []
label: HDFS Parquet
logo_url: /assets/logos/loaders/hdfs.png
maintenance_status: active
name: target-hdfs
namespace: target_hdfs
next_steps: ''
pip_url: git+https://github.com/Automattic/target-hdfs.git
quality: silver
repo: https://github.com/Automattic/target-hdfs
settings:
- description: The compression extension to include in the file names.
kind: string
label: Compression Extension
name: compression_extension
- description: The compression method to use.
kind: string
label: Compression Method
name: compression_method
- description: The prefix to add to each output file.
kind: string
label: File Prefix
name: file_prefix
- description: The size of the output files.
kind: integer
label: File Size Mb
name: file_size_mb
- description: The separator to use in the file name.
kind: string
label: Filename Separator
name: filename_separator
- description: The destination path in HDFS.
kind: string
label: Hdfs Destination Path
name: hdfs_destination_path
- description: The max size of the queue.
kind: integer
label: Max Queue Size
name: max_queue_size
- description: A partition value to use for separating output files.
kind: string
label: Partitions
name: partitions
- description: The number of rows per file.
kind: integer
label: Rows Per File
name: rows_per_file
- description: Whether to put stream data in separate folders or not.
kind: boolean
label: Streams In Separate Folder
name: streams_in_separate_folder
settings_group_validation:
- []
settings_preamble: ''
usage: ''
variant: automattic
Binary file added static/assets/logos/loaders/hdfs.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit d2c47a3

Please sign in to comment.