-
Notifications
You must be signed in to change notification settings - Fork 53
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Added docs for ktk cube functionality (#334)
* Added docs for ktk cube functionality * Added glossary file, updated all cube related files with minor corrections * Removed downpinning of Sphinx, as the issue is resolved in 3.2.1 release
- Loading branch information
1 parent
721cb7f
commit 6d8c9f4
Showing
9 changed files
with
1,233 additions
and
3 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,151 @@ | ||
|
|
||
| Command Line Features | ||
| --------------------- | ||
|
|
||
| .. raw:: html | ||
|
|
||
| <style> | ||
| .term-fg1 { | ||
| font-weight: bold; | ||
| } | ||
| .term-fg4 { | ||
| text-decoration: underline; | ||
| } | ||
| .term-fg33 { | ||
| color: orange; | ||
| } | ||
| </style> | ||
|
|
||
| Kartothek Cube also features a command line interface (CLI) for some cube operations. To use it, create a ``skv.yml`` file that | ||
| describes `storefact`_ stores: | ||
|
|
||
| .. code-block:: yaml | ||
| dataset: | ||
| type: hfs | ||
| path: path/to/data | ||
| Now use the ``kartothek_cube`` command to gather certain cube information: | ||
|
|
||
| .. code-block:: bash | ||
| kartothek_cube geodata info | ||
| .. raw:: html | ||
|
|
||
| <pre> | ||
| <span class="term-fg33 term-fg1 term-fg4">Infos</span> | ||
| <span class="term-fg1">UUID Prefix:</span> geodata | ||
| <span class="term-fg1">Dimension Columns:</span> | ||
| - city: string | ||
| - day: timestamp[ns] | ||
| <span class="term-fg1">Partition Columns:</span> | ||
| - country: string | ||
| <span class="term-fg1">Index Columns:</span> | ||
| | ||
| <span class="term-fg1">Seed Dataset:</span> seed | ||
| | ||
| <span class="term-fg33 term-fg1 term-fg4">Dataset: latlong</span> | ||
| <span class="term-fg1">Partition Keys:</span> | ||
| - country: string | ||
| <span class="term-fg1">Partitions:</span> 4 | ||
| <span class="term-fg1">Metadata:</span> | ||
| { | ||
| "creation_time": "2019-10-01T12:11:38.263496", | ||
| "klee_dimension_columns": [ | ||
| "city", | ||
| "day" | ||
| ], | ||
| "klee_is_seed": false, | ||
| "klee_partition_columns": [ | ||
| "country" | ||
| ] | ||
| } | ||
| <span class="term-fg1">Dimension Columns:</span> | ||
| - city: string | ||
| <span class="term-fg1">Payload Columns:</span> | ||
| - latitude: double | ||
| - longitude: double | ||
| | ||
| <span class="term-fg33 term-fg1 term-fg4">Dataset: seed</span> | ||
| <span class="term-fg1">Partition Keys:</span> | ||
| - country: string | ||
| <span class="term-fg1">Partitions:</span> 3 | ||
| <span class="term-fg1">Metadata:</span> | ||
| { | ||
| "creation_time": "2019-10-01T12:11:38.206653", | ||
| "klee_dimension_columns": [ | ||
| "city", | ||
| "day" | ||
| ], | ||
| "klee_is_seed": true, | ||
| "klee_partition_columns": [ | ||
| "country" | ||
| ] | ||
| } | ||
| <span class="term-fg1">Dimension Columns:</span> | ||
| - city: string | ||
| - day: timestamp[ns] | ||
| <span class="term-fg1">Payload Columns:</span> | ||
| - avg_temp: int64 | ||
| | ||
| <span class="term-fg33 term-fg1 term-fg4">Dataset: time</span> | ||
| <span class="term-fg1">Partitions:</span> 1 | ||
| <span class="term-fg1">Metadata:</span> | ||
| { | ||
| "creation_time": "2019-10-01T12:11:41.734913", | ||
| "klee_dimension_columns": [ | ||
| "city", | ||
| "day" | ||
| ], | ||
| "klee_is_seed": false, | ||
| "klee_partition_columns": [ | ||
| "country" | ||
| ] | ||
| } | ||
| <span class="term-fg1">Dimension Columns:</span> | ||
| - day: timestamp[ns] | ||
| <span class="term-fg1">Payload Columns:</span> | ||
| - month: int64 | ||
| - weekday: int64 | ||
| - year: int64 | ||
| </pre> | ||
|
|
||
| Some information is not available when reading the schema information and require a cube scan: | ||
|
|
||
| .. code-block:: bash | ||
| kartothek_cube geodata stats | ||
| .. raw:: html | ||
|
|
||
| <pre> | ||
| [########################################] | 100% Completed | 0.1s | ||
| <span class="term-fg33 term-fg1 term-fg4">latlong</span> | ||
| <span class="term-fg1">blobsize:</span> 5,690 | ||
| <span class="term-fg1">files:</span> 4 | ||
| <span class="term-fg1">partitions:</span> 4 | ||
| <span class="term-fg1">rows:</span> 4 | ||
| | ||
| <span class="term-fg33 term-fg1 term-fg4">seed</span> | ||
| <span class="term-fg1">blobsize:</span> 4,589 | ||
| <span class="term-fg1">files:</span> 3 | ||
| <span class="term-fg1">partitions:</span> 3 | ||
| <span class="term-fg1">rows:</span> 8 | ||
| | ||
| <span class="term-fg33 term-fg1 term-fg4">time</span> | ||
| <span class="term-fg1">blobsize:</span> 3,958 | ||
| <span class="term-fg1">files:</span> 1 | ||
| <span class="term-fg1">partitions:</span> 1 | ||
| <span class="term-fg1">rows:</span> 366 | ||
| | ||
| <span class="term-fg33 term-fg1 term-fg4">__total__</span> | ||
| <span class="term-fg1">blobsize:</span> 14,237 | ||
| <span class="term-fg1">files:</span> 8 | ||
| </pre> | ||
|
|
||
|
|
||
| Use ``kartothek_cube --help`` to get a list of all commands, or see :mod:`kartothek_cube.cli`. | ||
|
|
||
| .. _storefact: https://github.com/blue-yonder/storefact | ||
|
|
Oops, something went wrong.