Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add NetCDF and Zarr datasets #1

Draft
wants to merge 168 commits into
base: main
Choose a base branch
from
Draft

Add NetCDF and Zarr datasets #1

wants to merge 168 commits into from

Conversation

riley-brady
Copy link
Owner

Description

Development notes

Checklist

  • Opened this PR as a 'Draft Pull Request' if it is work-in-progress
  • Updated the documentation to reflect the code changes
  • Added a description of this change in the relevant RELEASE.md file
  • Added tests to cover my changes

merelcht and others added 30 commits October 2, 2023 14:40
@merelcht
style: Introduce ruff for linting in all plugins. (kedro-org#354)
0192170 
Signed-off-by: Merel Theisen <[email protected]>
@deepyaman
feat(datasets): create custom DeprecationWarning (kedro-org#356)
5f884e6 
* feat(datasets): create custom `DeprecationWarning`

Signed-off-by: Deepyaman Datta <[email protected]>

* feat(datasets): use the custom deprecation warning

Signed-off-by: Deepyaman Datta <[email protected]>

* chore(datasets): show Kedro's deprecation warnings

Signed-off-by: Deepyaman Datta <[email protected]>

* fix(datasets): remove unused imports in test files

Signed-off-by: Deepyaman Datta <[email protected]>

---------

Signed-off-by: Deepyaman Datta <[email protected]>
@deepyaman
docs(datasets): add note about DataSet deprecation (kedro-org#357)
ecb00f5
@deepyaman
test(datasets): skip tensorflow tests on Windows (kedro-org#363)
c4dc6e4 
Signed-off-by: Deepyaman Datta <[email protected]>
@ankatiyar
ci: Pin tables version (kedro-org#370)
b9c109f 
* Pin tables version

Signed-off-by: Ankita Katiyar <[email protected]>

* Also fix kedro-airflow

Signed-off-by: Ankita Katiyar <[email protected]>

* Revert trying to fix airflow

Signed-off-by: Ankita Katiyar <[email protected]>

---------

Signed-off-by: Ankita Katiyar <[email protected]>
@merelcht
build(datasets): Release 1.7.1 (kedro-org#378)
080c73a 
Signed-off-by: Merel Theisen <[email protected]>
@ankatiyar
docs: Update CONTRIBUTING.md and add one for kedro-datasets (kedro-…
7cbc265 
…org#379)

Update CONTRIBUTING.md + add one for kedro-datasets

Signed-off-by: Ankita Katiyar <[email protected]>
@merelcht
ci(datasets): Run tensorflow tests separately from other dataset tests (
8b47e1d 
kedro-org#377)

Signed-off-by: Merel Theisen <[email protected]>
@sbrugman
feat: Kedro-Airflow convert all pipelines option (kedro-org#335)
279da24 
* feat: kedro airflow convert --all option

Signed-off-by: Simon Brugman <[email protected]>

* docs: release docs

Signed-off-by: Simon Brugman <[email protected]>

---------

Signed-off-by: Simon Brugman <[email protected]>
@deepyaman
docs(datasets): blacken code in rst literal blocks (kedro-org#362)
78913d8 
Signed-off-by: Deepyaman Datta <[email protected]>
@hfwittmann
docs: cloudpickle is an interesting extension of the pickle functiona…
527706d 
…lity (kedro-org#361)

Signed-off-by: H. Felix Wittmann <[email protected]>
@merelcht
fix(datasets): Fix secret scan entropy error (kedro-org#383)
358fd95 
Fix secret scan entropy error

Signed-off-by: Merel Theisen <[email protected]>
@merelcht
style: Rename mentions of DataSet to Dataset in kedro-airflow a…
d8f1fd5 
…nd `kedro-telemetry` (kedro-org#384)


Signed-off-by: Merel Theisen <[email protected]>
@PtrBld @merelcht
feat(datasets): Migrated PartitionedDataSet and `IncrementalDataSet…
6d1fca4 
…` from main repository to kedro-datasets (kedro-org#253)

Signed-off-by: Peter Bludau <[email protected]>
Co-authored-by: Merel Theisen <[email protected]>
@sbrugman
fix: backwards compatibility for kedro-airflow (kedro-org#381)
cc75b40 
Signed-off-by: Simon Brugman <[email protected]>
@alamastor
fix(datasets): Don't warn for SparkDataset on Databricks when using s3 (
b946eec 
kedro-org#341)

Signed-off-by: Alistair McKelvie <[email protected]>
@riley-brady
initialize template and early additions
cfd040c 
Signed-off-by: Riley Brady <[email protected]>
@riley-brady
add placeholder for remote file system load
fa8f922 
Signed-off-by: Riley Brady <[email protected]>
@riley-brady
switch to versioned dataset
b3ec640 
Signed-off-by: Riley Brady <[email protected]>
@riley-brady
add initial remote -> local get for S3
3d1b1f3 
Signed-off-by: Riley Brady <[email protected]>
@riley-brady
further generalize remote retrieval
37ba9c2 
Signed-off-by: Riley Brady <[email protected]>
@riley-brady
add in credentials
0ccd58a 
Signed-off-by: Riley Brady <[email protected]>
@riley-brady
make temppath optional for remote datasets
de0b044 
Signed-off-by: Riley Brady <[email protected]>
@riley-brady
add initial idea for multifile glob
532fad8 
Signed-off-by: Riley Brady <[email protected]>
@merelcht @riley-brady
style: Introduce ruff for linting in all plugins. (kedro-org#354)
526a0ce 
Signed-off-by: Merel Theisen <[email protected]>
Signed-off-by: Riley Brady <[email protected]>
@riley-brady
add suggested style changes
84df521 
Signed-off-by: Riley Brady <[email protected]>
@riley-brady
add temppath to attributes
7bcef79 
Signed-off-by: Riley Brady <[email protected]>
@riley-brady
more temppath fixes
4dce2a5 
Signed-off-by: Riley Brady <[email protected]>
@riley-brady
more temppath updates
c9b320b 
Signed-off-by: Riley Brady <[email protected]>
@riley-brady
add better tempfile deletion and work on saving files
b67aabc 
Signed-off-by: Riley Brady <[email protected]>
astrojuanlu and others added 27 commits January 31, 2024 20:06
@astrojuanlu
Lint
8588573 
Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>
@astrojuanlu
Fix import
b6ae60b 
Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>
@rashidakanchwala
feat(datasets): Enable search-as-you type on Kedro-datasets docs (ked…
b675485 
…ro-org#532)

* done

Signed-off-by: rashidakanchwala <[email protected]>

* fix lint

Signed-off-by: rashidakanchwala <[email protected]>

---------

Signed-off-by: rashidakanchwala <[email protected]>
@merelcht
Merge branch 'main' into add_netcdf_zarr
83e523c
@SajidAlamQB @astrojuanlu
fix(datasets): Debug and fix kedro-datasets nightly build failures (k…
9ecbd27 
…edro-org#541)

* pin deltalake

* Update kedro-datasets/setup.py

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Update setup.py

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* sort order and compare

* Update setup.py

* lint

* pin deltalake

* add comment to pin

---------

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>
Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>
Co-authored-by: Juan Luis Cano Rodríguez <[email protected]>
@rashidakanchwala @ravi-kumar-pilla @SajidAlamQB
feat(datasets): Dataset Preview Refactor (kedro-org#504)
09409e3 
* test

* done

* change from _preview to preview

* fix lint and tests

* added docstrings

* rtd fix

* rtd fix

* fix rtd

Signed-off-by: rashidakanchwala <[email protected]>

* fix rtd

Signed-off-by: rashidakanchwala <[email protected]>

* fix rtd - pls"

Signed-off-by: rashidakanchwala <[email protected]>

* add nitpick ignore

Signed-off-by: rashidakanchwala <[email protected]>

* test again

Signed-off-by: rashidakanchwala <[email protected]>

* move tracking datasets to constant

Signed-off-by: rashidakanchwala <[email protected]>

* remove comma

Signed-off-by: rashidakanchwala <[email protected]>

* remove Newtype from json_dataset"

Signed-off-by: rashidakanchwala <[email protected]>

* pls work

Signed-off-by: rashidakanchwala <[email protected]>

* confirm rtd works locally

Signed-off-by: rashidakanchwala <[email protected]>

* juanlu's fix

Signed-off-by: rashidakanchwala <[email protected]>

* fix tests

Signed-off-by: rashidakanchwala <[email protected]>

* remove unnecessary stuff from conf.py

Signed-off-by: rashidakanchwala <[email protected]>

* fixes based on review

Signed-off-by: rashidakanchwala <[email protected]>

* changes based on review

Signed-off-by: rashidakanchwala <[email protected]>

* fix tests

Signed-off-by: rashidakanchwala <[email protected]>

* add suffix Preview

Signed-off-by: rashidakanchwala <[email protected]>

* change img return type to bytes

Signed-off-by: rashidakanchwala <[email protected]>

* fix tests

Signed-off-by: rashidakanchwala <[email protected]>

* update release note

* fix lint

---------

Signed-off-by: rashidakanchwala <[email protected]>
Co-authored-by: ravi-kumar-pilla <[email protected]>
Co-authored-by: Sajid Alam <[email protected]>
@felipemonroy @noklam
fix(datasets): Drop pyarrow constraint when using snowpark (kedro-org…
004888c 
…#538)

* Free pyarrow req

Signed-off-by: Felipe Monroy <[email protected]>

* Free pyarrow req

Signed-off-by: Felipe Monroy <[email protected]>

---------

Signed-off-by: Felipe Monroy <[email protected]>
Co-authored-by: Nok Lam Chan <[email protected]>
@DimedS @stichbury @astrojuanlu
docs: Update kedro-telemetry docs on which data is collected (kedro-o…
90c4314 
…rg#546)

* Update data being collected
---------

Signed-off-by: Dmitry Sorokin <[email protected]>
Signed-off-by: Dmitry Sorokin <[email protected]>
Co-authored-by: Jo Stichbury <[email protected]>
Co-authored-by: Juan Luis Cano Rodríguez <[email protected]>
@ankatiyar
ci(docker): Trying to fix e2e tests (kedro-org#548)
e849e7c 
* Pin psutil

Signed-off-by: Ankita Katiyar <[email protected]>

* Add no capture to test

Signed-off-by: Ankita Katiyar <[email protected]>

* Update pip version

Signed-off-by: Ankita Katiyar <[email protected]>

* Update call

Signed-off-by: Ankita Katiyar <[email protected]>

* Update pip

Signed-off-by: Ankita Katiyar <[email protected]>

* pip ruamel

Signed-off-by: Ankita Katiyar <[email protected]>

* change pip v

Signed-off-by: Ankita Katiyar <[email protected]>

* change pip v

Signed-off-by: Ankita Katiyar <[email protected]>

* show stdout

Signed-off-by: Ankita Katiyar <[email protected]>

* use no cache dir

Signed-off-by: Ankita Katiyar <[email protected]>

* revert extra changes

Signed-off-by: Ankita Katiyar <[email protected]>

* pin pip

Signed-off-by: Ankita Katiyar <[email protected]>

* gitpod

Signed-off-by: Ankita Katiyar <[email protected]>

* pip inside dockerfile

Signed-off-by: Ankita Katiyar <[email protected]>

* pip pip inside dockerfile

Signed-off-by: Ankita Katiyar <[email protected]>

---------

Signed-off-by: Ankita Katiyar <[email protected]>
@ankatiyar
chore: bump actions versions (kedro-org#539)
19741b1 
* Unpin pip and bump actions versions

Signed-off-by: Ankita Katiyar <[email protected]>

* remove version

Signed-off-by: Ankita Katiyar <[email protected]>

* Revert unpinning of pip

Signed-off-by: Ankita Katiyar <[email protected]>

---------

Signed-off-by: Ankita Katiyar <[email protected]>
@riley-brady
Merge branch 'main' into add_netcdf_zarr
a2caff8 
Signed-off-by: Riley Brady <[email protected]>
@astrojuanlu
Un-ignore NetCDF doctest
25c7c5c 
Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>
@astrojuanlu @stichbury
docs(telemetry): Direct readers to Kedro documentation for further in…
1cf69d2 
…formation on telemetry (kedro-org#555)

* Direct readers to Kedro documentation for further information on telemetry

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Wording improvements

Co-authored-by: Jo Stichbury <[email protected]>
Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Amend README section

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

---------

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>
Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>
Co-authored-by: Jo Stichbury <[email protected]>
@DimedS @astrojuanlu
fix: kedro-telemetry masking (kedro-org#552)
5d1c950 
* Fix masking

Signed-off-by: Dmitry Sorokin <[email protected]>

* Change masking approach

Signed-off-by: Dmitry Sorokin <[email protected]>

* Fix tests

Signed-off-by: Dmitry Sorokin <[email protected]>

* Fix tests

Signed-off-by: Dmitry Sorokin <[email protected]>

* Fix tests

Signed-off-by: Dmitry Sorokin <[email protected]>

* Fix tests

Signed-off-by: Dmitry Sorokin <[email protected]>

* Fix tests

Signed-off-by: Dmitry Sorokin <[email protected]>

* Fix tests

Signed-off-by: Dmitry Sorokin <[email protected]>

---------

Signed-off-by: Dmitry Sorokin <[email protected]>
Co-authored-by: Juan Luis Cano Rodríguez <[email protected]>
@ankatiyar
Add fixture
f838783 
Signed-off-by: Ankita Katiyar <[email protected]>
@DimedS @astrojuanlu
fix: telemetry data and add example_pipeline (kedro-org#557)
f33acbe 
* Fix telemetry data and add example_pipeline

---------

Signed-off-by: Dmitry Sorokin <[email protected]>
Signed-off-by: Dmitry Sorokin <[email protected]>
Co-authored-by: Juan Luis Cano Rodríguez <[email protected]>
@astrojuanlu
Mark problematic test as xfail
195be05 
Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>
@astrojuanlu
Merge branch 'main' into add_netcdf_zarr
120a757
@astrojuanlu
Skip problematic test instead of making it fail
fc57ba2 
Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>
@noklam
feat(airflow): make sure kedro airflow create is deterministic (ked…
3ff6f59 
…ro-org#525)

* sort dags dictionary

Signed-off-by: Nok Lam Chan <[email protected]>

* update release note

Signed-off-by: Nok Lam Chan <[email protected]>

---------

Signed-off-by: Nok Lam Chan <[email protected]>
@AhdraMeraliQB @ankatiyar
feat(telemetry): Add is_ci_env environment to reported project teleme…
d126e28 
…try (kedro-org#565)

* Add check for ci environment

Signed-off-by: Ahdra Merali <[email protected]>

* Adjust existing tests

Signed-off-by: Ahdra Merali <[email protected]>

* Lint

Signed-off-by: Ahdra Merali <[email protected]>

* Add tests

Signed-off-by: Ahdra Merali <[email protected]>

* Add changes to release notes

Signed-off-by: Ahdra Merali <[email protected]>

* Address review comments

Signed-off-by: Ankita Katiyar <[email protected]>

* Update tests

Signed-off-by: Ankita Katiyar <[email protected]>

* Debug test

Signed-off-by: Ankita Katiyar <[email protected]>

* Fix tests

Signed-off-by: Ankita Katiyar <[email protected]>

* Linting

Signed-off-by: Ankita Katiyar <[email protected]>

* Fix tests

Signed-off-by: Ankita Katiyar <[email protected]>

---------

Signed-off-by: Ahdra Merali <[email protected]>
Signed-off-by: Ankita Katiyar <[email protected]>
Signed-off-by: Ankita Katiyar <[email protected]>
Co-authored-by: Ankita Katiyar <[email protected]>
Co-authored-by: Ankita Katiyar <[email protected]>
@noklam @merelcht
chore(datasets): Update setup.py moto dependency (kedro-org#537)
afe4c98 
* Update setup.py moto dependency

Signed-off-by: Nok Lam Chan <[email protected]>

* Update mock_s3 with mock_aws

Signed-off-by: Merel Theisen <[email protected]>

* Try use same moto for all python versions

Signed-off-by: Merel Theisen <[email protected]>

* Clean up moto deps

Signed-off-by: Merel Theisen <[email protected]>

---------

Signed-off-by: Nok Lam Chan <[email protected]>
Co-authored-by: Merel Theisen <[email protected]>
@mspierenburg
feat(datasets): allow additional parameters for sqlalchemy engine whe…
dc0d2c7 
…n using sql dataset (kedro-org#564)

* allow additional sqlalchemy engine parameters

Signed-off-by: Manuel Spierenburg <[email protected]>
@ankatiyar
build(datasets): Release kedro-datasets 2.1.0 (kedro-org#575)
db92892 
* Bump versions kedro-datasets

Signed-off-by: Ankita Katiyar <[email protected]>

* Update release notes

Signed-off-by: Ankita Katiyar <[email protected]>

---------

Signed-off-by: Ankita Katiyar <[email protected]>
@ankatiyar
Merge branch 'main' into add_netcdf_zarr
16f906f 
Signed-off-by: Ankita Katiyar <[email protected]>
@ankatiyar
Skip problematic tests and fix failing tests
210e4ed 
Signed-off-by: Ankita Katiyar <[email protected]>
@ankatiyar
Remove comment
88a63ea 
Signed-off-by: Ankita Katiyar <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.