-
Notifications
You must be signed in to change notification settings - Fork 8
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* added examples of data versions * added example of code snapshotting * fixed docstrings examples formatting
- 0.28.1
- 0.28.0
- 0.27.3
- 0.27.2
- 0.27.1
- 0.27.0
- 0.26.0
- 0.25.0
- 0.24.9
- 0.24.8
- 0.24.7
- 0.24.6
- 0.24.5
- 0.24.4
- 0.24.3
- 0.24.2
- 0.24.1
- 0.24.0
- 0.23.4
- 0.23.3
- 0.23.2
- 0.23.1
- 0.23.0
- 0.22.0
- 0.21.1
- 0.21.0
- 0.20.3
- 0.20.2
- 0.20.1
- 0.20.0
- 0.19.0
- 0.18.4
- 0.18.3
- 0.18.2
- 0.18.0
- 0.17.0
- 0.16.3
- 0.16.2
- 0.16.1
- 0.16.0
- 0.15.3
- 0.15.2
- 0.15.1
- 0.15
- 0.14.0
- 0.13.9
- 0.13.8
- 0.13.7
- 0.13.6
- 0.13.5
- 0.13.4
- 0.13.3
- 0.13.2
- 0.13.1
- 0.12.4
- 0.12.3
- 0.12.2
- 0.12.1
- 0.12.0
- 0.11.0
- 0.10.1
- 0.9.2
- 0.9.1
- 0.9.0
- 0.8.0
- 0.7.1
- 0.7.0
- 0.6.1
1 parent
fc03a52
commit 5779417
Showing
22 changed files
with
687 additions
and
437 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,72 @@ | ||
{ | ||
"cells": [ | ||
{ | ||
"cell_type": "markdown", | ||
"metadata": {}, | ||
"source": [ | ||
"# Snapshoting code\n", | ||
"\n", | ||
"Neptune keeps track of your `.git` commit to make sure you know on which code you ran your experiment.\n", | ||
"But sometimes you don't want to commit everything and in those dirty, in between commit, situations you may want Neptune\n", | ||
"to snapshot your code and save it with experiment.\n", | ||
"\n", | ||
"In that case you need to pass a list of files you want to snaphsot to the `upload_source_files` argument of `neptune.create_experiment` method.\n", | ||
"We wrote a helper that lets you create this list getting all the names of files of certain extensions in your folder and subfolders." | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": null, | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"import neptune\n", | ||
"from neptunecontrib.api.utils import get_filepaths\n", | ||
"\n", | ||
"neptune.init('USER_NAME/PROJECT_NAME')\n", | ||
"\n", | ||
"with neptune.create_experiment(upload_source_files=get_filepaths(directory='.',extensions=['.py', '.yaml', '.yml'])):\n", | ||
" neptune.set_property('code_snapshot','yes!')" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"metadata": {}, | ||
"source": [ | ||
"Now you can view your code snapshot in Neptune.\n", | ||
"\n", | ||
"Check [this example experiment](https://ui.neptune.ml/neptune-ml/credit-default-prediction/e/CRED-108/source-code?path=src%2Fmodels%2F&file=train_lgbm.py):\n", | ||
" \n", | ||
"![img](https://gist.githubusercontent.com/jakubczakon/f754769a39ea6b8fa9728ede49b9165c/raw/e08d47e0af278225142eaa849c86964adfa7abf0/code_snapshots.png)" | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": null, | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [] | ||
} | ||
], | ||
"metadata": { | ||
"kernelspec": { | ||
"display_name": "Python 3", | ||
"language": "python", | ||
"name": "python3" | ||
}, | ||
"language_info": { | ||
"codemirror_mode": { | ||
"name": "ipython", | ||
"version": 3 | ||
}, | ||
"file_extension": ".py", | ||
"mimetype": "text/x-python", | ||
"name": "python", | ||
"nbconvert_exporter": "python", | ||
"pygments_lexer": "ipython3", | ||
"version": "3.6.8" | ||
} | ||
}, | ||
"nbformat": 4, | ||
"nbformat_minor": 2 | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,160 @@ | ||
{ | ||
"cells": [ | ||
{ | ||
"cell_type": "markdown", | ||
"metadata": {}, | ||
"source": [ | ||
"# Version Data\n", | ||
"\n", | ||
"With `log_data_version` and `log_s3_data_version` helpers you can log data location and data hash to Neptune.\n", | ||
"It will be stored as property and can be viewed both in the `Details` section of an experiment:\n", | ||
"\n", | ||
"![img](https://gist.githubusercontent.com/jakubczakon/f754769a39ea6b8fa9728ede49b9165c/raw/7b98d5a5ef9dc702e9b1cf47dd1019efffc32753/feature_versions.png)\n", | ||
"\n", | ||
"and in the experiment dashboard as a column.\n", | ||
"\n", | ||
"![img1](https://gist.githubusercontent.com/jakubczakon/f754769a39ea6b8fa9728ede49b9165c/raw/7b98d5a5ef9dc702e9b1cf47dd1019efffc32753/feature_versions_dashboard.png)\n", | ||
"\n", | ||
"Check [this example project](https://ui.neptune.ml/neptune-ml/credit-default-prediction/experiments) to see more.\n", | ||
"\n", | ||
"## Prerequisites\n", | ||
"Initialize Neptune" | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": null, | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"import neptune\n", | ||
"neptune.init('USER_NAME/PROJECT_NAME')" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"metadata": {}, | ||
"source": [ | ||
"## File data version" | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": null, | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"from neptunecontrib.versioning.data import log_data_version\n", | ||
"\n", | ||
"FILEPATH = '/path/to/data/my_data.csv'\n", | ||
"with neptune.create_experiment():\n", | ||
" log_data_version(FILEPATH)" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"metadata": {}, | ||
"source": [ | ||
"## Folder data version" | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": null, | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"from neptunecontrib.versioning.data import log_data_version\n", | ||
"\n", | ||
"DIRPATH = '/path/to/data/folder'\n", | ||
"with neptune.create_experiment():\n", | ||
" log_data_version(DIRPATH)" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"metadata": {}, | ||
"source": [ | ||
"## S3 bucket data version \n", | ||
"We can log both a version of a particular `key` which is similar to file versioning." | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": null, | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"BUCKET = 'my-bucket'\n", | ||
"PATH = 'training_dataset.csv'\n", | ||
"with neptune.create_experiment():\n", | ||
" log_s3_data_version(BUCKET, PATH)" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"metadata": {}, | ||
"source": [ | ||
"We can log a combined version of all the `keys` that start with a particular string which is similar to versioning a directory" | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": null, | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"BUCKET = 'my-bucket'\n", | ||
"PATH = 'train_dir/'\n", | ||
"with neptune.create_experiment():\n", | ||
" log_s3_data_version(BUCKET, PATH)" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"metadata": {}, | ||
"source": [ | ||
"## Prefixing\n", | ||
"If you want to track multiple data sources make sure to prefix them before logging.\n", | ||
"For example:" | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": null, | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"from neptunecontrib.versioning.data import log_data_version\n", | ||
"\n", | ||
"FILEPATH_TABLE_1 = '/path/to/data/my_table_1.csv'\n", | ||
"FILEPATH_TABLE_2 = '/path/to/data/my_table_2.csv'\n", | ||
"\n", | ||
"with neptune.create_experiment():\n", | ||
" log_data_version(FILEPATH_TABLE_1, prefix='table_1_')\n", | ||
" log_data_version(FILEPATH_TABLE_2, prefix='table_2_')" | ||
] | ||
} | ||
], | ||
"metadata": { | ||
"kernelspec": { | ||
"display_name": "Python 3", | ||
"language": "python", | ||
"name": "python3" | ||
}, | ||
"language_info": { | ||
"codemirror_mode": { | ||
"name": "ipython", | ||
"version": 3 | ||
}, | ||
"file_extension": ".py", | ||
"mimetype": "text/x-python", | ||
"name": "python", | ||
"nbconvert_exporter": "python", | ||
"pygments_lexer": "ipython3", | ||
"version": "3.6.8" | ||
} | ||
}, | ||
"nbformat": 4, | ||
"nbformat_minor": 2 | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,5 @@ | ||
Telegram bot | ||
=========== | ||
====================== | ||
|
||
.. automodule:: neptunecontrib.bots.telegram_bot | ||
:members: | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
Versioning data | ||
====================== | ||
|
||
.. automodule:: neptunecontrib.versioning.data | ||
:members: | ||
:show-inheritance: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.