Bug fix arg connected comp (#382)

* update obsolete flag

Signed-off-by: Walter Teng <[email protected]>

* build: Improve caching (#352)

Signed-off-by: Oliver Koenig <[email protected]>
Signed-off-by: Walter Teng <[email protected]>

* ci: Run on main (#354)

* ci: Run gpuci on main
* fix checkout

Signed-off-by: Oliver Koenig <[email protected]>
Signed-off-by: Walter Teng <[email protected]>

* ci: Run on merge commit (#355)

Signed-off-by: Oliver Koenig <[email protected]>
Signed-off-by: Walter Teng <[email protected]>

* build: Add conda env to `$PATH` (#357)

* build: Add conda env to `$PATH`

Signed-off-by: Oliver Koenig <[email protected]>

* test

Signed-off-by: Oliver Koenig <[email protected]>

* add newline

Signed-off-by: Oliver Koenig <[email protected]>

* run cleanup always

Signed-off-by: Oliver Koenig <[email protected]>

---------

Signed-off-by: Oliver Koenig <[email protected]>
Signed-off-by: Walter Teng <[email protected]>

* Add `build-test-publish-wheel` CI file (#356)

* Create build-test-publish-wheel.yml

Signed-off-by: Sarah Yurick <[email protected]>

* Create package_info.py

Signed-off-by: Sarah Yurick <[email protected]>

* run black

Signed-off-by: Sarah Yurick <[email protected]>

* Update __init__.py

Signed-off-by: Sarah Yurick <[email protected]>

* Update package_info.py

Signed-off-by: Sarah Yurick <[email protected]>

* Update .github/workflows/build-test-publish-wheel.yml

Signed-off-by: Sarah Yurick <[email protected]>

* remove extra version string

Signed-off-by: Sarah Yurick <[email protected]>

* Update __init__.py

Signed-off-by: Sarah Yurick <[email protected]>

* add `__all__`

Signed-off-by: Sarah Yurick <[email protected]>

* Fix version

Signed-off-by: oliver könig <[email protected]>

* Update .github/workflows/build-test-publish-wheel.yml

Signed-off-by: oliver könig <[email protected]>

* Ko3n1g/sarahyurick/ci/build test publish wheel (#358)

* fix

* fix

Signed-off-by: Oliver Koenig <[email protected]>

* fix

Signed-off-by: Oliver Koenig <[email protected]>

* fix

Signed-off-by: Oliver Koenig <[email protected]>

* fix

Signed-off-by: Oliver Koenig <[email protected]>

* fix

Signed-off-by: Oliver Koenig <[email protected]>

* fix

* fix

Signed-off-by: Oliver Koenig <[email protected]>

* fix

* fix

---------

Signed-off-by: Oliver Koenig <[email protected]>

* run black

Signed-off-by: Sarah Yurick <[email protected]>

* run isort

Signed-off-by: Sarah Yurick <[email protected]>

* Update __init__.py

Signed-off-by: Sarah Yurick <[email protected]>

* Update pyproject.toml

Signed-off-by: Sarah Yurick <[email protected]>

---------

Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: oliver könig <[email protected]>
Signed-off-by: Oliver Koenig <[email protected]>
Co-authored-by: oliver könig <[email protected]>
Signed-off-by: Walter Teng <[email protected]>

* Fix broken TestPyPi builder (#362)

* Update build-test-publish-wheel.yml

Signed-off-by: Sarah Yurick <[email protected]>

* Update Dockerfile

Signed-off-by: Sarah Yurick <[email protected]>

* Update build-test-publish-wheel.yml

Signed-off-by: Sarah Yurick <[email protected]>

---------

Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Walter Teng <[email protected]>

* chore: Add `CHANGELOG.md` file (#359)

* chore: Add `CHANGELOG.md` file

* fix

* add end of line

Signed-off-by: Oliver Koenig <[email protected]>

---------

Signed-off-by: Oliver Koenig <[email protected]>
Signed-off-by: Walter Teng <[email protected]>

* Release workflow (#360)

* add file

Signed-off-by: Sarah Yurick <[email protected]>

* trailing whitespace

Signed-off-by: Sarah Yurick <[email protected]>

---------

Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Walter Teng <[email protected]>

* ci: Bump release workflow to allow of `devN` semver (#366)

* ci: Bump release workflow for `devN`

Signed-off-by: Oliver Koenig <[email protected]>

* fix

Signed-off-by: Oliver Koenig <[email protected]>

* fix

Signed-off-by: Oliver Koenig <[email protected]>

* fix

Signed-off-by: Oliver Koenig <[email protected]>

---------

Signed-off-by: Oliver Koenig <[email protected]>
Signed-off-by: Walter Teng <[email protected]>

* ci: Add code-freeze workflow (#367)

Signed-off-by: Oliver Koenig <[email protected]>
Signed-off-by: Walter Teng <[email protected]>

* ci: Add cherry pick workflow (#368)

* ci: Add cherry pick workflow

Signed-off-by: Oliver Koenig <[email protected]>

* fix

Signed-off-by: Oliver Koenig <[email protected]>

---------

Signed-off-by: Oliver Koenig <[email protected]>
Signed-off-by: Walter Teng <[email protected]>

* Fix broken NeMo dependencies (#372)

* add packaging

Signed-off-by: Sarah Yurick <[email protected]>

* move to requires

Signed-off-by: Sarah Yurick <[email protected]>

* move to github ci file

Signed-off-by: Sarah Yurick <[email protected]>

* add pin

Signed-off-by: Sarah Yurick <[email protected]>

* add torch

Signed-off-by: Sarah Yurick <[email protected]>

* add suggestion from mamba readme

Signed-off-by: Sarah Yurick <[email protected]>

* try github install

Signed-off-by: Sarah Yurick <[email protected]>

* add comma

Signed-off-by: Sarah Yurick <[email protected]>

* another attempt

Signed-off-by: Sarah Yurick <[email protected]>

* remove nemo toolkit

Signed-off-by: Sarah Yurick <[email protected]>

* add datasets

Signed-off-by: Sarah Yurick <[email protected]>

* try removing cython

Signed-off-by: Sarah Yurick <[email protected]>

* remove cython

Signed-off-by: Sarah Yurick <[email protected]>

* sentencepiece

Signed-off-by: Sarah Yurick <[email protected]>

* run black

Signed-off-by: Sarah Yurick <[email protected]>

* apply ryan's suggestion

Signed-off-by: Sarah Yurick <[email protected]>

---------

Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Walter Teng <[email protected]>

* ci: Bump release workflow (#373)

Signed-off-by: Oliver Koenig <[email protected]>
Signed-off-by: Walter Teng <[email protected]>

* Skip reading files with incorrect extension (#318)

* filter_files_by_extension function

Signed-off-by: Sarah Yurick <[email protected]>

* add type checking

Signed-off-by: Sarah Yurick <[email protected]>

* add filter_by param to get_all_files_paths_under

Signed-off-by: Sarah Yurick <[email protected]>

* isort

Signed-off-by: Sarah Yurick <[email protected]>

* address ayush's comments

Signed-off-by: Sarah Yurick <[email protected]>

* run black

Signed-off-by: Sarah Yurick <[email protected]>

* trailing whitespace

Signed-off-by: Sarah Yurick <[email protected]>

* more whitespace

Signed-off-by: Sarah Yurick <[email protected]>

* address praateek's review

Signed-off-by: Sarah Yurick <[email protected]>

* praateek's review

Signed-off-by: Sarah Yurick <[email protected]>

---------

Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Walter Teng <[email protected]>

* remove deprecated convert_str_ids args  from ConnectedComponents

Signed-off-by: Walter Teng <[email protected]>

---------

Signed-off-by: Walter Teng <[email protected]>
Signed-off-by: Oliver Koenig <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: oliver könig <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Co-authored-by: oliver könig <[email protected]>
Co-authored-by: Sarah Yurick <[email protected]>

tutorials/pretraining-data-curation/red-pajama-v2-curation-tutorial.ipynb

@@ -2692,7 +2692,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": null,
    "id": "6bee85f3-5477-4b9c-b606-7bbbefbe6cfc",
    "metadata": {
     "tags": []
@@ -2749,7 +2749,6 @@
     "    cache_dir=cache_dir,\n",
     "    jaccard_pairs_path=jaccard_pairs_path,\n",
     "    id_column=id_field,\n",
-    "    convert_str_ids=True,\n",
     "    jaccard_threshold=jaccard_threshold,\n",
     ")\n",
     "components_stage.cc_workflow(output_path=output_path)\n",
@@ -4416,7 +4415,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": null,
    "id": "89398db9-d4e6-48ec-bad8-1d5ac553cadd",
    "metadata": {
     "tags": []
@@ -4454,7 +4453,6 @@
     "    cache_dir=cache_dir,\n",
     "    jaccard_pairs_path=jaccard_pairs_path,\n",
     "    id_column=id_field,\n",
-    "    convert_str_ids=True,\n",
     "    jaccard_threshold=jaccard_threshold,\n",
     ")\n",
     "components_stage.cc_workflow(output_path=output_path)\n",

tutorials/single_node_tutorial/single_gpu_tutorial.ipynb

@@ -1749,7 +1749,6 @@
     "    cache_dir=connected_component_cache_dir,\n",
     "    jaccard_pairs_path=jaccard_pairs_path,\n",
     "    id_column=input_id_field,\n",
-    "    convert_str_ids=True,\n",
     "    jaccard_threshold=jaccard_threshold,\n",
     ")\n",
     "\n",

tutorials/zyda2-tutorial/1_fuzzy_dedup/3_connected_components.py

@@ -41,7 +41,6 @@
         cache_dir=connected_component_cache_dir,
         jaccard_pairs_path=buckets_to_edges_out,
         id_column=input_id_field,
-        convert_str_ids=True,
     )
 
     # Load and run connected components