chore: add automatic cell removal, update contrib doc

docs/CONTRIBUTING_DOCS.md

@@ -116,11 +116,10 @@ To improve docs, basically follow FastAPI's instructions to create good Swagger
 
 ### Notebook Doc Gen
 
-Run `python scripts/generate_notebook.py` inside of `docs/` to automatically create a markdown version of the notebook. Add all new files to the commit.
+From within the `docs` directory, run `python ./scripts/generate_notebook.py`.
+This script will load all notebooks in `./docs/notebooks`, transforming them into viable markdown docs in `./docs/reference/gen_notebooks` which can be referenced by docusaurus just like any other markdown file. To publish the page, add the notebook and the markdown file to a PR.
 
-See `docs/scripts/generate_notebooks.py`, `./docs/notebooks`, and `./docs/reference/gen_notebooks`.
-
-This script will load all notebooks in `./docs/notebooks`, transforming them into viable markdown docs in `./docs/reference/gen_notebooks` which can be referenced by docusaurus just like any other markdown file. If you need header metadata, you can add a markdown block at the top of your notebook with:
+If you need header metadata, you can add a markdown block at the top of your notebook with:
 ```
 <!-- docusaurus_head_meta::start
 ---
@@ -129,8 +128,8 @@ title: Head Metadata
 docusaurus_head_meta::end -->
 ```
 
+For more details, see `docs/scripts/generate_notebooks.py`, `./docs/notebooks`, and `./docs/reference/gen_notebooks`.
+
 # Troubleshooting
+
 - Make sure to update your version of pip (tested on 24.2) before installing the requirements.dev.txt
-- Using the `docs/scripts/generate_notebook.py`
-    - Make sure to be in `docs/` when you run this command (the script is looking for `./notebooks`)
-    - The conversion might break because specific cell output might be misinterpreted - if you can delete the cell output of package installations

docs/scripts/generate_notebooks.py

@@ -1,5 +1,6 @@
 from nbconvert import MarkdownExporter
-
+from nbconvert.preprocessors import ClearOutputPreprocessor
+from nbconvert import NotebookExporter
 
 def make_header(notebook_path):
     github_uri = "wandb/weave/blob/master/docs"
@@ -22,8 +23,12 @@ def make_header(notebook_path):
 
 
 def export_notebook(notebook_path, output_path):
-    exporter = MarkdownExporter()
-    output, resources = exporter.from_filename(notebook_path)
+
+    notebook_exporter = NotebookExporter()
+    notebook_exporter.preprocessors = [ClearOutputPreprocessor()]
+    notebook_content, _ = notebook_exporter.from_filename(notebook_path)
+    markdown_exporter = MarkdownExporter()
+    output, _ = markdown_exporter.from_notebook_node(notebook_content)
 
     extract_meta = ""
     meta_mark_start = "<!-- docusaurus_head_meta::start\n"