init

docs/CONTRIBUTING_DOCS.md

@@ -18,6 +18,9 @@ Satisfy the following dependencies to create, build, and locally serve Weave Doc
   ```yarn
   npm install --global yarn
   ```
+- Make sure your python environment is setup by running the following from the repro root:
+  - `pip install -r requirements.dev.txt` 
+  - `pip install -e .` 
 - Install an IDE (e.g. VS Code) or Text Editor (e.g. Sublime)
 
  
@@ -77,3 +80,50 @@ git push origin <your-feature-branch>
 ```
 
 8. Open a pull request from the new branch to the original repo.
+
+## DocGen
+Currently, we have 3 forms of doc generation:
+1. Python Doc Gen
+2. Service Doc Gen
+3. Notebook Doc Gen
+
+Assuming you have node and python packages installed, these can all be generated by running `make generate_reference_docs`.
+
+Let's review some details about each process:
+
+### Python Doc Gen
+
+See: `docs/scripts/generate_python_sdk_docs.py` and `./docs/reference/python-sdk`
+
+Python doc gen uses `lazydocs` as the core library for building markdown docs from our symbols. There are a few things to keep in mind:
+
+1. `docs/scripts/generate_python_sdk_docs.py` contains an allow-list of modules to document. Since the Weave codebase is massive, it is far easier to just select what modules are useful for docs.
+2. If the file does now have a `__docspec__` list of symbols, all non-underscore symbols will be documented. However, if it does have a `__docspec__`, that will further narrow the symbols to just that selection.
+3. Documentation itself:
+   1. Module-level: Put a triple double quote (""") comment as the first line of the module to add module-level documentation
+   2. Classes: Put a triple double quote (""") comment as the first line of the class to add class-level docs
+   3. Methods / Functions: Put a triple double quote (""") comment as the first line of the implementation to add method/function-level docs
+      1. Currently attributes are not automatically documented. Instead, use the @property pattern.
+      2. `BaseModel`. For classes that inherit from `BaseModel`, we create a special field list automatically to overcome this limitation.
+
+### Service Doc Gen
+
+See `docs/scripts/generate_service_api_spec.py`  and `./docs/reference/service-api`
+
+Service doc generation loads the `openapi.json` file describing the server, processes it, then uses the `docusaurus-plugin-openapi-docs` plugin to generate markdown files from that specification.
+
+To improve docs, basically follow FastAPI's instructions to create good Swagger docs by adding field-level and endpoint-level documentation using their APIs. Assuming your have made some changes, `docs/scripts/generate_service_api_spec.py` needs a server to point to. You can either deploy to prod, or run the server locally and point to it in `docs/scripts/generate_service_api_spec.py`. From there, `docs/scripts/generate_service_api_spec.py` will download the spec, clean it up, and build the docs!
+
+### Notebook Doc Gen
+
+See `docs/scripts/generate_notebooks.py`, `./docs/notebooks`, and `./docs/reference/gen_notebooks`.
+
+This script will load all notebooks in `./docs/notebooks`, transforming them into viable markdown docs in `./docs/reference/gen_notebooks` which can be referenced by docusaurus just like any other markdown file. If you need header metadata, you can add a markdown block at the top of your notebook with:
+```
+<!-- docusaurus_head_meta::start
+---
+title: Head Metadata
+---
+docusaurus_head_meta::end -->
+```
+

requirements.dev.txt

@@ -22,3 +22,5 @@ types-Pillow
 types-Flask-Cors
 build
 twine
+lazydocs
+nbconvert