From 49de7dc6768e8a199f3c7f00d8342dff6f613206 Mon Sep 17 00:00:00 2001 From: Tim Sweeney Date: Fri, 9 Aug 2024 18:51:17 -0700 Subject: [PATCH] init --- docs/CONTRIBUTING_DOCS.md | 50 +++++++++++++++++++++++++++++++++++++++ requirements.dev.txt | 2 ++ 2 files changed, 52 insertions(+) diff --git a/docs/CONTRIBUTING_DOCS.md b/docs/CONTRIBUTING_DOCS.md index 9fdc1e5e203..31a0d04f13c 100644 --- a/docs/CONTRIBUTING_DOCS.md +++ b/docs/CONTRIBUTING_DOCS.md @@ -18,6 +18,9 @@ Satisfy the following dependencies to create, build, and locally serve Weave Doc ```yarn npm install --global yarn ``` +- Make sure your python environment is setup by running the following from the repro root: + - `pip install -r requirements.dev.txt` + - `pip install -e .` - Install an IDE (e.g. VS Code) or Text Editor (e.g. Sublime)   @@ -77,3 +80,50 @@ git push origin ``` 8. Open a pull request from the new branch to the original repo. + +## DocGen +Currently, we have 3 forms of doc generation: +1. Python Doc Gen +2. Service Doc Gen +3. Notebook Doc Gen + +Assuming you have node and python packages installed, these can all be generated by running `make generate_reference_docs`. + +Let's review some details about each process: + +### Python Doc Gen + +See: `docs/scripts/generate_python_sdk_docs.py` and `./docs/reference/python-sdk` + +Python doc gen uses `lazydocs` as the core library for building markdown docs from our symbols. There are a few things to keep in mind: + +1. `docs/scripts/generate_python_sdk_docs.py` contains an allow-list of modules to document. Since the Weave codebase is massive, it is far easier to just select what modules are useful for docs. +2. If the file does now have a `__docspec__` list of symbols, all non-underscore symbols will be documented. However, if it does have a `__docspec__`, that will further narrow the symbols to just that selection. +3. Documentation itself: + 1. Module-level: Put a triple double quote (""") comment as the first line of the module to add module-level documentation + 2. Classes: Put a triple double quote (""") comment as the first line of the class to add class-level docs + 3. Methods / Functions: Put a triple double quote (""") comment as the first line of the implementation to add method/function-level docs + 1. Currently attributes are not automatically documented. Instead, use the @property pattern. + 2. `BaseModel`. For classes that inherit from `BaseModel`, we create a special field list automatically to overcome this limitation. + +### Service Doc Gen + +See `docs/scripts/generate_service_api_spec.py` and `./docs/reference/service-api` + +Service doc generation loads the `openapi.json` file describing the server, processes it, then uses the `docusaurus-plugin-openapi-docs` plugin to generate markdown files from that specification. + +To improve docs, basically follow FastAPI's instructions to create good Swagger docs by adding field-level and endpoint-level documentation using their APIs. Assuming your have made some changes, `docs/scripts/generate_service_api_spec.py` needs a server to point to. You can either deploy to prod, or run the server locally and point to it in `docs/scripts/generate_service_api_spec.py`. From there, `docs/scripts/generate_service_api_spec.py` will download the spec, clean it up, and build the docs! + +### Notebook Doc Gen + +See `docs/scripts/generate_notebooks.py`, `./docs/notebooks`, and `./docs/reference/gen_notebooks`. + +This script will load all notebooks in `./docs/notebooks`, transforming them into viable markdown docs in `./docs/reference/gen_notebooks` which can be referenced by docusaurus just like any other markdown file. If you need header metadata, you can add a markdown block at the top of your notebook with: +``` + +``` + diff --git a/requirements.dev.txt b/requirements.dev.txt index c5fb9ca05fc..e0aad74fd36 100644 --- a/requirements.dev.txt +++ b/requirements.dev.txt @@ -22,3 +22,5 @@ types-Pillow types-Flask-Cors build twine +lazydocs +nbconvert