DOC: Auto-generate Pydantic model documentation #705
Conversation
mferrera
force-pushed
the
model-docs
branch
2 times, most recently
from
6a272df to
cdbe036
Compare
mferrera
changed the title
| # ----------- pydantic_autosummary change | ||
| from .generate import generate_autosummary_docs | ||
| # ----------/ pydantic_autosummary change |
There was a problem hiding this comment.
This file is a direct copy of autosummary, and this is the only change in it (it was previously sphinx.autosummary.generate or so), therefore I don't think the rest of this file needs to be reviewed.
There were very minor changes just to adapt to our ruff linting.
| # ----------- pydantic_autosummary change | ||
| from . import ( | ||
| ImportExceptionGroup, | ||
| get_documenter, | ||
| import_by_name, | ||
| import_ivar_by_name, | ||
| ) | ||
| from .pydantic import set_pydantic_model_fields | ||
|
|
||
| # ----------/ pydantic_autosummary change |
There was a problem hiding this comment.
This file is also a direct copy from autosummary, and all changes I made are marked here.
The first import set just made the path relative over sphinx.autosummary, but importing from .pydantic import set_pydantic_model_fields was my addition.
| # ----------- pydantic_autosummary change | ||
| system_templates_path = [os.path.join(os.path.dirname(__file__), "templates")] | ||
| # ----------/ pydantic_autosummary change |
There was a problem hiding this comment.
This changed the templates path to use the one embedded in the same directory over the one shipped with Sphinx.
| # ----------- pydantic_autosummary change | ||
| logger.warning( | ||
| __( | ||
| "pydantic_autosummary: failed to determine %r to be documented, " | ||
| "the following exception was raised:\n%s" | ||
| ), | ||
| name, | ||
| exc, | ||
| type="autosummary", | ||
| ) | ||
| # ----------/ pydantic_autosummary change |
There was a problem hiding this comment.
Just changed the logger to use pydantic_autosummary: instead of autosummary:
| # ----------- pydantic_autosummary change | ||
| ns["pydantic_models"], ns["all_pydantic_models"] = _get_members( | ||
| doc, app, obj, {"pydantic_model"}, imported=imported_members | ||
| ) | ||
| ns["pydantic_settings"], ns["all_pydantic_settings"] = _get_members( | ||
| doc, app, obj, {"pydantic_settings"}, imported=imported_members | ||
| ) | ||
| # ----------/ pydantic_autosummary change |
| # ----------- pydantic_autosummary change | ||
| if doc.objtype == "pydantic_model": | ||
| set_pydantic_model_fields(ns, obj) | ||
| # ----------/ pydantic_autosummary change |
There was a problem hiding this comment.
Takes us to where new code was added, pydantic.py
| # ----------- pydantic_autosummary change | ||
| logger.warning( | ||
| __( | ||
| "pydantic_autosummary: failed to determine %r to be documented, " | ||
| "the following exception was raised:\n%s" | ||
| ), | ||
| name, | ||
| exc, | ||
| type="autosummary", | ||
| ) | ||
| # ----------/ pydantic_autosummary change |
There was a problem hiding this comment.
Just changed the logger to use pydantic_autosummary: instead of autosummary:
| # ----------- pydantic_autosummary change | ||
| logger.info( | ||
| __("[pydantic_autosummary] generating autosummary for: %s") | ||
| % ", ".join(showed_sources) | ||
| ) | ||
| if output_dir: | ||
| logger.info(__("[pydantic_autosummary] writing to %s") % output_dir) | ||
| # ----------/ pydantic_autosummary change |
There was a problem hiding this comment.
Just changed the logger to use pydantic_autosummary: instead of autosummary:
| # ----------- pydantic_autosummary change | ||
| logger.warning( | ||
| __( | ||
| "[pydantic_autosummary] failed to import %s.\n" | ||
| "Possible hints:\n%s" | ||
| ), | ||
| entry.name, | ||
| "\n".join(errors), | ||
| ) | ||
| # ----------/ pydantic_autosummary change |
There was a problem hiding this comment.
Just changed the logger to use pydantic_autosummary: instead of autosummary:
| @@ -0,0 +1,64 @@ | |||
| from __future__ import annotations | |||
There was a problem hiding this comment.
Everything in this file was added by me, not pre-existing
| @@ -0,0 +1,5 @@ | |||
| {{ fullname | escape | underline}} | |||
There was a problem hiding this comment.
This is the default autosummary base template, not written by me
| @@ -0,0 +1,29 @@ | |||
| {{ fullname | escape | underline}} | |||
There was a problem hiding this comment.
Same, not written by me
| @@ -0,0 +1,60 @@ | |||
| {{ fullname | escape | underline}} | |||
| @@ -0,0 +1,51 @@ | |||
| {{ name | escape | underline}} | |||
There was a problem hiding this comment.
New template added by me
| :members: | ||
| :inherited-members: BaseModel | ||
| :model-show-config-summary: False | ||
| :model-show-json: False | ||
| :model-show-validator-members: False | ||
| :model-show-validator-summary: False | ||
| :field-list-validators: False |
There was a problem hiding this comment.
These options descriptions are documented here. If we make a parallel model documentation for ourselves, some of these may want to be removed, hence they were not set in conf.py instead.
| "navigation_depth": -1, | ||
| "collapse_navigation": False, | ||
| "titles_only": True, |
There was a problem hiding this comment.
These allow the sidebar menu expansion to function as we wanted them to
| The FMU data model is described using a `Pydantic <https://pydantic.dev/>`__ model | ||
| which programmatically generates a `JSON Schema <https://json-schema.org/>`__. |
There was a problem hiding this comment.
This text is mostly a copy of the previous datamodel.rst, with small changes like this
| Data model documentation | ||
| ======================== | ||
|
|
||
| There are two closely related data models represented here: metadata generated from | ||
| an FMU realization and metadata generated on a case level. The structure and | ||
| documentation of these two models can be inspected from here. | ||
|
|
||
| .. autosummary:: | ||
| :toctree: model/ | ||
| :recursive: | ||
|
|
||
|
|
||
| .. toctree:: | ||
| :maxdepth: -1 | ||
|
|
||
| fmu.dataio.datastructure.meta.meta.FMUDataClassMeta | ||
| fmu.dataio.datastructure.meta.meta.FMUCaseClassMeta |
| from fmu.dataio._definitions import SCHEMA, SOURCE, VERSION | ||
| from fmu.dataio.datastructure.meta import meta | ||
|
|
This implements a fork/copy of autosummary, stored internally, to generate hierarchically structured documentation for the Pydantic model that represents the metadata.
|
Thanks, and agreed. I made #711 and will continue digging into it cause I dislike the flood of messages too. |
Resolves #618
Resolves #591
Resolves #695
This PR adds auto-generated documentation for the Pydantic model, representing it as a nested hierarchy in the same way the Pydantic model describes it.
It does this by copying and modifying the Sphinx autosummary extension to handle Pydantic specific things.
It collects the previous datamodel documentation to the same place, removes a large section describing the model, and makes some small additions to include the links to the new documentation.
Unfortunately when building the docs a large number of warnings occur, and a smaller number of errors. The warnings are not avoidable because, f.eks., we want to generate a
Userdoc wherever a one appears in the model (multiple places), not have a single reference to it in the abstract.The errors are a result of
in
conf.py, so that Pydantic models don't display theBaseModel.__init__docstring from pydantic (see this). I think they are a false positive stemming from BaseModel, but am not certain.