Unrelated models showing up in extension due to AsdfPydanticConverter being a singleton

[ketozhang]

Because AsdfPydanticConverter is a module-scoped singleton, once it gets imported to Python's global scope, any models registered will still exists. What ends up happening is any downstream extension automatically includes the models of any extensions using this converter upstream.

from asdf_pydantic import AsdfPydanticConverter
AsdfPydanticConverter.add_models(Circle)

class FooExtension(Extension):
    extension_uri = "asdf://asdf-pydantic/examples/extensions/foo-1.0.0"
    converters = [AsdfPydanticConverter()]
    tags = [
        TagDefinition(Circle._tag, schema_uris=[Circle._tag + "/schema"]),
    ]

# In another file
from asdf_pydantic import AsdfPydanticConverter

AsdfPydanticConverter.add_models(Rectangle)

class BarExtension(Extension):
    extension_uri = "asdf://asdf-pydantic/examples/extensions/bar-1.0.0"
    converters = [AsdfPydanticConverter()]
    tags = [
        TagDefinition(Rectangle._tag, schema_uris=[Circle._tag + "/schema"]),
    ]

Here BarExtension will not only have Rectangle but also Circle

[ketozhang]

This has to be a breaking change as all users must move from the singleton usage (global scope) to a local scope.

converter = AsdfPydanticConverter([Circle, Rectangle])

class MyExtension(Extension):
    extension_uri = "asdf://asdf-pydantic/examples/extensions/my-1.0.0"
    converters = [converter]
    tags = [
        TagDefinition(Circle._tag, schema_uris=[Circle._tag + "/schema"]),
        TagDefinition(Rectangle._tag, schema_uris=[Rectangle._tag + "/schema"]),
    ]

To continue using the add_models() API, maybe something like this.

import functools

converter = functools.reduce(AsdfPydanticConverter.add_models, (Circle, Rectangle))

[ketozhang]

Import optimization

For import optimization where imports slows down asdf.open, there needs to be a way to initialize AsdfPydanticConverter without needing to import any data models directly since the converter is loaded during asdf.open.

ASDF's converter class uses fully qualified module names during definition of the class...

Details

from asdf.extension import Converter

class RectangleConverter(Converter):
    tags = ...
    types = ["example_package.shapes.Rectangle"]

    def to_yaml_tree(self, obj, tag, ctx): ...

    def from_yaml_tree(self, node, tag, ctx):
        from example_package.shapes import Rectangle
        return Rectangle(node["width"], node["height"])

...this could be moved to the instantiation of the class,

AsdfPydanticConverter(
  [
    {tag: "...", type: "example_package.shapes.Rectangle"}, 
    ...
  ]
)

However I quite dislike the number of hardcoding you have to do, especially with the tag showing up in AsdfPydanticModel, AsdfPydanticConverter, and Extension.

Solution: Depend on static objects

Details

Instead of extensions and converters importing the static metadata from the model, a static object could be in a lightweight module. This static object needs to contain the tag (URI), schema (URI), and model's qualified name.

flowchart LR
  subgraph foo[AsdfPydanticModelMetaDict]
    schema[Schema]
    tag[Tag]
    model_name["Model (name)"]
  end

  schema --> extension
  tag --> extension
  converter --> extension
  tag --> converter
  either{either} --> converter
  model[Model] --> either
  model_name --> either 

  schema --> model
  tag --> model

Loading

registry = AsdfPydanticMetaRegistry({"tag": ..., "schema_uris": [...], "types": [...]})
converter = AsdfPydanticConverter.from_registry(registry)

class MyExtension(Extension):
    extension_uri = "asdf://asdf-pydantic/examples/extensions/my-1.0.0"
    converters = [converter]
    tags = [
        registry.get_tag_definition("examples.shapes.Circle"),
        registry.get_tag_definition(Rectangle),   # <-- not import-optimized
    ]

Solution: Optimize imports in models

Or we could just optimize how imports are done on the model implementation.