Accessing Block values without the class definition

[zzstoatzz]

Background

this discussion is inspired by this issue #14899

---

Currently, to load and use an existing custom Prefect block, we need to:

0. have a custom block type created and registered, and an instance of that block saved

how to create and register a block type, then save an instance of that block

class CustomBlock(Block):
    _block_type_slug = "my-custom-block-type"

    foo: str
    secret: SecretStr
    some_dict: dict[str, str]


CustomBlock.register_type_and_schema()

CustomBlock(
    foo="bar",
    secret=SecretStr("test"),
    some_dict={"foo": "bar"},
).save("my-custom-block", overwrite=True)

1. have the block's class definition available when you want to load the block

from my_module import CustomBlock

CustomBlock.load("my-custom-block")

This means that practically, you need to ship your class definition to your runtime somehow, like packaging it into an internal library that you pip install in your image or otherwise make available in your python environment. This is a bit cumbersome if you only wanted to access the values from your saved block instance.

Today, it is possible to load the BlockDocument from the server without the class present, which looks like:

In [1]: from prefect import get_client

In [2]: async with get_client() as client:
   ...:     print(await client.read_block_document_by_name("my-custom-block", "my-custom-block-type"))

Out[2]: BlockDocument(id=..., name='my-custom-block', data={...}, ...)

but this is not a first-class experience (nor is using the private method Block._get_block_document) as:

• it requires a bespoke client method
• a user needs to know the shape of a "non-public" class BlockDocument to fetch their data
• there is no typing information associated with the values of the block

Below we'll go through some proposals to address some of the above.

Proposed enhancements

Important

No proposed solution will involve recovering methods on the block subclass, since we would strictly need the actual Block subclass to access/use these (as methods are not serializable / not stored on the server etc)

A more first-class experience might look like:

# say we saved some class earlier
# class CustomBlock(Block):
#    some_value: Any

# and later, we want to load a view of this block
from prefect.blocks.core import BlockView

block_view_A: BlockView = BlockView.from_block_name("custom-block-type/my-block-name")
print(block_view.some_value) # we could have attribute access here via `extra="allow"`


# or perhaps instead the view is just the raw contents of the block document
from prefect.blocks.core import get_block_view

block_view_B: dict = get_block_view("custom-block-type/my-block-name")
print(block_view.get("some_value"))

Comments on tradeoffs between these two options

version A

could look something like this

class BlockView(BaseModel):
    model_config = ConfigDict(extra="allow")

    @classmethod
    def from_block_name(cls, block_name: str) -> Self:
        block_document, _ = asyncio.run(Block._get_block_document(block_name))

        return cls.model_validate(block_document.data)

pro

• preserves some semblance of intuitive attribute / dot access of fields

con

• it gets tricky when you have nested blocks or basemodels on the original class, since recreating a faithful model where attribute access is preserved all the way down would require us to inspect the original schema and make an extra="allow" model for each such field we find
• not a straightforward way to get strong type hinting on the extra fields

version B

could look like this

def get_block_view(name: str) -> dict[str, Any]:
    block_document, _ = asyncio.run(Block._get_block_document(name))
    return block_document.data

pro

• simpler to implement and reason about, not pretending to be a "real" Block

con

• no attribute access, just a dict
• still no typing, as its just a dict

Important

On Schema to Model Conversion: It would be simple to create the view (i.e. "data portion") of the Block subclass without the original if we could reliably convert an OpenAPI schema into a real pydantic model, but the only reasonable way I'm familiar with to do this is using datamodel-code-generator, which is not something we want to hastily adopt just for this case.

Questions to discuss

1. Do you find it cumbersome to get custom block types to your runtime? i.e. does this proposal sound useful?

2. What level of type safety and familiar features (e.g. attribute access) would you expect from a BlockView-like solution?