middleware base #129

bekauz · 2024-12-20T15:55:50Z

introduces a base implementation of Valence Types to be used in programs.

new additions:

contracts/middleware/broker - type registry gateway meant to enable runtime upgrades to type registries
contracts/middleware/type-registries - actual type registries where developers can add new types to be used in valence
contracts/libraries/icq-querier - icq querier library which uses the broker for getting the KVKey needed for registering an ICQuery
packages/middleware-utils - utils used to define the common api for middleware system
local-interchaintest/examples/middleware_icq_test - e2e test showcasing the middleware usage to icq an osmosis gamm pool, parse it into a valence xyk type, and get its price

while some parts are still a little rough around the edges, I think it makes sense to put this pr up now to get some feedback around some key logic in this flow.

Broker

I expect the same broker to be used for any and all type registries or other types of plugins we may want to enable, as long as they comply with the expected interface.
Brokers wrap around the type registry api and relay the requests to the specified (or default=latest) version of the type registry. New type registries can be added to brokers during runtime to enable new types/functionality without requiring any upgrades from programs relying on a given broker.

Type Registries

This is where I spent the most time. In particular, contracts/middleware/type-registries/osmosis/osmo-26-0-0/src/definitions.

In mod.rs, we have a register_types! macro call. Currently it looks like this:

register_types! {
    "/osmosis.gamm.v1beta1.Pool" => {
        native_type: Pool,
        adapter: OsmosisXykPool,
        to_valence: ValenceType::XykPool,
    },
    "/cosmos.bank.v1beta1.QueryBalanceResponse" => {
        native_type: QueryBalanceResponse,
        adapter: OsmosisBankBalance,
        to_valence: ValenceType::BankBalance,
    }
}

Under the hood the macro generates a query handler that is used in the type registry contract:

pub fn query(_deps: Deps, _env: Env, msg: RegistryQueryMsg) -> StdResult<Binary> {
    handle_query(msg)
}

My thinking here was to minimize the lines of code needed to have a unified interface.
It can be seen as a map which maps the external type url (designator) to a set of functionality (plugins) we wish to enable for that type.
Zooming in on the gamm pool entry, we are basically saying:

"/osmosis.gamm.v1beta1.Pool" type url represents the osmosis::gamm::v1beta1::Pool native type which should map into the ValenceType::XykPool using the OsmosisXykPool adapter.

Valence Types are defined in utils packages and are out of developer (who are implementing the type registry) control.
Currently the set of enabled types is:

pub enum ValenceType {
    XykPool(ValenceXykPool),
    BankBalance(ValenceBankBalance),
}

Adapter is the only thing the developer needs to write, see example at contracts/middleware/type-registries/osmosis/osmo-26-0-0/src/definitions/gamm_pool.

Currently it feels like most (if not all) of those macro fields can be seen as optional. No type being registered should be expected to implement every adapter possible.
For example, we could introduce an evm_adapter field which could be used to specify the type/struct/function that knows how to do evm serde. It would only be useful for a small subset of all types we may be dealing with.

One thing to note here is that in order to maintain the unified api, these macros will embed every possible adapter for every type registry. Meaning, if we tried to call some evm_serde method on a type that does not have that adapter implemented, it would return an error along the lines of "evm adapter is not implemented for this type".

… 26+

… pool helper method

… queries

…trust issues

hxrts · 2024-12-23T09:44:17Z

I think the pretty name of the valence type should be an alias that we manage for convenience, but by default these types should be content addressed somehow. This allows anyone to add new valence types and external types to the registry and mapping them, but prevents squatting something like ValenceType:VaultDeposit.

So I'm an external developer I add the following to the registry:

sha256digest(valence_type)
sha256digest(evm_type)

The concatenation of these two content addresses concat(sha256digest(valence_type), "_", sha256digest(evm_type)) gives me a unique mapping between types which I can use in my program.

And then as a convenience, a multi-sig we own can create human readable aliases for a unique sha256digest(valence_type).

After which developers can use something like ValenceType:VaultDeposit_EVMType:VaultDeposit.

What's here is already a substantial addition, so we may want to address this is a separate PR.

hxrts · 2024-12-23T10:55:59Z

we should probably rename icq-querier to neutron-icq-querier to avoid future conflicts

keyleu · 2024-12-23T11:30:54Z

Cargo.toml

+prost                 = { version = "0.13.3", default-features = false, features = [ "prost-derive" ] }
+prost-types           = { version = "0.13.3", default-features = false }


nit: maybe add the default-features flag in the contract?

keyleu · 2024-12-23T11:31:30Z

contracts/libraries/icq-querier/README.md

@@ -0,0 +1 @@
+# ICQ-querier library


prob worth adding something here

keyleu · 2024-12-23T11:32:47Z

contracts/libraries/icq-querier/src/contract.rs

+    _info: MessageInfo,
+    _msg: InstantiateMsg,
+) -> Result<Response<NeutronMsg>, LibraryError> {
+    Ok(Response::default())


set contract version here?

keyleu · 2024-12-23T11:34:15Z

contracts/libraries/icq-querier/src/contract.rs

+        keys: vec![query_kv_key],
+        transactions_filter: String::new(),
+        connection_id,
+        update_period: 5,


should update_period be hardcoded?

keyleu · 2024-12-23T11:38:16Z

contracts/libraries/icq-querier/src/contract.rs

+            let mut resp = vec![];
+            for entry in ASSOCIATED_QUERY_IDS.range(deps.storage, None, None, Order::Ascending) {
+                resp.push(entry?);
+            }
+            to_json_binary(&resp)


nit: can just do

.filter_map(Result::ok) .map(|(_, entry)| entry) .collect()

for these queries

keyleu · 2024-12-23T13:10:47Z

contracts/middleware/broker/.cargo/config.toml

@@ -0,0 +1,3 @@
+[alias]
+wasm = "build --release --lib --target wasm32-unknown-unknown"
+schema = "run --bin schema"


can you create the schemas for both contracts? This way linter won't complain when we merge into main

keyleu · 2024-12-23T13:11:54Z

contracts/middleware/broker/Cargo.toml

+prost                 = { workspace = true }
+prost-types           = { workspace = true }


would add the default-features = false here like for serde-json-wasm (linked to prev comment)

keyleu · 2024-12-23T13:11:57Z

contracts/middleware/broker/Cargo.toml

+osmosis-std           = "0.26.0"
+osmosis-std-derive    = "0.26.0"


nit: we have these in workspace
nit2: indent.

keyleu · 2024-12-23T13:13:08Z

contracts/middleware/broker/src/contract.rs

+    _info: MessageInfo,
+    _msg: InstantiateMsg,
+) -> Result<Response, MiddlewareError> {
+    Ok(Response::default())


same as other contract. Add versioning for migration.

keyleu · 2024-12-24T11:49:27Z

packages/middleware-utils/src/canonical_types/pools/xyk.rs

+/*
+OSMOSIS POOL
+


bekauz added 30 commits November 14, 2024 18:42

init osmo gamm test; bump neutron_osmosis chainconfig osmo version to…

5d43765

… 26+

creating gamm pool

2661681

add osmo gamm libraries to program manager

db3f1fd

setting up polytone contracts pre-manager init

c7ea87b

storing polytone info in global config

284c1e7

wip: setting polytone config on non-main domains

824299f

wip: polytone integration in program manager

a6a090d

cleanup gc load in program manager

5ab9d5b

adjust get_bridge_info

34d7bd9

manager fixes

ce31c1a

clippyfmt

e73390f

funding input acc; wip: processing the lp message

c498862

wip: passing messages to osmosis processor

345325f

fix library denom order in test

1a9b6cb

follow liquidity provision with liquidity withdrawal works

6706b01

extend test with main domain authorization module callback validation

0daaa8d

fix manager processor instantiation

f7f15ca

rename polytone param in instantiate_processor fn

2c51981

confirm withdraw callback result is success on authorizations

7186127

remote domain processor queue length assertion helper

47b5464

more cleanups

8339f91

include workflow in ci

c310e7f

remove logs

765b1dd

merge

470097d

regen schema

9310661

final cleanup

e365d40

init cl e2e test

b7404af

update program manager library with osmo cl variants

479f6c7

alter osmo genesis to allow permissionless pool creation; creating CL…

4c23aeb

… pool helper method

asserting that CL position was created and transferred to output acc

ffe17c8

bekauz added 23 commits December 16, 2024 15:39

add canonical-types package; rework middleware impl

ef45e9b

merge packages into middleware-utils

31fd78d

cleanup org

9cdaa37

base bank balance integration

4b9c680

integrate balance query into valence types

9756fec

init broker

181f533

registry cntd; init type_registry type mod in utils

e3a00e0

replace individual type adapters with ValenceTypeAdapter

28caad3

testing mod in broker; integrate kv_key & proto_reconstruct carryover…

f293e04

… queries

cleanup broker

007d2dc

remove broker utils by moving logic into broker

6af3591

simplify broker query logic by wrapping the type registry query msg

d5a5ad3

rename middleware domains to type_registries

22616e1

middleware & type registry readmes

d6ecb31

integrate to & from canonical type conversions to broker

3ce96a3

remove hardcoded type urls; add middleware-utils readme

d46e8a3

wip: type registry unifying api macro seems to be working but i have …

8423e2f

…trust issues

cleanup

594bb4d

init middleware e2e test

31ff6a4

adjust icq querier to broker logic

ee9411a

e2e run to getting valence type and querying the price

579f11b

remove old type registry

81308f8

remove icq-querier-utils; cleanup

5eec62c

bekauz requested review from hxrts, uditvira, Art3miX and keyleu December 20, 2024 15:55

keyleu reviewed Dec 24, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

middleware base #129

middleware base #129

bekauz commented Dec 20, 2024

hxrts commented Dec 23, 2024 •

edited

Loading

hxrts commented Dec 23, 2024

keyleu Dec 23, 2024

keyleu Dec 23, 2024

keyleu Dec 23, 2024

keyleu Dec 23, 2024

keyleu Dec 23, 2024

keyleu Dec 23, 2024

keyleu Dec 23, 2024

keyleu Dec 23, 2024

keyleu Dec 23, 2024

keyleu Dec 24, 2024

		prost = { version = "0.13.3", default-features = false, features = [ "prost-derive" ] }
		prost-types = { version = "0.13.3", default-features = false }

		prost = { workspace = true }
		prost-types = { workspace = true }

		osmosis-std = "0.26.0"
		osmosis-std-derive = "0.26.0"

		/*
		OSMOSIS POOL

middleware base #129

Are you sure you want to change the base?

middleware base #129

Conversation

bekauz commented Dec 20, 2024

Broker

Type Registries

hxrts commented Dec 23, 2024 • edited Loading

hxrts commented Dec 23, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hxrts commented Dec 23, 2024 •

edited

Loading