LLM standardisation

[dylanratcliffe]

This is a pretty substantial piece of work that aims to standardise the methods we use to call out to LLMs everywhere in the product.

Why?

• As we look at ways to improve the assistant (better tools, learning, new RAG approaches) we also need to make sure that whatever work we do there, also improves risks and vice versa.
• We need to be able to evaluate new models more easily
• We need to support things like Amazon Bedrock for our enterprise customers to allow them to keep data completely in-house

How?

This has been achieved by creating a new interface for an assistant conversation, and for tools, that allows the actual LLM provider to be pluggable without changing anything else. The three interfaces that matter here are:

• Provider: This represents the actual LLM, I have created providers for OpenAI and Anthropic
• Conversation: This represents a back and forth conversation with one of the providers. This interface stores state which means that it not only simplifies the way we need to interact with it, but abstracts over some of the differences between OpenAI and Anthropic. Like for example the fact that Anthropic doesn't really do server-side storage of conversations, however there is prompt caching which seems kind of similar...?
• ToolImplementation: This is the one I'm the most proud of. It allows anyone to use the Tool struct with nothing but a name, description and a function. The JSON schema is automatically determined from the data type, which uses generics. Very nice.

[dylanratcliffe]

Note, this is failing linting due to a non-inherited context. I wanted to get a quick review on this. Is what I'm doing sane?

[tphoney]

seems sane so far,

• it would be good to see how this fits in with the assistant in Gateway. so there is a single place for an llm conversation.
• having shared otel metrics, for conversations.
• where the configuration is for linking your llm to a customer. how granular should this be.

[dylanratcliffe]

it would be good to see how this fits in with the assistant in Gateway. so there is a single place for an llm conversation.

Agreed. This has been designed so that we can replace all of the stuff in gateway with this too so that it's a unified implementation. All we need to do is refactor the tools to this (simpler IMO) format and we're good to go

where the configuration is for linking your llm to a customer. how granular should this be

I think that's outside the scope of this library, but you can see how I've done it in api-server i.e.

• Create a central location to store the providers: https://github.com/overmindtech/api-server/pull/1224/files#diff-5efd4fd9208dae00a924b10a8f6f9807c78e5522dbe087646a42ef726c493f5eR352-R359
• And look up the provider based on name at runtime: https://github.com/overmindtech/api-server/pull/1224/files#diff-e7aa10889abdf71acdaaea10f3273391598b92dc5126891bfd15f8a9b14744e7R170-R177

We can change the granularity as required for each use-case

[DavidS-ovm]

I like the overall approach and left some more detailed technical comments below.

If this works out as a model we could consider splitting it out into its own dedicated OSS project and make some promotional activities around it to see if there's interest from others.

[DavidS-ovm]

todo: review if there is overlap to metrics from anthropic we'd like to share attribute names (ovm.llm.*?)

[dylanratcliffe]

The only overlap is:

span.SetAttributes(
	attribute.Int64("ovm.anthropic.usage.inputTokens", response.Usage.InputTokens),
	attribute.Int64("ovm.anthropic.usage.outputTokens", response.Usage.OutputTokens),
	attribute.String("ovm.anthropic.model", response.Model),
)

I decided to use anthropic in the name so that we didn't get overlap as I thought it could be confusing, but I used the same names otherwise i.e. InputTokens etc.

[DavidS-ovm]

👨‍🍳 💋