Skip to content

Commit

Permalink
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'master' into kyle/fix-model-providers-path
Browse files Browse the repository at this point in the history
KyleGoyette authored Oct 30, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
2 parents e1d0c0d + 7a8a673 commit a9ad0b2
Showing 58 changed files with 4,316 additions and 598 deletions.
3 changes: 3 additions & 0 deletions .github/workflows/pr.yaml
Original file line number Diff line number Diff line change
@@ -30,7 +30,10 @@ jobs:
scopes: |
ui
weave
weave_ts
weave_query
app
dev
wip: false
requireScope: true
validateSingleCommit: false
373 changes: 373 additions & 0 deletions docs/docs/guides/core-types/prompts.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,373 @@
# Prompts

Creating, evaluating, and refining prompts is a core activity for AI engineers.
Small changes to a prompt can have big impacts on your application's behavior.
Weave lets you create prompts, save and retrieve them, and evolve them over time.
Some of the benefits of Weave's prompt management system are:

- Unopinionated core, with a batteries-included option for rapid development
- Versioning that shows you how a prompt has evolved over time
- The ability to update a prompt in production without redeploying your application
- The ability to evaluate a prompt against many inputs to evaluate performance

## Getting started

If you want complete control over how a Prompt is constructed, you can subclass the base class, `weave.Prompt`, `weave.StringPrompt`, or `weave.MessagesPrompt` and implement the corresponding `format` method. When you publish one of these objects with `weave.publish`, it will appear in your Weave project on the "Prompts" page.

```
class Prompt(Object):
def format(self, **kwargs: Any) -> Any:
...
class StringPrompt(Prompt):
def format(self, **kwargs: Any) -> str:
...
class MessagesPrompt(Prompt):
def format(self, **kwargs: Any) -> list:
...
```

Weave also includes a "batteries-included" class called `EasyPrompt` that can be simpler to start with, especially if you are working with APIs that are similar to OpenAI. This document highlights the features you get with EasyPrompt.

## Constructing prompts

You can think of the EasyPrompt object as a list of messages with associated roles, optional
placeholder variables, and an optional model configuration.
But constructing a prompt can be as simple as providing a single string:

```python
import weave

prompt = weave.EasyPrompt("What's 23 * 42?")
assert prompt[0] == {"role": "user", "content": "What's 23 * 42?"}
```

For terseness, the weave library aliases the `EasyPrompt` class to `P`.

```python
from weave import P
p = P("What's 23 * 42?")
```

It is common for a prompt to consist of multiple messages. Each message has an associated `role`.
If the role is omitted, it defaults to `"user"`.

**Some common roles**

| Role | Description |
| --------- | -------------------------------------------------------------------------------------------------------------------- |
| system | System prompts provide high level instructions and can be used to set the behavior, knowledge, or persona of the AI. |
| user | Represents input from a human user. (This is the default role.) |
| assistant | Represents the AI's generated replies. Can be used for historical completions or to show examples. |

For convenience, you can prefix a message string with one of these known roles:

```python
import weave

prompt = weave.EasyPrompt("system: Talk like a pirate")
assert prompt[0] == {"role": "system", "content": "Talk like a pirate"}

# An explicit role parameter takes precedence
prompt = weave.EasyPrompt("system: Talk like a pirate", role="user")
assert prompt[0] == {"role": "user", "content": "system: Talk like a pirate"}

```

Messages can be appended to a prompt one-by-one:

```python
import weave

prompt = weave.EasyPrompt()
prompt.append("You are an expert travel consultant.", role="system")
prompt.append("Give me five ideas for top kid-friendly attractions in New Zealand.")
```

Or you can append multiple messages at once, either with the `append` method or with the `Prompt`
constructor, which is convenient for constructing a prompt from existing messages.

```python
import weave

prompt = weave.EasyPrompt()
prompt.append([
{"role": "system", "content": "You are an expert travel consultant."},
"Give me five ideas for top kid-friendly attractions in New Zealand."
])

# Same
prompt = weave.EasyPrompt([
{"role": "system", "content": "You are an expert travel consultant."},
"Give me five ideas for top kid-friendly attractions in New Zealand."
])
```

The Prompt class is designed to be easily inserted into existing code.
For example, you can quickly wrap it around all of the arguments to the
OpenAI chat completion `create` call including its messages and model
configuration. If you don't wrap the inputs, Weave's integration would still
track all of the call's inputs, but it would not extract them as a separate
versioned object. Having a separate Prompt object allows you to version
the prompt, easily filter calls by that version, etc.

```python
from weave import init, P
from openai import OpenAI
client = OpenAI()

# Must specify a target project, otherwise the Weave code is a no-op
# highlight-next-line
init("intro-example")

# highlight-next-line
response = client.chat.completions.create(P(
model="gpt-4o-mini",
messages=[
{"role": "user", "content": "What's 23 * 42?"}
],
temperature=0.7,
max_tokens=64,
top_p=1
# highlight-next-line
))
```

:::note
Why this works: Weave's OpenAI integration wraps the OpenAI `create` method to make it a Weave Op.
When the Op is executed, the Prompt object in the input will get saved and associated with the Call.
However, it will be replaced with the structure the `create` method expects for the execution of the
underlying function.
:::

## Parameterizing prompts

When specifying a prompt, you can include placeholders for values you want to fill in later. These placeholders are called "Parameters".
Parameters are indicated with curly braces. Here's a simple example:

```python
import weave

prompt = weave.EasyPrompt("What's {A} + {B}?")
```

You will specify values for all of the parameters or "bind" them, when you [use the prompt](#using-prompts).

The `require` method of Prompt allows you to associate parameters with restrictions that will be checked at bind time to detect programming errors.

```python
import weave

prompt = weave.EasyPrompt("What's {A} + 42?")
prompt.require("A", type="int", min=0, max=100)

prompt = weave.EasyPrompt("system: You are a {profession}")
prompt.require("profession", oneof=('pirate', 'cartoon mouse', 'hungry dragon'), default='pirate')
```

## Using prompts

You use a Prompt by converting it into a list of messages where all template placeholders have been filled in. You can bind a prompt to parameter values with the `bind` method or by simply calling it as a function. Here's an example where the prompt has zero parameters.

```python
import weave
prompt = weave.EasyPrompt("What's 23 * 42?")
assert prompt() == prompt.bind() == [
{"role": "user", "content": "What's 23 * 42?"}
]
```

If a prompt has parameters, you would specify values for them when you use the prompt.
Parameter values can be passed in as a dictionary or as keyword arguments.

```python
import weave
prompt = weave.EasyPrompt("What's {A} + {B}?")
assert prompt(A=5, B="10") == prompt({"A": 5, "B": "10"})
```

If any parameters are missing, they will be left unsubstituted in the output.

Here's a complete example of using a prompt with OpenAI. This example also uses [Weave's OpenAI integration](../integrations/openai.md) to automatically log the prompt and response.

```python
import weave
from openai import OpenAI
client = OpenAI()

weave.init("intro-example")
prompt = weave.EasyPrompt()
prompt.append("You will be provided with a tweet, and your task is to classify its sentiment as positive, neutral, or negative.", role="system")
prompt.append("I love {this_thing}!")

response = client.chat.completions.create(
model="gpt-4o-mini",
messages=prompt(this_thing="Weave"),
temperature=0.7,
max_tokens=64,
top_p=1
)
```

## Publishing to server

Prompt are a type of [Weave object](../tracking/objects.md), and use the same methods for publishing to the Weave server.
You must specify a destination project name with `weave.init` before you can publish a prompt.

```python
import weave

prompt = weave.EasyPrompt()
prompt.append("What's 23 * 42?")

weave.init("intro-example") # Use entity/project format if not targeting your default entity
weave.publish(prompt, name="calculation-prompt")
```

Weave will automatically determine if the object has changed and only publish a new version if it has.
You can also specify a name or description for the Prompt as part of its constructor.

```python
import weave

prompt = weave.EasyPrompt(
"What's 23 * 42?",
name="calculation-prompt",
description="A prompt for calculating the product of two numbers.",
)

weave.init("intro-example")
weave.publish(prompt)
```

## Retrieving from server

Prompt are a type of [Weave object](../tracking/objects.md), and use the same methods for retrieval from the Weave server.
You must specify a source project name with `weave.init` before you can retrieve a prompt.

```python
import weave

weave.init("intro-example")
prompt = weave.ref("calculation-prompt").get()
```

By default, the latest version of the prompt is returned. You can make this explicit or select a specific version by providing its version id.

```python
import weave

weave.init("intro-example")
prompt = weave.ref("calculation-prompt:latest").get()
# "<prompt_name>:<version_digest>", for example:
prompt = weave.ref("calculation-prompt:QSLzr96CTzFwLWgFFi3EuawCI4oODz4Uax98SxIY79E").get()
```

It is also possible to retrieve a Prompt without calling `init` if you pass a fully qualified URI to `weave.ref`.

## Loading and saving from files

Prompts can be saved to files and loaded from files. This can be convenient if you want your Prompt to be versioned through
a mechanism other than Weave such as git, or as a fallback if Weave is not available.

To save a prompt to a file, you can use the `dump_file` method.

```python
import weave

prompt = weave.EasyPrompt("What's 23 * 42?")
prompt.dump_file("~/prompt.json")
```

and load it again later with `Prompt.load_file`.

```python
import weave

prompt = weave.EasyPrompt.load_file("~/prompt.json")
```

You can also use the lower level `dump` and `Prompt.load` methods for custom (de)serialization.

## Evaluating prompts

The [Parameter feature of prompts](#parameterizing-prompts) can be used to execute or evaluate variations of a prompt.

You can bind each row of a [Dataset](./datasets.md) to generate N variations of a prompt.

```python
import weave

# Create a dataset
dataset = weave.Dataset(name='countries', rows=[
{'id': '0', 'country': "Argentina"},
{'id': '1', 'country': "Belize"},
{'id': '2', 'country': "Canada"},
{'id': '3', 'country': "New Zealand"},
])

prompt = weave.EasyPrompt(name='travel_agent')
prompt.append("You are an expert travel consultant.", role="system")
prompt.append("Tell me the capital of {country} and about five kid-friendly attractions there.")


prompts = prompt.bind_rows(dataset)
assert prompts[2][1]["content"] == "Tell me the capital of Canada and about five kid-friendly attractions there."
```

You can extend this into an [Evaluation](./evaluations.md):

```python
import asyncio

import openai
import weave

weave.init("intro-example")

# Create a dataset
dataset = weave.Dataset(name='countries', rows=[
{'id': '0', 'country': "Argentina", 'capital': "Buenos Aires"},
{'id': '1', 'country': "Belize", 'capital': "Belmopan"},
{'id': '2', 'country': "Canada", 'capital': "Ottawa"},
{'id': '3', 'country': "New Zealand", 'capital': "Wellington"},
])

# Create a prompt
prompt = weave.EasyPrompt(name='travel_agent')
prompt.append("You are an expert travel consultant.", role="system")
prompt.append("Tell me the capital of {country} and about five kid-friendly attractions there.")

# Create a model, combining a prompt with model configuration
class TravelAgentModel(weave.Model):

model_name: str
prompt: weave.EasyPrompt

@weave.op
async def predict(self, country: str) -> dict:
client = openai.AsyncClient()

response = await client.chat.completions.create(
model=self.model_name,
messages=self.prompt(country=country),
)
result = response.choices[0].message.content
if result is None:
raise ValueError("No response from model")
return result

# Define and run the evaluation
@weave.op
def mentions_capital_scorer(capital: str, model_output: str) -> dict:
return {'correct': capital in model_output}

model = TravelAgentModel(model_name="gpt-4o-mini", prompt=prompt)
evaluation = weave.Evaluation(
dataset=dataset,
scorers=[mentions_capital_scorer],
)
asyncio.run(evaluation.evaluate(model))

```
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading

0 comments on commit a9ad0b2

Please sign in to comment.