Skip to content

Commit

Permalink
update structured outputs docs
Browse files Browse the repository at this point in the history
  • Loading branch information
@mrmer1
mrmer1 committed Dec 9, 2024
1 parent 10bf030 commit cfcd743
Show file tree
Hide file tree
Showing 5 changed files with 43 additions and 50 deletions.
14 changes: 7 additions & 7 deletions fern/pages/text-generation/structured-outputs.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ updatedAt: "Tue Jun 11 2024 02:43:00 GMT+0000 (Coordinated Universal Time)"

## Overview

Structured Outputs is a feature for forcing the LLM's output to follow a specified format 100% of the time. This increases the reliability of LLM in enterprise applications where downstream applications expect the LLM output to be correctly formatted. By forcing the model to follow a structured schema, hallucinated fields and entries in structured data can be reliably eliminated.
Structured Outputs is a feature that forces the LLM’s response to strictly follow a schema specified by the user. When Structured Outputs is turned on, the LLM will generate structured data that follows the desired schema, provided by the user, 100% of the time. This increases the reliability of LLMs in enterprise applications where downstream applications expect the LLM output to be correctly formatted. With Structured Outputs, hallucinated fields and entries in structured data can be reliably eliminated.

Compatible models:
- Command R 08 2024
Expand Down Expand Up @@ -74,8 +74,8 @@ By setting the `response_format` type to `"json_object"` in the Chat API, the ou
When using `{ "type": "json_object" }` your `message` should always explicitly instruct the model to generate a JSON (eg: _"Generate a JSON ..."_) . Otherwise the model may end up getting stuck generating an infinite stream of characters and eventually run out of context length.
</Info>

<Note title="Experimental">
This feature is currently not supported in RAG mode.
<Note title="Note">
This feature is currently not supported in [RAG](https://docs.cohere.com/v1/docs/retrieval-augmented-generation-rag) mode.
</Note>

#### JSON Schema mode
Expand Down Expand Up @@ -123,10 +123,6 @@ In this schema, we defined three keys ("title," "author," "publication_year") an
```

<Info title="Important">
Note: Each schema provided will incur a latency overhead required for processing the schema. This is only applicable for the first few requests.
</Info>

## Specifying a schema

### Generating nested objects
Expand Down Expand Up @@ -162,3 +158,7 @@ We do not support the entirety of the [JSON Schema specification](https://json-s
- Others:
- `uniqueItems`
- `additionalProperties`

<Info title="Important">
Note: Using Structured Outputs (in JSON Schema mode) will incur a latency overhead required for processing the structured schema. This increase in latency only applies for the first few requests, since the schema is cached afterwards.
</Info>
12 changes: 0 additions & 12 deletions fern/pages/text-generation/tools/tool-use.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -328,18 +328,6 @@ outputs = [{"number": 2343}] # Not Great
outputs = [{"sum": 2343}] # Better
```

## What's Next?

Here, we'll preview some of the functionality we plan on adding in the coming months.

### Cohere-hosted Tools

The model can currently handle any tool provided by the developer. That having been said, Cohere has implemented some pre-defined tools that users can leverage out-of-the-box.

Specifically we're going to roll out a **Python interpreter** tool and a **Web search** tool.

Please [reach out](mailto:[email protected]) to join the beta.

## Getting started

Check out [this notebook](https://github.com/cohere-ai/notebooks/blob/main/notebooks/agents/Vanilla_Tool_Use.ipynb) for a worked-out examples.
Expand Down
40 changes: 22 additions & 18 deletions fern/pages/v2/text-generation/structured-outputs.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ updatedAt: "Tue Jun 11 2024 02:43:00 GMT+0000 (Coordinated Universal Time)"

## Overview

Structured Outputs is a feature for forcing the LLM's output to follow a specified format 100% of the time. This increases the reliability of LLM in enterprise applications where downstream applications expect the LLM output to be correctly formatted. By forcing the model to follow a structured schema, hallucinated fields and entries in structured data can be reliably eliminated.
Structured Outputs is a feature that forces the LLM’s response to strictly follow a schema specified by the user. When Structured Outputs is turned on, the LLM will generate structured data that follows the desired schema, provided by the user, 100% of the time. This increases the reliability of LLMs in enterprise applications where downstream applications expect the LLM output to be correctly formatted. With Structured Outputs, hallucinated fields and entries in structured data can be reliably eliminated.

Compatible models:
- Command R 08 2024
Expand All @@ -27,7 +27,7 @@ Compatible models:

There are two ways to use Structured Outputs:
- **Structured Outputs (JSON)**. This is primarily used in text generation use cases.
- **Structured Outputs (Tools)**. This is primarily used in tool use and agents use cases via function calling.
- **Structured Outputs (Tools)**. Structured Outputs (Tools). This is primarily used in [tool use (or function calling)](https://docs.cohere.com/docs/tool-use) and [agents](https://docs.cohere.com/docs/multi-step-tool-use) use cases.

<Note title="API Compatibility">
Structured Outputs with Tools are only supported in [Chat API V2](https://docs.cohere.com/reference/chat#request.body.strict_tools) via the `strict_tools` parameter. This parameter is not supported in Chat API V1.
Expand Down Expand Up @@ -82,7 +82,7 @@ When using `{ "type": "json_object" }` your `message` should always explicitly
</Info>

<Note title="Note">
This feature is currently not supported in RAG mode.
This feature is currently not supported in [RAG](https://docs.cohere.com/docs/retrieval-augmented-generation-rag) mode.
</Note>

#### JSON Schema mode
Expand Down Expand Up @@ -135,24 +135,20 @@ In this schema, we defined three keys ("title," "author," "publication_year") an
```

<Info title="Important">
Note: Each schema provided (in both JSON and Tools modes) will incur a latency overhead required for processing the schema. This is only applicable for the first few requests.
</Info>

### Structured Outputs (Tools)
When you use the Chat API with `tools`, setting the `strict_tools` parameter to `true` will enforce that every generated tool call follows the specified tool schema.
When you use the Chat API with `tools` (see [tool use](https://docs.cohere.com/docs/tool-use) and [agents](https://docs.cohere.com/docs/multi-step-tool-use)), setting the `strict_tools` parameter to `True` will enforce that the tool calls generated by the mode strictly adhere to the tool descriptions you provided.

<Note title="Experimental">
`strict_tools` is currently an experimental parameter. We’ll be iterating on this feature and are looking for feedback. Share your experience with us in the `#api-discussions` channel on [discord](https://discord.gg/co-mmunity) or via [email](mailto:[email protected]).
</Note>
Concretely, this means:
- No hallucinated tool names
- No hallucinated tool parameters
- Every `required` parameter is included in the tool call
- All parameters produce the requested data types

With `strict_tools` enabled, the API will ensure that the tool names and tool parameters are generated according to the tool definitions. This eliminates tool name and parameter hallucinations, ensures that each parameter matches the specified data type, and that all required parameters are included in the model response.

Additionally, this results in faster development. You don’t need to spend a lot of time prompt engineering the model to avoid hallucinations.

In the example below, we create a tool that can retrieve weather data for a given location. The tool is called`get_weather` which contains a parameter called `location`. We then invoke the Chat API with `strict_tools` set to `true` to ensure that the generated tool calls always include the correct function and parameter names.

When the `strict_tools` parameter is set to `true`, you can define a maximum of 200 fields across all tools being passed to an API call.
In the example below, we create a tool that can retrieve weather data for a given location. The tool is called `get_weather` which contains a parameter called `location`. We then invoke the Chat API with `strict_tools` set to `true` to ensure that the generated tool calls always include the correct function and parameter names.

```python PYTHON {24}
tools = [
Expand Down Expand Up @@ -183,18 +179,22 @@ response = co.chat(model="command-r-plus-08-2024",
print(response.message.tool_calls)
```

#### Important notes on using `strict_tools`
- This parameter is only supported in Chat API V2 via the strict_tools parameter (not API V1).
- You must specify at least one `required` parameter. Tools with only optional parameters are not supported in this mode.
- You can define a maximum of 200 fields across all tools in a single Chat API call.

<Info title="Important">
When `strict_tools` is enabled, tool definitions that have optional parameters must specify at least one `required` parameter as well. Tools with only optional parameters are not supported in this mode.
</Info>
<Note title="Experimental">
`strict_tools` is currently an experimental parameter. We’ll be iterating on this feature and are looking for feedback. Share your experience with us in the `#api-discussions` channel on [discord](https://discord.gg/co-mmunity) or via [email](mailto:[email protected]).
</Note>

### When to Use Structured Outputs (JSON) vs. Structured Outputs (Tools)

Structured Outputs (JSON) are ideal for text generation use cases where you want to format the model's responses to users in a specific way.

For example, when building a travel planner application, you might want the LLM to generate itineraries in a specific JSON format, allowing the application to use the output in the other parts of the application.

Structured Outputs (Tools) are ideal for function calling, tool use or agents use cases where you need the model to interact with external data or services. For instance, you can grant the model access to functions that interact with databases or other APIs.
Structured Outputs (Tools) are ideal for [tool use (or function calling)](https://docs.cohere.com/docs/tool-use) and [agents](https://docs.cohere.com/docs/multi-step-tool-use) use cases where you need the model to interact with external data or services. For instance, you can grant the model access to functions that interact with databases or other APIs.

In summary, opt for:
- Structured Outputs (JSON) when you need the model's response to follow a specific structure.
Expand Down Expand Up @@ -236,3 +236,7 @@ We do not support the entirety of the [JSON Schema specification](https://json-s
- Others:
- `uniqueItems`
- `additionalProperties`

<Info title="Important">
Note: Using Structured Outputs (in both JSON Schema and Tools modes) will incur a latency overhead required for processing the structured schema. This increase in latency only applies for the first few requests, since the schema is cached afterwards.
</Info>
13 changes: 13 additions & 0 deletions fern/pages/v2/text-generation/tools/multi-step-tool-use.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,19 @@ Tool use is a natural extension of retrieval augmented generation (RAG). RAG is

Tool use pushes this further, allowing Cohere models to go far beyond information retrieval, interact with search engines, APIs, functions, databases, and many other tools.

## Structured Outputs (Tools)

Setting the `strict_tools` parameter to `True` will enforce each tool call to follow the specified tool schema. To learn more about this feature, visit the [Structured Outputs documentation](https://docs.cohere.com/v2/docs/structured-outputs).

Note that `strict_tools` is currently an experimental feature.

```python PYTHON {4}
response = co.chat(model="command-r-plus-08-2024",
messages=messages,
tools=tools,
strict_tools=True)
```

## A Further Example With Multiple Tools

This section provides another example of multi-step tool use, this time with multiple tools. The notebook for this example can be [found here](https://github.com/cohere-ai/notebooks/blob/main/notebooks/agents/Multi_Step_Tool_Use_Spotify_v2.ipynb).
Expand Down
14 changes: 1 addition & 13 deletions fern/pages/v2/text-generation/tools/tool-use.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -364,7 +364,7 @@ Start: 207 | End: 232 | Text: 'Tablet: $300, 25 in stock'

## Structured Outputs (Tools)

Setting the `strict_tools` parameter to True will enforce each tool call to follow the specified tool schema. To learn more about this feature, visit the [Structured Outputs documentation](https://docs.cohere.com/v2/docs/structured-outputs).
Setting the `strict_tools` parameter to `True` will enforce each tool call to follow the specified tool schema. To learn more about this feature, visit the [Structured Outputs documentation](https://docs.cohere.com/v2/docs/structured-outputs).

Note that `strict_tools` is currently an experimental feature.

Expand All @@ -386,18 +386,6 @@ outputs = [{"number": 2343}] # Not Great
outputs = [{"sum": 2343}] # Better
```

## What's Next?

Here, we'll preview some of the functionality we plan on adding in the coming months.

### Cohere-hosted Tools

The model can currently handle any tool provided by the developer. That having been said, Cohere has implemented some pre-defined tools that users can leverage out-of-the-box.

Specifically we're going to roll out a **Python interpreter** tool and a **Web search** tool.

Please [reach out](mailto:[email protected]) to join the beta.

## Getting started

Check out [this notebook](https://github.com/cohere-ai/notebooks/blob/main/notebooks/agents/Vanilla_Tool_Use_v2.ipynb) for a worked-out examples.
Expand Down

0 comments on commit cfcd743

Please sign in to comment.