From cfcd743ce3e380f727319e247e9505e5e51380a8 Mon Sep 17 00:00:00 2001 From: mrmer1 Date: Mon, 9 Dec 2024 18:32:39 +0800 Subject: [PATCH 1/2] update structured outputs docs --- .../text-generation/structured-outputs.mdx | 14 +++---- fern/pages/text-generation/tools/tool-use.mdx | 12 ------ .../v2/text-generation/structured-outputs.mdx | 40 ++++++++++--------- .../tools/multi-step-tool-use.mdx | 13 ++++++ .../v2/text-generation/tools/tool-use.mdx | 14 +------ 5 files changed, 43 insertions(+), 50 deletions(-) diff --git a/fern/pages/text-generation/structured-outputs.mdx b/fern/pages/text-generation/structured-outputs.mdx index bab8d112..0d038bd6 100644 --- a/fern/pages/text-generation/structured-outputs.mdx +++ b/fern/pages/text-generation/structured-outputs.mdx @@ -14,7 +14,7 @@ updatedAt: "Tue Jun 11 2024 02:43:00 GMT+0000 (Coordinated Universal Time)" ## Overview -Structured Outputs is a feature for forcing the LLM's output to follow a specified format 100% of the time. This increases the reliability of LLM in enterprise applications where downstream applications expect the LLM output to be correctly formatted. By forcing the model to follow a structured schema, hallucinated fields and entries in structured data can be reliably eliminated. +Structured Outputs is a feature that forces the LLM’s response to strictly follow a schema specified by the user. When Structured Outputs is turned on, the LLM will generate structured data that follows the desired schema, provided by the user, 100% of the time. This increases the reliability of LLMs in enterprise applications where downstream applications expect the LLM output to be correctly formatted. With Structured Outputs, hallucinated fields and entries in structured data can be reliably eliminated. Compatible models: - Command R 08 2024 @@ -74,8 +74,8 @@ By setting the `response_format` type to `"json_object"` in the Chat API, the ou When using `{ "type": "json_object" }` your `message` should always explicitly instruct the model to generate a JSON (eg: _"Generate a JSON ..."_) . Otherwise the model may end up getting stuck generating an infinite stream of characters and eventually run out of context length. - -This feature is currently not supported in RAG mode. + +This feature is currently not supported in [RAG](https://docs.cohere.com/v1/docs/retrieval-augmented-generation-rag) mode. #### JSON Schema mode @@ -123,10 +123,6 @@ In this schema, we defined three keys ("title," "author," "publication_year") an ``` - -Note: Each schema provided will incur a latency overhead required for processing the schema. This is only applicable for the first few requests. - - ## Specifying a schema ### Generating nested objects @@ -162,3 +158,7 @@ We do not support the entirety of the [JSON Schema specification](https://json-s - Others: - `uniqueItems` - `additionalProperties` + + +Note: Using Structured Outputs (in JSON Schema mode) will incur a latency overhead required for processing the structured schema. This increase in latency only applies for the first few requests, since the schema is cached afterwards. + \ No newline at end of file diff --git a/fern/pages/text-generation/tools/tool-use.mdx b/fern/pages/text-generation/tools/tool-use.mdx index e85aafa0..576c6044 100644 --- a/fern/pages/text-generation/tools/tool-use.mdx +++ b/fern/pages/text-generation/tools/tool-use.mdx @@ -328,18 +328,6 @@ outputs = [{"number": 2343}] # Not Great outputs = [{"sum": 2343}] # Better ``` -## What's Next? - -Here, we'll preview some of the functionality we plan on adding in the coming months. - -### Cohere-hosted Tools - -The model can currently handle any tool provided by the developer. That having been said, Cohere has implemented some pre-defined tools that users can leverage out-of-the-box. - -Specifically we're going to roll out a **Python interpreter** tool and a **Web search** tool. - -Please [reach out](mailto:MAXIMEVOISIN@COHERE.COM) to join the beta. - ## Getting started Check out [this notebook](https://github.com/cohere-ai/notebooks/blob/main/notebooks/agents/Vanilla_Tool_Use.ipynb) for a worked-out examples. diff --git a/fern/pages/v2/text-generation/structured-outputs.mdx b/fern/pages/v2/text-generation/structured-outputs.mdx index 2865aaa3..fe9db8c5 100644 --- a/fern/pages/v2/text-generation/structured-outputs.mdx +++ b/fern/pages/v2/text-generation/structured-outputs.mdx @@ -15,7 +15,7 @@ updatedAt: "Tue Jun 11 2024 02:43:00 GMT+0000 (Coordinated Universal Time)" ## Overview -Structured Outputs is a feature for forcing the LLM's output to follow a specified format 100% of the time. This increases the reliability of LLM in enterprise applications where downstream applications expect the LLM output to be correctly formatted. By forcing the model to follow a structured schema, hallucinated fields and entries in structured data can be reliably eliminated. +Structured Outputs is a feature that forces the LLM’s response to strictly follow a schema specified by the user. When Structured Outputs is turned on, the LLM will generate structured data that follows the desired schema, provided by the user, 100% of the time. This increases the reliability of LLMs in enterprise applications where downstream applications expect the LLM output to be correctly formatted. With Structured Outputs, hallucinated fields and entries in structured data can be reliably eliminated. Compatible models: - Command R 08 2024 @@ -27,7 +27,7 @@ Compatible models: There are two ways to use Structured Outputs: - **Structured Outputs (JSON)**. This is primarily used in text generation use cases. -- **Structured Outputs (Tools)**. This is primarily used in tool use and agents use cases via function calling. +- **Structured Outputs (Tools)**. Structured Outputs (Tools). This is primarily used in [tool use (or function calling)](https://docs.cohere.com/docs/tool-use) and [agents](https://docs.cohere.com/docs/multi-step-tool-use) use cases. Structured Outputs with Tools are only supported in [Chat API V2](https://docs.cohere.com/reference/chat#request.body.strict_tools) via the `strict_tools` parameter. This parameter is not supported in Chat API V1. @@ -82,7 +82,7 @@ When using `{ "type": "json_object" }` your `message` should always explicitly -This feature is currently not supported in RAG mode. +This feature is currently not supported in [RAG](https://docs.cohere.com/docs/retrieval-augmented-generation-rag) mode. #### JSON Schema mode @@ -135,24 +135,20 @@ In this schema, we defined three keys ("title," "author," "publication_year") an ``` - -Note: Each schema provided (in both JSON and Tools modes) will incur a latency overhead required for processing the schema. This is only applicable for the first few requests. - - ### Structured Outputs (Tools) -When you use the Chat API with `tools`, setting the `strict_tools` parameter to `true` will enforce that every generated tool call follows the specified tool schema. +When you use the Chat API with `tools` (see [tool use](https://docs.cohere.com/docs/tool-use) and [agents](https://docs.cohere.com/docs/multi-step-tool-use)), setting the `strict_tools` parameter to `True` will enforce that the tool calls generated by the mode strictly adhere to the tool descriptions you provided. - -`strict_tools` is currently an experimental parameter. We’ll be iterating on this feature and are looking for feedback. Share your experience with us in the `#api-discussions` channel on [discord](https://discord.gg/co-mmunity) or via [email](mailto:support@cohere.com). - +Concretely, this means: +- No hallucinated tool names +- No hallucinated tool parameters +- Every `required` parameter is included in the tool call +- All parameters produce the requested data types With `strict_tools` enabled, the API will ensure that the tool names and tool parameters are generated according to the tool definitions. This eliminates tool name and parameter hallucinations, ensures that each parameter matches the specified data type, and that all required parameters are included in the model response. Additionally, this results in faster development. You don’t need to spend a lot of time prompt engineering the model to avoid hallucinations. -In the example below, we create a tool that can retrieve weather data for a given location. The tool is called`get_weather` which contains a parameter called `location`. We then invoke the Chat API with `strict_tools` set to `true` to ensure that the generated tool calls always include the correct function and parameter names. - -When the `strict_tools` parameter is set to `true`, you can define a maximum of 200 fields across all tools being passed to an API call. +In the example below, we create a tool that can retrieve weather data for a given location. The tool is called `get_weather` which contains a parameter called `location`. We then invoke the Chat API with `strict_tools` set to `true` to ensure that the generated tool calls always include the correct function and parameter names. ```python PYTHON {24} tools = [ @@ -183,10 +179,14 @@ response = co.chat(model="command-r-plus-08-2024", print(response.message.tool_calls) ``` +#### Important notes on using `strict_tools` +- This parameter is only supported in Chat API V2 via the strict_tools parameter (not API V1). +- You must specify at least one `required` parameter. Tools with only optional parameters are not supported in this mode. +- You can define a maximum of 200 fields across all tools in a single Chat API call. - -When `strict_tools` is enabled, tool definitions that have optional parameters must specify at least one `required` parameter as well. Tools with only optional parameters are not supported in this mode. - + +`strict_tools` is currently an experimental parameter. We’ll be iterating on this feature and are looking for feedback. Share your experience with us in the `#api-discussions` channel on [discord](https://discord.gg/co-mmunity) or via [email](mailto:tools_feedback@cohere.com). + ### When to Use Structured Outputs (JSON) vs. Structured Outputs (Tools) @@ -194,7 +194,7 @@ Structured Outputs (JSON) are ideal for text generation use cases where you want For example, when building a travel planner application, you might want the LLM to generate itineraries in a specific JSON format, allowing the application to use the output in the other parts of the application. -Structured Outputs (Tools) are ideal for function calling, tool use or agents use cases where you need the model to interact with external data or services. For instance, you can grant the model access to functions that interact with databases or other APIs. +Structured Outputs (Tools) are ideal for [tool use (or function calling)](https://docs.cohere.com/docs/tool-use) and [agents](https://docs.cohere.com/docs/multi-step-tool-use) use cases where you need the model to interact with external data or services. For instance, you can grant the model access to functions that interact with databases or other APIs. In summary, opt for: - Structured Outputs (JSON) when you need the model's response to follow a specific structure. @@ -236,3 +236,7 @@ We do not support the entirety of the [JSON Schema specification](https://json-s - Others: - `uniqueItems` - `additionalProperties` + + +Note: Using Structured Outputs (in both JSON Schema and Tools modes) will incur a latency overhead required for processing the structured schema. This increase in latency only applies for the first few requests, since the schema is cached afterwards. + \ No newline at end of file diff --git a/fern/pages/v2/text-generation/tools/multi-step-tool-use.mdx b/fern/pages/v2/text-generation/tools/multi-step-tool-use.mdx index 2b0d9b66..e2c8c27b 100644 --- a/fern/pages/v2/text-generation/tools/multi-step-tool-use.mdx +++ b/fern/pages/v2/text-generation/tools/multi-step-tool-use.mdx @@ -164,6 +164,19 @@ Tool use is a natural extension of retrieval augmented generation (RAG). RAG is Tool use pushes this further, allowing Cohere models to go far beyond information retrieval, interact with search engines, APIs, functions, databases, and many other tools. +## Structured Outputs (Tools) + +Setting the `strict_tools` parameter to `True` will enforce each tool call to follow the specified tool schema. To learn more about this feature, visit the [Structured Outputs documentation](https://docs.cohere.com/v2/docs/structured-outputs). + +Note that `strict_tools` is currently an experimental feature. + +```python PYTHON {4} +response = co.chat(model="command-r-plus-08-2024", + messages=messages, + tools=tools, + strict_tools=True) +``` + ## A Further Example With Multiple Tools This section provides another example of multi-step tool use, this time with multiple tools. The notebook for this example can be [found here](https://github.com/cohere-ai/notebooks/blob/main/notebooks/agents/Multi_Step_Tool_Use_Spotify_v2.ipynb). diff --git a/fern/pages/v2/text-generation/tools/tool-use.mdx b/fern/pages/v2/text-generation/tools/tool-use.mdx index 1b7b8798..062abb64 100644 --- a/fern/pages/v2/text-generation/tools/tool-use.mdx +++ b/fern/pages/v2/text-generation/tools/tool-use.mdx @@ -364,7 +364,7 @@ Start: 207 | End: 232 | Text: 'Tablet: $300, 25 in stock' ## Structured Outputs (Tools) -Setting the `strict_tools` parameter to True will enforce each tool call to follow the specified tool schema. To learn more about this feature, visit the [Structured Outputs documentation](https://docs.cohere.com/v2/docs/structured-outputs). +Setting the `strict_tools` parameter to `True` will enforce each tool call to follow the specified tool schema. To learn more about this feature, visit the [Structured Outputs documentation](https://docs.cohere.com/v2/docs/structured-outputs). Note that `strict_tools` is currently an experimental feature. @@ -386,18 +386,6 @@ outputs = [{"number": 2343}] # Not Great outputs = [{"sum": 2343}] # Better ``` -## What's Next? - -Here, we'll preview some of the functionality we plan on adding in the coming months. - -### Cohere-hosted Tools - -The model can currently handle any tool provided by the developer. That having been said, Cohere has implemented some pre-defined tools that users can leverage out-of-the-box. - -Specifically we're going to roll out a **Python interpreter** tool and a **Web search** tool. - -Please [reach out](mailto:MAXIMEVOISIN@COHERE.COM) to join the beta. - ## Getting started Check out [this notebook](https://github.com/cohere-ai/notebooks/blob/main/notebooks/agents/Vanilla_Tool_Use_v2.ipynb) for a worked-out examples. From 60237e76d2f99c4181d8920564f6a7dff1777da3 Mon Sep 17 00:00:00 2001 From: mrmer1 Date: Mon, 9 Dec 2024 18:39:39 +0800 Subject: [PATCH 2/2] add notes on nesting --- fern/pages/text-generation/structured-outputs.mdx | 2 +- fern/pages/v2/text-generation/structured-outputs.mdx | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/fern/pages/text-generation/structured-outputs.mdx b/fern/pages/text-generation/structured-outputs.mdx index 0d038bd6..9075e23f 100644 --- a/fern/pages/text-generation/structured-outputs.mdx +++ b/fern/pages/text-generation/structured-outputs.mdx @@ -127,7 +127,7 @@ In this schema, we defined three keys ("title," "author," "publication_year") an ### Generating nested objects -The model can be configured to output objects with up to 5 levels of nesting. When a `schema` is specified, there are no limitations on the levels of nesting. +In JSON Schema mode, there are no limitations on the levels of nesting. However, in JSON mode (no schema specified), nesting is limited to 5 levels. ### Schema constraints diff --git a/fern/pages/v2/text-generation/structured-outputs.mdx b/fern/pages/v2/text-generation/structured-outputs.mdx index fe9db8c5..e9887e0a 100644 --- a/fern/pages/v2/text-generation/structured-outputs.mdx +++ b/fern/pages/v2/text-generation/structured-outputs.mdx @@ -205,7 +205,7 @@ In summary, opt for: ### Generating nested objects -The model can be configured to output objects with up to 5 levels of nesting. When a `schema` is specified, there are no limitations on the levels of nesting. +In JSON Schema mode, there are no limitations on the levels of nesting. However, in JSON mode (no schema specified), nesting is limited to 5 levels. ### Schema constraints