diff --git a/docs/docs/guides/tracking/feedback.md b/docs/docs/guides/tracking/feedback.md index 81022d31bc9..68e27776d44 100644 --- a/docs/docs/guides/tracking/feedback.md +++ b/docs/docs/guides/tracking/feedback.md @@ -3,33 +3,77 @@ import TabItem from '@theme/TabItem'; # Feedback -Evaluating LLM applications automatically is challenging. Teams often rely on direct user feedback, particularly from domain experts, who assess the content quality using simple indicators such as thumbs up or down. Developers also actively identify and resolve content issues. +Efficiently evaluating LLM applications requires robust tooling to collect and analyze feedback. Weave provides an integrated feedback system, allowing users to provide call feedback directly through the UI or programmatically via the SDK. Various feedback types are supported, including emoji reactions, textual comments, and structured data, enabling teams to: -Weave's feedback feature enables users to provide feedback directly within the Weave UI or through the API. You can add emoji reactions, textual notes, and structured data to calls. This feedback helps compile evaluation datasets, monitor application performance, and collect examples for advanced tasks like fine-tuning. +- Build evaluation datasets for performance monitoring. +- Identify and resolve LLM content issues effectively. +- Gather examples for advanced tasks like fine-tuning. -## View and Add Feedback within UI +This guide covers how to use Weave’s feedback functionality in both the UI and SDK, query and manage feedback, and use human annotations for detailed evaluations. -Reactions and notes are displayed in a column in the calls grid. Hovering over these indicators provides more detail. Use these buttons to add reactions or notes. +- [Provide feedback in the UI](#provide-feedback-in-the-ui) +- [Provide feedback via the SDK](#provide-feedback-via-the-sdk) +- [Add human annotations](#add-human-annotations) -![Screenshot of calls grid with feedback column](imgs/feedback_calls.png) +## Provide feedback in the UI -View and edit feedback in the header of the call details page. +In the Weave UI, you can add and view feedback [from the call details page](#from-the-call-details-page) or [using the icons](#use-the-icons). -![Screenshot of feedback controls in call details header](imgs/feedback_call_header.png) +### From the call details page -View the feedback for a call on the "Feedback" tab of the call details page. +1. In the sidebar, navigate to **Traces**. +2. Find the row for the call that you want to add feedback to. +3. Open the call details page. +4. Select the **Feedback** column for the call. +5. Add, view, or delete feedback: + - *[Add and view feedback using the icons](#use-the-icons)* located in the upper right corner of the call details feedback view. + - *View and delete feedback from the call details feedback table.* Delete feedback by clicking the trashcan icon in the rightmost column of the appropriate feedback row. ![Screenshot of Feedback tab in call details](imgs/feedback_tab.png) -Access copy-and-paste examples on the "Use" tab of the call details page to manipulate the feedback for that call using the SDK. +### Use the icons + +You can add or remove a reaction, and add a note using the icons that are located in both the call table and individual call details pages. + +- *Call table*: Located in **Feedback** column in the appropriate row in the call table. +- *Call details page*: Located in the upper right corner of each call details page. + +To add a reaction: + 1. Click the emoji icon. + 2. Add a thumbs up, thumbs down, or click the **+** icon for more emojis. + +To remove a reaction: + 1. Hover over the emoji reaction you want to remove. + 2. Click the reaction to remove it. + +> You can also delete feedback from the [**Feedback** column on the call details page.](#from-the-call-details-page). + +To add a comment: + 1. Click the comment bubble icon. + 2. In the text box, add your note. + 3. To save the note, press the **Enter** key. You can add additional notes. + +![Screenshot of calls grid with feedback column](imgs/feedback_calls.png) + +## Provide feedback via the SDK + +> You can find SDK usage examples for feedback in the UI under the **Use** tab in the call details page. -![Screenshot of Use tab in call details](imgs/feedback_use.png) +You can use the Weave SDK to programmatically add, remove, and query feedback on calls. -## SDK +### Query a project's feedback -Use the Weave SDK to programmatically add, remove, and query feedback on calls. +You can query the feedback for your Weave project using the SDK. The SDK supports the following feedback query operations: -### Querying a project's feedback +- `client.get_feedback()`: Returns all feedback in a project. +- `client.get_feedback("")`: Return a specific feedback object specified by `` as a collection. +- `client.get_feedback(reaction="")`: Returns all feedback objects for a specific reaction type. + +You can also get additional information for each feedback object in `client.get_feedback()`: +- `id`: The feedback object ID. +- `created_at`: The creation time information for the feedback object. +- `feedback_type`: The type of feedback (reaction, note, custom). +- `payload`: The feedback payload @@ -41,14 +85,13 @@ Use the Weave SDK to programmatically add, remove, and query feedback on calls. all_feedback = client.get_feedback() # Fetch a specific feedback object by id. - # Note that the API still returns a collection, which is expected - # to contain zero or one item(s). + # The API returns a collection, which is expected to contain at most one item. one_feedback = client.get_feedback("")[0] # Find all feedback objects with a specific reaction. You can specify offset and limit. thumbs_up = client.get_feedback(reaction="👍", limit=10) - # After retrieval you can view the details of individual feedback objects. + # After retrieval, view the details of individual feedback objects. for f in client.get_feedback(): print(f.id) print(f.created_at) @@ -64,7 +107,13 @@ Use the Weave SDK to programmatically add, remove, and query feedback on calls. -### Adding feedback to a call +### Add feedback to a call + +You can add feedback to a call using the call's UUID. To use the UUID to get a particular call, [retrieve it during or after call execution](#retrieve-the-call-uuid). The SDK supports the following operations for adding feedback to a call: + +- `call.feedback.add_reaction("")`: Add one of the supported `` (emojis), such as 👍. +- `call.feedback.add_note("")`: Add a note. +- `call.feedback.add("