Merge branch 'master' into griffin/nuke-project

dev_docs/BaseObjectClasses.md → dev_docs/BuiltinObjectClasses.md

@@ -1,4 +1,4 @@
-# BaseObjectClasses
+# BuiltinObjectClasses
 
 ## Refresher on Objects and object storage
 
@@ -79,11 +79,11 @@ While many Weave Objects are free-form and user-defined, there is often a need f
 
 Here's how to define and use a validated base object:
 
-1. **Define your schema** (in `weave/trace_server/interface/base_object_classes/your_schema.py`):
+1. **Define your schema** (in `weave/trace_server/interface/builtin_object_classes/your_schema.py`):
 
 ```python
 from pydantic import BaseModel
-from weave.trace_server.interface.base_object_classes import base_object_def
+from weave.trace_server.interface.builtin_object_classes import base_object_def
 
 class NestedConfig(BaseModel):
     setting_a: int
@@ -116,7 +116,7 @@ curl -X POST 'https://trace.wandb.ai/obj/create' \
       "project_id": "user/project",
       "object_id": "my_config",
       "val": {...},
-      "set_base_object_class": "MyConfig"
+      "object_class": "MyConfig"
     }
   }'
 
@@ -154,38 +154,38 @@ Run `make synchronize-base-object-schemas` to ensure the frontend TypeScript typ
 
 ### Architecture Flow
 
-1. Define your schema in a python file in the `weave/trace_server/interface/base_object_classes/test_only_example.py` directory. See `weave/trace_server/interface/base_object_classes/test_only_example.py` as an example.
-2. Make sure to register your schemas in `weave/trace_server/interface/base_object_classes/base_object_registry.py` by calling `register_base_object`.
+1. Define your schema in a python file in the `weave/trace_server/interface/builtin_object_classes/test_only_example.py` directory. See `weave/trace_server/interface/builtin_object_classes/test_only_example.py` as an example.
+2. Make sure to register your schemas in `weave/trace_server/interface/builtin_object_classes/builtin_object_registry.py` by calling `register_base_object`.
 3. Run `make synchronize-base-object-schemas` to generate the frontend types.
-    * The first step (`make generate_base_object_schemas`) will run `weave/scripts/generate_base_object_schemas.py` to generate a JSON schema in `weave/trace_server/interface/base_object_classes/generated/generated_base_object_class_schemas.json`.
-    * The second step (yarn `generate-schemas`) will read this file and use it to generate the frontend types located in `weave-js/src/components/PagePanelComponents/Home/Browse3/pages/wfReactInterface/generatedBaseObjectClasses.zod.ts`.
+    * The first step (`make generate_base_object_schemas`) will run `weave/scripts/generate_base_object_schemas.py` to generate a JSON schema in `weave/trace_server/interface/builtin_object_classes/generated/generated_builtin_object_class_schemas.json`.
+    * The second step (yarn `generate-schemas`) will read this file and use it to generate the frontend types located in `weave-js/src/components/PagePanelComponents/Home/Browse3/pages/wfReactInterface/generatedBuiltinObjectClasses.zod.ts`.
 4. Now, each use case uses different parts:
     1. `Python Writing`. Users can directly import these classes and use them as normal Pydantic models, which get published with `weave.publish`. The python client correct builds the requisite payload.
     2. `Python Reading`. Users can `weave.ref().get()` and the weave python SDK will return the instance with the correct type. Note: we do some special handling such that the returned object is not a WeaveObject, but literally the exact pydantic class.
-    3. `HTTP Writing`. In cases where the client/user does not want to add the special type information, users can publish base objects by setting the `set_base_object_class` setting on `POST obj/create` to the name of the class. The weave server will validate the object against the schema, update the metadata fields, and store the object.
+    3. `HTTP Writing`. In cases where the client/user does not want to add the special type information, users can publish builtin objects (set of weave.Objects provided by Weave) by setting the `builtin_object_class` setting on `POST obj/create` to the name of the class. The weave server will validate the object against the schema, update the metadata fields, and store the object.
     4. `HTTP Reading`. When querying for objects, the server will return the object with the correct type if the `base_object_class` metadata field is set.
-    5. `Frontend`. The frontend will read the zod schema from `weave-js/src/components/PagePanelComponents/Home/Browse3/pages/wfReactInterface/generatedBaseObjectClasses.zod.ts` and use that to provide compile time type safety when using `useBaseObjectInstances` and runtime type safety when using `useCreateBaseObjectInstance`.
+    5. `Frontend`. The frontend will read the zod schema from `weave-js/src/components/PagePanelComponents/Home/Browse3/pages/wfReactInterface/generatedBuiltinObjectClasses.zod.ts` and use that to provide compile time type safety when using `useBaseObjectInstances` and runtime type safety when using `useCreateBaseObjectInstance`.
 * Note: it is critical that all techniques produce the same digest for the same data - which is tested in the tests. This way versions are not thrashed by different clients/users.
 
 ```mermaid
 graph TD
     subgraph Schema Definition
         F["weave/trace_server/interface/<br>base_object_classes/your_schema.py"] --> |defines| P[Pydantic BaseObject]
-        P --> |register_base_object| R["base_object_registry.py"]
+        P --> |register_base_object| R["builtin_object_registry.py"]
     end
 
     subgraph Schema Generation
         M["make synchronize-base-object-schemas"] --> G["make generate_base_object_schemas"]
         G --> |runs| S["weave/scripts/<br>generate_base_object_schemas.py"]
         R --> |import registered classes| S
-        S --> |generates| J["generated_base_object_class_schemas.json"]
-        M --> |yarn generate-schemas| Z["generatedBaseObjectClasses.zod.ts"]
+        S --> |generates| J["generated_builtin_object_class_schemas.json"]
+        M --> |yarn generate-schemas| Z["generatedBuiltinObjectClasses.zod.ts"]
         J --> Z
     end
 
     subgraph "Trace Server"
         subgraph "HTTP API"
-            R --> |validates using| HW["POST obj/create<br>set_base_object_class"]
+            R --> |validates using| HW["POST obj/create<br>object_class"]
             HW --> DB[(Weave Object Store)]
             HR["POST objs/query<br>base_object_classes"] --> |Filters base_object_class| DB
         end
@@ -203,7 +203,7 @@ graph TD
         Z --> |import| UBI["useBaseObjectInstances"]
         Z --> |import| UCI["useCreateBaseObjectInstance"]
         UBI --> |Filters base_object_class| HR
-        UCI --> |set_base_object_class| HW
+        UCI --> |object_class| HW
         UI[React UI] --> UBI
         UI --> UCI
     end

docs/docs/guides/evaluation/scorers.md

@@ -224,9 +224,9 @@ In Weave, Scorers are used to evaluate AI outputs and return evaluation metrics.
     ```
 
     ### Mapping Column Names with `columnMapping`
-    :::warning
+    :::important
 
-    In TypeScript, this feature is currently on the `Evaluation` object, not individual scorers!
+    In TypeScript, this feature is currently on the `Evaluation` object, not individual scorers.
 
     :::
 
@@ -455,7 +455,7 @@ In Weave, Scorers are used to evaluate AI outputs and return evaluation metrics.
     from weave.scorers import OpenAIModerationScorer
     from openai import OpenAI
 
-    oai_client = OpenAI(api_key=...) # initialize your LLM client here
+    oai_client = OpenAI() # initialize your LLM client here
 
     scorer = OpenAIModerationScorer(
         client=oai_client,

docs/docs/guides/integrations/local_models.md

@@ -14,7 +14,6 @@ First and most important, is the `base_url` change during the `openai.OpenAI()`
 
 ```python
 client = openai.OpenAI(
-    api_key='fake',
     base_url="http://localhost:1234",
 )
 ```

docs/docs/guides/integrations/notdiamond.md

@@ -68,7 +68,6 @@ preference_id = train_router(
     response_column="actual",
     language="en",
     maximize=True,
-    api_key=api_key,
 )
 ```
 

docs/docs/guides/tools/playground.md

@@ -57,84 +57,119 @@ There are two ways to access the Playground:
 
 ## Select an LLM
 
-You can switch the LLM using the dropdown menu in the top left. Currently, the available models are:
+You can switch the LLM using the dropdown menu in the top left. The available models from various providers are listed below:
 
-- gpt-40-mini
-- claude-3-5-sonnet-20240620
+- [AI21](#ai21)
+- [Amazon](#amazon)
+- [Anthropic](#anthropic)
+- [Cohere](#cohere)
+- [Google](#google)
+- [Groq](#groq)
+- [Meta](#meta)
+- [Mistral](#mistral)
+- [OpenAI](#openai)
+- [X.AI](#xai)
+
+
+### AI21
+- ai21.j2-mid-v1
+- ai21.j2-ultra-v1
+
+### Amazon
+- amazon.nova-lite
+- amazon.nova-micro
+- amazon.nova-pro
+- amazon.titan-text-express-v1
+- amazon.titan-text-lite-v1
+
+### Anthropic
+- anthropic.claude-3-5-sonnet-20240620-v1:0
+- anthropic.claude-3-haiku-20240307-v1:0
+- anthropic.claude-3-opus-20240229-v1:0
+- anthropic.claude-3-sonnet-20240229-v1:0
+- anthropic.claude-instant-v1
+- anthropic.claude-v2
+- anthropic.claude-v2:1
 - claude-3-5-sonnet-20241022
 - claude-3-haiku-20240307
 - claude-3-opus-20240229
 - claude-3-sonnet-20240229
+
+### Cohere
+- cohere.command-light-text-v14
+- cohere.command-r-plus-v1:0
+- cohere.command-r-v1:0
+- cohere.command-text-v14
+
+### Google
+- gemini/gemini-1.5-flash
 - gemini/gemini-1.5-flash-001
 - gemini/gemini-1.5-flash-002
 - gemini/gemini-1.5-flash-8b-exp-0827
 - gemini/gemini-1.5-flash-8b-exp-0924
 - gemini/gemini-1.5-flash-exp-0827
 - gemini/gemini-1.5-flash-latest
-- gemini/gemini-1.5-flash
+- gemini/gemini-1.5-pro
 - gemini/gemini-1.5-pro-001
 - gemini/gemini-1.5-pro-002
 - gemini/gemini-1.5-pro-exp-0801
 - gemini/gemini-1.5-pro-exp-0827
 - gemini/gemini-1.5-pro-latest
-- gemini/gemini-1.5-pro
 - gemini/gemini-pro
+
+### Groq
+- groq/gemma-7b-it
+- groq/gemma2-9b-it
+- groq/llama-3.1-70b-versatile
+- groq/llama-3.1-8b-instant
+- groq/llama3-70b-8192
+- groq/llama3-8b-8192
+- groq/llama3-groq-70b-8192-tool-use-preview
+- groq/llama3-groq-8b-8192-tool-use-preview
+- groq/mixtral-8x7b-32768
+
+### Meta
+- meta.llama2-13b-chat-v1
+- meta.llama2-70b-chat-v1
+- meta.llama3-1-405b-instruct-v1:0
+- meta.llama3-1-70b-instruct-v1:0
+- meta.llama3-1-8b-instruct-v1:0
+- meta.llama3-70b-instruct-v1:0
+- meta.llama3-8b-instruct-v1:0
+
+### Mistral
+- mistral.mistral-7b-instruct-v0:2
+- mistral.mistral-large-2402-v1:0
+- mistral.mistral-large-2407-v1:0
+- mistral.mixtral-8x7b-instruct-v0:1
+
+### OpenAI
+- gpt-3.5-turbo
 - gpt-3.5-turbo-0125
 - gpt-3.5-turbo-1106
 - gpt-3.5-turbo-16k
-- gpt-3.5-turbo
+- gpt-4
 - gpt-4-0125-preview
 - gpt-4-0314
 - gpt-4-0613
 - gpt-4-1106-preview
 - gpt-4-32k-0314
+- gpt-4-turbo
 - gpt-4-turbo-2024-04-09
 - gpt-4-turbo-preview
-- gpt-4-turbo
-- gpt-4
 - gpt-40-2024-05-13
 - gpt-40-2024-08-06
+- gpt-40-mini
 - gpt-40-mini-2024-07-18
 - gpt-4o
-- groq/gemma-7b-it
-- groq/gemma2-9b-it
-- groq/llama-3.1-70b-versatile
-- groq/llama-3.1-8b-instant
-- groq/llama3-70b-8192
-- groq/llama3-8b-8192
-- groq/llama3-groq-70b-8192-tool-use-preview
-- groq/llama3-groq-8b-8192-tool-use-preview
-- groq/mixtral-8x7b-32768
-- o1-mini-2024-09-12
 - o1-mini
-- o1-preview-2024-09-12
+- o1-mini-2024-09-12
 - o1-preview
-- ai21.j2-mid-v1
-- ai21.j2-ultra-v1
-- amazon.titan-text-lite-v1
-- amazon.titan-text-express-v1
-- mistral.mistral-7b-instruct-v0:2
-- mistral.mixtral-8x7b-instruct-v0:1
-- mistral.mistral-large-2402-v1:0
-- mistral.mistral-large-2407-v1:0
-- anthropic.claude-3-sonnet-20240229-v1:0
-- anthropic.claude-3-5-sonnet-20240620-v1:0
-- anthropic.claude-3-haiku-20240307-v1:0
-- anthropic.claude-3-opus-20240229-v1:0
-- anthropic.claude-v2
-- anthropic.claude-v2:1
-- anthropic.claude-instant-v1
-- cohere.command-text-v14
-- cohere.command-light-text-v14
-- cohere.command-r-plus-v1:0
-- cohere.command-r-v1:0
-- meta.llama2-13b-chat-v1
-- meta.llama2-70b-chat-v1
-- meta.llama3-8b-instruct-v1:0
-- meta.llama3-70b-instruct-v1:0
-- meta.llama3-1-8b-instruct-v1:0
-- meta.llama3-1-70b-instruct-v1:0
-- meta.llama3-1-405b-instruct-v1:0
+- o1-preview-2024-09-12
+
+### X.AI
+- xai/grok-beta
+
 
 ## Adjust LLM parameters
 

docs/docs/guides/tracking/ops.md

@@ -116,6 +116,39 @@ A Weave op is a versioned function that automatically logs all calls.
   </TabItem>
 </Tabs>
 
+## Control sampling rate
+
+<Tabs groupId="programming-language">
+  <TabItem value="python" label="Python" default>
+    You can control how frequently an op's calls are traced by setting the `tracing_sample_rate` parameter in the `@weave.op` decorator. This is useful for high-frequency ops where you only need to trace a subset of calls.
+
+     Note that sampling rates are only applied to root calls. If an op has a sample rate, but is called by another op first, then that sampling rate will be ignored.
+
+    ```python
+    @weave.op(tracing_sample_rate=0.1)  # Only trace ~10% of calls
+    def high_frequency_op(x: int) -> int:
+        return x + 1
+
+    @weave.op(tracing_sample_rate=1.0)  # Always trace (default)
+    def always_traced_op(x: int) -> int:
+        return x + 1
+    ```
+
+    When an op's call is not sampled:
+    - The function executes normally
+    - No trace data is sent to Weave
+    - Child ops are also not traced for that call
+
+    The sampling rate must be between 0.0 and 1.0 inclusive.
+
+  </TabItem>
+  <TabItem value="typescript" label="TypeScript">
+    ```plaintext
+    This feature is not available in TypeScript yet. Stay tuned!
+    ```
+  </TabItem>
+</Tabs>
+
 ### Control call link output 
 
 If you want to suppress the printing of call links during logging, you can use the `WEAVE_PRINT_CALL_LINK` environment variable to `false`. This can be useful if you want to reduce  output verbosity and reduce clutter in your logs.

docs/docs/quickstart.md

@@ -50,7 +50,7 @@ _In this example, we're using openai so you will need to add an OpenAI [API key]
     import weave
     from openai import OpenAI
 
-    client = OpenAI(api_key="...")
+    client = OpenAI()
 
     # Weave will track the inputs, outputs and code of this function
     # highlight-next-line