Skip to content

Commit

Permalink
Merge pull request #42 from predictionguard/jacob/token-embedding
Browse files Browse the repository at this point in the history 
adding token inputs to embeddings and tokenize endpoint docs
  • Loading branch information
@jmansdorfer
jmansdorfer authored Oct 29, 2024
2 parents f2e5caa + de1b7d2 commit bb4b46f
Show file tree
Hide file tree
Showing 3 changed files with 268 additions and 9 deletions.
12 changes: 8 additions & 4 deletions fern/docs.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,8 @@ navigation:
path: ./docs/pages/reference/completions.mdx
- page: Embeddings
path: ./docs/pages/reference/embeddings.mdx
- page: Tokenize
path: ./docs/pages/reference/tokenize.mdx
- page: Factuality
path: ./docs/pages/reference/factuality.mdx
- page: Injection
Expand All @@ -118,8 +120,8 @@ navigation:
path: ./docs/pages/reference/PII.mdx
- page: Toxicity
path: ./docs/pages/reference/toxicity.mdx
- page: Translate
path: ./docs/pages/reference/translate.mdx
# - page: Translate
# path: ./docs/pages/reference/translate.mdx

- tab: sdk
layout:
Expand All @@ -141,6 +143,8 @@ navigation:
path: ./docs/pages/reference/completions.mdx
- page: Embeddings
path: ./docs/pages/reference/embeddings.mdx
- page: Tokenize
path: ./docs/pages/reference/tokenize.mdx
- page: Factuality
path: ./docs/pages/reference/factuality.mdx
- page: Injection
Expand All @@ -149,8 +153,8 @@ navigation:
path: ./docs/pages/reference/PII.mdx
- page: Toxicity
path: ./docs/pages/reference/toxicity.mdx
- page: Translate
path: ./docs/pages/reference/translate.mdx
# - page: Translate
# path: ./docs/pages/reference/translate.mdx

navbar-links:
- type: secondary
Expand Down
120 changes: 120 additions & 0 deletions fern/docs/pages/reference/tokenize.mdx
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
---
title: Tokenize
---

You can get privacy-conserving text and image embeddings from
[available models](/options/enumerations) using a call to the `/embeddings` REST
API endpoint or any of the official SDKs (Python, Go, Rust, JS, or cURL).

## Generate Embeddings

To generate embeddings, you can use the following code examples. Depending on your
preference or requirements, select the appropriate method for your application.
This functionality accepts text and image inputs, and supports batching multiple
inputs. The Go REST API only supports images that are input as a base64 encoded
string, whereas the python client supports image files, image urls, and images
encoded in a base64 string.

<CodeBlocks>
<CodeBlock title="Python">
```python
import os
import json

from predictionguard import PredictionGuard

# Set your Prediction Guard token as an environmental variable.
os.environ["PREDICTIONGUARD_API_KEY"] = "<api key>"

client = PredictionGuard()

response = client.tokenize.create(
model="Hermes-2-Pro-Llama-3-8B",
input="This is a tokenize example."
)

print(json.dumps(
response,
sort_keys=True,
indent=4,
separators=(',', ': ')
))
```
</CodeBlock>

<CodeBlock title="Go">
```go
package main

import (
"context"
"fmt"
"log"
"os"
"time"

"github.com/predictionguard/go-client"
)

func main() {
if err := run(); err != nil {
log.Fatalln(err)
}
```
</CodeBlock>
<CodeBlock title="Rust">
```rust
extern crate prediction_guard as pg_client;
```
</CodeBlock>
<CodeBlock title="NodeJS">
```js
import * as pg from 'predictionguard';

const client = new pg.Client('https://api.predictionguard.com', process.env.PREDICTIONGUARD_API_KEY);

```
</CodeBlock>
<CodeBlock title="cURL">
```bash
curl -i -X POST https://api.predictionguard.com/tokenize \
-H "Authorization: Bearer ${PREDICTIONGUARD_API_KEY}" \
-H "Content-Type: application/json" \
-d '{
"model": "Hermes-2-Pro-Llama-3-8B",
"input": "This is a tokenize example."
}'
```
</CodeBlock>
</CodeBlocks>
The output will look something like this.
```json
{
"id": "token-61dd03d2-76b4-4261-9543-736d82f86e8c",
"object": "tokens",
"created": 1729884596,
"model": "Hermes-2-Pro-Mistral-7B",
"tokens": [
{
"id": 1,
"start": 0,
"stop": 0,
"text": ""
},
{
"id": 851,
"start": 0,
"stop": 4,
"text": "This"
},
...
```
This approach presents a straightforward way for readers to choose and apply the
code example that best suits their needs for generating text completions using
either Python, Go, Rust, JS, or cURL.
145 changes: 140 additions & 5 deletions fern/openapi/Prediction-Guard-Prediction-Guard-API-1.0-resolved.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -759,12 +759,19 @@ paths:
oneOf:
- type: string
description: A string of text to be embedded.
- type: integer
description: A token to be embedded.
- type: array
description: An array of tokens to be embedded.
items:
type: integer
description: A token to be embedded.
- type: object
description: Use this option for multimodal input.
properties:
text:
type: string
description: The text to vectorize. You can choose to provide this or not.
description: The text to embed. You can choose to provide this or not.
image:
type: string
description: The base64 encoding of an image. You can choose to provide this or not.
Expand Down Expand Up @@ -877,6 +884,134 @@ paths:
security:
- bearerAuth: []

# /tokenize:
# post:
# summary: Tokenize
# description: Generates tokens for a string using a model's tokenizer.
# requestBody:
# content:
# application/json:
# schema:
# required:
# - model
# - input
# type: object
# properties:
# model:
# type: string
# description: The name of the model to use for tokenization.
# input:
# type: string
# description: The text to tokenize.
# examples:
# basic:
# summary: A basic example of using the API.
# value:
# model: "Hermes-3-Llama-3.1-8B"
# input: "Tell me a joke."
# responses:
# "403":
# description: Failed auth response.
# content:
# application/json:
# schema:
# type: object
# properties:
# error:
# type: string
# description: Description of the error.
# example:
# error: "api understands the request but refuses to authorize it"
# "400":
# description: General error response.
# content:
# application/json:
# schema:
# type: object
# properties:
# error:
# type: string
# description: Description of the error.
# example:
# error: "required fields are missing"
# "200":
# description: Successful response.
# content:
# application/json:
# schema:
# type: object
# properties:
# id:
# type: string
# description: Unique ID for the token.
# object:
# type: string
# description: Type of object (tokens).
# created:
# type: integer
# description: Timestamp of when the tokens were generated.
# model:
# type: string
# description: The name of the model used.
# tokens:
# type: array
# description: An array of token data.
# items:
# type: object
# properties:
# id:
# type: integer
# description: The converted token.
# start:
# type: integer
# description: The index the token starts at.
# stop:
# type: integer
# description: The index the token ends at.
# text:
# type: string
# description: The token as text.
# example:
# id: "token-ab046fcf-945f-421c-b9f0-1c75ff355203"
# object: "tokens"
# created: 1729871708
# model: "Hermes-2-Pro-Mistral-7B"
# tokens:
# - id: 1
# start: 0
# stop: 0
# text: ""
# - id: 851
# start: 0
# stop: 4
# text: "This"
# - id: 349
# start: 4
# stop: 7
# text: " is"
# - id: 264
# start: 7
# stop: 9
# text: " a"
# - id: 6029
# start: 9
# stop: 15
# text: " token"
# - id: 653
# start: 15
# stop: 18
# text: "ize"
# - id: 2757
# start: 18
# stop: 26
# text: " example"
# - id: 28723
# start: 26
# stop: 27
# text: "."
# security:
# - bearerAuth: []

/factuality:
post:
summary: Factuality
Expand Down Expand Up @@ -1253,7 +1388,7 @@ paths:
properties:
text:
type: string
description: The text to check for toxicity.
description: The text to translate.
source_lang:
type: string
description: The source language of the text.
Expand Down Expand Up @@ -1305,13 +1440,13 @@ paths:
properties:
id:
type: string
description: Unique ID for the chat completion.
description: Unique ID for the translation.
object:
type: string
description: Type of object (chat completion).
description: Type of object (translation).
created:
type: integer
description: Timestamp of when the chat completion was created.
description: Timestamp of when the translation was created.
best_translation:
type: string
description: The best translation of the input text.
Expand Down

0 comments on commit bb4b46f

Please sign in to comment.