Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adding rerank docs and removing reference sections #48

Merged
merged 4 commits into from
Nov 21, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# JetBrains Folder
.idea
106 changes: 54 additions & 52 deletions fern/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,8 @@ navigation:
path: ./docs/pages/options/lvms.mdx
- page: Embedding Models
path: ./docs/pages/options/embedding_models.mdx
- page: Reranker Models
path: ./docs/pages/options/reranker_models.mdx
- page: Enumerations
path: ./docs/pages/options/enumerations.mdx
- page: Prompt Formatting
Expand Down Expand Up @@ -102,32 +104,32 @@ navigation:
contents:
- page: Enumerations for API
path: ./docs/pages/options/enumerations.mdx
- section: SDK Reference
contents:
- page: Chat
path: ./docs/pages/reference/chat.mdx
- page: Chat SSE (Streaming)
path: ./docs/pages/reference/chat_sse.mdx
- page: Chat Vision
path: ./docs/pages/reference/chat_vision.mdx
- page: Completions
path: ./docs/pages/reference/completions.mdx
- page: Embeddings
path: ./docs/pages/reference/embeddings.mdx
# - page: Tokenize
# path: ./docs/pages/reference/tokenize.mdx
- page: Factuality
path: ./docs/pages/reference/factuality.mdx
- page: Injection
path: ./docs/pages/reference/injection.mdx
- page: PII
path: ./docs/pages/reference/PII.mdx
- page: Toxicity
path: ./docs/pages/reference/toxicity.mdx
- page: Translate
path: ./docs/pages/reference/translate.mdx
# - page: Models
# path: ./docs/pages/reference/models.mdx
# - section: SDK Reference
# contents:
# - page: Chat
# path: ./docs/pages/reference/chat.mdx
# - page: Chat SSE (Streaming)
# path: ./docs/pages/reference/chat_sse.mdx
# - page: Chat Vision
# path: ./docs/pages/reference/chat_vision.mdx
# - page: Completions
# path: ./docs/pages/reference/completions.mdx
# - page: Embeddings
# path: ./docs/pages/reference/embeddings.mdx
# - page: Tokenize
# path: ./docs/pages/reference/tokenize.mdx
# - page: Factuality
# path: ./docs/pages/reference/factuality.mdx
# - page: Injection
# path: ./docs/pages/reference/injection.mdx
# - page: PII
# path: ./docs/pages/reference/PII.mdx
# - page: Toxicity
# path: ./docs/pages/reference/toxicity.mdx
# - page: Translate
# path: ./docs/pages/reference/translate.mdx
# - page: Models
# path: ./docs/pages/reference/models.mdx

- tab: sdk
layout:
Expand All @@ -137,32 +139,32 @@ navigation:
path: ./docs/pages/sdks.mdx
- page: Quick Start
path: ./docs/pages/sdkquickstart.mdx
- section: SDK Reference
contents:
- page: Chat
path: ./docs/pages/reference/chat.mdx
- page: Chat SSE (Streaming)
path: ./docs/pages/reference/chat_sse.mdx
- page: Chat Vision
path: ./docs/pages/reference/chat_vision.mdx
- page: Completions
path: ./docs/pages/reference/completions.mdx
- page: Embeddings
path: ./docs/pages/reference/embeddings.mdx
# - page: Tokenize
# path: ./docs/pages/reference/tokenize.mdx
- page: Factuality
path: ./docs/pages/reference/factuality.mdx
- page: Injection
path: ./docs/pages/reference/injection.mdx
- page: PII
path: ./docs/pages/reference/PII.mdx
- page: Toxicity
path: ./docs/pages/reference/toxicity.mdx
- page: Translate
path: ./docs/pages/reference/translate.mdx
# - page: Models
# path: ./docs/pages/reference/models.mdx
# - section: SDK Reference
# contents:
# - page: Chat
# path: ./docs/pages/reference/chat.mdx
# - page: Chat SSE (Streaming)
# path: ./docs/pages/reference/chat_sse.mdx
# - page: Chat Vision
# path: ./docs/pages/reference/chat_vision.mdx
# - page: Completions
# path: ./docs/pages/reference/completions.mdx
# - page: Embeddings
# path: ./docs/pages/reference/embeddings.mdx
# - page: Tokenize
# path: ./docs/pages/reference/tokenize.mdx
# - page: Factuality
# path: ./docs/pages/reference/factuality.mdx
# - page: Injection
# path: ./docs/pages/reference/injection.mdx
# - page: PII
# path: ./docs/pages/reference/PII.mdx
# - page: Toxicity
# path: ./docs/pages/reference/toxicity.mdx
# - page: Translate
# path: ./docs/pages/reference/translate.mdx
# - page: Models
# path: ./docs/pages/reference/models.mdx

redirects:
- source: "/options/models"
Expand Down
36 changes: 36 additions & 0 deletions fern/docs/pages/options/reranker_models.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
---
title: Reranker Models
description: Reliable, future proof AI predictions
slug: options/reranker_models
---

This page provides information on the Reranker models that are available in the Prediction Guard API.
These models are designed for semantically ranking text, and are used in the `/rerank` endpoint.

## Models

| Model Name | Type | Use Case | Context Length | More Info |
| ---------------------- | ------------ | ----------------------------------------------- | -------------- | ------------------------------------------------------|
| bge-reranker-v2-m3 | Reranker | Used for semantically ranking queries | 512 | [link](/options/reranker_models#bge-reranker-v2-m3) |

## Model Descriptions

### bge-reranker-v2-m3

BAAI/bge-reranker-v2-m3 is a lightweight, multilingual reranker model designed for efficient and accurate text retrieval tasks.

**Type:** Reranker\
**Use Case:** Used for Semantically Ranking Documents

https://huggingface.co/BAAI/bge-reranker-v2-m3

Unlike embedding models, rerankers take a query and document (or passage) as input and directly output a similarity score.
The output relevance score can be converted to a float value in the range [0,1] using a sigmoid function.

Key Features:
• Multilingual Support: Excels across multiple languages with strong cross-lingual capabilities.
• Efficiency: Lightweight design ensures fast inference and easy deployment.
• Versatility: Supports a range of use cases and scenarios.

This model offers an excellent balance between performance and deployment efficiency,
making it a powerful choice for a wide range of text retrieval scenarios.
115 changes: 114 additions & 1 deletion fern/openapi/Prediction-Guard-Prediction-Guard-API-1.0-resolved.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -817,6 +817,119 @@ paths:
security:
- bearerAuth: []

/rerank:
post:
summary: Rerank
description: Rerank sorts text inputs by semantic relevance to a specified query.
requestBody:
content:
application/json:
schema:
required:
- model
- query
- documents
type: object
properties:
model:
type: string
description: The model to use for reranking.
query:
type: string
description: The query to rank against.
documents:
type: array
description: Array of documents to rank.
items:
type: string
description: Documents to rank.
return_documents:
type: boolean
description: Boolean setting whether to return documents in output.
examples:
basic:
summary: A text example of using the API.
value:
model: "bge-reranker-v2-m3"
query: "What is Deep Learning?"
documents:
- "Deep Learning is pizza."
- "Deep Learning is not pizza."
return_documents: true
responses:
"403":
description: Failed auth response.
content:
application/json:
schema:
type: object
properties:
error:
type: string
description: Description of the error.
example:
error: "api understands the request but refuses to authorize it"
"400":
description: General error response.
content:
application/json:
schema:
type: object
properties:
error:
type: string
description: Description of the error.
example:
error: "required fields are missing"
"200":
description: Successful response.
content:
application/json:
schema:
type: object
properties:
id:
type: string
description: Unique ID for the rerank response.
object:
type: string
description: Type of object (list).
created:
type: integer
description: Timestamp of when the rerank response was created.
model:
type: string
description: The rerank model used.
results:
type: array
description: The set of rankings.
items:
type: object
properties:
index:
type: integer
description: The index position in the collection.
relevance_score:
type: number
description: The relevance ranking score.
text:
type: string
description: The returned document
example:
id: "rerank-39bf1cf3-66cd-43d5-9185-b52baf742414"
object: "list"
created: 1732039610
model: "bge-reranker-v2-m3"
results:
- index: 0
relevance_score: 0.25128007
text: "Deep Learning is not pizza."
- index: 1
relevance_score: 0.5100618
text: "Deep Learning is not pizza."
security:
- bearerAuth: []

/tokenize:
post:
summary: Tokenize
Expand Down Expand Up @@ -1127,7 +1240,7 @@ paths:

/PII:
post:
summary: Replace Personal Information
summary: PII
description: Replace personal information such as names, SSNs, and emails in a given text.
requestBody:
content:
Expand Down
Loading