Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding modalities #91

Merged
merged 27 commits into from
Oct 22, 2024
Merged

Adding modalities #91

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
6edcdd2
Adding modalities
Sep 2, 2024
0acfc8d
Testing markdown changes.
Sep 2, 2024
d0d5c75
Testing markdown changes.
Sep 2, 2024
9508dc5
Testing markdown changes.
Sep 2, 2024
aad6607
Testing markdown changes.
Sep 2, 2024
60a318a
Testing markdown changes.
Sep 2, 2024
d187100
Changing the embedding table.
Sep 2, 2024
c88506b
Adding the command model tables.
Sep 2, 2024
b673768
Adding the rerank model tables.
Sep 2, 2024
8205162
Updating the embeddings doc.
Sep 2, 2024
ee9cda3
Adding imports to the new code snippet, so as to avoid excoriation fr…
Sep 2, 2024
6d6b332
Fixing a typo.
Sep 2, 2024
52f5249
added release notes
mahjongmen Sep 11, 2024
23ff0dc
added the guide for multimodal embeddings
mahjongmen Sep 11, 2024
f0084b2
add IA
mahjongmen Sep 11, 2024
eafdd9d
add IA to yamlv2
mahjongmen Sep 11, 2024
f596c47
Merge branch 'main' into multi-modal-embeddings
trentfowlercohere Sep 23, 2024
5fa0220
rerank table not rendering.
Sep 23, 2024
d9ad24b
Adding modalities columns.
Sep 23, 2024
a48c164
changed the date for multimodal embed and added webp and gif support …
mahjongmen Oct 1, 2024
0aeb5a2
added to release notes
mahjongmen Oct 1, 2024
dc1981f
Adding information to the AWS Sagemaker page.
Oct 9, 2024
06b1cb1
Adding information to the Azure page.
Oct 9, 2024
ea3bcbd
Fixing a link, a typo.
Oct 22, 2024
65e4737
Merge branch 'main' into multi-modal-embeddings
trentfowlercohere Oct 22, 2024
e1c37de
Shortened metadescription.
Oct 22, 2024
7b39a4a
Adding a description to the changelog.
Oct 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added fern/assets/images/multi-modal-guide-header.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
36 changes: 36 additions & 0 deletions fern/pages/changelog/2024-10-22-Embed-v3-is-multimodal.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
---
title: "Embed v3.0 Models are now Multimodal"
slug: "changelog/embed-v3-is-multimodal"
createdAt: "Tues Oct 22 2024 05:30:00 (MST)"
hidden: false
description: >-
Launch of multimodal embeddings for our Embed models, plus some code to help get started.
---

Today we’re announcing updates to our embed-v3.0 family of models. These models now have the ability to process images into embeddings. There is no change to existing text capabilities which means there is no need to re-embed texts you have already processed with our `embed-v3.0` models.

In the rest of these release notes, we’ll provide more details about technical enhancements, new features, and new pricing.

## Technical Details
### API Changes:
The Embed API has two major changes:
- Introduced a new `input_type` called `image`
- Introduced a new parameter called `images`

Example request on how to process

```Text cURL
POST https://api.cohere.ai/v1/embed
{
"model": "embed-multilingual-v3.0",
"input_type": "image",
"embedding_types": ["float"],
"images": [enc_img]
}
```
### Restrictions:
- The API only accepts images in the base format of the following: `png`, `jpeg`,`Webp`, and `gif`
- Image embeddings currently does not support batching so the max images sent per request is 1
- The maximum image sizez is `5mb`
- The `images` parameter only accepts a base64 encoded image formatted as a Data Url

2 changes: 2 additions & 0 deletions fern/pages/deployment-options/cohere-on-aws/amazon-sagemaker-setup-guide.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,8 @@ result = co.embed(
print(result)
```

Note that we've released multimodal embeddings models that are able to handle images in addition to text. Find [more information here](https://docs.cohere.com/docs/multimodal-embeddings).

## Text Generation

You can use this code to invoke Cohere's Command models on Amazon SageMaker:
Expand Down
1 change: 1 addition & 0 deletions fern/pages/deployment-options/cohere-on-microsoft-azure.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,7 @@ Though this section is called "Text Generation", it's worth pointing out that th
We expose two routes for Embed v3 - English and Embed v3 - Multilingual inference:

- `v1/embeddings` adheres to the Azure AI Generative Messages API schema;
- Use `v1/images/embeddings` if you want to use one of our [multimodal embeddings models](/docs/multimodal-embeddings).
- ` v1/embed` supports Cohere's native API schema.

You can find more information about Azure's API [here](https://learn.microsoft.com/en-us/azure/ai-studio/how-to/deploy-models-cohere-embed#embed-api-reference-for-cohere-embed-models-deployed-as-a-service).
Expand Down
22 changes: 11 additions & 11 deletions fern/pages/models/cohere-embed.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,22 +14,22 @@ Embed models can be used to generate embeddings from text or classify it based o

## English Models

| Latest Model | Description | Dimensions | Max Tokens (Context Length) | Similarity Metric | Endpoints |
|-----------------------------|------------------------------------------------------------------------------------------------------|------------|-----------------------------|-------------------|-------------------------------------------------------------------------------------------|
| `embed-english-v3.0` | A model that allows for text to be classified or turned into embeddings. English only. | 1024 | 512 | Cosine Similarity, Dot Product Similarity, Euclidean Distance | [Embed](/reference/embed), <br/>[Embed Jobs](/reference/embed-jobs) |
| `embed-english-light-v3.0` | A smaller, faster version of `embed-english-v3.0`. Almost as capable, but a lot faster. English only.| 384 | 512 | Cosine Similarity, Dot Product Similarity, Euclidean Distance | [Embed](/reference/embed), <br/>[Embed Jobs](/reference/embed-jobs) |
| `embed-english-v2.0` | Our older embeddings model that allows for text to be classified or turned into embeddings. English only | 4096 | 512 | Cosine Similarity | [Classify](/reference/classify), [Embed](/reference/embed) |
| `embed-english-light-v2.0` | A smaller, faster version of embed-english-v2.0. Almost as capable, but a lot faster. English only. | 1024 | 512 | Cosine Similarity | [Classify](/reference/classify), [Embed](/reference/embed) |
| Latest Model | Description | Modality | Dimensions | Max Tokens (Context Length) | Similarity Metric | Endpoints |
|-----------------------------|----------------------------------------------------------------------------------------------------------|--------------|------------|-----------------------------|---------------------------------------------------------------|------------------------------------------------------------------------------------|
| `embed-english-v3.0` | A model that allows for text to be classified or turned into embeddings. English only. | Text, Images | 1024 | 512 | Cosine Similarity, Dot Product Similarity, Euclidean Distance | [Embed](/reference/embed), <br/>[Embed Jobs](/reference/embed-jobs) |
| `embed-english-light-v3.0` | A smaller, faster version of `embed-english-v3.0`. Almost as capable, but a lot faster. English only. | Text, Images | 384 | 512 | Cosine Similarity, Dot Product Similarity, Euclidean Distance | [Embed](/reference/embed), <br/>[Embed Jobs](/reference/embed-jobs) |
| `embed-english-v2.0` | Our older embeddings model that allows for text to be classified or turned into embeddings. English only.| Text | 4096 | 512 | Cosine Similarity | [Classify](/reference/classify), [Embed](/reference/embed) |
| `embed-english-light-v2.0` | A smaller, faster version of embed-english-v2.0. Almost as capable, but a lot faster. English only. | Text | 1024 | 512 | Cosine Similarity | [Classify](/reference/classify), [Embed](/reference/embed) |



## Multi-Lingual Models

| Latest Model | Description | Dimensions | Max Tokens (Context Length) | Similarity Metric | Endpoints |
|----------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------|------------|-----------------------------|-------------------------|---------------------------------------------------------------------------------------------------|
| `embed-multilingual-v3.0` | Provides multilingual classification and embedding support. [See supported languages here.](/docs/supported-languages) | 1024 | 512 | Cosine Similarity, Dot Product Similarity, Euclidean Distance | [Embed](/reference/embed), [Embed Jobs](/reference/embed-jobs) |
| `embed-multilingual-light-v3.0` | A smaller, faster version of `embed-multilingual-v3.0`. Almost as capable, but a lot faster. Supports multiple languages. | 384 | 512 | Cosine Similarity, Dot Product Similarity, Euclidean Distance | [Embed](/reference/embed), <br/>[Embed Jobs](/reference/embed-jobs) |
| `embed-multilingual-v2.0` | Provides multilingual classification and embedding support. [See supported languages here.](/docs/supported-languages) | 768 | 256 | Dot Product Similarity | [Classify](/reference/classify), [Embed](/reference/embed) |
| Latest Model | Description | Modality | Dimensions | Max Tokens (Context Length) | Similarity Metric | Endpoints |
|----------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------|-------------------|------------|-----------------------------|---------------------------------------------------------------------|------------------------------------------------------------------------|
| `embed-multilingual-v3.0` | Provides multilingual classification and embedding support. [See supported languages here.](/docs/supported-languages) | Text, Images | 1024 | 512 | Cosine Similarity, Dot Product Similarity, Euclidean Distance | [Embed](/reference/embed), [Embed Jobs](/reference/embed-jobs) |
| `embed-multilingual-light-v3.0` | A smaller, faster version of `embed-multilingual-v3.0`. Almost as capable, but a lot faster. Supports multiple languages. | Text, Images | 384 | 512 | Cosine Similarity, Dot Product Similarity, Euclidean Distance | [Embed](/reference/embed), <br/>[Embed Jobs](/reference/embed-jobs) |
| `embed-multilingual-v2.0` | Provides multilingual classification and embedding support. [See supported languages here.](/docs/supported-languages) | Text | 768 | 256 | Dot Product Similarity | [Classify](/reference/classify), [Embed](/reference/embed) |



Expand Down
Loading
Loading