Skip to content

Commit

Permalink
add support for o1 models
Browse files Browse the repository at this point in the history
Added support for OpenAI o1-mini and o1-preview models by mapping them to the `o200k_base` encoding. This update ensures compatibility with the latest tiktoken changes as introduced in version 0.8.0 of the official library.

Fixes compatibility issues with models not yet included in tiktoken-php.
  • Loading branch information
iamarsenibragimov authored Oct 11, 2024
1 parent 780b874 commit 4431360
Showing 1 changed file with 1 addition and 0 deletions.
1 change: 1 addition & 0 deletions src/EncoderProvider.php
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ final class EncoderProvider implements ResetInterface
],
];
private const MODEL_PREFIX_TO_ENCODING = [

Check failure on line 49 in src/EncoderProvider.php

View workflow job for this annotation

GitHub Actions / Static Analysis

ParseError

src/EncoderProvider.php:49:46: ParseError: Array element cannot be empty (see https://psalm.dev/173)
"o1-": "o200k_base",

Check failure on line 50 in src/EncoderProvider.php

View workflow job for this annotation

GitHub Actions / Static Analysis

ParseError

src/EncoderProvider.php:50:14: ParseError: Syntax error, unexpected ':', expecting ',' or ']' or ')' on line 50 (see https://psalm.dev/173)
'gpt-4o-' => 'o200k_base',
'gpt-4-' => 'cl100k_base',
'gpt-3.5-turbo-' => 'cl100k_base',
Expand Down

0 comments on commit 4431360

Please sign in to comment.