Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OpenAI Codex tokens limit - identify and help the user workaround the limit by reducing the file set #14

Open
RobotSail opened this issue Mar 9, 2022 · 5 comments
Open

OpenAI Codex tokens limit - identify and help the user workaround the limit by reducing the file set #14

RobotSail opened this issue Mar 9, 2022 · 5 comments
Assignees
@RobotSail
Labels
enhancement New feature or request
Milestone
TBD

Comments

@RobotSail
Copy link
Collaborator

With the OpenAI Codex model, we are only able to use 4096 tokens in total (prompt + generation), so we need to either develop a scheme to chain multiple completions together (maybe some sort of stitching mechanism), or error out when the #tokens(input files) > (MAX_TOKENS - #tokens(prompt)) / 2
@RobotSail RobotSail added enhancement New feature or request help wanted Extra attention is needed question Further information is requested labels Mar 9, 2022
@guymguym guymguym changed the title Handle completion overflow OpenAI Codex tokens limit - identify and help the user workaround the limit by reducing the file set Apr 26, 2022
@guymguym
Copy link
Contributor

@RobotSail assigning to you to check for options using a tokenizer lib or better api response from openai.

@guymguym guymguym removed enhancement New feature or request help wanted Extra attention is needed labels Apr 27, 2022
@RobotSail
Copy link
Collaborator Author

This is related to #17 since splitting jobs between the two endpoints will result in less tokens being used overall.

@guymguym guymguym added this to the TBD milestone May 25, 2022
@RobotSail
Copy link
Collaborator Author

#71 seems related

@RobotSail
Copy link
Collaborator Author

Related to #13

@RobotSail
Copy link
Collaborator Author

We have a tokenizer code which cannot be published publicly, we could work around this by creating an API or some other workaround. Keeping this at TBD until requirements become clearer

@guymguym guymguym added enhancement New feature or request and removed question Further information is requested labels Aug 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@RobotSail RobotSail

Labels
enhancement New feature or request
Projects
None yet
Milestone
TBD
Development

No branches or pull requests
2 participants
@guymguym @RobotSail