fix TextCallbackStreamer when characters are eaten by regex #1302
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
pavel-esir
commented
Dec 4, 2024
pavel-esir
commented
- In some cases adding the next token can shorten the text, e.g. when apostrophe removing regex had worked after adding new tokens. Need to hold on before flushing if apostrophe is the last symbol.
- ticket CVS-157216
apaniukov
approved these changes
Dec 4, 2024
Wovchena
requested changes
Dec 4, 2024
ilya-lavrenov
added
the
port to master
pavel-esir
commented
Dec 4, 2024
Wovchena
reviewed
Dec 5, 2024
Comment on lines
+100
to
+101
| # Also, in some cases adding the next token can shorten the text, | ||
| # e.g. when apostrophe removing regex had worked after adding new tokens. |
There was a problem hiding this comment.
Suggested change
| # Also, in some cases adding the next token can shorten the text, | |
| # e.g. when apostrophe removing regex had worked after adding new tokens. | |
| # E.g. when apostrophe removing regex had worked after adding new tokens. |
This statements are equivalent:
It is possible to have a shorter text after adding new token.
and
in some cases adding the next token can shorten the text
Wovchena
approved these changes
Dec 5, 2024
There was a problem hiding this comment.
There's no much time left. Address https://github.com/openvinotoolkit/openvino.genai/pull/1302/files#r1870703054 when porting to master
Merged
via the queue into
openvinotoolkit:releases/2024/5
with commit Dec 5, 2024
15f6a90
52 checks passed
pavel-esir
pushed a commit
to pavel-esir/openvino.genai
that referenced
this pull request
Dec 10, 2024
…toolkit#1302) - In some cases adding the next token can shorten the text, e.g. when apostrophe removing regex had worked after adding new tokens. Need to hold on before flushing if apostrophe is the last symbol. - ticket CVS-157216
pavel-esir
removed
the
port to master
pavel-esir
added a commit
to pavel-esir/openvino.genai
that referenced
this pull request
Dec 11, 2024
ilya-lavrenov
added
the
port to master
ilya-lavrenov
added a commit
that referenced
this pull request
Dec 11, 2024
- ~#1302 (didn't port this PR because of the issue CVS-159227) - #1262 - #1336 - #1331 --------- Co-authored-by: Andrei Kochin <[email protected]> Co-authored-by: Vladimir Zlobin <[email protected]> Co-authored-by: Ilya Lavrenov <[email protected]>
sungeunk
pushed a commit
to sungeunk/openvino.genai
that referenced
this pull request
Dec 16, 2024
- ~openvinotoolkit#1302 (didn't port this PR because of the issue CVS-159227) - openvinotoolkit#1262 - openvinotoolkit#1336 - openvinotoolkit#1331 --------- Co-authored-by: Andrei Kochin <[email protected]> Co-authored-by: Vladimir Zlobin <[email protected]> Co-authored-by: Ilya Lavrenov <[email protected]>
ScottZhang812
pushed a commit
to ScottZhang812/_openvino.genai
that referenced
this pull request
Dec 23, 2024
- ~openvinotoolkit/openvino.genai#1302 (didn't port this PR because of the issue CVS-159227) - openvinotoolkit/openvino.genai#1262 - openvinotoolkit/openvino.genai#1336 - openvinotoolkit/openvino.genai#1331 --------- Co-authored-by: Andrei Kochin <[email protected]> Co-authored-by: Vladimir Zlobin <[email protected]> Co-authored-by: Ilya Lavrenov <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.