Skip to content

Commit

Permalink
Try and always use at least one chunk
Browse files Browse the repository at this point in the history
  • Loading branch information
Weves committed Sep 5, 2023
1 parent 94f452b commit 15ea927
Showing 1 changed file with 8 additions and 0 deletions.
8 changes: 8 additions & 0 deletions backend/danswer/direct_qa/qa_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -270,6 +270,14 @@ def _get_usable_chunks(

usable_chunks.append(chunk)

# try and return at least one chunk if possible. This chunk will
# get truncated later on in the pipeline. This would only occur if
# the first chunk is larger than the token limit (usually due to character
# count -> token count mismatches caused by special characters / non-ascii
# languages)
if not usable_chunks and chunks:
usable_chunks = [chunks[0]]

return usable_chunks


Expand Down

0 comments on commit 15ea927

Please sign in to comment.