Skip to content

Commit

Permalink
[BUG]: Fixed BF index overflow issue with subsequent delete (chroma-c…
Browse files Browse the repository at this point in the history
…ore#1150)

Refs: chroma-core#989

## Description of changes

*Summarize the changes made by this PR.*
 - Improvements & Bug fixes
- When the BF index overflows (batch_size upon insertion of large batch
it is cleared, if a subsequent delete request comes to delete Ids which
were in the cleared BF index a warning is raised for non-existent
embedding. The issue was resolved by separately checking if BF the
record exists in the BF index and conditionally execute the BF removal

## Test plan
*How are these changes tested?*

- [x] Tests pass locally with `pytest` for python

## Documentation Changes
N/A
  • Loading branch information
tazarov authored Sep 19, 2023
1 parent 9e05cb3 commit 3aed7b7
Showing 1 changed file with 3 additions and 1 deletion.
4 changes: 3 additions & 1 deletion chromadb/segment/impl/vector/local_persistent_hnsw.py
Original file line number Diff line number Diff line change
Expand Up @@ -225,11 +225,13 @@ def _write_records(self, records: Sequence[EmbeddingRecord]) -> None:
exists_in_index = self._id_to_label.get(
id, None
) is not None or self._brute_force_index.has_id(id)
exists_in_bf_index = self._brute_force_index.has_id(id)

if op == Operation.DELETE:
if exists_in_index:
self._curr_batch.apply(record)
self._brute_force_index.delete([record])
if exists_in_bf_index:
self._brute_force_index.delete([record])
else:
logger.warning(f"Delete of nonexisting embedding ID: {id}")

Expand Down

0 comments on commit 3aed7b7

Please sign in to comment.