Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ENH] Add rust hnswlib bindings, index interface #1516

Merged
merged 17 commits into from
Jan 16, 2024
Merged

Conversation

HammadB
Copy link
Collaborator

@HammadB HammadB commented Dec 13, 2023

Description of changes

Summarize the changes made by this PR.

  • Improvements & Bug fixes
    • Fixes ChromaError to Trait Bound on std Error
  • New functionality
    • Adds the index module with traits for Index and Persistent Index types
    • Adds bindings to chroma-hnswlib c++, along with a rust-y interface for it.
    • Adds basic config injection for the index. In the future we can add dynamic/static field + watch behavior. I sketched out a plan for that while implementing this.

Test plan

How are these changes tested?
Rudimentary unit tests.

  • Tests pass locally with cargo test

Documentation Changes

None required.

Copy link

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

  • Can you think of any use case in which the code does not behave as intended? Have they been tested?
  • Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
  • If appropriate, are there adequate property based tests?
  • If appropriate, are there adequate unit tests?
  • Should any logging, debugging, tracing information be added or removed?
  • Are error messages user-friendly?
  • Have all documentation changes needed been made?
  • Have all non-obvious changes been commented?

System Compatibility

  • Are there any potential impacts on other parts of the system or backward compatibility?
  • Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

  • Is this code of a unexpectedly high quality (Readbility, Modularity, Intuitiveness)

rust/worker/build.rs Outdated Show resolved Hide resolved
rust/worker/build.rs Outdated Show resolved Hide resolved
@HammadB HammadB requested a review from beggers December 13, 2023 10:59
@HammadB HammadB force-pushed the hammad/rust_types_proto branch from 1dd5d74 to 9c7dfab Compare December 13, 2023 11:03
@HammadB HammadB force-pushed the hammad/rust_hnsw_index branch from 5763f1d to 6c4a2e8 Compare December 13, 2023 11:04
@@ -19,6 +19,7 @@ num_cpus = "1.16.0"
murmur3 = "0.5.2"
thiserror = "1.0.50"
num-bigint = "0.4.4"
tempfile = "3.8.1"
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move to [dev-dependencies]

@HammadB
Copy link
Collaborator Author

HammadB commented Dec 13, 2023

Tests are failing because we need to checkout chroma-hnswlib in the worker/ CI test

Copy link
Contributor

@beggers beggers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reasonable and understandable. LGTM

@HammadB HammadB force-pushed the hammad/rust_hnsw_index branch from 06a48d6 to faed752 Compare January 16, 2024 01:14
@HammadB HammadB changed the base branch from hammad/rust_types_proto to main January 16, 2024 01:14
@HammadB HammadB merged commit 878f91a into main Jan 16, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants