Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Vector Embedding Search Optimization #62

Open
6 tasks
devansh-shah-11 opened this issue Oct 7, 2024 · 3 comments
Open
6 tasks

Vector Embedding Search Optimization #62

devansh-shah-11 opened this issue Oct 7, 2024 · 3 comments

Comments

@devansh-shah-11
Copy link
Collaborator

devansh-shah-11 commented Oct 7, 2024

Is your feature request related to a problem? Please describe.
Embedding searches in vector databases for face recognition can be slow, especially with large datasets. Faster retrieval methods are essential for efficient model evaluation and experimentation.

Describe the solution you'd like
Implement optimizations for embedding search by leveraging indexing strategies like Approximate Nearest Neighbor (ANN) and caching mechanisms. These optimizations should be adaptable across different vector DBs (such as FAISS, Pinecone, Milvus) to ensure faster face recognition queries.

Additional context
These optimizations will enhance overall system performance and enable more efficient searches, thereby improving the response time.

Checklist

  • Research Approximate Nearest Neighbor (ANN) libraries

    • FAISS, Pinecone, Milvus, or other relevant vector DBs.
  • Ensure support across different vector DBs

    • Test optimizations with different databases (FAISS, Pinecone, Milvus).
  • Integrate ANN search capabilities

    • Implement ANN indexing in vector DBs used by the system.
  • Implement caching mechanisms for frequent searches

    • Add caching for common queries to further improve performance.
  • Test the optimized embedding search on large datasets

    • Benchmark and compare performance before and after the optimizations.
  • Document the embedding search optimization process

    • Add detailed documentation detailing how to configure and optimize searches.
@Sai-ganesh-0004
Copy link

Hello I would like to work on this Can you assign it to me

@Devasy23
Copy link
Owner

Hey @Sai-ganesh-0004 , I hope the issue you working on is going well, If you need any help you may reach out to us, we'll be happy to guide you

@Soumyals
Copy link

Hey, I would love to work on this project. I have worked on fine tuning LLM'S and well as RAG's and have a fair understanding of vector embeddings and vector databases. I would be able to effectively contribute to this project. Could you please assign this to me.
Thank you.

@Sai-ganesh-0004 Sai-ganesh-0004 removed their assignment Nov 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants