Batch query / search

[fyfrey]

Sometimes you don't want to find the nearest 1000 vectors to a single query vector. Instead, you look for e.g. the closest vector for 1000 different query vectors.
Currently, this requires looping over those 1000 vectors and making 1000 network requests unless I am mistaken...

This can be done a lot more efficiently if batch searches are supported.
For some indices (at least flat), this can also be computationally more efficient: one matrix-matrix multiplication compared to 1000 matrix-vector multiplications (for dot product).
Batching vector searches is supported in some vector databases like typesense or qdrant (maybe called multi search etc.)

Hope you'd like the feature as well! In any case, thanks for building this awesome software :-)

[gaocegege]

Hello, could you please provide more details about your specific use case? Are you utilizing vector search within an Online Analytical Processing (OLAP) scenario?

[fyfrey]

Sure! I've got two different use cases:

1. Scientific research: We build an interactive application to semi-automate manual text annotation in large text collections. We use embeddings for this and developed an algorithm to provide suggestions for the user / quantify results on the entire text corpus. This algorithm works by fetching the 1 closest embedding (except itself) for each embedding in a positive/negative support set. These support sets grow over time and typically become hundreds or a few thousand embeddings. To provide interactive suggestions for the user, we'd currently need to do many individual searches which becomes too slow for interactive usage when the support set grows to a few hundred vectors.

2. Immich: We use embeddings for images and detected faces. Currently, all processing works on one embedding at a time (even the clustering of face embeddings...) For new features, e.g. deduplicated auto-creation of an album for a time span (e.g. your last vacation), we'd need to find the closest vectors for X vectors. Another feature is automatically adding / suggesting images to album. You've got a set of query vectors (embeddings of all images in the album) and look for the closest to these.
   Looping is always an option, but it takes (too) long for smooth interactive usage.

[VoVAllen]

Hi, I'd like to share my thoughts on this feature:

• With the current algorithm, batching won't boost the performance a lot. HNSW is a graph algorithm that each points will have different trajectory on the graph. Usually they don't have common datas, so rare computation can be batched together. It might help if with GPU, but CPU can handle non-batched requests efficiently. My estimation is that the batched request will be less than 30% faster than the unbatched requests.
• For the network issue, the problem can be solved with client usages. To fully utilize the server resources, user need to initiate multiple sessions from the client. Postgres will start one process for one session, to utilize the CPU core. Then user can use prepared statement, with the binary format instead of string format to minimize the parsing cost. Sending multiple requests on each session will reuse the TCP connection with the database. Therefore the network cost should be similar between 1000 requests for each vector and one batched request. We might provide an example for this later.
• Due to the SQL interface, it's hard to design the batching syntax. I don't have any idea now.

And still thanks for your feedback! Making real world use cases easier and faster is always our target!