feat: Qdrant vectorstore support (#4689)

* feat: Qdrant vectorstore support Signed-off-by: Anush008 <[email protected]> * chore: make build-ui again Signed-off-by: Anush008 <[email protected]> --------- Signed-off-by: Anush008 <[email protected]>
feast-dev · Oct 27, 2024 · 86573d2 · 86573d2
1 parent a439b93
commit 86573d2
Show file tree

Hide file tree

Showing 16 changed files with 558 additions and 24 deletions.
diff --git a/Makefile b/Makefile
@@ -351,6 +351,14 @@ test-python-universal-singlestore-online:
 				not test_snowflake" \
 			sdk/python/tests
 
+ test-python-universal-qdrant-online:
+	PYTHONPATH='.' \
+		FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.online_stores.contrib.qdrant_repo_configuration \
+		PYTEST_PLUGINS=sdk.python.tests.integration.feature_repos.universal.online_store.qdrant \
+		python -m pytest -n 8 --integration \
+ 			-k "test_retrieve_online_documents" \
+ 			sdk/python/tests/integration/online_store/test_universal_online.py
+
 test-python-universal:
 	python -m pytest -n 8 --integration sdk/python/tests
 

diff --git a/docs/reference/alpha-vector-database.md b/docs/reference/alpha-vector-database.md
@@ -14,8 +14,9 @@ Below are supported vector databases and implemented features:
 | Milvus          | [ ]       | [ ]      |
 | Faiss           | [ ]       | [ ]      |
 | SQLite          | [x]       | [ ]      |
+| Qdrant          | [x]       | [x]      |
 
-Note: SQLite is in limited access and only working on Python 3.10. It will be updated as [sqlite_vec](https://github.com/asg017/sqlite-vec/) progresses.   
+Note: SQLite is in limited access and only working on Python 3.10. It will be updated as [sqlite_vec](https://github.com/asg017/sqlite-vec/) progresses.
 
 ## Example
 
@@ -113,9 +114,11 @@ print_online_features(features)
 ```
 
 ### Configuration
-We offer two Online Store options for Vector Databases. PGVector and SQLite.
+
+We offer [PGVector](https://github.com/pgvector/pgvector), [SQLite](https://github.com/asg017/sqlite-vec), [Elasticsearch](https://www.elastic.co) and [Qdrant](https://qdrant.tech/) as Online Store options for Vector Databases.
 
 #### Installation with SQLite
+
 If you are using `pyenv` to manage your Python versions, you can install the SQLite extension with the following command:
 ```bash
 PYTHON_CONFIGURE_OPTS="--enable-loadable-sqlite-extensions" \
@@ -124,6 +127,19 @@ PYTHON_CONFIGURE_OPTS="--enable-loadable-sqlite-extensions" \
     pyenv install 3.10.14
 ```
 And you can the Feast install package via:
+
 ```bash
 pip install feast[sqlite_vec]
-```
+```
+
+#### Installation with Elasticsearch
+
+```bash
+pip install feast[elasticsearch]
+```
+
+#### Installation with Qdrant
+
+```bash
+pip install feast[qdrant]
+```
diff --git a/docs/reference/online-stores/qdrant.md b/docs/reference/online-stores/qdrant.md
@@ -0,0 +1,81 @@
+# Qdrant online store (contrib)
+
+## Description
+
+[Qdrant](http://qdrant.tech) is a vector similarity search engine. It provides a production-ready service with a convenient API to store, search, and manage vectors with additional payload and extended filtering support. It makes it useful for all sorts of neural network or semantic-based matching, faceted search, and other applications.
+
+## Getting started
+
+In order to use this online store, you'll need to run `pip install 'feast[qdrant]'`.
+
+## Example
+
+{% code title="feature_store.yaml" %}
+
+```yaml
+project: my_feature_repo
+registry: data/registry.db
+provider: local
+online_store:
+    type: qdrant
+    host: localhost
+    port: 6333
+    vector_len: 384
+    write_batch_size: 100
+```
+
+{% endcode %}
+
+The full set of configuration options is available in [QdrantOnlineStoreConfig](https://rtd.feast.dev/en/master/#feast.infra.online_stores.contrib.qdrant.QdrantOnlineStoreConfig).
+
+## Functionality Matrix
+
+|                                                           | Qdrant   |
+| :-------------------------------------------------------- | :------- |
+| write feature values to the online store                  | yes      |
+| read feature values from the online store                 | yes      |
+| update infrastructure (e.g. tables) in the online store   | yes      |
+| teardown infrastructure (e.g. tables) in the online store | yes      |
+| generate a plan of infrastructure changes                 | no       |
+| support for on-demand transforms                          | yes      |
+| readable by Python SDK                                    | yes      |
+| readable by Java                                          | no       |
+| readable by Go                                            | no       |
+| support for entityless feature views                      | yes      |
+| support for concurrent writing to the same key            | no       |
+| support for ttl (time to live) at retrieval               | no       |
+| support for deleting expired data                         | no       |
+| collocated by feature view                                | yes      |
+| collocated by feature service                             | no       |
+| collocated by entity key                                  | no       |
+
+To compare this set of functionality against other online stores, please see the full [functionality matrix](overview.md#functionality-matrix).
+
+## Retrieving online document vectors
+
+The Qdrant online store supports retrieving document vectors for a given list of entity keys. The document vectors are returned as a dictionary where the key is the entity key and the value is the document vector. The document vector is a dense vector of floats.
+
+{% code title="python" %}
+
+```python
+from feast import FeatureStore
+
+feature_store = FeatureStore(repo_path="feature_store.yaml")
+
+query_vector = [1.0, 2.0, 3.0, 4.0, 5.0]
+top_k = 5
+
+# Retrieve the top k closest features to the query vector
+# Since Qdrant supports multiple vectors per entry,
+# the vector to use can be specified in the repo config.
+# Reference: https://qdrant.tech/documentation/concepts/vectors/#named-vectors
+feature_values = feature_store.retrieve_online_documents(
+    feature="my_feature",
+    query=query_vector,
+    top_k=top_k
+)
+```
+
+{% endcode %}
+
+These APIs are subject to change in future versions of Feast to improve performance and usability.
diff --git a/sdk/python/docs/source/feast.infra.online_stores.contrib.rst b/sdk/python/docs/source/feast.infra.online_stores.contrib.rst
@@ -40,6 +40,22 @@ feast.infra.online\_stores.contrib.elasticsearch\_repo\_configuration module
    :undoc-members:
    :show-inheritance:
 
+feast.infra.online\_stores.contrib.qdrant module
+-------------------------------------------------------
+
+.. automodule:: feast.infra.online_stores.contrib.qdrant
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+feast.infra.online\_stores.contrib.qdrant\_repo\_configuration module
+----------------------------------------------------------------------------
+
+.. automodule:: feast.infra.online_stores.contrib.qdrant_repo_configuration
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
 feast.infra.online\_stores.contrib.hazelcast\_repo\_configuration module
 ------------------------------------------------------------------------
 

diff --git a/sdk/python/feast/infra/offline_stores/contrib/trino_offline_store/connectors/upload.py b/sdk/python/feast/infra/offline_stores/contrib/trino_offline_store/connectors/upload.py
@@ -45,6 +45,7 @@
     "thrift",
     "tpcds",
     "tpch",
+    "qdrant",
 }
 CONNECTORS_WITHOUT_WITH_STATEMENTS: Set[str] = {
     "bigquery",