-
Notifications
You must be signed in to change notification settings - Fork 917
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Implement a GemFireVectorStore implementing the VectorStore interface. - Add unit and integration test. - Add antora documentation. - Add to BOM.
- Loading branch information
Showing
9 changed files
with
948 additions
and
2 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
122 changes: 122 additions & 0 deletions
122
spring-ai-docs/src/main/antora/modules/ROOT/pages/api/vectordbs/gemfire.adoc
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,122 @@ | ||
= GemFire Vector Store | ||
|
||
This section walks you through setting up the GemFire VectorStore to store document embeddings and perform similarity searches. | ||
|
||
link:https://tanzu.vmware.com/gemfire[GemFire] is an ultra high speed in-memory data and compute grid, with vector extensions to store and search vectors efficiently. | ||
|
||
link:https://docs.vmware.com/en/VMware-GemFire-VectorDB/1.0/gemfire-vectordb/overview.html[GemFire VectorDB] extends GemFire's capabilities, serving as a versatile vector database that efficiently stores, retrieves, and performs vector searches through a distributed and resilient infrastructure: | ||
|
||
Capabilities: | ||
- Create Indexes | ||
- Store vectors and the associated metadata | ||
- Perform vector searches based on similarity | ||
|
||
== Prerequisites | ||
|
||
Access to a GemFire cluster with the link:https://docs.vmware.com/en/VMware-GemFire-VectorDB/1.0/gemfire-vectordb/install.html[GemFire Vector Database] extension installed. | ||
You can download the GemFire VectorDB extension from the link:https://network.pivotal.io/products/gemfire-vectordb/[VMware Tanzu Network] after signing in. | ||
|
||
== Dependencies | ||
|
||
Add these dependencies to your project: | ||
|
||
- Embedding Client boot starter, required for calculating embeddings. | ||
- Transformers Embedding (Local) and follow the ONNX Transformers Embedding instructions. | ||
|
||
[source,xml] | ||
---- | ||
<dependency> | ||
<groupId>org.springframework.ai</groupId> | ||
<artifactId>spring-ai-transformers</artifactId> | ||
</dependency> | ||
---- | ||
|
||
- Add the GemFire VectorDB dependencies | ||
|
||
[source,xml] | ||
---- | ||
<dependency> | ||
<groupId>org.springframework.ai</groupId> | ||
<artifactId>spring-ai-gemfire</artifactId> | ||
</dependency> | ||
---- | ||
|
||
|
||
TIP: Refer to the xref:getting-started.adoc#dependency-management[Dependency Management] section to add the Spring AI BOM to your build file. | ||
|
||
|
||
== Sample Code | ||
|
||
- To configure GemFire in your application, use the following setup: | ||
|
||
[source,java] | ||
---- | ||
@Bean | ||
public GemFireVectorStoreConfig gemFireVectorStoreConfig() { | ||
return GemFireVectorStoreConfig.builder() | ||
.withUrl("http://localhost:8080") | ||
.withIndexName("spring-ai-test-index") | ||
.build(); | ||
} | ||
---- | ||
|
||
- Create a GemFireVectorStore instance connected to your GemFire VectorDB: | ||
|
||
[source,java] | ||
---- | ||
@Bean | ||
public VectorStore vectorStore(GemFireVectorStoreConfig config, EmbeddingClient embeddingClient) { | ||
return new GemFireVectorStore(config, embeddingClient); | ||
} | ||
---- | ||
- Create a Vector Index which will configure GemFire region. | ||
|
||
[source,java] | ||
---- | ||
public void createIndex() { | ||
try { | ||
CreateRequest createRequest = new CreateRequest(); | ||
createRequest.setName(INDEX_NAME); | ||
createRequest.setBeamWidth(20); | ||
createRequest.setMaxConnections(16); | ||
ObjectMapper objectMapper = new ObjectMapper(); | ||
String index = objectMapper.writeValueAsString(createRequest); | ||
client.post() | ||
.contentType(MediaType.APPLICATION_JSON) | ||
.bodyValue(index) | ||
.retrieve() | ||
.bodyToMono(Void.class) | ||
.block(); | ||
} | ||
catch (Exception e) { | ||
logger.warn("An unexpected error occurred while creating the index"); | ||
} | ||
} | ||
---- | ||
|
||
- Create some documents: | ||
|
||
[source,java] | ||
---- | ||
List<Document> documents = List.of( | ||
new Document("1", getText("classpath:/test/data/spring.ai.txt"), Map.of("meta1", "meta1")), | ||
new Document("2", getText("classpath:/test/data/time.shelter.txt"), Map.of()), | ||
new Document("3", getText("classpath:/test/data/great.depression.txt"), Map.of("meta2", "meta2"))); | ||
---- | ||
|
||
- Add the documents to GemFire VectorDB: | ||
|
||
[source,java] | ||
---- | ||
vectorStore.add(List.of(document)); | ||
---- | ||
|
||
- And finally, retrieve documents similar to a query: | ||
|
||
[source,java] | ||
---- | ||
List<Document> results = vectorStore.similaritySearch("Spring", 5); | ||
---- | ||
|
||
If all goes well, you should retrieve the document containing the text "Spring AI rocks!!". | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -323,7 +323,6 @@ | |
<scope>test</scope> | ||
</dependency> | ||
|
||
|
||
</dependencies> | ||
</dependencies> | ||
|
||
</project> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
[GemFire Vector Store Documentation](https://docs.spring.io/spring-ai/reference/api/vectordbs/gemfire.html) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,81 @@ | ||
<?xml version="1.0" encoding="UTF-8"?> | ||
<project xmlns="http://maven.apache.org/POM/4.0.0" | ||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"> | ||
<modelVersion>4.0.0</modelVersion> | ||
<parent> | ||
<groupId>org.springframework.ai</groupId> | ||
<artifactId>spring-ai</artifactId> | ||
<version>1.0.0-SNAPSHOT</version> | ||
<relativePath>../../pom.xml</relativePath> | ||
</parent> | ||
<artifactId>spring-ai-gemfire</artifactId> | ||
<packaging>jar</packaging> | ||
<name>Spring AI Vector Store - GemFire</name> | ||
<description>Spring AI GemFire Vector Store</description> | ||
<url>https://github.com/spring-projects/spring-ai</url> | ||
|
||
<scm> | ||
<url>https://github.com/spring-projects/spring-ai</url> | ||
<connection>git://github.com/spring-projects/spring-ai.git</connection> | ||
<developerConnection>[email protected]:spring-projects/spring-ai.git</developerConnection> | ||
</scm> | ||
|
||
<properties> | ||
<maven.compiler.target>17</maven.compiler.target> | ||
<maven.compiler.source>17</maven.compiler.source> | ||
</properties> | ||
|
||
<dependencies> | ||
<dependency> | ||
<groupId>org.springframework.ai</groupId> | ||
<artifactId>spring-ai-core</artifactId> | ||
<version>${project.parent.version}</version> | ||
</dependency> | ||
|
||
<dependency> | ||
<groupId>org.springframework</groupId> | ||
<artifactId>spring-webflux</artifactId> | ||
</dependency> | ||
|
||
<!-- TESTING --> | ||
<dependency> | ||
<groupId>org.springframework.ai</groupId> | ||
<artifactId>spring-ai-openai</artifactId> | ||
<version>${parent.version}</version> | ||
<scope>test</scope> | ||
</dependency> | ||
|
||
<dependency> | ||
<groupId>org.springframework.ai</groupId> | ||
<artifactId>spring-ai-test</artifactId> | ||
<version>${parent.version}</version> | ||
<scope>test</scope> | ||
</dependency> | ||
|
||
<dependency> | ||
<groupId>org.springframework.ai</groupId> | ||
<artifactId>spring-ai-transformers</artifactId> | ||
<version>${parent.version}</version> | ||
<scope>test</scope> | ||
</dependency> | ||
|
||
<dependency> | ||
<groupId>org.springframework.boot</groupId> | ||
<artifactId>spring-boot-starter-test</artifactId> | ||
<scope>test</scope> | ||
</dependency> | ||
|
||
<dependency> | ||
<groupId>org.awaitility</groupId> | ||
<artifactId>awaitility</artifactId> | ||
<version>3.0.0</version> | ||
<scope>test</scope> | ||
</dependency> | ||
<dependency> | ||
<groupId>org.apache.logging.log4j</groupId> | ||
<artifactId>log4j-core</artifactId> | ||
</dependency> | ||
|
||
</dependencies> | ||
|
||
</project> |
Oops, something went wrong.