[FEA]: Improve the VectorDBService and WriteToVectorDBStage to handle similarity search

[mdemoret-nv]

Is this a new feature, an improvement, or a change to existing functionality?

Improvement

How would you describe the priority of this feature request

High

Please provide a clear description of problem this feature solves

When working with the VectorDBService and WriteToVectorDBStage for LLM pipelines, there are a few deficiencies which have come up:

• There is no API for performing a similarity search from a FLOAT_VECTOR index type
• The resource name needs to be specified for every call to the service API.
• No information can be saved in the service for a particular resource (i.e. the schema or configuration options)
• The WriteToVectorDBStage does not have any capability to upload embedding vectors

Describe your ideal solution

The following improvements should be added to the VectorDBService class:

1. A new sub object (VectorDBServiceResource) can be created which limits the API to a single resource.
   1. All of the APIs which take a name parameter should be available on the VectorDBServiceResource class.
   2. i.e. VectorDBService.insert(name: str, data: list, **kwargs) -> VectorDBServiceResource.insert(data: list, **kwargs)
2. A new method get_resource_client(self, name: str, **kwargs: dict[str, typing.Any]) -> VectorDBServiceResource will provide the mechanism for getting the resource client object
3. Reorganizing the files:
   1. Make a subdirectory in morpheus/service called vdb
   2. Move milvus_vector_db_service.py to morpheus/service/vdb
   3. Move vector_db_service.py to morpheus/service/vdb
   4. Move vector_db_service_utils.py to morpheus/service/vdb/utils.py

These will then need to be implemented for all VDB implementations.

The following improvements should be added to the WriteToVectorDBStage:

1. Ability to specify which column should be treated as an embedding vector column for the purpose of uploading an embedding with each document
2. The ability to recreate the VDB resource on startup
3. Separation of the service_kwargs (the arguments needed for connecting to the service) from the resource_kwargs (the args required to build and setup the resource)
4. The service_kwargs will always be required since they are necessary to connect to the service
5. The resource_kwargs are only necessary when creating the resource (or recreating). If we are just connecting to an existing resource, then the parameters can be determined from the service.
6. Ability to use both ControlMessage or MultiMessage or MultiMessageResponse (if an embedding column is specified) types

Additional context

See the following files for partial implementations of the above:

• https://github.com/nv-morpheus/Morpheus/blob/de35ac300e7061bcfd3599c2c7f9b3373e71d703/morpheus/stages/output/write_to_vector_db.py
• https://github.com/nv-morpheus/Morpheus/blob/de35ac300e7061bcfd3599c2c7f9b3373e71d703/morpheus/service/milvus_vector_db_service.py
• https://github.com/nv-morpheus/Morpheus/blob/b324ed725c51d38f8b8437dbfbd3d55788b5d2a3/morpheus/service/vector_db_service.py
• https://github.com/nv-morpheus/Morpheus/blob/b324ed725c51d38f8b8437dbfbd3d55788b5d2a3/morpheus/utils/vector_db_service_utils.py

Code of Conduct

• I agree to follow this project's Code of Conduct
• I have searched the open feature requests and have found no duplicates for this feature request