Pinecone update to Readme and docker compose for ChatQnA #540

pallavijaini0525 · 2024-08-07T06:26:28Z

Description

Added the Readme and Docker compose files for Pinecone DB for ChatQnA usecase

Issues

#411

Type of change

List the type of change like below. Please delete options that are not relevant.

Others (enhancement, documentation, validation, etc.)

chensuyue · 2024-08-12T05:52:46Z

Do we need a separate readme?
ChatQnA/docker/xeon/docker_compose_pinecone.yaml --> ChatQnA/docker/xeon/compose_pinecone.yaml
Please add an example e2e test, like this one, https://github.com/opea-project/GenAIExamples/blob/main/ChatQnA/tests/test_chatqna_on_xeon.sh

pallavijaini0525 · 2024-08-12T16:54:44Z

@chensuyue -> Sure i will add the tests, can you please help in adding the PINECONE_KEY to the project variables simialr to HF_TOKEN, so i can use that key for the tests.

@ZePan110 has added for GenAIComps

chensuyue · 2024-08-13T01:37:24Z

@chensuyue -> Sure i will add the tests, can you please help in adding the PINECONE_KEY to the project variables simialr to HF_TOKEN, so i can use that key for the tests.

@ZePan110 has added for GenAIComps

#579

pallavijaini0525 · 2024-08-13T22:26:38Z

Tests are added and file is renamed to compose_pinecone.yaml

ChatQnA/tests/_test_chatqna_pinecone_on_xeon.sh

arun-gupta · 2024-09-05T23:43:13Z

What is the current status of this merge?

In addition, can opea/*-pinecone Docker images be pushed to Docker Hub? This will simplify the overall developer experience.

pallavijaini0525 · 2024-09-06T05:11:26Z

Only the docker compose and Readme files are not merged which are part of this PR, In GenAIComps, pinecone related changes are merged

arun-gupta · 2024-09-06T19:09:52Z

Seems like @chensuyue requested some changes. Can we make them so that it can be integrated?

pallavijaini0525 · 2024-09-11T18:04:43Z

@chensuyue -> All issues are resolved, please review

ChatQnA/tests/test_chatqna_pinecone_on_xeon.sh

chensuyue · 2024-09-13T02:22:58Z

@chensuyue -> All issues are resolved, please review

Because the test scripts name not adapt to the new structure, the test is actually not triggered.

Signed-off-by: Pallavi Jaini <[email protected]> Signed-off-by: Pallavi Jaini <pallavi,[email protected]>

for more information, see https://pre-commit.ci Signed-off-by: Pallavi Jaini <pallavi,[email protected]>

Signed-off-by: Pallavi Jaini <[email protected]> Signed-off-by: Pallavi Jaini <pallavi,[email protected]>

Signed-off-by: Pallavi Jaini <pallavi,[email protected]>

for more information, see https://pre-commit.ci Signed-off-by: Pallavi Jaini <pallavi,[email protected]>

…pdate

Signed-off-by: Pallavi Jaini <pallavi,[email protected]>

Signed-off-by: root <[email protected]>

Signed-off-by: Pallavi Jaini <[email protected]>

ChatQnA/tests/test_compose_pinecone_on_xeon.sh

Signed-off-by: Pallavi Jaini <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: Pallavi Jaini <[email protected]>

for more information, see https://pre-commit.ci

* fix get file error & refine logs Signed-off-by: letonghan <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: letonghan <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

…e for v0.9 (opea-project#538) * clip embedding support Signed-off-by: srinarayan-srikanthan <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: srinarayan-srikanthan <[email protected]> * test script for embedding Signed-off-by: srinarayan-srikanthan <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: srinarayan-srikanthan <[email protected]> * fix freeze workflow (opea-project#522) Signed-off-by: Sun, Xuehao <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]> * Fix Dataprep Potential Error in get_file (opea-project#540) * fix get file error & refine logs Signed-off-by: letonghan <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: letonghan <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: srinarayan-srikanthan <[email protected]> * Support SearchedDoc input type in LLM for No Rerank Pipeline (opea-project#541) Signed-off-by: letonghan <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]> * Add dependency for pdf2image and OCR processing (opea-project#421) Signed-off-by: srinarayan-srikanthan <[email protected]> * Add local_embedding return 768 length to align with chatqna example (opea-project#313) Signed-off-by: Chendi.Xue <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]> * add telemetry doc (opea-project#536) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: srinarayan-srikanthan <[email protected]> * Add video-llama LVM microservice under lvms (opea-project#495) Signed-off-by: BaoHuiling <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: srinarayan-srikanthan <[email protected]> * Fix the data load issue for structured files (opea-project#505) Signed-off-by: XuhuiRen <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]> * Add finetuning component (opea-project#502) Signed-off-by: Xinyu Ye <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: lkk <[email protected]> Co-authored-by: test <[email protected]> Co-authored-by: root <[email protected]> Co-authored-by: Letong Han <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]> * add torchvision into requirements (opea-project#546) Signed-off-by: chensuyue <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]> * Use Gaudi base images from Dockerhub (opea-project#526) * Use Gaudi base images from Dockerhub Signed-off-by: Abolfazl Shahbazi <[email protected]> * Fixing the malformed tag Signed-off-by: Abolfazl Shahbazi <[email protected]> * fix another malformed tag Signed-off-by: Abolfazl Shahbazi <[email protected]> --------- Signed-off-by: Abolfazl Shahbazi <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]> * Add toxicity detection microservice (opea-project#338) * Add toxicity detection microservice Signed-off-by: Qun Gao <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Modification to toxicity plugin PR (opea-project#432) * changed microservice to use Service.GUARDRAILS and input/output to TextDoc Signed-off-by: Tyler Wilbers <[email protected]> * simplify dockerfile to use langchain Signed-off-by: Tyler Wilbers <[email protected]> * sort requirements Signed-off-by: Tyler Wilbers <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Tyler Wilbers <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Minor SPDX header update (opea-project#434) Signed-off-by: Abolfazl Shahbazi <[email protected]> * Remove 'langsmith' per code review (opea-project#534) Signed-off-by: Abolfazl Shahbazi <[email protected]> * Add toxicity detection microservices with E2E testing Signed-off-by: Qun Gao <[email protected]> --------- Signed-off-by: Qun Gao <[email protected]> Signed-off-by: Tyler Wilbers <[email protected]> Signed-off-by: Abolfazl Shahbazi <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Abolfazl Shahbazi <[email protected]> Co-authored-by: Tyler W <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]> * rename script and use 5xxx Signed-off-by: BaoHuiling <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]> * add proxy for build Signed-off-by: BaoHuiling <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]> * fixed commit issues Signed-off-by: srinarayan-srikanthan <[email protected]> * Fix docarray constraint Signed-off-by: srinarayan-srikanthan <[email protected]> * updated docarray Signed-off-by: srinarayan-srikanthan <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: srinarayan-srikanthan <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * rm telemetry which cause error in mega Signed-off-by: BaoHuiling <[email protected]> * renamed dirs Signed-off-by: srinarayan-srikanthan <[email protected]> * renamed test Signed-off-by: srinarayan-srikanthan <[email protected]> --------- Signed-off-by: srinarayan-srikanthan <[email protected]> Signed-off-by: Sun, Xuehao <[email protected]> Signed-off-by: letonghan <[email protected]> Signed-off-by: Chendi.Xue <[email protected]> Signed-off-by: BaoHuiling <[email protected]> Signed-off-by: XuhuiRen <[email protected]> Signed-off-by: Xinyu Ye <[email protected]> Signed-off-by: chensuyue <[email protected]> Signed-off-by: Abolfazl Shahbazi <[email protected]> Signed-off-by: Qun Gao <[email protected]> Signed-off-by: Tyler Wilbers <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Sun, Xuehao <[email protected]> Co-authored-by: Letong Han <[email protected]> Co-authored-by: Zaili Wang <[email protected]> Co-authored-by: Chendi.Xue <[email protected]> Co-authored-by: Sihan Chen <[email protected]> Co-authored-by: Huiling Bao <[email protected]> Co-authored-by: XuhuiRen <[email protected]> Co-authored-by: XinyuYe-Intel <[email protected]> Co-authored-by: lkk <[email protected]> Co-authored-by: test <[email protected]> Co-authored-by: root <[email protected]> Co-authored-by: chen, suyue <[email protected]> Co-authored-by: Abolfazl Shahbazi <[email protected]> Co-authored-by: qgao007 <[email protected]> Co-authored-by: Tyler W <[email protected]>

pallavijaini0525 requested a review from lvliang-intel as a code owner August 7, 2024 06:26

pallavijaini0525 mentioned this pull request Aug 7, 2024

Managed vector database example #411

Open

pallavijaini0525 force-pushed the pinecone_update branch from c7f1c5b to b5b7c6d Compare August 7, 2024 06:38

pallavijaini0525 force-pushed the pinecone_update branch from 43f68a2 to 7220e10 Compare August 12, 2024 16:52

chensuyue mentioned this pull request Aug 13, 2024

Add secrets for CI test #579

Merged

4 tasks

pallavijaini0525 force-pushed the pinecone_update branch from 7220e10 to da944ae Compare August 13, 2024 20:55

pallavijaini0525 force-pushed the pinecone_update branch from b3c2e3f to 5e0b6e5 Compare August 14, 2024 15:33

chensuyue reviewed Aug 17, 2024

View reviewed changes

ChatQnA/tests/_test_chatqna_pinecone_on_xeon.sh Outdated Show resolved Hide resolved

pallavijaini0525 force-pushed the pinecone_update branch 3 times, most recently from 331e481 to 2088b19 Compare September 11, 2024 15:43

pallavijaini0525 requested a review from chensuyue September 11, 2024 18:05

chensuyue reviewed Sep 13, 2024

View reviewed changes

ChatQnA/tests/test_chatqna_pinecone_on_xeon.sh Outdated Show resolved Hide resolved

chensuyue reviewed Sep 13, 2024

View reviewed changes

ChatQnA/tests/test_chatqna_pinecone_on_xeon.sh Outdated Show resolved Hide resolved

pallavijaini0525 force-pushed the pinecone_update branch 7 times, most recently from e13bce5 to a5ade94 Compare September 18, 2024 15:50

pallavijaini0525 and others added 7 commits October 16, 2024 04:48

Commented out dataprep as not able to connect to pineconeserver

05a6b65

Signed-off-by: Pallavi Jaini <[email protected]> Signed-off-by: Pallavi Jaini <pallavi,[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

70b7159

for more information, see https://pre-commit.ci Signed-off-by: Pallavi Jaini <pallavi,[email protected]>

Added more comments

7628f33

Signed-off-by: Pallavi Jaini <[email protected]> Signed-off-by: Pallavi Jaini <pallavi,[email protected]>

Validated in IDC and enabled all the tests in pinecone

98d8eb4

Signed-off-by: Pallavi Jaini <pallavi,[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

19f95bf

for more information, see https://pre-commit.ci Signed-off-by: Pallavi Jaini <pallavi,[email protected]>

Merge remote-tracking branch 'origin/pinecone_update' into pinecone_u…

1a59f91

…pdate

Implemented all review comments

6aa1720

Signed-off-by: Pallavi Jaini <pallavi,[email protected]>

chensuyue mentioned this pull request Oct 16, 2024

add PINECONE_KEY_LANGCHAIN_TEST for CI test #959

Merged

4 tasks

root and others added 5 commits October 16, 2024 07:09

Updated the pinecone key

822d109

Signed-off-by: root <[email protected]>

Merge branch 'main' into pinecone_update

a83b36f

Merge branch 'main' into pinecone_update

9b14a97

Made changes to reflect the new framework

15362dc

Signed-off-by: Pallavi Jaini <[email protected]>

Fixed the typos

0b559c6

Signed-off-by: Pallavi Jaini <[email protected]>

chensuyue reviewed Oct 29, 2024

View reviewed changes

ChatQnA/tests/test_compose_pinecone_on_xeon.sh Outdated Show resolved Hide resolved

pallavijaini0525 and others added 7 commits October 28, 2024 21:52

added more logs

a722ba1

Signed-off-by: Pallavi Jaini <[email protected]>

Updated the log path

700671c

Updated the log path

810a70b

Signed-off-by: Pallavi Jaini <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

9b5a7e6

for more information, see https://pre-commit.ci

Merge branch 'main' into pinecone_update

f44b9ed

Updated the llm image

53e5013

Updated the llm image

2dbdffa

Signed-off-by: Pallavi Jaini <[email protected]>

chensuyue added this to the v1.1 milestone Nov 12, 2024

pallavijaini0525 and others added 3 commits November 12, 2024 08:25

uncommented building the images

2dd0339

uncommented building the images

de13ffe

Signed-off-by: Pallavi Jaini <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

d7b040a

for more information, see https://pre-commit.ci

chensuyue approved these changes Nov 13, 2024

View reviewed changes

ZePan110 approved these changes Nov 13, 2024

View reviewed changes

lvliang-intel merged commit 0d52c2f into opea-project:main Nov 13, 2024
15 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pinecone update to Readme and docker compose for ChatQnA #540

Pinecone update to Readme and docker compose for ChatQnA #540

pallavijaini0525 commented Aug 7, 2024

chensuyue commented Aug 12, 2024

pallavijaini0525 commented Aug 12, 2024

chensuyue commented Aug 13, 2024

pallavijaini0525 commented Aug 13, 2024

arun-gupta commented Sep 5, 2024

pallavijaini0525 commented Sep 6, 2024

arun-gupta commented Sep 6, 2024

pallavijaini0525 commented Sep 11, 2024

chensuyue commented Sep 13, 2024

Pinecone update to Readme and docker compose for ChatQnA #540

Pinecone update to Readme and docker compose for ChatQnA #540

Conversation

pallavijaini0525 commented Aug 7, 2024

Description

Issues

Type of change

chensuyue commented Aug 12, 2024

pallavijaini0525 commented Aug 12, 2024

chensuyue commented Aug 13, 2024

pallavijaini0525 commented Aug 13, 2024

arun-gupta commented Sep 5, 2024

pallavijaini0525 commented Sep 6, 2024

arun-gupta commented Sep 6, 2024

pallavijaini0525 commented Sep 11, 2024

chensuyue commented Sep 13, 2024