Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add finetuning component #502

Merged
merged 15 commits into from
Aug 21, 2024
Merged

Add finetuning component #502

merged 15 commits into from
Aug 21, 2024

Conversation

XinyuYe-Intel
Copy link
Collaborator

Description

Continue the work of #236.

Issues

n/a.

Type of change

List the type of change like below. Please delete options that are not relevant.

  • New feature (non-breaking change which adds new functionality)

Dependencies

List the newly introduced 3rd party dependency if exists.

pre-commit-ci bot and others added 3 commits August 19, 2024 06:23
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
f3949ce 
for more information, see https://pre-commit.ci
@XinyuYe-Intel
added docker for cpu part
7d5da06 
Signed-off-by: Xinyu Ye <[email protected]>
@lkk12014402 @pre-commit-ci
update finetuning api with openai format. (#535)
1ff81da 
* update finetuning api with openai format.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update doc and use 8001 port.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: test <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: root <[email protected]>
@lkk12014402 lkk12014402 mentioned this pull request Aug 21, 2024
Closed
@codecov Codecov
Copy link

codecov bot commented Aug 21, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Files Coverage Δ
comps/cores/mega/micro_service.py 91.13% <100.00%> (ø)
comps/cores/proto/api_protocol.py 95.54% <100.00%> (+1.31%) ⬆️

... and 2 files with indirect coverage changes

letonghan and others added 3 commits August 21, 2024 16:59
@letonghan
Merge branch 'main' into xinyuye/finetune
168c7d2
@lkk12014402 @pre-commit-ci
Add finetuning gaudi (#544)
bd991e4 
* enable finetuning on Gaudi

* add license Intel.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: test <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
@lkk12014402
fix license and typo (#545)
a73c0c4 
* remove token.

* fix license.

---------

Co-authored-by: root <[email protected]>
@chensuyue chensuyue added this to the v0.9 milestone Aug 21, 2024
@chensuyue chensuyue merged commit ad0bb7c into main Aug 21, 2024
14 checks passed
@chensuyue chensuyue deleted the xinyuye/finetune branch August 21, 2024 15:46
@chensuyue
Copy link
Collaborator

Please add e2e test in another PR.

chensuyue pushed a commit that referenced this pull request Aug 21, 2024
@XinyuYe-Intel @chensuyue
Add finetuning component (#502)
59fc1c4 
Signed-off-by: Xinyu Ye <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: lkk <[email protected]>
Co-authored-by: test <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: Letong Han <[email protected]>
(cherry picked from commit ad0bb7c)
@kevinintel kevinintel mentioned this pull request Aug 22, 2024
Closed
This was referenced Aug 22, 2024
Closed
Closed
tileintel pushed a commit to siddhivelankar23/GenAIComps that referenced this pull request Aug 22, 2024
@XinyuYe-Intel @pre-commit-ci
@lkk12014402 @letonghan @siddhivelankar23
Add finetuning component (opea-project#502)
165013c 
Signed-off-by: Xinyu Ye <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: lkk <[email protected]>
Co-authored-by: test <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: Letong Han <[email protected]>
Signed-off-by: siddhivelankar23 <[email protected]>
srinarayan-srikanthan pushed a commit to siddhivelankar23/GenAIComps that referenced this pull request Sep 1, 2024
@XinyuYe-Intel @pre-commit-ci
@lkk12014402 @letonghan @srinarayan-srikanthan
Add finetuning component (opea-project#502)
73a9a12 
Signed-off-by: Xinyu Ye <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: lkk <[email protected]>
Co-authored-by: test <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: Letong Han <[email protected]>
Signed-off-by: srinarayan-srikanthan <[email protected]>
sharanshirodkar7 pushed a commit to predictionguard/pg-GenAIComps that referenced this pull request Sep 3, 2024
@XinyuYe-Intel @pre-commit-ci
@lkk12014402 @letonghan @sharanshirodkar7
Add finetuning component (opea-project#502)
ab4b239 
Signed-off-by: Xinyu Ye <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: lkk <[email protected]>
Co-authored-by: test <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: Letong Han <[email protected]>
yinghu5 pushed a commit that referenced this pull request Sep 4, 2024
@srinarayan-srikanthan @pre-commit-ci
@XuehaoSun @letonghan @ZailiWang @xuechendi @Spycsh @BaoHuiling @XuhuiRen @XinyuYe-Intel @lkk12014402 @chensuyue @ashahba @qgao007
adding embedding support for CLIP based models for VideoRAGQnA exampl…
2a53e25 
…e for v0.9 (#538)

* clip embedding support

Signed-off-by: srinarayan-srikanthan <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: srinarayan-srikanthan <[email protected]>

* test script for embedding

Signed-off-by: srinarayan-srikanthan <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: srinarayan-srikanthan <[email protected]>

* fix freeze workflow (#522)

Signed-off-by: Sun, Xuehao <[email protected]>
Signed-off-by: srinarayan-srikanthan <[email protected]>

* Fix Dataprep Potential Error in get_file (#540)

* fix get file error & refine logs

Signed-off-by: letonghan <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: letonghan <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: srinarayan-srikanthan <[email protected]>

* Support SearchedDoc input type in LLM for No Rerank Pipeline (#541)

Signed-off-by: letonghan <[email protected]>
Signed-off-by: srinarayan-srikanthan <[email protected]>

* Add dependency for pdf2image and OCR processing (#421)

Signed-off-by: srinarayan-srikanthan <[email protected]>

* Add local_embedding return 768 length to align with chatqna example (#313)

Signed-off-by: Chendi.Xue <[email protected]>
Signed-off-by: srinarayan-srikanthan <[email protected]>

* add telemetry doc (#536)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: srinarayan-srikanthan <[email protected]>

* Add video-llama LVM microservice under lvms  (#495)

Signed-off-by: BaoHuiling <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: srinarayan-srikanthan <[email protected]>

* Fix the data load issue for structured files (#505)

Signed-off-by: XuhuiRen <[email protected]>
Signed-off-by: srinarayan-srikanthan <[email protected]>

* Add finetuning component (#502)

Signed-off-by: Xinyu Ye <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: lkk <[email protected]>
Co-authored-by: test <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: Letong Han <[email protected]>
Signed-off-by: srinarayan-srikanthan <[email protected]>

* add torchvision into requirements (#546)

Signed-off-by: chensuyue <[email protected]>
Signed-off-by: srinarayan-srikanthan <[email protected]>

* Use Gaudi base images from Dockerhub (#526)

* Use Gaudi base images from Dockerhub

Signed-off-by: Abolfazl Shahbazi <[email protected]>

* Fixing the malformed tag

Signed-off-by: Abolfazl Shahbazi <[email protected]>

* fix another malformed tag

Signed-off-by: Abolfazl Shahbazi <[email protected]>

---------

Signed-off-by: Abolfazl Shahbazi <[email protected]>
Signed-off-by: srinarayan-srikanthan <[email protected]>

* Add toxicity detection microservice (#338)

* Add toxicity detection microservice

Signed-off-by: Qun Gao <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Modification to toxicity plugin PR  (#432)

* changed microservice to use Service.GUARDRAILS and input/output to TextDoc

Signed-off-by: Tyler Wilbers <[email protected]>

* simplify dockerfile to use langchain

Signed-off-by: Tyler Wilbers <[email protected]>

* sort requirements

Signed-off-by: Tyler Wilbers <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Tyler Wilbers <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Minor SPDX header update (#434)

Signed-off-by: Abolfazl Shahbazi <[email protected]>

* Remove 'langsmith' per code review (#534)

Signed-off-by: Abolfazl Shahbazi <[email protected]>

* Add toxicity detection microservices with E2E testing

Signed-off-by: Qun Gao <[email protected]>

---------

Signed-off-by: Qun Gao <[email protected]>
Signed-off-by: Tyler Wilbers <[email protected]>
Signed-off-by: Abolfazl Shahbazi <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Abolfazl Shahbazi <[email protected]>
Co-authored-by: Tyler W <[email protected]>
Signed-off-by: srinarayan-srikanthan <[email protected]>

* rename script and use 5xxx

Signed-off-by: BaoHuiling <[email protected]>
Signed-off-by: srinarayan-srikanthan <[email protected]>

* add proxy for build

Signed-off-by: BaoHuiling <[email protected]>
Signed-off-by: srinarayan-srikanthan <[email protected]>

* fixed commit issues

Signed-off-by: srinarayan-srikanthan <[email protected]>

* Fix docarray constraint

Signed-off-by: srinarayan-srikanthan <[email protected]>

* updated docarray

Signed-off-by: srinarayan-srikanthan <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: srinarayan-srikanthan <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* rm telemetry which cause error in mega

Signed-off-by: BaoHuiling <[email protected]>

* renamed dirs

Signed-off-by: srinarayan-srikanthan <[email protected]>

* renamed test

Signed-off-by: srinarayan-srikanthan <[email protected]>

---------

Signed-off-by: srinarayan-srikanthan <[email protected]>
Signed-off-by: Sun, Xuehao <[email protected]>
Signed-off-by: letonghan <[email protected]>
Signed-off-by: Chendi.Xue <[email protected]>
Signed-off-by: BaoHuiling <[email protected]>
Signed-off-by: XuhuiRen <[email protected]>
Signed-off-by: Xinyu Ye <[email protected]>
Signed-off-by: chensuyue <[email protected]>
Signed-off-by: Abolfazl Shahbazi <[email protected]>
Signed-off-by: Qun Gao <[email protected]>
Signed-off-by: Tyler Wilbers <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sun, Xuehao <[email protected]>
Co-authored-by: Letong Han <[email protected]>
Co-authored-by: Zaili Wang <[email protected]>
Co-authored-by: Chendi.Xue <[email protected]>
Co-authored-by: Sihan Chen <[email protected]>
Co-authored-by: Huiling Bao <[email protected]>
Co-authored-by: XuhuiRen <[email protected]>
Co-authored-by: XinyuYe-Intel <[email protected]>
Co-authored-by: lkk <[email protected]>
Co-authored-by: test <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: chen, suyue <[email protected]>
Co-authored-by: Abolfazl Shahbazi <[email protected]>
Co-authored-by: qgao007 <[email protected]>
Co-authored-by: Tyler W <[email protected]>
@hshen14 hshen14 mentioned this pull request Sep 4, 2024
Closed
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lkk12014402 lkk12014402 lkk12014402 left review comments

@kevinintel kevinintel kevinintel approved these changes

@XuhuiRen XuhuiRen XuhuiRen approved these changes

@lvliang-intel lvliang-intel Awaiting requested review from lvliang-intel lvliang-intel is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v0.9
Development

Successfully merging this pull request may close these issues.
6 participants
@XinyuYe-Intel @chensuyue @lkk12014402 @XuhuiRen @kevinintel @letonghan