Skip to content
This repository has been archived by the owner on Aug 30, 2024. It is now read-only.

miagrate pr: Calculate accuracy of runtime #46

Merged
merged 7 commits into from
Jan 19, 2024
Merged

miagrate pr: Calculate accuracy of runtime #46

merged 7 commits into from
Jan 19, 2024

Conversation

zhenwei-intel
Copy link
Contributor

@zhenwei-intel zhenwei-intel commented Jan 11, 2024

Type of Change

feature
intel/intel-extension-for-transformers#1123

Description

  • add script of calculating diff
  • add script of calculating accuracy

usage

python scripts/cal_acc.py --model_name ~/models/llama/Llama-2-7b-chat-hf/ --model_format runtime --tasks piqa

Meta-llama/Llama-2-7b-chat-hf on lamabda_openai:

acc ppl
Fp32
0.6986 3.3715
Q4j 0.6839 3.6585
GPTQ 0.6953 3.5189

Expected Behavior & Potential Risk

the expected behavior that triggered by this PR

How has this PR been tested?

how to reproduce the test (including hardware information)

Dependency Change?

any library dependency introduced or removed

@zhenwei-intel zhenwei-intel changed the title Calculate accuracy of runtime miagrate pr: Calculate accuracy of runtime Jan 18, 2024
@a32543254
Copy link
Contributor

add reuqirment.txt

Copy link
Contributor

@a32543254 a32543254 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zhenwei-intel zhenwei-intel marked this pull request as draft January 18, 2024 07:39
@zhenwei-intel zhenwei-intel marked this pull request as ready for review January 19, 2024 01:34
Signed-off-by: zhenwei-intel <[email protected]>
Signed-off-by: zhenwei-intel <[email protected]>
Signed-off-by: zhenwei-intel <[email protected]>
Signed-off-by: zhenwei-intel <[email protected]>
Signed-off-by: zhenwei-intel <[email protected]>
Signed-off-by: zhenwei-intel <[email protected]>
Signed-off-by: zhenwei-intel <[email protected]>
Copy link
Contributor

@a32543254 a32543254 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add extension test for acc test

@VincyZhang
Copy link
Contributor

Could we add extension test for acc test
We will later.
Please provide your local result first @zhenwei-intel

@zhenwei-intel
Copy link
Contributor Author

Could we add extension test for acc test
We will later.
Please provide your local result first @zhenwei-intel

updated

@VincyZhang VincyZhang merged commit 7b124aa into main Jan 19, 2024
10 checks passed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants