[Neural Speed] Fix Baichuan, chatGLM1&2&3 acc issue

[zhentaoyu]

Type of Change

feature or bug fix or documentation or others
API changed or not

Description

detail description
Issues: xxx

due to logits_all memory copy

ne_baichuan_q_int4_bestla_cint8_sym_sfp32_g32.bin    
|    Tasks     |Version|Filter|n-shot|  Metric  |Value |   |Stderr|
|--------------|------:|------|-----:|----------|-----:|---|-----:|
|lambada_openai|      1|none  |     0|perplexity|4.0944|±  |0.1164|
|              |       |none  |     0|acc       |0.6662|±  |0.0066|

ne_chatglm3_q_int4_bestla_cint8_sym_sfp32_g32.bin   
|    Tasks     |Version|Filter|n-shot|  Metric  |Value|   |Stderr|
|--------------|------:|------|-----:|----------|----:|---|-----:|
|lambada_openai|      1|none  |     0|perplexity|9.339|±  |0.4476|
|              |       |none  |     0|acc       |0.596|±  |0.0068|

ne_chatglm2_q_int4_bestla_cint8_sym_sfp32_g32.bin
|    Tasks     |Version|Filter|n-shot|  Metric  | Value |   |Stderr|
|--------------|------:|------|-----:|----------|------:|---|-----:|
|lambada_openai|      1|none  |     0|perplexity|13.0181|±  |0.6216|
|              |       |none  |     0|acc       | 0.5263|±  |0.0070|

due to tokenizer

ne_chatglm_q_int4_bestla_cint8_sym_sfp32_g32.bin
|    Tasks     |Version|Filter|n-shot|  Metric  |  Value  |   | Stderr |
|--------------|------:|------|-----:|----------|--------:|---|-------:|
|lambada_openai|      1|none  |     0|perplexity|1181.2761|±  |178.7728|
|              |       |none  |     0|acc       |   0.4194|±  |  0.0069|

|Tasks|Version|Filter|n-shot| Metric |Value|   |Stderr|
|-----|------:|------|-----:|--------|----:|---|-----:|
|piqa |      1|none  |     0|acc     |0.506|±  |0.0117|
|     |       |none  |     0|acc_norm|0.488|±  |0.0117|

|  Tasks   |Version|Filter|n-shot|Metric|Value |   |Stderr|
|----------|------:|------|-----:|------|-----:|---|-----:|
|winogrande|      1|none  |     0|acc   |0.4996|±  |0.0141|

Expected Behavior & Potential Risk

the expected behavior that triggered by this PR

How has this PR been tested?

how to reproduce the test (including hardware information)

Dependency Change?

any library dependency introduced or removed

[a32543254]

LGTM

[intellinjun]

https://inteltf-jenk.sh.intel.com/job/neural_speed_extension/170/

[zhentaoyu]

https://inteltf-jenk.sh.intel.com/job/neural_speed_extension/170/

waiting for ext test results.

[zhentaoyu]

ne_chatglm_f32.bin
|    Tasks     |Version|Filter|n-shot|  Metric  |  Value  |   | Stderr |
|--------------|------:|------|-----:|----------|--------:|---|-------:|
|lambada_openai|      1|none  |     0|perplexity|1089.0576|±  |166.4332|
|              |       |none  |     0|acc       |   0.4236|±  |  0.0069|