Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Verify minicpm 2b dpo #738

Closed
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 49 additions & 0 deletions .github/workflows/causal_lm_cpp.yml
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There was

  1. Extend the nightly_models list with the given model and run nightly tests for that model: https://github.com/openvinotoolkit/openvino.genai/tree/master/tests/python_tests#customise-tests-run. Report if there are failing tests and comment out the model in the nightly_models list. Add this change to your pull request (PR).

recently added. Please do that after MiniCPM-2B-dpo-fp16 gets a compatible license.

Original file line number Diff line number Diff line change
Expand Up @@ -349,6 +349,55 @@ jobs:
&& timeout 50s ./build/samples/cpp/beam_search_causal_lm/beam_search_causal_lm ./notus-7b-v1/ 69
| diff <(timeout 50s ./samples/python/beam_search_causal_lm/beam_search_causal_lm.py ./notus-7b-v1/ 69) -

cpp-beam_search_causal_lm-MiniCPM-2B-dpo:
runs-on: ubuntu-20.04-16-cores
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
runs-on: ubuntu-20.04-16-cores
runs-on: ubuntu-20.04

Try a smaller runner. If it fails, switch to ubuntu-20.04-4-cores

steps:
- uses: actions/checkout@v4
with:
submodules: recursive
- uses: actions/setup-python@v4
with:
python-version: 3.8
- name: Install OpenVINO
run: |
mkdir ./ov/
curl ${{ env.l_ov_link }} | tar --directory ./ov/ --strip-components 1 -xz
sudo ./ov/install_dependencies/install_openvino_dependencies.sh
- name: Download, convert and build
run: |
source ./ov/setupvars.sh
python -m pip install --upgrade-strategy eager -r ./samples/requirements.txt --pre --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly
python -m pip install ./thirdparty/openvino_tokenizers/[transformers] --pre --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly
optimum-cli export openvino --trust-remote-code --weight-format fp16 --model openbmb/MiniCPM-2B-dpo-fp16 MiniCPM-2B-dpo-fp16
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The model doesn't have a license. https://huggingface.co/openbmb/MiniCPM-2B-dpo-fp16/tree/main. We can't test it until it gets a compatible license. Please, submit an issue to https://huggingface.co/openbmb/MiniCPM-2B-dpo-fp16/ asking to add the license. I didn't think about it earlier. Sorry. I updated #259 to reflect the license question.

cmake -DCMAKE_BUILD_TYPE=Release -S ./ -B ./build/
cmake --build ./build/ --config Release -j
- name: Run Generation CPP
run: |
source ./ov/setupvars.sh
timeout 50s ./build/samples/cpp/beam_search_causal_lm/beam_search_causal_lm ./MiniCPM-2B-dpo-fp16/ "你好! 你好嗎?" > ./pred.txt
- name: Run Generation PY
run: >
. ./ov/setupvars.sh
&& export PYTHONPATH=./build/:$PYTHONPATH
&& timeout 50s samples/python/beam_search_causal_lm/beam_search_causal_lm.py ./MiniCPM-2B-dpo-fp16/ "你好! 你好嗎?"
| diff ./pred.txt -
- name: Compare
run: |
python -c "
import transformers
with open('pred.txt', 'r') as file:
predictions = file.read()
tokenizer = transformers.AutoTokenizer.from_pretrained('openbmb/MiniCPM-2B-dpo-fp16')
tokenized = tokenizer('你好! 你好嗎?', return_tensors='pt')
for beam in transformers.AutoModelForCausalLM.from_pretrained('openbmb/MiniCPM-2B-dpo-fp16', trust_remote_code=True).generate(**tokenized, num_beam_groups=3, num_beams=15, num_return_sequences=15, diversity_penalty=1.0, max_new_tokens=20, early_stopping=False, length_penalty=1.0, no_repeat_ngram_size=9**9, do_sample=False):
ref = ': ' + tokenizer.decode(beam[tokenized['input_ids'].numel():], skip_special_tokens=True)
idx = predictions.find(ref)
if -1 == idx:
raise RuntimeError(f'Missing "{ref=}" from predictions')
predictions = predictions[:idx] + predictions[idx + len(ref):]
"
echo "你好! 你好嗎?" passed

cpp-speculative_decoding_lm-ubuntu:
runs-on: ubuntu-20.04-16-cores
steps:
Expand Down
9 changes: 9 additions & 0 deletions src/docs/SUPPORTED_MODELS.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,15 @@
</ul>
</td>
</tr>
<tr>
<td><code>MiniCPMForCausalLM</code></td>
<td>MiniCPM</td>
<td>
<ul>
<li><a href="https://huggingface.co/openbmb/MiniCPM-2B-dpo-fp16"><code>openbmb/MiniCPM-2B-dpo-fp16</code></a></li>
</ul>
</td>
</tr>
</tbody>
</table>

Expand Down
Loading