[Good First Issue]: Verify red-pajama-3b-chat with GenAI text_generation

[p-wysocki]

Context

This task regards enabling tests for red-pajama-3b-chat. You can find more details under openvino_notebooks LLM chatbot README.md.

Please ask general questions in the main issue at #259

What needs to be done?

Described in the main Discussion issue at: #259

Example Pull Requests

Described in the main Discussion issue at: #259

Resources

• Contribution guide - start here!
• Intel DevHub Discord channel - engage in discussions, ask questions and talk to OpenVINO developers

Contact points

Described in the main Discussion issue at: #259

Ticket

[tranchung163]

.take

[github-actions]

Thank you for looking into this issue! Please let us know if you have any questions or require any help.

[p-wysocki]

Hello @tranchung163, are you still working on this? Is there anything we could help you with?

[tranchung163]

Hi @p-wysocki, yes i am still working on this.

I have tried Convert a model to OpenVINO IR and benchmark for red-pajama model. This is the output I got, it seems fine to me.

But I ran into some issues with cmake files [Text_generation]
(https://github.com/openvinotoolkit/openvino.genai/tree/master/text_generation/causal_lm/cpp)

I build greedy_causal_lm <MODEL_DIR> ""

I tried cmake -S .\ -B .\build\ && cmake --build .\build\ --config Release -j
and got this

I am trying to install all packages that are required and fix the issues. Sorry for the late response, I will try figure it out. Thanks

[p-wysocki]

@pavel-esir, @Wovchena could you please take a look?

[pavel-esir]

@tranchung163 thanks a lot for your analysis! According to logs looks like activate setupvars.sh was not executed or it was done with errors.

Could you please try to make cd to OPENVINO_INSTALL_DIR path and call source setupvars.sh form there?

cd <OPENVINO_INSTALL_DIR>
source setupvars.sh
cd - (back to genai repo)
cmake -DCMAKE_BUILD_TYPE=Release -S ./ -B ./build/ && cmake --build ./build/ -j

For some reason setupvars.sh works correctly only if you call it from the folder it's located

[tranchung163]

Hi @pavel-esir, thank you for your help. I set up source setupvars.sh and after building, installing tokenizers from openvino_tokenizers I was able to test beam search and greedy search. I wonder if the output should look like this ??

./build/beam_search_causal_lm

./build/greedy_causal_lm

[pavel-esir]

@tranchung163 thanks for the update. Outputs do not look meaningful. Could you please play around with different questions and what are the results?

[tranchung163]

Hi @pavel-esir, there is an issue while running
python3 -m pip install --upgrade-strategy eager "transformers<4.38" -r ../../../llm_bench/python/requirements.txt ../../../thirdparty/openvino_tokenizers/"[transformers]" --extra-index-url https://download.pytorch.org/whl/cpu

It seems like file ./../../llm_bench/python/requirements.txt can not install auto_gptq version 0.5.1. When I changed one line in requirements.txt file with auto_gptq==0.2.0 instead of auto_gptq>=0.5.1. It works and I was able to installed all packages including auto_gpt 0.2.0. (I wonder is it because I am using MacOS M1, so I run into this error ??)

Then I run these commands without having any issue

python3 ../../../llm_bench/python/convert.py --model_id TinyLlama/TinyLlama-1.1B-Chat-v1.0 --output_dir ./TinyLlama-1.1B-Chat-v1.0/ --precision FP16
convert_tokenizer ./TinyLlama-1.1B-Chat-v1.0/pytorch/dldt/FP16/ --output ./TinyLlama-1.1B-Chat-v1.0/pytorch/dldt/FP16/ --with-detokenizer --trust-remote-code

But I build my model again with ./build/beam_search_causal_lm /opt/intel/openvino_2024.0.0/openvino.genai/text_generation/causal_lm/cpp/RedPajama-INCITE-Chat-3B-v1/pytorch/dldt/FP32/ "What is openvino?"

..And I got the same result

[pavel-esir]

thanks for reporting the issue with installing requirements. It can be because of the M1. I'll check on x86 the versions of auto_gpt as well a bit later.

Hmm, outputs again look strange. But i noticed that you are using togethercomputer/RedPajama-INCITE-Chat-3B-v1 but issue is for another model https://huggingface.co/ikala/redpajama-3b-chat. I quickly looked and results look correct. Please check on that model and if you also get correct results add tests to .github/workflows/causal_lm_cpp.yml as well

[tranchung163]

Hi @pavel-esir, Thank you for your feedback. I think Auto-gptq new versions do not support M1 MacOS, so the output did not make sense. After I changed my laptop, I was able to run the test and got expected result. I also switched to model ikala/redpajama-3b-chat

I am going to add tests to .github/workflows/causal_lm_cpp.yml.

[DaBaoDIY]

#WLB#+ .take

[github-actions]

Thanks for being interested in this issue. It looks like this ticket is already assigned to a contributor. Please communicate with the assigned contributor to confirm the status of the issue.