[Feature] Add minigpt4 gradio demo and training script

[hmtbgc]

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

Implement gradio demo and training script for MiniGPT-4.

Modification

• Implement minigpt4_demo.py and another auxiliary file (conversation.py) under projects/gradio_demo.
• Modify tools/train.py to support DeepSpeed training.
• Provide dataset for training (mmpretrain/datasets/minigpt4_dataset.py).
• Modify minigpt4.py to train properly.
• Provide a new script (minigpt-4_baichuan-7b_caption.py) for training minigpt4. This script replaces vicuna with other LLM to improve Chinese-generating ability.

BC-breaking (Optional)

Does the modification introduce changes that break the backward compatibility of the downstream repositories?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.

No

Use cases (Optional)

If this PR introduces a new feature, it is better to list some use cases here and update the documentation.

Training guide

• Download dataset from this url. There are two datasets: minigpt4 and llava. Both could be used individually for training. Modify line 37 of configs/minigpt4/minigpt-4_baichuan-7b_caption.py as path to minigpt4/llava dataset, and line 38 as minigpt4_official_data.json/llava_official_data.json.

• Download baichuan7b weight from this url and modify line 104 and 108 as path to baichuan7b weight.

• Run the following command in terminal:

  export CONFIG_FILE=configs/minigpt4/minigpt-4_baichuan-7b_caption.py
  bash ./tools/dist_train.sh ${CONFIG_FILE} ${NUM_GPUS}

  Each GPU may take up 28GB memory. It take ~34mins to train 6 epochs. 4 GPUs are used for training by default. If you only have one GPU, the learning rate should be reduced appropriately (e.g. from 1e-3 to 5e-4).

• After training, you will get a file named mp_rank_00_model_states.pt under work_dirs/minigpt-4_baichuan-7b_caption/epoch_6.pth. Post process mp_rank_00_model_states.ptwith the following python code:

  import torch
  from collections import OrderedDict
  file_path = '/path/to/mp_rank_00_model_states.pt'
  
  data = torch.load(filename)['module']
  
  destination = OrderedDict()
  destination['llama_proj.weight'] = data['llama_proj.weight']
  destination['llama_proj.bias'] = data['llama_proj.bias']
  torch.save(destination, '/path/to/save/model.pth')

Testing guide

• Run the following command in terminal:

  python projects/gradio_demo/minigpt4_demo.py /path/to/minigpt-4_baichuan-7b_caption.py /path/to/save/model.pth

  It needs 20GB GPU memory to start a chat with minigpt4.

Examples

• BaiChuan7B + minigpt4_official_data
  
  
  
  
• BaiChuan7B + llava_official_data
  
  
  
  
• Original Vicuna7B
  
  
  
  

Checklist

Before PR:

• Pre-commit or other linting tools are used to fix the potential lint issues.
• Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests.
• The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
• The documentation has been modified accordingly, like docstring or example tutorials.

After PR:

• If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects, like MMDet or MMSeg.
• CLA has been signed and all committers have signed the CLA in this PR.

[CLAassistant]

All committers have signed the CLA.

[OpenMMLab-Assistant-004]

Hi @hmtbgc,

We'd like to express our appreciation for your valuable contributions to the mmpretrain. Your efforts have significantly aided in enhancing the project's quality.
It is our pleasure to invite you to join our community thorugh Discord_Special Interest Group (SIG) channel. This is a great place to share your experiences, discuss ideas, and connect with other like-minded people. To become a part of the SIG channel, send a message to the moderator, OpenMMLab, briefly introduce yourself and mention your open-source contributions in the #introductions channel. Our team will gladly facilitate your entry. We eagerly await your presence. Please follow this link to join us: ​https://discord.gg/UjgXkPWNqA.

If you're on WeChat, we'd also love for you to join our community there. Just add our assistant using the WeChat ID: openmmlabwx. When sending the friend request, remember to include the remark "mmsig + Github ID".

Thanks again for your awesome contribution, and we're excited to have you as part of our community!