MotionAgent

Introduction

如果您熟悉中文，可以阅读中文版本的README。

MotionAgent is a deep learning model tool that can generate videos from user-created scripts. Users can create scripts, generate movie stills, generate images/videos, and compose background music through our provided toolset.

The model of MotionAgent is powered by the open-source model community ModelScope.

Features

• Script Generation
  • Users can generate scripts by specifying the story theme and background
  • The script generation model is based on LLM (such as Qwen-7B-Chat), which can generate scripts of various styles
• Movie still Generation
  • Generate corresponding movie still scene images
• Video Generation
  • Generate videos from images
  • Support high-resolution video generation
• Music Generation
  • Custom style background music

Quick Start

Compatibility Verification

Verified environments:

• python3.8
• torch2.0.1
• CUDA11.7
• OS: Ubuntu 20.04
• Nvidia-A100 40G

Resource Requirements

• GPU memory: 36GB
• Disk: It is recommended to reserve more than 50GB of storage space

Installation Guide

conda virtual environment

Use the conda virtual environment, refer to Anaconda to manage your dependencies, after installation, execute the following commands:

conda create -n motion_agent python=3.8
conda activate motion_agent

GIT_LFS_SKIP_SMUDGE=1 git clone https://github.com/modelscope/motionagent.git --depth 1
cd motionagent

# Install dependencies
pip3 install -r requirements.txt

# Run the application
python3 app.py

# Note: MotionAgent currently supports single-card GPU, if your environment has multiple cards, please use the following command
# CUDA_VISIBLE_DEVICES=0 python3 app.py
# Note: If you are using the Modelscope community Notebook or if your disk memory is less than 100GB, please turn on the clear_cache switch. Each run will result in re-downloading the model, causing a significant decrease in speed. Please be patient and wait.
# python3 app.py --clear_cache

# Finally, click on the URL generated in the log to access the page.

Model List

[1] Qwen-7B-Chat： Model | Space

[2] SDXL 1.0：Model | Space

[3] I2VGen-XL： Model | Space

[4] MusicGen： Model | Space

More Information

• ModelScope library

  ModelScope Library is a model ecosystem repository hosted on github, belonging to the Damo Academy Moda project.

• Contribute models to ModelScope

License

This project is licensed under the Apache License (Version 2.0).