mirror of
https://github.com/Wan-Video/Wan2.1.git
synced 2025-11-04 22:26:36 +00:00
Added Ollama support for prompt extension
This commit is contained in:
Egar Almeida
parent
a326079926
commit
362fbc4ff3
833
README.md
833
README.md
@ -1,404 +1,429 @@
|
|||||||
# Wan2.1
|
# Wan2.1
|
||||||
|
|
||||||
<p align="center">
|
<p align="center">
|
||||||
<img src="assets/logo.png" width="400"/>
|
<img src="assets/logo.png" width="400"/>
|
||||||
<p>
|
<p>
|
||||||
|
|
||||||
<p align="center">
|
<p align="center">
|
||||||
💜 <a href=""><b>Wan</b></a>    ｜    🖥️ <a href="https://github.com/Wan-Video/Wan2.1">GitHub</a>    |   🤗 <a href="https://huggingface.co/Wan-AI/">Hugging Face</a>   |   🤖 <a href="https://modelscope.cn/organization/Wan-AI">ModelScope</a>   |    📑 <a href="">Paper (Coming soon)</a>    |    📑 <a href="https://wanxai.com">Blog</a>    |   💬 <a href="https://gw.alicdn.com/imgextra/i2/O1CN01tqjWFi1ByuyehkTSB_!!6000000000015-0-tps-611-1279.jpg">WeChat Group</a>   |    📖 <a href="https://discord.gg/p5XbdQV7">Discord</a>  
|
💜 <a href=""><b>Wan</b></a>    ｜    🖥️ <a href="https://github.com/Wan-Video/Wan2.1">GitHub</a>    |   🤗 <a href="https://huggingface.co/Wan-AI/">Hugging Face</a>   |   🤖 <a href="https://modelscope.cn/organization/Wan-AI">ModelScope</a>   |    📑 <a href="">Paper (Coming soon)</a>    |    📑 <a href="https://wanxai.com">Blog</a>    |   💬 <a href="https://gw.alicdn.com/imgextra/i2/O1CN01tqjWFi1ByuyehkTSB_!!6000000000015-0-tps-611-1279.jpg">WeChat Group</a>   |    📖 <a href="https://discord.gg/p5XbdQV7">Discord</a>  
|
||||||
<br>
|
<br>
|
||||||
|
|
||||||
-----
|
-----
|
||||||
|
|
||||||
[**Wan: Open and Advanced Large-Scale Video Generative Models**]("") <be>
|
[**Wan: Open and Advanced Large-Scale Video Generative Models**]("") <be>
|
||||||
|
|
||||||
In this repository, we present **Wan2.1**, a comprehensive and open suite of video foundation models that pushes the boundaries of video generation. **Wan2.1** offers these key features:
|
In this repository, we present **Wan2.1**, a comprehensive and open suite of video foundation models that pushes the boundaries of video generation. **Wan2.1** offers these key features:
|
||||||
- 👍 **SOTA Performance**: **Wan2.1** consistently outperforms existing open-source models and state-of-the-art commercial solutions across multiple benchmarks.
|
- 👍 **SOTA Performance**: **Wan2.1** consistently outperforms existing open-source models and state-of-the-art commercial solutions across multiple benchmarks.
|
||||||
- 👍 **Supports Consumer-grade GPUs**: The T2V-1.3B model requires only 8.19 GB VRAM, making it compatible with almost all consumer-grade GPUs. It can generate a 5-second 480P video on an RTX 4090 in about 4 minutes (without optimization techniques like quantization). Its performance is even comparable to some closed-source models.
|
- 👍 **Supports Consumer-grade GPUs**: The T2V-1.3B model requires only 8.19 GB VRAM, making it compatible with almost all consumer-grade GPUs. It can generate a 5-second 480P video on an RTX 4090 in about 4 minutes (without optimization techniques like quantization). Its performance is even comparable to some closed-source models.
|
||||||
- 👍 **Multiple Tasks**: **Wan2.1** excels in Text-to-Video, Image-to-Video, Video Editing, Text-to-Image, and Video-to-Audio, advancing the field of video generation.
|
- 👍 **Multiple Tasks**: **Wan2.1** excels in Text-to-Video, Image-to-Video, Video Editing, Text-to-Image, and Video-to-Audio, advancing the field of video generation.
|
||||||
- 👍 **Visual Text Generation**: **Wan2.1** is the first video model capable of generating both Chinese and English text, featuring robust text generation that enhances its practical applications.
|
- 👍 **Visual Text Generation**: **Wan2.1** is the first video model capable of generating both Chinese and English text, featuring robust text generation that enhances its practical applications.
|
||||||
- 👍 **Powerful Video VAE**: **Wan-VAE** delivers exceptional efficiency and performance, encoding and decoding 1080P videos of any length while preserving temporal information, making it an ideal foundation for video and image generation.
|
- 👍 **Powerful Video VAE**: **Wan-VAE** delivers exceptional efficiency and performance, encoding and decoding 1080P videos of any length while preserving temporal information, making it an ideal foundation for video and image generation.
|
||||||
|
|
||||||
## Video Demos
|
## Video Demos
|
||||||
|
|
||||||
<div align="center">
|
<div align="center">
|
||||||
<video src="https://github.com/user-attachments/assets/4aca6063-60bf-4953-bfb7-e265053f49ef" width="70%" poster=""> </video>
|
<video src="https://github.com/user-attachments/assets/4aca6063-60bf-4953-bfb7-e265053f49ef" width="70%" poster=""> </video>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
## 🔥 Latest News!!
|
## 🔥 Latest News!!
|
||||||
|
|
||||||
* Feb 25, 2025: 👋 We've released the inference code and weights of Wan2.1.
|
* Feb 25, 2025: 👋 We've released the inference code and weights of Wan2.1.
|
||||||
* Feb 27, 2025: 👋 Wan2.1 has been integrated into [ComfyUI](https://comfyanonymous.github.io/ComfyUI_examples/wan/). Enjoy!
|
* Feb 27, 2025: 👋 Wan2.1 has been integrated into [ComfyUI](https://comfyanonymous.github.io/ComfyUI_examples/wan/). Enjoy!
|
||||||
|
|
||||||
|
|
||||||
## 📑 Todo List
|
## 📑 Todo List
|
||||||
- Wan2.1 Text-to-Video
|
- Wan2.1 Text-to-Video
|
||||||
- [x] Multi-GPU Inference code of the 14B and 1.3B models
|
- [x] Multi-GPU Inference code of the 14B and 1.3B models
|
||||||
- [x] Checkpoints of the 14B and 1.3B models
|
- [x] Checkpoints of the 14B and 1.3B models
|
||||||
- [x] Gradio demo
|
- [x] Gradio demo
|
||||||
- [x] ComfyUI integration
|
- [x] ComfyUI integration
|
||||||
- [ ] Diffusers integration
|
- [ ] Diffusers integration
|
||||||
- Wan2.1 Image-to-Video
|
- Wan2.1 Image-to-Video
|
||||||
- [x] Multi-GPU Inference code of the 14B model
|
- [x] Multi-GPU Inference code of the 14B model
|
||||||
- [x] Checkpoints of the 14B model
|
- [x] Checkpoints of the 14B model
|
||||||
- [x] Gradio demo
|
- [x] Gradio demo
|
||||||
- [X] ComfyUI integration
|
- [X] ComfyUI integration
|
||||||
- [ ] Diffusers integration
|
- [ ] Diffusers integration
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
## Quickstart
|
## Quickstart
|
||||||
|
|
||||||
#### Installation
|
#### Installation
|
||||||
Clone the repo:
|
Clone the repo:
|
||||||
```
|
```
|
||||||
git clone https://github.com/Wan-Video/Wan2.1.git
|
git clone https://github.com/Wan-Video/Wan2.1.git
|
||||||
cd Wan2.1
|
cd Wan2.1
|
||||||
```
|
```
|
||||||
|
|
||||||
Install dependencies:
|
Install dependencies:
|
||||||
```
|
```
|
||||||
# Ensure torch >= 2.4.0
|
# Ensure torch >= 2.4.0
|
||||||
pip install -r requirements.txt
|
pip install -r requirements.txt
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
||||||
#### Model Download
|
#### Model Download
|
||||||
|
|
||||||
| Models | Download Link | Notes |
|
| Models | Download Link | Notes |
|
||||||
| --------------|-------------------------------------------------------------------------------|-------------------------------|
|
| --------------|-------------------------------------------------------------------------------|-------------------------------|
|
||||||
| T2V-14B | 🤗 [Huggingface](https://huggingface.co/Wan-AI/Wan2.1-T2V-14B) 🤖 [ModelScope](https://www.modelscope.cn/models/Wan-AI/Wan2.1-T2V-14B) | Supports both 480P and 720P
|
| T2V-14B | 🤗 [Huggingface](https://huggingface.co/Wan-AI/Wan2.1-T2V-14B) 🤖 [ModelScope](https://www.modelscope.cn/models/Wan-AI/Wan2.1-T2V-14B) | Supports both 480P and 720P
|
||||||
| I2V-14B-720P | 🤗 [Huggingface](https://huggingface.co/Wan-AI/Wan2.1-I2V-14B-720P) 🤖 [ModelScope](https://www.modelscope.cn/models/Wan-AI/Wan2.1-I2V-14B-720P) | Supports 720P
|
| I2V-14B-720P | 🤗 [Huggingface](https://huggingface.co/Wan-AI/Wan2.1-I2V-14B-720P) 🤖 [ModelScope](https://www.modelscope.cn/models/Wan-AI/Wan2.1-I2V-14B-720P) | Supports 720P
|
||||||
| I2V-14B-480P | 🤗 [Huggingface](https://huggingface.co/Wan-AI/Wan2.1-I2V-14B-480P) 🤖 [ModelScope](https://www.modelscope.cn/models/Wan-AI/Wan2.1-I2V-14B-480P) | Supports 480P
|
| I2V-14B-480P | 🤗 [Huggingface](https://huggingface.co/Wan-AI/Wan2.1-I2V-14B-480P) 🤖 [ModelScope](https://www.modelscope.cn/models/Wan-AI/Wan2.1-I2V-14B-480P) | Supports 480P
|
||||||
| T2V-1.3B | 🤗 [Huggingface](https://huggingface.co/Wan-AI/Wan2.1-T2V-1.3B) 🤖 [ModelScope](https://www.modelscope.cn/models/Wan-AI/Wan2.1-T2V-1.3B) | Supports 480P
|
| T2V-1.3B | 🤗 [Huggingface](https://huggingface.co/Wan-AI/Wan2.1-T2V-1.3B) 🤖 [ModelScope](https://www.modelscope.cn/models/Wan-AI/Wan2.1-T2V-1.3B) | Supports 480P
|
||||||
|
|
||||||
> 💡Note: The 1.3B model is capable of generating videos at 720P resolution. However, due to limited training at this resolution, the results are generally less stable compared to 480P. For optimal performance, we recommend using 480P resolution.
|
> 💡Note: The 1.3B model is capable of generating videos at 720P resolution. However, due to limited training at this resolution, the results are generally less stable compared to 480P. For optimal performance, we recommend using 480P resolution.
|
||||||
|
|
||||||
|
|
||||||
Download models using huggingface-cli:
|
Download models using huggingface-cli:
|
||||||
```
|
```
|
||||||
pip install "huggingface_hub[cli]"
|
pip install "huggingface_hub[cli]"
|
||||||
huggingface-cli download Wan-AI/Wan2.1-T2V-14B --local-dir ./Wan2.1-T2V-14B
|
huggingface-cli download Wan-AI/Wan2.1-T2V-14B --local-dir ./Wan2.1-T2V-14B
|
||||||
```
|
```
|
||||||
|
|
||||||
Download models using modelscope-cli:
|
Download models using modelscope-cli:
|
||||||
```
|
```
|
||||||
pip install modelscope
|
pip install modelscope
|
||||||
modelscope download Wan-AI/Wan2.1-T2V-14B --local_dir ./Wan2.1-T2V-14B
|
modelscope download Wan-AI/Wan2.1-T2V-14B --local_dir ./Wan2.1-T2V-14B
|
||||||
```
|
```
|
||||||
#### Run Text-to-Video Generation
|
#### Run Text-to-Video Generation
|
||||||
|
|
||||||
This repository supports two Text-to-Video models (1.3B and 14B) and two resolutions (480P and 720P). The parameters and configurations for these models are as follows:
|
This repository supports two Text-to-Video models (1.3B and 14B) and two resolutions (480P and 720P). The parameters and configurations for these models are as follows:
|
||||||
|
|
||||||
<table>
|
<table>
|
||||||
<thead>
|
<thead>
|
||||||
<tr>
|
<tr>
|
||||||
<th rowspan="2">Task</th>
|
<th rowspan="2">Task</th>
|
||||||
<th colspan="2">Resolution</th>
|
<th colspan="2">Resolution</th>
|
||||||
<th rowspan="2">Model</th>
|
<th rowspan="2">Model</th>
|
||||||
</tr>
|
</tr>
|
||||||
<tr>
|
<tr>
|
||||||
<th>480P</th>
|
<th>480P</th>
|
||||||
<th>720P</th>
|
<th>720P</th>
|
||||||
</tr>
|
</tr>
|
||||||
</thead>
|
</thead>
|
||||||
<tbody>
|
<tbody>
|
||||||
<tr>
|
<tr>
|
||||||
<td>t2v-14B</td>
|
<td>t2v-14B</td>
|
||||||
<td style="color: green;">✔️</td>
|
<td style="color: green;">✔️</td>
|
||||||
<td style="color: green;">✔️</td>
|
<td style="color: green;">✔️</td>
|
||||||
<td>Wan2.1-T2V-14B</td>
|
<td>Wan2.1-T2V-14B</td>
|
||||||
</tr>
|
</tr>
|
||||||
<tr>
|
<tr>
|
||||||
<td>t2v-1.3B</td>
|
<td>t2v-1.3B</td>
|
||||||
<td style="color: green;">✔️</td>
|
<td style="color: green;">✔️</td>
|
||||||
<td style="color: red;">❌</td>
|
<td style="color: red;">❌</td>
|
||||||
<td>Wan2.1-T2V-1.3B</td>
|
<td>Wan2.1-T2V-1.3B</td>
|
||||||
</tr>
|
</tr>
|
||||||
</tbody>
|
</tbody>
|
||||||
</table>
|
</table>
|
||||||
|
|
||||||
|
|
||||||
##### (1) Without Prompt Extension
|
##### (1) Without Prompt Extension
|
||||||
|
|
||||||
To facilitate implementation, we will start with a basic version of the inference process that skips the [prompt extension](#2-using-prompt-extention) step.
|
To facilitate implementation, we will start with a basic version of the inference process that skips the [prompt extension](#2-using-prompt-extention) step.
|
||||||
|
|
||||||
- Single-GPU inference
|
- Single-GPU inference
|
||||||
|
|
||||||
```
|
```
|
||||||
python generate.py --task t2v-14B --size 1280*720 --ckpt_dir ./Wan2.1-T2V-14B --prompt "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage."
|
python generate.py --task t2v-14B --size 1280*720 --ckpt_dir ./Wan2.1-T2V-14B --prompt "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage."
|
||||||
```
|
```
|
||||||
|
|
||||||
If you encounter OOM (Out-of-Memory) issues, you can use the `--offload_model True` and `--t5_cpu` options to reduce GPU memory usage. For example, on an RTX 4090 GPU:
|
If you encounter OOM (Out-of-Memory) issues, you can use the `--offload_model True` and `--t5_cpu` options to reduce GPU memory usage. For example, on an RTX 4090 GPU:
|
||||||
|
|
||||||
```
|
```
|
||||||
python generate.py --task t2v-1.3B --size 832*480 --ckpt_dir ./Wan2.1-T2V-1.3B --offload_model True --t5_cpu --sample_shift 8 --sample_guide_scale 6 --prompt "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage."
|
python generate.py --task t2v-1.3B --size 832*480 --ckpt_dir ./Wan2.1-T2V-1.3B --offload_model True --t5_cpu --sample_shift 8 --sample_guide_scale 6 --prompt "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage."
|
||||||
```
|
```
|
||||||
|
|
||||||
> 💡Note: If you are using the `T2V-1.3B` model, we recommend setting the parameter `--sample_guide_scale 6`. The `--sample_shift parameter` can be adjusted within the range of 8 to 12 based on the performance.
|
> 💡Note: If you are using the `T2V-1.3B` model, we recommend setting the parameter `--sample_guide_scale 6`. The `--sample_shift parameter` can be adjusted within the range of 8 to 12 based on the performance.
|
||||||
|
|
||||||
|
|
||||||
- Multi-GPU inference using FSDP + xDiT USP
|
- Multi-GPU inference using FSDP + xDiT USP
|
||||||
|
|
||||||
```
|
```
|
||||||
pip install "xfuser>=0.4.1"
|
pip install "xfuser>=0.4.1"
|
||||||
torchrun --nproc_per_node=8 generate.py --task t2v-14B --size 1280*720 --ckpt_dir ./Wan2.1-T2V-14B --dit_fsdp --t5_fsdp --ulysses_size 8 --prompt "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage."
|
torchrun --nproc_per_node=8 generate.py --task t2v-14B --size 1280*720 --ckpt_dir ./Wan2.1-T2V-14B --dit_fsdp --t5_fsdp --ulysses_size 8 --prompt "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage."
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
||||||
##### (2) Using Prompt Extension
|
##### (2) Using Prompt Extension
|
||||||
|
|
||||||
Extending the prompts can effectively enrich the details in the generated videos, further enhancing the video quality. Therefore, we recommend enabling prompt extension. We provide the following two methods for prompt extension:
|
Extending the prompts can effectively enrich the details in the generated videos, further enhancing the video quality. Therefore, we recommend enabling prompt extension. We provide the following two methods for prompt extension:
|
||||||
|
|
||||||
- Use the Dashscope API for extension.
|
- Use the Dashscope API for extension.
|
||||||
- Apply for a `dashscope.api_key` in advance ([EN](https://www.alibabacloud.com/help/en/model-studio/getting-started/first-api-call-to-qwen) | [CN](https://help.aliyun.com/zh/model-studio/getting-started/first-api-call-to-qwen)).
|
- Apply for a `dashscope.api_key` in advance ([EN](https://www.alibabacloud.com/help/en/model-studio/getting-started/first-api-call-to-qwen) | [CN](https://help.aliyun.com/zh/model-studio/getting-started/first-api-call-to-qwen)).
|
||||||
- Configure the environment variable `DASH_API_KEY` to specify the Dashscope API key. For users of Alibaba Cloud's international site, you also need to set the environment variable `DASH_API_URL` to 'https://dashscope-intl.aliyuncs.com/api/v1'. For more detailed instructions, please refer to the [dashscope document](https://www.alibabacloud.com/help/en/model-studio/developer-reference/use-qwen-by-calling-api?spm=a2c63.p38356.0.i1).
|
- Configure the environment variable `DASH_API_KEY` to specify the Dashscope API key. For users of Alibaba Cloud's international site, you also need to set the environment variable `DASH_API_URL` to 'https://dashscope-intl.aliyuncs.com/api/v1'. For more detailed instructions, please refer to the [dashscope document](https://www.alibabacloud.com/help/en/model-studio/developer-reference/use-qwen-by-calling-api?spm=a2c63.p38356.0.i1).
|
||||||
- Use the `qwen-plus` model for text-to-video tasks and `qwen-vl-max` for image-to-video tasks.
|
- Use the `qwen-plus` model for text-to-video tasks and `qwen-vl-max` for image-to-video tasks.
|
||||||
- You can modify the model used for extension with the parameter `--prompt_extend_model`. For example:
|
- You can modify the model used for extension with the parameter `--prompt_extend_model`. For example:
|
||||||
```
|
```
|
||||||
DASH_API_KEY=your_key python generate.py --task t2v-14B --size 1280*720 --ckpt_dir ./Wan2.1-T2V-14B --prompt "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage" --use_prompt_extend --prompt_extend_method 'dashscope' --prompt_extend_target_lang 'ch'
|
DASH_API_KEY=your_key python generate.py --task t2v-14B --size 1280*720 --ckpt_dir ./Wan2.1-T2V-14B --prompt "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage" --use_prompt_extend --prompt_extend_method 'dashscope' --prompt_extend_target_lang 'ch'
|
||||||
```
|
```
|
||||||
|
|
||||||
- Using a local model for extension.
|
- Using a local model for extension.
|
||||||
|
|
||||||
- By default, the Qwen model on HuggingFace is used for this extension. Users can choose Qwen models or other models based on the available GPU memory size.
|
- By default, the Qwen model on HuggingFace is used for this extension. Users can choose Qwen models or other models based on the available GPU memory size.
|
||||||
- For text-to-video tasks, you can use models like `Qwen/Qwen2.5-14B-Instruct`, `Qwen/Qwen2.5-7B-Instruct` and `Qwen/Qwen2.5-3B-Instruct`.
|
- For text-to-video tasks, you can use models like `Qwen/Qwen2.5-14B-Instruct`, `Qwen/Qwen2.5-7B-Instruct` and `Qwen/Qwen2.5-3B-Instruct`.
|
||||||
- For image-to-video tasks, you can use models like `Qwen/Qwen2.5-VL-7B-Instruct` and `Qwen/Qwen2.5-VL-3B-Instruct`.
|
- For image-to-video tasks, you can use models like `Qwen/Qwen2.5-VL-7B-Instruct` and `Qwen/Qwen2.5-VL-3B-Instruct`.
|
||||||
- Larger models generally provide better extension results but require more GPU memory.
|
- Larger models generally provide better extension results but require more GPU memory.
|
||||||
- You can modify the model used for extension with the parameter `--prompt_extend_model` , allowing you to specify either a local model path or a Hugging Face model. For example:
|
- You can modify the model used for extension with the parameter `--prompt_extend_model` , allowing you to specify either a local model path or a Hugging Face model. For example:
|
||||||
|
|
||||||
```
|
```
|
||||||
python generate.py --task t2v-14B --size 1280*720 --ckpt_dir ./Wan2.1-T2V-14B --prompt "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage" --use_prompt_extend --prompt_extend_method 'local_qwen' --prompt_extend_target_lang 'ch'
|
python generate.py --task t2v-14B --size 1280*720 --ckpt_dir ./Wan2.1-T2V-14B --prompt "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage" --use_prompt_extend --prompt_extend_method 'local_qwen' --prompt_extend_target_lang 'ch'
|
||||||
```
|
```
|
||||||
|
- Using Ollama for extension (Local API).
|
||||||
##### (3) Running local gradio
|
- Ensure Ollama is installed and running
|
||||||
|
- Pull your desired model: `ollama pull qwen2.5`
|
||||||
```
|
- The model name is case-sensitive and should match exactly what you pulled in Ollama
|
||||||
cd gradio
|
- For text-to-video tasks:
|
||||||
# if one uses dashscope’s API for prompt extension
|
```bash
|
||||||
DASH_API_KEY=your_key python t2v_14B_singleGPU.py --prompt_extend_method 'dashscope' --ckpt_dir ./Wan2.1-T2V-14B
|
python generate.py --task t2v-14B --size 1280*720 --ckpt_dir ./Wan2.1-T2V-14B \
|
||||||
|
--prompt "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage" \
|
||||||
# if one uses a local model for prompt extension
|
--use_prompt_extend \
|
||||||
python t2v_14B_singleGPU.py --prompt_extend_method 'local_qwen' --ckpt_dir ./Wan2.1-T2V-14B
|
--prompt_extend_method 'ollama' \
|
||||||
```
|
--prompt_extend_model 'qwen2.5'
|
||||||
|
```
|
||||||
|
- For image-to-video tasks:
|
||||||
#### Run Image-to-Video Generation
|
```bash
|
||||||
|
python generate.py --task i2v-14B --size 1280*720 --ckpt_dir ./Wan2.1-I2V-14B-720P \
|
||||||
Similar to Text-to-Video, Image-to-Video is also divided into processes with and without the prompt extension step. The specific parameters and their corresponding settings are as follows:
|
--image examples/i2v_input.JPG \
|
||||||
<table>
|
--prompt "Your prompt here" \
|
||||||
<thead>
|
--use_prompt_extend \
|
||||||
<tr>
|
--prompt_extend_method 'ollama' \
|
||||||
<th rowspan="2">Task</th>
|
--prompt_extend_model 'qwen2.5'
|
||||||
<th colspan="2">Resolution</th>
|
```
|
||||||
<th rowspan="2">Model</th>
|
- Optional: Customize the Ollama API URL if not using default (http://localhost:11434):
|
||||||
</tr>
|
```bash
|
||||||
<tr>
|
python generate.py [...other args...] --ollama_api_url 'http://your-ollama-server:11434'
|
||||||
<th>480P</th>
|
```
|
||||||
<th>720P</th>
|
|
||||||
</tr>
|
##### (3) Running local gradio
|
||||||
</thead>
|
|
||||||
<tbody>
|
```
|
||||||
<tr>
|
cd gradio
|
||||||
<td>i2v-14B</td>
|
# if one uses dashscope’s API for prompt extension
|
||||||
<td style="color: green;">❌</td>
|
DASH_API_KEY=your_key python t2v_14B_singleGPU.py --prompt_extend_method 'dashscope' --ckpt_dir ./Wan2.1-T2V-14B
|
||||||
<td style="color: green;">✔️</td>
|
|
||||||
<td>Wan2.1-I2V-14B-720P</td>
|
# if one uses a local model for prompt extension
|
||||||
</tr>
|
python t2v_14B_singleGPU.py --prompt_extend_method 'local_qwen' --ckpt_dir ./Wan2.1-T2V-14B
|
||||||
<tr>
|
```
|
||||||
<td>i2v-14B</td>
|
|
||||||
<td style="color: green;">✔️</td>
|
|
||||||
<td style="color: red;">❌</td>
|
#### Run Image-to-Video Generation
|
||||||
<td>Wan2.1-T2V-14B-480P</td>
|
|
||||||
</tr>
|
Similar to Text-to-Video, Image-to-Video is also divided into processes with and without the prompt extension step. The specific parameters and their corresponding settings are as follows:
|
||||||
</tbody>
|
<table>
|
||||||
</table>
|
<thead>
|
||||||
|
<tr>
|
||||||
|
<th rowspan="2">Task</th>
|
||||||
##### (1) Without Prompt Extension
|
<th colspan="2">Resolution</th>
|
||||||
|
<th rowspan="2">Model</th>
|
||||||
- Single-GPU inference
|
</tr>
|
||||||
```
|
<tr>
|
||||||
python generate.py --task i2v-14B --size 1280*720 --ckpt_dir ./Wan2.1-I2V-14B-720P --image examples/i2v_input.JPG --prompt "Summer beach vacation style, a white cat wearing sunglasses sits on a surfboard. The fluffy-furred feline gazes directly at the camera with a relaxed expression. Blurred beach scenery forms the background featuring crystal-clear waters, distant green hills, and a blue sky dotted with white clouds. The cat assumes a naturally relaxed posture, as if savoring the sea breeze and warm sunlight. A close-up shot highlights the feline's intricate details and the refreshing atmosphere of the seaside."
|
<th>480P</th>
|
||||||
```
|
<th>720P</th>
|
||||||
|
</tr>
|
||||||
> 💡For the Image-to-Video task, the `size` parameter represents the area of the generated video, with the aspect ratio following that of the original input image.
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr>
|
||||||
- Multi-GPU inference using FSDP + xDiT USP
|
<td>i2v-14B</td>
|
||||||
|
<td style="color: green;">❌</td>
|
||||||
```
|
<td style="color: green;">✔️</td>
|
||||||
pip install "xfuser>=0.4.1"
|
<td>Wan2.1-I2V-14B-720P</td>
|
||||||
torchrun --nproc_per_node=8 generate.py --task i2v-14B --size 1280*720 --ckpt_dir ./Wan2.1-I2V-14B-720P --image examples/i2v_input.JPG --dit_fsdp --t5_fsdp --ulysses_size 8 --prompt "Summer beach vacation style, a white cat wearing sunglasses sits on a surfboard. The fluffy-furred feline gazes directly at the camera with a relaxed expression. Blurred beach scenery forms the background featuring crystal-clear waters, distant green hills, and a blue sky dotted with white clouds. The cat assumes a naturally relaxed posture, as if savoring the sea breeze and warm sunlight. A close-up shot highlights the feline's intricate details and the refreshing atmosphere of the seaside."
|
</tr>
|
||||||
```
|
<tr>
|
||||||
|
<td>i2v-14B</td>
|
||||||
##### (2) Using Prompt Extension
|
<td style="color: green;">✔️</td>
|
||||||
|
<td style="color: red;">❌</td>
|
||||||
|
<td>Wan2.1-T2V-14B-480P</td>
|
||||||
The process of prompt extension can be referenced [here](#2-using-prompt-extention).
|
</tr>
|
||||||
|
</tbody>
|
||||||
Run with local prompt extension using `Qwen/Qwen2.5-VL-7B-Instruct`:
|
</table>
|
||||||
```
|
|
||||||
python generate.py --task i2v-14B --size 1280*720 --ckpt_dir ./Wan2.1-I2V-14B-720P --image examples/i2v_input.JPG --use_prompt_extend --prompt_extend_model Qwen/Qwen2.5-VL-7B-Instruct --prompt "Summer beach vacation style, a white cat wearing sunglasses sits on a surfboard. The fluffy-furred feline gazes directly at the camera with a relaxed expression. Blurred beach scenery forms the background featuring crystal-clear waters, distant green hills, and a blue sky dotted with white clouds. The cat assumes a naturally relaxed posture, as if savoring the sea breeze and warm sunlight. A close-up shot highlights the feline's intricate details and the refreshing atmosphere of the seaside."
|
|
||||||
```
|
##### (1) Without Prompt Extension
|
||||||
|
|
||||||
Run with remote prompt extension using `dashscope`:
|
- Single-GPU inference
|
||||||
```
|
```
|
||||||
DASH_API_KEY=your_key python generate.py --task i2v-14B --size 1280*720 --ckpt_dir ./Wan2.1-I2V-14B-720P --image examples/i2v_input.JPG --use_prompt_extend --prompt_extend_method 'dashscope' --prompt "Summer beach vacation style, a white cat wearing sunglasses sits on a surfboard. The fluffy-furred feline gazes directly at the camera with a relaxed expression. Blurred beach scenery forms the background featuring crystal-clear waters, distant green hills, and a blue sky dotted with white clouds. The cat assumes a naturally relaxed posture, as if savoring the sea breeze and warm sunlight. A close-up shot highlights the feline's intricate details and the refreshing atmosphere of the seaside."
|
python generate.py --task i2v-14B --size 1280*720 --ckpt_dir ./Wan2.1-I2V-14B-720P --image examples/i2v_input.JPG --prompt "Summer beach vacation style, a white cat wearing sunglasses sits on a surfboard. The fluffy-furred feline gazes directly at the camera with a relaxed expression. Blurred beach scenery forms the background featuring crystal-clear waters, distant green hills, and a blue sky dotted with white clouds. The cat assumes a naturally relaxed posture, as if savoring the sea breeze and warm sunlight. A close-up shot highlights the feline's intricate details and the refreshing atmosphere of the seaside."
|
||||||
```
|
```
|
||||||
|
|
||||||
##### (3) Running local gradio
|
> 💡For the Image-to-Video task, the `size` parameter represents the area of the generated video, with the aspect ratio following that of the original input image.
|
||||||
|
|
||||||
```
|
|
||||||
cd gradio
|
- Multi-GPU inference using FSDP + xDiT USP
|
||||||
# if one only uses 480P model in gradio
|
|
||||||
DASH_API_KEY=your_key python i2v_14B_singleGPU.py --prompt_extend_method 'dashscope' --ckpt_dir_480p ./Wan2.1-I2V-14B-480P
|
```
|
||||||
|
pip install "xfuser>=0.4.1"
|
||||||
# if one only uses 720P model in gradio
|
torchrun --nproc_per_node=8 generate.py --task i2v-14B --size 1280*720 --ckpt_dir ./Wan2.1-I2V-14B-720P --image examples/i2v_input.JPG --dit_fsdp --t5_fsdp --ulysses_size 8 --prompt "Summer beach vacation style, a white cat wearing sunglasses sits on a surfboard. The fluffy-furred feline gazes directly at the camera with a relaxed expression. Blurred beach scenery forms the background featuring crystal-clear waters, distant green hills, and a blue sky dotted with white clouds. The cat assumes a naturally relaxed posture, as if savoring the sea breeze and warm sunlight. A close-up shot highlights the feline's intricate details and the refreshing atmosphere of the seaside."
|
||||||
DASH_API_KEY=your_key python i2v_14B_singleGPU.py --prompt_extend_method 'dashscope' --ckpt_dir_720p ./Wan2.1-I2V-14B-720P
|
```
|
||||||
|
|
||||||
# if one uses both 480P and 720P models in gradio
|
##### (2) Using Prompt Extension
|
||||||
DASH_API_KEY=your_key python i2v_14B_singleGPU.py --prompt_extend_method 'dashscope' --ckpt_dir_480p ./Wan2.1-I2V-14B-480P --ckpt_dir_720p ./Wan2.1-I2V-14B-720P
|
|
||||||
```
|
|
||||||
|
The process of prompt extension can be referenced [here](#2-using-prompt-extention).
|
||||||
|
|
||||||
#### Run Text-to-Image Generation
|
Run with local prompt extension using `Qwen/Qwen2.5-VL-7B-Instruct`:
|
||||||
|
```
|
||||||
Wan2.1 is a unified model for both image and video generation. Since it was trained on both types of data, it can also generate images. The command for generating images is similar to video generation, as follows:
|
python generate.py --task i2v-14B --size 1280*720 --ckpt_dir ./Wan2.1-I2V-14B-720P --image examples/i2v_input.JPG --use_prompt_extend --prompt_extend_model Qwen/Qwen2.5-VL-7B-Instruct --prompt "Summer beach vacation style, a white cat wearing sunglasses sits on a surfboard. The fluffy-furred feline gazes directly at the camera with a relaxed expression. Blurred beach scenery forms the background featuring crystal-clear waters, distant green hills, and a blue sky dotted with white clouds. The cat assumes a naturally relaxed posture, as if savoring the sea breeze and warm sunlight. A close-up shot highlights the feline's intricate details and the refreshing atmosphere of the seaside."
|
||||||
|
```
|
||||||
##### (1) Without Prompt Extension
|
|
||||||
|
Run with remote prompt extension using `dashscope`:
|
||||||
- Single-GPU inference
|
```
|
||||||
```
|
DASH_API_KEY=your_key python generate.py --task i2v-14B --size 1280*720 --ckpt_dir ./Wan2.1-I2V-14B-720P --image examples/i2v_input.JPG --use_prompt_extend --prompt_extend_method 'dashscope' --prompt "Summer beach vacation style, a white cat wearing sunglasses sits on a surfboard. The fluffy-furred feline gazes directly at the camera with a relaxed expression. Blurred beach scenery forms the background featuring crystal-clear waters, distant green hills, and a blue sky dotted with white clouds. The cat assumes a naturally relaxed posture, as if savoring the sea breeze and warm sunlight. A close-up shot highlights the feline's intricate details and the refreshing atmosphere of the seaside."
|
||||||
python generate.py --task t2i-14B --size 1024*1024 --ckpt_dir ./Wan2.1-T2V-14B --prompt '一个朴素端庄的美人'
|
```
|
||||||
```
|
|
||||||
|
##### (3) Running local gradio
|
||||||
- Multi-GPU inference using FSDP + xDiT USP
|
|
||||||
|
```
|
||||||
```
|
cd gradio
|
||||||
torchrun --nproc_per_node=8 generate.py --dit_fsdp --t5_fsdp --ulysses_size 8 --base_seed 0 --frame_num 1 --task t2i-14B --size 1024*1024 --prompt '一个朴素端庄的美人' --ckpt_dir ./Wan2.1-T2V-14B
|
# if one only uses 480P model in gradio
|
||||||
```
|
DASH_API_KEY=your_key python i2v_14B_singleGPU.py --prompt_extend_method 'dashscope' --ckpt_dir_480p ./Wan2.1-I2V-14B-480P
|
||||||
|
|
||||||
##### (2) With Prompt Extention
|
# if one only uses 720P model in gradio
|
||||||
|
DASH_API_KEY=your_key python i2v_14B_singleGPU.py --prompt_extend_method 'dashscope' --ckpt_dir_720p ./Wan2.1-I2V-14B-720P
|
||||||
- Single-GPU inference
|
|
||||||
```
|
# if one uses both 480P and 720P models in gradio
|
||||||
python generate.py --task t2i-14B --size 1024*1024 --ckpt_dir ./Wan2.1-T2V-14B --prompt '一个朴素端庄的美人' --use_prompt_extend
|
DASH_API_KEY=your_key python i2v_14B_singleGPU.py --prompt_extend_method 'dashscope' --ckpt_dir_480p ./Wan2.1-I2V-14B-480P --ckpt_dir_720p ./Wan2.1-I2V-14B-720P
|
||||||
```
|
```
|
||||||
|
|
||||||
- Multi-GPU inference using FSDP + xDiT USP
|
|
||||||
```
|
#### Run Text-to-Image Generation
|
||||||
torchrun --nproc_per_node=8 generate.py --dit_fsdp --t5_fsdp --ulysses_size 8 --base_seed 0 --frame_num 1 --task t2i-14B --size 1024*1024 --ckpt_dir ./Wan2.1-T2V-14B --prompt '一个朴素端庄的美人' --use_prompt_extend
|
|
||||||
```
|
Wan2.1 is a unified model for both image and video generation. Since it was trained on both types of data, it can also generate images. The command for generating images is similar to video generation, as follows:
|
||||||
|
|
||||||
|
##### (1) Without Prompt Extension
|
||||||
## Manual Evaluation
|
|
||||||
|
- Single-GPU inference
|
||||||
##### (1) Text-to-Video Evaluation
|
```
|
||||||
|
python generate.py --task t2i-14B --size 1024*1024 --ckpt_dir ./Wan2.1-T2V-14B --prompt '一个朴素端庄的美人'
|
||||||
Through manual evaluation, the results generated after prompt extension are superior to those from both closed-source and open-source models.
|
```
|
||||||
|
|
||||||
<div align="center">
|
- Multi-GPU inference using FSDP + xDiT USP
|
||||||
<img src="assets/t2v_res.jpg" alt="" style="width: 80%;" />
|
|
||||||
</div>
|
```
|
||||||
|
torchrun --nproc_per_node=8 generate.py --dit_fsdp --t5_fsdp --ulysses_size 8 --base_seed 0 --frame_num 1 --task t2i-14B --size 1024*1024 --prompt '一个朴素端庄的美人' --ckpt_dir ./Wan2.1-T2V-14B
|
||||||
|
```
|
||||||
##### (2) Image-to-Video Evaluation
|
|
||||||
|
##### (2) With Prompt Extention
|
||||||
We also conducted extensive manual evaluations to evaluate the performance of the Image-to-Video model, and the results are presented in the table below. The results clearly indicate that **Wan2.1** outperforms both closed-source and open-source models.
|
|
||||||
|
- Single-GPU inference
|
||||||
<div align="center">
|
```
|
||||||
<img src="assets/i2v_res.png" alt="" style="width: 80%;" />
|
python generate.py --task t2i-14B --size 1024*1024 --ckpt_dir ./Wan2.1-T2V-14B --prompt '一个朴素端庄的美人' --use_prompt_extend
|
||||||
</div>
|
```
|
||||||
|
|
||||||
|
- Multi-GPU inference using FSDP + xDiT USP
|
||||||
## Computational Efficiency on Different GPUs
|
```
|
||||||
|
torchrun --nproc_per_node=8 generate.py --dit_fsdp --t5_fsdp --ulysses_size 8 --base_seed 0 --frame_num 1 --task t2i-14B --size 1024*1024 --ckpt_dir ./Wan2.1-T2V-14B --prompt '一个朴素端庄的美人' --use_prompt_extend
|
||||||
We test the computational efficiency of different **Wan2.1** models on different GPUs in the following table. The results are presented in the format: **Total time (s) / peak GPU memory (GB)**.
|
```
|
||||||
|
|
||||||
|
|
||||||
<div align="center">
|
## Manual Evaluation
|
||||||
<img src="assets/comp_effic.png" alt="" style="width: 80%;" />
|
|
||||||
</div>
|
##### (1) Text-to-Video Evaluation
|
||||||
|
|
||||||
> The parameter settings for the tests presented in this table are as follows:
|
Through manual evaluation, the results generated after prompt extension are superior to those from both closed-source and open-source models.
|
||||||
> (1) For the 1.3B model on 8 GPUs, set `--ring_size 8` and `--ulysses_size 1`;
|
|
||||||
> (2) For the 14B model on 1 GPU, use `--offload_model True`;
|
<div align="center">
|
||||||
> (3) For the 1.3B model on a single 4090 GPU, set `--offload_model True --t5_cpu`;
|
<img src="assets/t2v_res.jpg" alt="" style="width: 80%;" />
|
||||||
> (4) For all testings, no prompt extension was applied, meaning `--use_prompt_extend` was not enabled.
|
</div>
|
||||||
|
|
||||||
> 💡Note: T2V-14B is slower than I2V-14B because the former samples 50 steps while the latter uses 40 steps.
|
|
||||||
|
##### (2) Image-to-Video Evaluation
|
||||||
|
|
||||||
## Community Contributions
|
We also conducted extensive manual evaluations to evaluate the performance of the Image-to-Video model, and the results are presented in the table below. The results clearly indicate that **Wan2.1** outperforms both closed-source and open-source models.
|
||||||
- [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio) provides more support for **Wan2.1**, including video-to-video, FP8 quantization, VRAM optimization, LoRA training, and more. Please refer to [their examples](https://github.com/modelscope/DiffSynth-Studio/tree/main/examples/wanvideo).
|
|
||||||
|
<div align="center">
|
||||||
-------
|
<img src="assets/i2v_res.png" alt="" style="width: 80%;" />
|
||||||
|
</div>
|
||||||
## Introduction of Wan2.1
|
|
||||||
|
|
||||||
**Wan2.1** is designed on the mainstream diffusion transformer paradigm, achieving significant advancements in generative capabilities through a series of innovations. These include our novel spatio-temporal variational autoencoder (VAE), scalable training strategies, large-scale data construction, and automated evaluation metrics. Collectively, these contributions enhance the model’s performance and versatility.
|
## Computational Efficiency on Different GPUs
|
||||||
|
|
||||||
|
We test the computational efficiency of different **Wan2.1** models on different GPUs in the following table. The results are presented in the format: **Total time (s) / peak GPU memory (GB)**.
|
||||||
##### (1) 3D Variational Autoencoders
|
|
||||||
We propose a novel 3D causal VAE architecture, termed **Wan-VAE** specifically designed for video generation. By combining multiple strategies, we improve spatio-temporal compression, reduce memory usage, and ensure temporal causality. **Wan-VAE** demonstrates significant advantages in performance efficiency compared to other open-source VAEs. Furthermore, our **Wan-VAE** can encode and decode unlimited-length 1080P videos without losing historical temporal information, making it particularly well-suited for video generation tasks.
|
|
||||||
|
<div align="center">
|
||||||
|
<img src="assets/comp_effic.png" alt="" style="width: 80%;" />
|
||||||
<div align="center">
|
</div>
|
||||||
<img src="assets/video_vae_res.jpg" alt="" style="width: 80%;" />
|
|
||||||
</div>
|
> The parameter settings for the tests presented in this table are as follows:
|
||||||
|
> (1) For the 1.3B model on 8 GPUs, set `--ring_size 8` and `--ulysses_size 1`;
|
||||||
|
> (2) For the 14B model on 1 GPU, use `--offload_model True`;
|
||||||
##### (2) Video Diffusion DiT
|
> (3) For the 1.3B model on a single 4090 GPU, set `--offload_model True --t5_cpu`;
|
||||||
|
> (4) For all testings, no prompt extension was applied, meaning `--use_prompt_extend` was not enabled.
|
||||||
**Wan2.1** is designed using the Flow Matching framework within the paradigm of mainstream Diffusion Transformers. Our model's architecture uses the T5 Encoder to encode multilingual text input, with cross-attention in each transformer block embedding the text into the model structure. Additionally, we employ an MLP with a Linear layer and a SiLU layer to process the input time embeddings and predict six modulation parameters individually. This MLP is shared across all transformer blocks, with each block learning a distinct set of biases. Our experimental findings reveal a significant performance improvement with this approach at the same parameter scale.
|
|
||||||
|
> 💡Note: T2V-14B is slower than I2V-14B because the former samples 50 steps while the latter uses 40 steps.
|
||||||
<div align="center">
|
|
||||||
<img src="assets/video_dit_arch.jpg" alt="" style="width: 80%;" />
|
|
||||||
</div>
|
## Community Contributions
|
||||||
|
- [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio) provides more support for **Wan2.1**, including video-to-video, FP8 quantization, VRAM optimization, LoRA training, and more. Please refer to [their examples](https://github.com/modelscope/DiffSynth-Studio/tree/main/examples/wanvideo).
|
||||||
|
|
||||||
| Model | Dimension | Input Dimension | Output Dimension | Feedforward Dimension | Frequency Dimension | Number of Heads | Number of Layers |
|
-------
|
||||||
|--------|-----------|-----------------|------------------|-----------------------|---------------------|-----------------|------------------|
|
|
||||||
| 1.3B | 1536 | 16 | 16 | 8960 | 256 | 12 | 30 |
|
## Introduction of Wan2.1
|
||||||
| 14B | 5120 | 16 | 16 | 13824 | 256 | 40 | 40 |
|
|
||||||
|
**Wan2.1** is designed on the mainstream diffusion transformer paradigm, achieving significant advancements in generative capabilities through a series of innovations. These include our novel spatio-temporal variational autoencoder (VAE), scalable training strategies, large-scale data construction, and automated evaluation metrics. Collectively, these contributions enhance the model’s performance and versatility.
|
||||||
|
|
||||||
|
|
||||||
##### Data
|
##### (1) 3D Variational Autoencoders
|
||||||
|
We propose a novel 3D causal VAE architecture, termed **Wan-VAE** specifically designed for video generation. By combining multiple strategies, we improve spatio-temporal compression, reduce memory usage, and ensure temporal causality. **Wan-VAE** demonstrates significant advantages in performance efficiency compared to other open-source VAEs. Furthermore, our **Wan-VAE** can encode and decode unlimited-length 1080P videos without losing historical temporal information, making it particularly well-suited for video generation tasks.
|
||||||
We curated and deduplicated a candidate dataset comprising a vast amount of image and video data. During the data curation process, we designed a four-step data cleaning process, focusing on fundamental dimensions, visual quality and motion quality. Through the robust data processing pipeline, we can easily obtain high-quality, diverse, and large-scale training sets of images and videos.
|
|
||||||
|
|
||||||
|
<div align="center">
|
||||||
|
<img src="assets/video_vae_res.jpg" alt="" style="width: 80%;" />
|
||||||
|
</div>
|
||||||
##### Comparisons to SOTA
|
|
||||||
We compared **Wan2.1** with leading open-source and closed-source models to evaluate the performance. Using our carefully designed set of 1,035 internal prompts, we tested across 14 major dimensions and 26 sub-dimensions. We then compute the total score by performing a weighted calculation on the scores of each dimension, utilizing weights derived from human preferences in the matching process. The detailed results are shown in the table below. These results demonstrate our model's superior performance compared to both open-source and closed-source models.
|
|
||||||
|
##### (2) Video Diffusion DiT
|
||||||
|
|
||||||
|
**Wan2.1** is designed using the Flow Matching framework within the paradigm of mainstream Diffusion Transformers. Our model's architecture uses the T5 Encoder to encode multilingual text input, with cross-attention in each transformer block embedding the text into the model structure. Additionally, we employ an MLP with a Linear layer and a SiLU layer to process the input time embeddings and predict six modulation parameters individually. This MLP is shared across all transformer blocks, with each block learning a distinct set of biases. Our experimental findings reveal a significant performance improvement with this approach at the same parameter scale.
|
||||||
|
|
||||||
## Citation
|
<div align="center">
|
||||||
If you find our work helpful, please cite us.
|
<img src="assets/video_dit_arch.jpg" alt="" style="width: 80%;" />
|
||||||
|
</div>
|
||||||
```
|
|
||||||
@article{wan2.1,
|
|
||||||
title = {Wan: Open and Advanced Large-Scale Video Generative Models},
|
| Model | Dimension | Input Dimension | Output Dimension | Feedforward Dimension | Frequency Dimension | Number of Heads | Number of Layers |
|
||||||
author = {Wan Team},
|
|--------|-----------|-----------------|------------------|-----------------------|---------------------|-----------------|------------------|
|
||||||
journal = {},
|
| 1.3B | 1536 | 16 | 16 | 8960 | 256 | 12 | 30 |
|
||||||
year = {2025}
|
| 14B | 5120 | 16 | 16 | 13824 | 256 | 40 | 40 |
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## License Agreement
|
##### Data
|
||||||
The models in this repository are licensed under the Apache 2.0 License. We claim no rights over the your generated contents, granting you the freedom to use them while ensuring that your usage complies with the provisions of this license. You are fully accountable for your use of the models, which must not involve sharing any content that violates applicable laws, causes harm to individuals or groups, disseminates personal information intended for harm, spreads misinformation, or targets vulnerable populations. For a complete list of restrictions and details regarding your rights, please refer to the full text of the [license](LICENSE.txt).
|
|
||||||
|
We curated and deduplicated a candidate dataset comprising a vast amount of image and video data. During the data curation process, we designed a four-step data cleaning process, focusing on fundamental dimensions, visual quality and motion quality. Through the robust data processing pipeline, we can easily obtain high-quality, diverse, and large-scale training sets of images and videos.
|
||||||
|
|
||||||
## Acknowledgements
|
|
||||||
|
|
||||||
We would like to thank the contributors to the [SD3](https://huggingface.co/stabilityai/stable-diffusion-3-medium), [Qwen](https://huggingface.co/Qwen), [umt5-xxl](https://huggingface.co/google/umt5-xxl), [diffusers](https://github.com/huggingface/diffusers) and [HuggingFace](https://huggingface.co) repositories, for their open research.
|
|
||||||
|
##### Comparisons to SOTA
|
||||||
|
We compared **Wan2.1** with leading open-source and closed-source models to evaluate the performance. Using our carefully designed set of 1,035 internal prompts, we tested across 14 major dimensions and 26 sub-dimensions. We then compute the total score by performing a weighted calculation on the scores of each dimension, utilizing weights derived from human preferences in the matching process. The detailed results are shown in the table below. These results demonstrate our model's superior performance compared to both open-source and closed-source models.
|
||||||
|
|
||||||
## Contact Us
|
|
||||||
If you would like to leave a message to our research or product teams, feel free to join our [Discord](https://discord.gg/p5XbdQV7) or [WeChat groups](https://gw.alicdn.com/imgextra/i2/O1CN01tqjWFi1ByuyehkTSB_!!6000000000015-0-tps-611-1279.jpg)!
|
|
||||||
|
|
||||||
|
## Citation
|
||||||
|
If you find our work helpful, please cite us.
|
||||||
|
|
||||||
|
```
|
||||||
|
@article{wan2.1,
|
||||||
|
title = {Wan: Open and Advanced Large-Scale Video Generative Models},
|
||||||
|
author = {Wan Team},
|
||||||
|
journal = {},
|
||||||
|
year = {2025}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## License Agreement
|
||||||
|
The models in this repository are licensed under the Apache 2.0 License. We claim no rights over the your generated contents, granting you the freedom to use them while ensuring that your usage complies with the provisions of this license. You are fully accountable for your use of the models, which must not involve sharing any content that violates applicable laws, causes harm to individuals or groups, disseminates personal information intended for harm, spreads misinformation, or targets vulnerable populations. For a complete list of restrictions and details regarding your rights, please refer to the full text of the [license](LICENSE.txt).
|
||||||
|
|
||||||
|
|
||||||
|
## Acknowledgements
|
||||||
|
|
||||||
|
We would like to thank the contributors to the [SD3](https://huggingface.co/stabilityai/stable-diffusion-3-medium), [Qwen](https://huggingface.co/Qwen), [umt5-xxl](https://huggingface.co/google/umt5-xxl), [diffusers](https://github.com/huggingface/diffusers) and [HuggingFace](https://huggingface.co) repositories, for their open research.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## Contact Us
|
||||||
|
If you would like to leave a message to our research or product teams, feel free to join our [Discord](https://discord.gg/p5XbdQV7) or [WeChat groups](https://gw.alicdn.com/imgextra/i2/O1CN01tqjWFi1ByuyehkTSB_!!6000000000015-0-tps-611-1279.jpg)!
|
||||||
|
|||||||
834
generate.py
834
generate.py
@ -1,411 +1,423 @@
|
|||||||
# Copyright 2024-2025 The Alibaba Wan Team Authors. All rights reserved.
|
# Copyright 2024-2025 The Alibaba Wan Team Authors. All rights reserved.
|
||||||
import argparse
|
# Changelog:
|
||||||
from datetime import datetime
|
# 2025-03-01: Added Ollama support for prompt extension
|
||||||
import logging
|
|
||||||
import os
|
import argparse
|
||||||
import sys
|
from datetime import datetime
|
||||||
import warnings
|
import logging
|
||||||
|
import os
|
||||||
warnings.filterwarnings('ignore')
|
import sys
|
||||||
|
import warnings
|
||||||
import torch, random
|
|
||||||
import torch.distributed as dist
|
warnings.filterwarnings('ignore')
|
||||||
from PIL import Image
|
|
||||||
|
import torch, random
|
||||||
import wan
|
import torch.distributed as dist
|
||||||
from wan.configs import WAN_CONFIGS, SIZE_CONFIGS, MAX_AREA_CONFIGS, SUPPORTED_SIZES
|
from PIL import Image
|
||||||
from wan.utils.prompt_extend import DashScopePromptExpander, QwenPromptExpander
|
|
||||||
from wan.utils.utils import cache_video, cache_image, str2bool
|
import wan
|
||||||
|
from wan.configs import WAN_CONFIGS, SIZE_CONFIGS, MAX_AREA_CONFIGS, SUPPORTED_SIZES
|
||||||
EXAMPLE_PROMPT = {
|
from wan.utils.prompt_extend import DashScopePromptExpander, QwenPromptExpander, OllamaPromptExpander
|
||||||
"t2v-1.3B": {
|
from wan.utils.utils import cache_video, cache_image, str2bool
|
||||||
"prompt": "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage.",
|
|
||||||
},
|
EXAMPLE_PROMPT = {
|
||||||
"t2v-14B": {
|
"t2v-1.3B": {
|
||||||
"prompt": "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage.",
|
"prompt": "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage.",
|
||||||
},
|
},
|
||||||
"t2i-14B": {
|
"t2v-14B": {
|
||||||
"prompt": "一个朴素端庄的美人",
|
"prompt": "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage.",
|
||||||
},
|
},
|
||||||
"i2v-14B": {
|
"t2i-14B": {
|
||||||
"prompt":
|
"prompt": "一个朴素端庄的美人",
|
||||||
"Summer beach vacation style, a white cat wearing sunglasses sits on a surfboard. The fluffy-furred feline gazes directly at the camera with a relaxed expression. Blurred beach scenery forms the background featuring crystal-clear waters, distant green hills, and a blue sky dotted with white clouds. The cat assumes a naturally relaxed posture, as if savoring the sea breeze and warm sunlight. A close-up shot highlights the feline's intricate details and the refreshing atmosphere of the seaside.",
|
},
|
||||||
"image":
|
"i2v-14B": {
|
||||||
"examples/i2v_input.JPG",
|
"prompt":
|
||||||
},
|
"Summer beach vacation style, a white cat wearing sunglasses sits on a surfboard. The fluffy-furred feline gazes directly at the camera with a relaxed expression. Blurred beach scenery forms the background featuring crystal-clear waters, distant green hills, and a blue sky dotted with white clouds. The cat assumes a naturally relaxed posture, as if savoring the sea breeze and warm sunlight. A close-up shot highlights the feline's intricate details and the refreshing atmosphere of the seaside.",
|
||||||
}
|
"image":
|
||||||
|
"examples/i2v_input.JPG",
|
||||||
|
},
|
||||||
def _validate_args(args):
|
}
|
||||||
# Basic check
|
|
||||||
assert args.ckpt_dir is not None, "Please specify the checkpoint directory."
|
|
||||||
assert args.task in WAN_CONFIGS, f"Unsupport task: {args.task}"
|
def _validate_args(args):
|
||||||
assert args.task in EXAMPLE_PROMPT, f"Unsupport task: {args.task}"
|
# Basic check
|
||||||
|
assert args.ckpt_dir is not None, "Please specify the checkpoint directory."
|
||||||
# The default sampling steps are 40 for image-to-video tasks and 50 for text-to-video tasks.
|
assert args.task in WAN_CONFIGS, f"Unsupport task: {args.task}"
|
||||||
if args.sample_steps is None:
|
assert args.task in EXAMPLE_PROMPT, f"Unsupport task: {args.task}"
|
||||||
args.sample_steps = 40 if "i2v" in args.task else 50
|
|
||||||
|
# The default sampling steps are 40 for image-to-video tasks and 50 for text-to-video tasks.
|
||||||
if args.sample_shift is None:
|
if args.sample_steps is None:
|
||||||
args.sample_shift = 5.0
|
args.sample_steps = 40 if "i2v" in args.task else 50
|
||||||
if "i2v" in args.task and args.size in ["832*480", "480*832"]:
|
|
||||||
args.sample_shift = 3.0
|
if args.sample_shift is None:
|
||||||
|
args.sample_shift = 5.0
|
||||||
# The default number of frames are 1 for text-to-image tasks and 81 for other tasks.
|
if "i2v" in args.task and args.size in ["832*480", "480*832"]:
|
||||||
if args.frame_num is None:
|
args.sample_shift = 3.0
|
||||||
args.frame_num = 1 if "t2i" in args.task else 81
|
|
||||||
|
# The default number of frames are 1 for text-to-image tasks and 81 for other tasks.
|
||||||
# T2I frame_num check
|
if args.frame_num is None:
|
||||||
if "t2i" in args.task:
|
args.frame_num = 1 if "t2i" in args.task else 81
|
||||||
assert args.frame_num == 1, f"Unsupport frame_num {args.frame_num} for task {args.task}"
|
|
||||||
|
# T2I frame_num check
|
||||||
args.base_seed = args.base_seed if args.base_seed >= 0 else random.randint(
|
if "t2i" in args.task:
|
||||||
0, sys.maxsize)
|
assert args.frame_num == 1, f"Unsupport frame_num {args.frame_num} for task {args.task}"
|
||||||
# Size check
|
|
||||||
assert args.size in SUPPORTED_SIZES[
|
args.base_seed = args.base_seed if args.base_seed >= 0 else random.randint(
|
||||||
args.
|
0, sys.maxsize)
|
||||||
task], f"Unsupport size {args.size} for task {args.task}, supported sizes are: {', '.join(SUPPORTED_SIZES[args.task])}"
|
# Size check
|
||||||
|
assert args.size in SUPPORTED_SIZES[
|
||||||
|
args.
|
||||||
def _parse_args():
|
task], f"Unsupport size {args.size} for task {args.task}, supported sizes are: {', '.join(SUPPORTED_SIZES[args.task])}"
|
||||||
parser = argparse.ArgumentParser(
|
|
||||||
description="Generate a image or video from a text prompt or image using Wan"
|
|
||||||
)
|
def _parse_args():
|
||||||
parser.add_argument(
|
parser = argparse.ArgumentParser(
|
||||||
"--task",
|
description="Generate a image or video from a text prompt or image using Wan"
|
||||||
type=str,
|
)
|
||||||
default="t2v-14B",
|
parser.add_argument(
|
||||||
choices=list(WAN_CONFIGS.keys()),
|
"--task",
|
||||||
help="The task to run.")
|
type=str,
|
||||||
parser.add_argument(
|
default="t2v-14B",
|
||||||
"--size",
|
choices=list(WAN_CONFIGS.keys()),
|
||||||
type=str,
|
help="The task to run.")
|
||||||
default="1280*720",
|
parser.add_argument(
|
||||||
choices=list(SIZE_CONFIGS.keys()),
|
"--size",
|
||||||
help="The area (width*height) of the generated video. For the I2V task, the aspect ratio of the output video will follow that of the input image."
|
type=str,
|
||||||
)
|
default="1280*720",
|
||||||
parser.add_argument(
|
choices=list(SIZE_CONFIGS.keys()),
|
||||||
"--frame_num",
|
help="The area (width*height) of the generated video. For the I2V task, the aspect ratio of the output video will follow that of the input image."
|
||||||
type=int,
|
)
|
||||||
default=None,
|
parser.add_argument(
|
||||||
help="How many frames to sample from a image or video. The number should be 4n+1"
|
"--frame_num",
|
||||||
)
|
type=int,
|
||||||
parser.add_argument(
|
default=None,
|
||||||
"--ckpt_dir",
|
help="How many frames to sample from a image or video. The number should be 4n+1"
|
||||||
type=str,
|
)
|
||||||
default=None,
|
parser.add_argument(
|
||||||
help="The path to the checkpoint directory.")
|
"--ckpt_dir",
|
||||||
parser.add_argument(
|
type=str,
|
||||||
"--offload_model",
|
default=None,
|
||||||
type=str2bool,
|
help="The path to the checkpoint directory.")
|
||||||
default=None,
|
parser.add_argument(
|
||||||
help="Whether to offload the model to CPU after each model forward, reducing GPU memory usage."
|
"--offload_model",
|
||||||
)
|
type=str2bool,
|
||||||
parser.add_argument(
|
default=None,
|
||||||
"--ulysses_size",
|
help="Whether to offload the model to CPU after each model forward, reducing GPU memory usage."
|
||||||
type=int,
|
)
|
||||||
default=1,
|
parser.add_argument(
|
||||||
help="The size of the ulysses parallelism in DiT.")
|
"--ulysses_size",
|
||||||
parser.add_argument(
|
type=int,
|
||||||
"--ring_size",
|
default=1,
|
||||||
type=int,
|
help="The size of the ulysses parallelism in DiT.")
|
||||||
default=1,
|
parser.add_argument(
|
||||||
help="The size of the ring attention parallelism in DiT.")
|
"--ring_size",
|
||||||
parser.add_argument(
|
type=int,
|
||||||
"--t5_fsdp",
|
default=1,
|
||||||
action="store_true",
|
help="The size of the ring attention parallelism in DiT.")
|
||||||
default=False,
|
parser.add_argument(
|
||||||
help="Whether to use FSDP for T5.")
|
"--t5_fsdp",
|
||||||
parser.add_argument(
|
action="store_true",
|
||||||
"--t5_cpu",
|
default=False,
|
||||||
action="store_true",
|
help="Whether to use FSDP for T5.")
|
||||||
default=False,
|
parser.add_argument(
|
||||||
help="Whether to place T5 model on CPU.")
|
"--t5_cpu",
|
||||||
parser.add_argument(
|
action="store_true",
|
||||||
"--dit_fsdp",
|
default=False,
|
||||||
action="store_true",
|
help="Whether to place T5 model on CPU.")
|
||||||
default=False,
|
parser.add_argument(
|
||||||
help="Whether to use FSDP for DiT.")
|
"--dit_fsdp",
|
||||||
parser.add_argument(
|
action="store_true",
|
||||||
"--save_file",
|
default=False,
|
||||||
type=str,
|
help="Whether to use FSDP for DiT.")
|
||||||
default=None,
|
parser.add_argument(
|
||||||
help="The file to save the generated image or video to.")
|
"--save_file",
|
||||||
parser.add_argument(
|
type=str,
|
||||||
"--prompt",
|
default=None,
|
||||||
type=str,
|
help="The file to save the generated image or video to.")
|
||||||
default=None,
|
parser.add_argument(
|
||||||
help="The prompt to generate the image or video from.")
|
"--prompt",
|
||||||
parser.add_argument(
|
type=str,
|
||||||
"--use_prompt_extend",
|
default=None,
|
||||||
action="store_true",
|
help="The prompt to generate the image or video from.")
|
||||||
default=False,
|
parser.add_argument(
|
||||||
help="Whether to use prompt extend.")
|
"--use_prompt_extend",
|
||||||
parser.add_argument(
|
action="store_true",
|
||||||
"--prompt_extend_method",
|
default=False,
|
||||||
type=str,
|
help="Whether to use prompt extend.")
|
||||||
default="local_qwen",
|
parser.add_argument(
|
||||||
choices=["dashscope", "local_qwen"],
|
"--prompt_extend_method",
|
||||||
help="The prompt extend method to use.")
|
type=str,
|
||||||
parser.add_argument(
|
default="local_qwen",
|
||||||
"--prompt_extend_model",
|
choices=["dashscope", "local_qwen", "ollama"],
|
||||||
type=str,
|
help="The prompt extend method to use.")
|
||||||
default=None,
|
parser.add_argument(
|
||||||
help="The prompt extend model to use.")
|
"--ollama_api_url",
|
||||||
parser.add_argument(
|
type=str,
|
||||||
"--prompt_extend_target_lang",
|
default="http://localhost:11434",
|
||||||
type=str,
|
help="The URL of the Ollama API (only used with ollama method).")
|
||||||
default="ch",
|
parser.add_argument(
|
||||||
choices=["ch", "en"],
|
"--prompt_extend_model",
|
||||||
help="The target language of prompt extend.")
|
type=str,
|
||||||
parser.add_argument(
|
default=None,
|
||||||
"--base_seed",
|
help="The prompt extend model to use.")
|
||||||
type=int,
|
parser.add_argument(
|
||||||
default=-1,
|
"--prompt_extend_target_lang",
|
||||||
help="The seed to use for generating the image or video.")
|
type=str,
|
||||||
parser.add_argument(
|
default="ch",
|
||||||
"--image",
|
choices=["ch", "en"],
|
||||||
type=str,
|
help="The target language of prompt extend.")
|
||||||
default=None,
|
parser.add_argument(
|
||||||
help="The image to generate the video from.")
|
"--base_seed",
|
||||||
parser.add_argument(
|
type=int,
|
||||||
"--sample_solver",
|
default=-1,
|
||||||
type=str,
|
help="The seed to use for generating the image or video.")
|
||||||
default='unipc',
|
parser.add_argument(
|
||||||
choices=['unipc', 'dpm++'],
|
"--image",
|
||||||
help="The solver used to sample.")
|
type=str,
|
||||||
parser.add_argument(
|
default=None,
|
||||||
"--sample_steps", type=int, default=None, help="The sampling steps.")
|
help="The image to generate the video from.")
|
||||||
parser.add_argument(
|
parser.add_argument(
|
||||||
"--sample_shift",
|
"--sample_solver",
|
||||||
type=float,
|
type=str,
|
||||||
default=None,
|
default='unipc',
|
||||||
help="Sampling shift factor for flow matching schedulers.")
|
choices=['unipc', 'dpm++'],
|
||||||
parser.add_argument(
|
help="The solver used to sample.")
|
||||||
"--sample_guide_scale",
|
parser.add_argument(
|
||||||
type=float,
|
"--sample_steps", type=int, default=None, help="The sampling steps.")
|
||||||
default=5.0,
|
parser.add_argument(
|
||||||
help="Classifier free guidance scale.")
|
"--sample_shift",
|
||||||
|
type=float,
|
||||||
args = parser.parse_args()
|
default=None,
|
||||||
|
help="Sampling shift factor for flow matching schedulers.")
|
||||||
_validate_args(args)
|
parser.add_argument(
|
||||||
|
"--sample_guide_scale",
|
||||||
return args
|
type=float,
|
||||||
|
default=5.0,
|
||||||
|
help="Classifier free guidance scale.")
|
||||||
def _init_logging(rank):
|
|
||||||
# logging
|
args = parser.parse_args()
|
||||||
if rank == 0:
|
|
||||||
# set format
|
_validate_args(args)
|
||||||
logging.basicConfig(
|
|
||||||
level=logging.INFO,
|
return args
|
||||||
format="[%(asctime)s] %(levelname)s: %(message)s",
|
|
||||||
handlers=[logging.StreamHandler(stream=sys.stdout)])
|
|
||||||
else:
|
def _init_logging(rank):
|
||||||
logging.basicConfig(level=logging.ERROR)
|
# logging
|
||||||
|
if rank == 0:
|
||||||
|
# set format
|
||||||
def generate(args):
|
logging.basicConfig(
|
||||||
rank = int(os.getenv("RANK", 0))
|
level=logging.INFO,
|
||||||
world_size = int(os.getenv("WORLD_SIZE", 1))
|
format="[%(asctime)s] %(levelname)s: %(message)s",
|
||||||
local_rank = int(os.getenv("LOCAL_RANK", 0))
|
handlers=[logging.StreamHandler(stream=sys.stdout)])
|
||||||
device = local_rank
|
else:
|
||||||
_init_logging(rank)
|
logging.basicConfig(level=logging.ERROR)
|
||||||
|
|
||||||
if args.offload_model is None:
|
|
||||||
args.offload_model = False if world_size > 1 else True
|
def generate(args):
|
||||||
logging.info(
|
rank = int(os.getenv("RANK", 0))
|
||||||
f"offload_model is not specified, set to {args.offload_model}.")
|
world_size = int(os.getenv("WORLD_SIZE", 1))
|
||||||
if world_size > 1:
|
local_rank = int(os.getenv("LOCAL_RANK", 0))
|
||||||
torch.cuda.set_device(local_rank)
|
device = local_rank
|
||||||
dist.init_process_group(
|
_init_logging(rank)
|
||||||
backend="nccl",
|
|
||||||
init_method="env://",
|
if args.offload_model is None:
|
||||||
rank=rank,
|
args.offload_model = False if world_size > 1 else True
|
||||||
world_size=world_size)
|
logging.info(
|
||||||
else:
|
f"offload_model is not specified, set to {args.offload_model}.")
|
||||||
assert not (
|
if world_size > 1:
|
||||||
args.t5_fsdp or args.dit_fsdp
|
torch.cuda.set_device(local_rank)
|
||||||
), f"t5_fsdp and dit_fsdp are not supported in non-distributed environments."
|
dist.init_process_group(
|
||||||
assert not (
|
backend="nccl",
|
||||||
args.ulysses_size > 1 or args.ring_size > 1
|
init_method="env://",
|
||||||
), f"context parallel are not supported in non-distributed environments."
|
rank=rank,
|
||||||
|
world_size=world_size)
|
||||||
if args.ulysses_size > 1 or args.ring_size > 1:
|
else:
|
||||||
assert args.ulysses_size * args.ring_size == world_size, f"The number of ulysses_size and ring_size should be equal to the world size."
|
assert not (
|
||||||
from xfuser.core.distributed import (initialize_model_parallel,
|
args.t5_fsdp or args.dit_fsdp
|
||||||
init_distributed_environment)
|
), f"t5_fsdp and dit_fsdp are not supported in non-distributed environments."
|
||||||
init_distributed_environment(
|
assert not (
|
||||||
rank=dist.get_rank(), world_size=dist.get_world_size())
|
args.ulysses_size > 1 or args.ring_size > 1
|
||||||
|
), f"context parallel are not supported in non-distributed environments."
|
||||||
initialize_model_parallel(
|
|
||||||
sequence_parallel_degree=dist.get_world_size(),
|
if args.ulysses_size > 1 or args.ring_size > 1:
|
||||||
ring_degree=args.ring_size,
|
assert args.ulysses_size * args.ring_size == world_size, f"The number of ulysses_size and ring_size should be equal to the world size."
|
||||||
ulysses_degree=args.ulysses_size,
|
from xfuser.core.distributed import (initialize_model_parallel,
|
||||||
)
|
init_distributed_environment)
|
||||||
|
init_distributed_environment(
|
||||||
if args.use_prompt_extend:
|
rank=dist.get_rank(), world_size=dist.get_world_size())
|
||||||
if args.prompt_extend_method == "dashscope":
|
|
||||||
prompt_expander = DashScopePromptExpander(
|
initialize_model_parallel(
|
||||||
model_name=args.prompt_extend_model, is_vl="i2v" in args.task)
|
sequence_parallel_degree=dist.get_world_size(),
|
||||||
elif args.prompt_extend_method == "local_qwen":
|
ring_degree=args.ring_size,
|
||||||
prompt_expander = QwenPromptExpander(
|
ulysses_degree=args.ulysses_size,
|
||||||
model_name=args.prompt_extend_model,
|
)
|
||||||
is_vl="i2v" in args.task,
|
|
||||||
device=rank)
|
if args.use_prompt_extend:
|
||||||
else:
|
if args.prompt_extend_method == "dashscope":
|
||||||
raise NotImplementedError(
|
prompt_expander = DashScopePromptExpander(
|
||||||
f"Unsupport prompt_extend_method: {args.prompt_extend_method}")
|
model_name=args.prompt_extend_model, is_vl="i2v" in args.task)
|
||||||
|
elif args.prompt_extend_method == "local_qwen":
|
||||||
cfg = WAN_CONFIGS[args.task]
|
prompt_expander = QwenPromptExpander(
|
||||||
if args.ulysses_size > 1:
|
model_name=args.prompt_extend_model,
|
||||||
assert cfg.num_heads % args.ulysses_size == 0, f"`num_heads` must be divisible by `ulysses_size`."
|
is_vl="i2v" in args.task,
|
||||||
|
device=rank)
|
||||||
logging.info(f"Generation job args: {args}")
|
elif args.prompt_extend_method == "ollama":
|
||||||
logging.info(f"Generation model config: {cfg}")
|
prompt_expander = OllamaPromptExpander(
|
||||||
|
model_name=args.prompt_extend_model,
|
||||||
if dist.is_initialized():
|
api_url=args.ollama_api_url,
|
||||||
base_seed = [args.base_seed] if rank == 0 else [None]
|
is_vl="i2v" in args.task)
|
||||||
dist.broadcast_object_list(base_seed, src=0)
|
else:
|
||||||
args.base_seed = base_seed[0]
|
raise NotImplementedError(
|
||||||
|
f"Unsupport prompt_extend_method: {args.prompt_extend_method}")
|
||||||
if "t2v" in args.task or "t2i" in args.task:
|
|
||||||
if args.prompt is None:
|
cfg = WAN_CONFIGS[args.task]
|
||||||
args.prompt = EXAMPLE_PROMPT[args.task]["prompt"]
|
if args.ulysses_size > 1:
|
||||||
logging.info(f"Input prompt: {args.prompt}")
|
assert cfg.num_heads % args.ulysses_size == 0, f"`num_heads` must be divisible by `ulysses_size`."
|
||||||
if args.use_prompt_extend:
|
|
||||||
logging.info("Extending prompt ...")
|
logging.info(f"Generation job args: {args}")
|
||||||
if rank == 0:
|
logging.info(f"Generation model config: {cfg}")
|
||||||
prompt_output = prompt_expander(
|
|
||||||
args.prompt,
|
if dist.is_initialized():
|
||||||
tar_lang=args.prompt_extend_target_lang,
|
base_seed = [args.base_seed] if rank == 0 else [None]
|
||||||
seed=args.base_seed)
|
dist.broadcast_object_list(base_seed, src=0)
|
||||||
if prompt_output.status == False:
|
args.base_seed = base_seed[0]
|
||||||
logging.info(
|
|
||||||
f"Extending prompt failed: {prompt_output.message}")
|
if "t2v" in args.task or "t2i" in args.task:
|
||||||
logging.info("Falling back to original prompt.")
|
if args.prompt is None:
|
||||||
input_prompt = args.prompt
|
args.prompt = EXAMPLE_PROMPT[args.task]["prompt"]
|
||||||
else:
|
logging.info(f"Input prompt: {args.prompt}")
|
||||||
input_prompt = prompt_output.prompt
|
if args.use_prompt_extend:
|
||||||
input_prompt = [input_prompt]
|
logging.info("Extending prompt ...")
|
||||||
else:
|
if rank == 0:
|
||||||
input_prompt = [None]
|
prompt_output = prompt_expander(
|
||||||
if dist.is_initialized():
|
args.prompt,
|
||||||
dist.broadcast_object_list(input_prompt, src=0)
|
tar_lang=args.prompt_extend_target_lang,
|
||||||
args.prompt = input_prompt[0]
|
seed=args.base_seed)
|
||||||
logging.info(f"Extended prompt: {args.prompt}")
|
if prompt_output.status == False:
|
||||||
|
logging.info(
|
||||||
logging.info("Creating WanT2V pipeline.")
|
f"Extending prompt failed: {prompt_output.message}")
|
||||||
wan_t2v = wan.WanT2V(
|
logging.info("Falling back to original prompt.")
|
||||||
config=cfg,
|
input_prompt = args.prompt
|
||||||
checkpoint_dir=args.ckpt_dir,
|
else:
|
||||||
device_id=device,
|
input_prompt = prompt_output.prompt
|
||||||
rank=rank,
|
input_prompt = [input_prompt]
|
||||||
t5_fsdp=args.t5_fsdp,
|
else:
|
||||||
dit_fsdp=args.dit_fsdp,
|
input_prompt = [None]
|
||||||
use_usp=(args.ulysses_size > 1 or args.ring_size > 1),
|
if dist.is_initialized():
|
||||||
t5_cpu=args.t5_cpu,
|
dist.broadcast_object_list(input_prompt, src=0)
|
||||||
)
|
args.prompt = input_prompt[0]
|
||||||
|
logging.info(f"Extended prompt: {args.prompt}")
|
||||||
logging.info(
|
|
||||||
f"Generating {'image' if 't2i' in args.task else 'video'} ...")
|
logging.info("Creating WanT2V pipeline.")
|
||||||
video = wan_t2v.generate(
|
wan_t2v = wan.WanT2V(
|
||||||
args.prompt,
|
config=cfg,
|
||||||
size=SIZE_CONFIGS[args.size],
|
checkpoint_dir=args.ckpt_dir,
|
||||||
frame_num=args.frame_num,
|
device_id=device,
|
||||||
shift=args.sample_shift,
|
rank=rank,
|
||||||
sample_solver=args.sample_solver,
|
t5_fsdp=args.t5_fsdp,
|
||||||
sampling_steps=args.sample_steps,
|
dit_fsdp=args.dit_fsdp,
|
||||||
guide_scale=args.sample_guide_scale,
|
use_usp=(args.ulysses_size > 1 or args.ring_size > 1),
|
||||||
seed=args.base_seed,
|
t5_cpu=args.t5_cpu,
|
||||||
offload_model=args.offload_model)
|
)
|
||||||
|
|
||||||
else:
|
logging.info(
|
||||||
if args.prompt is None:
|
f"Generating {'image' if 't2i' in args.task else 'video'} ...")
|
||||||
args.prompt = EXAMPLE_PROMPT[args.task]["prompt"]
|
video = wan_t2v.generate(
|
||||||
if args.image is None:
|
args.prompt,
|
||||||
args.image = EXAMPLE_PROMPT[args.task]["image"]
|
size=SIZE_CONFIGS[args.size],
|
||||||
logging.info(f"Input prompt: {args.prompt}")
|
frame_num=args.frame_num,
|
||||||
logging.info(f"Input image: {args.image}")
|
shift=args.sample_shift,
|
||||||
|
sample_solver=args.sample_solver,
|
||||||
img = Image.open(args.image).convert("RGB")
|
sampling_steps=args.sample_steps,
|
||||||
if args.use_prompt_extend:
|
guide_scale=args.sample_guide_scale,
|
||||||
logging.info("Extending prompt ...")
|
seed=args.base_seed,
|
||||||
if rank == 0:
|
offload_model=args.offload_model)
|
||||||
prompt_output = prompt_expander(
|
|
||||||
args.prompt,
|
else:
|
||||||
tar_lang=args.prompt_extend_target_lang,
|
if args.prompt is None:
|
||||||
image=img,
|
args.prompt = EXAMPLE_PROMPT[args.task]["prompt"]
|
||||||
seed=args.base_seed)
|
if args.image is None:
|
||||||
if prompt_output.status == False:
|
args.image = EXAMPLE_PROMPT[args.task]["image"]
|
||||||
logging.info(
|
logging.info(f"Input prompt: {args.prompt}")
|
||||||
f"Extending prompt failed: {prompt_output.message}")
|
logging.info(f"Input image: {args.image}")
|
||||||
logging.info("Falling back to original prompt.")
|
|
||||||
input_prompt = args.prompt
|
img = Image.open(args.image).convert("RGB")
|
||||||
else:
|
if args.use_prompt_extend:
|
||||||
input_prompt = prompt_output.prompt
|
logging.info("Extending prompt ...")
|
||||||
input_prompt = [input_prompt]
|
if rank == 0:
|
||||||
else:
|
prompt_output = prompt_expander(
|
||||||
input_prompt = [None]
|
args.prompt,
|
||||||
if dist.is_initialized():
|
tar_lang=args.prompt_extend_target_lang,
|
||||||
dist.broadcast_object_list(input_prompt, src=0)
|
image=img,
|
||||||
args.prompt = input_prompt[0]
|
seed=args.base_seed)
|
||||||
logging.info(f"Extended prompt: {args.prompt}")
|
if prompt_output.status == False:
|
||||||
|
logging.info(
|
||||||
logging.info("Creating WanI2V pipeline.")
|
f"Extending prompt failed: {prompt_output.message}")
|
||||||
wan_i2v = wan.WanI2V(
|
logging.info("Falling back to original prompt.")
|
||||||
config=cfg,
|
input_prompt = args.prompt
|
||||||
checkpoint_dir=args.ckpt_dir,
|
else:
|
||||||
device_id=device,
|
input_prompt = prompt_output.prompt
|
||||||
rank=rank,
|
input_prompt = [input_prompt]
|
||||||
t5_fsdp=args.t5_fsdp,
|
else:
|
||||||
dit_fsdp=args.dit_fsdp,
|
input_prompt = [None]
|
||||||
use_usp=(args.ulysses_size > 1 or args.ring_size > 1),
|
if dist.is_initialized():
|
||||||
t5_cpu=args.t5_cpu,
|
dist.broadcast_object_list(input_prompt, src=0)
|
||||||
)
|
args.prompt = input_prompt[0]
|
||||||
|
logging.info(f"Extended prompt: {args.prompt}")
|
||||||
logging.info("Generating video ...")
|
|
||||||
video = wan_i2v.generate(
|
logging.info("Creating WanI2V pipeline.")
|
||||||
args.prompt,
|
wan_i2v = wan.WanI2V(
|
||||||
img,
|
config=cfg,
|
||||||
max_area=MAX_AREA_CONFIGS[args.size],
|
checkpoint_dir=args.ckpt_dir,
|
||||||
frame_num=args.frame_num,
|
device_id=device,
|
||||||
shift=args.sample_shift,
|
rank=rank,
|
||||||
sample_solver=args.sample_solver,
|
t5_fsdp=args.t5_fsdp,
|
||||||
sampling_steps=args.sample_steps,
|
dit_fsdp=args.dit_fsdp,
|
||||||
guide_scale=args.sample_guide_scale,
|
use_usp=(args.ulysses_size > 1 or args.ring_size > 1),
|
||||||
seed=args.base_seed,
|
t5_cpu=args.t5_cpu,
|
||||||
offload_model=args.offload_model)
|
)
|
||||||
|
|
||||||
if rank == 0:
|
logging.info("Generating video ...")
|
||||||
if args.save_file is None:
|
video = wan_i2v.generate(
|
||||||
formatted_time = datetime.now().strftime("%Y%m%d_%H%M%S")
|
args.prompt,
|
||||||
formatted_prompt = args.prompt.replace(" ", "_").replace("/",
|
img,
|
||||||
"_")[:50]
|
max_area=MAX_AREA_CONFIGS[args.size],
|
||||||
suffix = '.png' if "t2i" in args.task else '.mp4'
|
frame_num=args.frame_num,
|
||||||
args.save_file = f"{args.task}_{args.size}_{args.ulysses_size}_{args.ring_size}_{formatted_prompt}_{formatted_time}" + suffix
|
shift=args.sample_shift,
|
||||||
|
sample_solver=args.sample_solver,
|
||||||
if "t2i" in args.task:
|
sampling_steps=args.sample_steps,
|
||||||
logging.info(f"Saving generated image to {args.save_file}")
|
guide_scale=args.sample_guide_scale,
|
||||||
cache_image(
|
seed=args.base_seed,
|
||||||
tensor=video.squeeze(1)[None],
|
offload_model=args.offload_model)
|
||||||
save_file=args.save_file,
|
|
||||||
nrow=1,
|
if rank == 0:
|
||||||
normalize=True,
|
if args.save_file is None:
|
||||||
value_range=(-1, 1))
|
formatted_time = datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||||
else:
|
formatted_prompt = args.prompt.replace(" ", "_").replace("/", "_")[:50]
|
||||||
logging.info(f"Saving generated video to {args.save_file}")
|
suffix = '.png' if "t2i" in args.task else '.mp4'
|
||||||
cache_video(
|
args.save_file = f"{args.task}_{args.size}_{args.ulysses_size}_{args.ring_size}_{formatted_prompt}_{formatted_time}" + suffix
|
||||||
tensor=video[None],
|
|
||||||
save_file=args.save_file,
|
if "t2i" in args.task:
|
||||||
fps=cfg.sample_fps,
|
logging.info(f"Saving generated image to {args.save_file}")
|
||||||
nrow=1,
|
cache_image(
|
||||||
normalize=True,
|
tensor=video.squeeze(1)[None],
|
||||||
value_range=(-1, 1))
|
save_file=args.save_file,
|
||||||
logging.info("Finished.")
|
nrow=1,
|
||||||
|
normalize=True,
|
||||||
|
value_range=(-1, 1))
|
||||||
if __name__ == "__main__":
|
else:
|
||||||
args = _parse_args()
|
logging.info(f"Saving generated video to {args.save_file}")
|
||||||
generate(args)
|
cache_video(
|
||||||
|
tensor=video[None],
|
||||||
|
save_file=args.save_file,
|
||||||
|
fps=cfg.sample_fps,
|
||||||
|
nrow=1,
|
||||||
|
normalize=True,
|
||||||
|
value_range=(-1, 1))
|
||||||
|
logging.info("Finished.")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
args = _parse_args()
|
||||||
|
generate(args)
|
||||||
|
|||||||
@ -1,16 +1,16 @@
|
|||||||
torch>=2.4.0
|
torch>=2.4.0
|
||||||
torchvision>=0.19.0
|
torchvision>=0.19.0
|
||||||
opencv-python>=4.9.0.80
|
opencv-python>=4.9.0.80
|
||||||
diffusers>=0.31.0
|
diffusers>=0.31.0
|
||||||
transformers>=4.49.0
|
transformers>=4.49.0
|
||||||
tokenizers>=0.20.3
|
tokenizers>=0.20.3
|
||||||
accelerate>=1.1.1
|
accelerate>=1.1.1
|
||||||
tqdm
|
tqdm
|
||||||
imageio
|
imageio
|
||||||
easydict
|
easydict
|
||||||
ftfy
|
ftfy
|
||||||
dashscope
|
dashscope
|
||||||
imageio-ffmpeg
|
imageio-ffmpeg
|
||||||
flash_attn
|
flash_attn
|
||||||
gradio>=5.0.0
|
gradio>=5.0.0
|
||||||
numpy>=1.23.5,<2
|
numpy>=1.23.5,<2
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
Loading…
Reference in New Issue
Block a user