[core] LTX Video

[a-r-r-o-w]

T2V:

import torch
from diffusers import LTXPipeline
from diffusers.utils import export_to_video

pipe = LTXPipeline.from_pretrained("a-r-r-o-w/LTX-Video-diffusers", torch_dtype=torch.bfloat16)
pipe.to("cuda")

prompt = "A woman with long brown hair and light skin smiles at another woman with long blonde hair. The woman with brown hair wears a black jacket and has a small, barely noticeable mole on her right cheek. The camera angle is a close-up, focused on the woman with brown hair's face. The lighting is warm and natural, likely from the setting sun, casting a soft glow on the scene. The scene appears to be real-life footage"
negative_prompt = "worst quality, inconsistent motion, blurry, jittery, distorted"

video = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    width=704,
    height=480,
    num_frames=161,
    num_inference_steps=50,
).frames[0]
export_to_video(video, "output.mp4", fps=24)

I2V:

import torch
from diffusers import LTXImageToVideoPipeline
from diffusers.utils import export_to_video, load_image

pipe = LTXImageToVideoPipeline.from_pretrained("a-r-r-o-w/LTX-Video-diffusers", torch_dtype=torch.bfloat16)
pipe.to("cuda")

image = load_image(
    "https://huggingface.co/datasets/a-r-r-o-w/tiny-meme-dataset-captioned/resolve/main/images/8.png"
)
prompt = "A young girl stands calmly in the foreground, looking directly at the camera, as a house fire rages in the background. Flames engulf the structure, with smoke billowing into the air. Firefighters in protective gear rush to the scene, a fire truck labeled '38' visible behind them. The girl's neutral expression contrasts sharply with the chaos of the fire, creating a poignant and emotionally charged scene."
negative_prompt = "worst quality, inconsistent motion, blurry, jittery, distorted"

video = pipe(
    image=image,
    prompt=prompt,
    negative_prompt=negative_prompt,
    width=704,
    height=480,
    num_frames=161,
    num_inference_steps=50,
).frames[0]
export_to_video(video, "output.mp4", fps=24)

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[a-r-r-o-w]

cc @yiyixux for the shift_terminal change

[a-r-r-o-w]

@DN6 Should I follow your approach with Mochi and create a separate attention class for LTX?

[yiyixuxu]

ok but we want to be more careful, ideally, we do that as part of carefully planned-out refactor
but maybe it would be safe to just inherit form Attention for now? e.g. we wrote code like this with the assumption in mind we only have one attention class
https://github.com/huggingface/diffusers/blob/e47cc1fc1a89a5375c322d296cd122fe71ab859f/src/diffusers/pipelines/pag/pag_utils.py#L57C39-L57C48

cc @DN6 here too

[a-r-r-o-w]

cc @yiyixuxu

[stevhliu]

Thanks for adding!

[yiyixuxu]

is there any reason we reshape from 5d -> 3d and then 3d -> 5d on every iteration?

[yiyixuxu]

think it will be nice to:

1. have a _pack_latent and _unpack_latent like we did for flux
2. maybe we can move the rotary embedding in the pipeline so we only pack/unpack once (I know we currently have some discrepancy here, so open to discussion); or we can just configure the rotary pos embeds class in pipeline, so we do not need to give the shape info each time

[a-r-r-o-w]

Yes, I'll add the pack and unpack latent methods.

Regarding the RoPE, I think a separate layer approach is okay despite it requiring recomputation at every step. This is because we're planning to work on caching hooks that would enable the outputs of any layer to be cached and re-used. Since RoPE is an integral part of many models, there can be some opt-out code we could default to for enabling caching by default on these kinds of model specific RoPE layers. WDYT?

[a-r-r-o-w]

@yiyixuxu They use per latent frame timesteps (actually, it's per-token timesteps but all tokens corresponding to the same frame have the same timesteps), but since we don't support it in our schedulers, we can't really do the normal scheduler.step(). These changes were required to make the pipeline atleast generate reasonable results. The quality of generations looks similar to me but will try and numerically match.