[Python] How to improve temporal consistency in frame-by-frame video generation using...

I'm using a pretrained image-to-image diffusion model (a custom UNet+DDPM) to generate each frame of a video from a sequence of conditioning inputs (e.g. pose keypoints or masks). The results look good per-frame, but the generated video flickers a lot due to inconsistency across frames.

Here's the simplified code that loads keypoints and generates images:

from diffusers import UNet2DConditionModel, DDPMScheduler
import torch
import numpy as np
from PIL import Image
import cv2

model = UNet2DConditionModel.from_pretrained("my-custom-checkpoint")
scheduler = DDPMScheduler(num_train_timesteps=1000)
model.eval()

def generate_frame(condition_tensor):
noise = torch.randn((1, 3, 64, 64))
for t in reversed(range(1000)):
noise = scheduler.step(model(noise, t, condition_tensor).sample, t, noise).prev_sample
return noise

video = []

for i in range(30): # 30 frames
cond = get_condition_for_frame(i) # e.g. keypoints converted to tensor
image_tensor = generate_frame(cond)
image = (image_tensor.squeeze().permute(1, 2, 0).numpy() * 255).astype(np.uint8)
video.append(image)

# Write to video
out = cv2.VideoWriter('output.mp4', cv2.VideoWriter_fourcc(*'mp4v'), 10, (64, 64))
for frame in video:
out.write(frame)
out.release()


My problem is that the video I’m generating flickers a lot. Each frame looks okay by itself, but when I play them as a video, they don’t transition smoothly — it feels jittery. I’m using a diffusion model with smoothly changing inputs (like pose keypoints), but the output still jumps around between frames. I’m trying to figure out how to make the video look more stable and consistent.

What I tried


I generated each frame one by one using a diffusion model, with small changes in the conditioning input (like pose keypoints).
I also tried:

• 
  Using the previous frame as extra input (not sure how to use it effectively)
  
  
• 
  Adding simple smoothing between frames (helped a bit but looked blurry)
  
  
• 
  Warping frames with optical flow (added ghosting artifacts)
  

What I expected


I expected the video to look smooth and consistent — like a natural animation where each frame blends into the next.
Since the inputs change gradually, I thought the outputs would too. But the result still flickers and feels unstable.

Continue reading...