[Python] How to improve ControlNet-based Stable Diffusion pipeline for frame-wise...

I'm working on a task where I use Stable Diffusion with ControlNet to perform frame-wise regeneration of video frames.

For each frame:

• 
  I have a reference image (original frame from the video).
  
  
• 
  I also have a pose image (OpenPose skeleton of the same person with modified pose).
  
  
• 
  I want to preserve the original appearance, style, and background, but only update the pose.
  

I'm trying to regenerate individual video frames using StableDiffusionControlNetImg2ImgPipeline with OpenPose as the control input.
My goal is to preserve the original image (face, clothing, background) while adjusting only the human pose to match a new skeleton.

Here’s the pipeline code I’m using:

controlnet_pose = ControlNetModel.from_pretrained(
"lllyasviel/control_v11p_sd15_openpose", torch_dtype=torch.float16
).to("cuda")

pipe = StableDiffusionControlNetImg2ImgPipeline.from_pretrained(
"runwayml/stable-diffusion-v1-5",
controlnet=controlnet_pose,
torch_dtype=torch.float16
).to("cuda")

pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)
pipe.enable_model_cpu_offload()
pipe.enable_xformers_memory_efficient_attention()

def regenerate_frame(frame_number):
image_path = os.path.join(image_folder, f"frame_{frame_number}.png")
pose_path = os.path.join(image_folder, f"pose_{frame_number}.png")

ref_image = Image.open(image_path).convert("RGB").resize((512, 512))
pose_image = Image.open(pose_path).convert("RGB").resize((512, 512))

prompt = "same person, same style, same clothes, same background, realistic"
negative_prompt = "blurry, deformed, inconsistent face, wrong pose, bad anatomy"

try:
result = pipe(
prompt=prompt,
negative_prompt=negative_prompt,
image=ref_image,
control_image=pose_image,
strength=0.3,
num_inference_steps=30,
guidance_scale=7.5
).images[0]
except Exception as e:
print("Error occurred:", e)
return

result.save(...)


I expected the regenerated frame to match the original input frame in appearance, and only change the pose. But in many cases:

• 
  Face identity is not preserved.
  
  
• 
  Background or clothing changes unexpectedly.
  
  
• 
  The result is blurrier or worse than the original.
  

I tried:

• 
  Lowering strength (to 0.2–0.4)
  
  
• 
  Tuning guidance_scale
  
  
• 
  Rewriting the prompt
  But the result is still unstable.
  

Continue reading...