[Python] How to reduce GPU memory usage when running Stable Diffusion with HuggingFace...

I'm trying to run Stable Diffusion using HuggingFace's diffusers library, but I keep getting CUDA out of memory errors on my RTX 3060 (12GB VRAM). I'm using the standard StableDiffusionPipeline from the "CompVis/stable-diffusion-v1-4" checkpoint.

Here's my code:

from diffusers import StableDiffusionPipeline
import torch

# Load the pre-trained Stable Diffusion model
pipe = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4")
pipe = pipe.to("cuda")

# Define the prompt
prompt = "a futuristic cityscape, high resolution, ultra detailed"

# Generate the image
image = pipe(prompt).images[0]

# Show the image
image.show()


I tried running the code as shown above, expecting it to generate a high-quality image from the text prompt without memory issues. I also tried adding torch_dtype=torch.float16 when loading the pipeline to reduce memory usage, like this:

pipe = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4", torch_dtype=torch.float16)
pipe = pipe.to("cuda")


However, I still encounter CUDA out of memory errors, especially with larger prompts or higher resolutions. I expected the float16 setting to help, but it didn't fully solve the issue.

I also expected to control the resolution by adjusting parameters, but I'm not sure how to properly do that with this pipeline.

Continue reading...