Stable Diffusion Image Generation with Diffusers: A Practical Guide

Stable Diffusion turns text into images, and with the HuggingFace Diffusers library you can run it locally, adapt it to your own workflows, and control the output far beyond a simple prompt. This guide walks through the core capabilities, from basic text-to-image to inpainting, ControlNet, and LoRA adapters, with the exact code you need to get started.

What it does

Diffusers gives you a unified interface to several generations of Stable Diffusion models. You can describe a scene and get an image, transform an existing photo with text guidance, fill in masked areas, or extend an image beyond its borders. The library handles the heavy lifting: loading model weights, running the denoising loop, and decoding the result into a viewable image. You pick a pipeline, point it at a model, and call it with a prompt.

The real strength is flexibility. You can swap schedulers to trade speed for quality, load fine-tuned LoRA adapters to apply a specific style, and use ControlNet to condition the output on edges, poses, or depth maps. This makes Diffusers suitable for everything from quick concept sketches to production pipelines that need reproducible, batch-generated images.

Before you start

This skill is optional and installed on demand. It runs on Linux, macOS, and Windows. You will need Python and pip, and a GPU with CUDA support is strongly recommended for reasonable performance. The installation commands pull in the core dependencies:

pip install diffusers transformers accelerate torch
pip install xformers  # Optional: memory-efficient attention

The first line installs the Diffusers library, the Transformers library for text encoding, Accelerate for device management, and PyTorch itself. The second installs xFormers, which is optional but can significantly reduce memory usage during attention computation. If you are on a machine without a GPU, the code examples that call .to("cuda") will not work as written; you would need to adapt them for CPU, which is slow but possible for small images.

Quick start

Installation

Run the commands above in your terminal. This is a one-time setup. After that, you can import Diffusers in your Python scripts.

Basic text-to-image

The simplest path to an image is a text prompt. The following code loads the Stable Diffusion v1.5 pipeline, moves it to the GPU, and generates an image:

from diffusers import DiffusionPipeline
import torch

# Load pipeline (auto-detects model type)
pipe = DiffusionPipeline.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5",
    torch_dtype=torch.float16
)
pipe.to("cuda")

# Generate image
image = pipe(
    "A serene mountain landscape at sunset, highly detailed",
    num_inference_steps=50,
    guidance_scale=7.5
).images[0]

image.save("output.png")

This is the canonical example. The pipeline downloads the model weights on first use, which can take several gigabytes. The torch_dtype=torch.float16 argument halves the memory footprint. The num_inference_steps controls how many denoising iterations run; more steps generally mean better quality but longer generation time. guidance_scale controls how closely the output follows the prompt; values around 7.5 are typical. The result is saved as output.png in the current directory.

Using SDXL (higher quality)

For higher resolution and more detailed outputs, SDXL is the next step up. The code is nearly identical, but uses a different pipeline class and model identifier:

from diffusers import AutoPipelineForText2Image
import torch

pipe = AutoPipelineForText2Image.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
    variant="fp16"
)
pipe.to("cuda")

# Enable memory optimization
pipe.enable_model_cpu_offload()

image = pipe(
    prompt="A futuristic city with flying cars, cinematic lighting",
    height=1024,
    width=1024,
    num_inference_steps=30
).images[0]

Note the variant="fp16" argument, which selects the half-precision version of the model files. The enable_model_cpu_offload() call moves model components to the CPU when they are not in use, which reduces GPU memory pressure at the cost of some speed. SDXL natively generates at 1024x1024, so you do not need to upscale.

Architecture overview

Three-pillar design

Diffusers organizes generation around three components: a pipeline that orchestrates the process, a model that contains the neural networks, and a scheduler that controls denoising. The model itself has three parts: a UNet or Transformer that predicts noise, a VAE that encodes and decodes latents, and a text encoder (CLIP or T5) that turns your prompt into embeddings.

Pipeline (orchestration)
├── Model (neural networks)
│   ├── UNet / Transformer (noise prediction)
│   ├── VAE (latent encoding/decoding)
│   └── Text Encoder (CLIP/T5)
└── Scheduler (denoising algorithm)

Pipeline inference flow

When you call the pipeline, the text prompt goes through the text encoder to produce embeddings. Simultaneously, random noise is generated. The scheduler then runs a denoising loop: at each step, the UNet predicts the noise component, the scheduler removes it, and this repeats. Finally, the VAE decoder converts the denoised latents into a pixel image.

Text Prompt → Text Encoder → Text Embeddings
                                    ↓
Random Noise → [Denoising Loop] ← Scheduler
                      ↓
               Predicted Noise
                      ↓
              VAE Decoder → Final Image

Understanding this flow helps when things go wrong. For example, if you get black images, the problem is often in the VAE or a dtype mismatch, not in the prompt.

Core concepts

Pipelines

Pipelines are the high-level entry points. Each one bundles the model and scheduler for a specific task. The table below lists the main ones:

Pipeline	Purpose
StableDiffusionPipeline	Text-to-image (SD 1.x/2.x)
StableDiffusionXLPipeline	Text-to-image (SDXL)
StableDiffusion3Pipeline	Text-to-image (SD 3.0)
FluxPipeline	Text-to-image (Flux models)
StableDiffusionImg2ImgPipeline	Image-to-image
StableDiffusionInpaintPipeline	Inpainting

You can also use the AutoPipelineForText2Image class, which automatically selects the correct pipeline based on the model ID. This is convenient when you switch between SD 1.5, SDXL, or Flux.

Schedulers

Schedulers determine how the denoising loop progresses. Different schedulers offer different trade-offs between speed and quality. The table summarizes common choices:

Scheduler	Steps	Quality	Use Case
EulerDiscreteScheduler	20-50	Good	Default choice
EulerAncestralDiscreteScheduler	20-50	Good	More variation
DPMSolverMultistepScheduler	15-25	Excellent	Fast, high quality
DDIMScheduler	50-100	Good	Deterministic
LCMScheduler	4-8	Good	Very fast
UniPCMultistepScheduler	15-25	Excellent	Fast convergence

For most purposes, the default scheduler is fine. But if you want faster generation, switching to DPMSolver or LCM can cut steps dramatically.

Swapping schedulers

You can replace the scheduler on an existing pipeline at any time. The new scheduler is created from the pipeline's current configuration, so it is compatible with the model:

from diffusers import DPMSolverMultistepScheduler

# Swap for faster generation
pipe.scheduler = DPMSolverMultistepScheduler.from_config(
    pipe.scheduler.config
)

# Now generate with fewer steps
image = pipe(prompt, num_inference_steps=20).images[0]

After swapping, you can reduce the number of inference steps from 50 to 20 and still get good results. This is a common optimization for interactive use.

Generation parameters

Key parameters

The pipeline call accepts several parameters that control the output. The most important are listed below:

Parameter	Default	Description
prompt	Required	Text description of desired image
negative_prompt	None	What to avoid in the image
num_inference_steps	50	Denoising steps (more = better quality)
guidance_scale	7.5	Prompt adherence (7-12 typical)
height, width	512/1024	Output dimensions (multiples of 8)
generator	None	Torch generator for reproducibility
num_images_per_prompt	1	Batch size

Note that height and width must be multiples of 8, because the VAE operates on 8x8 patches. If you request a size that is not a multiple of 8, the pipeline may error or produce unexpected results.

Reproducible generation

To get the same image from the same prompt, you need to seed the random number generator. The following code sets a seed of 42:

import torch

generator = torch.Generator(device="cuda").manual_seed(42)

image = pipe(
    prompt="A cat wearing a top hat",
    generator=generator,
    num_inference_steps=50
).images[0]

With the same seed, the same prompt, and the same settings, you will get the same image. This is essential for debugging and for reproducible research.

Negative prompts

Negative prompts tell the model what to avoid. This is useful for reducing common artifacts like blurriness or distortion:

image = pipe(
    prompt="Professional photo of a dog in a garden",
    negative_prompt="blurry, low quality, distorted, ugly, bad anatomy",
    guidance_scale=7.5
).images[0]

The negative prompt is encoded and used to steer the denoising process away from those concepts. It is a powerful tool for improving output quality without changing the main prompt.

Image-to-image

Image-to-image takes an existing image and transforms it according to a text prompt. The strength parameter controls how much the original image is altered, with 0 meaning no change and 1 meaning a completely new image. A value of 0.75 is a good starting point for style transfer:

from diffusers import AutoPipelineForImage2Image
from PIL import Image

pipe = AutoPipelineForImage2Image.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5",
    torch_dtype=torch.float16
).to("cuda")

init_image = Image.open("input.jpg").resize((512, 512))

image = pipe(
    prompt="A watercolor painting of the scene",
    image=init_image,
    strength=0.75,  # How much to transform (0-1)
    num_inference_steps=50
).images[0]

Note that the input image is resized to 512x512, which matches the model's native resolution. If you use a different size, you may need to adjust it. The output will be the same size as the input after resizing.

Inpainting

Inpainting fills in masked regions of an image. You provide an image and a mask where white pixels indicate the area to repaint. The model generates content that matches the prompt and blends with the surrounding context:

from diffusers import AutoPipelineForInpainting
from PIL import Image

pipe = AutoPipelineForInpainting.from_pretrained(
    "runwayml/stable-diffusion-inpainting",
    torch_dtype=torch.float16
).to("cuda")

image = Image.open("photo.jpg")
mask = Image.open("mask.png")  # White = inpaint region

result = pipe(
    prompt="A red car parked on the street",
    image=image,
    mask_image=mask,
    num_inference_steps=50
).images[0]

The mask must be the same size as the image. The white areas are replaced, and the black areas are preserved. This is useful for removing objects, changing backgrounds, or adding new elements.

ControlNet

ControlNet adds spatial conditioning, letting you guide the composition with edges, poses, or depth maps. For example, you can use a Canny edge map to preserve the structure of an input image while changing the style:

from diffusers import StableDiffusionControlNetPipeline, ControlNetModel
import torch

# Load ControlNet for edge conditioning
controlnet = ControlNetModel.from_pretrained(
    "lllyasviel/control_v11p_sd15_canny",
    torch_dtype=torch.float16
)

pipe = StableDiffusionControlNetPipeline.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5",
    controlnet=controlnet,
    torch_dtype=torch.float16
).to("cuda")

# Use Canny edge image as control
control_image = get_canny_image(input_image)

image = pipe(
    prompt="A beautiful house in the style of Van Gogh",
    image=control_image,
    num_inference_steps=30
).images[0]

The get_canny_image function is not provided; you need to implement it using OpenCV or another library. The control image is passed as the image argument, and the pipeline uses it to condition the generation.

Available ControlNets

The table lists the common ControlNet models and their inputs:

ControlNet	Input Type	Use Case
canny	Edge maps	Preserve structure
openpose	Pose skeletons	Human poses
depth	Depth maps	3D-aware generation
normal	Normal maps	Surface details
mlsd	Line segments	Architectural lines
scribble	Rough sketches	Sketch-to-image

Each ControlNet is a separate model that you load and attach to the pipeline. The choice depends on what kind of control you need.

LoRA adapters

LoRA (Low-Rank Adaptation) is a technique for fine-tuning models with a small number of parameters. Diffusers supports loading LoRA weights to apply a specific style or character without retraining the whole model:

from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5",
    torch_dtype=torch.float16
).to("cuda")

# Load LoRA weights
pipe.load_lora_weights("path/to/lora", weight_name="style.safetensors")

# Generate with LoRA style
image = pipe("A portrait in the trained style").images[0]

# Adjust LoRA strength
pipe.fuse_lora(lora_scale=0.8)

# Unload LoRA
pipe.unload_lora_weights()

The load_lora_weights method loads the adapter, fuse_lora merges it into the model with a given scale, and unload_lora_weights removes it. You can adjust the scale to control how strongly the style is applied.

Multiple LoRAs

You can load multiple LoRAs and combine them with different weights. This is useful for mixing a style and a character:

# Load multiple LoRAs
pipe.load_lora_weights("lora1", adapter_name="style")
pipe.load_lora_weights("lora2", adapter_name="character")

# Set weights for each
pipe.set_adapters(["style", "character"], adapter_weights=[0.7, 0.5])

image = pipe("A portrait").images[0]

Each LoRA is given a name, and set_adapters controls the blend. This allows fine-grained control over the output.

Memory optimization

Running Stable Diffusion on a GPU can be memory-intensive. Diffusers provides several methods to reduce memory usage, at the cost of some speed.

Enable CPU offloading

CPU offloading moves model components to the CPU when they are not in use. This frees up GPU memory for the active computation:

# Model CPU offload - moves models to CPU when not in use
pipe.enable_model_cpu_offload()

# Sequential CPU offload - more aggressive, slower
pipe.enable_sequential_cpu_offload()

enable_model_cpu_offload() is a good balance. enable_sequential_cpu_offload() is more aggressive and can run on very limited GPUs, but is noticeably slower.

Attention slicing

Attention slicing computes attention in chunks, reducing peak memory usage:

# Reduce memory by computing attention in chunks
pipe.enable_attention_slicing()

# Or specific chunk size
pipe.enable_attention_slicing("max")

The "max" option uses the smallest chunk size, which minimizes memory but increases computation time.

xFormers memory-efficient attention

If you installed xFormers, you can enable its memory-efficient attention implementation:

# Requires xformers package
pipe.enable_xformers_memory_efficient_attention()

This is often the single most effective optimization for memory usage.

VAE slicing for large images

For large images, the VAE can decode latents in tiles to avoid running out of memory:

# Decode latents in tiles for large images
pipe.enable_vae_slicing()
pipe.enable_vae_tiling()

enable_vae_slicing() splits the batch dimension, while enable_vae_tiling() splits the spatial dimensions. Both reduce memory at the cost of speed.

Model variants

Loading different precisions

You can load models in different precisions to trade memory for quality. FP16 is the default recommendation for GPU inference. BF16 offers better numerical stability but requires an Ampere or newer GPU:

# FP16 (recommended for GPU)
pipe = DiffusionPipeline.from_pretrained(
    "model-id",
    torch_dtype=torch.float16,
    variant="fp16"
)

# BF16 (better precision, requires Ampere+ GPU)
pipe = DiffusionPipeline.from_pretrained(
    "model-id",
    torch_dtype=torch.bfloat16
)

The variant="fp16" argument selects the pre-converted FP16 weights from the model repository. For BF16, you do not need a variant because the conversion happens on the fly.

Loading specific components

Sometimes you want to replace a component, such as the VAE, with a better one. You can load it separately and pass it to the pipeline:

from diffusers import UNet2DConditionModel, AutoencoderKL

# Load custom VAE
vae = AutoencoderKL.from_pretrained("stabilityai/sd-vae-ft-mse")

# Use with pipeline
pipe = DiffusionPipeline.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5",
    vae=vae,
    torch_dtype=torch.float16
)

This is useful when a model's default VAE produces slightly blurry images. The sd-vae-ft-mse is a common improvement.

Batch generation

To generate multiple images efficiently, you can pass a list of prompts or set num_images_per_prompt:

# Multiple prompts
prompts = [
    "A cat playing piano",
    "A dog reading a book",
    "A bird painting a picture"
]

images = pipe(prompts, num_inference_steps=30).images

# Multiple images per prompt
images = pipe(
    "A beautiful sunset",
    num_images_per_prompt=4,
    num_inference_steps=30
).images

The first call generates one image per prompt. The second generates four variations of the same prompt. Batching is more efficient than calling the pipeline in a loop because it uses the GPU more effectively.

Common workflows

Workflow 1: High-quality generation

For the best quality, combine SDXL with a fast scheduler and memory optimization:

from diffusers import StableDiffusionXLPipeline, DPMSolverMultistepScheduler
import torch

# 1. Load SDXL with optimizations
pipe = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
    variant="fp16"
)
pipe.to("cuda")
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
pipe.enable_model_cpu_offload()

# 2. Generate with quality settings
image = pipe(
    prompt="A majestic lion in the savanna, golden hour lighting, 8k, detailed fur",
    negative_prompt="blurry, low quality, cartoon, anime, sketch",
    num_inference_steps=30,
    guidance_scale=7.5,
    height=1024,
    width=1024
).images[0]

This setup produces high-resolution, detailed images in about 30 steps. The negative prompt helps avoid common artifacts.

Workflow 2: Fast prototyping

For quick iterations, use LCM (Latent Consistency Models) to generate images in just a few steps:

from diffusers import AutoPipelineForText2Image, LCMScheduler
import torch

# Use LCM for 4-8 step generation
pipe = AutoPipelineForText2Image.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16
).to("cuda")

# Load LCM LoRA for fast generation
pipe.load_lora_weights("latent-consistency/lcm-lora-sdxl")
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)
pipe.fuse_lora()

# Generate in ~1 second
image = pipe(
    "A beautiful landscape",
    num_inference_steps=4,
    guidance_scale=1.0
).images[0]

With LCM, you can generate an image in about a second, which is ideal for exploring ideas quickly. Note that the guidance scale is set to 1.0, which is typical for LCM.

Common issues

CUDA out of memory:

If you run out of GPU memory, enable the memory optimizations described above:

# Enable memory optimizations
pipe.enable_model_cpu_offload()
pipe.enable_attention_slicing()
pipe.enable_vae_slicing()

# Or use lower precision
pipe = DiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16)

Black/noise images:

Black or noisy outputs often indicate a VAE problem or a dtype mismatch. The source suggests checking the VAE configuration and, if needed, bypassing the safety checker:

# Check VAE configuration
# Use safety checker bypass if needed
pipe.safety_checker = None

# Ensure proper dtype consistency
pipe = pipe.to(dtype=torch.float16)

Note that disabling the safety checker removes a filter that blocks NSFW content. Only do this if you understand the implications.

Slow generation:

If generation is slow, switch to a faster scheduler and reduce the number of steps:

# Use faster scheduler
from diffusers import DPMSolverMultistepScheduler
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)

# Reduce steps
image = pipe(prompt, num_inference_steps=20).images[0]

When not to use it

Stable Diffusion is not always the right tool. The source lists several alternatives for specific situations:

• DALL-E 3: If you do not have a GPU and want an API-based solution.
• Midjourney: If you want artistic, stylized outputs without managing infrastructure.
• Imagen: If you are already on Google Cloud and want tight integration.
• Leonardo.ai: If you prefer a web-based creative workflow.

These services handle the infrastructure and often provide a simpler interface, but they come with usage costs and less control. If you need to run locally, customize the model, or integrate with your own code, Diffusers is the better choice.

Limits and gotchas

• GPU requirement: The examples assume a CUDA GPU. Without one, generation is slow or impossible for large models.
• Memory: SDXL and Flux models are large. Use the memory optimization techniques to fit them on consumer GPUs.
• Resolution multiples of 8: The height and width parameters must be multiples of 8, or the pipeline may fail.
• Safety checker: By default, a safety checker may block certain content. Disabling it (pipe.safety_checker = None) is possible but removes that protection.
• Model downloads: The first run downloads model weights, which can be several gigabytes. Ensure you have disk space and a stable connection.
• Reproducibility: To get identical results, you must set the generator seed and use the same model version, scheduler, and parameters.

References and further reading

The source points to two reference documents in the Hermes Agent repository:

• Advanced Usage - Custom pipelines, fine-tuning, deployment
• Troubleshooting - Common issues and solutions

For the official Diffusers documentation, model hub, and community, see the resources section in the source page. These are the best places to go when you want to move beyond the basics covered here.