Stable Diffusion Image Generation with Diffusers: Text-to-Image, Inpainting, and Img2Img

This guide covers generating images with Stable Diffusion using the HuggingFace Diffusers library, as implemented in the Hermes Agent MLOps skill. You will learn how to run text-to-image, image-to-image, and inpainting pipelines, swap schedulers, apply ControlNet and LoRA adapters, and optimize memory usage. This is for practitioners who want to integrate local image generation into their autonomous agent workflows.

What it does

This skill gives Hermes Agent the ability to generate, transform, and edit images using Stable Diffusion models. You can create images from text prompts, modify existing images with text guidance, fill in masked areas of a photo, and apply spatial controls like edge maps or pose skeletons. The underlying library is Diffusers, which provides a unified interface for multiple model architectures (SD 1.5, SDXL, SD 3.0, Flux) and scheduler algorithms. The skill is designed to run on a local GPU, making it suitable for offline or privacy-sensitive pipelines.

Before you start

You need a machine with a CUDA-capable GPU and enough VRAM to hold the model. For SD 1.5, 8 GB is a practical minimum; for SDXL, 12 GB or more is recommended. The skill is installed on demand and is optional. It runs on Linux, macOS, and Windows. You must have Python and pip available. The core dependencies are diffusers, transformers, accelerate, and torch. The xformers package is optional but recommended for memory-efficient attention.

Installation

pip install diffusers transformers accelerate torch
pip install xformers  # Optional: memory-efficient attention

Basic text-to-image

The simplest workflow loads a pipeline and calls it with a prompt. The example below uses the SD 1.5 base model with FP16 precision.

from diffusers import DiffusionPipeline
import torch

# Load pipeline (auto-detects model type)
pipe = DiffusionPipeline.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5",
    torch_dtype=torch.float16
)
pipe.to("cuda")

# Generate image
image = pipe(
    "A serene mountain landscape at sunset, highly detailed",
    num_inference_steps=50,
    guidance_scale=7.5
).images[0]

image.save("output.png")

Using SDXL (higher quality)

SDXL produces higher-resolution, more detailed images. It requires more VRAM but can be run with memory optimizations.

from diffusers import AutoPipelineForText2Image
import torch

pipe = AutoPipelineForText2Image.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
    variant="fp16"
)
pipe.to("cuda")

# Enable memory optimization
pipe.enable_model_cpu_offload()

image = pipe(
    prompt="A futuristic city with flying cars, cinematic lighting",
    height=1024,
    width=1024,
    num_inference_steps=30
).images[0]

Architecture overview

Three-pillar design

Diffusers is built around three core components:

Pipeline (orchestration)
├── Model (neural networks)
│   ├── UNet / Transformer (noise prediction)
│   ├── VAE (latent encoding/decoding)
│   └── Text Encoder (CLIP/T5)
└── Scheduler (denoising algorithm)

Pipeline inference flow

Text Prompt → Text Encoder → Text Embeddings
                                    ↓
Random Noise → [Denoising Loop] ← Scheduler
                      ↓
               Predicted Noise
                      ↓
              VAE Decoder → Final Image

Core concepts

Pipelines

Pipelines orchestrate complete workflows:

Pipeline	Purpose
StableDiffusionPipeline	Text-to-image (SD 1.x/2.x)
StableDiffusionXLPipeline	Text-to-image (SDXL)
StableDiffusion3Pipeline	Text-to-image (SD 3.0)
FluxPipeline	Text-to-image (Flux models)
StableDiffusionImg2ImgPipeline	Image-to-image
StableDiffusionInpaintPipeline	Inpainting

Schedulers

Schedulers control the denoising process:

Scheduler	Steps	Quality	Use Case
EulerDiscreteScheduler	20-50	Good	Default choice
EulerAncestralDiscreteScheduler	20-50	Good	More variation
DPMSolverMultistepScheduler	15-25	Excellent	Fast, high quality
DDIMScheduler	50-100	Good	Deterministic
LCMScheduler	4-8	Good	Very fast
UniPCMultistepScheduler	15-25	Excellent	Fast convergence

Swapping schedulers

from diffusers import DPMSolverMultistepScheduler

# Swap for faster generation
pipe.scheduler = DPMSolverMultistepScheduler.from_config(
    pipe.scheduler.config
)

# Now generate with fewer steps
image = pipe(prompt, num_inference_steps=20).images[0]

Generation parameters

Key parameters

Parameter	Default	Description
prompt	Required	Text description of desired image
negative_prompt	None	What to avoid in the image
num_inference_steps	50	Denoising steps (more = better quality)
guidance_scale	7.5	Prompt adherence (7-12 typical)
height, width	512/1024	Output dimensions (multiples of 8)
generator	None	Torch generator for reproducibility
num_images_per_prompt	1	Batch size

Reproducible generation

import torch

generator = torch.Generator(device="cuda").manual_seed(42)

image = pipe(
    prompt="A cat wearing a top hat",
    generator=generator,
    num_inference_steps=50
).images[0]

Negative prompts

image = pipe(
    prompt="Professional photo of a dog in a garden",
    negative_prompt="blurry, low quality, distorted, ugly, bad anatomy",
    guidance_scale=7.5
).images[0]

Image-to-image

Transform existing images with text guidance:

from diffusers import AutoPipelineForImage2Image
from PIL import Image

pipe = AutoPipelineForImage2Image.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5",
    torch_dtype=torch.float16
).to("cuda")

init_image = Image.open("input.jpg").resize((512, 512))

image = pipe(
    prompt="A watercolor painting of the scene",
    image=init_image,
    strength=0.75,  # How much to transform (0-1)
    num_inference_steps=50
).images[0]

Inpainting

Fill masked regions:

from diffusers import AutoPipelineForInpainting
from PIL import Image

pipe = AutoPipelineForInpainting.from_pretrained(
    "runwayml/stable-diffusion-inpainting",
    torch_dtype=torch.float16
).to("cuda")

image = Image.open("photo.jpg")
mask = Image.open("mask.png")  # White = inpaint region

result = pipe(
    prompt="A red car parked on the street",
    image=image,
    mask_image=mask,
    num_inference_steps=50
).images[0]

ControlNet

Add spatial conditioning for precise control:

from diffusers import StableDiffusionControlNetPipeline, ControlNetModel
import torch

# Load ControlNet for edge conditioning
controlnet = ControlNetModel.from_pretrained(
    "lllyasviel/control_v11p_sd15_canny",
    torch_dtype=torch.float16
)

pipe = StableDiffusionControlNetPipeline.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5",
    controlnet=controlnet,
    torch_dtype=torch.float16
).to("cuda")

# Use Canny edge image as control
control_image = get_canny_image(input_image)

image = pipe(
    prompt="A beautiful house in the style of Van Gogh",
    image=control_image,
    num_inference_steps=30
).images[0]

Available ControlNets

ControlNet	Input Type	Use Case
canny	Edge maps	Preserve structure
openpose	Pose skeletons	Human poses
depth	Depth maps	3D-aware generation
normal	Normal maps	Surface details
mlsd	Line segments	Architectural lines
scribble	Rough sketches	Sketch-to-image

LoRA adapters

Load fine-tuned style adapters:

from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5",
    torch_dtype=torch.float16
).to("cuda")

# Load LoRA weights
pipe.load_lora_weights("path/to/lora", weight_name="style.safetensors")

# Generate with LoRA style
image = pipe("A portrait in the trained style").images[0]

# Adjust LoRA strength
pipe.fuse_lora(lora_scale=0.8)

# Unload LoRA
pipe.unload_lora_weights()

Multiple LoRAs

# Load multiple LoRAs
pipe.load_lora_weights("lora1", adapter_name="style")
pipe.load_lora_weights("lora2", adapter_name="character")

# Set weights for each
pipe.set_adapters(["style", "character"], adapter_weights=[0.7, 0.5])

image = pipe("A portrait").images[0]

Memory optimization

Enable CPU offloading

# Model CPU offload - moves models to CPU when not in use
pipe.enable_model_cpu_offload()

# Sequential CPU offload - more aggressive, slower
pipe.enable_sequential_cpu_offload()

Attention slicing

# Reduce memory by computing attention in chunks
pipe.enable_attention_slicing()

# Or specific chunk size
pipe.enable_attention_slicing("max")

xFormers memory-efficient attention

# Requires xformers package
pipe.enable_xformers_memory_efficient_attention()

VAE slicing for large images

# Decode latents in tiles for large images
pipe.enable_vae_slicing()
pipe.enable_vae_tiling()

Model variants

Loading different precisions

# FP16 (recommended for GPU)
pipe = DiffusionPipeline.from_pretrained(
    "model-id",
    torch_dtype=torch.float16,
    variant="fp16"
)

# BF16 (better precision, requires Ampere+ GPU)
pipe = DiffusionPipeline.from_pretrained(
    "model-id",
    torch_dtype=torch.bfloat16
)

Loading specific components

from diffusers import UNet2DConditionModel, AutoencoderKL

# Load custom VAE
vae = AutoencoderKL.from_pretrained("stabilityai/sd-vae-ft-mse")

# Use with pipeline
pipe = DiffusionPipeline.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5",
    vae=vae,
    torch_dtype=torch.float16
)

Batch generation

Generate multiple images efficiently:

# Multiple prompts
prompts = [
    "A cat playing piano",
    "A dog reading a book",
    "A bird painting a picture"
]

images = pipe(prompts, num_inference_steps=30).images

# Multiple images per prompt
images = pipe(
    "A beautiful sunset",
    num_images_per_prompt=4,
    num_inference_steps=30
).images

Common workflows

Workflow 1: High-quality generation

from diffusers import StableDiffusionXLPipeline, DPMSolverMultistepScheduler
import torch

# 1. Load SDXL with optimizations
pipe = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
    variant="fp16"
)
pipe.to("cuda")
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
pipe.enable_model_cpu_offload()

# 2. Generate with quality settings
image = pipe(
    prompt="A majestic lion in the savanna, golden hour lighting, 8k, detailed fur",
    negative_prompt="blurry, low quality, cartoon, anime, sketch",
    num_inference_steps=30,
    guidance_scale=7.5,
    height=1024,
    width=1024
).images[0]

Workflow 2: Fast prototyping

from diffusers import AutoPipelineForText2Image, LCMScheduler
import torch

# Use LCM for 4-8 step generation
pipe = AutoPipelineForText2Image.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16
).to("cuda")

# Load LCM LoRA for fast generation
pipe.load_lora_weights("latent-consistency/lcm-lora-sdxl")
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)
pipe.fuse_lora()

# Generate in ~1 second
image = pipe(
    "A beautiful landscape",
    num_inference_steps=4,
    guidance_scale=1.0
).images[0]

Common issues

CUDA out of memory:

# Enable memory optimizations
pipe.enable_model_cpu_offload()
pipe.enable_attention_slicing()
pipe.enable_vae_slicing()

# Or use lower precision
pipe = DiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16)

Black/noise images:

# Check VAE configuration
# Use safety checker bypass if needed
pipe.safety_checker = None

# Ensure proper dtype consistency
pipe = pipe.to(dtype=torch.float16)

Slow generation:

# Use faster scheduler
from diffusers import DPMSolverMultistepScheduler
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)

# Reduce steps
image = pipe(prompt, num_inference_steps=20).images[0]

When not to use it

If you do not have a local GPU, or if you prefer a managed API, consider alternatives. DALL-E 3 works without GPU hardware. Midjourney is better for artistic, stylized outputs. Imagen integrates with Google Cloud. Leonardo.ai provides a web-based creative workflow. This skill is for local, scriptable, and customizable generation.

Limits and gotchas

• The skill requires a CUDA GPU. CPU-only inference is not covered here.
• Memory usage scales with model size, image resolution, and batch size. Use the memory optimization methods if you hit limits.
• The strength parameter in img2img controls how much of the original image is preserved. A value of 0.75 means 75% of the denoising process is applied to the new prompt.
• For inpainting, the mask image must have white pixels where you want to fill and black pixels elsewhere.
• ControlNet models are model-specific. The example uses a Canny ControlNet for SD 1.5. SDXL ControlNets exist but are not shown here.
• LoRA adapters must be compatible with the base model. Loading a LoRA trained on SD 1.5 into an SDXL pipeline will fail.
• The safety_checker can be set to None if you see black images, but this disables content filtering.

What pairs with this

For custom pipelines, fine-tuning, and deployment, see the Advanced Usage reference. For more common issues and solutions, see the Troubleshooting reference. The upstream Diffusers documentation is at https://huggingface.co/docs/diffusers. The Diffusers repository is at https://github.com/huggingface/diffusers. Models are listed on the HuggingFace Model Hub. Community support is available on the Diffusers Discord.