Stable Diffusion

Stable Diffusion

Purpose

• Use this skill to generate or edit images locally with Hugging Face diffusers.
• Prefer it for text-to-image, img2img, inpainting, and model composition workflows.
• This skill covers both classic Stable Diffusion models and SDXL.
• It is useful for scripted generation, reproducible experiments, and GPU-backed image pipelines.

Install

pip install diffusers transformers accelerate torch

• You typically also need a compatible CUDA-enabled PyTorch build for GPU inference.
• Confirm the install in Python before pulling large checkpoints.

Core Libraries

• diffusers for pipeline abstractions
• transformers for text encoders and related model components
• accelerate for efficient device loading and memory movement
• torch for runtime execution

Text-to-Image With SD 1.5

• Classic Stable Diffusion 1.5 uses StableDiffusionPipeline.

import torch
from diffusers import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    torch_dtype=torch.float16,
)
pipe = pipe.to("cuda")

image = pipe(
    prompt="a cinematic photo of a mountain observatory at sunrise",
    negative_prompt="blurry, low quality, distorted",
    num_inference_steps=30,
    guidance_scale=7.5,
).images[0]

image.save("output.png")

SDXL

• SDXL typically uses StableDiffusionXLPipeline.

import torch
from diffusers import StableDiffusionXLPipeline

pipe = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
)
pipe = pipe.to("cuda")

image = pipe(
    prompt="a highly detailed editorial photo of a futuristic library interior",
    negative_prompt="low resolution, deformed, extra limbs",
    num_inference_steps=35,
    guidance_scale=6.5,
).images[0]

image.save("sdxl-output.png")

• SDXL generally produces stronger prompt fidelity and image quality than SD 1.5, but it also requires more VRAM.

Key Parameters

• prompt: the main text instruction
• negative_prompt: what to suppress
• num_inference_steps: denoising step count
• guidance_scale: classifier-free guidance strength
• seed: random seed for reproducibility

Seeded Generation

• Use a seed when you need repeatable outputs:

import torch
from diffusers import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    torch_dtype=torch.float16,
).to("cuda")

generator = torch.Generator(device="cuda").manual_seed(42)

image = pipe(
    prompt="a clean product photo of a ceramic mug on a wood table",
    negative_prompt="blurry, noisy, warped",
    num_inference_steps=28,
    guidance_scale=7.0,
    generator=generator,
).images[0]

image.save("seeded-output.png")

• The same seed and settings help with debugging prompt and LoRA changes.

Save Output

• Save the generated image with PIL:

image.save("output.png")

• Always save prompt metadata separately if you need auditability or experiment tracking.

Img2Img

• Use StableDiffusionImg2ImgPipeline to transform an existing image while preserving composition.

import torch
from diffusers import StableDiffusionImg2ImgPipeline
from PIL import Image

pipe = StableDiffusionImg2ImgPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    torch_dtype=torch.float16,
).to("cuda")

init_image = Image.open("input.png").convert("RGB").resize((768, 768))

image = pipe(
    prompt="turn this concept sketch into a polished sci-fi matte painting",
    negative_prompt="blurry, low contrast, artifacts",
    image=init_image,
    strength=0.65,
    num_inference_steps=30,
    guidance_scale=7.5,
).images[0]

image.save("img2img-output.png")

• Lower strength preserves more of the input image.
• Higher strength pushes the result further away from the source.

Inpainting

• Use StableDiffusionInpaintPipeline to replace or repair masked regions.

import torch
from diffusers import StableDiffusionInpaintPipeline
from PIL import Image

pipe = StableDiffusionInpaintPipeline.from_pretrained(
    "runwayml/stable-diffusion-inpainting",
    torch_dtype=torch.float16,
).to("cuda")

image = Image.open("scene.png").convert("RGB").resize((512, 512))
mask = Image.open("mask.png").convert("RGB").resize((512, 512))

result = pipe(
    prompt="replace the missing area with a wooden chair",
    negative_prompt="blurry, malformed, duplicate objects",
    image=image,
    mask_image=mask,
    num_inference_steps=30,
    guidance_scale=7.5,
).images[0]

result.save("inpaint-output.png")

• White mask regions are typically where edits are applied.
• Good masks matter as much as prompts for reliable inpainting.

ControlNet

• ControlNet is useful when you want stronger control over pose, depth, edges, or composition.
• Typical uses include pose-guided character generation, depth-aware edits, and line-art conditioning.
• Pair ControlNet with SD 1.5 or SDXL depending on the model combination you are using.

Memory Optimization

• Reduce memory pressure with built-in helpers:

pipe.enable_model_cpu_offload()
pipe.enable_attention_slicing()

• pipe.enable_model_cpu_offload() is often helpful on constrained GPUs.
• pipe.enable_attention_slicing() can reduce peak memory at some performance cost.
• These settings are practical for laptops and single-GPU consumer machines.

LoRA Loading

• Load LoRA adapters to specialize style, subject, or composition behavior:

pipe.load_lora_weights("./lora.safetensors")

• Keep the base model and LoRA pairing compatible.
• Track LoRA names, weights, and prompts in experiment logs.

Negative Prompts

• Common negative prompts include:

• blurry

• low quality

• worst quality

• deformed

• extra limbs

• bad anatomy

• artifact

• text

• watermark

• Use concise negative prompts first.

• Overly long negative prompts can produce unstable or muddled outputs.

Prompting Guidance

• Be concrete about subject, style, lighting, framing, and medium.
• Use short prompt iterations during tuning rather than changing many variables at once.
• Record prompt, negative prompt, seed, and model version together.

SDXL vs SD 1.5

• Use SDXL when:

• prompt fidelity matters

• you need stronger detail and composition

• you have enough VRAM

• Use SD 1.5 when:

• you need a lighter model

• you rely on mature community tooling

• you need broad LoRA and ControlNet ecosystem support