Training Your Own Stable Diffusion LoRA for Self-Portraits

• 22/3/2025
• 8-minute read

The ability to generate photorealistic images of yourself in different styles, environments, and scenarios has become increasingly accessible through AI image generation. Low-Rank Adaptation (LoRA) offers an efficient way to personalize Stable Diffusion models with just a handful of personal photos. In this tutorial, I’ll walk you through creating your own LoRA model to generate high-quality, diverse self-portraits that actually look like you.

What is LoRA and Why Use It for Self-Portraits?

When I first tried creating AI versions of myself, the results were… well, let’s just say they looked like my distant cousin from another dimension! That’s because general models don’t know what makes you uniquely you. This is where LoRA comes in.

LoRA is a parameter-efficient fine-tuning method that allows you to customize large language and image generation models without modifying all their parameters. For self-portraits, this means you can “teach” Stable Diffusion what you look like using just 15-30 photos, while preserving the model’s ability to generate diverse, creative outputs.

The advantages of using LoRA include:

• Efficiency: Requires significantly less computational resources than full model fine-tuning (my laptop thanked me!)
• Small file size: Resulting models are typically only 40-150MB (perfect for sharing with friends)
• Faster training: Complete training in 1-2 hours on consumer GPUs (enough time to make a nice meal while waiting)
• Blend control: Adjust how strongly your likeness appears in generations (from subtle resemblance to “definitely you”)

Preparing Your Training Dataset

The quality of your training images directly impacts the success of your LoRA model. Trust me on this one—I learned it the hard way after my first attempt gave me three eyes! Here’s how to create an optimal dataset:

Image Selection Guidelines

• Quantity: 15-30 images minimum (20-25 is ideal for most uses). I used 22 in my successful model.
• Diversity: Include various angles, expressions, lighting conditions. Remember that time I only used forward-facing photos? The side profile generations were… creative!
• Consistency: All photos should be clearly of the same person (no group photos where the model gets confused about who you are)
• Quality: High-resolution, well-lit images work best. Those dim selfies from 2 AM? Maybe leave those out.
• Background: Simple or removable backgrounds are preferred. I spent an extra hour removing backgrounds, and it was worth every minute.

Image Preparation Process

1. Resolution: Resize images to 512×512 or 768×768 pixels (square aspect ratio). I prefer 768×768 for more detail.
2. Background removal: Use tools like remove.bg or Photoshop. This was a game-changer for my training quality!
3. Face prominence: Your face should occupy approximately 40-60% of the frame. Too small and the model won’t learn your features; too large and you’ll get weird cropping in generations.
4. File format: Save as PNG or high-quality JPG. I stick with PNG for training—why compress what you don’t need to?
5. Organization: Create a dedicated folder for your training images. Future you will thank present you for staying organized!

Remember to avoid overtraining by ensuring diversity in your dataset. Including too many similar images can lead to overfitting and reduced creativity in the results. My first model could only generate me with the exact same expression because I used too many similar photos!

Setting Up Your Training Environment

You can train a LoRA model using various platforms. Here are the most popular options—I’ve personally tried all of them and will share my experiences:

Option 1: Using Google Colab (Beginner-Friendly)

Google Colab offers free GPU access, making it ideal for those without powerful hardware. This is how I created my first successful LoRA:

1. Visit TheLastBen’s fast-stable-diffusion Colab
2. Run the setup cells (get a coffee while it installs dependencies)
3. Enable the training tab
4. Upload your prepared images (this took me about 10 minutes with my internet speed)
5. Configure parameters as shown in the table below (I’ll share what worked for me)

Option 2: Using Kohya SS GUI (Intermediate)

After outgrowing Colab, I moved to this option for more control:

1. Download Kohya SS
2. Install dependencies following the README (prepare for some troubleshooting if you’re on Windows like me!)
3. Launch the GUI and navigate to the LoRA tab
4. Configure parameters and point to your image folder

Option 3: Using FluxGym (Beginner to Intermediate)

FluxGym is a user-friendly web UI specifically designed for training FLUX LoRAs with support for lower VRAM GPUs (12GB/16GB/20GB). I found this option perfect when I upgraded from Colab but wasn’t ready for command-line work yet:

1. Clone the repository: git clone https://github.com/cocktailpeanut/fluxgym
2. Navigate to the folder: cd fluxgym
3. Clone the Kohya scripts: git clone -b sd3 https://github.com/kohya-ss/sd-scripts
4. Run with Docker (recommended for simple setup):

   docker compose up -d --build
   

5. Open your browser and navigate to: http://localhost:7860

Using FluxGym is straightforward:

1. Enter your LoRA information (name, trigger word)
2. Upload your prepared images and caption them with your trigger word
3. Click “Start” and let it train

What I love about FluxGym is its automatic sample image generation—you can configure it to show progress images every N steps, which helped me catch training issues early. If you want more control, there’s also an “Advanced” tab that gives you access to all of Kohya’s training options.

FluxGym automatically downloads the models when you select them, supporting Flux1-dev, Flux1-dev2pro, and others without manual downloads. It’s been a fantastic middle ground between the simplicity of Colab and the power of command-line options!

Option 4: Command Line (Advanced)

For you fellow terminal enthusiasts (I eventually graduated to this method):

# Clone the repository
git clone https://github.com/kohya-ss/sd-scripts
cd sd-scripts

# Install dependencies
pip install -r requirements.txt

# Run training
accelerate launch --num_cpu_threads_per_process 8 train_network.py \
  --pretrained_model_name_or_path="runwayml/stable-diffusion-v1-5" \
  --train_data_dir="./your_training_images" \
  --output_dir="./output" \
  --caption_extension=".txt" \
  --max_train_epochs=15 \
  --learning_rate=1e-4 \
  --lr_scheduler="cosine_with_restarts" \
  --network_module="networks.lora" \
  --network_dim=32 \
  --network_alpha=16

Critical Training Parameters

Tuning these parameters significantly affects your results. I’ve spent countless hours experimenting with these values (so you don’t have to!):

Example Training Configuration

For most self-portrait LoRAs, these settings work well (and have been tested through my many trials and errors):

Base model: runwayml/stable-diffusion-v1-5
Network Dim: 64
Network Alpha: 32
Learning rate: 2e-4
Epochs: 15-20
Clip skip: 2
Resolution: 768×768
Caption: "photo of sanj" (replace with your chosen trigger word)

Testing and Using Your LoRA

Once training completes, you’ll have a .safetensors file containing your LoRA weights. I still remember the excitement of testing my first successful model! Here’s how you can use yours:

1. Install the AUTOMATIC1111 WebUI
2. Place your .safetensors file in the models/Lora directory
3. Restart the WebUI if it’s running
4. In the text prompt, add your trigger word and <lora:your_file_name:0.8> where 0.8 is the strength

Effective Prompting Techniques

The key to getting diverse yet accurate self-portraits lies in the prompt. After hundreds of generations, here are my tried-and-true prompting patterns:

• Basic format: photo of sanj, [style description], [setting], [lighting], [camera details] <lora:sanj_lora:0.8>
• Style examples: portrait photography, cinematic, film grain, cyberpunk, anime, oil painting
• Lighting examples: golden hour, blue hour, studio lighting, dramatic shadows
• Camera details: 85mm portrait lens, Leica, Canon, bokeh, shallow depth of field

My personal favorite prompt? photo of sanj, wearing explorer outfit, in dense jungle, cinematic lighting, National Geographic style, 85mm lens, detailed, 4k <lora:sanj_lora:0.85>

Negative prompt example: deformed, distorted, disfigured, poorly drawn, bad anatomy, wrong anatomy, extra limb, missing limb, floating limbs, disconnected limbs, mutation, mutated, ugly, disgusting, blurry, out of focus

Troubleshooting Common Issues

Been there, fixed that! Here are solutions to problems I’ve personally encountered:

Ethical Considerations and Best Practices

When creating AI self-portraits, I always follow these guidelines and recommend you do too:

• Keep your LoRA model secure, especially if you plan to create photorealistic images
• Be transparent when sharing AI-generated images of yourself (I use #AIGenerated in my posts)
• Respect copyright when using your model with celebrity or character prompts
• Consider watermarking AI-generated content if sharing publicly

This technology opens creative possibilities but comes with responsibilities. Use it ethically and safely. I’ve had so much fun creating alternate versions of myself—from “me as a cyberpunk hacker” to “me in Renaissance painting style”—without crossing ethical lines.

Further Reading and Resources

Want to dive deeper? These are the resources that helped me master LoRA training:

• Hugging Face LoRA Training Documentation
• Original LoRA Paper and Implementation
• Effective Training for LoRA Models in Stable Diffusion
• Stable Diffusion Art: LoRA Training Guide
• Kohya’s LoRA Training Scripts