Running Stable Diffusion Image Generation on Your CPU

Run Stable Diffusion on Your CPU

AI image generation has been making a lot of buzz lately. It’s an exciting space due to the systems like DALL·E and Midjourney becoming available for people outside the AI world to play with. However, the one I find compelling, from a tech nerd viewpoint, is Stable Diffusion because it is open source and can run on local hardware. So, of course, I have to try it out and see how it worked. 

Getting Stable Diffusion running on my hardware is more complicated than at first glance. Out of the box, the project is designed to run on the PyTorch machine learning framework. That can be a problem because PyTorch only supports hardware acceleration using NVIDIA’s CUDA API, AMD’s ROCM API, or Apple’s Metal API (using MPS). What if you don’t have a system supported by those APIs, like most laptops? There is another option.

PyTorch can also be run on your CPU, assuming the programming has considered that. Unfortunately, the programmers of Stable Diffusion did not. Running the txt2img script results in an error saying, “Found no NVIDIA driver on your system.” All hope is not lost. Other users have looked at the code and created a pull request in the GitHub repository that fixes Stable Diffusion to work CPUs. Most of the work involves adding IF statements only to use CUDA if a NIVIDIA GPU is detected.

The Setup

  1. Install a python environment. This can be as simple as installing Python on your system and running everything in a single environment. However, using Python virtual environments is recommended to isolate projects from each other and reduce the risk of package conflicts.
    1. One of the easiest methods for setting up virtual environments is to use Conda. That is what the rest of the instructions will use.
  2. Install Git. Git is version control software for tracking code changes and pushing/pulling those changes from a central repository.
  3. Go to https://huggingface.co/. This is where the Stable Diffusion weights are hosted.
    1. First, you will need to create a free account on the site.
    1. Go to https://huggingface.co/CompVis/stable-diffusion-v-1-4-original and click the “Access repository” button. By doing this, you are sharing your contact info with the Stable Diffusion authors.
    1. Download the sd-v1-4.ckpt file.
  4. Launch a Conda command prompt.
  5. Download the Stable Diffusion repository from GitHub using Git and go to the directory.
    https://github.com/CompVis/stable-diffusion.git
    cd stable-diffusion
  6. Use an already created pull request to modify Stable Diffusion to work on CPUs.
    git fetch origin pull/56/head:pr56
    git checkout pr56
  7. Follow the rest of the setup process from Stable Diffusion’s README.
  8. Setup the Python environment in Conda
    conda env create -f environment.yaml
    conda activate ldm
  9. Create a new folder called stable-diffusion-v1 under .\models\ldm
  10. Copy the sd-v1-4.ckpt file from earlier into the stable-diffusion-v1 folder and rename it to model.ckpt
  11. The environment setup is now complete.

How to generate an image

Now that the Python environment is setup, the last thing to do is try to generate an image. That’s where my script makes it easy.

  1. Launch a Conda command prompt.
  2. Activate the Python environment created earlier.
    conda activate ldm
    cd stable-diffusion
  3. Run the following command to generate an image. The prompt can be change to whatever text you want.
    python scripts\txt2img.pny –prompt "astronaut riding a horse" --plms

Troubleshooting/FAQ

  • Why are my images of Rick Astley?
    • The NSFW safety checker thought the image had naughty bits. Try a different seed number or prompt. I’ve also seen that reducing the image size can cause the safety checker to flag the image.

Have fun.
-Tony