The Handy Guide to Stable Diffusion Prompts

Stable Diffusion can take AI image generation to the next level with plenty of detailed settings to explore, but this also means that getting good results isn’t exactly intuitive. Understanding just a handful of key techniques, however, will help you get past this hurdle.

A Quick Overview of Stable Diffusion: Getting Good Images

There are plenty of AI image generators out there and Stable Diffusion is among the most popular owing to its open-source nature and the advanced control you have over image generation.

But this also means that it can be difficult to get good-looking images right away. I’ve spent a lot of time using the AI generator Midjourney, which produces aesthetically pleasing images almost effortlessly. By comparison, Stable Diffusion takes a bit more effort to learn.

The quickest way to get good results is to write a detailed description, add a negative prompt to avoid unwanted elements, and then refine the image by changing the weighting for keywords and ideas in your prompt.

Master these basics, and you’ll grow confident that you can create really great images every time.

Writing a Detailed Description

Let’s start with the description. With Stable Diffusion, you need to write detailed prompts if you want to produce stunning images. Studying other people’s prompts is a fantastic way to see how it’s done.

We’ll use the video above as an example. The prompt looks like this:

Portrait of a cosmic goddess, suit made out of stars and galaxies and cosmic energy, intricate, headshot, highly detailed, digital painting, artstation, concept art, sharp focus, cinematic lighting, illustration, art by argerm and greg rutkewski, alphonese much, cgsociety

This prompt is made up of 38 words, which gives you a good idea of how far you need to take your description. Thankfully, it’s not too complicated to grasp what type of words are being used in this example. I did a quick analysis of this prompt and found that there are seven categories you could split the keywords into.

• Subject: Portrait of a cosmic goddess, a suit made out of stars and galaxies and cosmic energy
• Shot: Headshot
• Lighting: Cinematic lighting
• Detail: Intricate, highly detailed, sharp focus
• Art style: Digital painting, concept art, illustration
• Artist: Art by Argerm and Greg Rutkewski, Alphonse Mucha
• Community References: ArtStation, CGsociety

Using a prompt template can help a lot with coming up with an initial idea. Here’s one that the author of the video uses:

Subject, action, background, lighting, style, artist

There’s no hard rule about what keywords to include, or in what order, with the exception that words at the start of the prompt are given greater importance and will feature more prominently in the image.

Here’s one more template you can experiment with:

• Subject: e.g. woman, man, centaur
• Pose: e.g. standing, sitting, looking out the window, sleeping
• Details: e.g. clothing, hair color, accessories, props
• Environment: e.g. office, beach, forest, medieval castle
• Quality descriptors: fine details, intricate

Over time, you might settle on a template that gets the best results for you. Until then, however, you can use these templates to get started.

Using Negative Prompts

Negative prompts is a tool you can use to improve your images by telling the AI system what you don’t want to see. If there are random people in your image, you can write “no people” in the negative prompt to try to remove them.

At first, it might seem like a magic fix, but that’s not exactly true. Sometimes adding just one extra word to a negative prompt will suddenly and drastically alter your image in ways you wouldn’t expect.

The relationship between the negative prompt and the image can often be difficult to understand. There seem to be no universal negative prompts that work every time. However, there are some generally popular words that the Stable Diffusion community have come to use.

This YouTuber offers the following negative prompt template for you to try out:

kitsch, ugly, oversaturated, grain, low-res, blurry, bad anatomy, poorly drawn face, extra limb, poorly drawn hands, missing limb, floating limbs, disconnected limbs, malformed hands, blur, out of focus, long neck, long body, ugly, poorly drawn, childish, mangled, old, surreal

In another interesting case, this Reddit user explains how and why they use the following negative prompt:

Caricature, body horror, mutant, facebook, youtube, food

The first three should be obvious for why I don’t want them influencing my pictures of regular people. The latter three are an attempt to filter out a large part of the search engine optimization (seo) tag cloud that contaminates lots of the images in the dataset.

Try some of these negative prompt tags for yourself to see how they influence your images. It’s helpful to start small, with only a few words in your negative prompt, so you can see how the words are affecting the image.

Refining Your Results

Once you get the hang of writing detailed image prompts and making use of the negative prompt tool to remove unwanted elements in the image, the next best thing to learn is how to “weight” certain words to emphasize parts of your image.

You can do this by adding brackets to any word in the prompt that you want the AI system to focus on, relative to the other words in your description. If you want to emphasize “banana”, for example, you might write the following prompt: A bowl of fruit, (bananas), apples, strawberry.

In this video, you can see multiple examples of what happens to an image when the weight is placed on the different fruit in the description. Adding brackets to a word will increase the importance of that element in your image. You can add multiple brackets to add greater weight.

You can also highlight the word and press Ctrl or Command, plus the Up or Down arrow to change the weight. If you use this shortcut, it will also add a value next to the word.

By default, each keyword in your prompt will have a weight of 1. You can use that as a starting point for increasing or decreasing the weight. A weight of 2 or 3 will produce extreme results, for example, so it’s best to move in small increments like 1.2 or 1.3, etc. Playing with word weighting is one of the easiest ways to craft your image further.

Starting with a strong description with plenty of details, then moving on to negative prompts and keyword weighting, are some of the best fundamentals to learn in Stable Diffusion. With these points in mind, you’ll be able to start creating the exact images you want.