Stable Diffusion: Exploring trained models and unique tokens for more accurate text-to-image results

Before I get start­ed, I’ve got a pre­am­ble to this post. I’m only in the ear­ly dis­cov­ery process of AI-pow­ered text-to-image tech­nol­o­gy. There’s a good chance you know more than I do about this top­ic! I’m writ­ing this to doc­u­ment what I’ve learned. If you’re read­ing this, you might be on the same jour­ney. If so, I hope you find this help­ful.

If you peaked ahead in this post, you’d see all kinds of images based on the prompt, a tiger sitting on a log, and some results are not that impres­sive. I’ve used this basic prompt inten­tion­al­ly because it helps illus­trate two thoughts I want­ed to share. 

The first take­away is that the prompt you give these tools is incred­i­bly impor­tant. For exam­ple, take a peek at a col­lec­tion of sea otter images cre­at­ed with DALL‑E 2. If you hov­er over the images on that page, you’ll see the text that gen­er­at­ed the images.

DALL-E 2 - Sea Otters prompt example

The text you pro­vide will sig­nif­i­cant­ly impact the image you get out. That makes sense, and this ​“sea otter” page offers ample proof.

The sec­ond take­away I want to address here is that the mod­el is even more impor­tant. This is a con­cept I didn’t grasp until I start­ed exper­i­ment­ing with the same basic prompt and dif­fer­ent data mod­els. Sta­ble Dif­fu­sion, which I will get to in a moment, is how I’ve been exper­i­ment­ing with mod­els.

In short, there’s a lot I don’t know, but I’m learn­ing. That’s the end of the pre­am­ble. Let’s get start­ed.

Before Sta­ble Dif­fu­sion, DALL‑E #

In ear­ly 2022, text-to-image cre­ation tools seemed to be every­where online. I couldn’t wait to try them myself. The first ver­sion I tried was DALL‑E, aka DALL‑E ver­sion 1. You can eas­i­ly try DALL‑E 1 in your brows­er at https://​www​.craiy​on​.com/. If you type in ​“a tiger sit­ting on a log,” you will get some­thing like what you see below.

Prompt: "a tiger sitting on a log" | Engine: DALL-E 1

Tech­nol­o­gy has raced beyond this ear­ly ver­sion of DALL‑E, but these images are still impres­sive. Seem­ing­ly like mag­ic, some images appear that are rough rep­re­sen­ta­tions of what you typed. Yes, they look a lit­tle wonky, but they didn’t exist until you typed in that phrase. You are a pow­er­ful wiz­ard. :mage:‍♂️

In 2022, the suc­ces­sor to DALL‑E was released by Ope­nAI. DALL‑E 2 was a giant leap for­ward. It had improved image qual­i­ty and a bet­ter text inter­preter. The same prompt, a tiger sitting on a log, pro­duced great­ly improved results.

Prompt: "a tiger sitting on a log" | Engine: DALL-E 2

Sta­ble Dif­fu­sion #

Last week I had time to exper­i­ment with Sta­ble Dif­fu­sion. What’s the dif­fer­ence? I’ll quote from the Sta­ble Dif­fu­sion arti­cle on Wikipedia.

Unlike mod­els like DALL‑E, Sta­ble Dif­fu­sion makes its source code avail­able, along with pre­trained weights. […] The user owns the rights to their gen­er­at­ed out­put images, and is free to use them com­mer­cial­ly.

In oth­er words, you can run Sta­ble Dif­fu­sion on your com­put­er and tin­ker with it. Sta­ble Dif­fu­sion claims no copy­right on the images you cre­ate. 

Dur­ing DALL‑E 2 pri­vate beta, the images were not avail­able for com­mer­cial use, but when DALL‑E 2 opened up into pub­lic beta, the cre­ators, Ope­nAI, updat­ed their terms of use, giv­ing users own­er­ship of the images they cre­ate. Their press release said, ​“users get full usage rights to com­mer­cial­ize the images they cre­ate with DALL·E, includ­ing the right to reprint, sell, and mer­chan­dise.”

DALL‑E 2 has an easy-to-use web inter­face and gen­er­ates won­der­ful results, but I don’t see any way to manip­u­late the mod­el. (Could I be wrong about that? Absolute­ly!)

Sta­ble Dif­fu­sion, being open, has a grow­ing com­mu­ni­ty of hack­ers push­ing it for­ward. I installed Sta­ble Dif­fu­sion on my Mac to explore some of these exper­i­ments.

Installing Sta­ble Dif­fu­sion #

There are a vari­ety of ways of installing Sta­ble Dif­fu­sion on your com­put­er. As of Novem­ber 2022, the moment I’m writ­ing this, my advice is to not do this unless you are com­fort­able work­ing on your computer’s com­mand line. Installing this is exper­i­men­tal at best, and you could break things. Seri­ous­ly. Pro­ceed with cau­tion. If you can wait, these tools will become eas­i­er to use over time. 

If you want to exper­i­ment with­out installing this on your machine, a search engine called Lex­i­ca will allow you to search for images oth­ers have cre­at­ed, and if you make a free account, you can use Sta­ble Dif­fu­sion on their site. You won’t be able to use any cus­tom mod­els, but you won’t have to install any­thing on your machine.

Lexica, a Stable Diffusion search engine

Ok, now that you’ve decid­ed to install this stuff any­way, good luck! :stuck_​out_​tongue_​winking_​eye:

I’ve installed sev­er­al ver­sions of Sta­ble Dif­fu­sion dur­ing my explo­ration, and InvokeAI is the one that has the eas­i­est instal­la­tion process. There are instruc­tions for many plat­forms. Once installed, Invoke can be used from the com­mand line or via the web inter­face it pro­vides. (Since writ­ing this post, I’ve heard about Charl‑e, for the Mac. I’ve not tried it, though. Thanks for the tip, Rudy.)

The Invoke AI web interface

What does Sta­ble Dif­fu­sion do with the same prompt we tried with DALL‑E? Here are three results.

Prompt: "a tiger sitting on a log" | model: sd-v1-5.ckpt | Engine: Stable Diffusion

I think these are impres­sive results. Notice I’ve includ­ed an addi­tion­al note about the ​“mod­el” above. The mod­el here, sd-v1-5.ckpt, is the model’s file­name con­tain­ing the AI​’s text and image train­ing data. This mod­el lives at https://huggingface.co/runwayml/stable-diffusion‑v1‑5. You can see exact­ly what this mod­el is sup­posed to do.

Sta­ble Dif­fu­sion is a latent text-to-image dif­fu­sion mod­el capa­ble of gen­er­at­ing pho­­to-real­is­tic images giv­en any text input.

It’s trained for pho­­to-real­ism. That’s prob­a­bly why the basic prompt I’ve used is return­ing some­thing that looks like a pho­to instead of an illus­tra­tion.

If you want a deep dive into what mod­els do, check out Sta­ble Dif­fu­sion with Dif­fusers, where the authors ​“explain how the mod­el works and final­ly dive a bit deep­er into how dif­fusers allow one to cus­tomize the image gen­er­a­tion pipeline.” I’ll let them explain the tech­ni­cal details because they’re good at it.

Adding style detail to the prompt #

As you saw ear­li­er with the sea otter exam­ple from DALL‑E, you can tai­lor your prompts with more descrip­tive words to get bet­ter results. If we don’t want pho­to real­ism as the style of our images, we need to be more detailed with our request. 

Let’s say instead pho­­to-real­is­tic tigers, you want­ed your tigers to look as if they were drawn by Hol­lie Mengert. Below is an exam­ple of her char­ac­ter design.

Piper Coyote by Hollie Mengert

Own­er­ship and ethics #

Before we go any fur­ther, we must acknowl­edge this art­work is Hollie’s pro­fes­sion. Cre­at­ing her art­work is how she makes a liv­ing. For con­text, please read Inva­sive Dif­fu­sion: How one unwill­ing illus­tra­tor found her­self turned into an AI mod­el. I’m writ­ing anoth­er piece regard­ing the top­ic, which I plan to post to my blog. I will update this post with a link when it’s ready.

Adding style detail to the prompt (cont.) #

Using the same pho­­to-real­is­tic mod­el we’ve been using so far, chang­ing our prompt to hollie mengert artstyle tiger sitting on a log cre­ates a very dif­fer­ent out­come.

Prompt: "hollie mengert artstyle tiger sitting on a log" | model: sd-v1-5.ckpt | Engine: Stable Diffusion

Although I like that first tiger’s hand-sketched style, these images do not look like Hollie’s work. The sd‑v1‑5.ckpt mod­el doesn’t know what Hollie’s style is beyond the fact that it’s not a pho­to­graph­ic style. That leads to an obvi­ous ques­tion. Can you teach an old mod­el new tricks? Yes, you can. 

Dream­Booth and train­ing mod­els #

There’s a project called Dream­Booth that offers ​“a new approach for ​‘per­son­al­iza­tion’ of text-to-image dif­fu­sion mod­els” to ​“fine-tune a pre-trained text-to-image mod­el.”

The Illus­­tra­­tion-Dif­­fu­­sion mod­el used some of Hollie’s work and trained it to mim­ic her style. The doc­u­men­ta­tion for this mod­el says, ​“the cor­rect token is hol­liemengert art­style.”

You might won­der if there is a typo in the token. Is it miss­ing a space? It’s not. This miss­ing space is inten­tion­al. The token is hol­liemengert and not hol­lie mengert. This mod­el con­tains data derived from images tagged with the made-up word hol­liemengert. You’ll see many of these made-up words if you use cus­tom mod­els. Using unique words allows the new data to be iso­lat­ed and not get pol­lut­ed with oth­er data. Imag­ine the images had been tagged with the word draw­ing which already con­tains lots of data. You’d have no way to tell Sta­ble Dif­fu­sion that you meant the ​“hol­lie mengert” style with­out it being pol­lut­ed with many oth­er draw­ing styles.

In the images above, where we asked for ​“hol­lie mengert art­style”, I’m a lit­tle sur­prised we didn’t end up with some hol­ly (not Hol­lie) in the images.

Christmas card with an illustration of holly

Let’s run our new prompt,holliemengert artstyle tiger sitting on a log, through Sta­ble Dif­fu­sion using the Illus­­tra­­tion-Dif­­fu­­sion mod­el, aka hollie-mengert.ckpt.

Prompt: "holliemengert artstyle tiger sitting on a log" | model: hollie-mengert.ckpt | Engine: Stable Diffusion

The results are remark­ably dif­fer­ent. These images are not the same qual­i­ty as would have been cre­at­ed by Hol­lie, but they have a much more dis­tinc­tive style. These results are heav­i­ly influ­enced by the new mod­el we’re using.

Using the model with the wrong ​“token” #

How impor­tant is the token holliemengert artstyle in the prompt now that we’re using this new­ly trained mod­el? What if you want­ed the tiger to be influ­enced by clas­sic Dis­ney movies instead of Hollie’s work?

Here’s what hap­pens if we try the token classic disney style as part of the prompt but still use the Illus­tra­tion-Dif­fu­sion mod­el trained on the Hol­lie Mengert art.

Prompt: "classic disney style tiger sitting on a log" | model: hollie-mengert.ckpt | Engine: Stable Diffusion

The results are not exact­ly in the style of Hol­lie Mengert, but they’re not real­ly in the Dis­ney style either. The mod­el is pick­ing up some influ­ences to Dis­ney style but doesn’t seem quite right either. 

Clas­sic Ani­ma­tion Dif­fu­sion #

Let’s take a look at a dif­fer­ent mod­el. The clas­sic-anim-dif­­fu­­sion mod­el, a ​“mod­el trained on screen­shots from a pop­u­lar ani­ma­tion stu­dio,” (cough, cough) uses the token classic disney style as a style prompt. We’ll switch the mod­el to this Dis­­ney-trained mod­el using the same prompt that gen­er­at­ed the pre­vi­ous images.

Prompt: "classic disney artstyle tiger sitting on a log" | model: classicAnim-v1.ckpt | Engine: Stable Diffusion

Once again, there is an obvi­ous shift in the style. The first two images are not keep­ers. The third is accept­able. They look like Dis­ney tigers though, so the exper­i­ment worked.

Super­hero Dif­fu­sion #

To wrap up these exper­i­ments, we’ll try anoth­er mod­el called Super­hero-Dif­­fu­­sion trained on Pepe Larraz’s work (you can see real exam­ples of his work with this Google Image search) that uses the token comicmay artstyle. Again, we’re keep­ing the same basic prompt and chang­ing the token and the mod­el.

Prompt: "comicmay artstyle tiger sitting on a log" | model: superhero-diffusion.ckpt | Engine: Stable Diffusion

Whoa. These look cool to me. I notice that the tiger we asked to be sit­ting is actu­al­ly stand­ing, but the tiger is at least perched on a log. I’m sure if Pepe Lar­raz could cre­ate a sit­ting tiger on his first attempt, but that’s a dif­fer­ent dis­cus­sion entire­ly.

What about Mid­jour­ney? #

There is anoth­er text-to-image project that I’m aware of called Mid­jour­ney. I have seen impres­sive results post­ed on Twit­ter with the hash­tag #mid­jour­ney, but I have not used Mid­jour­ney dur­ing my explo­ration. If you’re explor­ing, you may want to check it out also.

Wrap­ping up #

I would be sur­prised if you’ve made it this far and haven’t felt the urge to try mak­ing some images of your own. Lex­i­ca is prob­a­bly the eas­i­est place to begin. Exper­i­ment with your prompt, and you’ll be amazed at your results even with the base mod­el. Also, real­ize that ver­sion 1.5 is the ver­sion of the mod­el that’s cur­rent as I write this, but I expect that to evolve quick­ly. The pace of change with these mod­els is incred­i­bly rapid, and the results will only improve with time. 

If you want to exper­i­ment with dead artists whose work has seeped deeply into the cul­ture and prob­a­bly into the data mod­els, try adding Leonardo da Vinci art style or frida kahlo art style and see what hap­pens. 

Last­ly, I would encour­age you to think about liv­ing artists whose work influ­ence data mod­els. Are you com­fort­able cre­at­ing art based on a liv­ing artist? Does it mat­ter if it’s only for per­son­al explo­ration? Does it cross an eth­i­cal line for you if you used this art­work in a way that you made mon­ey from? Would you pay for access to a mod­el?