Running Google’s Gemma 4 Locally on macOS with Ollama

Large lan­guage mod­els don’t have to live in the cloud. With Olla­ma and a recent Mac, you can run Google’s Gem­ma 4 mod­els entire­ly on your own machine. No API keys. No usage lim­its. No data leav­ing your lap­top.

Here’s how I set it up.

Installing Ollama

Olla­ma is the eas­i­est way to run open-weight LLMs local­ly. On macOS, it’s a sin­gle Home­brew com­mand:

brew install ollama

Once installed, start the Olla­ma ser­vice:

brew services start ollama

That’s it. Olla­ma is now run­ning in the back­ground and ready to pull and serve mod­els.

Any guess­es on how to stop olla­ma? Easy!

brew services stop ollama

Pulling Gemma 4

Google’s Gem­ma 4 fam­i­ly comes in four sizes. The right choice depends on your avail­able RAM and what you need the mod­el for:

A good rule of thumb: you’ll want at least as much free RAM as the model’s disk size, plus a few extra giga­bytes for the sys­tem. The e2b and e4b vari­ants run well on 16 GB machines. The 26b and 31b vari­ants are more com­fort­able with 32 GB or more.

I have a Mac­Book Pro with 64 GB of RAM, so I went with the 26b vari­ant — best quality/​speed trade­off for my set­up. If you’re not sure, start­ing with gemma4:e2b is a quick way to test things out before com­mit­ting to a big­ger down­load.

ollama pull gemma4:26b

This down­loads rough­ly 17 GB, so give it a few min­utes depend­ing on your con­nec­tion. Once the down­load fin­ish­es, you can start chat­ting imme­di­ate­ly:

ollama run gemma4:26b

You’re now talk­ing to a 26-bil­lion para­me­ter mod­el run­ning entire­ly on your hard­ware.

Making It a One-Word Command

I got tired of typ­ing ollama run gemma4:26b every time, so I added a small func­tion to my .zshrc:

gemma() {
    ollama run gemma4:26b "$@"
}

After run­ning source ~/.zshrc, I can just type:

gemma

Tada! I’m instant­ly in a con­ver­sa­tion.

If I want to send a quick one-off prompt with­out enter­ing inter­ac­tive mode:

gemma "Explain the difference between concurrency and parallelism"

Why a Local LLM?

Cloud-host­ed mod­els are great… until they’re not. 😬 Here are a few sit­u­a­tions where hav­ing a local LLM pays off.

Sensitive documents stay sensitive

Some things shouldn’t leave your machine. Say you need to review an NDA or a con­tract. With a local mod­el, you can pipe it straight in with­out wor­ry­ing about where your data ends up. 

gemma "Summarize this contract in a markdown table with columns: Clause, What It Means, and Watch Out For" < nda.txt

By the way, this works well with text doc­u­ments. Com­plex files like PDFs don’t work based on my expe­ri­ence.

What about pro­pri­etary code, inter­nal docs, client data? Using a local LLM means none of it ever hits an exter­nal serv­er. There’s no terms-of-ser­vice fine print to parse, no trust required. It just stays on your lap­top.

No internet? No problem.

Once you’ve pulled a mod­el, it’s on your disk. Air­plane mode, spot­ty cof­fee shop Wi-Fi, work­ing from a cab­in with no sig­nal — doesn’t mat­ter. Your local LLM works the same whether you’re online or off. Where I live, we lose pow­er and inter­net some­times and hav­ing a local LLM comes in handy.

No credits, no quotas, no surprises

Cloud APIs charge per token. Usage caps reset month­ly. Rate lim­its kick in at the worst pos­si­ble time. A local mod­el has none of that. Run as many prompts as you want, as often as you want. It’s your hard­ware and there’s no meter run­ning.

What It’s Like

Response qual­i­ty from the 26b mod­el is gen­uine­ly impres­sive for a local set­up. It han­dles cod­ing ques­tions, writ­ing tasks, and gen­er­al knowl­edge well. Respons­es start stream­ing with­in a few sec­onds on Apple Sil­i­con, and the expe­ri­ence isn’t as fast as you’d get from using a cloud-host­ed mod­el but it’s still impres­sive.

Wrapping Up

The whole set­up takes about five min­utes:

1. brew install ollama
2. brew services start ollama
3. ollama pull gemma4:26b
4. Add the gemma short­cut to your shell con­fig

No accounts to cre­ate, no tokens to man­age, no month­ly bills. Just a local LLM ready when­ev­er you are.

In Part 2, I’ll show how I upgrad­ed this basic set­up with auto-start­ing Olla­ma, live stream­ing out­put, mark­down ren­der­ing, and full con­ver­sa­tion his­to­ry — all with­in the same shell func­tion.