Day 3 - Install Ollama and Run Local LLMs: Llama, GPT-OSS and Gemma
Week 1 · Day 3 · Lecture 15. Lab file: 1_foundations/2_lab2.ipynb. Optional but recommended if your machine can handle it: install Ollama, pull a couple of small open-source models, and call them locally through the same OpenAI-compatible pattern from the last lecture.
This step is optional. Running local models needs a reasonably capable computer. If yours isn't, skip this lecture entirely and carry on with the cloud providers — nothing later in the course depends on having done this.
Installing Ollama
- Go to ollama.com and click Download for your OS (Mac, Linux or Windows).
- Run the installer. It briefly opens a chat window you can close — Ollama keeps running in the background after that.
(Deeper, step-by-step installation instructions are covered in more detail in the companion AI Engineering Core Track, if you need them.)
Choosing a model: size is the whole game
Browse available models on Ollama's models page — but ignore the "applications" section it shows you (that's for using Ollama inside other apps; here, you'll call it directly from your own code). What matters is the size listed next to each model.
Rule of thumb: stick to models around 3GB or smaller, unless you know your machine can handle more. Bigger models eat memory and can make your computer unusable while running. A common trap:
- Llama 3.2 — the usual starting point, small and manageable. Good default choice.
- Llama 3.3 — looks like a natural "upgrade" from 3.2, but it's 70 billion parameters and ~43GB on disk. It will overwhelm most computers. Avoid it (and Llama 4, also too large) unless you specifically know your hardware can cope.
- Gemma (Google) — comes in several sizes; some are reasonable, others still quite large. Check the size before pulling.
- Qwen (Alibaba Cloud) — a strong model family; Qwen 3.6 doesn't offer small variants, but Qwen 3.5 does have genuinely tiny options worth using.
- GPT-OSS — the 20B version is about 14GB, right at the edge of what a well-equipped laptop can handle in memory. This is the largest model demonstrated in this lecture.
Watch for "-cloud" in a model's name. A model ending in "cloud" doesn't run on your computer at all — like Groq, it's served remotely. If your goal is genuinely local inference, avoid anything named that way.
Once you've picked a model, copy its exact name from the Ollama site — you'll need it verbatim for the pull command below.
Managing models from the terminal
Open a terminal (Ctrl/Cmd + backtick in Cursor) and use:
ollama ls— list models already downloaded (empty if you just installed).ollama serve— start the Ollama service if it isn't already running ("address already in use" means it's already running — that's fine).ollama pull <model-name>— download a model (this can also be run from inside a Jupyter cell by prefixing the line with!, which tells Jupyter to run it as a shell command).ollama rm <model-name>— remove a downloaded model to free up disk space.
If your terminal says ollama isn't recognized, open a fresh terminal (or restart Cursor, or your computer) so it picks up the newly installed program.
Verifying Ollama is running
Ollama listens on port 11434. Two quick sanity checks in your browser:
http://localhost:11434— should respond with "Ollama is running".http://localhost:11434/v1/models— lists every model you've pulled, using the same OpenAI-compatible/v1/modelsshape you'd see from a cloud provider.
Calling local models — same pattern as before
Exactly like Anthropic, Google, DeepSeek, Groq and OpenRouter in the previous lecture, Ollama gets its own OpenAI-compatible client — just point base_url at localhost:11434. The call itself is identical: ollama.chat.completions.create(model=..., messages=...).
Three local models were added to the running comparison from the last lecture:
| Local model | Answer (paraphrased) |
|---|---|
| Llama 3.2 (1B version) | Humility, over-optimized, becomes excessive self-importance — prioritizing your own needs over others'. |
| GPT-OSS (20B, run locally) | Perseverance can become stubbornness — relentless pursuit of a goal blinding you to changing circumstances. |
| Gemma 4 (latest, Google's open-source model) | Diligence, over-optimized, becomes workaholism — sidelining self-care and personal interests. |
Worth noting: the smallest model here (Llama 3.2 1B) produced a noticeably more generic, boilerplate-feeling answer than the larger models — a useful, concrete reminder that model size still matters even when the prompt and task are identical. The 20B GPT-OSS model, by contrast, is demanding enough to visibly load a GPU while it runs (and took roughly 20-25 seconds per answer in this lecture's run, similar to Gemma 4).
Nine competitors, one question, run independently
With these three local models added to the six cloud calls from the previous lecture, the running comparison now has nine competitors answering the same LLM-generated question: GPT nano, Claude Sonnet, Gemini, DeepSeek, GPT-OSS-120B (via Groq), Kimi K2.6 (via OpenRouter), Llama 3.2 (local), GPT-OSS 20B (local), and Gemma 4 (local). Each call ran independently and sequentially in this walkthrough — they could equally have been run in parallel, since none of them depends on another's output.
What you should now understand
- Running local models is optional and hardware-dependent — nothing later in the course requires it.
- Model size is the main thing to check before pulling — stick to roughly 3GB or under unless you know your machine can handle more; Llama 3.3/4 and other 70B+ models will overwhelm most computers.
- A model name ending in "cloud" is NOT local — it's served remotely, similar to Groq.
ollama ls,ollama serve,ollama pull, andollama rmcover the basic model lifecycle from the terminal (or a Jupyter cell prefixed with!).- Ollama exposes the same OpenAI-compatible
/v1shape as cloud providers — the only change is pointingbase_urlatlocalhost:11434. - Smaller local models can produce noticeably more generic/boilerplate answers than larger models — a visible, practical illustration of the capability-vs-size trade-off.
Resource links
- Ollama — download and browse models
- Artificial Analysis — compare model intelligence, speed and cost
- Course resources page
- This lecture on Udemy
Next
With nine answers now collected, it's time to pick a winner — using an LLM itself as the judge.