Running a local LLM on your own machine.

What a local LLM runner actually is CORE

Running a model on your own machine sounds complicated. It is not — mostly because someone else has already done the hard parts. The hard parts are: loading billions of internal numbers from disk into memory, wiring them up to run on whatever hardware you have (CPU, Apple Silicon, NVIDIA GPU, AMD GPU), and exposing a simple way for you to send it text and get text back.

A local LLM runner is the class of tool that handles all of that. You install it once; it gives you two things. First, a command that downloads model weights — ten gigabytes or so for a typical 8 B (8-billion) model — and stores them on your disk. Second, an interface — a terminal prompt, a chat window, or an API on localhost — that lets you send prompts to the model and receive completions. The details differ by product, but every runner in this category does the same job: bridge “model weights” and “your prompt.”

Understanding this helps you sit through any tool change. Specific products come and go. The job the tool is doing is durable.

Why you should run one even if you prefer cloud CORE

Even if Lesson 2.1 left you convinced that cloud is your default, you should still run a local model. Three reasons.

Grounding. Running a model locally makes it concrete. You see the file. You watch memory rise when you start inference. You feel the difference between a 7 B and a 13 B model on your machine. Students who skip this step end up with a vague sense of “the model is somewhere” that will bite them later when they are designing real systems.

Directing practice. Local models are smaller, and smaller models are more exacting to direct. A weaker model fails in more visible ways than a frontier cloud model. Directing a local 8 B model teaches you — fast — the difference between a prompt that works and a prompt that does not. That feedback transfers directly to making you better at cloud directing.

The fallback. There will be a day — a power outage, a trip, a provider incident — when cloud is not there. If you can still reach a local model, you can still work.

You do not need to like local models more than cloud models. You do need one installed and working.

Hardware vocabulary you need before the install CORE

Five terms appear all over this lesson and the recipes. Read them once now and the rest of the lesson will be much easier to follow.

• CPU — the main processor in your laptop. Every laptop has one. Runs everything. Not specialized for AI math, so model inference on a CPU alone is slower than on a GPU.
• GPU (graphics processing unit) — a chip designed for the kind of parallel math that AI models love. Originally built for video games, now the default hardware for AI work. A laptop either has a GPU or it does not.
• Discrete GPU — a separate, dedicated GPU chip. Common on gaming laptops (NVIDIA RTX, AMD Radeon RX). Uncommon on thin-and-light or budget laptops. If your laptop runs hot under load and has a fan, it likely has one; if it is silent and thin and cheap, it likely does not.
• Apple Silicon — Apple's M1, M2, M3, and M4 chips, found in Macs from late 2020 onward. Designed so the CPU and GPU share the same pool of RAM, which makes Apple Silicon Macs surprisingly capable for local AI even without a “discrete” GPU in the traditional sense. If you have a Mac from the last four years, you almost certainly have Apple Silicon.
• CPU-only — running the model on your CPU because no compatible GPU is available. Still works for the smaller models in this lesson; just slower (plan for 5–15 tokens per second on a quantized 7–8 B model rather than 30+ on a GPU).

And one more concept that determines whether a model fits on your machine at all:

• Quantization — compressing a model's numerical weights to lower precision (for example, 4-bit instead of 16-bit) so the model fits in less RAM and runs faster. Costs roughly 3–5% of the model's quality and saves roughly 3× the memory. The recipes below default to a q4_K_M quantization, which is the standard “small but good” version.

If you are on a Chromebook, an iPad, or a laptop with less than 8 GB of RAM, the local-model recipes in this lesson will not run reliably. Skip the install, do the activity in paper mode, and treat Lesson 2.4 (cloud accounts) as your real workstation. The rest of the course works on cloud alone.

Decoding the model-name format CORE

You're about to type a command like ollama pull llama3.1:8b-instruct-q4_K_M✓. That string isn't random. Here's what each piece means:

llama3.1 : 8b - instruct - q4_K_M
   |       |       |          |
   |       |       |          └─ Quantization: 4-bit, "K_M" variant
   |       |       └──────────── Purpose: trained to follow instructions
   |       └──────────────────── Size: 8 billion parameters
   └──────────────────────────── Family: Meta's Llama 3.1

• Family (llama3.1, mistral, qwen2, etc.) — who made the model and which generation.
• Size (8b, 7b, 3b, 13b) — billions of parameters. More is more capable; more also requires more RAM.
• Purpose (instruct, chat, or absent) — how the model was fine-tuned. instruct and chat models are trained to answer questions and follow directions. A model with no purpose suffix is a “base” model that just continues text — usually not what you want.
• Quantization (q4_K_M, q5_K_M, q8_0, etc.) — the compression flavor. Higher numbers = bigger file, slightly better quality. q4_K_M is the standard balance.

You don't have to memorize every variant. You do need to know that the model name is exact — typing llama3.1:8b without the rest will fail, and the registry won't auto-correct.

Latency, memory, and tokens per second CORE

Three numbers matter when you run a local model. You do not need to measure them with a stopwatch; your runner prints them.

Latency is how long between when you hit Enter and when the first token appears. For a local 7–8 B model on a modern laptop, this is usually under two seconds on Apple Silicon or a discrete GPU, and three to eight seconds on CPU-only.

Memory use is how much of your RAM the model is consuming while loaded. For a quantized 8 B model, expect roughly 5–7 GB. If your laptop has 8 GB total, you are running tight; close other apps. If it has 16 GB or more, you have room for real multitasking while the model runs.

Tokens per second (tok/s) is how fast the model generates output once it has started. To make the numbers concrete: a fast human types about 5 tokens per second; you read at maybe 4–6 tokens per second; a comfortable AI chat experience feels like 20–40 tok/s. So:

• 30+ tok/s (Apple Silicon or discrete GPU): faster than you can read. Comfortable.
• 8–15 tok/s (CPU-only on a recent laptop): slower than reading speed. Sluggish but usable for short tasks.
• Under 5 tok/s: one word every two seconds. Too slow for real iterative work — try a smaller model or move that workload to cloud.

A local setup is usable for real work if your typical task finishes in under a minute. If a one-paragraph summary takes four minutes, the tool is technically installed but practically unusable. Knowing this in advance prevents wasted effort.