What “reasoning” actually means for a language model CORE
The word reasoning is misleading when applied to a language model. The model is not thinking the way you think. It is not weighing evidence, consulting a memory of facts, or checking its conclusions against the world. It is doing one thing, over and over, very fast: predicting the next small chunk of text, given everything in front of it.
That sounds almost insultingly simple. It is not. With enough training data, a next-chunk prediction machine can produce outputs that are indistinguishable from reasoning most of the time — writing a proof, debugging code, summarizing an argument, composing a persuasive letter. The machine is not secretly doing something else on hard problems. It is doing the same prediction, at higher skill, in a denser region of the patterns it learned.
This lesson gives you just enough mechanism to predict when that prediction will succeed and when it will fail. Not enough to build a model. Not enough to train one. Just enough to direct one without being surprised.
What the model is really doing CORE
Strip away the marketing language and the model is doing one thing, repeated very fast: predicting the next chunk of text, given the text it has seen so far. It is not consulting a database of facts. It is not searching the internet. It is producing the most likely next chunk based on patterns it learned during training.
That sounds insultingly simple, and in a way it is. But with enough training data and enough patterns, a next-chunk predictor produces output that is indistinguishable from reasoning most of the time — writing a proof, summarizing an argument, drafting a persuasive letter. The machine is not secretly doing something else on hard problems. It is doing the same prediction at higher skill, in a denser region of the patterns it learned.
Two things follow, and both shape how you direct an agent. First, the model is reasoning over patterns of text, not over facts — which means fluent output is not the same as accurate output. Second, the model has no internal alarm that says “I'm guessing here.” It produces fluent text either way. Those two facts set up the rest of the lesson.
Why confidence ≠ correctness CORE
The model picks the next chunk based on probability. But a high-probability sequence is not the same as a true sequence. The model is not matching your prompt against a database of facts. It is completing a pattern of text.
If the sequence “the Battle of Trafalgar was fought in ____” was followed in training data, overwhelmingly, by “1805,” the model will confidently produce “1805.” Good — that one happens to be right.
If the sequence “the neurologist Dr. Amelia Reeves published her 2021 study on ____” was never in training data — because Dr. Amelia Reeves is made up, or the study does not exist — the model still produces something. Whatever continuation the patterns favor next. It will produce it with the same fluent, confident tone it used for Trafalgar. It might invent a plausible-sounding journal. It might invent co-authors. It will not stop and say “I do not have a pattern here.” It has no mechanism to say that.
This is hallucination. It is not lying. The model has no concept of lying or truth. It has a next-chunk predictor doing exactly what it was trained to do: complete the pattern.
Your job as a director is to notice when you are in a low-data region — obscure topics, recent events past training, specific numbers, specific citations, specific names — and stop treating tone as evidence of accuracy. Fluency is cheap. Groundedness is expensive, and you usually have to arrange it yourself.
Why “think step by step” helps CORE
Here is a surprising consequence of the mechanism. Ask a model, cold, “What is 47 × 89?” and it often gets it wrong — because the exact pattern “47 × 89 =” is uncommon in training data, and the model is essentially guessing the chunk that usually comes next.
Now ask: “Think through this step by step. What is 47 × 89?” If the model produces intermediate steps — “47 × 89 = 47 × 90 − 47. 47 × 90 = 4,230. 4,230 − 47 = 4,183.” — accuracy jumps significantly.
Why? Not because the model is suddenly “thinking harder.” Because each intermediate step is itself a common pattern the model has seen many times. The pattern “47 × 90 =” is much denser in training data than “47 × 89 =”. Subtracting 47 from a number is a pattern it has seen thousands of times. You are routing the problem through terrain the model has walked before, one easy step at a time, instead of asking it to leap the whole distance.
This is the mechanism behind chain-of-thought prompting, and it is why modern “reasoning models” — the ones advertised as taking time to think before answering — perform better on hard problems. They do this step-by-step work internally before producing the final answer. The insight is the same: break a hard prediction into a sequence of easy ones.
What this means for how you direct CORE
Three directing moves follow directly from the mechanism, and you will use all three for the rest of the course.
Give the model context. The more of the right input is visible in the model's window, the denser the pattern region it is predicting in, the better the output. Vague input produces low-density predictions, which produce confident nonsense. If you want a good rewrite of your essay, paste the essay — and the rubric.
For anything that must be exact, do not trust what the model produces from memory — use a tool. Calculations, citations, dates, names, current events: these are the regions where confidence and correctness drift apart. Give the agent a calculator, a search, a database, a file lookup. Direct it to use that tool rather than answer from memory. Module 2 sets up those tools; Modules 3 through 6 teach you to wield them.
On hard tasks, ask for intermediate steps. Not because it is magic. Because each step is an easier prediction than the whole leap, and the model's own previous words become useful context for the next ones. “Think step by step” is not a superstition; it is a mechanism-level move.
In Lesson 1.3, we will zoom back out and watch these predictions happen inside the agent loop — which is where all of this starts to do real work.
Try it CORE
Catch a hallucination in the wild
Goal. See, in your own AI of choice, where confidence and correctness pull apart — and notice the tone the AI uses when it's making something up.
Instructions.
- Open any AI chat tool you have access to (ChatGPT, Claude, Gemini, the chat box on your phone, the assistant in your favorite app). Pick a topic you care about — a hobby, a sport, a favorite author, a video game, a niche industry, anything moderately specific.
- Ask the AI for five book recommendations on that topic, with the author and publisher of each.
- Then go look each one up in an outside source — a library catalog, Google Books, your library's website, or a major bookstore. Don't ask the AI to verify itself; check elsewhere.
- Label each one: real, partly real (real author but the book doesn't exist, or vice versa), or fully invented.
Observe. What did the AI's tone sound like for the made-up ones vs. the real ones? Could you have told the difference from the answer alone, without checking? That feeling — fluent and wrong sounds exactly like fluent and right — is the most important takeaway in this lesson.
If everything came back real: congratulations, you picked a topic well represented in training data. Try again with something more obscure — a niche local business, a specific technical term in a hobby, a less-famous person, a recent event. The point is to find the line where the AI starts making things up.
Done with the hands-on?
When the recipe steps and any activity above are complete, mark this stage to unlock the assessment, reflection, and project checkpoint.
Key concepts CORE
- A language model generates text by predicting one chunk of text at a time, based on what's most likely given the text so far.
- The model reasons over patterns of text, not over a list of facts. Fluent output is not the same as accurate output.
- Hallucination is not the model lying. It is the model doing its normal job (continue the pattern) in a region of low or conflicting training data. Fluent tone is produced either way.
- Chain-of-thought helps because breaking a hard prediction into several easy predictions routes the problem through terrain the model has seen before.
- Directing moves that follow: give rich context, use external tools for anything that must be exact, and ask for intermediate steps on hard tasks.
Quick check CORE
Four questions. Pick the best answer, then reveal the explanation — the why matters more than the letter.
Show explanation
Answer: B. The model is a next-chunk predictor — on each step, it produces the most likely continuation of the text in front of it. A and C describe tools an agent might be given, but the bare model does neither. D is tempting but wrong: the model was not trained to memorize passages but to learn patterns. Sometimes patterns produce near-memorization; usually they produce plausible completion.
Show explanation
Answer: C. Hallucination is the normal mechanism behaving normally in a region of low or conflicting training data. The output is fluent because fluency is cheap — the model always produces a probable-sounding continuation. Nothing in the mechanism alerts the model that it is guessing. A and B anthropomorphize. D confuses hallucination with a different problem (context window limits, covered in Lesson 1.4).
Show explanation
Answer: B. This is the mechanism-level answer and the whole point of the block on chain-of-thought. A is nonsense — the model does not warm up between prompts. C is wrong unless you have given it a calculator tool (Module 2). D confuses the prompt with an architectural switch; some newer “reasoning models” do have internal chain-of-thought built in, but the mechanism is the same.
Show explanation
Answer: C. Citations are the paradigm case of “confidence ≠ correctness” — a model will produce author names, journal titles, years, and DOIs that sound exactly right and do not exist. The only reliable safeguard is grounding: a real search tool plus a directive to cite only from retrieved sources. A is a polite phrase; the model has no mechanism to distinguish what it “knows” from what it is pattern-completing. B is precisely the failure mode. D is superstition.
Reflection prompt
When did tone last mislead you?
In 5–7 sentences, answer: Think of one time in the last month that an AI tool gave you a confidently-stated answer that turned out to be wrong or made up. Describe the situation briefly. Then, using the mechanism from this lesson, explain what was probably going on — what was the model doing, what pattern region was it in, and where did tone mislead you? Finally, describe one concrete thing you will do differently next time in that kind of situation.
If you cannot think of an example, you have either been very lucky or you have not checked. Try to check one factual claim from a recent AI conversation and see what happens.
Project checkpoint
Name where confidence would bite you.
Open the note you started in Lesson 1.1 — the one-sentence candidate task for an agent to help you with. Underneath it, write one more sentence:
“The part of this task where confidence ≠ correctness could bite me is ____.”
Be specific. Is it a date you cannot afford to get wrong? A person's name? A price? A citation? A code change that has to compile? Whatever it is, name it in one short phrase. You are identifying, in advance, the place where you will want a tool — not parameters — doing the work. You will use this in Lesson 1.5 when we design your first loop.
That's the end of the free preview.
The full course picks up at Lesson 1.3 and runs through Module 10. Capstone, credit documentation for parents, and a year of Recipe Book updates included.
30-day no-questions refund. ← Re-read Lesson 1.1