---
title: "A simulated student, an AI editor and one shared log: how I test my courses with agents"
date: 2026-10-17
description: Agent orchestration patterns, and the setup I use to check the courses on ihor.sh. A Gemini agent plays a beginner who knows only school arithmetic, a Claude agent works as instructor and editor under written guardrails, an append-only log connects them, and verification tools turn the teaching rules into checks.
tags: ai, agents, orchestration, education, skills
---

I write three courses on [ihor.sh](https://ihor.sh) with a lot of help from AI agents:
[AI from scratch](https://ihor.sh/ai/) (73 lessons), [Radio from scratch](https://ihor.sh/radio/course/) (22
chapters) and [Electronics](https://ihor.sh/learn/) (16 chapters). Agents are good at writing a lesson about a topic.
They're bad at noticing that the lesson uses a word, a symbol or a line of Python the learner hasn't met yet. That
kind of gap is invisible to whoever wrote the text, because it isn't in the text. It's in the reader's head.

So I stopped asking one agent to write and check its own work. Now several agents do different jobs, each held to
written rules. This article covers the orchestration patterns behind that, and the exact setup I use: a Gemini
agent that plays a student, a Claude agent that plays the course's editor, a shared log that connects them, and
tools that turn the course's teaching rules into checks.

## Orchestration and swarming, in one page

Once you have more than one agent, you have to decide who decides. Most multi-agent setups are a mix of a few
patterns:

| Pattern | How it works | Good for | Watch out for |
|---|---|---|---|
| Orchestrator and workers | One agent splits the task and hands pieces to workers, then merges the results | Big tasks that split cleanly (research, code review across many files) | The orchestrator becomes the bottleneck, and the workers lose context |
| Pipeline | Each agent does one stage and passes its output on | Fixed processes: draft → edit → check | An early mistake flows all the way through |
| Generator and critic | One agent produces, another judges, and they loop until the critic passes the work | Anything with a quality bar | A critic that shares the generator's blind spots just agrees with it |
| Blackboard | Agents never talk directly; they read and write a shared store and react to what appears | Loosely coupled roles working at their own speed | The store needs a format everyone respects |
| Swarm (handoffs) | Peers pass control to whichever agent fits the next step, with no central boss | Conversations that change topic, like a support desk | Hard to see why something happened |

"Swarming" usually means the last two: many agents, little central control, and coordination through shared state
or handoffs instead of a master plan. It's attractive because each agent stays small and focused. It's risky because
nobody holds the whole picture, and agents can confidently agree with each other about something wrong.

Three things matter more than the pattern you pick:

1. **Different eyes.** A critic made from the same model, with the same prompt and context, tends to share the
   author's blind spots. Give the checker a different model, a different role, or less information.
2. **Rules the agents can't argue with.** "Write clearly" can't be checked. "Every answer quotes, word for word,
   the text that justifies it" can be checked by a script.
3. **A record of who did what.** When two agents disagree, or a fix doesn't work, you need to replay what each one
   saw and did.

My course review is a generator–critic loop wired through a blackboard, with a human at the end.

## The setup on ihor.sh

| Role | Who | What it sees | What it does |
|---|---|---|---|
| Student | Gemini, with the [`course-student`](https://github.com/ikruchynskyi/ihor.sh/blob/main/.agents/skills/course-student/SKILL.md) skill | Only what the course has shown so far, in path order, through a real browser | Reads, predicts, answers every question, runs every code box and notebook, logs every struggle, asks when stuck |
| Instructor and editor | Claude, with the [`course-editor`](https://github.com/ikruchynskyi/ihor.sh/blob/main/.claude/skills/course-editor/SKILL.md) skill and the [course guardrail](https://github.com/ikruchynskyi/ihor.sh/blob/main/.claude/skills/course-guardrail/SKILL.md) | The student's log, the lesson in question, the repository | Answers questions from the course only, sorts real gaps from misreadings, fixes the real ones, writes a report per stage |
| Auditor | A script ([`learner.py audit`](https://github.com/ikruchynskyi/ihor.sh/blob/main/ai/review/learner.py)) | The student's log and every page it was shown | Checks that every quote the student cites is really in what it saw |
| Owner | Me | One review packet per stage | Proofreads the changes, makes the product decisions |

Two different model families are deliberate. The student is the role that burns the most tokens (it reads every
lesson and runs every exercise), and that's the one I give to Gemini. More importantly, two different models don't
share the same blind spots: the editor sees the course through someone else's eyes, and the student can't lean on
the editor's way of explaining things.

### The student: a skill that forbids knowing things

The hardest part of simulating a beginner with a large model is that the model isn't a beginner. Ask it about
gradient descent and it knows. So the student skill is mostly a list of things it may not do:

- **You know:** whole numbers, negatives, decimals, fractions, percentages, reading a bar chart, using a browser.
- **You don't know:** algebra beyond "a letter stands for a number", exponents, logarithms, any programming, any
  statistics, any machine-learning word, until a lesson teaches it.
- **Never read the course's source files.** They contain the answers. The student sees lessons only through a tool
  that hides answers until it commits, and through screenshots of the real pages.
- **Every answer quotes, word for word, the passage that justifies it.** If nothing it was shown justifies the
  answer, it still answers, but logs a `guess`.
- **Log every struggle the moment it happens**, with a type from a fixed list (`unknown term or symbol`, `step too
  big`, `unclear wording`, `example missing`, `widget confusing or broken`, …), the exact quote where it went
  wrong, and `quit: true` if a real beginner would close the tab there.

The skill also tells it to use the course's own words: if it's about to write a term the lesson never used, that's
its training data talking, and it should rephrase or log that the course never named it. That rule came from a
real slip, below.

### The editor: a skill plus guardrails

The editor's skill is about attention. It never reads the whole log. It runs `learner.py watch` in the background,
which blocks until the student logs something that needs a person: a question, a struggle, a quit, a guess, an
error, an answer that's wrong twice, a solution it had to reveal, or a finished lesson. Then:

- **A question** gets answered first and fixed later, because the student waits up to five minutes. The answer
  starts with where the course already covers it ("Covered in §2: …") or says plainly "Not covered yet". A "not
  covered" answer is a gap.
- **A struggle** is either real (the lesson uses something untaught, skips a step, words it badly) or a misreading
  (the course clearly covers it). A misreading gets a pointer and no change, unless two struggles hit the same
  spot, which makes it real.
- **A real gap** is fixed by the guardrail's rules, in one commit per fix. Then the editor tells the student
  `fixed: re-read §3`, and the student's next entry says whether the fix worked.
- **Anything bigger than a section** (moving content between lessons, dropping a topic) isn't done live. It goes
  into the stage packet as a proposal for me.

The guardrail is the part every agent loads, whatever it's doing in a course. It's one file with three rules:

1. **No gaps:** use only what earlier lessons taught, and look each prerequisite up instead of recalling it.
2. **See it before naming it:** a question and a picture or widget first, then words, then formulas, then code.
3. **Fit it in:** search the course and read every hit before adding anything. Extend an existing lesson rather
   than repeat one.

Before any change, the agent writes a short fit plan: what's already covered (with file and line), the exact gap,
where the fix goes and why there, and which earlier lesson teaches each term it needs. The guardrail ends with a
table of mistakes agents have already made in this repo: a roadmap that re-planned five topics that were already
taught, prerequisites cited from memory that turned out to be in a different lesson, a widget whose demo broke the
very rule the lesson taught, an `import torch` in a code box that runs in the browser where PyTorch can't. Every new
mistake adds a row. That table is the team's memory, and it's cheaper than every agent rediscovering it.

### The shared log: a blackboard made of files

The two agents never talk to each other directly. They share a folder, `ai/review/run/`:

| File | Written by | Read by |
|---|---|---|
| `log.jsonl` | the student (every view, answer, code run, note, question) | the editor's `watch`, the audit, the report |
| `replies.jsonl` | the editor (answers and "fixed, re-read" notes) | the student, on its next command |
| `seen.txt` | the tool, every time it shows the student a lesson | the citation audit |
| `state.json`, `editor.json` | the tool | where each agent left off |

A struggle entry has this shape (this one is the example from the student's skill; the tool adds the time and the lesson):

```json
{"kind": "struggle", "type": "unknown term or symbol", "section": "3", "quote": "the gradient ∇L points uphill", "expected": "what ∇ means",
 "happened": "∇ was never explained", "quit": false}
```

Append-only JSON lines beat a framework here for boring reasons. Either agent can stop and the next session picks
up from the files. I can read exactly what the student saw and when. Any agent, from any vendor, can take part
by running a command-line tool. And `watch` costs nothing while the student is busy: it polls the file and prints
only the entries that need attention, so the editor spends tokens on decisions, not on reading.

The same tool, [`learner.py`](https://github.com/ikruchynskyi/ihor.sh/blob/main/ai/review/learner.py), is the
student's whole interface to the course (`show`, `answer`, `run`, `note`, `ask`, `nb`, `hint`, `done`) and the
editor's (`watch`, `reply`, `audit`, `report`). A Playwright helper,
[`browse.py`](https://github.com/ikruchynskyi/ihor.sh/blob/main/ai/review/browse.py), lets the student look at
the real pages, drag sliders, press Run, and check a page at phone width. It logs JavaScript errors and sideways
scrolling automatically.

### Verification tools: rules turned into checks

Agents follow rules better when a program checks them. Each course has its own:

- **The ledger linter** ([`lint.py`](https://github.com/ikruchynskyi/ihor.sh/blob/main/ai/notebooks/lint.py)) walks
  the AI course in order and keeps a ledger of what a learner has been taught (Python syntax, math notation) and
  shown (library calls in code they've run). Every notebook exercise is checked against it: an exercise that needs
  a `for` loop before any lesson taught one fails. `lint.py --ledger s1` prints what a learner knows at that point,
  so an agent can look prerequisites up instead of guessing.
- **The depth check** (`lint.py --depth`) requires each lesson's core idea in layers: a one-sentence gist, an
  optional "Why …?" box, a "Where it breaks" box with numbers, and a "why" question. All 73 lessons pass today.
- **The radio course's ledger** ([`course-ledger.ts`](https://github.com/ikruchynskyi/ihor.sh/blob/main/radio/scripts/course-ledger.ts))
  maps each term to the chapter that teaches it. A term used earlier fails the course check, unless that paragraph points
  to the later chapter. It also requires a gist in every chapter, and a "Before you start" box from chapter 2 on.
- **The citation audit** checks every quote in the student's log against everything it was shown. A quote that
  isn't there means the student used outside knowledge.
- **The self-test** parses all 73 lessons and checks that no explanation leaks into the student's view before it
  answers, and that every answer key is valid.
- **The site check** (`npm run check`) fetches every page and API.

None of these replace the student. A linter can tell that `∇` appears before the lesson that defines it. It can't
tell that a step is too big, or that an example is missing. The student finds those, and the tools make sure a fix
for one thing doesn't break another.

## What actually happened

The first stage, four lessons, ran on October 15–16. Here's the packet's summary:

| Lesson | Questions | Right first try | Hints | Struggles | Errors |
|---|---|---|---|---|---|
| s1 What is ML | 4 | 4/4 | 0 | 1 | 0 |
| s2 Just enough code | 7 | 7/7 | 0 | 0 | 0 |
| s3 Math warm-up | 13 | 10/13 | 0 | 1 | 0 |
| s4 Your first shared model | 5 | 5/5 | 2 | 2 | 1 |

The real findings were the kind I'd never have caught by reading my own lessons:

- **s1's notebook** had a "real model in five minutes" part that downloads about 800 MB and trains for about 25
  minutes on a laptop CPU, and nothing said it was optional. Fixed: marked optional, with its real cost and how to
  turn on a GPU.
- **A chart on s1** had its axis labels on the wrong ticks and text cut off at 390 px wide. The student saw it in a
  phone-width screenshot. Fixed in the chart library, so every chart got it.
- **s4's photo download** came from a server running at 160–370 KB/s, so the download alone takes 35–80 minutes.
  The editor proposed a faster mirror; I chose a warning in the notebook instead. That's the kind of decision that
  stays with the owner.

### The student cheated, and the tools caught it

The most useful finding was about the student itself. In s2 it completed nine notebook exercises in under two
minutes without a hint or a single struggle, ran all nine code boxes in one burst, reported 25 minutes for what
took 4.5, and explained something using the word "broadcasting", which the lesson never uses. It wasn't learning
from the course. It was answering as the expert it is.

The fix wasn't a sterner prompt. It was making the rules checkable:

- `done` now refuses to close a lesson until every notebook exercise has a note quoting where the course showed how
  to do it, and those quotes are audited like answers.
- If the student names something as "hardest" without having logged a struggle for it, the tool logs one.
- Real minutes are recorded next to the student's estimate.
- The skill gained a "stay in character" section: use the course's words, work section by section, run each code
  box when you reach it.

s2 is marked provisional and gets rerun under the new rules. Even after the fix, s3's answers came about five
seconds apart, still expert speed, and that's written down as an open item instead of being declared solved.

### The orchestration had bugs too

Messages from the editor only reached the student when it opened the next lesson. So a correction about s2 sat
unread for all of s3. Now replies are shown on every command the student runs. A notebook that needs a GPU timed
out after 30 minutes on the student's machine, so GPU notebooks are no longer run there; the student still fills
in and cites every exercise. Neither was a course problem, and the editor's skill says so: infrastructure gets the
smallest fix that unblocks the student, and the effort goes into content and structure.

## The same idea, without the student

The full student loop is expensive, so the other courses get the cheaper half first: the guardrail, a ledger and a
cold read by an editor agent. The radio course got a strict ledger check, a gist and a "Before you start" box in
every chapter, and "Why" and "Where it breaks" boxes in all 22 chapters, each with numbers from the code that
really runs the receiver.

Today I did the same pass on the electronics course, which has no ledger yet, so the agent read all 16 chapters as
a learner would. It found that four later chapters (oscillators, the linear regulator, the phototransistor and the
audio amplifier classes) leaned on a transistor running half on as an amplifier. Chapter 4 had only taught the
transistor as a switch, and had told the learner never to run one half on. Chapters 14 and 16 pointed back to
chapter 3 for how a moving magnet makes a voltage, which chapter 3 never explained. One chapter cited "chapter 1's
vectors", which don't exist. Each fix went where the guardrail says: in the earliest chapter that should teach the
idea, so every later chapter benefits. The new amplifier section has a widget solved by the course's own circuit
simulator, and every number on the page came from running it. A ledger file for electronics is the next step, so a
script catches the next gap instead of a reader.

## What I'd tell anyone building something similar

1. **Split the roles by what each one is allowed to know.** The student's value comes from its ignorance, which
   has to be enforced: hide the sources, hide the answers, and demand a quote for every step.
2. **Use different models for the maker and the checker.** Same-model review mostly confirms itself.
3. **Coordinate through files, not chat.** An append-only log is a blackboard any agent can join. It survives
   restarts, and it's an audit trail for free.
4. **Wake the expensive agent only when it's needed.** A blocking `watch` that filters for questions and struggles
   keeps the editor's context for decisions.
5. **Turn every rule into a check.** "Stay in character" failed as a sentence. It worked as a tool that won't
   close a lesson without cited exercises.
6. **Write the mistakes down where every agent will read them.** The guardrail's table of past mistakes keeps them
   from being made twice.
7. **Keep a human at the decision points.** Agents fix gaps inside a lesson. Moving lessons, dropping topics and
   choosing between a mirror and a warning stay with the owner, through a short packet per stage.

The review continues lesson by lesson, and it ends with a fresh student, with no notes from earlier runs, going
from the first lesson to the capstone. The bar for "done" is written in the plan: no "I'd quit here" left, no
unresolved citation failures, at least 80% of checkpoint questions right on the first try, and every fix approved
by me. The courses are free at [ihor.sh](https://ihor.sh), and the skills and tools linked above are in the
site's public repository if you want to borrow the setup.