The tiny factory has a hole in it, and the median spec found it. Station 1 refused SPEC-ALGO-103 partly because it asked for a body called median_sorted, and the pattern's catalog holds only prefix_sum and clamp_each. Fix the other five errors and the spec is still refused. Example A can build only what someone already wrote as a template.
For two requests a week, the fix is a person who writes the new body, reviews it and adds it to the catalog. For two thousand requests, each needing its own logic, that person is the bottleneck, and this is where a language model looks attractive: it reads "return the median of a list" and writes the function.
So a model, or a stand-in for one, takes the station. Its first answer arrives with a note: "done, all tests pass". An independent check runs the answer on 30 cases. Twenty-one pass. The note was wrong, and it was written with exactly the same confidence as the note on the second, correct answer.
The learner's question: what does a model actually contribute to a factory, what does it take away, and which stations should get one while the rest stay plain functions, generators, compilers and tests?
What a model adds#
Plain English. A model can read a description written for people and produce new code or text from it. No template can do that. It can also be wrong while sounding sure, and it may give a different answer when asked the same thing twice.
Beginner definition: a model belongs at a station when the station's job is to interpret a business description that varies from unit to unit and write logic nobody could prepare in advance; everywhere else, code does the job more reliably.
A semantic station is a station whose work is interpreting variable logic written in prose (a requirement, a problem statement, a business rule) and turning it into a new artifact. Writing the body of median from "return the median of a list" is semantic. The non-AI analogy is a contractor hired to write a custom module from a brief: you cannot predict their exact code, only check it against the brief. The operational consequence is that the station's output cannot be predicted, so it must be checked by something else. The trade-off is that you buy coverage of requests no catalog anticipated and pay in checking.
A deterministic station is one that gives the same output every time for the same input: validation, rendering from templates, compiling, running tests, writing a manifest. Every station of Example A is deterministic, and that is why its second run was byte-identical.
Before AI. Code generators, from a Maven archetype to a protocol-buffer compiler, never interpreted prose. They mapped a structured input to output by fixed rules. What was new in a model is the ability to start from a description written for people; what is old is everything around it: the spec, the tests, the build, the record.
What drops, and what comes with it#
Four properties change when a model takes a station. Each has a name worth knowing.
Non-determinism means the same input can produce different outputs on different runs. Ask the same model the same question twice and the code may differ in names, structure, or correctness. The analogy is asking two equally skilled engineers for the same function. The consequence for this site's factory is concrete: the byte-identical rerun of chapter 4 no longer holds at that station. The manifest still records what was produced and tested, by hash, but "run it again and get the same bytes" becomes "keep the bytes you accepted, with their evidence". The examples on this site cannot show non-determinism, because their stand-in is scripted, and they say so.
The context window is the amount of text a model can take into account at once: the instructions, the spec, any files it reads, and the conversation so far. The analogy is a desk of fixed size: whatever does not fit is not in view. The consequence is that a station worker given a long task gradually has less room for the task itself, and anything that falls out of view is invisible to it. The usual design response is to keep each task small and start a fresh worker for each one. Long-running agent execution is the home of context growth, checkpoints and compaction; Context Engineering covers the working set itself.
A hallucination, in this site's narrow sense, is output that is wrong and presented as right: a function that fails cases it claims to pass, a citation that does not exist, an expected value nobody computed. What makes it dangerous in a factory is not the error, which every worker makes, but the confidence: nothing in the output marks it as doubtful.
A self-report is a worker's own statement about its work: "done", "all tests pass", "verified". It costs nothing to produce and it is the first thing a busy line is tempted to trust. This site's third refrain applies here word for word: a self-report of "done" is not evidence; a test the producer did not write is.
Diagram: Example B · one semantic station between deterministic ones
Left to right. The oracle station is code: it builds 30 cases from fixtures and seeded random inputs, each checked against a brute-force scan. The solution station is the only semantic one; a scripted stand-in plays the model and attaches the self-report 'done, all tests pass' to every answer. The self-report is stored and ignored. The oracle's evidence decides: attempt 1 passes 21 of 30 and goes back as rework with the failing cases; attempt 2 passes 30 of 30. Only the accepted solution reaches the visualization station, which is code again and checks its own trace invariants.
- Deterministic
- S1 build oracle: Code, written ahead of time; a fixture with a wrong expected value is refused.
- Semantic
- S2 generate solution: Where a model would write new logic from the statement. Here a fixed script plays it.
- self-report: Stored with each attempt; never part of the acceptance condition.
- Gate
- oracle evaluates: Runs the candidate on every case; the verdict comes from this evidence only.
- rework: At most 3 attempts, budget checked before each call, then escalation.
- Deterministic
- S3 trace visualization: Runs only after acceptance; every trace must satisfy the invariants.
The diagram is Example B's line. Two deterministic stations surround one semantic station. The oracle is built first, by code, from fixtures and seeded random cases. The scripted stand-in writes a solution and attaches its self-report. The self-report is recorded and ignored; the oracle's 30 cases decide. A rejected attempt goes back with the failing cases as rework. Only an accepted solution reaches the visualization station, which is code again and checks its own invariants.
The stand-in at work#
Example B (examples/b-algorithm-factory/) builds one toy unit: lower_bound(values, target), which returns the first index whose value is at least the target, or the length of the list when there is none. For values = [1, 2, 2, 2, 3] and target = 2 the answer is 1. The run is factory run FR-0002, building spec SPEC-ALGO-001 with pattern PAT-ALGO-UNIT 1.3.0.
A stand-in is the scripted replacement for a model in this site's examples: a class that returns canned outputs in a fixed order, with no model call, no network and no key. It lets every page show the factory's controls working on a known failure. The agent here is a scripted stand-in, so these results show how the factory mechanics work, not how good any model is. The class, quoted verbatim from generator.py:
class ScriptedGenerator:
"""Plays the model. `script` lists which canned source each attempt returns."""
def __init__(self, script=("buggy", "correct", "correct"), cost_per_call=400):
self.script = list(script)
self.cost_per_call = cost_per_call # illustrative "token units", not real tokens or money
self.calls = 0
self.feedback_seen = []
def generate(self, spec, attempt, feedback=None):
"""Return {"source", "self_report", "cost_units"} for this attempt (1-based)."""
self.calls += 1
self.feedback_seen.append(feedback)
kind = self.script[min(attempt - 1, len(self.script) - 1)]
return {"source": SOURCES[kind], "self_report": SELF_REPORT, "cost_units": self.cost_per_call}
SELF_REPORT is the constant "done, all tests pass", returned with every answer whether it is true or not. The first canned source has an off-by-one bug: it starts with hi = len(values) - 1, so it can never return len(values).
The station that uses it is do_solution in factory.py, quoted verbatim:
def do_solution(cp, out, spec, gen, emit, verbose):
rec = cp["items"]["solution"]
cases = load_json(out / cp["items"]["oracle"]["artifact"])["cases"]
max_attempts = spec["limits"]["max_attempts"]
feedback = None
while len(rec["attempts"]) < max_attempts:
attempt = len(rec["attempts"]) + 1
if cp["cost_units_spent"] + gen.cost_per_call > cp["budget_token_units"]:
rec["status"] = "budget_exhausted"
emit("[stop] %s solution: attempt %d would cost %d units; spent %d of %d: budget_exhausted"
% (rec["station_id"], attempt, gen.cost_per_call, cp["cost_units_spent"], cp["budget_token_units"]))
return
answer = gen.generate(spec, attempt, feedback)
cp["cost_units_spent"] += answer["cost_units"]
source = answer["source"]
rel = "artifacts/solution.attempt-%d.py" % attempt
write_atomic(out / rel, source)
evidence = oracle.evaluate(load_module(source)["lower_bound"], cases)
verdict = "accepted" if evidence["failed"] == 0 else "rejected"
rec["attempts"].append({"attempt": attempt, "self_report": answer["self_report"], "verdict": verdict,
"source": rel, "source_sha256": sha256_text(source),
"evidence": {k: evidence[k] for k in ("total", "passed", "failed", "failures")}})
emit("[try ] %s solution attempt %d: stand-in says \"%s\"; oracle: %d/%d passed -> %s"
% (rec["station_id"], attempt, answer["self_report"], evidence["passed"], evidence["total"], verdict))
if verbose and evidence["failed"]:
emit(" failed cases: %s" % ", ".join(evidence["failed_names"]))
for f in evidence["failures"]:
emit(" %s values=%s target=%d expected=%d got=%s"
% (f["name"], f["values"], f["target"], f["expected"], f["got"]))
if verdict == "accepted":
save_artifact(out, "artifacts/solution.py", source, rec)
rec["status"] = "done"
return
feedback = evidence["failures"]
rec["status"] = "escalated"
emit("[stop] %s solution: %d attempts used, none accepted: escalated to a person"
% (rec["station_id"], max_attempts))
Read the order of operations. The budget is checked before each call, not after. Every attempt's source is written to disk and hashed, accepted or not. The self-report is stored in the attempt record, and the verdict is computed from the oracle's evidence alone; the self-report never appears in the condition. A rejected attempt sets feedback to the failing cases, which is rework. After max_attempts (3 in the spec) the station escalates to a person instead of looping. The generated code runs in-process through exec to keep the example short; the README says a real factory runs it in a sandboxed subprocess with resource limits.
Output of /opt/homebrew/bin/python3 run.py --verbose in a scratch copy on 2026-11-20, the first part:
start FR-0002 spec SPEC-ALGO-001 pattern PAT-ALGO-UNIT@1.3.0 spec_hash 44d76f9399ab budget 2000 units
order: oracle -> solution -> visualization
[run ] S1-build-oracle oracle: 30 cases (10 fixtures incl. edge cases, 20 seeded random), sha256 45d4895d56b7
[try ] S2-generate-solution solution attempt 1: stand-in says "done, all tests pass"; oracle: 21/30 passed -> rejected
failed cases: all-smaller, single-above, random-06, random-09, random-10, random-14, random-17, random-18, random-20
all-smaller values=[1, 2, 3] target=9 expected=3 got=2
single-above values=[4] target=5 expected=1 got=0
random-06 values=[-13, -11, -1, 14, 15, 16] target=18 expected=6 got=5
[try ] S2-generate-solution solution attempt 2: stand-in says "done, all tests pass"; oracle: 30/30 passed -> accepted
[run ] S3-trace-visualization visualization: 10 traces, 28 states, invariants hold, sha256 2bf5ff26c1a1
end FR-0002 status=succeeded cost=800 units generator_calls=2
Every failed case has a target larger than every value, which is exactly where the off-by-one bites. Two self-reports, identical in wording; one true, one false. The oracle, built by oracle.py : brute_force as a linear scan that is too slow to ship and easy to trust, is what told them apart. The 800 units are illustrative: 400 per generator call, invented for the demo, not real tokens or money.
How good is a 30-case oracle? For this page a script ran three implementations against it in a scratch copy of Example B: the stand-in's two sources, and a third variant written for the verify lab (not part of Example B) that returns any matching index instead of the first.
buggy self-report: 'done, all tests pass' oracle: 21/30 passed
failed: all-smaller, single-above, random-06, random-09, random-10, random-14, random-17, random-18, random-20
all-smaller values=[1, 2, 3] target=9 expected=3 got=2
single-above values=[4] target=5 expected=1 got=0
correct self-report: 'done, all tests pass' oracle: 30/30 passed
failed: none
dupes self-report: 'done, all tests pass' oracle: 28/30 passed
failed: duplicates, negatives
duplicates values=[1, 2, 2, 2, 3] target=2 expected=1 got=2
negatives values=[-5, -3, -3, 0, 2] target=-3 expected=1 got=2
The duplicates bug is caught by only two cases, both hand-written fixtures; none of the 20 random cases happened to contain a duplicated target. That is a fact about this oracle, not about oracles in general, and it is the reason Evidence of correctness adds properties and differential tests on top of examples.
Which stations stay plain code#
Putting a model at one station does not change the others. In Example B two of three stations are deterministic code, and in Example D three of six are. The table below sorts the kinds of station work this site has met, with the recommendation for each.
| Station work | Who does it | Why | Example |
|---|---|---|---|
| Validate the spec | code | Rules are known; a model would add doubt to the gate itself | Example A validate_spec |
| Render from templates | code (a generator) | The logic is already written | Example A render_artifact |
| Compile, type-check, lint | code (a compiler, a static analyzer) | Exact, fast, and the same every time | Example A py_compile |
| Run the tests and the oracle | code | The decision must not come from the producer | Example B oracle.evaluate |
| Write new logic from prose | a model or a person | Nobody could write it in advance | Example B S2-generate-solution (stand-in) |
| Build the oracle | code where possible; otherwise a different producer, reviewed | A check written by the same producer shares its blind spots | Example B brute_force |
| Produce a trace or visualization | code over the accepted solution | A trace of execution should come from execution | Example B S3-trace-visualization |
| Write the manifest and the run record | code | It is the evidence; it must be exact | Example A station 4, Example D station_attempts |
This site's recommendation: a model belongs at a station only where the work is semantic interpretation of variable business logic, and every station that judges, records or packages stays code. The checks that decide acceptance must be computational wherever the rule can be written as code; chapter 1's folded block on check kinds gives the vocabulary.
One agent, specialized agents, or functions?#
This page owns one of the site's decisions. Once a model is allowed in, the next temptation is to let it do more: the whole unit, or a team of agents with roles.
The question. For one unit of work, which parts go to one agent, which to separate specialized agents, and which to deterministic functions?
Diagram: One agent, specialized agents, or functions · asked once per part of a unit
Top to bottom, for each part of the unit. Can the part be written as code ahead of time? Then it is a function, and the question ends. If it is semantic, can its output be checked by code the agent cannot edit? If not, a person or a separately owned evaluation must stand behind it before any agent takes it. If it can be checked, does it need instructions, tools, permissions or context different from the other semantic parts? Then it gets its own specialized agent. Otherwise one agent can do the semantic parts in sequence, with checks between them.
- Questions
- writable in advance?: If someone can write the logic before the unit arrives, a model adds only doubt.
- checkable by code the agent cannot edit?: Acceptance must not come from the producer.
- different brief, tools, permissions, context?: Different needs justify a separate, narrowly scoped worker.
- Answer
- function: Exact, rerunnable, no model cost.
- person or separate evaluation: No computational check exists yet; build one or put a person at the gate.
- specialized agent: One worker per semantic station, each with a narrow scope.
- one agent: Fewer handoffs; never let it grade its own work.
The diagram reads top to bottom, once for each part of the unit. If the part can be written as code ahead of time, it is a function, and that ends the question. If it is semantic, ask whether its output can be checked by code the agent cannot edit; if not, it needs a person or a separately owned evaluation before any agent takes it. If it can be checked, ask whether it needs different instructions, tools, permissions or context from the other semantic parts; if so, give it its own specialized agent with its own brief; if not, one agent can do the semantic parts in sequence, with checks between them.
| Option | Choose it when | It buys | It risks |
|---|---|---|---|
| Deterministic functions | The logic can be written in advance (validate, render, compile, test, package) | Exactness, reruns, no model cost | Nothing new: a catalog gap stays a gap |
| One agent for the semantic parts | Few semantic parts, similar context, checks between them | Less coordination, one brief | Context grows across parts; the agent may grade its own work if you let it |
| Specialized agents per semantic station | Parts need different briefs, tools, permissions or context | Small focused contexts; narrow permissions per station | Handoffs, more briefs to maintain, more places to lose information |
The criteria, in order: can the part be written as code ahead of time; can its output be checked by code the agent cannot edit; do the parts need different instructions, tools, permissions or context; and how large is each part's context. Underneath all four sits the same rule: the producer never writes the check that accepts it.
The counterexample, from WatchAlgo. Observed in the repository (docs/EVIDENCE.md, matrix row 9): in September the pipeline used two kinds of worker, solution workers and tracer workers, with different prompts and briefs and the same model (claude -p --model sonnet), and acceptance was deterministic code, not an agent. "Specialized agents" there meant specialization by instructions, not different models and not a team that negotiates. The repository also holds a separate engine with roles named planner, worker, gate, reviewer and consensus per stage (CosmicKeys backend, engine.py, commit 93e2a0b1), and no evidence that it produced published content. The more elaborate design is not the one that filled the corpus.
The failure modes. One agent for the whole unit, writing the code and its tests and reporting success: the self-report problem in its purest form. Specialized agents where functions would do: a "validator agent" that judges output a function could check exactly, adding non-determinism to the gate. Functions where semantic work is needed: the median spec, refused forever. And specialized agents with handoffs that lose information, where each agent sees a summary of the previous one's work rather than the artifact.
A coding agent as a station worker#
The site's second, smaller example is a coding agent running headless in a repository: it receives one work item and a scoped checkout, writes code, the factory's own tests decide whether the artifact advances, and a person with merge rights reviews. This site uses two products in that seat as examples. The facts below come from their documentation as read on 2026-11-20; nothing here was run against either product.
Product (Claude Code, as documented on 2026-11-20): adding -p to a claude command runs it non-interactively, and the documentation presents this as using the Agent SDK through the CLI. --output-format json returns the result with a session id and metadata, and --json-schema puts output that conforms to a schema in structured_output. --allowedTools pre-approves tools using permission-rule syntax such as Bash(git diff *). From version 2.1.302, a claude -p run with no permission mode set starts in the default mode and denies calls that need approval. --bare skips auto-discovery of hooks, skills, subagents, plugins, MCP servers, auto memory and CLAUDE.md; the documentation recommends it for scripted calls and says it will become the default for -p in a future release.
Product (OpenAI Codex, as documented on 2026-11-20): codex exec "<task>" runs without the interactive interface, streams progress to stderr and prints only the final message to stdout. It defaults to a read-only sandbox; --sandbox workspace-write lets it edit the workspace. --json emits JSON Lines events and --output-schema <file> constrains the final output. The documentation says --full-auto is no longer supported for codex exec.
Product (Claude Agent SDK, as documented on 2026-11-20): the SDK gives the tools, agent loop and context management that power Claude Code, programmable in Python and TypeScript; for other languages, the documentation says to run the CLI as a subprocess with -p and --output-format json.
What the factory adds is the same for both, and none of it comes from the agent. The spec passes station 1 before the agent starts. The agent gets one work item and only the permissions that item needs. Its structured output is checked for shape, and then the factory's own tests, written from the spec and run outside the agent's reach, decide acceptance. The manifest records the commit and the evidence. The agent's summary of what it did is a self-report and is stored as one.
Security note. Running headless is not the same as running isolated. Product (Claude Code, as documented on 2026-11-20): without--bare, a-psession runs a project's.claude/settings.jsonhooks and.mcp.jsonservers even in an untrusted folder, with no trust dialog. A factory that points a headless agent at a repository it did not write should treat those files as untrusted input. The controls around the agent's run belong to Harness Engineering.
In a coding agent. WatchAlgo's September workers were exactly this shape, and its evidence shows both halves. Observed in the repository (docs/EVIDENCE.md, section 5): each worker ran asclaude -p --model sonnet --dangerously-skip-permissionswith a 60-minute timeout, and the documentation read today says that flag equals--permission-mode bypassPermissions. The workers' permissions were not limited; the line's safety came from deterministic acceptance after each worker exited. A factory built today would keep the acceptance and narrow the permissions too.
What this is not. This page decides where a model belongs in a factory. The controls around one agent's run (permissions, hooks, context, the loop) are Harness Engineering. Graders, rubrics and evaluations for output that code cannot check are Eval Engineering; the factory invokes them as gates. What the model sees in its window is Context Engineering. And this is not about an agent that operates inside the product: the insurance factory's underwriting agent recommends and a person decides, which Insurance factory, part 2 keeps apart from building it.
Try it#
The verify lab runs Example B's oracle on a faulty or correct implementation. On the left, the stand-in's self-report; on the right, the oracle's results; the verdict comes from the evidence only. Switch between buggy (off by one), dupes (wrong on duplicates; this lab's own variant) and correct.
The same results for a reader without JavaScript, from a scratch copy of Example B on 2026-11-20:
| Variant | Self-report | Oracle (30 cases) | Failed cases | Verdict |
|---|---|---|---|---|
buggy (Example B attempt 1) | done, all tests pass | 21 passed | all-smaller, single-above and 7 random cases | rejected |
dupes (the lab's own variant) | done, all tests pass | 28 passed | duplicates, negatives | rejected |
correct (Example B attempt 2) | done, all tests pass | 30 passed | none | accepted |
The conveyor lab with preset b steps Example B's line and shows each station's contract: the deterministic oracle, the stand-in's solution with its rework, and the visualization that waits for an accepted solution.
| Station | Worker | Attempt | State | Note |
|---|---|---|---|---|
S1-build-oracle | deterministic code | 1 | passed | 30 cases, brute force agrees |
S2-generate-solution | scripted stand-in | 1 | failed, rework | stand-in says "done, all tests pass"; oracle 21/30 |
S2-generate-solution | scripted stand-in | 2 | passed | oracle 30/30 |
S3-trace-visualization | code over the accepted solution | 1 | passed | 10 traces, 28 states, invariants hold |
The agent here is a scripted stand-in, so these results show how the factory mechanics work, not how good any model is.
WatchAlgo in practice#
WatchAlgo's visualization station went through the change this page recommends, and the repository records both versions.
Observed in the repository (zenalgo-frontend, scripts/autonomous_visualization_assembly.py, commit b29dd499): in April 2026 a model wrote each visualization.json directly: one claude -p --model opus agent per problem, checked afterwards for shape only (the file parses, at least three test cases, all three languages, all three narrative flavors). Nothing checked that the animation states matched the algorithm. On 2026-04-02 the run produced 69 visualizations, and 1,431 attempts failed because the account was out of usage (docs/EVIDENCE.md, section 8).
Observed in the repository (scripts/viz/generate.py, scripts/viz/vizlib.py, commit d08ba4f0; first introduced in a0a0cbd6): in September the station was redesigned. A model worker now writes a small Python tracer program per problem that re-implements the algorithm and records its state at each step, and code, not the model, writes the visualization file from that program. Before the file is written, vizlib.validate checks every state against the data contract, and a Node cross-check runs the problem's real JavaScript solution on each test input and compares the result with the tracer's. Only if all of it passes does the file get written. 2,144 current visualizations come from tracers; 191 of 2,160 tracers opt out of the Node cross-check with a written reason, such as solutions that change their input in place (docs/evidence/station-contracts.md, S2b).
Observed in the repository (commit a0a0cbd6; .ai-context/VIZ_SOLUTION_BUGS_FOUND.md, commit 4426f119): the cross-check found 86 defects in solution.json files that had already passed the solution validator, about 80 wrong expected outputs and about 20 wrong algorithms in one category. The solution station's brief asks workers to compute expected outputs by running the JavaScript with Node, and no code checks that they did (docs/evidence/station-contracts.md, S1). An instruction is not a check. The evidence notes do not settle which stage produced the 86 defective files; they do show that an executable check, added later, caught what the solution validator had let through.
Diagram: WatchAlgo's visualization station · April against September
Two rows, observed in the repository. In April a model wrote each visualization file directly and the checks after it looked at shape only: nothing compared the animation states with the algorithm. In September a model writes a tracer program, code writes the visualization from that program, and the file is written only if the data contract holds for every state and a Node run of the real JavaScript solution agrees with the tracer. The cross-check also exposed 86 defects in solution files that earlier checks had passed.
- April 2026
- model writes visualization JSON: The artifact itself comes from the model.
- shape checks only: No check that the states match the algorithm.
- September 2026
- model writes a tracer program: The model writes a generator, not the artifact.
- code writes the visualization: The trace comes from executing the tracer.
- contract + Node cross-check: Every state checked; the real JavaScript solution run on each input and compared.
The diagram sets the two designs side by side. In the top row the model's output is the artifact, and the checks look only at its shape. In the bottom row the model's output is a program, code produces the artifact by running it, and the artifact is compared with an independent execution before it is written.
The pattern generalizes, and the evidence notes name it: the model writes a generator, code writes the artifact, and the artifact is checked against an execution the model did not perform. Reported by the author: specialized agents handled individual station responsibilities across 3,246 problems; the WatchAlgo story reconciles that account with the repository, and The visualizer factory develops the tracer contract.
Enterprise translation#
In Part III's insurance factory, Example C, the factory itself has no model station: it assembles the intake service from a pattern and a spec, and every station is code. Teaching reconstruction: in a real enterprise factory the semantic station would be the one that drafts a product's business-rule module from the spec's prose, for example a new eligibility rule written by an underwriting team, because no template can anticipate it. Everything around it would stay deterministic: the spec validator, the schema-compatibility gate, the generated tests for 401, 403 and 422, the secret scan, the manifest. A person with merge rights reviews the drafted module, and the release approver signs.
Keep two models apart, because they are easy to confuse. The factory's model writes code, and gates judge the code. The product's underwriting agent, a scripted stand-in in Example C, recommends at run time, and the model recommends; a person decides: an underwriter records every decision. The factory never makes an underwriting decision. In this setting the model station is rarely what dominates the critical path; the review of a regulated rule, security approval of the service, and integration with the systems around it take longer than writing the code.
Failure test#
The failure: the model station keeps producing the same wrong answer, and each answer says "done, all tests pass".
How the factory catches it: the oracle rejects every attempt, the attempt limit stops the loop, and the station escalates to a person instead of retrying forever. The run is never marked succeeded, and the visualization station never runs on an unaccepted solution. In a scratch copy of Example B:
$ /opt/homebrew/bin/python3 run.py --script buggy,buggy,buggy
[try ] S2-generate-solution solution attempt 1: stand-in says "done, all tests pass"; oracle: 21/30 passed -> rejected
[try ] S2-generate-solution solution attempt 2: stand-in says "done, all tests pass"; oracle: 21/30 passed -> rejected
[try ] S2-generate-solution solution attempt 3: stand-in says "done, all tests pass"; oracle: 21/30 passed -> rejected
[stop] S2-generate-solution solution: 3 attempts used, none accepted: escalated to a person
end FR-0002 status=escalated cost=1200 units (checkpoint kept)
generator calls in this process: 3
S1-build-oracle oracle done attempts=0
S2-generate-solution solution escalated attempts=3
S3-trace-visualization visualization pending attempts=0
(The output starts at the first attempt; the start and oracle lines are the same as in the fresh run above.) Three self-reports, three rejections, 1,200 illustrative units spent, and a person called. Notice the cost: failed attempts cost as much as successful ones, so a model station's cost is counted per accepted unit, not per attempt (Factory quality engineering).
What to run or toggle: in a scratch copy of Example B, run the command above, then /opt/homebrew/bin/python3 run.py --budget 500 to watch the budget guard stop the run before the second call. In the verify lab, pick dupes and find which two cases catch it.
Where to go deeper#
- The WatchAlgo production story (next): what the repository shows, and how the author's account reconciles with it.
- Evidence of correctness: properties, differential and metamorphic tests, and self-report against evidence.
- The visualizer factory: the "model writes a generator, code writes the artifact" pattern in detail.
- Long-running agent execution: context growth, checkpoints and resumable runs.
- Harness Engineering: permissions, hooks and the loop around one agent.
- Eval Engineering: graders and rubrics for output that code cannot check.
Checklist for Monday#
- For each step of your pipeline, ask: could this be written as code in advance? If yes, it is not a model's job.
- Put the model only where logic is written fresh from prose; keep every judging, recording and packaging step as code.
- Store every self-report, and never let it appear in an acceptance condition.
- Build the oracle from a different producer than the code it checks, and write down what it covers.
- Bound every model station: maximum attempts, a budget checked before each call, escalation to a named person.
- Run headless agents with narrow permissions and isolated settings, and treat repository configuration as untrusted input.
Exercises#
- A team proposes a "reviewer agent" that reads each generated function and approves it. What is wrong with it as an acceptance gate?
Answer 1An inferential check where code could decide
Its approval is another non-deterministic judgement, possibly sharing the producer's blind spots. Where the rule can be a test or an oracle, use that; keep the reviewer as an extra signal with its own measured error rate.
- Example B's oracle passes the
dupesvariant on 28 of 30 cases. What single change would make the random cases catch it?
Answer 2Generate duplicated targets
Make the random generator draw the target from the list's own values often, with repeats in the list. Duplicate targets are where the bug lives.
- Why does a model station make the byte-identical rerun of Example A meaningless at that station, but not the manifest?
Answer 3Regeneration differs; the record does not
A rerun may produce different code. The manifest still records, by hash, what was produced and which evidence accepted it, so the accepted artifact stays identifiable and reusable.
Advanced perspective#
Context limits in practice: small tasks, fresh workersWhat WatchAlgo's runs show, and what they do not
The standard answer to a finite context window is architectural: give each worker a small task and start a fresh worker for the next one, so context never accumulates. Observed in the repository (docs/EVIDENCE.md, section 8): each WatchAlgo task started a fresh claude -p process, so context growth was bounded by the task size (5 solution problems or 8 visualization problems), and nothing shows compaction inside workers. The runs did stop, but the logs show account usage limits (weekly, monthly spend, session) as the cause, not context limits.
That distinction matters for design. A context limit is per worker and is solved by task size and checkpoints. A usage limit is shared by every worker on the account, so the right response is a pause for all workers and a clean, resumable stop, which is why a worker count is bounded by the slowest shared resource, not by the machine. Long-running agent execution covers both, and Looping over a large backlog covers the concurrency limit.
The cost of a model station is per accepted unitFailed attempts, review, and escaped defects
Example B's stand-in costs 400 illustrative units per call. The fresh run spends 800 units for one accepted solution, and the escalated run spends 1,200 units for none. Both numbers are invented for the demo; the shape is not. A model station's cost per accepted unit is the cost per attempt divided by its first-pass yield, plus the review time for what it produces, plus the expected cost of the defects that escape its checks.
That arithmetic is why the decision table above prefers functions wherever logic can be written in advance: their cost per accepted unit is near zero and their escape rate is whatever their tests allow. It is also why a stronger oracle is often the cheapest improvement to a model station: it raises the denominator honestly by refusing bad work early, instead of letting it reach a reviewer or a user. Factory quality engineering works the numbers, simulated and illustrative, including a case where adding agents makes accepted output fall.
Headless permission defaults change; pin themTwo documented changes and one design rule
Both coding-agent products in this chapter document recent changes to how a headless run behaves. Product (Claude Code, as documented on 2026-11-20): from version 2.1.302, a -p run with no mode set starts in the default mode and denies calls that need approval; earlier versions could start in an automatic mode. Product (OpenAI Codex, as documented on 2026-11-20): --full-auto is no longer supported for codex exec; the documentation points to --sandbox workspace-write or --approve-for-me instead.
The design rule that follows does not depend on either product: a factory states the worker's permission mode, sandbox and allowed tools explicitly in the station's configuration, versioned with the pattern, and never relies on a default. Defaults change between versions; a station that silently gains or loses the right to write files is a different station. Record the agent version and its permission settings in the manifest next to the output, so a later reader can tell what the worker was allowed to do.
In one breath#
A model adds one thing a factory could not do before: interpret variable business logic written in prose and write new code from it. It takes away determinism, brings a finite context window, and produces confident wrong answers whose self-reports read exactly like the right ones, as Example B's stand-in shows with "done, all tests pass" against an oracle score of 21 of 30. So the model gets the semantic station only, and every station that validates, renders, compiles, tests, records or packages stays code, which is the change WatchAlgo made when its visualizations moved from model-written files to code-written traces checked against real runs.
Part I ends here: a primitive, a pattern, a validated spec, a reproducible line, typed station contracts, and a model confined to the one station that needs it. Part II tests those ideas against a real factory. The WatchAlgo production story opens the repository: what it shows, what the author reports, and where the two differ.