All chapters
  1. Start hereA software factory, from first principles
  2. Primitive · 1A factory, before AI: input, transformation, check, output
  3. Primitive · 2From repetition to a pattern: what stays, what varies, what to leave alone
  4. Primitive · 3The specification as an executable contract
  5. Primitive · 4Build the smallest factory: four stations, two specs, one manifest
  6. Primitive · 5Assembly stations: typed contracts, owners and failure handling
  7. Primitive · 6AI enters the factory: what a model adds, and what stays deterministic
  8. WatchAlgo · 7The WatchAlgo production story: what the repository shows
  9. WatchAlgo · 8A problem as a production unit: the files one unit must contain
  10. WatchAlgo · 9Station by station: description, solution, implementations, visualization
  11. WatchAlgo · 10The visualizer factory: from algorithm state to renderable steps
  12. WatchAlgo · 11Three flavors and three languages: two different contracts
  13. WatchAlgo · 12Looping over a large backlog: scheduling, limits and a failed-case queue
  14. WatchAlgo · 13Long-running agent execution: context, checkpoints and token limits
  15. WatchAlgo · 14Evidence of correctness: tests the producer cannot talk its way past
  16. WatchAlgo · 15The factory learning loop: defects change the pattern, not only the unit
  17. Enterprise · 16Classify the enterprise demand before choosing a pattern
  18. Enterprise · 17Reference enterprise patterns and their variation points
  19. Enterprise · 18Platform foundations: golden paths, templates, registries and catalogs
  20. Enterprise · 19Pattern versus configuration versus business logic
  21. Enterprise · 20Insurance factory, part 1: building the intake contract
  22. Enterprise · 21Insurance factory, part 2: building the underwriting workflow
  23. Enterprise · 22Insurance factory, part 3: trust and observability built in
  24. Enterprise · 23Data and integration engineering: contracts that survive change
  25. Enterprise · 24The CI/CD and deployment factory: from pull request to production
  26. Enterprise · 25Enterprise governance: owners, approved patterns and provenance
  27. Enterprise · 26Scaling safely: many teams, many agents, one set of boundaries
  28. Operations · 27Factory quality engineering: measuring accepted output, not activity
  29. Operations · 28Failure and recovery scenarios: what breaks a factory, and what restores it
  30. Operations · 29The landscape: which tool for which job, read from its own documentation
  31. Operations · 30Design an enterprise software factory: the blueprint and the rollout
  32. ReferenceSources and verification
  33. ReferenceGlossary
  34. AboutAbout this site
Primitive · 5

Assembly stations: typed contracts, owners and failure handling

A station is a responsibility with declared inputs, outputs, validators and an owner. Dependencies, rework, and what a station does when its input or its own work fails.

You will write
A station contract with seven fields: inputs, outputs, validator, owner, dependencies, failure handling, and the line between retry and rework.
You will tell apart
A station, a pipeline and a harness, and who answers for each when it fails.
You will decide
How many stations a line should have: modularity against coordination cost, with a counterexample from Example A itself.

Late on 3 September 2026 the queue that was filling WatchAlgo with visualizations met a message it had never seen. The account behind its workers had reached its weekly usage limit, and every worker that started a task got back a limit message instead of doing the work. The pipeline knew how to handle a rate limit: pause every worker for 12 minutes and put the task back without counting it as a failed attempt. But it recognised rate limits by matching known phrases, and "weekly limit" was not one of them.

So the pipeline treated the message as bad work. Tasks "finished in 0 min", were recorded as failures, and failed problems got new retry tasks. The shell supervisor that watches the driver restarted it 18 times in about 13.5 hours. When the storm was over the queue held 11,896 tasks, of which 11,503 were failed ones. No content was produced in those hours, and none was corrupted.

Observed in the repository (zenalgo-frontend, scripts/viz/pipeline.py, commit 971863f2; local run.log; queue.json at 1b183564): the retry storm above, as reconstructed in docs/EVIDENCE.md (matrix rows 12 and 13, section 8). Every station did its work correctly. What failed was one sentence that no one had written down: what the station should do when its worker fails, and how to tell "the work was wrong" from "the worker could not work".

The learner's question: what must each station declare, in writing, so the line knows what to feed it, how to check it, who answers for it, and what to do when it fails?

A station is a promise with a checker#

Plain English. A station is one job on the line, with a written promise: what it takes in, what it hands on, how its output is checked, who is responsible for it, what it waits for, and what happens when something goes wrong.

Beginner definition: a station is a responsibility, not a worker. Code, a person, a model or a mix can do the work; the station's contract stays the same whoever does it.

The previous chapter ran Example A's four stations as four function calls. Each one already had a contract, but the contract lived in the code and in the README. A station contract is that promise written down as data the line can read: inputs, outputs, validator, owner, dependencies, failure handling, and the rule that separates a retry from rework. The analogy is a workstation on a manufacturing line with a card pinned above it: which parts arrive, which part leaves, which gauge checks it, which supervisor to call, and what to do with a part that fails the gauge. The consequence is that the line can act on a failure without asking anyone what was meant. The trade-off is that every field must be kept true as the station changes, and a contract that drifts from the code is worse than none.

Before AI. A Jenkins pipeline stage has a name, steps, a post block for failure and an owner in someone's head. A Make target declares its prerequisites and fails the build when its recipe exits non-zero. Both hold part of a station contract. Neither says who owns the stage, or separates "the network hiccuped" from "the code is wrong".

Here is Example A's line with its contracts filled in. The inputs, outputs, validators, dependencies and failure handling come from the example's README and from labcore's a_conveyor.py, which builds the conveyor lab from the example's own definitions. The owner column is this site's recommendation; Example A names no owners.

StationInputsOutputsValidatorOwner (recommended)Depends onOn failure
S1-validate-speca spec JSON filethe spec, or an exact error listschema.json fields and types, id formats, pattern id and version, body in the catalog, examples match params, known checkspattern teamnonerejected_at_S1; S2 and S3 skipped; errors go to the spec author
S2-render-artifactvalid spec + PAT-ALGO-UNIT 1.3.0module and test filenone: rendering is deterministicpattern teamS1not expected: same inputs give the same bytes
S3-compile-testthe two rendered filestest evidencepy_compile, then unittest in a subprocesspattern teamS2rejected_at_S3; the unit is never accepted
S4-write-manifesteverything abovemanifest.jsonalways runs, also for a rejectionpattern teamS3the rejection is recorded, never silent

Two cells deserve a second look. Station 2 has no validator, which the decision section below comes back to. And no station in Example A retries: every station is deterministic, so running it again on the same input would fail in the same way. Retries only make sense where something outside the input can change between attempts.

The smallest primitive: a contract as a type#

A contract becomes useful when the line can read it. The script below was written for this page and run with Python 3.14 in a scratch folder. It turns station 3's contract into a frozen dataclass and adds one function, next_step, that decides what the line does after a failed attempt. It is a teaching sketch: the retry fields and the three-attempt limit are added to show the decision, and Example A itself has neither.

from dataclasses import dataclass


@dataclass(frozen=True)
class StationContract:
    station_id: str
    inputs: tuple
    outputs: tuple
    validator: str
    owner: str
    depends_on: tuple
    max_attempts: int
    retry_on: frozenset    # transient: same input, try again
    rework_on: frozenset   # the output was wrong: send it back with the evidence


S3 = StationContract(
    station_id="S3-compile-test",
    inputs=("running_sum.py", "test_running_sum.py"),
    outputs=("test_evidence",),
    validator="py_compile both files, then unittest in a subprocess",
    owner="pattern team (PAT-ALGO-UNIT)",
    depends_on=("S2-render-artifact",),
    max_attempts=3,
    retry_on=frozenset({"transient_timeout"}),
    rework_on=frozenset({"test_failed", "compile_error"}),
)


def next_step(contract, error_class, failures_so_far):
    """What the line does after one failed attempt. Anything unclassified stops the line."""
    if failures_so_far >= contract.max_attempts:
        return "escalate to the owner"
    if error_class in contract.retry_on:
        return "retry: same input"
    if error_class in contract.rework_on:
        return "rework: send back with the failing evidence"
    return "stop and ask the owner: unknown failure"


for error_class, n in [("transient_timeout", 1), ("test_failed", 1), ("test_failed", 3), ("weekly limit", 1)]:
    print(f"{error_class:18} failures={n}  ->  {next_step(S3, error_class, n)}")
transient_timeout  failures=1  ->  retry: same input
test_failed        failures=1  ->  rework: send back with the failing evidence
test_failed        failures=3  ->  escalate to the owner
weekly limit       failures=1  ->  stop and ask the owner: unknown failure

The last line is the WatchAlgo night in miniature. The pipeline there had two buckets, "rate limit" and "everything else", and "everything else" meant bad content. A third bucket, "I do not recognise this failure", that stops and asks, would have turned 13.5 hours of retries into one alert. The general rule: a failure the contract does not name is not a content failure by default. It is a question for the owner.

Diagram: One attempt at one station · checked, then classified, then routed

Left to right. Inputs arrive from the stations this one depends on. The worker, code or a person or a model, produces an output. The validator, which the worker did not write, checks it. A pass hands the artifact on. A failure is classified before anything else happens: a transient failure is retried with the same input, a defect goes back as rework with the failing evidence, and an unknown failure or an exhausted attempt budget goes to the station owner. Every path ends in a record.

  1. Inputs
    1. artifacts from dependencies: Only the outputs of stations this one declares as dependencies.
  2. Station
    1. worker: Does the work. Interchangeable; the contract stays the same.
    2. validator: Checks the output against the contract.
  3. Routing
    1. classify the failure: The class decides the path; an unnamed class is never treated as bad content.
    2. retry: Something outside the work went wrong; run the attempt again.
    3. rework: The output had a defect; send it back with what failed.
    4. escalate: Attempts used up, or the failure has no name: a person decides.
  4. Outcome
    1. artifact handed on: Recorded with its hash and test evidence, then read by the next station.
    2. attempt recorded: Every attempt leaves a row, including the ones that failed.

The diagram follows one attempt through one station. Inputs arrive from the stations it depends on. The worker, whoever it is, produces an output. The validator, which the worker did not write, checks it. A pass hands the artifact on. A failure is classified first, and the class decides the path: retry with the same input, rework with the evidence attached, or escalation to the owner when attempts run out or the failure has no name. Every path ends in a record.

Owner, defect, rework#

A station owner is the named person or team who answers for a station: its contract, its validator, and the queue of work it refused. The owner is not the worker. In Example A every worker is code, but code cannot decide whether a failing example means the code or the spec is wrong; a person can. A good analogy is the shift supervisor of a workstation, who does not operate the machine but is the one called when the gauge keeps rejecting parts. The consequence is that every refusal has somewhere to go. The trade-off is that owners must exist and respond; an owner named on paper who never looks at the refusals is the same as no owner.

Common confusion. The conveyor lab's contract view has a row labelled "owner" that names who does the work: "code (no model)" for Example A, "scripted stand-in generator" for Example B's solution station. On this page that row is the worker. The owner is whoever answers when that worker's output is refused.

A defect is a property of an artifact that violates its specification or fails one of its checks. Example B's first solution attempt has one: it starts its search window with hi = len(values) - 1, so it can never return len(values), and every case where the target is larger than every value fails. Rework is sending work back to the station that produced it, or to the author of its input, because a check found a defect, together with the evidence of what failed. Rework is not a retry. A retry runs the same attempt again because something outside the work was temporarily wrong, such as a timeout; nothing about the work needs to change. Rework expects the work to change, so it carries the failing evidence with it.

The difference matters for three reasons. A retry of a defect produces the same defect and wastes the attempt, as WatchAlgo's night showed at scale. Rework without the evidence gives the producer nothing to fix. And rework and retries usually belong in separate counters: an interruption should not use up the attempts reserved for real defects. Example D, the supervisor of Part II, records exactly this distinction, and its output shows all three paths.

The examples' real failure paths#

Example D (examples/d-supervisor/) runs the station DAG of PAT-ALGO-UNIT over four factory runs and records every attempt in SQLite. Its stations are scripted stand-ins with scripted failures, so the output below shows how the supervisor handles failure, not how good any model is. From /opt/homebrew/bin/python3 run.py in a scratch copy on 2026-11-20, three of the four runs:

== FR-0004 SPEC-ALGO-001: clean run, one transient failure retried
   run status: succeeded; physical artifact writes by this process: 6
  station                  att  status     cost  artifact / error
  S1-validate-spec           1  succeeded     0  artifact://FR-0004/S1-validate-spec/171a648da216
  S2-generate-oracle         1  failed       30  transient_timeout
  S2-generate-oracle         2  succeeded    30  artifact://FR-0004/S2-generate-oracle/6981bd4abb75
  S2-generate-solution       1  succeeded    40  artifact://FR-0004/S2-generate-solution/4545fa833c8a
  S3-build-visualization     1  succeeded    25  artifact://FR-0004/S3-build-visualization/6e1f6f8ac7ea
  S3-run-tests               1  succeeded     0  artifact://FR-0004/S3-run-tests/94b662066656
  S4-package-manifest        1  succeeded     0  artifact://FR-0004/S4-package-manifest/122f893ee740
  tasks: T1=succeeded T2=succeeded T3=succeeded T4=succeeded T5=succeeded T6=succeeded

== FR-0005 SPEC-ALGO-002: resume run (new supervisor, same database)
   run status: succeeded; physical artifact writes by this process: 3
  station                  att  status     cost  artifact / error
  S1-validate-spec           1  succeeded     0  artifact://FR-0005/S1-validate-spec/6b1ff6027d36
  S2-generate-oracle         1  succeeded    30  artifact://FR-0005/S2-generate-oracle/17a636457aeb
  S2-generate-solution       1  abandoned     0  interrupted
  S2-generate-solution       2  succeeded    40  artifact://FR-0005/S2-generate-solution/b03036903131
  S3-build-visualization     1  succeeded    25  artifact://FR-0005/S3-build-visualization/f176f95de42b
  S3-run-tests               1  succeeded     0  artifact://FR-0005/S3-run-tests/192abe0c57d5
  S4-package-manifest        1  succeeded     0  artifact://FR-0005/S4-package-manifest/0212923625ed
  tasks: T1=succeeded T2=succeeded T3=succeeded T4=succeeded T5=succeeded T6=succeeded

== FR-0007 SPEC-ALGO-003: bounded failures and escalation
   run status: incomplete; physical artifact writes by this process: 4
  station                  att  status     cost  artifact / error
  S1-validate-spec           1  succeeded     0  artifact://FR-0007/S1-validate-spec/367a9daba2a1
  S2-generate-oracle         1  succeeded    30  artifact://FR-0007/S2-generate-oracle/d74b7cef763b
  S2-generate-solution       1  succeeded    40  artifact://FR-0007/S2-generate-solution/47cabbdf3454
  S3-build-visualization     1  failed       25  invariant_violation
  S3-build-visualization     2  failed       25  invariant_violation
  S3-build-visualization     3  failed       25  invariant_violation
  S3-run-tests               1  succeeded     0  artifact://FR-0007/S3-run-tests/ae596d71be22
  tasks: T1=succeeded T2=succeeded T3=succeeded T4=escalated T5=succeeded T6=blocked

Each run teaches one path. In FR-0004 the oracle station fails once with the class transient_timeout and passes on attempt 2: a retry, and the failed attempt still costs its units (illustrative units, not prices). In FR-0005 the process died after the solution station wrote its file but before the record was updated; the new supervisor marks that attempt abandoned with the class interrupted, which does not use up an attempt, and runs it again without writing the file twice. In FR-0007 the visualization station fails three times with invariant_violation, is escalated to a person, its dependent manifest station is blocked, and the run ends incomplete, never succeeded. The station that runs the tests does not depend on the visualization, so it still finishes.

Example B shows rework with evidence. Its solution station sends the failing cases back to the generator, which is the scripted stand-in for a model (factory.py : do_solution passes feedback = evidence["failures"] into the next call). Attempt 1 is rejected with 21 of 30 oracle cases passing; attempt 2 passes all 30. AI enters the factory reads that run closely.

Dependencies: the build graph#

A station's dependencies say which other stations' outputs it reads. Put every station and every dependency together and you get the line's build graph: a directed graph with no cycles (a DAG), in which an arrow from A to B means B cannot start until A has finished. The analogy is a Make file, which builds a target only after its prerequisites. The consequence is the one Example D uses: stations with no path between them can run at the same time. The trade-off is that every arrow must be true. A missing arrow lets a station read an output that does not exist yet; an unnecessary arrow makes the line slower than it needs to be.

Example D's graph is a tuple in dag.py, quoted verbatim:

PATTERN_TASKS = (
    Task("T1", "S1-validate-spec", (), "deterministic", 0),
    Task("T2", "S2-generate-solution", ("S1-validate-spec",), "model-stand-in", 40),
    Task("T3", "S2-generate-oracle", ("S1-validate-spec",), "model-stand-in", 30),
    Task("T4", "S3-build-visualization", ("S2-generate-solution",), "model-stand-in", 25),
    Task("T5", "S3-run-tests", ("S2-generate-solution", "S2-generate-oracle"), "deterministic", 0),
    Task("T6", "S4-package-manifest", ("S3-build-visualization", "S3-run-tests"), "deterministic", 0),
)

The planner (dag.py : topo_levels) turns it into four levels: S1, then the oracle and the solution together, then the visualization and the tests together, then the manifest. Tasks inside one level run in parallel on a bounded thread pool; a level starts only when the one before it has finished. Example B adds a distinction worth keeping: its solution station does not need the oracle to generate code, but it needs the oracle to be accepted. The factory therefore builds the oracle first (oracle -> solution -> visualization). A dependency can be on data or on a gate, and both belong in the graph.

Diagram: Example D's build graph · four levels, parallel where nothing connects

Columns are the planner's levels. Level 1 validates the spec. Level 2 generates the oracle and the solution in parallel; both read only the validated spec. Level 3 builds the visualization from the solution and runs the tests against solution and oracle, again in parallel. Level 4 packages the manifest after both. In FR-0007 the visualization fails three times and is escalated: the manifest is blocked, the tests still finish, and the run ends incomplete.

  1. Level 1
    1. S1-validate-spec: Validates the spec; costs 0 illustrative units.
  2. Level 2 (parallel)
    1. S2-generate-oracle: Depends on S1 only. In FR-0004 it fails once with transient_timeout and is retried.
    2. S2-generate-solution: Depends on S1 only.
  3. Level 3 (parallel)
    1. S3-build-visualization: Reads the solution. In FR-0007 it fails three times: escalated.
    2. S3-run-tests: Reads solution and oracle; independent of the visualization, so it still finishes in FR-0007.
  4. Level 4
    1. S4-package-manifest: Needs both level-3 stations. Blocked in FR-0007, so the run ends incomplete.

The diagram shows Example D's six stations in their four levels. Arrows are dependencies. The two pairs in the middle levels run in parallel; the dashed path shows FR-0007, where an escalated visualization blocks the manifest but not the tests.

Assembly line, conveyor, pipeline, harness#

An assembly line is the ordered set of stations a unit passes through, with a handoff between each. A conveyor is the mechanism that moves work and its artifacts from one station to the next. In Example A the conveyor is a function call and a folder. In Example D it is the scheduler reading SQLite. In WatchAlgo it was a file-locked queue.json. The point of naming it: when a line is slow or loses work, the fault is often in the conveyor, not in any station.

Three more words are easy to mix up. A pipeline is the execution order and dependency graph across stations; a workflow engine is one way to run it. A harness is the set of runtime controls around one agent's execution: its permissions, tools, context, hooks and loop. Harness Engineering is the sister site about it.

TermScopeWho answers for itExample
stationone responsibility, one contractthe station ownerS3-compile-test
pipeline (line)order and dependencies of all stationsthe pattern's ownersExample D's four levels
harnessthe controls around one agent's run at one stationwhoever operates that agentthe permissions and tools a headless coding agent gets
factorypattern, spec, pipeline, checks, artifacts, run record, feedbackthe platform team that runs itFR-0001 from spec to manifest
In a coding agent. A coding agent such as Claude Code or Codex, running headless as the worker at one station, sits inside a harness inside a station. The harness limits what the agent may touch while it works. The station contract decides what it receives, which check judges its output, and what happens when that check fails. The factory's own tests, not the agent's, decide whether the work advances, and a person with merge rights reviews the change. A harness that lets the agent run its own tests is useful feedback for the agent; it is not the station's validator.

How many stations? Modularity against coordination cost#

This page owns one of the site's decisions. Every station boundary buys something and costs something, and teams rarely count both.

The question. Should this piece of work be its own station, or part of a neighbouring one?

Diagram: Should this be its own station? · four questions about one boundary

Ask the questions in order about a proposed boundary. Does the work have its own independent check? Does a different person or team answer for it? Does it fail differently, so it needs its own retry or rework rule? Is its output reused, cached by input hash, or produced in parallel with something else? Any yes earns a station; write that reason into the contract. Four noes mean the boundary only adds a handoff, a record, a wait and a contract to maintain: merge it into its neighbour.

  1. Questions
    1. own validator?: A check the worker did not write, run at this boundary.
    2. own owner?: Someone else answers for refusals here.
    3. own failure policy?: It fails in a different way from its neighbours.
    4. reused or parallel?: Its output is cached or it runs beside another station.
  2. Answer
    1. a station: The boundary buys something; the contract records what.
    2. merge it: No check, owner, policy or reuse: only coordination cost.

The diagram reads top to bottom as four questions about a candidate boundary. If the work has its own independent check, or its own owner, or its own failure and retry policy, or an output worth reusing or running in parallel, it earns a station. If none is true, the boundary costs coordination and buys nothing, and the work belongs inside its neighbour.

OptionChoose it whenIt buysIt costs
One station for the whole jobThe job is small, has one check at the end, one owner, and failures are rare and cheap to redoLeast coordination; one record rowA failure says only "the job failed"; a retry redoes everything; nothing is reused
A few coarse stationsEach boundary has its own check or owner (Example A)Failures point at a station; refusals have an ownerA contract and an artifact handoff per boundary
Many fine stationsParts can run in parallel, fail differently, or produce reusable outputs (Example D)Parallelism, targeted retries, reuse by input hashMore contracts to keep true, more records, more places for the conveyor to fail

The criteria, in the order to ask them: does this boundary have an independent validator? Does a different person or team answer for it? Does it fail differently, so it needs its own retry or rework rule? Is its output reused, cached, or produced in parallel with something else?

The counterexample, from Example A itself. Station 2, rendering, has no validator: its contract says "none; rendering from templates is deterministic". By the first criterion it should not be a station; it could be the first half of station 3. It earns its place on a different criterion: its output is an artifact that station 4 hashes and a person may open, and keeping it separate means a compile failure points at the rendered file rather than at the renderer. That is a judgment, and a team could reasonably merge the two. A station with no validator, no owner of its own and no reused output is a stage in a script, and it is fine to call it that.

The failure modes. Too few stations: a large step fails and nobody can tell which part, every retry repeats finished work, and the record says nothing useful. Too many: stations with no check pass their input through, contracts multiply until nobody keeps them true, and the coordination itself (handoffs, records, waiting for the slowest parallel branch) becomes the cost that dominates. Factory quality engineering works an arithmetic case where adding parallel workers makes a line slower and more expensive.

What this is not. This page defines stations and their contracts. The controls around one agent's run belong to Harness Engineering. Retries, checkpoints and recovery as a discipline belong to Agent Reliability Engineering. The control plane that schedules many agents belongs to Agent Orchestration Engineering. The durable record of every attempt is introduced briefly below and developed on Long-running agent execution.

A durable record of every attempt#

Example D keeps its station contracts' results in a database table, station_attempts in state.py. The schema is quoted from the file; it ran in SQLite during the example's run, and the build does not check SQL:

CREATE TABLE IF NOT EXISTS station_attempts (
  run_id TEXT NOT NULL,
  task_id TEXT NOT NULL,
  station_id TEXT NOT NULL,
  attempt INTEGER NOT NULL,
  status TEXT NOT NULL,
  input_hash TEXT NOT NULL,
  artifact_ref TEXT,
  test_evidence_ref TEXT,
  error_class TEXT,
  cost_units INTEGER NOT NULL DEFAULT 0,
  started_at TEXT NOT NULL,
  finished_at TEXT,
  PRIMARY KEY (run_id, station_id, attempt),
  FOREIGN KEY (run_id) REFERENCES factory_runs(run_id)
);

Read the columns as the station contract's fields at run time. input_hash records what the station consumed, so an identical input can reuse a recorded artifact. artifact_ref and test_evidence_ref point at the output and its check. error_class holds the failure class that decides between retry, rework and escalation. attempt is bounded. A row is written as running before the work starts and updated after, which is why the crash in FR-0005 left an honest record. Long-running agent execution is the home of this record, resume and idempotency; here it is enough to see that a station contract is not finished until its outcomes are stored somewhere a scheduler can query.

Try it#

The conveyor lab steps a line of stations one event at a time: queued, running, passed, failed, rework. Pick a station to see its contract. The first embed is Example A's line; the second injects one simulated failure at station 3 (data-fail), so you can watch rework happen.

The events for a reader without JavaScript, from labcore a_conveyor.py (real runs of the examples; the injected failure is simulated):

LineStationAttemptStateNote
A, defaulteach of S1 to S41running, passed0 errors; 2 files; 4 tests run, 0 failed; status accepted
A, data-fail="S3-compile-test"S3-compile-test1failed, reworkinjected failure; sent back to the spec author, who fixes the input
A, data-fail="S3-compile-test"S3-compile-test2passed4 tests run, 0 failed
B, defaultS2-generate-solution1failed, reworkstand-in says "done, all tests pass"; oracle 21/30
B, defaultS2-generate-solution2passedoracle 30/30
B, data-fail="S2-generate-solution"S2-generate-solution1 to 3failed, failed, passedinjected failure, then 21/30, then 30/30 (the third and last allowed attempt)

Try data-fail on Example B's solution station in your head first: the injected failure uses one of the three attempts, so the real fix arrives on the last one. One more failure and the station would escalate. The agent in these examples is a scripted stand-in, so these results show how the factory mechanics work, not how good any model is.

WatchAlgo in practice#

"Station" is this site's word; the WatchAlgo code does not use it. Observed in the repository (zenalgo-frontend, scripts/viz/pipeline.py, commit 971863f2): the September pipeline has two task types, solution (5 problems per task) and viz (8 per task), and a viz task cannot be claimed until every one of its problems has a solution.json, which is a dependency between stations. Acceptance is deterministic code in the driver, after the worker exits: run_task ignores the worker's own success lines and calls verify_solution or verify_viz for every assigned problem, from disk. A problem that fails gets a retry task, up to MAX_ATTEMPTS = 3. Output matching the rate-limit pattern pauses all workers for PAUSE_MINUTES = 12 and re-queues without spending an attempt. The evidence note is docs/evidence/station-contracts.md (section S3).

Read against this page, that is most of a station contract: inputs, outputs, an independent validator, a dependency, bounded attempts, and a separate path for one failure class. Two fields were missing. There was no explicit "unknown failure" class, which is how one unrecognised message became a content failure, as the opening told. And there was no recorded human owner of refusals: Observed in the repository: the human role visible in commit messages is starting runs, setting concurrency, requeueing by hand and standing workers down (docs/EVIDENCE.md, section 5). Teaching reconstruction: the station-contracts note proposes classifying every non-success exit as ok, content_fail, limit or unknown, and pausing on unknown. Reported by the author: specialized agents handled individual station responsibilities; the WatchAlgo story sets that report beside what the repository shows.

Enterprise translation#

In an enterprise the contract fields stay the same and the owner column fills with different teams. In Example C, the factory that builds the insurance intake service, the stations are S1-validate-spec, S2-generate-contracts, S3-assemble-service and S4-write-manifest, followed by CI gates such as the schema-compatibility check and the secret scan. This site's recommendation for owners: the platform team owns the pattern stations and the conveyor; the domain team that owns the intake product owns rework on its spec; the security team owns the secret-scan gate and its exceptions; a release approver, a person, owns the last gate. When the schema gate refuses a breaking change, the refusal goes to the domain team with the failing field list, not to the platform team, and not back into a retry loop.

Two enterprise facts change the decision on station count. Ownership boundaries are real organizational boundaries, so a station that crosses two teams usually needs splitting even if the work is small, because each team needs its own refusal queue. And some stations wait on people for hours or days, so the conveyor must be durable: a function call cannot wait for a security approval. That is where the record above, and the workflow engines on the landscape page, come in. What dominates the critical path in this setting is rarely a station's compute time; it is the queues in front of human-owned stations.

Failure test#

The failure: a station's worker fails for a reason the contract does not name, and the line treats it as bad work. Retries pile up, each one a wasted attempt, while the real cause (a limit, an outage, a revoked credential) goes unreported.

How the factory catches it: a contract with an explicit unknown class stops instead of retrying. The primitive above shows it: weekly limit gets "stop and ask the owner: unknown failure", while transient_timeout gets a retry and test_failed gets rework. Example D shows the neighbouring rules working: an interruption is abandoned, not failed, so it costs no attempt; three real failures escalate; a blocked dependent keeps the run incomplete.

What to run or toggle: run the primitive with a failure class of your own and read the decision. In the lab, set data-fail on Example B's solution station and count how many attempts are left for the real fix. In a scratch copy of Example D, /opt/homebrew/bin/python3 run.py --verbose prints every supervisor event, including the escalation in FR-0007.

Where to go deeper#

Checklist for Monday#

Exercises#

  1. A station's worker times out three times in a row. Retry, rework or escalate?
Answer 1Retry, then escalate

Timeouts are a retry class, but the attempt limit still applies: after the third, escalate to the owner. Repeated timeouts are a signal about the environment, not the work.

  1. In Example D, why can S3-run-tests finish in FR-0007 when the visualization is escalated?
Answer 2No path between them

It depends only on the solution and the oracle. The manifest depends on both S3 stations, so it is blocked, and the run ends incomplete.

  1. Your rendering step has no validator. Name one reason to keep it as a separate station anyway.
Answer 3Its output is an artifact

Its output is hashed, recorded and opened by people, and a failure downstream then points at a specific rendered file. Without such a reason, merge it.

Advanced perspective#

Failure classification is the hard part of failure handlingMost retry storms are classification bugs

A retry policy is easy to write: try up to three times, wait between attempts. The hard part is deciding which failures deserve it. Failures fall into at least four classes with different correct responses. Content failures (the output is wrong) need rework with evidence. Transient infrastructure failures (a timeout, a dropped connection) need a retry, ideally with backoff. Capacity failures (a rate limit, a quota, a spend limit) need every worker to pause together, because retrying one worker against an exhausted shared limit only burns attempts. Unknown failures need a stop and a person.

WatchAlgo's pipeline had the capacity class and applied it correctly to the messages it knew. Its storm came from a capacity failure that fell into the content bucket. The design lesson is that the default bucket must be the safe one: an unrecognised failure should pause, not retry. A second lesson: classification belongs to the station contract, versioned with it, because new failure messages appear when vendors, models or limits change, and the list has an owner like any other part of the contract.

Rework fixes the unit; a recurring defect should fix the stationFrom the artifact to the pattern

Rework repairs one artifact. When the same defect class appears again and again at one station, repairing artifacts one by one is the expensive response; the cheaper one is to change the station: a stricter validator, a better template, a clearer brief for the worker. Kief Morris makes the same distinction for coding agents in Humans and Agents in Software Engineering Loops (martinfowler.com, 4 March 2026, read 2026-11-20): when output is bad, a person working in the loop fixes the artifact, while a person working on the loop fixes the harness that produced it.

WatchAlgo's history shows the station-level fix happening. The April visualizations were written by a model and checked for shape only; in September the station was redesigned so that a model writes a tracer program and code writes the visualization, checked against a real run of the solution (AI enters the factory tells that story). Record defects with their station and class, so the pattern can be fixed where the defects come from. The factory learning loop is the home of that practice.

Station contracts drift, so version them with the patternValidators tightened after the files were written

A validator that gets stricter is a contract change. Artifacts accepted under the old contract do not become invalid by themselves, but a re-check under the new contract will refuse some of them, and without a record nobody can tell which contract accepted what. WatchAlgo's evidence has a measured case: on a scratch copy of every tenth unit, 70 of 281 solutions fail the solution validator as it stands today, all of them numbered wa-001761 or lower, written before the rules were tightened (docs/EVIDENCE.md, section 10.3).

Three practices keep this manageable. Version each station's contract as part of the pattern version. Record in each manifest which contract version accepted each artifact. And when a contract tightens, decide explicitly between re-checking old artifacts, grandfathering them with a label, or reworking them, and record the decision. The choice is a governance one, not a technical one; Enterprise governance covers who makes it.

What a station boundary costs, countedHandoff, record, wait, contract upkeep

Each boundary adds four costs. A handoff: the output must be written as an artifact, hashed and read back. A record: at least one row per attempt, written before and after the work. A wait: in a parallel level, the next level starts only when the slowest branch finishes, so a level is as slow as its slowest station. And upkeep: one more contract whose fields must stay true as the code changes.

In Example A these costs are tiny, because everything runs in one process in milliseconds. In a line where stations are separate processes on separate machines, or wait on people, they dominate. The useful habit is to ask, for each proposed boundary, which of the four benefits it buys (an independent check, an owner, a failure policy, reuse or parallelism) and to write that reason into the contract. A boundary whose reason nobody can state is a candidate for merging at the next pattern release.

In one breath#

A station is a responsibility with a written contract: inputs, outputs, a validator the worker did not write, an owner who answers for refusals, dependencies, and failure handling that tells a retry (something outside the work went wrong) from rework (the work was wrong, so it goes back with the evidence). Unknown failures stop and ask instead of retrying, which is the lesson of WatchAlgo's 13.5-hour retry storm. A boundary earns a station when it has its own check, owner, failure policy or reusable output; otherwise it is coordination cost.

So far every worker on the line has been code or a scripted stand-in that behaves the same way every time. The next chapter, AI enters the factory, puts a model at one station: what it adds (interpretation of variable business logic), what it takes away (determinism), and why every other station stays a plain function, a compiler or a test.