The most impressive orchestration code in the WatchAlgo repositories is a run engine built in June. It has an orchestrator agent with the role planner, a worker per stage, a quality-gates agent, a council of reviewers that reaches consensus, and a human approval step with a 24-hour timeout. Its stages are named description, solution and visualization. It has an admin screen in the frontend, and a public "recorded run" that replays one run step by step.
A reader who starts there will draw a clean station diagram from it and call that "how WatchAlgo was built". The diagram would be wrong. No content commit falls between 2026-04-02 and 2026-09-02, the engine was built on 2026-06-10, and the public recorded run is a demonstration assembled from one real unit's shipped files plus logs written for the demo. The corpus was filled by something much plainer: a queue file, short command-line worker processes, and checks written in code.
The learner's question: when you inherit a factory, or study someone else's, how do you divide it into stations from what actually produced the output, without inventing the parts you cannot see?
Evidence first, names second#
Plain English. Before calling anything a station, find the module that does the work, the input it reads, the file it writes, and the check that decides whether that file is accepted. Name it after that, not after the diagram you wish it had.
Assembly stations defined a station as a responsibility with declared inputs, outputs, a validator and an owner. This page applies that definition backwards. A station map is a table that matches each station to the evidence for it: the real module, who or what does the work, the check that accepts the output, and a status of observed, partly observed or unknown. A factory topology is the picture of the whole line built from that map: where work comes from, what coordinates it, which stations run in which order, how acceptance and checkpoints work, and how output reaches readers.
The word "station" is this site's teaching term. Observed in the repository (docs/evidence/station-contracts.md): the WatchAlgo code does not use it. The September pipeline speaks of task types (solution, viz); the June engine speaks of stages and agent roles. A station map translates from the code's words into this site's, and says so.
Before AI. Engineers who inherit a nightly batch system do this without thinking of it as a method. They read the scheduler's job list, look at which files land in which folders at what time, find the job that checks row counts before the load, and only then draw boxes. The job called FINAL_RECON_V2 that nobody has run since 2019 stays off the diagram, however good its name is.
The September queue pipeline#
Start with what filled most of the corpus. Observed in the repository (scripts/viz/pipeline.py, commit 971863f2; scripts/viz/supervise.sh, commit 017ed6b9): between 2026-09-02 and 2026-09-05 this pipeline produced 2,163 visualizations and 639 solutions.
The driver keeps a queue of tasks in one JSON file, .ai-context/pipeline/queue.json, protected by a file lock (fcntl.flock) and replaced atomically on every write. A task has a type, a list of problem numbers, a status and an attempt count. There are two task types. A solution task covers five problems (SOL_BATCH = 5); a viz task covers eight (VIZ_BATCH = 8). A viz task cannot be claimed until every one of its problems has a solution.json, which is a real dependency between stations.
Worker threads (w01 to wNN, eight by default under the supervisor) claim tasks and start one external process each. The worker command, as written in the driver:
cmd = [str(CLAUDE), "-p", prompt_for(t), "--model", "sonnet", "--dangerously-skip-permissions",
"--output-format", "text"]
So each worker is a headless Claude Code process with the task's prompt, one model setting for both task types, permission prompts skipped, and plain text output. Each has a 60-minute timeout (TASK_TIMEOUT). The two kinds of worker differ by instructions, not by model: the solution prompt from prompt_for, and for visualizations the brief in scripts/viz/WORKER_BRIEF.md.
After the process exits, the driver ignores whatever the worker printed about its own success. It calls verify_solution or verify_viz for every assigned problem, reading the files from disk. Verified problems are credited. The rest go into a retry task, up to MAX_ATTEMPTS = 3. Output that matches the rate-limit pattern pauses all workers for PAUSE_MINUTES = 12 and re-queues the task without spending an attempt. On restart, tasks left running by a dead driver are verified and their unfinished problems re-queued.
The supervisor wraps all of this. It restarts the driver when no driver process is found, commits verified content and the queue every 1,800 seconds, and exits only after the queue reports drained twice and a rebuild from disk adds nothing.
Security note. The observed workers ran with --dangerously-skip-permissions, so nothing stopped a worker from running any command on the author's machine. On a single-author machine with content-only tasks that was the author's call; it is not a pattern to copy. This site's recommendation for any station worker is a scoped checkout, a command allowlist and no credentials beyond the task, with Harness Engineering as the home of those controls.
The deterministic tracers: the model writes a generator, code writes the artifact#
The visualization station changed design between April and September, and the change is the most transferable idea in the repository.
Observed (scripts/autonomous_visualization_assembly.py, commit b29dd499): in April, one agent per problem wrote visualization.json by hand. The checks after it exited were shape checks: the file exists, parses, has at least three test cases, three languages and three flavors. Nothing checked that the recorded states matched the algorithm.
Observed (scripts/viz/generate.py, scripts/viz/vizlib.py, scripts/viz/WORKER_BRIEF.md, commit d08ba4f0): in September, the worker writes a Python module per problem, scripts/viz/problems/wa_NNNNNN.py, that re-implements the algorithm and records a state at each interesting line. The brief says, in so many words, not to write JSON by hand. Then code runs the tracer, builds the visualization, and checks it three ways: vizlib.validate checks every state's keys, the shared line ids and the allowed statuses; every test case must have at least two states; and the real JavaScript solution runs in Node on each test input and must give the tracer's output. Only if all three pass does vizlib.write write the file.
That design turns a model's output from the artifact itself into a program whose output can be executed and compared. 2,144 current visualizations have a tracer; 236 come from the earlier era without one. 191 tracers opt out of the Node cross-check with SKIP_NODE_VERIFY = True and a written reason, such as an in-place mutation or a class-based input the harness cannot build.
Earlier solution assembly#
Observed (.ai-context/SESSION_STATE.md, commit 0309048f): before and shortly after the first commit, solutions were generated in batches of 50 problems by 10 sub-agents of 5 problems each, first with one model and from batch 15 with a smaller one, with "gap-fills" by the larger. An hourly cron watchdog (.ai-context/cron-watchdog.sh, commit cb6e7ac3) started a claude -p session whenever it found gaps; its log shows the solution count rising from 1,615 to 2,154 between 2026-03-23 and 2026-04-03. SESSION_STATE.md also records a "recovered assembly pattern" in which Codex workers generated solution files on disjoint write sets, the main agent ran the authoritative disk-validation sweep, and Gemini only reviewed.
No session durations or transcripts survive for this phase. The instructions survive (.claude/SOLUTION_ASSEMBLY_LINE.md, 478 lines), which shows what workers were asked to do; it does not show what each session did.
The June orchestration platform#
Observed (~/Projects/cosmickeys/backend/app/orchestration/engine.py, commit 93e2a0b1; council.py, commit 06bf8363): a run engine with STAGE_ORDER = ("description", "solution", "visualization"), agents orchestrator (role planner), <stage>-worker (role worker) and quality-gates (role gate), council roles reviewer and consensus, gate checkers per stage, tables for runs and steps, an approval timeout that counts as a rejection, and tests. It is the clearest code evidence of specialized agent roles per stage in any WatchAlgo repository.
Nothing shows that it produced published content. No content commit falls in its period, and the engine was built after the last earlier content commit. The public recorded run is explicitly a demonstration: per .ai-context/REPLAY_FEATURE.md (commit 7be9297e) its structure was cloned from a local run and filled with wa-000001's shipped files plus logs written in the engine's voice. The evidence that would settle the question, the engine's real run records, would live in the backend database and was not inspected.
So the June engine belongs on the map with its own status: built, tested, not shown to produce output. That status is a fact worth teaching. A factory's most elaborate component is not necessarily its producing one.
The station map#
With the modules found, the stations can be named. Each row says what the evidence supports, and the last column says how much.
| Station (this site's name) | Real module | Who does the work | Accepted by | Status |
|---|---|---|---|---|
| S0 Problem statement | none found | unknown; all 3,246 files predate the first commit | scripts/validate_problems.py (structure and wording) | producer unknown, validator observed |
| S1 Solution | pipeline.py task type solution; earlier the cron watchdog | claude -p worker, 5 problems per task | verify_solution, which calls validate_payload | observed |
| S2a Visualization, April | autonomous_visualization_assembly.py | one agent per problem writes the JSON | shape checks only | observed |
| S2b Visualization, September | pipeline.py task type viz, generate.py, vizlib.py | claude -p worker writes a tracer; code writes the JSON | vizlib.validate + at least 2 states + Node cross-check | observed |
| S3 Acceptance | verify_solution, verify_viz in the driver | deterministic code after each worker exits | (it is the check) | observed |
| S4 Checkpoint | supervise.sh commit_progress | shell supervisor, every 30 minutes | none | observed |
| S5 Publication | npm run build; cache headers in next.config.ts | the author; deploy mechanism inferred | none recorded | partly inferred |
| S6 Review | reviews/*.md; instructions in the assembly-line spec | other models, early batches only | consensus by instruction | instruction observed, execution partial |
Three rows carry an unknown, and each unknown is a sentence a reader can rely on: nobody knows who wrote the problem statements, nobody recorded which commit went live, and nobody has shown that review continued past the early batches. Acceptance at S3 is code, not an agent. "Specialized agents per station", as the author reports it, holds in the September code as two worker kinds with different instructions running the same model.
Diagram: WatchAlgo factory topology, September 2026, with unknowns marked
Five lanes from work source to readers. Work comes from a scan of the disk for missing files, so disk is the source of truth and the queue can be rebuilt. The driver and supervisor coordinate: a locked queue, worker threads, a shared pause on usage limits, restarts. Stations are stateless command-line workers of two kinds; a visualization task waits until its solutions exist. Acceptance is code that runs after each worker exits; only accepted files reach the product folder; checkpoints commit every 30 minutes. Publication is a build and a deploy whose record is unknown. Pink boxes are unknowns, drawn in place.
- Work source
- scan disk for missing facets: The work list is rebuilt from public/content in a fixed category order; the queue is an index, not the truth.
- who chose the 3,246 problems: How the problem ids and titles were chosen is not recorded anywhere found.
- Coordinator
- driver + locked queue: pipeline.py run: worker threads claim tasks from queue.json under flock; retries up to 3; a 12-minute pause for all on a recognised limit.
- supervisor: supervise.sh restarts a dead driver and commits verified files and the queue every 1,800 seconds.
- Stations (workers)
- S1 solution worker: A headless command-line process with the solution prompt; writes solution.json files.
- S2b tracer worker: Writes one Python tracer per problem; code, not the model, writes the visualization.
- tool use inside workers: Workers ran with text output only; what they did inside a task is not logged beyond stdout.
- Acceptance and checkpoint
- S3 verify from disk: verify_solution and verify_viz check every assigned problem; the worker's own report is ignored.
- S4 checkpoint commit: Verified files and the queue committed to git; 64 checkpoint commits.
- expected outputs correct?: No validator checks that a solution's expected outputs are right; the cross-check at S2b found 86 defects later.
- Publication
- S5 build + deploy: Build from the working tree, inferred from probes; no deployment record was inspected.
Read the topology left to right. Work comes from a scan of the disk, which makes disk the source of truth and the queue a disposable index. The coordinator hands tasks to stateless workers, and acceptance runs outside the workers, after they exit. Checkpoints and publication sit at the end. The pink boxes are the unknowns, drawn in place so the picture never looks more complete than the evidence.
Common confusion. A queue is not a station, and neither is a supervisor. They move and protect work; stations transform it and checks accept it. Mixing the two produces maps where "the queue" seems to do the solving.
Example B: the teaching shape#
The WatchAlgo map is real but tangled across four phases. To study the shape in isolation, this site uses Example B (examples/b-algorithm-factory/). Teaching reconstruction: a toy algorithm unit, lower bound in a sorted list, with three logically independent work items: a solution, a test oracle, and a visualization. A work item is one deliverable the factory schedules, produces and accepts as a whole; it is the thing that moves through stations. Example B is not a claim about how WatchAlgo was built; its README says so in its first table.
An item dependency is a declared rule that one item cannot start, or cannot be accepted, until another is done. The specification declares the items and their dependencies as data, quoted from spec.json:
{"work_items": [
{"id": "oracle", "station_id": "S1-build-oracle", "depends_on": [], "produced_by": "deterministic code"},
{"id": "solution", "station_id": "S2-generate-solution", "depends_on": [], "accepted_by": "oracle", "produced_by": "scripted stand-in generator"},
{"id": "visualization", "station_id": "S3-trace-visualization", "depends_on": ["solution"], "produced_by": "deterministic code over the accepted solution"}
]}
The factory orders them with a stable topological sort, quoted from factory.py : topo_order:
def topo_order(work_items):
"""Kahn's algorithm; ready items are taken in sorted order so the order is stable."""
deps = {w["id"]: set(w["depends_on"]) for w in work_items}
done, order = set(), []
while len(order) < len(deps):
ready = sorted(i for i, d in deps.items() if i not in done and d <= done)
if not ready:
raise ValueError("dependency cycle among work items")
order.append(ready[0])
done.add(ready[0])
return order
Diagram: Example B: three work items and their dependencies (teaching reconstruction)
Left to right. The specification declares three work items. The oracle is built first by deterministic code. The solution is produced by the scripted stand-in; its generation needs nothing, but its acceptance needs the oracle, a dependency written as accepted_by and not read by the ordering function. Attempt 1 says done and fails 9 of 30 oracle cases; attempt 2 passes all 30. The visualization depends on the accepted solution and is produced by code, checked by state invariants. A checkpoint is written after each completed item. This is a teaching shape, not a description of how WatchAlgo was built.
- Specification
- spec.json: The toy lower-bound problem with its work items, dependencies and limits as data.
- Work items
- oracle: Brute force plus 10 fixtures and 20 seeded random cases; accepted when the fixtures agree with brute force.
- solution: The scripted stand-in writes the function; the oracle decides. At most 3 attempts, then escalation to a person.
- visualization: Traces of the accepted solution, checked by invariants such as 0 <= lo <= hi <= n.
- Evidence
- checkpoint.json: Written atomically after each completed item; resume skips items whose artifact hash still matches.
The diagram shows the two kinds of dependency. The visualization depends on the solution to be produced at all: it traces the accepted solution. The solution's generation needs nothing, but its acceptance needs the oracle. In spec.json that second dependency is written as accepted_by, which topo_order does not read; the oracle comes first because ready items are taken in sorted order, and "oracle" sorts before "solution". The run is correct, and it carries a lesson: an acceptance dependency belongs in the graph as much as a production one, or a change to the tie-breaking rule could silently reorder the line.
Here is the run, executed on 2026-11-20 with /opt/homebrew/bin/python3 run.py --verbose --no-cross-lang in a scratch copy. The agent here is a scripted stand-in, so these results show how the factory mechanics work, not how good any model is.
start FR-0002 spec SPEC-ALGO-001 pattern PAT-ALGO-UNIT@1.3.0 spec_hash 44d76f9399ab budget 2000 units
order: oracle -> solution -> visualization
[run ] S1-build-oracle oracle: 30 cases (10 fixtures incl. edge cases, 20 seeded random), sha256 45d4895d56b7
[try ] S2-generate-solution solution attempt 1: stand-in says "done, all tests pass"; oracle: 21/30 passed -> rejected
failed cases: all-smaller, single-above, random-06, random-09, random-10, random-14, random-17, random-18, random-20
all-smaller values=[1, 2, 3] target=9 expected=3 got=2
single-above values=[4] target=5 expected=1 got=0
random-06 values=[-13, -11, -1, 14, 15, 16] target=18 expected=6 got=5
[try ] S2-generate-solution solution attempt 2: stand-in says "done, all tests pass"; oracle: 30/30 passed -> accepted
[run ] S3-trace-visualization visualization: 10 traces, 28 states, invariants hold, sha256 2bf5ff26c1a1
trace for duplicates values=[1, 2, 2, 2, 3] target=2:
step 0 lo=0 hi=5 mid=None start
step 1 lo=0 hi=2 mid=2 values[2]=2 >= 2: hi = 2
step 2 lo=0 hi=1 mid=1 values[1]=2 >= 2: hi = 1
step 3 lo=1 hi=1 mid=0 values[0]=1 < 2: lo = 1
end FR-0002 status=succeeded cost=800 units generator_calls=2
generator calls in this process: 2
S1-build-oracle oracle done attempts=0
S2-generate-solution solution done attempts=2
S3-trace-visualization visualization done attempts=0
Compare it line by line with the WatchAlgo map. The stand-in's first attempt says "done, all tests pass" and the oracle finds 21 of 30; the factory acts on the oracle, as WatchAlgo's driver acted on verify_solution rather than on the worker's printout. The visualization runs only over an accepted solution, as a WatchAlgo viz task waited for its solutions. And the visualization is produced by code from the solution, the September tracer idea in miniature. The cost units (400 per generator call) are illustrative, not tokens or prices.
In a coding agent. The observed WatchAlgo workers were command-line coding-agent sessions, which makes the series' second example concrete here: a coding agent such as Claude Code running headless as a station worker receives one work item and a scoped checkout, writes code, and the factory's own tests decide whether the artifact advances. Example B's solution item is exactly such a slot, filled here by a scripted stand-in so the run is reproducible. A person with merge rights reviews what the gate accepts; Diff to Deploy follows that change to production.
What this is not. This page is not about the control plane that schedules and supervises many agents in general; that is Agent Orchestration. It maps one factory's stations to evidence and shows a teaching shape. Which orchestration product to use belongs to the landscape.
Try it#
The explorer shows the artifacts each station delivers for four real units: the problem from S0, the solution from S1, and the visualization from S2, with the commit that last changed each. Step a visualization to see what S2b's tracer recorded.
Which station delivered each facet of the four explorer units, by the commit that last changed the file:
| Unit | Problem (S0) | Solution (S1) | Visualization (S2) |
|---|---|---|---|
wa-000001 | cb6e7ac3, producer unknown | cb6e7ac3, before git | cb6e7ac3, the single visualization of the first commit, no tracer era |
wa-001378 | cb6e7ac3, producer unknown | cb6e7ac3, before git | d08ba4f0, September tracer era; wa_001378.py is a reference tracer |
wa-001689 | cb6e7ac3, producer unknown | cb6e7ac3, before git | d08ba4f0, September tracer era; wa_001689.py is a reference tracer |
wa-001830 | cb6e7ac3, producer unknown | f88d3c22, 2026-03-25 | d08ba4f0, September tracer era; wa_001830.py is a reference tracer |
The conveyor steps Example B's line. Use Step or Run to move the three items through queued, running, passed, failed and rework, and pick a station to read its contract.
Example B's station contracts and the run above, for a reader without JavaScript (from examples/b-algorithm-factory/README.md and the real output):
| Station | Work item | Depends on | Produced by | Accepted by | In the run |
|---|---|---|---|---|---|
S1-build-oracle | oracle | nothing | deterministic code: brute force, 10 fixtures, 20 seeded random cases | its fixtures agree with brute force | done, 30 cases, sha256 45d4895d56b7 |
S2-generate-solution | solution | nothing to generate; the oracle to be accepted | scripted stand-in generator | all 30 oracle cases pass; at most 3 attempts, then escalation | attempt 1 rejected (21/30), attempt 2 accepted (30/30) |
S3-trace-visualization | visualization | solution | deterministic code over the accepted solution | state invariants on every trace | done, 10 traces, 28 states, sha256 2bf5ff26c1a1 |
WatchAlgo in practice#
Reported by the author: specialized agents handled individual assembly-station responsibilities. Observed in the repository: in September, specialization was by instructions (a solution prompt, a tracer brief) on one model setting, with acceptance in code; in March and April, by role (Codex workers generating, Gemini reviewing only, per SESSION_STATE.md); in June, by declared agent roles in an engine with no shown output. The report holds, with the evidence spread across three designs.
Observed: review by other models is instructed in .claude/SOLUTION_ASSEMBLY_LINE.md ("until consensus (both Gemini + Codex approve)") and evidenced for early batches only (reviews/solution_review_batch1.md to batch3.md, reviewer "Claude Opus 4.6", 2026-03-21). No human review record exists.
Teaching reconstruction: the station map's recommendation for a line like this one is one role per station with its own brief, tool allowlist and acceptance test; specialization by instructions is often enough, and a different model per station is an option to justify with evidence, not a requirement.
Enterprise translation#
An enterprise team rarely starts with a blank page. It inherits a delivery process: tickets, build jobs, review rules, release steps, some scripts nobody owns. The method of this page applies first. Build the station map from evidence (CI logs, ticket transitions, the folders where artifacts land), mark unknowns, and only then decide which stations a factory should automate.
In the insurance factory of Part III, Example C's factory run FR-0003 has four stations that a station map can name without guessing, because each one prints its result: S1-validate-spec, S2-generate-contracts, S3-assemble-service, S4-write-manifest. Its work items are the five artifacts the run lists, from openapi.json to rollout_manifest.json, and its dependencies are real: contracts are generated from the validated specification, the service is assembled from the contracts. What a station map of a real enterprise line adds is the stations that are people: a security reviewer, a release approver, the underwriting team that signs off on rule changes. They belong on the map with their inputs, outputs and the evidence they leave, or the map hides the slowest part of the line.
Failure test#
The failure: the factory process dies after some items are accepted. A careless resume redoes accepted work, calls the generator again, or overwrites an accepted artifact with a new, different one.
How the factory catches it: a checkpoint after each completed item, and a resume that skips an item only when its recorded artifact hash still matches. WatchAlgo's driver did the coarse version: on restart, tasks left running were verified from disk and only unfinished problems were re-queued (19 orphan recoveries in the local run.log). Example B does the fine version, run on 2026-11-20 in a scratch copy:
$ /opt/homebrew/bin/python3 run.py --stop-after solution
start FR-0002 spec SPEC-ALGO-001 pattern PAT-ALGO-UNIT@1.3.0 spec_hash 44d76f9399ab budget 2000 units
[run ] S1-build-oracle oracle: 30 cases (10 fixtures incl. edge cases, 20 seeded random), sha256 45d4895d56b7
[try ] S2-generate-solution solution attempt 1: stand-in says "done, all tests pass"; oracle: 21/30 passed -> rejected
[try ] S2-generate-solution solution attempt 2: stand-in says "done, all tests pass"; oracle: 30/30 passed -> accepted
[kill] simulated crash after solution (checkpoint written, process gone)
checkpoint left at out/checkpoint.json; rerun with --resume
$ /opt/homebrew/bin/python3 run.py --resume --no-cross-lang
resume FR-0002 from checkpoint: done = oracle, solution
[skip] S1-build-oracle oracle: done in checkpoint, artifact sha256 matches
[skip] S2-generate-solution solution: done in checkpoint, artifact sha256 matches
[run ] S3-trace-visualization visualization: 10 traces, 28 states, invariants hold, sha256 2bf5ff26c1a1
end FR-0002 status=succeeded cost=800 units generator_calls=0
generator calls in this process: 0
S1-build-oracle oracle done attempts=0
S2-generate-solution solution done attempts=2
S3-trace-visualization visualization done attempts=0
The resumed process calls the generator zero times and produces the same visualization hash as the uninterrupted run. What to run or toggle: cp -R examples/b-algorithm-factory to a scratch folder and run the two commands above; then edit out/artifacts/solution.py between them and watch the resume rebuild the item instead of skipping it. Long-running agent execution is the home of checkpoints and resume.
Decision: one agent per unit, specialized workers, or a deterministic line with agents at semantic stations#
Every decomposition answers one question from the blueprint's decision set: who does the work at each step? WatchAlgo tried three answers over its phases, which makes it a useful case.
| Option | What it looks like | Choose it when | Counterexample in the evidence | Failure mode |
|---|---|---|---|---|
| One agent per whole unit | one session writes problem, solution and visualization | units are small, checks are cheap, and you only need a prototype | the April visualization run wrote JSON directly; no check tied states to the algorithm | unverifiable output that looks complete |
| Specialized agent per station | separate briefs per facet, a role per station | facets need different instructions and have separate checks | September used two briefs on one model; specialization did not need different models | coordination and context costs without a check between stations |
| Deterministic line, agents only at semantic stations | code coordinates, verifies, writes artifacts; agents write the parts that need interpretation | an executable oracle exists or can be built | the September tracers: the agent writes a generator, code writes and checks the visualization | an opted-out or missing check (191 tracers skip the Node cross-check) |
The criterion that decides between them is not the number of agents. It is where the independent check sits. This site's recommendation: start from the third option and move a step to an agent only where the work is semantic interpretation that code cannot do, keeping a check the agent did not write at the exit of that step.
Where to go deeper#
- The visualizer factory (next): the state contract behind S2b's tracers, with invariants you can break.
- Assembly stations: the station contract this page applies backwards.
- AI enters the factory: which stations should get a model at all.
- Long-running agent execution: checkpoints, resume and session limits.
- Looping over a large backlog: queues, WIP limits and retries at the scale of thousands of units.
- Agent Orchestration: the control plane for agents, beyond one factory.
- Harness Engineering: the controls around one worker's run, including permissions.
Checklist for Monday#
- Before naming a station, find its module, its input, its output file and the check that accepts it.
- Give every station a status: observed, partly observed, or unknown. Draw unknowns on the diagram.
- Do not take the most elaborate component as the producing one; check the commits for what produced output.
- Keep acceptance outside the worker, and run it after the worker exits, from the stored artifact.
- Declare acceptance dependencies in the graph, not only production ones.
- Put the human stations on the map, with what they receive and what evidence they leave.
Exercises#
- Why is the June engine not on the producing path of the station map, even though its stages match the facets?
Answer 1No output in its period
No content commit falls between 2026-04-02 and 2026-09-02, the engine was built on 2026-06-10, and its public replay is a demonstration; nothing shows it produced a published unit.
- Suppose
topo_ordertook ready items in reverse sorted order instead. What would happen in Example B?
Answer 2The solution would run first and fail
solution would be ready before oracle (neither declares a dependency on the other), so do_solution would try to load oracle cases that do not exist yet. Declaring the acceptance dependency in depends_on makes the order independent of tie-breaking.
- The September design had one model for both worker kinds. In what sense were its stations "specialized"?
Answer 3By instructions and checks
By instructions (a solution prompt and a tracer brief) and by acceptance checks (verify_solution and verify_viz), not by model.
Advanced perspective#
Task size as a design choiceFive or eight problems per worker, and why it bounded context
Observed: solution tasks carried five problems and visualization tasks eight, with a median of 14 minutes per task. Each task started a fresh process, so the context a worker accumulated was bounded by its task, and the evidence notes find no sign of context compaction inside workers. The batch size was the context policy.
The trade-off runs both ways. Larger tasks amortize the startup cost of a session and let a worker reuse what it learned on the first problem; they also lose more work when a usage limit cuts a task off mid-way, and a single bad problem can sink the whole task's attempt count. Smaller tasks retry precisely and checkpoint finely, at the price of more process starts and more coordination writes to the queue. WatchAlgo's driver softened the cost of large tasks by crediting each problem separately: a task cut off after four of five problems re-queued only the fifth, as a local log line shows ("cut off by a rate limit after 18 min (4 ok)"). Per-unit credit inside a multi-unit task is the detail that makes batching safe.
Disjoint write sets instead of locksThe March pattern and the September queue
The March "recovered assembly pattern" in SESSION_STATE.md gave each Codex worker a disjoint set of solution files to write, and left the authoritative validation sweep to one main agent. The September pipeline instead used one shared queue under a file lock, with each task's problem list as its write set.
Both avoid the same failure: two workers editing the same file. Disjoint write sets need no lock at run time but require the partition to be correct up front, and they cannot rebalance when one worker is slow. A locked queue rebalances naturally, since idle workers claim the next task, but the lock becomes a shared point that must be held briefly and released on crash; pipeline.py keeps critical sections short and writes atomically through a temporary file. In an enterprise factory the same choice appears as repository partitioning between teams versus a central scheduler with leases. Scaling safely is the home of locks, leases and isolation.
What would turn the unknowns into observationsExact evidence per unknown station
Each unknown on the map has a specific artifact that would resolve it. S0 needs the generator, prompts and logs for the 3,246 problem statements, from earlier sessions or a repository not found under the inspected folders. S5 needs deployment records with the source commit, from the hosting provider's dashboard or command history. S6 needs review records past the early batches, or a human acceptance sample with verdicts. The June engine's status needs its real run records (orch_runs, orch_steps) in the backend database, as opposed to the replay demonstration.
The design lesson for a new factory is cheap to state: every station should write the record that would let an outsider fill its row of the station map. A station that leaves no evidence of having run is, for any later reader, indistinguishable from a station that never existed.
In one breath#
WatchAlgo's stations, mapped from evidence before they are named, are a problem station with an unknown producer, a solution station and two generations of visualization station run by short command-line workers, acceptance written in code that ignores the workers' own reports, 30-minute checkpoint commits, and a publication step whose deploy record is unknown. The June engine, the most elaborate component, has no shown output and sits on the map with that status. Example B reduces the shape to three work items with explicit dependencies, and shows in a few lines both the payoff (the oracle overrules the stand-in's "done") and a gap worth closing (an acceptance dependency the graph does not declare).
The map is drawn and the dependencies are known. The station that changed most between April and September, the visualization, now needs its own contract. The visualizer factory follows one algorithm's state transitions into renderable steps and shows which invariants a trace must keep, because a visualization that looks right and traces the wrong algorithm is the defect this line was redesigned to stop.