Open WatchAlgo's learning page and its description says "Explore 3,247 algorithm problems". The file it publishes for AI crawlers says "3,200+ problems". The author's own account says 3,246. The live sitemap lists 3,246 problem pages. Then open one of the later problems and the Solution tab says "Coming Soon".
None of these numbers is a lie, and none of them alone tells you what the factory produced. A reader who repeats the biggest one has learned nothing about the factory. A reader who dismisses all of them has thrown away a real case. The useful move is to ask, for each number, what it counts and where it comes from.
The learner's question: how do you turn one project's production story into evidence a skeptical engineer can check, without inflating it and without discarding it?
Three columns before any number#
Plain English. Before quoting any fact about WatchAlgo, this site sorts it into one of three piles: something a file or log on disk shows, something the author says, or something this site proposes to explain the idea.
WatchAlgo is the author's own algorithm-learning product: one page per algorithm problem, each with a problem statement, a worked solution in code, and an animated visualization of the algorithm running. It is the main case study of this site because its content was produced by a repeatable process with agents at some of its steps. A case study is only useful if the reader can tell what was seen from what was said.
The first pile is observed evidence: a statement backed by a file, commit or log, cited by path and commit so anyone with the repository can open it. For example, "3,246 folders named wa-000001 to wa-003246, each with a problem.json" is observed (public/content/, HEAD 1b183564).
The second pile is the author report: the operational account the author gave, recorded as a report and never as a measurement. It says WatchAlgo contains 3,246 implemented algorithm problems with descriptions, solutions and visualizations in three flavors; that specialized agents handled individual station responsibilities; and that a repeatable process ran for 30+ hours in long-running sessions until token limits required intervention.
The third pile is the teaching reconstruction: an architecture, toy or simulator this site builds to explain the idea, such as Example B. It is a design, not a description of WatchAlgo, and it says so every time it appears.
Put the three piles side by side, one row per claim, and you have an evidence matrix. Matching each reported number against what the files count, and explaining every gap, is a reconciliation.
Before AI. An auditor does this with every set of accounts. Management asserts a revenue figure; the auditor samples invoices and bank records; the opinion says which assertions the evidence supports. Nobody calls the assertion false because it is an assertion, and nobody signs it because it is confident.
The source of every WatchAlgo statement on this site is docs/EVIDENCE.md and its four supporting notes in docs/evidence/. They were written from read-only inspection of the repositories, the factory's own local logs and the public site. Corpus code ran only on copies.
What the repository holds#
The smallest useful primitive here is a count that a script makes, not a person. This site ships a read-only scanner, examples/watchalgo-adapter/inventory.py, that reads a WatchAlgo content folder and prints what is really there. It never writes to the scanned folder; its own test checks that.
Here it is, run on 2026-11-20 against the author's working tree (copied adapter in a scratch folder, WatchAlgo read in place, read-only). The first block of its output:
WatchAlgo content inventory (read-only)
root: /Users/.../zenalgo-frontend/public/content
entries in root : 3247
well-formed wa-NNNNNN : 3246 range ['wa-000001', 'wa-003246']
number gaps in range : 0 []
other entries : 1
- 'wa-000{925,926,927,928,929,930,931,932,933,934}' (dir, children=0)
facet files present (units holding each file)
problem.json 3246
solution.json 2773
visualization.json 2380
facet combinations per unit
problem+solution+visualization 2380
problem 473
problem+solution 393
JSON parse errors : 0
Read it slowly. Every unit is one folder with up to three files, called facets here: a problem statement, a solution, a visualization. All 3,246 folders have a problem. Fewer have a solution, fewer still a visualization. One directory entry is not a unit at all.
Observed in the repository (zenalgo-frontend working tree, commit 1b183564): git ls-tree -r HEAD public/content lists the same 3,246 / 2,773 / 2,380 files and git status shows no uncommitted change under public/content, so the working tree equals that commit. The branch holding it, feature/admin-orchestration, has no upstream; the pushed main branch (ea250663) holds 2,134 solutions and 217 visualizations.
The facet counts also answer what "three flavors" means. Observed in the repository: every present facet carries three narrative styles of the same problem, technical, fun and spiritual; wa-000001 is "Remove Duplicates from Sorted Array", "Remove Extra Fruits" and "One of Each Stone". Code languages (javascript, python, java) are a separate axis. The multi-flavor contract is the home of that distinction.
Reconciling 3,246#
Here is the problem the opening story raised, now with every number traced to what it counts.
| Number | Where it appears | What it counts |
|---|---|---|
| 3,246 | the author's report; scripts/viz/pipeline.py and supervise.sh; checkpoint commit messages | problem units with a problem.json in three flavors |
| 3,247 | src/app/learn/page.tsx, the comment in src/app/sitemap.ts, the architecture notes | the raw directory listing, including one empty folder |
| 3,200+ | public/llms.txt, src/app/layout.tsx | a rounded public figure |
| 3,246 | the live sitemap: 3,266 URLs, of which 3,246 are problem pages | problem pages published |
| 2,773 | working tree, commit 1b183564 | problems with a solution |
| 2,380 | working tree, commit 1b183564 | problems with a visualization, which here means complete units |
| about 2,287 and 2,162 | the live site, inferred from probes | solutions and visualizations publicly served |
The 3,247 comes from one empty folder literally named wa-000{925,926,927,928,929,930,931,932,933,934}. That is what a shell leaves behind when a command such as mkdir wa-000{925,...,934} runs where brace expansion is off. Git does not track empty folders, so the repository never shows it. Every script that matters filters it with the strict pattern ^wa-\d{6}$ (scripts/generate-sitemap-manifest.mjs line 31; scripts/generate_content_manifest.mjs line 21, commit 3b4c4a48). The live sitemap, fetched again on 2026-11-20, lists 3,266 URLs and 3,246 distinct problem pages, wa-000001 through wa-003246.
So the reconciliation is one sentence with one qualifier. 3,246 is a correct count of problems described in three flavors; it is not a count of problems solved and visualized. On the author's machine 2,773 have solutions (85.4%) and 2,380 have visualizations (73.3%). A problem without a solution shows a "Coming Soon" placeholder rather than an error (ProblemPageClient.tsx line 692).
The live site serves fewer still. Sequential read-only probes found every file added up to checkpoint c641a13b (2026-09-03 10:26 PDT) live, and every file added from checkpoint 38946471 (13:26 the same day) missing. At c641a13b the corpus held 2,287 solutions and 2,162 visualizations. That the deployment was built from the working tree in that window is an inference: no deployment record was inspected.
Common confusion. Reconciling a number is not correcting it. This site keeps 3,246 as the author's figure, because it is right for what it counts. What changes is the sentence around it: three numbers (registered, complete, published), each with its denominator, never one.
Four phases of production#
The git history and the local logs divide the work into four phases. The diagram puts them in order with what each produced.
Diagram: Four phases of production, as the history shows them
Four lanes in time order. Phase 0, before the first commit, produced every problem statement and 1,820 solutions, and left no generator in any repository. Phase 1 in March and April added solutions and the first 216 visualizations with command-line agent runs and an hourly watchdog. Phase 2 in June and July built a run engine with planner, worker, gate and reviewer roles and produced no content. Phase 3, three days in September, filled most visualizations with a file-locked queue, short worker processes and acceptance written in code. The amount of machinery in a phase says nothing about how much it produced.
- 0 · Before git (to 2026-03-25)
- 3,246 problem.json 1,820 solution.json: Observed: the first commit adds every problem and 1,820 solutions in one go.
- generator: not found: No script, prompt or log for the problem statements exists in any repository found.
- 1 · March to April
- solution batches + hourly watchdog: Observed: batches of 50 problems by 10 sub-agents of 5; a cron watchdog relaunched a command-line session when it found gaps.
- first visualizations: Observed: an April run of 5 parallel agents wrote visualization files by hand; 69 done, 1,431 failed on usage limits.
- 2 · June to July
- orchestration engine + admin UI: Observed: a run engine in the shared backend with planner, worker, gate, reviewer and consensus roles per stage. No content commit falls in this phase.
- 3 · 2026-09-02 to 09-05
- queue + tracers + supervisor: Observed: a file-locked queue, claude -p workers, deterministic tracers that write visualization files, verification in code, 30-minute checkpoint commits.
- solutions 2,134 to 2,773: Observed: solution tasks of 5 problems each, accepted by the solution validator after the worker exits.
Read the lanes left to right. The first phase produced the largest single block, every problem statement and 1,820 solutions, and left no generator behind: all of it arrived in the first commit, cb6e7ac3 (2026-03-25). The third phase built the most elaborate machinery, a run engine with planner, worker, gate and reviewer roles, and produced no content. The fourth phase, three days in September, filled most of the visualizations with a queue, short worker processes and checks written in code. The size of a phase's machinery says nothing about how much it produced.
Observed in the repository: content-adding commits sum exactly to the final counts. Solutions: 1,865 on 2026-03-25, 269 on 2026-04-02, 639 in September, total 2,773. Visualizations: 1, then 216, then 2,163, total 2,380. Reported by the author: specialized agents handled individual station responsibilities. The September code shows two worker kinds with different instructions running the same model; Station by station maps them.
What "30+ hours" means, read from the logs#
The author reports a repeatable process that ran for 30+ hours in long-running sessions until token limits required intervention. The logs let a reader test that sentence without turning it into a measurement it never was.
A timestamp says when a line was written. The distance between two timestamps is a span, not a measured run time, and it says nothing about tokens or about how long a person was away. With that rule, here is what the September logs show (local, untracked files written by the factory's own scripts on one machine):
| Measure | Value | Source |
|---|---|---|
run.log first to last line | 2026-09-02 19:28 to 2026-09-05 13:13, a 65.8-hour span | local log |
| the same, minus silent gaps over 20 minutes | 59.2 hours | arithmetic |
supervisor.log span | 62.3 hours | local log |
| productive windows | about 09-02 22:10 to 09-03 18:40, and 09-04 13:05 to 09-05 13:13 | first to last verified task |
| between them | about 4 hours of recognised rate-limit pauses, then about 13.5 hours with no output | local log |
| one worker task | median 14 minutes, p90 19 to 21 minutes | productive tasks only |
The span is consistent with "30+ hours". The word "session" needs care. The workers were not one long agent conversation. Each task started a fresh command-line process, claude -p, that worked on five or eight problems and exited; code restarted the next one. What ran for days was the supervisor and its queue, not any single agent.
"Until token limits required intervention" holds, with one correction of wording. The last lines of 10,657 worker logs are account usage messages: "You've hit your weekly limit" (9,674), a monthly spend limit (976), a session limit (7). These are the account's usage allowance running out, not a context window filling up. Nothing in the logs shows context compaction inside a worker; each task started fresh, so context growth was bounded by the task size. The intervention a usage limit requires is waiting, or raising the limit.
The retry storm: a pause that turned into 11,503 failures#
Here is the failure the logs make visible, and it is the most useful thing in this case study.
The September driver, scripts/viz/pipeline.py, had a sensible rule. When a worker's output matched a rate-limit pattern, it paused every worker for 12 minutes and put the task back without spending an attempt. Any other unsuccessful exit counted as a failed attempt; after three, the task was marked failed.
The pattern was a list of phrases: usage limit, rate limit, spend limit, session limit and a few more. On 2026-09-03 at about 23:05 the account reached its weekly limit. Every worker exited at once with "You've hit your weekly limit". The phrase "weekly limit" matches none of the listed phrases.
Diagram: How one unrecognised message turned a pause into a retry storm
Left to right. A worker exits with the account's weekly-limit message. The driver checks the output against its rate-limit pattern; the phrase weekly limit is not on the list, so the exit is treated as a content failure. The task spends an attempt and gets a retry, which hits the same limit seconds later; after three attempts the task is marked failed. When the queue drains into failures the supervisor restarts the driver and rebuilds the queue, every 47 minutes or so for about 13.5 hours. The lower path is what protected the corpus: acceptance reads the disk after every worker, and nothing unverified is credited.
- Worker exit
- "You've hit your weekly limit": The last line of 9,674 worker logs. An account usage limit, not a context window.
- Classification
- rate-limit pattern: A list of phrases in pipeline.py. A match pauses all workers 12 minutes and spends no attempt.
- no match: content failure: Anything unmatched counts as a failed attempt. There is no unknown class that would pause instead.
- Retry loop
- retry, up to 3: Each retry hits the same limit within seconds; after three attempts the task is marked failed.
- queue drained, supervisor restarts: The driver exits on an empty queue; the supervisor restarts it and rebuilds 272 tasks from disk.
- What held
- acceptance reads disk: verify_solution and verify_viz check each claimed file; a worker that wrote nothing gets nothing credited.
Follow the arrows. The unrecognised message was classified as a content failure, so each task spent an attempt and got a retry task. Retries hit the same limit within seconds. Tasks reached three attempts and were marked failed. The queue drained into failures, the driver exited with "queue drained", the supervisor restarted it and rebuilt the queue from disk, 272 tasks each time, every 47 minutes or so for about 13.5 hours. The queue file ended with 11,896 tasks, 11,503 of them failed.
Now the part that matters. No content was lost and none was corrupted. The verifier checked every claimed file on disk after each worker exited, and a worker that wrote nothing had nothing credited. Verification protected the content; only the bookkeeping inflated. The same thing had already happened once on 2026-09-02 with "monthly spend limit"; commit 971863f2 added the spend and session phrases. The fix covered the message seen that day, not the next one.
The lesson generalizes beyond WatchAlgo: an unclassified failure class turns a pause into a retry storm. A retry policy is only as good as the classification in front of it, and the safe default for an exit nobody classified is to stop and ask, not to retry.
The defect logs are evidence of verification#
A production story that reports no defects should worry a reader more than one that does. WatchAlgo's repository holds its own defect logs, and they show checks catching work that earlier steps had accepted.
Observed in the repository (.ai-context/VIZ_SOLUTION_BUGS_FOUND.md, commit 4426f119): 86 solution.json defects in one category (Hashing & Prefix), found when the September visualization generator ran each JavaScript solution in Node and compared it with its own trace: about 80 wrong expected outputs, about 20 wrong algorithms, and one solution copied from another problem. Observed (.ai-context/SESSION_STATE.md, commit 0309048f): 15 problem examples with wrong outputs, found while writing solutions; wa-000379 duplicates wa-000378 ("Two Sum"); 13 units blocked from visualization by broken reference code.
Two things follow. These defects had passed the solution validator, which checks shape and wording but not whether an expected output is right. And they were found by a check that ran code, written by something other than the producer. That is the central claim of this site seen in someone else's logs.
The evidence matrix, reader-sized#
The full matrix in docs/EVIDENCE.md has seventeen rows. These eight carry the case study.
| Claim | Observed in the repository | Reported by the author | Teaching reconstruction |
|---|---|---|---|
| Problem count | 3,246 folders, no gaps; all added in cb6e7ac3 | 3,246 implemented problems | count units by script with a strict filter |
| Complete units | 2,380 with all three facets; 393 problem and solution; 473 problem only | descriptions, solutions and visualizations | report registered, complete and published, each with a denominator |
| Three flavors | technical, fun, spiritual in every present facet | in three flavors | flavors are variants of one station's output, checked for the same meaning |
| Specialized agents | two worker kinds with different instructions, same model; acceptance by code | specialized agents per station | one role per station with its own brief and acceptance test |
| Generator code | September queue and tracers present; no generator for problem.json | a repeatable process | coordinator, durable queue, stateless workers, independent acceptance |
| 30+ hours | log spans of 62 to 66 hours with two productive windows; not measured | 30+ hours in long sessions | a run ledger with start, end, active time and pauses |
| Limits | account usage limits in 10,657 worker logs | token limits required intervention | budget per accepted unit; stop cleanly with resumable state |
| Human review | no human review record; reviews in reviews/ are model-written | no manual review | sample-based human acceptance with a recorded rate |
The pattern across the rows is consistent. The author's report survives contact with the files, with narrower wording in three places: "implemented" means described, not finished; "sessions" means many short processes under a supervisor; "token limits" means account usage limits.
The system the repository describes#
The modules below are the ones the files show, with their connections. Nothing is drawn that the evidence notes do not cite.
Diagram: The discovered modules and their connections
Five lanes, each holding only modules the evidence notes cite. Instructions are files the workers read. The control lane is the supervisor and the queue driver, which spawns workers and runs acceptance in code after each one exits. Workers are command-line processes of two kinds. Only files that pass acceptance reach the product folder; the renderer reads them as static files and the site is published from the working tree, an inference because no deployment record was inspected. The last lane holds the two places a reader is tempted to fill gaps: a June engine that exists but shows no published output, and a problem generator found in no repository.
- Instructions
- assembly-line specs: Committed instruction files read by solution workers and the watchdog prompt.
- tracer brief: The 100-line brief for September tracer workers: write a Python tracer, never JSON by hand.
- Control
- supervise.sh: Keeps the driver alive and commits verified content and the queue every 1,800 seconds.
- pipeline.py: The driver: a queue file under flock, worker threads, acceptance after each exit, retries up to 3, a shared pause on limits.
- queue.json + run.log: Queue state committed; driver log local. The queue can be rebuilt by scanning for missing files.
- Workers (CLI)
- solution worker: A claude -p process with the solution prompt; writes solution.json files.
- tracer worker: A claude -p process that writes one Python tracer per problem; code then writes the visualization.
- Product and publication
- wa-NNNNNN folders: The product: three JSON facets per unit under public/content.
- renderer: Reads the static files; a missing facet shows Coming Soon.
- build + deploy: Sitemap manifest, then the site build; the live snapshot matches the working tree of 2026-09-03 (inferred).
- Where readers fill gaps
- June run engine: Planner, worker, gate, reviewer and consensus roles in the shared backend; nothing shows it produced published content.
- problem generator: The producer of the 3,246 problem statements is unknown.
Instructions feed workers; the driver spawns workers, holds the queue and runs acceptance; only accepted files reach the product folder; the renderer reads static files; publication goes out from the working tree. The two boxes on the right edge, the June engine and the problem generator, are where a reader is most tempted to fill gaps: one exists but shows no published output, the other does not exist in any repository found.
The quality denominator#
3,246 units, or 2,380 complete ones, do not by themselves establish that the content is correct, that it teaches well, or that the problems are of comparable difficulty. A count is a numerator waiting for its denominator.
What the evidence does establish is narrower and more useful: schema validity everywhere the validators ran, executed-output agreement for the visualizations made by deterministic tracers, and known defect pockets in earlier solutions. On a scratch copy of every tenth unit, 70 of 281 solutions fail today's solution validator, all numbered wa-001761 or lower: the rules were tightened after those files were written. What is missing is a human sample with recorded verdicts. Evidence of correctness says what that sample would need to measure.
In a coding agent. The observed workers were command-line coding-agent sessions: Claude Code started headless with claude -p (non-interactive mode, which the documentation now describes as running the Agent SDK through the command line), one work item per process, the driver's own checks deciding what counted. That is the shape this site recommends for a coding agent at a station: a scoped work item in, the factory's tests and not the agent's deciding, a person with merge rights reviewing. The runtime around one such agent belongs to Harness Engineering.
What this is not. This page is not a benchmark of WatchAlgo or of any model: no throughput figure here is a measurement of model quality, and none should be compared across products. It is not a release-quality evaluation either; rubrics, golden sets and graders belong to Eval Engineering. It is a reading of one project's evidence, with every claim labelled.
Check it yourself: the reproducibility notebook#
Everything below runs on a reader's machine without modifying WatchAlgo. The commands are the ones docs/EVIDENCE.md gives; this page ran them again on 2026-11-20 on copies in a scratch folder, and the outputs are pasted as they printed. A successful run proves the counts and the sample checks; it does not prove that any unit outside the copied sample is correct.
First, the inventory and its tests (the scanner reads, never writes):
/opt/homebrew/bin/python3 examples/watchalgo-adapter/inventory.py ~/Projects/zenalgo-frontend/public/content
/opt/homebrew/bin/python3 -m unittest discover -s examples/watchalgo-adapter/tests -v
entries in root : 3247
well-formed wa-NNNNNN : 3246 range ['wa-000001', 'wa-003246']
units complete (3 facets x 3 flavors x 3 languages, all parse): 2380
Ran 8 tests in 0.020s
OK
Second, run four units' JavaScript solutions against the structured test cases of their visualizations, on copies. The runner, run_js_cases.mjs, is printed in full in docs/EVIDENCE.md section 10.2; it follows the same input conversions as WatchAlgo's own cross-check and runs each case in a node:vm sandbox with a five-second timeout. Node v26.3.1:
wa-000001 removeDuplicates(nums)
tc0 Basic: small array with one duplicate: PASS output=2
tc1 Edge: all identical elements: PASS output=1
tc2 Complex: longer array with multiple duplicates: PASS output=5
wa-001378 inorderTraversal(root)
tc0 Basic: right child with a left grandchild: PASS output=[1,3,2]
tc1 Edge: empty tree: PASS output=[]
tc2 Complex: full binary search tree: PASS output=[1,2,3,4,5,6,7]
wa-001689 countComponents(n, edges)
tc0 Basic: two clusters: PASS output=2
tc1 Edge: no edges at all: PASS output=4
tc2 Complex: a redundant edge closes a cycle: PASS output=2
wa-001830 floodFill(canvas, startRow, startCol, paintColor)
tc0 Basic: 3×3 image, fill from the centre: PASS output=[[2,2,2],[2,2,0],[2,0,1]]
tc1 Edge: new color equals the old color: PASS output=[[0,0],[0,0]]
tc2 Complex: 4×4 with a winding region: PASS output=[[3,3,0,1],[3,0,0,1],[3,3,3,0],[0,3,0,1]]
units run: 4, skipped (no solution/viz or no top-level function): 0
test cases: 12 pass, 0 fail
failing units: 0
Third, count the facets at the checkpoint the live site appears to match, with read-only git:
git -C ~/Projects/zenalgo-frontend ls-tree -r --name-only c641a13b public/content | awk -F/ '{print $4}' | sort | uniq -c
3246 problem.json
2287 solution.json
2162 visualization.json
docs/EVIDENCE.md section 10.3 adds the factory's own validators run on copies; A problem as a production unit runs the solution validator on a small sample and shows what it misses.
Try it#
The explorer holds four real units, chosen for different algorithm shapes. Pick a problem, switch between the three narrative flavors and the three code languages, and step the visualization states. Every view shows the source path and the commit that last changed the file. The lab invents nothing: the excerpts come from examples/watchalgo-adapter/fixtures/watchalgo-excerpts.json, extracted read-only at commit 1b183564.
The same four units for a reader without JavaScript (states are kept up to 12 per test case in the fixture):
| Unit | Technical · fun · spiritual title | Algorithm type and structure | Last changed (problem · solution · visualization) | States in test case 0 |
|---|---|---|---|---|
wa-000001 | Remove Duplicates from Sorted Array · Remove Extra Fruits · One of Each Stone | two-pointer, array | cb6e7ac3 · cb6e7ac3 · cb6e7ac3 | 8 |
wa-001378 | Binary Tree Inorder Traversal · Visit Family Left-First · Walking the Left Path First | recursive-dfs, tree | cb6e7ac3 · cb6e7ac3 · d08ba4f0 | 13 (12 kept) |
wa-001689 | Number of Connected Components in an Undirected Graph · Count the Neighborhoods · Islands of Stillness | union-find, graph | cb6e7ac3 · cb6e7ac3 · d08ba4f0 | 11 |
wa-001830 | Flood Fill · Painting Desk Groups · Color Flowing Through Stones | bfs, matrix | cb6e7ac3 · f88d3c22 · d08ba4f0 | 21 (12 kept) |
Notice the last-changed column. Three of the four visualizations date from d08ba4f0, the September commit that added the tree, graph and grid renderer; wa-000001's visualization is the single one in the first commit.
WatchAlgo in practice#
This whole page is WatchAlgo in practice, so this module answers the question a reader brings to any case study: what carries over, and what was particular to this project?
Observed in the repository, and general: one folder per unit with fixed facets and a strict id pattern; a work list rebuilt from disk, so disk is the source of truth; a file-locked queue of small tasks handled by stateless command-line workers; acceptance by code after the worker exits, with the worker's own "done" ignored; an executable cross-check that runs the reference code; a station dependency (a visualization waits for its solution); a limit-aware pause shared by all workers; periodic checkpoint commits.
Observed, and specific to WatchAlgo: the facet names and the wa-NNNNNN scheme, the claude -p command and its flags, the vendor's limit messages, the 30-minute cadence, the technical, fun and spiritual voices.
Teaching reconstruction, where the analogy stops: algorithm problems usually have one right answer that code can check, and most enterprise work has no cheap oracle of that kind. WatchAlgo is a solo author's content site with no external users, no change approval and no production data. It is one content factory, not an enterprise delivery process; the enterprise blueprint holds the full table of where the analogy fails.
Enterprise translation#
The three-column discipline transfers directly. An enterprise team reporting on its own factory, or evaluating a vendor's claim about one, faces the same three piles: what the systems record, what people report, and what the proposal says will happen. In the insurance factory of Part III, a statement such as "the factory built the intake service in one run" is observed only if the factory run FR-0003 has a record with its stations, attempts and artifacts; a figure in a slide is a report.
The reconciliation transfers too. "Number of services generated" is a 3,247-style number: it counts directory entries. The numbers an enterprise needs are units registered, units that passed every gate, and units in production, each with its denominator, and a defect log that shows checks catching things. Factory quality engineering defines those measures and computes cost per accepted unit rather than per generated one.
The retry storm transfers most directly of all. Any enterprise line that retries on failure needs a failure classification with an explicit "unknown" class, a cap on retries per unit and per hour, and an alarm when attempts rise while accepted output stays flat.
Failure test#
The failure: a worker exits with a message the retry policy has never seen, the policy treats it as a content failure, and every retry hits the same wall.
How a factory catches it: classify every non-success exit before deciding what to do with it, and make "unknown" a class that pauses the line and asks a person. The script below holds WatchAlgo's observed pattern, copied as written from scripts/viz/pipeline.py (commit 971863f2), and a four-class version this site proposes (teaching reconstruction). It is a simplified demonstration: it models only the classification step, not the driver's verification from disk. The three limit messages are the last lines of real worker logs, reset times removed; the last two messages are invented for contrast.
import re
# The pattern as written in scripts/viz/pipeline.py (RATE_LIMIT_RE, commit 971863f2).
OBSERVED = re.compile(
r"usage limit|rate limit|rate limited|out of extra usage|spend limit|session limit"
r"|429|overloaded|capacity|quota|temporarily limiting", re.I)
# Last lines of worker logs, as the evidence notes give them (reset times removed).
MESSAGES = [
"You've hit your monthly spend limit",
"You've hit your session limit",
"You've hit your weekly limit",
"SyntaxError in tracer wa_001234.py", # invented content error, for contrast
"", # invented: the worker exited and printed nothing
]
def observed(text):
"""Two outcomes: a recognised limit pauses; anything else costs an attempt."""
return "pause 12 min, no attempt spent" if OBSERVED.search(text) else "content failure, attempt spent"
def classified(text):
"""Teaching reconstruction: four classes, and the unknown class pauses too."""
if OBSERVED.search(text) or re.search(r"\blimit\b", text, re.I):
return "limit: pause all workers"
if re.search(r"Error|Traceback", text):
return "content: retry with the error as feedback"
return "unknown: pause and ask a person"
for m in MESSAGES:
print(f"{m!r:44} observed -> {observed(m):32} classified -> {classified(m)}")
"You've hit your monthly spend limit" observed -> pause 12 min, no attempt spent classified -> limit: pause all workers
"You've hit your session limit" observed -> pause 12 min, no attempt spent classified -> limit: pause all workers
"You've hit your weekly limit" observed -> content failure, attempt spent classified -> limit: pause all workers
'SyntaxError in tracer wa_001234.py' observed -> content failure, attempt spent classified -> content: retry with the error as feedback
'' observed -> content failure, attempt spent classified -> unknown: pause and ask a person
Line three is the storm in one row. The generic limit rule would have caught the weekly message, and the unknown class would have stopped the empty exit from costing three attempts. What to run or toggle: save the script and run it with /opt/homebrew/bin/python3; then delete the \blimit\b rule and watch line three go back to "content failure". Looping over a large backlog is the home of failed-case queues and retry caps.
Where to go deeper#
- A problem as a production unit (next): the three files inside one unit folder, field by field.
- Station by station: the stations mapped to modules, with unknowns marked, and Example B's teaching shape.
- Long-running agent execution: checkpoints, resume and session limits as a design topic.
- Evidence of correctness: what a human acceptance sample and an executable oracle add to the defect logs above.
- The enterprise blueprint: the full table of where the WatchAlgo analogy fails.
- Agent Reliability: retries, checkpoints and recovery as a discipline.
- Observability: the telemetry this case study lacks, such as per-task tokens and a run ledger.
Checklist for Monday#
- Label every number you quote about a factory: observed (with a path or record), reported (with who said it), or reconstructed.
- Report three counts for any batch: registered, complete, published. Never one.
- Generate counts with a script and a strict id filter; never type them by hand.
- Read a log span as a span. If you need run time, record start, end and pauses in a run ledger.
- Classify every non-success exit before retrying, and pause on the class nobody has seen.
- Keep the defect logs. A factory that shows none has not shown it checks.
Exercises#
- A colleague says "the factory produced 3,247 problems". What two corrections does the evidence support?
Answer 1One folder is empty; most counts are partial
The 3,247th entry is an empty brace-expansion folder, so 3,246 units exist; and 3,246 counts problem statements, of which 2,773 have solutions and 2,380 have visualizations.
- The supervisor log spans 62.3 hours. Why is "the agent ran for 62 hours" wrong twice?
Answer 2Span is not run time; workers were short
A span between timestamps includes about 17.5 hours of pauses and retry storm with no output, and the workers were many short claude -p processes (median 14 minutes per task), not one agent.
- During the retry storm, 11,503 tasks failed. How many verified files were corrupted, and why?
Answer 3None: acceptance read the disk
None. The driver credited only files that passed verification on disk after each worker exited; workers that wrote nothing had nothing credited, so only the queue bookkeeping inflated.
Advanced perspective#
What evidence would upgrade this case studyA run ledger, usage export, deployment records, a human sample
Every "reported" cell in the matrix names the artifact that would move it to "observed". The generator and logs for the 3,246 problem statements would turn the first station from unknown into observed. A per-task token and cost export from the provider would turn "token limits" into a number and make cost per accepted unit computable. A run ledger with start, stop and pause per session would turn "30+ hours" from reported to measured; it was not recorded, though it could be rebuilt partly from worker log file times. Deployment records with the source commit would turn the publication path from inferred to observed. A human acceptance sample with recorded verdicts would turn "no manual review" into a measured quality rate.
The design lesson is to decide, before a factory runs, which of these records it must write. Each one is cheap at run time and impossible to recover afterwards. A factory that intends to make claims about itself needs telemetry designed for those claims, not logs written for debugging.
A partial failure recorded as successThe watchdog that still fires
Observed (local cron and the tracked file .ai-context/cron-watchdog.log): an hourly cron entry still runs the March watchdog. On 2026-11-20 at 10:00 it found four "gaps", failed to start the claude binary at its old path (the binary is gone), and then logged "Resume triggered" anyway. The log claims success after a failed launch.
Two operational points come out of it. First, a launcher that reports its intent ("triggered") instead of its result ("process started with id N") turns every downstream reader into a victim of the same mistake. Second, schedulers outlive the work they were created for; this one would start generation sessions on its own if a binary reappeared at that path. An enterprise factory needs an inventory of its schedules with an owner and an expiry, and launch steps that verify their own effect. Nothing was changed; the evidence notes report it for the author to decide.
Where the concurrency ceiling came fromThe account, not the machine
Observed (.ai-context/SESSION_STATE.md, commit 0309048f): "keep total concurrent Claude sessions ≤ ~8"; running 22 at once (8 driver workers plus 14 in-session teammates) triggered account-wide throttling that killed every teammate within seconds. Commit 971863f2 reports one run with 24 concurrent workers; the supervisor's default became 8 (supervise.sh). These are recorded observations from the author's sessions, not a measured capacity curve.
The engineering point is where the limit lives. CPU and memory on the author's machine were not the constraint; the shared account was. A WIP limit set from local resources would have been wrong in both directions. In an enterprise factory the same pattern appears as a shared API quota, a license pool, or a review team: the slowest shared resource sets the useful concurrency, and adding workers past it adds only queue time and wasted attempts. Looping over a large backlog and Scaling safely treat this as a design input.
First-run yield, read with its caveats91% by the factory's own validators
Observed (local run.log): of the problems assigned to productive tasks, 91.4% of visualization codes and 91.9% of solution codes were verified when the task ran; for first-attempt tasks, 91.1% and 88.2%. A worker-hour produced about 34 visualization codes or 21 solution codes while running, arithmetic from the median task time that excludes pauses.
Three caveats keep these figures honest. Acceptance means the factory's own validators, which for solutions do not check expected outputs (the 86 later defects passed them). The figures exclude the zero-minute tasks of the retry storm, which would otherwise dominate any average. And they describe one author's account, one model setting and one corpus; they are not a benchmark and do not transfer to another line. Their use is internal: a yield that drops between batches is a signal to look, whatever its absolute level.
In one breath#
WatchAlgo's story holds up when every claim is put in one of three columns: what the repository shows, what the author reports, and what this site reconstructs. 3,246 is a correct count of problems described in three flavors, of which 2,380 are complete on the author's machine and fewer are live. The "30+ hours" were a supervisor and a queue running short worker processes over a 62-hour span until account usage limits stopped them, and the sharpest lesson inside them is the retry storm: an unclassified failure turned a pause into 11,503 failed tasks while verification kept the content intact.
The case study now has honest numbers and a map of its phases. What it does not yet have is a clear picture of the thing being produced. A problem as a production unit opens one unit folder and reads its three files field by field, because no station can be named, checked or trusted until the contract of what it must deliver is written down.