At 22:14 on 2 September 2026 a commit landed in WatchAlgo with an unusual message. Workers writing visualization tracers had been cut off mid-task when the account hit a usage limit, and nine half-written tracer files were left on disk. The commit removed them, and its message ends with four words: "the generator gate held".
Observed in the repository (zenalgo-frontend, commit d901dd6b; docs/evidence/git-history.md, section 3): the nine stubs never became visualizations. A tracer is a program a worker writes; the visualization file is written by code only after the tracer passes its checks (docs/evidence/station-contracts.md, S2b). A worker killed halfway left a broken program behind, and the line treated it as nothing, which is what it was.
That night the factory lost the work in flight and nothing else. No half-written file reached the product, and no finished file was overwritten. A long run is mostly a question of making that the normal outcome of every interruption.
The learner's question: when a run lasts longer than any single worker, session or process, and something stops it in the middle, how does the factory lose nothing that was finished, redo nothing that was verified, and write nothing twice?
Five ways a long run ends early#
Plain English. A long run will be interrupted. The only question is whether, afterwards, you can tell exactly what was finished, and continue from there without doing anything twice.
Beginner definition: a run is resumable when its progress lives in durable records and immutable artifacts outside any worker, so a new process can read them and continue.
The causes differ in who notices, how much they take with them, and what the right response is. The table sorts the five this chapter treats.
| Cause | Scope | Who notices | Lost without a checkpoint | Right response |
|---|---|---|---|---|
| Context growth | one worker | the worker, often silently | the worker's grip on early instructions | small tasks, fresh workers, checkpoint before compaction |
| Session or usage limit | every worker on an account | the provider, all at once | all work in flight | pause everyone; stop cleanly; resume later |
| Process crash, closed terminal | one process | nobody, until a restart | everything only in memory | durable "running" rows; resume from the record |
| Tool or terminal failure | one call | the worker or the station | the attempt | classify, then retry or rework (see Assembly stations) |
| Token budget exhaustion | one unit or run | the factory's own guard | nothing, if the guard checks before the call | stop with a named status; keep the checkpoint |
The context window, the text a model can take into account at once, was introduced in AI enters the factory. As a worker reads files, receives tool results and accumulates conversation, its context grows, and anything pushed out of view stops influencing its work. A session limit is different: a cap set by the provider or the account on how much one session, or one account, may use in a period. The first is a property of one worker's task; the second is shared by every worker on the account.
Before AI. A database writes every change to a log before it changes the data, so after a crash it replays the log and knows exactly which transactions committed. A nightly batch job that records the last committed chunk can restart from there instead of from row one. Make rebuilds only the targets older than their inputs. Every long-running system before AI already answered this chapter's question the same way: progress lives in a durable record, written before the work is considered done.
Checkpoint, resume, idempotency#
A checkpoint is a durable record of what a run has finished, written after each completed unit and before any risky step. The analogy is a save point in a long game. Its consequence is that an interruption costs at most the work since the last checkpoint. Its trade-off is a write per unit, and a format that must stay readable by the next version of the code.
To resume is to continue a run from its checkpoint: read what is finished, verify that each finished artifact is still intact, skip it, and do the rest. A resume that trusts the checkpoint without checking the artifacts can skip work that a crash or a person has damaged.
Idempotency means that doing an operation twice has the same effect as doing it once. Writing a file named by the hash of its content is idempotent: the second write finds the file already there and changes nothing. Appending a row "attempt succeeded" is not, unless the row has a key that refuses the duplicate. Idempotency matters because a resumed run cannot always know whether the last write before the crash happened.
A persistent artifact is a station's output stored outside the worker's memory, immutable and addressed by its hash, so that a later process can find it, check it and reuse it. A transcript is not one; a file in a content-addressed store is.
The smallest primitive: a checkpoint after every item#
Example B (examples/b-algorithm-factory/) builds one toy unit, a lower-bound function, through three items: an oracle, a solution and a visualization. After each completed item it writes a checkpoint. Its writing function, quoted verbatim from factory.py : write_atomic:
def write_atomic(path, text):
tmp = Path(str(path) + ".tmp")
tmp.write_text(text, encoding="utf-8")
os.replace(tmp, path)
The temporary file plus rename means a crash during the write leaves either the old checkpoint or the new one, never half of one. The resume decision is three lines inside factory.py : run, quoted verbatim:
for item_id in order:
rec = cp["items"][item_id]
if rec["status"] == "done" and artifact_intact(out, rec):
emit("[skip] %s %s: done in checkpoint, artifact sha256 matches" % (rec["station_id"], item_id))
continue
An item is skipped only when the checkpoint says it is done and artifact_intact finds the artifact on disk with the sha256 the checkpoint recorded. Run in a scratch copy on 2026-11-20, a simulated crash right after the solution item, then a resume:
$ /opt/homebrew/bin/python3 run.py --stop-after solution --no-cross-lang
start FR-0002 spec SPEC-ALGO-001 pattern PAT-ALGO-UNIT@1.3.0 spec_hash 44d76f9399ab budget 2000 units
[run ] S1-build-oracle oracle: 30 cases (10 fixtures incl. edge cases, 20 seeded random), sha256 45d4895d56b7
[try ] S2-generate-solution solution attempt 1: stand-in says "done, all tests pass"; oracle: 21/30 passed -> rejected
[try ] S2-generate-solution solution attempt 2: stand-in says "done, all tests pass"; oracle: 30/30 passed -> accepted
[kill] simulated crash after solution (checkpoint written, process gone)
checkpoint left at out/checkpoint.json; rerun with --resume
$ /opt/homebrew/bin/python3 run.py --resume --no-cross-lang
resume FR-0002 from checkpoint: done = oracle, solution
[skip] S1-build-oracle oracle: done in checkpoint, artifact sha256 matches
[skip] S2-generate-solution solution: done in checkpoint, artifact sha256 matches
[run ] S3-trace-visualization visualization: 10 traces, 28 states, invariants hold, sha256 2bf5ff26c1a1
end FR-0002 status=succeeded cost=800 units generator_calls=0
generator calls in this process: 0
S1-build-oracle oracle done attempts=0
S2-generate-solution solution done attempts=2
S3-trace-visualization visualization done attempts=0
The resumed process called the generator zero times. The 800 units in the last line are the run's total, carried in the checkpoint; nothing new was spent. Without the checkpoint, a restart would have rebuilt the oracle and called the generator twice again, another 800 illustrative units for work already accepted. The agent here is a scripted stand-in, so these results show how the factory mechanics work, not how good any model is.
The same example has a budget guard, which is the fifth cause of the table handled well. It checks the budget before each call, so it stops with a named status instead of overspending:
$ /opt/homebrew/bin/python3 run.py --budget 500 --no-cross-lang
start FR-0002 spec SPEC-ALGO-001 pattern PAT-ALGO-UNIT@1.3.0 spec_hash 44d76f9399ab budget 500 units
[run ] S1-build-oracle oracle: 30 cases (10 fixtures incl. edge cases, 20 seeded random), sha256 45d4895d56b7
[try ] S2-generate-solution solution attempt 1: stand-in says "done, all tests pass"; oracle: 21/30 passed -> rejected
[stop] S2-generate-solution solution: attempt 2 would cost 400 units; spent 400 of 500: budget_exhausted
end FR-0002 status=budget_exhausted cost=400 units (checkpoint kept)
budget_exhausted is not succeeded and not failed: it is a resumable stop with the checkpoint kept. Units are illustrative, not tokens or money.
A durable record of a whole run#
A checkpoint file serves one run of one unit. A factory running many units, stations and attempts needs a record it can query. Example D (examples/d-supervisor/) keeps that record in SQLite and schedules from it. The schema, quoted verbatim from state.py : SCHEMA (it ran in SQLite during the example's run; the build does not check SQL):
CREATE TABLE IF NOT EXISTS factory_runs (
run_id TEXT PRIMARY KEY,
pattern_version TEXT NOT NULL,
specification_hash TEXT NOT NULL,
spec_id TEXT NOT NULL,
status TEXT NOT NULL,
cost_units_total INTEGER NOT NULL DEFAULT 0,
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS tasks (
run_id TEXT NOT NULL,
task_id TEXT NOT NULL,
station_id TEXT NOT NULL,
depends_on TEXT NOT NULL,
status TEXT NOT NULL,
PRIMARY KEY (run_id, task_id),
FOREIGN KEY (run_id) REFERENCES factory_runs(run_id)
);
CREATE TABLE IF NOT EXISTS station_attempts (
run_id TEXT NOT NULL,
task_id TEXT NOT NULL,
station_id TEXT NOT NULL,
attempt INTEGER NOT NULL,
status TEXT NOT NULL,
input_hash TEXT NOT NULL,
artifact_ref TEXT,
test_evidence_ref TEXT,
error_class TEXT,
cost_units INTEGER NOT NULL DEFAULT 0,
started_at TEXT NOT NULL,
finished_at TEXT,
PRIMARY KEY (run_id, station_id, attempt),
FOREIGN KEY (run_id) REFERENCES factory_runs(run_id)
);
CREATE TABLE IF NOT EXISTS artifacts (
uri TEXT PRIMARY KEY,
sha256 TEXT NOT NULL,
bytes INTEGER NOT NULL,
run_id TEXT NOT NULL,
station_id TEXT NOT NULL,
created_at TEXT NOT NULL
);
A durable record is this kind of store: one row per run, per task and per attempt, written before and after each piece of work, holding hashes and pointers to immutable artifacts. factory_runs binds a run to its pattern version and specification hash, so a resume with another spec can be refused. tasks holds the station graph and each task's status. station_attempts holds every attempt, its input_hash, its artifact and its failure class; Assembly stations reads those columns as a station contract at run time. artifacts maps each artifact URI to the sha256 of its bytes.
The idempotency logic, verbatim#
Three pieces of code make a crash at any point safe. They are quoted verbatim from Example D; the two methods of Supervisor are shown with their class indentation removed.
The first is the artifact store, artifacts.py : ArtifactStore.put. A file is named by the sha256 of its bytes and written only if it does not exist yet:
def put(self, data: bytes) -> tuple:
"""Store bytes; return (digest, wrote). Safe to call from worker threads."""
digest = sha256(data)
p = self.path(digest)
with self._lock:
if os.path.exists(p):
return digest, False
tmp = p + ".tmp"
with open(tmp, "wb") as f:
f.write(data)
os.replace(tmp, p) # atomic rename: a crash never leaves half a file under the real name
self.writes += 1
return digest, True
The second decides whether a station needs to run at all, scheduler.py : Supervisor.input_hash and Supervisor.find_reusable:
def input_hash(self, run_id, spec, t):
"""Hash of everything the station reads: pattern, spec, station and dependency digests."""
inputs = {}
for d in t.deps:
ref = self.conn.execute(
"SELECT artifact_ref FROM station_attempts WHERE run_id=? AND station_id=? "
"AND status IN ('succeeded','reused') ORDER BY attempt DESC LIMIT 1", (run_id, d)).fetchone()
inputs[d] = self.conn.execute("SELECT sha256 FROM artifacts WHERE uri=?",
(ref["artifact_ref"],)).fetchone()["sha256"]
material = {"pattern": f"{PATTERN_ID} {PATTERN_VERSION}", "spec_hash": spec_hash(spec),
"station_id": t.station_id, "inputs": inputs}
return hashlib.sha256(canonical(material)).hexdigest(), inputs
def find_reusable(self, station_id, ih):
return self.conn.execute(
"SELECT artifact_ref, test_evidence_ref FROM station_attempts WHERE station_id=? AND input_hash=? "
"AND status='succeeded' ORDER BY run_id, attempt LIMIT 1", (station_id, ih)).fetchone()
The third turns attempts left running by a dead process into abandoned, scheduler.py : Supervisor.recover_interrupted:
def recover_interrupted(self, run_id: str) -> None:
"""Attempts left 'running' by a crash become 'abandoned'. They do not count as failures."""
rows = self.conn.execute(
"SELECT station_id, attempt FROM station_attempts WHERE run_id=? AND status='running' "
"ORDER BY station_id, attempt", (run_id,)).fetchall()
for r in rows:
self.conn.execute(
"UPDATE station_attempts SET status='abandoned', error_class='interrupted', finished_at=? "
"WHERE run_id=? AND station_id=? AND attempt=?",
(self.clock.now(), run_id, r["station_id"], r["attempt"]))
self.log.append(f"{run_id} {r['station_id']:24s} attempt {r['attempt']} abandoned (interrupted)")
self.conn.commit()
The order of operations ties them together, and Supervisor.run_round enforces it: an attempt row with status running is inserted and committed before any worker starts (the comment in the code reads "'running' rows are durable BEFORE any work starts"); the worker computes the artifact and calls put; only then is the row updated to succeeded with the artifact's URI. A crash before put leaves a running row and no file. A crash after put leaves a running row and a file. Either way, resume marks the row abandoned, runs the station again, and put either writes the file for the first time or finds it and writes nothing.
The executable demonstration: crash, resume, reuse#
Output of /opt/homebrew/bin/python3 run.py in a scratch copy of Example D on 2026-11-20, the crash run, its resume, and the rerun of an earlier spec:
== FR-0005 SPEC-ALGO-002: process dies after S2-generate-solution writes its artifact
run status: CRASHED at S2-generate-solution; physical artifact writes by this process: 3
station att status cost artifact / error
S1-validate-spec 1 succeeded 0 artifact://FR-0005/S1-validate-spec/6b1ff6027d36
S2-generate-oracle 1 succeeded 30 artifact://FR-0005/S2-generate-oracle/17a636457aeb
S2-generate-solution 1 running 0
tasks: T1=succeeded T2=running T3=succeeded T4=pending T5=pending T6=pending
== FR-0005 SPEC-ALGO-002: resume run (new supervisor, same database)
run status: succeeded; physical artifact writes by this process: 3
station att status cost artifact / error
S1-validate-spec 1 succeeded 0 artifact://FR-0005/S1-validate-spec/6b1ff6027d36
S2-generate-oracle 1 succeeded 30 artifact://FR-0005/S2-generate-oracle/17a636457aeb
S2-generate-solution 1 abandoned 0 interrupted
S2-generate-solution 2 succeeded 40 artifact://FR-0005/S2-generate-solution/b03036903131
S3-build-visualization 1 succeeded 25 artifact://FR-0005/S3-build-visualization/f176f95de42b
S3-run-tests 1 succeeded 0 artifact://FR-0005/S3-run-tests/192abe0c57d5
S4-package-manifest 1 succeeded 0 artifact://FR-0005/S4-package-manifest/0212923625ed
tasks: T1=succeeded T2=succeeded T3=succeeded T4=succeeded T5=succeeded T6=succeeded
== FR-0006 SPEC-ALGO-001: same spec as FR-0004: idempotent rerun
run status: succeeded; physical artifact writes by this process: 0
station att status cost artifact / error
S1-validate-spec 1 reused 0 artifact://FR-0004/S1-validate-spec/171a648da216
S2-generate-oracle 1 reused 0 artifact://FR-0004/S2-generate-oracle/6981bd4abb75
S2-generate-solution 1 reused 0 artifact://FR-0004/S2-generate-solution/4545fa833c8a
S3-build-visualization 1 reused 0 artifact://FR-0004/S3-build-visualization/6e1f6f8ac7ea
S3-run-tests 1 reused 0 artifact://FR-0004/S3-run-tests/94b662066656
S4-package-manifest 1 reused 0 artifact://FR-0004/S4-package-manifest/122f893ee740
tasks: T1=reused T2=reused T3=reused T4=reused T5=reused T6=reused
== checks
FR-0005 writes: 3 before crash + 3 after resume = 6 (6 stations, 6 distinct artifacts)
FR-0005 succeeded attempts per station: [1] (never a duplicate)
FR-0006 station executions: 0, writes: 0
The event log of the resume (run.py --verbose) shows the one line that proves idempotency: the solution station ran again and found its file already present.
FR-0005 S2-generate-solution CRASH after artifact write, before record
FR-0005 resumed status was running
FR-0005 S2-generate-solution attempt 1 abandoned (interrupted)
FR-0005 S1-validate-spec skip (already succeeded)
FR-0005 S2-generate-oracle skip (already succeeded)
FR-0005 S2-generate-solution attempt 2 succeeded artifact://FR-0005/S2-generate-solution/b03036903131 (file already present, no write)
FR-0005 S3-build-visualization attempt 1 succeeded artifact://FR-0005/S3-build-visualization/f176f95de42b (wrote)
A record is only durable if a different program can read it. This page's writer opened the database the run left behind with a short script (run beside the copy, after run.py) and asked it the two questions a person asks after an interruption:
"""read_record.py: open Example D's SQLite record after run.py and answer two questions from it alone."""
import sqlite3
conn = sqlite3.connect("out/factory.db")
print("What happened in FR-0005?")
for row in conn.execute("SELECT station_id, attempt, status, error_class, input_hash, artifact_ref "
"FROM station_attempts WHERE run_id='FR-0005' ORDER BY started_at, station_id, attempt"):
sid, att, status, err, ih, ref = row
print(f" {sid:23} {att} {status:9} {err or '':11} in={ih[:12]} {ref or ''}")
print("What is left to do in each run?")
for run_id, status in conn.execute("SELECT run_id, status FROM factory_runs ORDER BY run_id"):
left = [r[0] for r in conn.execute("SELECT station_id FROM tasks WHERE run_id=? AND status NOT IN "
"('succeeded','reused') ORDER BY task_id", (run_id,))]
print(f" {run_id} {status:10} open tasks: {', '.join(left) or 'none'}")
What happened in FR-0005?
S1-validate-spec 1 succeeded in=7e9515d900fd artifact://FR-0005/S1-validate-spec/6b1ff6027d36
S2-generate-oracle 1 succeeded in=6bf12a168880 artifact://FR-0005/S2-generate-oracle/17a636457aeb
S2-generate-solution 1 abandoned interrupted in=5401f2a0768d
S2-generate-solution 2 succeeded in=5401f2a0768d artifact://FR-0005/S2-generate-solution/b03036903131
S3-build-visualization 1 succeeded in=2761aa60acaf artifact://FR-0005/S3-build-visualization/f176f95de42b
S3-run-tests 1 succeeded in=09f211241a8e artifact://FR-0005/S3-run-tests/192abe0c57d5
S4-package-manifest 1 succeeded in=3fa50402e7cf artifact://FR-0005/S4-package-manifest/0212923625ed
What is left to do in each run?
FR-0004 succeeded open tasks: none
FR-0005 succeeded open tasks: none
FR-0006 succeeded open tasks: none
FR-0007 incomplete open tasks: S3-build-visualization, S4-package-manifest
Three things are visible in the rows. The abandoned attempt and the successful one carry the same input hash, 5401f2a0768d: the same input ran twice and produced one artifact. FR-0007, the run whose visualization station failed three times and was escalated, is incomplete with two open tasks, never succeeded. And everything above came from the record alone, with no process running. sh check_repro.sh in the same copy printed byte-identical: stdout, out/state_dump.json, out/events.log, out/artifacts/*: two full runs, crash and resume included, produce the same record. This is the site's demonstration that durable run state, resumability and independently validated artifacts work as described; the stations are scripted stand-ins, so it shows the supervisor's mechanics, not any model's quality.
Diagram: One station attempt through a crash · a running row first, the artifact by hash, the final row last
Top lane, the normal path: the supervisor commits a running row before any work starts; the worker computes the artifact; the content-addressed store writes it under its hash only if the file does not exist; then the row becomes succeeded. A crash anywhere in that path leaves a running row. Bottom lane, the resume: a new supervisor reads the same database, marks the running row abandoned, which costs no attempt, skips stations already succeeded, and runs the interrupted station again with the same input hash. The store finds the file already present and writes nothing. In FR-0005 that gives 3 writes before the crash, 3 after, and 6 distinct artifacts.
- Normal path
- running row: The attempt is recorded as running before the worker starts, so a crash leaves an honest trace.
- worker computes: The station's work; in Example D a scripted stand-in, so the bytes depend only on the input.
- store by hash: The file is named by the sha256 of its bytes and written atomically, once.
- succeeded row: Only now does the record point at the artifact.
- Crash and resume
- crash: The process dies; the row still says running.
- mark abandoned: Interrupted, not failed: no attempt is used up.
- skip finished: Stations with a succeeded row are not run again.
- rerun, no write: Same input hash, same bytes; the store writes nothing the second time.
The diagram follows one station attempt through a crash. Work starts only after a running row is committed. The artifact goes into a content-addressed store, and only then is the row marked succeeded. A crash anywhere in between leaves a running row. On resume, that row becomes abandoned, which costs no attempt; finished stations are skipped; the interrupted station runs again with the same input hash; and the store, finding the file already present, writes nothing.
Compaction, and what it must never replace#
Compaction is replacing the earlier part of a long conversation with a shorter summary, so a worker that is running out of context room can continue. The analogy is minutes that replace the recording of a long meeting: much shorter, mostly right, and silent about whatever the note-taker thought unimportant. The operational consequence is that compaction is lossy. A constraint stated once at the start, an exception agreed halfway through, or the exact list of files already finished can disappear from the summary, and the worker will not know it has lost them. The trade-off is continuity of one worker's session against the risk of acting on a summary.
This site's recommendation: never let a compacted conversation be the record a run resumes from. Before compaction, write a checkpoint: the finished units, their hashes, and the next step, into the durable record. After compaction, the worker reads the record again. Product (Claude Code, as documented on 2026-11-20): the hooks reference lists a PreCompact event among its hook events, which is a natural place to run a script that writes such a checkpoint. The documentation read here describes the event's existence, not a recommended checkpoint script; the script is this site's suggestion.
The cheaper answer, where the work allows it, is to avoid long conversations altogether: give each worker one small task and start a fresh worker for the next. Context then never grows past one task, and nothing needs compacting. That is the design WatchAlgo used, as the next section shows.
In a coding agent. A headless coding agent can resume a conversation. Product (Claude Code, as documented on 2026-11-20):--continuecontinues the most recent conversation and--resume <session-id>a specific one. Product (Claude Agent SDK, as documented on 2026-11-20): sessions persist the conversation, not the filesystem. Product (Codex, as documented on 2026-11-20):codex exec resume --lastorcodex exec resume <SESSION_ID>resumes non-interactive work. Each of these restores what the agent was told and said. None of them is the factory's record of which units are accepted: that lives in the factory's own store, checked by the factory's own tests, and a person with merge rights still reviews what the agent produced.
A durable record is not long-term agent memory#
The two are easy to confuse because both survive the end of a process. They answer different questions.
Example D's database answers "what happened in run FR-0005, and what is left to do". It is keyed by run, station and attempt, holds hashes and pointers to immutable artifacts, and is read by the scheduler with exact queries. Nothing in it goes into a model's prompt, gets summarized, or is forgotten. Its job ends when the run ends; after that it is an audit trail.
Long-term agent memory answers "what should an agent know next time, across many runs". It is selected, summarized and retrieved by relevance; it can be wrong or stale; it needs its own rules for what to keep, who may write it and when to forget. Mixing the two goes wrong in both directions. A resume decision taken from a fuzzy memory can redo finished work or skip unfinished work. A memory store fed every attempt row floods the agent with noise.
Diagram: Durable execution record against long-term agent memory · two stores, two questions
Left, the durable record: written by the scheduler before and after each attempt, keyed exactly by run, station and attempt, read by code with exact queries, complete by design. It answers what happened and what is left, and it decides what runs next. Right, long-term memory: written by or for an agent, selected and summarized, retrieved by relevance, allowed to be incomplete or stale. It answers what an agent should know next time. Memory may shape how an agent works; it must never decide whether a unit counts as done.
- Durable record
- written by the scheduler: Every run, task and attempt, with hashes and artifact pointers.
- exact keys: Queried exactly; nothing is summarized or forgotten while the run lives.
- decides what runs: The scheduler acts only on the record and the artifacts it points to.
- Agent memory
- written for an agent: Chosen notes meant to enter a future context.
- retrieved by relevance: Summarized and ranked; can be wrong, outdated or missing pieces.
- shapes how work is done: Guides behavior; it is not evidence that a unit is accepted.
The diagram sets the two side by side. The record is written by the scheduler, keyed exactly, read by code, and complete; it decides what runs next. Memory is written by or for an agent, selected and summarized, retrieved by relevance, and allowed to be incomplete; it shapes how an agent works, never whether a unit counts as done.
What this is not. This page covers checkpoints, resume and idempotent writes inside a factory run. Durable memory as a discipline, and how it differs from saved execution state, belongs to Memory Engineering. Retries, checkpoints and recovery as a reliability discipline belong to Agent Reliability Engineering. The working set a model sees in its window belongs to Context Engineering. The catalog of failures and their recoveries is Failure and recovery scenarios.
Try it#
The recovery lab runs a long job over many units with a context budget. Context grows with each unit; at the limit the session ends. Switch between three modes, no checkpoint, a checkpoint per unit, and a checkpoint plus compaction, and see units persisted, work lost, the resume point and duplicate writes. The second embed starts in the no-checkpoint mode.
docs/LABS.md has no plain table for this lab yet, so the table below was written for this page from the real outputs of Examples B and D shown above. The lab's own numbers are simulated, illustrative; the rows below say which example each fact comes from.
| Mode | What survives an interruption | Work redone on resume | Duplicate writes | Source |
|---|---|---|---|---|
| No checkpoint | only what the restarted run rebuilds | everything: Example B's fresh run calls the generator twice, 800 illustrative units | possible for any non-idempotent write | Example B, fresh run output |
| Checkpoint per unit | every item recorded as done, with its sha256 | only unfinished items: Example B's resume makes 0 generator calls | 0 in Example D: 3 writes before the crash + 3 after = 6 artifacts | Examples B and D, crash and resume output |
| Checkpoint + compaction | as above, plus a shorter session that can continue | as above, if the record and not the summary decides | as above | the lab only; neither example compacts, so this row is simulated |
WatchAlgo in practice#
Observed in the repository (docs/EVIDENCE.md, matrix rows 11 and 13, section 8): the long September runs were stopped by account usage limits, not by context windows. The last lines of 10,657 worker logs are usage messages: weekly limit (9,674), monthly spend limit (976), session limit (7). Each task started a fresh claude -p process working on 5 or 8 problems, so context growth was bounded by the task size, and nothing in the logs shows compaction inside a worker. The WatchAlgo story reads the author's "30+ hours" against these logs; this page uses only the mechanisms.
Observed (scripts/viz/pipeline.py, commit 971863f2; scripts/viz/supervise.sh, commit 017ed6b9): the queue lived on disk under a file lock and was written atomically; it could be rebuilt from the files on disk, so the disk was the source of truth. When the driver started, tasks left running by a dead driver were verified from disk, finished problems were credited, and the rest re-queued without spending an attempt. The supervisor restarted the driver when it was gone and committed verified content and the queue every 1,800 seconds. The local run.log records 19 orphan recoveries and 18 supervisor restarts. Checkpoint commits run every 30 minutes from 5e6c6040 to 1b183564. Observed (.claude/SOLUTION_ASSEMBLY_LINE.md, .ai-context/AUTONOMOUS_PROGRESS.md): earlier, session-level resume was done by instruction: a "Session Recovery" section and an "authoritative resume marker" told the next session where to start.
Observed (local cron, tracked file .ai-context/cron-watchdog.log): the March watchdog still runs hourly. On 2026-11-20 at 10:00 it failed to start the claude binary, which no longer exists at its old path, and logged "Resume triggered" anyway: a partial failure recorded as success. The design lesson for resume is that a launcher must record its result, not its intent. Reported by the author: the process ran until token limits required intervention; read against the logs, the intervention a usage limit requires is waiting or raising the limit, and the resumable queue is what made waiting cheap. Teaching reconstruction: Example D's per-attempt record with input hashes is this site's design; WatchAlgo's record was a task queue and git commits, with verification from disk standing in for input hashes.
Enterprise translation#
In the insurance factory of Part III, a factory run that builds the intake service may wait days at a security review or a release approval. No process stays alive that long, so the run must be a record, not a process. Product (Temporal, as documented on 2026-11-20): progress is recorded in an Event History; after a crash a new worker replays the workflow code against that history to rebuild its state; workflow code must be deterministic, and non-deterministic work, including model calls, belongs in Activities, which Temporal retries with timeouts. Product (LangGraph, as documented on 2026-11-20): retries re-run a whole node, so side effects such as payments or emails can repeat, and the documentation advises deriving idempotency keys from durable state. Both say, in their own terms, what Example D shows: replay is safe only when every effect is idempotent.
The same rule appears inside the product the factory builds. The insurance service's write tool, record_recommendation(submission_id, recommendation, idempotency_key), takes an idempotency key so that a retried call cannot record a recommendation twice; that is the underwriting workflow operating, which Insurance factory, part 2 keeps apart from the factory building it. This site's recommendation: checkpoint before every step that touches the outside world or waits for a person; key every external write; and treat a run as finished only when its record says every task is accepted, never because a process exited cleanly. What dominates the critical path here is the waiting at human gates, and a durable record is what makes waiting free.
Failure test#
The failure: a station writes its artifact, and the process dies before the record is updated. Without care, the resumed run either thinks the station never ran and writes a second artifact, or thinks it finished and skips a file nobody checked.
How the factory catches it: the running row committed before the work leaves an honest trace; resume turns it into abandoned, which costs no attempt; the station runs again with the same input hash; the content-addressed store finds the file and writes nothing. FR-0005 shows the result: 3 writes before the crash, 3 after, 6 distinct artifacts, one success per station.
What to run or toggle: in a scratch copy of examples/d-supervisor/, run /opt/homebrew/bin/python3 run.py --verbose and find the line "file already present, no write"; then run sh check_repro.sh. In a copy of Example B, run run.py --stop-after solution and then run.py --resume. In the lab, switch to "no checkpoint" and compare the work lost with the per-unit mode.
Where to go deeper#
- Evidence of correctness (next): what makes an accepted artifact worth keeping.
- Looping over a large backlog: queues, WIP limits and the failed-case queue.
- Assembly stations: the station contract whose results the record stores.
- Failure and recovery scenarios: crashes, partial artifacts and escalation across the factory.
- Memory Engineering: durable memory, and how it differs from saved execution state.
- Agent Reliability Engineering: retries, checkpoints and recovery as a discipline.
- Context Engineering: what fits in a model's window, and what to leave out.
Checklist for Monday#
- Write a durable "running" record before any work starts, and the final record only after the artifact exists.
- Store artifacts by content hash and write them atomically (temporary file, then rename).
- Compute an input hash per station; reuse a recorded success with the same hash.
- On resume, verify each finished artifact's hash before skipping it.
- Count interruptions separately from failures; an interrupted attempt should not use up a retry.
- Checkpoint before compaction, before any external write, and before any step that waits for a person.
- Give every external write an idempotency key derived from durable state.
- Report a run as succeeded only when every task is accepted; a launcher records results, not intentions.
Exercises#
- A station crashed after writing its artifact. Why does Example D not count that attempt as a failure?
Answer 1Interruption is not a defect
Nothing showed the work was wrong; the process died. Marking it abandoned keeps the three attempts for real defects.
- Why does
FR-0006execute zero stations?
Answer 2Same input hashes
Its spec equals FR-0004's, so every station's input hash matches a recorded success and the artifact is reused.
- A worker's conversation was compacted and it now says six units are finished. What should the scheduler believe?
Answer 3The record, not the summary
The durable record and the artifact hashes. A summary is a self-report written under context pressure.
Advanced perspective#
Exactly once is a property you build, not one you getAt-least-once execution plus idempotent effects
A process can always die between doing something and recording that it did it. No ordering of two writes removes that gap; it only moves it. So distributed systems, workflow engines and Example D all settle for at-least-once execution and make every effect idempotent, which together behave like exactly once from the outside. Example D's version: execution may repeat (the solution station ran twice in FR-0005), but the effect, a file named by its hash, happens once.
The hard cases are effects that are not naturally idempotent: an email, a payment, a ticket in another team's tracker, a pull request. For those, the factory needs a key the receiving system honors, or a record written before the call that the resume checks, or a person. A factory that cannot make an effect idempotent should put a human gate in front of it, and the governance chapter should say so.
Checkpoint granularity: per unit, per step, per time windowWhat an interruption costs against what a checkpoint costs
WatchAlgo checkpointed at two levels at once. Per unit: every verified problem was credited in the queue, and its files were on disk. Per time window: the supervisor committed to git every 30 minutes. The first decides what a resume redoes; the second decides what survives a lost machine or a bad edit. Example B checkpoints per item; Example D per attempt.
Finer checkpoints cost writes and a schema to maintain, and coarser ones cost redone work. The useful rule is to checkpoint at the boundary of anything expensive or irreversible: after each accepted unit, before a model call that costs real money, before an external write, and before compaction. A time-window checkpoint is a good second layer but a poor only layer, because a crash a minute before the next commit loses everything in between.
Resume must re-verify, not trustA checkpoint is a claim about the disk
A checkpoint says "item X was finished, and its artifact had hash H". Between the checkpoint and the resume, a person may have edited the file, a disk may have failed, or a half-finished operation may have left something behind. Example B handles this with artifact_intact: it skips an item only if the file exists and its sha256 still matches, and its tests show a tampered artifact being rebuilt. WatchAlgo's driver did the same for orphaned tasks: it verified each problem from disk and credited only what passed.
The principle is the site's refrain applied to the factory's own records: a checkpoint is a self-report from an earlier process. It is a very good hint, and resuming from it is the point, but the artifact on disk is the evidence. When the two disagree, the artifact wins and the item is redone.
In one breath#
A long run will be interrupted, by a full context, a usage limit, a crash, a failing tool or an exhausted budget, so its progress must live outside any worker: a durable record written before and after each piece of work, and artifacts stored by content hash. Resume reads the record, re-verifies each finished artifact, skips it, and reruns the rest; idempotent writes make a repeated step harmless, which Example D demonstrates with a crash between an artifact write and its record, and a resume with no duplicate and no loss. Compaction and resumed conversations help one worker continue, but the record, not a summary or a memory, decides what counts as done.
A run that resumes perfectly can still preserve a wrong answer perfectly. The next chapter, Evidence of correctness, asks what makes an accepted artifact worth keeping: properties, randomized and differential tests, metamorphic relations, trace checks, regression cases, and where a person still has to look.