All chapters
  1. Start hereA software factory, from first principles
  2. Primitive · 1A factory, before AI: input, transformation, check, output
  3. Primitive · 2From repetition to a pattern: what stays, what varies, what to leave alone
  4. Primitive · 3The specification as an executable contract
  5. Primitive · 4Build the smallest factory: four stations, two specs, one manifest
  6. Primitive · 5Assembly stations: typed contracts, owners and failure handling
  7. Primitive · 6AI enters the factory: what a model adds, and what stays deterministic
  8. WatchAlgo · 7The WatchAlgo production story: what the repository shows
  9. WatchAlgo · 8A problem as a production unit: the files one unit must contain
  10. WatchAlgo · 9Station by station: description, solution, implementations, visualization
  11. WatchAlgo · 10The visualizer factory: from algorithm state to renderable steps
  12. WatchAlgo · 11Three flavors and three languages: two different contracts
  13. WatchAlgo · 12Looping over a large backlog: scheduling, limits and a failed-case queue
  14. WatchAlgo · 13Long-running agent execution: context, checkpoints and token limits
  15. WatchAlgo · 14Evidence of correctness: tests the producer cannot talk its way past
  16. WatchAlgo · 15The factory learning loop: defects change the pattern, not only the unit
  17. Enterprise · 16Classify the enterprise demand before choosing a pattern
  18. Enterprise · 17Reference enterprise patterns and their variation points
  19. Enterprise · 18Platform foundations: golden paths, templates, registries and catalogs
  20. Enterprise · 19Pattern versus configuration versus business logic
  21. Enterprise · 20Insurance factory, part 1: building the intake contract
  22. Enterprise · 21Insurance factory, part 2: building the underwriting workflow
  23. Enterprise · 22Insurance factory, part 3: trust and observability built in
  24. Enterprise · 23Data and integration engineering: contracts that survive change
  25. Enterprise · 24The CI/CD and deployment factory: from pull request to production
  26. Enterprise · 25Enterprise governance: owners, approved patterns and provenance
  27. Enterprise · 26Scaling safely: many teams, many agents, one set of boundaries
  28. Operations · 27Factory quality engineering: measuring accepted output, not activity
  29. Operations · 28Failure and recovery scenarios: what breaks a factory, and what restores it
  30. Operations · 29The landscape: which tool for which job, read from its own documentation
  31. Operations · 30Design an enterprise software factory: the blueprint and the rollout
  32. ReferenceSources and verification
  33. ReferenceGlossary
  34. AboutAbout this site
WatchAlgo · 13

Long-running agent execution: context, checkpoints and token limits

Context grows, sessions end and terminals fail. Checkpoints before risky work, resumable runs and persistent artifacts make sure an interruption loses minutes, not hours.

You will name
Five ways a long run ends early, from a full context to an account usage limit, and what each one destroys.
You will run
A crash between an artifact write and its record, then a resume with no duplicate write and no lost work (Example D).
You will separate
A durable execution record from long-term agent memory, and a resumed conversation from a resumed run.

At 22:14 on 2 September 2026 a commit landed in WatchAlgo with an unusual message. Workers writing visualization tracers had been cut off mid-task when the account hit a usage limit, and nine half-written tracer files were left on disk. The commit removed them, and its message ends with four words: "the generator gate held".

Observed in the repository (zenalgo-frontend, commit d901dd6b; docs/evidence/git-history.md, section 3): the nine stubs never became visualizations. A tracer is a program a worker writes; the visualization file is written by code only after the tracer passes its checks (docs/evidence/station-contracts.md, S2b). A worker killed halfway left a broken program behind, and the line treated it as nothing, which is what it was.

That night the factory lost the work in flight and nothing else. No half-written file reached the product, and no finished file was overwritten. A long run is mostly a question of making that the normal outcome of every interruption.

The learner's question: when a run lasts longer than any single worker, session or process, and something stops it in the middle, how does the factory lose nothing that was finished, redo nothing that was verified, and write nothing twice?

Five ways a long run ends early#

Plain English. A long run will be interrupted. The only question is whether, afterwards, you can tell exactly what was finished, and continue from there without doing anything twice.

Beginner definition: a run is resumable when its progress lives in durable records and immutable artifacts outside any worker, so a new process can read them and continue.

The causes differ in who notices, how much they take with them, and what the right response is. The table sorts the five this chapter treats.

CauseScopeWho noticesLost without a checkpointRight response
Context growthone workerthe worker, often silentlythe worker's grip on early instructionssmall tasks, fresh workers, checkpoint before compaction
Session or usage limitevery worker on an accountthe provider, all at onceall work in flightpause everyone; stop cleanly; resume later
Process crash, closed terminalone processnobody, until a restarteverything only in memorydurable "running" rows; resume from the record
Tool or terminal failureone callthe worker or the stationthe attemptclassify, then retry or rework (see Assembly stations)
Token budget exhaustionone unit or runthe factory's own guardnothing, if the guard checks before the callstop with a named status; keep the checkpoint

The context window, the text a model can take into account at once, was introduced in AI enters the factory. As a worker reads files, receives tool results and accumulates conversation, its context grows, and anything pushed out of view stops influencing its work. A session limit is different: a cap set by the provider or the account on how much one session, or one account, may use in a period. The first is a property of one worker's task; the second is shared by every worker on the account.

Before AI. A database writes every change to a log before it changes the data, so after a crash it replays the log and knows exactly which transactions committed. A nightly batch job that records the last committed chunk can restart from there instead of from row one. Make rebuilds only the targets older than their inputs. Every long-running system before AI already answered this chapter's question the same way: progress lives in a durable record, written before the work is considered done.

Checkpoint, resume, idempotency#

A checkpoint is a durable record of what a run has finished, written after each completed unit and before any risky step. The analogy is a save point in a long game. Its consequence is that an interruption costs at most the work since the last checkpoint. Its trade-off is a write per unit, and a format that must stay readable by the next version of the code.

To resume is to continue a run from its checkpoint: read what is finished, verify that each finished artifact is still intact, skip it, and do the rest. A resume that trusts the checkpoint without checking the artifacts can skip work that a crash or a person has damaged.

Idempotency means that doing an operation twice has the same effect as doing it once. Writing a file named by the hash of its content is idempotent: the second write finds the file already there and changes nothing. Appending a row "attempt succeeded" is not, unless the row has a key that refuses the duplicate. Idempotency matters because a resumed run cannot always know whether the last write before the crash happened.

A persistent artifact is a station's output stored outside the worker's memory, immutable and addressed by its hash, so that a later process can find it, check it and reuse it. A transcript is not one; a file in a content-addressed store is.

The smallest primitive: a checkpoint after every item#

Example B (examples/b-algorithm-factory/) builds one toy unit, a lower-bound function, through three items: an oracle, a solution and a visualization. After each completed item it writes a checkpoint. Its writing function, quoted verbatim from factory.py : write_atomic:

def write_atomic(path, text):
    tmp = Path(str(path) + ".tmp")
    tmp.write_text(text, encoding="utf-8")
    os.replace(tmp, path)

The temporary file plus rename means a crash during the write leaves either the old checkpoint or the new one, never half of one. The resume decision is three lines inside factory.py : run, quoted verbatim:

for item_id in order:
    rec = cp["items"][item_id]
    if rec["status"] == "done" and artifact_intact(out, rec):
        emit("[skip] %s %s: done in checkpoint, artifact sha256 matches" % (rec["station_id"], item_id))
        continue

An item is skipped only when the checkpoint says it is done and artifact_intact finds the artifact on disk with the sha256 the checkpoint recorded. Run in a scratch copy on 2026-11-20, a simulated crash right after the solution item, then a resume:

$ /opt/homebrew/bin/python3 run.py --stop-after solution --no-cross-lang
start FR-0002 spec SPEC-ALGO-001 pattern PAT-ALGO-UNIT@1.3.0 spec_hash 44d76f9399ab budget 2000 units
[run ] S1-build-oracle oracle: 30 cases (10 fixtures incl. edge cases, 20 seeded random), sha256 45d4895d56b7
[try ] S2-generate-solution solution attempt 1: stand-in says "done, all tests pass"; oracle: 21/30 passed -> rejected
[try ] S2-generate-solution solution attempt 2: stand-in says "done, all tests pass"; oracle: 30/30 passed -> accepted
[kill] simulated crash after solution (checkpoint written, process gone)
checkpoint left at out/checkpoint.json; rerun with --resume

$ /opt/homebrew/bin/python3 run.py --resume --no-cross-lang
resume FR-0002 from checkpoint: done = oracle, solution
[skip] S1-build-oracle oracle: done in checkpoint, artifact sha256 matches
[skip] S2-generate-solution solution: done in checkpoint, artifact sha256 matches
[run ] S3-trace-visualization visualization: 10 traces, 28 states, invariants hold, sha256 2bf5ff26c1a1
end FR-0002 status=succeeded cost=800 units generator_calls=0
generator calls in this process: 0
  S1-build-oracle        oracle        done             attempts=0
  S2-generate-solution   solution      done             attempts=2
  S3-trace-visualization visualization done             attempts=0

The resumed process called the generator zero times. The 800 units in the last line are the run's total, carried in the checkpoint; nothing new was spent. Without the checkpoint, a restart would have rebuilt the oracle and called the generator twice again, another 800 illustrative units for work already accepted. The agent here is a scripted stand-in, so these results show how the factory mechanics work, not how good any model is.

The same example has a budget guard, which is the fifth cause of the table handled well. It checks the budget before each call, so it stops with a named status instead of overspending:

$ /opt/homebrew/bin/python3 run.py --budget 500 --no-cross-lang
start FR-0002 spec SPEC-ALGO-001 pattern PAT-ALGO-UNIT@1.3.0 spec_hash 44d76f9399ab budget 500 units
[run ] S1-build-oracle oracle: 30 cases (10 fixtures incl. edge cases, 20 seeded random), sha256 45d4895d56b7
[try ] S2-generate-solution solution attempt 1: stand-in says "done, all tests pass"; oracle: 21/30 passed -> rejected
[stop] S2-generate-solution solution: attempt 2 would cost 400 units; spent 400 of 500: budget_exhausted
end FR-0002 status=budget_exhausted cost=400 units (checkpoint kept)

budget_exhausted is not succeeded and not failed: it is a resumable stop with the checkpoint kept. Units are illustrative, not tokens or money.

A durable record of a whole run#

A checkpoint file serves one run of one unit. A factory running many units, stations and attempts needs a record it can query. Example D (examples/d-supervisor/) keeps that record in SQLite and schedules from it. The schema, quoted verbatim from state.py : SCHEMA (it ran in SQLite during the example's run; the build does not check SQL):

CREATE TABLE IF NOT EXISTS factory_runs (
  run_id TEXT PRIMARY KEY,
  pattern_version TEXT NOT NULL,
  specification_hash TEXT NOT NULL,
  spec_id TEXT NOT NULL,
  status TEXT NOT NULL,
  cost_units_total INTEGER NOT NULL DEFAULT 0,
  created_at TEXT NOT NULL,
  updated_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS tasks (
  run_id TEXT NOT NULL,
  task_id TEXT NOT NULL,
  station_id TEXT NOT NULL,
  depends_on TEXT NOT NULL,
  status TEXT NOT NULL,
  PRIMARY KEY (run_id, task_id),
  FOREIGN KEY (run_id) REFERENCES factory_runs(run_id)
);
CREATE TABLE IF NOT EXISTS station_attempts (
  run_id TEXT NOT NULL,
  task_id TEXT NOT NULL,
  station_id TEXT NOT NULL,
  attempt INTEGER NOT NULL,
  status TEXT NOT NULL,
  input_hash TEXT NOT NULL,
  artifact_ref TEXT,
  test_evidence_ref TEXT,
  error_class TEXT,
  cost_units INTEGER NOT NULL DEFAULT 0,
  started_at TEXT NOT NULL,
  finished_at TEXT,
  PRIMARY KEY (run_id, station_id, attempt),
  FOREIGN KEY (run_id) REFERENCES factory_runs(run_id)
);
CREATE TABLE IF NOT EXISTS artifacts (
  uri TEXT PRIMARY KEY,
  sha256 TEXT NOT NULL,
  bytes INTEGER NOT NULL,
  run_id TEXT NOT NULL,
  station_id TEXT NOT NULL,
  created_at TEXT NOT NULL
);

A durable record is this kind of store: one row per run, per task and per attempt, written before and after each piece of work, holding hashes and pointers to immutable artifacts. factory_runs binds a run to its pattern version and specification hash, so a resume with another spec can be refused. tasks holds the station graph and each task's status. station_attempts holds every attempt, its input_hash, its artifact and its failure class; Assembly stations reads those columns as a station contract at run time. artifacts maps each artifact URI to the sha256 of its bytes.

The idempotency logic, verbatim#

Three pieces of code make a crash at any point safe. They are quoted verbatim from Example D; the two methods of Supervisor are shown with their class indentation removed.

The first is the artifact store, artifacts.py : ArtifactStore.put. A file is named by the sha256 of its bytes and written only if it does not exist yet:

def put(self, data: bytes) -> tuple:
    """Store bytes; return (digest, wrote). Safe to call from worker threads."""
    digest = sha256(data)
    p = self.path(digest)
    with self._lock:
        if os.path.exists(p):
            return digest, False
        tmp = p + ".tmp"
        with open(tmp, "wb") as f:
            f.write(data)
        os.replace(tmp, p)  # atomic rename: a crash never leaves half a file under the real name
        self.writes += 1
        return digest, True

The second decides whether a station needs to run at all, scheduler.py : Supervisor.input_hash and Supervisor.find_reusable:

def input_hash(self, run_id, spec, t):
    """Hash of everything the station reads: pattern, spec, station and dependency digests."""
    inputs = {}
    for d in t.deps:
        ref = self.conn.execute(
            "SELECT artifact_ref FROM station_attempts WHERE run_id=? AND station_id=? "
            "AND status IN ('succeeded','reused') ORDER BY attempt DESC LIMIT 1", (run_id, d)).fetchone()
        inputs[d] = self.conn.execute("SELECT sha256 FROM artifacts WHERE uri=?",
                                      (ref["artifact_ref"],)).fetchone()["sha256"]
    material = {"pattern": f"{PATTERN_ID} {PATTERN_VERSION}", "spec_hash": spec_hash(spec),
                "station_id": t.station_id, "inputs": inputs}
    return hashlib.sha256(canonical(material)).hexdigest(), inputs

def find_reusable(self, station_id, ih):
    return self.conn.execute(
        "SELECT artifact_ref, test_evidence_ref FROM station_attempts WHERE station_id=? AND input_hash=? "
        "AND status='succeeded' ORDER BY run_id, attempt LIMIT 1", (station_id, ih)).fetchone()

The third turns attempts left running by a dead process into abandoned, scheduler.py : Supervisor.recover_interrupted:

def recover_interrupted(self, run_id: str) -> None:
    """Attempts left 'running' by a crash become 'abandoned'. They do not count as failures."""
    rows = self.conn.execute(
        "SELECT station_id, attempt FROM station_attempts WHERE run_id=? AND status='running' "
        "ORDER BY station_id, attempt", (run_id,)).fetchall()
    for r in rows:
        self.conn.execute(
            "UPDATE station_attempts SET status='abandoned', error_class='interrupted', finished_at=? "
            "WHERE run_id=? AND station_id=? AND attempt=?",
            (self.clock.now(), run_id, r["station_id"], r["attempt"]))
        self.log.append(f"{run_id} {r['station_id']:24s} attempt {r['attempt']} abandoned (interrupted)")
    self.conn.commit()

The order of operations ties them together, and Supervisor.run_round enforces it: an attempt row with status running is inserted and committed before any worker starts (the comment in the code reads "'running' rows are durable BEFORE any work starts"); the worker computes the artifact and calls put; only then is the row updated to succeeded with the artifact's URI. A crash before put leaves a running row and no file. A crash after put leaves a running row and a file. Either way, resume marks the row abandoned, runs the station again, and put either writes the file for the first time or finds it and writes nothing.

The executable demonstration: crash, resume, reuse#

Output of /opt/homebrew/bin/python3 run.py in a scratch copy of Example D on 2026-11-20, the crash run, its resume, and the rerun of an earlier spec:

== FR-0005 SPEC-ALGO-002: process dies after S2-generate-solution writes its artifact
   run status: CRASHED at S2-generate-solution; physical artifact writes by this process: 3
  station                  att  status     cost  artifact / error
  S1-validate-spec           1  succeeded     0  artifact://FR-0005/S1-validate-spec/6b1ff6027d36
  S2-generate-oracle         1  succeeded    30  artifact://FR-0005/S2-generate-oracle/17a636457aeb
  S2-generate-solution       1  running       0
  tasks: T1=succeeded T2=running T3=succeeded T4=pending T5=pending T6=pending

== FR-0005 SPEC-ALGO-002: resume run (new supervisor, same database)
   run status: succeeded; physical artifact writes by this process: 3
  station                  att  status     cost  artifact / error
  S1-validate-spec           1  succeeded     0  artifact://FR-0005/S1-validate-spec/6b1ff6027d36
  S2-generate-oracle         1  succeeded    30  artifact://FR-0005/S2-generate-oracle/17a636457aeb
  S2-generate-solution       1  abandoned     0  interrupted
  S2-generate-solution       2  succeeded    40  artifact://FR-0005/S2-generate-solution/b03036903131
  S3-build-visualization     1  succeeded    25  artifact://FR-0005/S3-build-visualization/f176f95de42b
  S3-run-tests               1  succeeded     0  artifact://FR-0005/S3-run-tests/192abe0c57d5
  S4-package-manifest        1  succeeded     0  artifact://FR-0005/S4-package-manifest/0212923625ed
  tasks: T1=succeeded T2=succeeded T3=succeeded T4=succeeded T5=succeeded T6=succeeded

== FR-0006 SPEC-ALGO-001: same spec as FR-0004: idempotent rerun
   run status: succeeded; physical artifact writes by this process: 0
  station                  att  status     cost  artifact / error
  S1-validate-spec           1  reused        0  artifact://FR-0004/S1-validate-spec/171a648da216
  S2-generate-oracle         1  reused        0  artifact://FR-0004/S2-generate-oracle/6981bd4abb75
  S2-generate-solution       1  reused        0  artifact://FR-0004/S2-generate-solution/4545fa833c8a
  S3-build-visualization     1  reused        0  artifact://FR-0004/S3-build-visualization/6e1f6f8ac7ea
  S3-run-tests               1  reused        0  artifact://FR-0004/S3-run-tests/94b662066656
  S4-package-manifest        1  reused        0  artifact://FR-0004/S4-package-manifest/122f893ee740
  tasks: T1=reused T2=reused T3=reused T4=reused T5=reused T6=reused

== checks
   FR-0005 writes: 3 before crash + 3 after resume = 6 (6 stations, 6 distinct artifacts)
   FR-0005 succeeded attempts per station: [1] (never a duplicate)
   FR-0006 station executions: 0, writes: 0

The event log of the resume (run.py --verbose) shows the one line that proves idempotency: the solution station ran again and found its file already present.

FR-0005 S2-generate-solution     CRASH after artifact write, before record
FR-0005 resumed  status was running
FR-0005 S2-generate-solution     attempt 1 abandoned (interrupted)
FR-0005 S1-validate-spec         skip (already succeeded)
FR-0005 S2-generate-oracle       skip (already succeeded)
FR-0005 S2-generate-solution     attempt 2 succeeded artifact://FR-0005/S2-generate-solution/b03036903131 (file already present, no write)
FR-0005 S3-build-visualization   attempt 1 succeeded artifact://FR-0005/S3-build-visualization/f176f95de42b (wrote)

A record is only durable if a different program can read it. This page's writer opened the database the run left behind with a short script (run beside the copy, after run.py) and asked it the two questions a person asks after an interruption:

"""read_record.py: open Example D's SQLite record after run.py and answer two questions from it alone."""
import sqlite3

conn = sqlite3.connect("out/factory.db")
print("What happened in FR-0005?")
for row in conn.execute("SELECT station_id, attempt, status, error_class, input_hash, artifact_ref "
                        "FROM station_attempts WHERE run_id='FR-0005' ORDER BY started_at, station_id, attempt"):
    sid, att, status, err, ih, ref = row
    print(f"  {sid:23} {att} {status:9} {err or '':11} in={ih[:12]} {ref or ''}")
print("What is left to do in each run?")
for run_id, status in conn.execute("SELECT run_id, status FROM factory_runs ORDER BY run_id"):
    left = [r[0] for r in conn.execute("SELECT station_id FROM tasks WHERE run_id=? AND status NOT IN "
                                       "('succeeded','reused') ORDER BY task_id", (run_id,))]
    print(f"  {run_id} {status:10} open tasks: {', '.join(left) or 'none'}")
What happened in FR-0005?
  S1-validate-spec        1 succeeded             in=7e9515d900fd artifact://FR-0005/S1-validate-spec/6b1ff6027d36
  S2-generate-oracle      1 succeeded             in=6bf12a168880 artifact://FR-0005/S2-generate-oracle/17a636457aeb
  S2-generate-solution    1 abandoned interrupted in=5401f2a0768d 
  S2-generate-solution    2 succeeded             in=5401f2a0768d artifact://FR-0005/S2-generate-solution/b03036903131
  S3-build-visualization  1 succeeded             in=2761aa60acaf artifact://FR-0005/S3-build-visualization/f176f95de42b
  S3-run-tests            1 succeeded             in=09f211241a8e artifact://FR-0005/S3-run-tests/192abe0c57d5
  S4-package-manifest     1 succeeded             in=3fa50402e7cf artifact://FR-0005/S4-package-manifest/0212923625ed
What is left to do in each run?
  FR-0004 succeeded  open tasks: none
  FR-0005 succeeded  open tasks: none
  FR-0006 succeeded  open tasks: none
  FR-0007 incomplete open tasks: S3-build-visualization, S4-package-manifest

Three things are visible in the rows. The abandoned attempt and the successful one carry the same input hash, 5401f2a0768d: the same input ran twice and produced one artifact. FR-0007, the run whose visualization station failed three times and was escalated, is incomplete with two open tasks, never succeeded. And everything above came from the record alone, with no process running. sh check_repro.sh in the same copy printed byte-identical: stdout, out/state_dump.json, out/events.log, out/artifacts/*: two full runs, crash and resume included, produce the same record. This is the site's demonstration that durable run state, resumability and independently validated artifacts work as described; the stations are scripted stand-ins, so it shows the supervisor's mechanics, not any model's quality.

Diagram: One station attempt through a crash · a running row first, the artifact by hash, the final row last

Top lane, the normal path: the supervisor commits a running row before any work starts; the worker computes the artifact; the content-addressed store writes it under its hash only if the file does not exist; then the row becomes succeeded. A crash anywhere in that path leaves a running row. Bottom lane, the resume: a new supervisor reads the same database, marks the running row abandoned, which costs no attempt, skips stations already succeeded, and runs the interrupted station again with the same input hash. The store finds the file already present and writes nothing. In FR-0005 that gives 3 writes before the crash, 3 after, and 6 distinct artifacts.

  1. Normal path
    1. running row: The attempt is recorded as running before the worker starts, so a crash leaves an honest trace.
    2. worker computes: The station's work; in Example D a scripted stand-in, so the bytes depend only on the input.
    3. store by hash: The file is named by the sha256 of its bytes and written atomically, once.
    4. succeeded row: Only now does the record point at the artifact.
  2. Crash and resume
    1. crash: The process dies; the row still says running.
    2. mark abandoned: Interrupted, not failed: no attempt is used up.
    3. skip finished: Stations with a succeeded row are not run again.
    4. rerun, no write: Same input hash, same bytes; the store writes nothing the second time.

The diagram follows one station attempt through a crash. Work starts only after a running row is committed. The artifact goes into a content-addressed store, and only then is the row marked succeeded. A crash anywhere in between leaves a running row. On resume, that row becomes abandoned, which costs no attempt; finished stations are skipped; the interrupted station runs again with the same input hash; and the store, finding the file already present, writes nothing.

Compaction, and what it must never replace#

Compaction is replacing the earlier part of a long conversation with a shorter summary, so a worker that is running out of context room can continue. The analogy is minutes that replace the recording of a long meeting: much shorter, mostly right, and silent about whatever the note-taker thought unimportant. The operational consequence is that compaction is lossy. A constraint stated once at the start, an exception agreed halfway through, or the exact list of files already finished can disappear from the summary, and the worker will not know it has lost them. The trade-off is continuity of one worker's session against the risk of acting on a summary.

This site's recommendation: never let a compacted conversation be the record a run resumes from. Before compaction, write a checkpoint: the finished units, their hashes, and the next step, into the durable record. After compaction, the worker reads the record again. Product (Claude Code, as documented on 2026-11-20): the hooks reference lists a PreCompact event among its hook events, which is a natural place to run a script that writes such a checkpoint. The documentation read here describes the event's existence, not a recommended checkpoint script; the script is this site's suggestion.

The cheaper answer, where the work allows it, is to avoid long conversations altogether: give each worker one small task and start a fresh worker for the next. Context then never grows past one task, and nothing needs compacting. That is the design WatchAlgo used, as the next section shows.

In a coding agent. A headless coding agent can resume a conversation. Product (Claude Code, as documented on 2026-11-20): --continue continues the most recent conversation and --resume <session-id> a specific one. Product (Claude Agent SDK, as documented on 2026-11-20): sessions persist the conversation, not the filesystem. Product (Codex, as documented on 2026-11-20): codex exec resume --last or codex exec resume <SESSION_ID> resumes non-interactive work. Each of these restores what the agent was told and said. None of them is the factory's record of which units are accepted: that lives in the factory's own store, checked by the factory's own tests, and a person with merge rights still reviews what the agent produced.

A durable record is not long-term agent memory#

The two are easy to confuse because both survive the end of a process. They answer different questions.

Example D's database answers "what happened in run FR-0005, and what is left to do". It is keyed by run, station and attempt, holds hashes and pointers to immutable artifacts, and is read by the scheduler with exact queries. Nothing in it goes into a model's prompt, gets summarized, or is forgotten. Its job ends when the run ends; after that it is an audit trail.

Long-term agent memory answers "what should an agent know next time, across many runs". It is selected, summarized and retrieved by relevance; it can be wrong or stale; it needs its own rules for what to keep, who may write it and when to forget. Mixing the two goes wrong in both directions. A resume decision taken from a fuzzy memory can redo finished work or skip unfinished work. A memory store fed every attempt row floods the agent with noise.

Diagram: Durable execution record against long-term agent memory · two stores, two questions

Left, the durable record: written by the scheduler before and after each attempt, keyed exactly by run, station and attempt, read by code with exact queries, complete by design. It answers what happened and what is left, and it decides what runs next. Right, long-term memory: written by or for an agent, selected and summarized, retrieved by relevance, allowed to be incomplete or stale. It answers what an agent should know next time. Memory may shape how an agent works; it must never decide whether a unit counts as done.

  1. Durable record
    1. written by the scheduler: Every run, task and attempt, with hashes and artifact pointers.
    2. exact keys: Queried exactly; nothing is summarized or forgotten while the run lives.
    3. decides what runs: The scheduler acts only on the record and the artifacts it points to.
  2. Agent memory
    1. written for an agent: Chosen notes meant to enter a future context.
    2. retrieved by relevance: Summarized and ranked; can be wrong, outdated or missing pieces.
    3. shapes how work is done: Guides behavior; it is not evidence that a unit is accepted.

The diagram sets the two side by side. The record is written by the scheduler, keyed exactly, read by code, and complete; it decides what runs next. Memory is written by or for an agent, selected and summarized, retrieved by relevance, and allowed to be incomplete; it shapes how an agent works, never whether a unit counts as done.

What this is not. This page covers checkpoints, resume and idempotent writes inside a factory run. Durable memory as a discipline, and how it differs from saved execution state, belongs to Memory Engineering. Retries, checkpoints and recovery as a reliability discipline belong to Agent Reliability Engineering. The working set a model sees in its window belongs to Context Engineering. The catalog of failures and their recoveries is Failure and recovery scenarios.

Try it#

The recovery lab runs a long job over many units with a context budget. Context grows with each unit; at the limit the session ends. Switch between three modes, no checkpoint, a checkpoint per unit, and a checkpoint plus compaction, and see units persisted, work lost, the resume point and duplicate writes. The second embed starts in the no-checkpoint mode.

docs/LABS.md has no plain table for this lab yet, so the table below was written for this page from the real outputs of Examples B and D shown above. The lab's own numbers are simulated, illustrative; the rows below say which example each fact comes from.

ModeWhat survives an interruptionWork redone on resumeDuplicate writesSource
No checkpointonly what the restarted run rebuildseverything: Example B's fresh run calls the generator twice, 800 illustrative unitspossible for any non-idempotent writeExample B, fresh run output
Checkpoint per unitevery item recorded as done, with its sha256only unfinished items: Example B's resume makes 0 generator calls0 in Example D: 3 writes before the crash + 3 after = 6 artifactsExamples B and D, crash and resume output
Checkpoint + compactionas above, plus a shorter session that can continueas above, if the record and not the summary decidesas abovethe lab only; neither example compacts, so this row is simulated

WatchAlgo in practice#

Observed in the repository (docs/EVIDENCE.md, matrix rows 11 and 13, section 8): the long September runs were stopped by account usage limits, not by context windows. The last lines of 10,657 worker logs are usage messages: weekly limit (9,674), monthly spend limit (976), session limit (7). Each task started a fresh claude -p process working on 5 or 8 problems, so context growth was bounded by the task size, and nothing in the logs shows compaction inside a worker. The WatchAlgo story reads the author's "30+ hours" against these logs; this page uses only the mechanisms.

Observed (scripts/viz/pipeline.py, commit 971863f2; scripts/viz/supervise.sh, commit 017ed6b9): the queue lived on disk under a file lock and was written atomically; it could be rebuilt from the files on disk, so the disk was the source of truth. When the driver started, tasks left running by a dead driver were verified from disk, finished problems were credited, and the rest re-queued without spending an attempt. The supervisor restarted the driver when it was gone and committed verified content and the queue every 1,800 seconds. The local run.log records 19 orphan recoveries and 18 supervisor restarts. Checkpoint commits run every 30 minutes from 5e6c6040 to 1b183564. Observed (.claude/SOLUTION_ASSEMBLY_LINE.md, .ai-context/AUTONOMOUS_PROGRESS.md): earlier, session-level resume was done by instruction: a "Session Recovery" section and an "authoritative resume marker" told the next session where to start.

Observed (local cron, tracked file .ai-context/cron-watchdog.log): the March watchdog still runs hourly. On 2026-11-20 at 10:00 it failed to start the claude binary, which no longer exists at its old path, and logged "Resume triggered" anyway: a partial failure recorded as success. The design lesson for resume is that a launcher must record its result, not its intent. Reported by the author: the process ran until token limits required intervention; read against the logs, the intervention a usage limit requires is waiting or raising the limit, and the resumable queue is what made waiting cheap. Teaching reconstruction: Example D's per-attempt record with input hashes is this site's design; WatchAlgo's record was a task queue and git commits, with verification from disk standing in for input hashes.

Enterprise translation#

In the insurance factory of Part III, a factory run that builds the intake service may wait days at a security review or a release approval. No process stays alive that long, so the run must be a record, not a process. Product (Temporal, as documented on 2026-11-20): progress is recorded in an Event History; after a crash a new worker replays the workflow code against that history to rebuild its state; workflow code must be deterministic, and non-deterministic work, including model calls, belongs in Activities, which Temporal retries with timeouts. Product (LangGraph, as documented on 2026-11-20): retries re-run a whole node, so side effects such as payments or emails can repeat, and the documentation advises deriving idempotency keys from durable state. Both say, in their own terms, what Example D shows: replay is safe only when every effect is idempotent.

The same rule appears inside the product the factory builds. The insurance service's write tool, record_recommendation(submission_id, recommendation, idempotency_key), takes an idempotency key so that a retried call cannot record a recommendation twice; that is the underwriting workflow operating, which Insurance factory, part 2 keeps apart from the factory building it. This site's recommendation: checkpoint before every step that touches the outside world or waits for a person; key every external write; and treat a run as finished only when its record says every task is accepted, never because a process exited cleanly. What dominates the critical path here is the waiting at human gates, and a durable record is what makes waiting free.

Failure test#

The failure: a station writes its artifact, and the process dies before the record is updated. Without care, the resumed run either thinks the station never ran and writes a second artifact, or thinks it finished and skips a file nobody checked.

How the factory catches it: the running row committed before the work leaves an honest trace; resume turns it into abandoned, which costs no attempt; the station runs again with the same input hash; the content-addressed store finds the file and writes nothing. FR-0005 shows the result: 3 writes before the crash, 3 after, 6 distinct artifacts, one success per station.

What to run or toggle: in a scratch copy of examples/d-supervisor/, run /opt/homebrew/bin/python3 run.py --verbose and find the line "file already present, no write"; then run sh check_repro.sh. In a copy of Example B, run run.py --stop-after solution and then run.py --resume. In the lab, switch to "no checkpoint" and compare the work lost with the per-unit mode.

Where to go deeper#

Checklist for Monday#

Exercises#

  1. A station crashed after writing its artifact. Why does Example D not count that attempt as a failure?
Answer 1Interruption is not a defect

Nothing showed the work was wrong; the process died. Marking it abandoned keeps the three attempts for real defects.

  1. Why does FR-0006 execute zero stations?
Answer 2Same input hashes

Its spec equals FR-0004's, so every station's input hash matches a recorded success and the artifact is reused.

  1. A worker's conversation was compacted and it now says six units are finished. What should the scheduler believe?
Answer 3The record, not the summary

The durable record and the artifact hashes. A summary is a self-report written under context pressure.

Advanced perspective#

Exactly once is a property you build, not one you getAt-least-once execution plus idempotent effects

A process can always die between doing something and recording that it did it. No ordering of two writes removes that gap; it only moves it. So distributed systems, workflow engines and Example D all settle for at-least-once execution and make every effect idempotent, which together behave like exactly once from the outside. Example D's version: execution may repeat (the solution station ran twice in FR-0005), but the effect, a file named by its hash, happens once.

The hard cases are effects that are not naturally idempotent: an email, a payment, a ticket in another team's tracker, a pull request. For those, the factory needs a key the receiving system honors, or a record written before the call that the resume checks, or a person. A factory that cannot make an effect idempotent should put a human gate in front of it, and the governance chapter should say so.

Checkpoint granularity: per unit, per step, per time windowWhat an interruption costs against what a checkpoint costs

WatchAlgo checkpointed at two levels at once. Per unit: every verified problem was credited in the queue, and its files were on disk. Per time window: the supervisor committed to git every 30 minutes. The first decides what a resume redoes; the second decides what survives a lost machine or a bad edit. Example B checkpoints per item; Example D per attempt.

Finer checkpoints cost writes and a schema to maintain, and coarser ones cost redone work. The useful rule is to checkpoint at the boundary of anything expensive or irreversible: after each accepted unit, before a model call that costs real money, before an external write, and before compaction. A time-window checkpoint is a good second layer but a poor only layer, because a crash a minute before the next commit loses everything in between.

Resume must re-verify, not trustA checkpoint is a claim about the disk

A checkpoint says "item X was finished, and its artifact had hash H". Between the checkpoint and the resume, a person may have edited the file, a disk may have failed, or a half-finished operation may have left something behind. Example B handles this with artifact_intact: it skips an item only if the file exists and its sha256 still matches, and its tests show a tampered artifact being rebuilt. WatchAlgo's driver did the same for orphaned tasks: it verified each problem from disk and credited only what passed.

The principle is the site's refrain applied to the factory's own records: a checkpoint is a self-report from an earlier process. It is a very good hint, and resuming from it is the point, but the artifact on disk is the evidence. When the two disagree, the artifact wins and the item is redone.

A usage limit is a fleet-wide eventPause everyone, stop cleanly, resume on a schedule

A context limit hits one worker. A usage limit hits every worker on the account in the same minute, so a per-worker retry multiplies the damage: every worker burns an attempt at the same time against a limit that will not reset for hours. WatchAlgo's driver handled the messages it recognised correctly, with one pause for all workers and no attempt spent, and its storm came from one message it did not recognise.

The robust design treats any limit-class failure as a run-level event: record it in the run's record with the reset time if known, pause the whole line, stop cleanly with a resumable status such as paused_on_limit, and resume on a schedule rather than in a retry loop. Example B's budget_exhausted is the same idea for the factory's own budget. A run that pauses this way loses nothing; the record says exactly where to continue.

In one breath#

A long run will be interrupted, by a full context, a usage limit, a crash, a failing tool or an exhausted budget, so its progress must live outside any worker: a durable record written before and after each piece of work, and artifacts stored by content hash. Resume reads the record, re-verifies each finished artifact, skips it, and reruns the rest; idempotent writes make a repeated step harmless, which Example D demonstrates with a crash between an artifact write and its record, and a resume with no duplicate and no loss. Compaction and resumed conversations help one worker continue, but the record, not a summary or a memory, decides what counts as done.

A run that resumes perfectly can still preserve a wrong answer perfectly. The next chapter, Evidence of correctness, asks what makes an accepted artifact worth keeping: properties, randomized and differential tests, metamorphic relations, trace checks, regression cases, and where a person still has to look.