At 10:00 this morning, 20 November 2026, a scheduled job on the author's machine woke up as it does every hour. It is WatchAlgo's March watchdog, and its job was to find units with missing files and start a generation session to fill them. It found four gaps. It tried to start the coding agent at its old path, and the start failed, because that program is no longer installed there. Then it wrote one more line to its log: "Resume triggered".
Observed (local cron entry and the tracked file .ai-context/cron-watchdog.log, docs/EVIDENCE.md section 13): the log records a success that did not happen. Nothing was harmed this time, because nothing ran. But a reader of that log, a person or a dashboard, would believe the gaps were being filled. And if a program reappeared at that path, the job would start generation sessions on its own.
The job recorded what it intended, not what happened. That one habit, multiplied across stations, is how a factory reports a batch as done while half of it is missing.
The learner's question: when something in the line breaks, how does the factory notice, keep the damage inside one unit, restore a known good state, and never report a broken run as done?
Every failure has four questions#
Plain English. A failure is not only something that crashes. It is anything that leaves the factory's record saying something different from the truth: a half-written file, a test that lies, a loop that never ends, an instruction hidden in a spec, a write that cannot be undone.
Beginner definition: failure handling in a factory means that, for each known way the line can break, someone has answered four questions in advance: how does it show up, what detects it, what restores a known good state, and who decides when the automatic answer is not enough.
A partial artifact is the simplest example: a station started writing its output and stopped halfway. A tiny example: a tracer file with a function signature and no body. The consequence is that any later station, or any person, that trusts the file's existence builds on nothing. The trade-off of defending against it is a rule that costs a little on every write: an artifact counts only once its record says so, and the record is written after the artifact is complete.
Before AI. Operations teams keep runbooks: for each alert, what it means, how to confirm it, how to restore service, whom to call. Manufacturing engineers do a failure mode and effects analysis before a line opens, listing every way a part can fail, how it would be detected, and how bad the effect would be. A Kafka consumer sends a message it cannot process to a dead-letter topic instead of retrying it forever. A factory for software needs the same discipline, with three failure modes that are new: a model that confidently reports success, untrusted text that tries to give instructions, and a worker that can loop without ever crashing.
Four safety rules, in plain words#
This site's spec turns those lessons into rules that every factory run follows. They are short because they must be checkable.
| Rule | In plain words | What it prevents | Where the examples prove it |
|---|---|---|---|
| Partial failure is never success | A run with any task failed, escalated, blocked or unverified ends incomplete, never succeeded | the watchdog's "Resume triggered" | Example D: FR-0007 ends incomplete with 5 of 6 tasks done |
| Idempotency for retryable side effects | Any write that may be retried carries a key, so doing it twice has the effect of doing it once | duplicate records after a retry or a crash | Example C: the same idempotency key replays without a second recommendation |
| Durable checkpoint before long operations | Progress is written to a durable record before risky or long work starts | losing hours of finished work to one interruption | Example B and Example D: resume skips finished work |
| Human review for irreversible actions | Authorization changes, insurance decisions, schema destruction, credential handling and irreversible deployments need a person | an automated line doing what no one can undo | Example C: only the underwriter role can record a decision |
Two more rules from the same section of the spec run through the catalog below. Untrusted text, such as a problem statement, a pull request description, repository docs, tool results or a dependency, cannot override the factory's own instructions. And an agent's report that it is done is never accepted without artifacts and tests it did not write.
The smallest primitive: a run status that cannot lie#
The first rule fits in a few lines. Example D's supervisor decides a run's final status from the status of every task, not from the last thing that happened. Quoted verbatim from examples/d-supervisor/scheduler.py : Supervisor.finish_run (dedented by one level so it compiles alone; OK_STATUSES is ("succeeded", "reused") in the same file):
def finish_run(self, run_id: str) -> str:
statuses = [r["status"] for r in self.conn.execute(
"SELECT status FROM tasks WHERE run_id=? ORDER BY task_id", (run_id,))]
final = "succeeded" if all(s in OK_STATUSES for s in statuses) else "incomplete"
cost = self.conn.execute("SELECT COALESCE(SUM(cost_units),0) FROM station_attempts WHERE run_id=?",
(run_id,)).fetchone()[0]
self.conn.execute("UPDATE factory_runs SET status=?, cost_units_total=?, updated_at=? WHERE run_id=?",
(final, cost, self.clock.now(), run_id))
self.conn.commit()
self.log.append(f"{run_id} finished status={final} cost_units={cost}")
return final
Read the fourth line. "Succeeded" is the exception that must be earned by every task; anything else, including a task nobody classified, gives "incomplete". The default points toward the truth. The watchdog's default pointed the other way: it logged success unless something stopped it.
The executable demonstration: crash, escalate, and read the record#
Example D (examples/d-supervisor/) runs four factory runs into a SQLite record. Two of them fail on purpose. In FR-0005 the process dies after a station wrote its artifact but before the record said so; a new supervisor resumes the run. In FR-0007 the visualization station fails its invariant check three times and is escalated to a person. Long-running agent execution walks through the crash and resume step by step; this page reads what the failures leave behind. The agent here is a scripted stand-in, so these results show how the factory mechanics work, not how good any model is.
Run in a scratch copy with /opt/homebrew/bin/python3 run.py --verbose on 2026-11-20, the lines for the two failing runs:
FR-0005 S2-generate-solution CRASH after artifact write, before record
FR-0005 resumed status was running
FR-0005 S2-generate-solution attempt 1 abandoned (interrupted)
FR-0005 S1-validate-spec skip (already succeeded)
FR-0005 S2-generate-oracle skip (already succeeded)
FR-0005 S2-generate-solution attempt 2 succeeded artifact://FR-0005/S2-generate-solution/b03036903131 (file already present, no write)
FR-0005 finished status=succeeded cost_units=95
FR-0007 S3-build-visualization attempt 1 failed invariant_violation; retry (1/3 failures)
FR-0007 S3-run-tests attempt 1 succeeded artifact://FR-0007/S3-run-tests/ae596d71be22 (wrote)
FR-0007 S3-build-visualization attempt 2 failed invariant_violation; retry (2/3 failures)
FR-0007 S3-build-visualization attempt 3 failed invariant_violation; 3/3 failures: ESCALATED to a person
FR-0007 S4-package-manifest blocked by S3-build-visualization
FR-0007 finished status=incomplete cost_units=145
Two different failures, two different records. The crash is an interruption, not a defect, so the interrupted attempt becomes abandoned and does not use up a retry. The visualization failure is a defect, so each attempt counts, the third one escalates, and the manifest that depends on it is blocked. S3-run-tests does not depend on the visualization and finishes anyway: one failure stays inside its own branch of the graph.
A log is for reading once. The record is for answering questions later, so this page wrote a short read-only script that queries the database the way an on-call person would, run in the same scratch copy:
"""failures_view.py: read Example D's record after run.py, the way an on-call person would.
Run in a scratch COPY of examples/d-supervisor after `run.py`. Read-only queries on out/factory.db.
For each factory run: final status, attempts by status, and every task that is not finished.
"""
import sqlite3
conn = sqlite3.connect("out/factory.db")
conn.row_factory = sqlite3.Row
for run in conn.execute("SELECT run_id, status, cost_units_total FROM factory_runs ORDER BY run_id"):
rid = run["run_id"]
counts = {r["status"]: r["n"] for r in conn.execute(
"SELECT status, COUNT(*) AS n FROM station_attempts WHERE run_id=? GROUP BY status ORDER BY status", (rid,))}
errors = [f"{r['station_id']}#{r['attempt']}={r['error_class']}" for r in conn.execute(
"SELECT station_id, attempt, error_class FROM station_attempts "
"WHERE run_id=? AND error_class IS NOT NULL ORDER BY station_id, attempt", (rid,))]
open_tasks = [f"{r['task_id']}:{r['status']}" for r in conn.execute(
"SELECT task_id, status FROM tasks WHERE run_id=? AND status NOT IN ('succeeded', 'reused') "
"ORDER BY task_id", (rid,))]
print(f"{rid} status={run['status']:10} cost_units={run['cost_units_total']:3} attempts={counts}")
if errors:
print(f" errors: {', '.join(errors)}")
if open_tasks:
print(f" not finished: {', '.join(open_tasks)}")
FR-0004 status=succeeded cost_units=125 attempts={'failed': 1, 'succeeded': 6}
errors: S2-generate-oracle#1=transient_timeout
FR-0005 status=succeeded cost_units= 95 attempts={'abandoned': 1, 'succeeded': 6}
errors: S2-generate-solution#1=interrupted
FR-0006 status=succeeded cost_units= 0 attempts={'reused': 6}
FR-0007 status=incomplete cost_units=145 attempts={'failed': 3, 'succeeded': 4}
errors: S3-build-visualization#1=invariant_violation, S3-build-visualization#2=invariant_violation, S3-build-visualization#3=invariant_violation
not finished: T4:escalated, T6:blocked
Every failure has a class: transient_timeout, interrupted, invariant_violation. Every unfinished task has a status a person can act on. And FR-0007 cannot be mistaken for a success by anyone who reads the record, which is exactly what the watchdog's log could not promise.
Diagram: Any failure, four steps · detect, classify, respond, record honestly
Left to right. A signal detects the failure: a check that fails, a record that still says running after a crash, or a run whose spend rises while accepted output stays flat. Classification decides the response, and it comes before any retry. A transient error, such as a timeout, is retried within a bound. A defect goes back to its station as rework, and after the attempt cap it is escalated to a person. A shared limit, such as an account usage limit, pauses every worker at once. Untrusted input is quarantined as data. Anything nobody classified stops and asks a person. Last, the record carries an honest status: abandoned, escalated, blocked or incomplete, never succeeded unless every task succeeded. A recurring class goes to the learning loop, which fixes the pattern.
- Detect
- a signal: A failed check, a running row after a crash, or spend without accepted output.
- Classify, then respond
- transient: Timeouts and brief unavailability; retry a few times, then treat as an outage.
- defect: The artifact is wrong; send it back, and after three attempts give it to a person.
- shared limit: One cause for all workers; retrying each unit separately only spends attempts.
- unknown: No class matches; the safe default is a person, not a retry.
- Record and learn
- honest status: abandoned, escalated, blocked or incomplete; succeeded only when every task succeeded.
- learning loop: A class that recurs changes the pattern, its checks or its catalog, not only the unit.
The diagram follows any failure through four steps. A signal detects it: a check that fails, a record still saying running, a run that spends without progress. Classification decides the response: a transient error is retried within a bound, a defect goes back as rework, a shared limit pauses every worker, untrusted input is quarantined, and anything unclassified stops and asks a person. The record then carries an honest status, and a recurring class goes to the learning loop to fix the pattern, not only the unit.
The catalog: ten failures#
The table is the catalog in one view. The sections after it explain each family: why it happens, and why the obvious fix is not enough.
| Failure | Shows up as | Detected by | Restored by | Demonstrated in |
|---|---|---|---|---|
| Partial artifact | a file exists, its record is missing or running | record and artifact disagree; content hash check | mark the attempt abandoned, rerun; content-addressed write | Example D FR-0005; recovery lab, crash preset |
| Flaky test | the same artifact passes and fails on reruns | rerun history per gate | quarantine with owner and expiry; fix the test | not demonstrated; Example D FR-0004 shows the transient retry to keep apart |
| Agent loop | attempts and spend rise, accepted output does not | attempt cap, budget guard, progress check | escalate to a person with the evidence | Example B escalated and budget_exhausted; Example D FR-0007 |
| Prompt injection in specs and passages | input text tries to give the factory or agent instructions | spec validation; golden injection cases | treat as data; least privilege; human gate | Insurance factory, part 3 UW-005-style test: agent gets 403 |
| Malicious dependency | a new or changed package runs code at build or run time | approved-dependency list, lockfile hashes, provenance checks | pin, remove, rebuild from known inputs | not demonstrated; Governance |
| Merge conflict | parallel stations edit the same files | merge check at integration; conflict rate | partition writes; sequence the work | economics shared-files preset; batch contention preset |
| Model outage | every call fails or times out at once | error class across workers | bounded retry, then pause the line; resume later | Example C UW-F03 503 after 3 attempts (provider stub) |
| Stale pattern | units built against an old pattern or validator version | version recorded per artifact; revalidation | selective regression; rebuild affected units | blast lab; resume refused for a different spec |
| Broken deployment | errors rise after a release | canary threshold | automatic rollback to the previous digest | gates lab, rollback preset |
| Irreversible external write | a retried write would happen twice, or cannot be undone | idempotency key; approval gate | the key refuses the duplicate; a person approves first | Example C UW-F02 idempotent replay |
Work that stops halfway: partial artifacts, model outages, agent loops#
A partial artifact comes from the gap between doing and recording. A process can die after writing a file and before updating the record, or halfway through the file itself. The obvious fix, "check whether the file exists", is the bug: existence says nothing about completeness. Example D's design closes the gap three ways: a durable running row before the work, files named by the hash of their content so a repeat write changes nothing, and a final row only after the file exists. On resume, any running attempt becomes abandoned and the station runs again.
A model outage is the provider being unavailable: timeouts, errors, or a usage limit that stops every worker at once. It looks like many failures and is one. Retrying each unit separately spends attempts against a cause that no unit can fix. Example C shows the bounded part with its stub provider (a roof-data provider, not a model; the shape is the same): UW-F02 times out twice and succeeds on the third attempt, while UW-F03 always times out and stops with a 503 after 3 attempts, writes nothing, and emits assessment.failed.v1. The fleet-wide part, a pause shared by all workers, is the rule WatchAlgo's driver had and the WatchAlgo story shows failing on one unrecognised message.
An agent loop is a worker that keeps trying without converging: rewriting the same file, rerunning the same failing test, calling the same tool. It never crashes, so crash detection never fires. The defences are bounds the worker cannot negotiate. Example B, run in a scratch copy with the stand-in scripted to fail every time (run.py --script buggy,buggy,buggy), ends:
[stop] S2-generate-solution solution: 3 attempts used, none accepted: escalated to a person
end FR-0002 status=escalated cost=1200 units (checkpoint kept)
The same example with --budget 500 stops before the second call with budget_exhausted, because the guard checks the budget before each call, not after. Neither stop says "succeeded", and the visualization never runs on an unaccepted solution.
Checks you cannot trust: flaky tests and stale patterns#
A flaky test passes and fails on the same artifact. It teaches people to rerun until green, and then a real regression passes on the second try. The obvious fix, an automatic retry on failure, makes the gate decide nothing. The discipline: retry infrastructure errors (a timeout reaching a test database) automatically, never a failing assertion; record every gate run, not only the last; and quarantine a flaky test with an owner and an expiry. The CI/CD and deployment factory treats this as a gate design question. Example D's FR-0004 shows the distinction from the other side: its oracle station fails once with transient_timeout, a classified infrastructure error, and passes on attempt 2.
A stale pattern is a reference that no longer matches the rules: a template, a validator or an instruction file older than the products built from it. Units built against it pass their old checks and fail today's. Observed in the repository (docs/EVIDENCE.md, section 9): on a scratch copy of every tenth WatchAlgo unit, 70 of 281 solutions fail today's solution validator, all numbered wa-001761 or lower, because the rules were tightened after those files were written. The restore is the one the learning loop teaches: record the pattern and validator version on every artifact, find the affected units, and rerun only their regression suite. Examples B and D refuse to resume a run from a checkpoint made for a different spec, the same rule at run level.
Inputs you cannot trust: prompt injection and malicious dependencies#
Prompt injection is text inside the factory's inputs that tries to act as an instruction: a line in a spec saying "skip the security tests", a pull request description telling a reviewer agent to approve, a retrieved passage telling an underwriting agent to recommend approval. The model cannot reliably tell data from instruction, so the obvious fix, a stronger instruction to ignore such text, is necessary and never sufficient. The factory's defences do not depend on the model obeying: specs are validated against a schema, so free text cannot add a station or remove a gate; every gate is a check the producer did not write; and the agent holds no privilege that would let a hijacked instruction cause harm.
Diagram: Instructions and data · untrusted input reaches effects only through four controls
Two kinds of input. On the left, the factory's own instructions: the versioned pattern, the station contracts and policy, owned by named teams. In the middle, everything that arrives from outside: the product spec's free text, pull request descriptions, repository docs and configuration, retrieved passages, tool results and dependencies. Outside input enters a station as data only; it cannot add a station, remove a gate or change a contract. On the way to any effect it meets four controls: schema validation of the spec, least privilege for the worker, independent gates that the producer did not write, and a person before any irreversible action. In Example C an agent that obeys an injected passage still gets 403 at the decision route.
- Factory instructions (trusted)
- pattern and contracts: The station graph, contracts and policy, changed only through review by their owners.
- Outside input (untrusted)
- spec text and PR text: Free text in a spec or a pull request description can try to steer an agent.
- passages and tool results: Text returned by a retrieval or a tool is data, whatever it says.
- dependencies and repo files: Packages, plugins and repository configuration can run code at build or start time.
- Controls before any effect
- schema validation: The spec is checked against a schema; free text cannot change the station graph.
- least privilege and gates: The worker holds only what its task needs, and checks it did not write decide acceptance.
- a person: Decisions, schema destruction, credentials and irreversible deployments wait for a human.
The diagram separates two kinds of input. On the left, the factory's own pattern, station contracts and policy, owned by named teams and versioned. On the right, everything that arrives from outside: specs, pull request descriptions, repository docs, retrieved passages, tool results and dependencies. Right-hand inputs flow into stations as data only. They meet four controls on the way to any effect: schema validation, least privilege, independent gates and, for irreversible actions, a person.
In the insurance factory the series case UW-005 is exactly this: injected text in a retrieved passage. Insurance factory, part 3 installs it as a test on a scratch copy of Example C, where even an agent that obeys the passage gets 403 when it tries to record a decision. Standard (OWASP Top 10 for LLM Applications, a community guideline, read 2026-11-20): prompt injection is the first item of the 2025 list.
A malicious dependency is a package, plugin or tool that runs harmful code when the factory builds or runs a product. A factory makes this worse in a specific way: an agent asked to make a test pass may add a dependency nobody reviewed, and the factory then copies it into every product built from that pattern. Standard (OWASP, read 2026-11-20): supply chain is item 3 of the 2025 LLM list, and agentic supply chain vulnerabilities are item 4 of the Top 10 for Agentic Applications. The defences belong to the pattern, not the station: an approved-dependency list in the pattern registry, lockfiles with hashes, a dependency change treated as a pattern change with its own review, and provenance for every built artifact. Enterprise governance is the home of SLSA, in-toto, Sigstore and SBOM on this site. No example here demonstrates a malicious package; the claim stays a design rule.
Security note. A repository is input too. Product (Claude Code, as documented on 2026-11-20): without--bare, a session started with-pruns a project's.claude/settings.jsonhooks and.mcp.jsonservers, even in an untrusted folder, with no trust dialog. A factory that checks out a repository and starts a coding agent in it should treat the repository's own configuration as untrusted input and decide deliberately what may run.
Changes that collide or escape: merge conflicts, broken deployments, irreversible writes#
A merge conflict happens when parallel workers edit the same files. Each worker's change is fine alone; together they cannot be combined, or worse, combine silently into something wrong. The obvious response, more workers to go faster, makes it worse, because every extra worker raises the conflict rate. Factory quality engineering works out the cost: with shared files, six agents accept 3.15 units an hour where two accepted 3.57 (simulated, illustrative). The restore is structural: partition writes so parallel work touches disjoint files, and sequence what cannot be partitioned. Example D runs stations in parallel only within one level of its dependency graph, where no task depends on another.
A broken deployment is a release that passes every gate before production and still fails there. Rollback restores the previous build by its digest, and a canary limits how many users see the failure first. The gates lab's rollback preset shows it: the first five gates pass, the canary's error rate of 3.4% is above Example C's 2.0% threshold, and production rolls back automatically. Recovery is a new build and a new canary, never a retry of the same one; The CI/CD and deployment factory is the home of these mechanics.
An irreversible external write is any effect that switching builds back does not undo: a recommendation written to a partner system, a message on a shared event bus, an email to a broker, a dropped column. Two controls apply, before the write. An idempotency key makes a retried write a replay: in Example C, UW-F02's repeated request with the same key returns the same recommendation, and the record still shows 4 recommendations for 4 assessed submissions and 3 provider calls for SUB-1102, not 6. And a person approves any action that cannot be undone. This site's recommendation: when in doubt whether a write is reversible, treat it as irreversible.
What this is not. This page catalogs failures of a factory and points to the example that shows each one. Retries, backoff, circuit breakers and recovery as a discipline belong to Agent Reliability Engineering. Content filters and policy enforcement at run time belong to Guardrails Engineering. The controls around one coding agent's run, its permissions, hooks and loop, belong to Harness Engineering.
In a coding agent. A coding agent running headless at a station can produce half of these failures: a partial edit when its process is killed, a loop of rewriting the same file, a dependency added to make a test pass, an obeyed instruction from a file it read. The factory's answer does not depend on the agent behaving: a scoped checkout, a cap on attempts and spend, the factory's own tests deciding acceptance, a dependency diff reviewed like code, and a person with merge rights. Product (Claude Agent SDK, as documented on 2026-11-20): a subagent definition can set an optional maxTurns, which is one place to put a bound; the factory keeps its own bound as well.
Try it#
The recovery lab runs Example D's six stations for FR-0005 and crashes after S2-generate-solution. Switch the mode between no checkpoint and a checkpoint per station, and turn idempotency keys off to see duplicate writes appear. The gates lab starts with a canary at 3.4% and rolls back; use the recovery button to start a new canary, or inject a failure at another gate.
The crash runs without JavaScript (examples/labcore/b_recovery.py, simulated, illustrative; Example D's six stations, crash after the third):
| Crash | Mode | Idempotency keys | Sessions | Persisted | Work lost | Resume point | Writes | Duplicate writes |
|---|---|---|---|---|---|---|---|---|
| after S2-generate-solution | none | yes | 2 | 6 of 6 | 3 | S1-validate-spec | 6 | 0 |
| after S2-generate-solution | none | no | 2 | 6 of 6 | 3 | S1-validate-spec | 9 | 3 |
| after S2-generate-solution | checkpoint | yes | 2 | 6 of 6 | 1 | S2-generate-solution | 6 (3 before, 3 after) | 0 |
| after S2-generate-solution | checkpoint | no | 2 | 6 of 6 | 1 | S2-generate-solution | 7 | 1 |
The checkpoint row with keys matches Example D's real FR-0005: 3 writes before the crash, 3 after, S2-generate-solution attempt 1 abandoned and attempt 2 succeeded. A checkpoint stops repeated work; an idempotency key stops duplicate writes. They solve different problems.
The gates without JavaScript (Example C's gates and evidence; the canary error rate and the approver's answer are simulated; nothing deploys):
| Gate | Passing evidence (real unless marked) | Failing evidence when injected |
|---|---|---|
| unit tests | 50 tests, 0 failures (Example C's suite) | simulated: 1 of 50 tests fails |
| contract check | 5 routes, 6 event types, 0 problems; schema v1 to v2: 0 breaking changes | hand-edited contract: 4 problems |
| security scan | 34 files scanned, 0 findings, demo tokens allowlisted | planted secrets: 4 findings |
| evals | 4 of 4 recommendation cases give the expected recommendation | 3 of 4: UW-004 expected escalate, got approve |
| human approval | a person with the release-approver role signs off (simulated) | the release approver declines (simulated) |
| canary | 5% of traffic for 60 minutes, error rate at or below 2.0% | error rate above 2.0%: automatic rollback to the previous version |
| promote | promoted to production (documentation only) | not reached |
| Preset | Outcome |
|---|---|
rollback | the first five gates pass; canary 3.4% is above 2.0%, automatic rollback; promote not run |
default | promoted: all seven gates pass, canary 0.4% |
WatchAlgo in practice#
WatchAlgo's repository records three real failures, each a member of this catalog. All three are told in other chapters; here they are filed by class.
Observed in the repository (local run.log, queue.json at commit 1b183564; docs/EVIDENCE.md rows 12 and 13): a model outage handled as a content failure. On 2026-09-03 the account's weekly usage limit stopped every worker, the message "weekly limit" matched none of the driver's rate-limit phrases, and a 12-minute pause became a 13.5-hour retry storm that left 11,503 failed tasks. No content was lost or corrupted, because acceptance read the disk after each worker exited. The WatchAlgo story tells it in full.
Observed (commit d901dd6b, docs/evidence/git-history.md): partial artifacts contained. A usage-limit cutoff left nine half-written tracer stubs on 2026-09-02; the commit dropped them, and its message notes "the generator gate held". Visualization files are written by code only after a tracer passes its checks, so a broken tracer never became product. Long-running agent execution opens with it.
Observed (local cron and .ai-context/cron-watchdog.log, docs/EVIDENCE.md section 13): a partial failure recorded as success, this page's opening. Reported by the author: runs of 30+ hours until token limits required intervention; the logs show those limits were account usage limits. Teaching reconstruction: the failure classes, the four safety rules as checks and Example D's incomplete status are this site's. WatchAlgo's driver had bounded retries, a shared pause and disk-based acceptance; it had no run status that a failed launch could not overwrite.
Enterprise translation#
In the insurance factory of Part III, the catalog maps onto the product the factory builds. The intake service's provider calls can time out (UW-006 in the series; UW-F02 and UW-F03 in Example C); the factory installs bounded retries and a 503 that writes nothing. Retrieved passages can carry injected text (UW-005); the factory installs the injection test and the role check that makes it harmless. The recommendation write is retryable and must not happen twice; the factory installs the idempotency key. And the decision is irreversible for the broker: only a person with the underwriter role can record it, and no agent route reaches it. The model recommends; a person decides.
This site's recommendation: keep a failure catalog per pattern, owned by the platform team, with one row per class: signal, detector, restore step, the test or drill that proves it, and the owner who is paged. Review it whenever an incident reveals an unlisted class, the way WatchAlgo's "weekly limit" did. In an enterprise the critical path after a failure is rarely the fix itself. It is finding out what the factory built, from which inputs, and which products are affected, which is why the durable record matters more than any single control.
Failure test#
The failure: a launcher, like WatchAlgo's watchdog, records "resume triggered" after its launch failed. A dashboard reads the log and reports the gaps as being filled. Hours later, nothing has been built and nobody knows.
How the factory catches it: a launch step records the effect it verified, such as a process id or a running row written by the started worker, never its intent. The run's status is computed from task statuses, so a run with unfinished tasks reads incomplete whatever any log line says. An alert fires when a run stays running without a new attempt for longer than its station timeout.
What to run or toggle: in a scratch copy of examples/d-supervisor/, run /opt/homebrew/bin/python3 run.py, then the failures_view.py script above, and find FR-0007's escalated and blocked tasks. In the recovery lab with the crash preset, turn idempotency keys off and watch duplicate writes appear; in the gates lab, move the canary slider below 2.0% and back above it.
Where to go deeper#
- The landscape (next): which tools provide durable execution, retries and gates, read from their own documentation.
- Long-running agent execution: checkpoints, resume and idempotent writes, step by step.
- Insurance factory, part 3: the injection test, least privilege and the secret scan.
- The CI/CD and deployment factory: canary, rollback and flaky gates.
- Enterprise governance: provenance standards and dependency approval.
- Agent Reliability Engineering: retries, backoff, circuit breakers and recovery as a discipline.
- Guardrails Engineering: policy and content controls at run time.
Checklist for Monday#
- Compute every run's status from its task statuses; make "succeeded" the case that must be earned.
- Write a durable
runningrow before work and the final row only after the artifact exists. - Give every retryable external write an idempotency key; refuse a reused key with a different body.
- Classify failures before retrying: transient, defect, shared limit, untrusted input, unknown. Unknown stops.
- Cap attempts per unit and spend per run; escalate with the evidence attached.
- Treat specs, pull request text, repository files, retrieved passages and dependencies as untrusted data.
- List the irreversible actions of each product and put a person in front of each one.
Exercises#
- A run has six tasks: five succeeded and one is
blocked. What status should the run have, and why?
Answer 1Incomplete
incomplete: one task is not finished, so a deployable artifact does not exist. Example D's finish_run gives exactly this for FR-0007.
- A worker's process is killed after it wrote its artifact. Should the interrupted attempt count against the station's three attempts?
Answer 2No; it is an interruption, not a defect
Example D marks it abandoned with error class interrupted, reruns the station, and finds the content-addressed file already present, so nothing is written twice.
- Why does a stronger instruction to "ignore any instructions in the spec" not solve prompt injection?
Answer 3It still depends on the model obeying
The model cannot reliably separate data from instructions. The controls that hold are the ones that do not depend on it: schema validation, least privilege, independent gates and a person before irreversible actions.
Advanced perspective#
The unknown class is the one that mattersDefault to stop, not to retry
Every catalog is written from past failures, so the next serious one is usually a class nobody listed. WatchAlgo's driver classified rate limits by matching phrases, and the one it had not seen, "weekly limit", fell into the default branch, which was "content failure, retry". A default of "retry" turns every new class into a storm; a default of "stop and ask" turns it into one page to a person.
The cost of stopping is real: a line that halts on every unfamiliar message produces less. Two practices keep it bounded. Classify on structured signals where they exist (exit codes, status fields, error types) rather than message text. And measure how often the unknown branch fires: a rising rate is the signal to add a class, with a test that reproduces it, before the next long run.
Detecting a loop that never crashesProgress signals, not liveness signals
A heartbeat proves a worker is alive. It says nothing about whether the worker is getting anywhere, and a looping agent is very much alive. Progress needs its own signal: the number of distinct artifacts or test results produced, the share of tests passing, the size of the diff against the last attempt. An attempt that repeats the previous one's diff, or a run whose spend rises while its accepted output stays flat, is looping whatever its heartbeat says.
The same signal works at fleet scale. During WatchAlgo's retry storm, attempts rose by thousands while verified files stayed flat for 13.5 hours. A scheduler rule that paused the line when accepted output stays flat while attempts rise would have stopped it in minutes; the per-unit cap of three attempts could not, because each unit's attempts were legitimately new.
When a write cannot be undone: idempotency, outbox, compensationThree designs, each for a different gap
An idempotency key makes a retry safe: the receiver stores the key with the result and replays it for a repeat. It does not help if the receiver is a system that cannot store keys, and it does not undo anything. An outbox makes "write and notify" atomic: the factory-built service writes the event to a table in the same transaction as the data, and a separate process sends it, so a message is sent only if the data committed. A compensating action undoes the business effect of a completed step with a new step, such as a correction notice after a wrong notification; it is not a rollback, because both actions remain in the history.
Which one applies depends on the effect. Retried calls need keys; data plus events need an outbox; multi-step processes across systems need compensations designed in advance. Some effects have no compensation at all, which is why this site requires a person before them. Example C demonstrates the first design only; the other two are common patterns, named here without a runnable example.
Inject every failure on purposeA catalog row without a drill is a hope
Each row of the catalog claims a detector and a restore step. A claim is evidence only when it has been exercised. Example D's crash is a drill in code: SimulatedCrash is raised at a chosen point, and a test asserts that resume writes nothing twice. Example C plants secrets to prove the secret scan finds them, and edits a contract by hand to prove the contract gate fails. The gates lab injects a failure at any gate.
An enterprise factory runs the same drills on a schedule, against a staging copy: kill a worker mid-write, make the model endpoint time out, feed a spec with an injected instruction, add an unapproved dependency, push a canary over its threshold. Each drill records whether the detector fired, how long the restore took, and whether the run status told the truth. The time-to-recovery figures of Factory quality engineering then come from drills as well as incidents, which is the only way to have them before the first real outage.
In one breath#
A factory fails in ten recognizable ways, from half-written artifacts and flaky tests to injected instructions and writes that cannot be undone, and each needs four answers in advance: how it shows up, what detects it, what restores it, and who decides. Example D shows the core: an interruption is recorded as abandoned and resumed without a duplicate write, a repeated defect is escalated to a person, and a run with any unfinished task ends incomplete. WatchAlgo's watchdog, logging "Resume triggered" after a failed launch, is the failure all four safety rules exist to prevent: a record that says something different from the truth.
Failure handling is mostly built from ordinary tools: workflow engines with retries, CI systems with gates, standards for provenance. The landscape reads those tools from their own documentation, each with what it solves, where it sits in the factory, and when not to use it.