All chapters
  1. Start hereA software factory, from first principles
  2. Primitive · 1A factory, before AI: input, transformation, check, output
  3. Primitive · 2From repetition to a pattern: what stays, what varies, what to leave alone
  4. Primitive · 3The specification as an executable contract
  5. Primitive · 4Build the smallest factory: four stations, two specs, one manifest
  6. Primitive · 5Assembly stations: typed contracts, owners and failure handling
  7. Primitive · 6AI enters the factory: what a model adds, and what stays deterministic
  8. WatchAlgo · 7The WatchAlgo production story: what the repository shows
  9. WatchAlgo · 8A problem as a production unit: the files one unit must contain
  10. WatchAlgo · 9Station by station: description, solution, implementations, visualization
  11. WatchAlgo · 10The visualizer factory: from algorithm state to renderable steps
  12. WatchAlgo · 11Three flavors and three languages: two different contracts
  13. WatchAlgo · 12Looping over a large backlog: scheduling, limits and a failed-case queue
  14. WatchAlgo · 13Long-running agent execution: context, checkpoints and token limits
  15. WatchAlgo · 14Evidence of correctness: tests the producer cannot talk its way past
  16. WatchAlgo · 15The factory learning loop: defects change the pattern, not only the unit
  17. Enterprise · 16Classify the enterprise demand before choosing a pattern
  18. Enterprise · 17Reference enterprise patterns and their variation points
  19. Enterprise · 18Platform foundations: golden paths, templates, registries and catalogs
  20. Enterprise · 19Pattern versus configuration versus business logic
  21. Enterprise · 20Insurance factory, part 1: building the intake contract
  22. Enterprise · 21Insurance factory, part 2: building the underwriting workflow
  23. Enterprise · 22Insurance factory, part 3: trust and observability built in
  24. Enterprise · 23Data and integration engineering: contracts that survive change
  25. Enterprise · 24The CI/CD and deployment factory: from pull request to production
  26. Enterprise · 25Enterprise governance: owners, approved patterns and provenance
  27. Enterprise · 26Scaling safely: many teams, many agents, one set of boundaries
  28. Operations · 27Factory quality engineering: measuring accepted output, not activity
  29. Operations · 28Failure and recovery scenarios: what breaks a factory, and what restores it
  30. Operations · 29The landscape: which tool for which job, read from its own documentation
  31. Operations · 30Design an enterprise software factory: the blueprint and the rollout
  32. ReferenceSources and verification
  33. ReferenceGlossary
  34. AboutAbout this site
WatchAlgo · 15

The factory learning loop: defects change the pattern, not only the unit

A defect taxonomy, rework causes, versioned pattern changes, the blast radius of a template edit, and measuring the effect on the next batch.

You will sort
Defects into classes with a station and a cause, so one fix can retire a whole class instead of one instance.
You will release
A pattern change as a versioned release with a computed blast radius and a selective regression suite.
You will measure
Whether the next batch is better with a fair comparison, and name what confounds the raw number.

On the evening of 2 September 2026, WatchAlgo's queue pipeline started recording failures that took zero minutes. Every worker exited with "You've hit your monthly spend limit", the verifier found no new file, and each exit counted as a failed attempt. 964 tasks failed in about seventeen minutes. The author fixed it the same evening: the pipeline's list of rate-limit phrases gained "spend limit" and "session limit", and 1,104 failed tasks were put back in the queue by hand.

The next night, at about 23:05, the account reached its weekly limit. The message was "You've hit your weekly limit". It matched none of the phrases on the list, so the same thing happened again, this time for about thirteen and a half hours, until the queue file held 11,896 tasks, 11,503 of them failed. No content was lost or corrupted, because acceptance read the disk. But the fix of the first night had repaired the message seen that day. It had not repaired the reason the failure was possible.

The learner's question: when the factory finds a defect, how do you fix the thing that produced it rather than the one artifact or message in front of you, release that fix safely to every product built from the pattern, and prove the next batch is better?

From one defect to a defect class#

Plain English. Rework repairs one bad artifact. Learning repairs the line, so the next thousand artifacts do not need the same repair.

The previous chapters gave the factory a way to catch defects: independent checks at every station (Evidence of correctness). Catching is half the loop. The other half starts with a question a bug report rarely asks: what kind of defect is this, and where did it enter?

A defect taxonomy is a short, fixed list of defect classes that every recorded defect is sorted into, so that counts per class show where the line leaks. A tiny example: "wrong expected output", "wrong algorithm", "shape drift" and "unclassified exit" are four classes; the 86 solution defects in WatchAlgo's own log fall almost entirely into the first two. The operational consequence is that a class with many members points at a station worth changing. The trade-off: a taxonomy with too many classes spreads defects thinly and hides patterns; one with too few lumps different causes together.

A rework cause is the reason an artifact was sent back, recorded with the rework: which check failed, at which station, and which input or instruction let the defect through. A defect class says what went wrong; a rework cause says why this station let it happen. "Expected output wrong, because the solution validator checks shape and wording but never runs the code" is a rework cause. It names its own fix.

Before AI. Manufacturing calls this a Pareto analysis: count defects by cause, fix the few causes that produce most of them. Software teams do the same with bug-tracker components and root-cause fields in a postmortem. The factory version adds one field those tools often lack: the pattern version that produced the defect, so a fix can be tied to a release.

The table sorts WatchAlgo's recorded defects this way. The "where to fix" column is this site's recommendation, taken from the teaching contracts in docs/evidence/station-contracts.md; the rest is observed.

Defect classWatchAlgo instance (observed)Caught atRework causeWhere to fix the class
wrong expected outputabout 80 of the 86 solution defectsvisualization station, by running the JS in Nodethe solution validator never runs the codesolution station: the validator runs the code on every test input
wrong algorithmabout 20 of the 86the same Node cross-checksamesame
duplicate unitwa-000379 duplicates wa-000378; one solution copied from another problemwhile writing solutions; the cross-checkno uniqueness check on titles or coderegistry: a duplicate check before a unit enters the queue
wrong example in a problem15 problem exampleswhile writing solutionsthe problem validator checks structure, not outputsproblem station: execute each example against a reference
unclassified exit"weekly limit" read as a content failurenot caught; it caused the stormthe failure classes were a phrase listacceptance: an exit that produced nothing and matches no class is "unknown" and pauses
validator drift70 of 281 sampled solutions fail today's validatora re-run on a copyrules tightened after files were acceptedrecord the validator version per unit; see below

Read the last column against the second. Every row is a class, and every fix in the last column changes a station, a validator or the pattern, never one file.

The smallest primitive: a defect record#

Teaching reconstruction. A defect becomes useful to the loop when it is a record, not a sentence in a log. This is the least a record needs; the field names are this site's own.

{
  "defect_id": "DEF-0042",
  "unit": "wa-000417",
  "found_at_station": "S2b-visualization",
  "introduced_at_station": "S1-solution",
  "class": "wrong_expected_output",
  "rework_cause": "solution validator checks shape, never runs the code",
  "found_by": "node cross-check of the JS solution",
  "pattern_version": "PAT-ALGO-UNIT 1.3.0",
  "validator_version": "validate_payload 2026-03",
  "factory_run": "FR-0007"
}

The unit id and run id are illustrative. Two fields carry the loop. introduced_at_station differs from found_at_station, which is the normal case: the defect entered at the solution station and was found two stations later. And pattern_version lets someone ask, after a fix ships, whether defects of this class still come from the new version or only from units built under the old one.

A pattern change is a release#

Suppose the class "wrong expected output" is fixed the right way: the solution station's validator now runs the reference code on every test input. That is a change to a shared part of the pattern. Every product built from that pattern is touched by it, and some of them were built months ago under the old rule.

A pattern version is the identifier of one released state of a pattern, recorded in every manifest built from it, so any artifact can be traced to the exact templates, validators and station graph that produced it. Example A already does this: its pattern file says "version": "1.3.0", and station 4 records a digest over the pattern and every template it names (From repetition to a pattern introduced it).

Semantic versioning is a convention for what a version number promises. Standard (Semantic Versioning 2.0.0, semver.org, read 2026-11-20): increment MAJOR for incompatible API changes, MINOR for added functionality that stays compatible, PATCH for compatible bug fixes; software using it must declare a public API; versions 0.y.z are initial development, where anything may change; and the contents of a released version must not be modified, so any change needs a new version.

The phrase "public API" needs a translation for a pattern. A pattern's public API is everything a product spec or a downstream consumer relies on: the extension points a spec fills, the fields of the audit record, the event names, the shape of the test harness. Renaming an audit field is incompatible for anyone reading audit records, so it is MAJOR. Adding a field to every log line is compatible, so MINOR. Making expired tokens fail the same way on every route is a fix, so PATCH. The rule is in the blast lab's own logic, examples/labcore/b_blast.py : bump, quoted verbatim:

BUMP = {"breaking": "major", "additive": "minor", "fix": "patch"}


def bump(version: str, kind: str) -> str:
    major, minor, patch = (int(x) for x in version.split("."))
    b = BUMP.get(kind, "patch")
    if b == "major":
        return f"{major + 1}.0.0"
    if b == "minor":
        return f"{major}.{minor + 1}.0"
    return f"{major}.{minor}.{patch + 1}"

Notice what the function cannot decide: the kind. A person classifies a change as breaking, additive or a fix, and that judgement is where pattern releases go wrong. A field rename called "a fix" ships as a patch and breaks a consumer that pinned the major version.

Diagram: The learning loop · from one defect to a better next batch

Three lanes, read left to right and then back to the start. In the find lane, an independent check finds a defect and records it with a class, the station that let it in, and the pattern version. In the fix lane, a class that keeps recurring is fixed in the pattern, released as a new semantic version, and its blast radius is computed from the catalog. In the prove lane, the selective regression suite runs for every affected product and consumer, and the next batch is compared with the last one on the same unit mix and the same checker. Its defects start the next turn of the loop.

  1. Find
    1. independent check finds a defect: A check the producer did not write rejects an artifact.
    2. record class, station, cause: Sort the defect into a fixed class and record why the station let it through.
  2. Fix the class
    1. fix in the pattern, not the artifact: A recurring class is fixed where it enters, so later units never need the repair.
    2. release a new pattern version: The version number states how far the change may reach.
    3. compute the blast radius: List the products, consumers and approvals the change touches.
  3. Prove it
    1. selective regression: Run the tests that exercise the changed part, plus consumers' contract tests.
    2. measure the next batch fairly: Compare per category or on a fixed mix, with the validator version held constant.

Follow the loop from the top left. A check finds a defect and records it with a class and a cause. When a class keeps recurring, the fix goes into the pattern, not into the artifact. The fix is released as a new pattern version, and the version bump says how far it may reach. The blast radius lists the products that receive it, the selective regression suite proves they still pass, and the next batch built under the new version is measured against the last one, fairly. Its defects start the next turn.

Blast radius and selective regression#

The blast radius of a change is the set of products, consumers and approvals it touches: who must rebuild, who must re-test, who must sign. Selective regression is running only the tests that exercise the changed part, plus the contract tests of anyone who consumes it, instead of every test of every product.

The blast lab computes both over a synthetic catalog. Teaching reconstruction: eight products on three pattern lines; only the names marked as coming from Examples A and C are real, every test count and owner is the lab's own. The rule, as the module's docstring states it: a product is affected when it pins a version of a touched pattern with the same major version as the current one, and that version ships the changed component; products pinned to an older major line keep their version until they upgrade. The script below calls the module's radius function for one breaking change; it was run on a scratch copy of examples/labcore/.

audit-field-rename: breaking -> major
   pattern PAT-EVENT-WORKER 1.4.0 -> 2.0.0
   pattern PAT-INTAKE-API 2.1.0 -> 3.0.0
   affected broker-submission-intake (2.1.0): 3 of 50 tests
   affected claim-notice-intake (2.1.0): 3 of 40 tests
   affected endorsement-request-intake (2.0.0): 2 of 29 tests
   affected policy-event-relay (1.4.0): 2 of 24 tests
   affected audit-export-job (1.4.0): 5 of 18 tests
   consumer audit-export-job reads from 3: 4 contract tests
   not affected loss-run-request-intake: pinned to PAT-INTAKE-API 1.4.0, an older major line; it keeps that version until it upgrades
   not affected running-sum: PAT-ALGO-UNIT does not ship audit_record
   not affected clamp-list: PAT-ALGO-UNIT does not ship audit_record
   suite 19 of 206; approvals: 8

Three lines repay a slow read. The suite runs 19 of 206 tests: the tests tagged with the audit component in each affected product, plus the 4 contract tests of the export job, which reads the audit records of three intake products. Selection saves most of the run, and it is only as good as the tags. Second, loss-run-request-intake is not affected, because it pins the older major line 1.4.0. That is correct today and a debt tomorrow: the day it upgrades, the rename reaches it and the export job that reads its records. Third, a breaking change needs eight approvals, one from each owner it touches plus the compliance reviewer, because the audit record is a governed component. A patch to the auth middleware needs three.

Diagram: Blast radius of one breaking change · audit field renamed

Rows are the pattern lines, the products that receive the change, and the products that do not. Columns are the intake pattern, the event-worker pattern and the algorithm pattern. The audit record component is renamed, a breaking change, so both patterns that ship it get a new major version. Every product pinned to their current major line is affected and runs its tagged tests; the audit export job also runs four contract tests because it reads the intake products' audit records. The loss-run intake pins the older major line 1.4.0 and is not affected until it upgrades; the algorithm products never ship an audit record. The catalog is the blast lab's own synthetic one.

  1. Pattern line
    1. PAT-INTAKE-API 2.1.0 to 3.0.0: Its current version ships the audit record, so the breaking change raises the major number.
    2. PAT-EVENT-WORKER 1.4.0 to 2.0.0: A second pattern line that shares the audit component.
  2. Affected
    1. 3 intake products: Products pinned to 2.x that ship the audit record.
    2. relay + export job: The export job also reads the intake products' audit records, so its contract tests run.
  3. Not affected
    1. loss-run intake: Not affected today; receives every change at once when it upgrades.
    2. running-sum, clamp-list: Their pattern does not ship an audit record.

The diagram lays the same result out by lane. The changed component sits in the pattern lane; arrows reach the products that ship it on the current major line and the consumer that reads its output. The older major line and the algorithm pattern stay grey: not affected, each with its reason.

Measuring the next batch, fairly#

The pattern now carries the fix. Did it work? The tempting measurement is to compare first-pass yield, the share of units accepted on their first attempt, in the batch before the change and the batch after. Here is why that comparison lies, run as code. The numbers are simulated, illustrative; no real batch is described.

BEFORE = {"pattern": "PAT-ALGO-UNIT 1.3.0",
          "units": {"arrays": 30, "hashing": 70},          # units per category in the batch
          "first_pass": {"arrays": 27, "hashing": 42}}     # units accepted on attempt 1
AFTER = {"pattern": "PAT-ALGO-UNIT 1.4.0",
         "units": {"arrays": 70, "hashing": 30},
         "first_pass": {"arrays": 64, "hashing": 19}}


def raw_yield(batch):
    return sum(batch["first_pass"].values()) / sum(batch["units"].values())


def yield_on_mix(batch, mix):
    """Per-category yield of `batch`, weighted by another batch's category mix."""
    total = sum(mix.values())
    return sum(mix[c] / total * batch["first_pass"][c] / batch["units"][c] for c in mix)


for b in (BEFORE, AFTER):
    per_cat = {c: round(b["first_pass"][c] / b["units"][c], 3) for c in b["units"]}
    print(f"{b['pattern']}: raw first-pass yield {raw_yield(b):.2f}  per category {per_cat}")

print(f"raw change:               {raw_yield(AFTER) - raw_yield(BEFORE):+.3f}")
same_mix = yield_on_mix(AFTER, BEFORE["units"]) - yield_on_mix(BEFORE, BEFORE["units"])
print(f"change on the before mix: {same_mix:+.3f}")
PAT-ALGO-UNIT 1.3.0: raw first-pass yield 0.69  per category {'arrays': 0.9, 'hashing': 0.6}
PAT-ALGO-UNIT 1.4.0: raw first-pass yield 0.83  per category {'arrays': 0.914, 'hashing': 0.633}
raw change:               +0.140
change on the before mix: +0.028

The raw yield rose by 14 points. Almost all of it is the batch, not the fix: the second batch held mostly easy array problems, the first mostly hard hashing problems. Hold the category mix fixed and the improvement is about 3 points. Both batches are real batches of the same factory; only the comparison changed.

A confound is anything other than the change that differs between the two batches and moves the measure. For a factory, the usual ones are these:

ConfoundHow it moves first-pass yieldHow to control it
unit mixeasier units raise yield with no fix at allcompare per category, or weight to one fixed mix
the checker changeda stricter validator lowers yield; a looser one raises itrecord the validator version; re-run the old batch under the new checker
the model or worker changeda model switch moves yield on its ownchange one thing per release, or record both and split the batches
capacity eventslimit pauses and retry storms turn into failed attemptscount only attempts that ran; classify exits first
small batchesa few units swing the rate by pointsreport the rate with its unit count; repeat over several batches
escaped defects found latertoday's yield counts defects nobody has found yetre-measure the batch after downstream checks run

The second row deserves emphasis because it fools teams that measure carefully. If the fix itself is a stricter validator, first-pass yield will fall after the release, because the line now rejects defects it used to accept. That fall is the improvement. The fair measure is then escaped defects per accepted unit, found by an independent check run on both batches, not yield alone.

Common confusion. Rework in this site's sense is work sent back by a station's check inside one factory run. Guideline (DORA, read 2026-11-20): DORA's deployment rework rate, added in 2024, is the share of deployments that are unplanned bug-fix work, an instability measure of a delivery process. The two share a word and differ in denominator: station attempts here, production deployments there. Factory quality engineering keeps both apart.

One reference pattern or a family of patterns?#

This page owns one of the site's decision trees. Every class fix raises it: should the fix go into the one shared pattern, or is it time to split the pattern into a family?

The question. Should one versioned reference pattern serve every product of a kind, or should a family of related patterns each serve a narrower kind?

OptionChoose it whenIt costsIt fails when
One pattern, one version lineproducts share their station graph, checks and public API; changes are mostly fixesevery change has the full blast radius; one owner becomes a queueproduct needs pull it in opposite directions and switches pile up
One pattern with declared variantsa few products need an optional station or check, and the rest is sharedthe validator must model the variants, or they look like defectsvariants are undeclared, as in validator drift
A family of patterns sharing componentskinds differ in station graph or risk tier, but reuse audit, auth or loggingcomponents need their own versions; the blast radius crosses patternsthe shared components are copied, not versioned, and drift apart
Separate patterns, no sharingkinds share almost nothing, or are owned by unrelated teamsduplicated plumbing and duplicated fixesa class fix has to be made five times, and one is forgotten

The criteria, in order: do the products share a station graph; do they change for the same reasons and on the same schedule; who owns each part; and how expensive is a missed fix compared with a wide blast radius. The blast catalog is a family: three pattern lines share components such as audit_record and logging_format, which is why one rename touched two patterns and five products.

The counterexample. WatchAlgo looks like one pattern, and for the problem and solution facets it is. Its visualization station is the exception: in April a model wrote the visualization JSON, checked for shape only; in September the station was redesigned so that a model writes a tracer program and code writes the JSON. That is a major version of one station, and both versions' outputs still sit side by side in the corpus (2,144 visualizations with a tracer, 236 without). A single pattern with no recorded version per unit cannot tell them apart.

The failure modes. Too few patterns: switches multiply until the pattern is a configuration language nobody can test in all combinations. Too many: a class fix lands in one pattern and not its siblings. Unrecorded versions: every option fails, because no one can compute the blast radius.

In a coding agent. When a coding agent staffs a station, its instructions file is part of the pattern and belongs under the same version as the templates and validators. A changed instruction is a pattern change with a blast radius: the factory's own checks, not the agent's account of its work, decide whether the next batch improved. The controls around the agent's run are Harness Engineering.
What this is not. This page versions and measures the factory's pattern. Designing graders, golden sets and release quality bars is Eval Engineering; the factory invokes them as gates. Retry and recovery as a discipline is Agent Reliability. Releasing one change of one product through CI is Diff to Deploy.

Try it#

Pick a shared template change and read its semantic version bump, the products it reaches, the selective regression suite and the approvals; the lab also lists what is not affected and why. Preset default opens on this page's catalog.

The table holds the same results for a reader without JavaScript. It was written for this page from a scratch run of examples/labcore/b_blast.py : radius over the lab's own synthetic catalog (docs/LABS.md has no plain table for this lab yet). Counts are of the catalog's synthetic tests.

ChangeKind and bumpPatterns touchedProducts affectedNot affected (reason)Tests selectedApprovals
Logging format: every line gains trace_idadditive, minorPAT-INTAKE-API 2.1.0 to 2.2.0; PAT-EVENT-WORKER 1.4.0 to 1.5.05loss-run intake (older major line); running-sum, clamp-list (pattern lacks it)7 of 2063, incl. security reviewer
Auth middleware: expired tokens rejected alikefix, patchsame two, to 2.1.1 and 1.4.15same three15 of 2063, incl. security reviewer
Audit record: actor renamed principalbreaking, majorsame two, to 3.0.0 and 2.0.05, plus the export job as consumersame three19 of 206 (4 contract tests)8, incl. compliance reviewer
Test harness bump, same assertionsfix, patchall three, incl. PAT-ALGO-UNIT 1.3.0 to 1.3.17loss-run intake (older major line)9 of 206 (0 in five products)3
Idempotency keys kept 7 daysadditive, minorintake and event-worker patterns5same three16 of 2062

No model takes part in this lab or in the examples it draws on; where an example puts an agent at a station, the agent there is a scripted stand-in, so these results show how the factory mechanics work, not how good any model is.

WatchAlgo in practice#

WatchAlgo is the last Part II chapter's case and its best evidence for this page, because the repository holds the factory's own defect logs and both kinds of fix.

Observed in the repository (.ai-context/VIZ_SOLUTION_BUGS_FOUND.md, commit 4426f119): 86 solution.json defects in one category, Hashing & Prefix (516 units), found when the September visualization generator ran each JavaScript solution in Node: about 80 wrong expected outputs, about 20 wrong algorithms, and one solution copied verbatim from another problem. Observed (.ai-context/SESSION_STATE.md, commit 0309048f): 15 problem examples with wrong outputs, found while writing solutions; wa-000379 duplicates wa-000378, both "Two Sum"; 13 units blocked from visualization by broken reference code. These are a defect taxonomy in all but name. The repository does not show the class fix for the first two: as observed, the solution validator still checks shape and wording, not outputs.

Observed (docs/evidence/git-history.md section 6; commit 971863f2): the instance fix described at the top of this page. The commit message says the "rate-limit regex covers spend/session limits"; the next limit message, "weekly limit", was not covered, and the pipeline's pattern still lacks it. A class fix would have changed the default: an exit that produced nothing and matched no known class pauses the line instead of spending an attempt.

Observed, with a correction (docs/EVIDENCE.md, metrics inventory, validator drift): 70 of 281 sampled solutions fail today's validate_payload. Most are files accepted before the rules were tightened. Some are not drift at all: wa-001378 declares its brute-force approach as not applicable, with a reason, and the validator does not model that variant, so it raises KeyError('complexity'). And failures exist outside the sample above wa-002134, for example wa-002157, committed in a September checkpoint. Without a validator version per unit, "broken", "older than the rule" and "a declared variant" all look the same. That is the blast-radius problem in reverse: the rule changed and nobody could compute which units it touched.

Reported by the author, as an estimate (.ai-context/SESSION_STATE.md): from batch 15 the solution sub-agents switched from Opus to Sonnet, recorded as about 25K tokens per 5 problems against about 170K, "7x fewer tokens, identical quality". The repository holds no per-task token log and no quality comparison between the batches, so "identical quality" is a judgement, not a measurement; a fair test would compare escaped defects per category between batches 1 to 14 and 15 onward.

Teaching reconstruction: this site's teaching contracts propose the class fixes the logs motivate: run every example against a reference at the problem station, run the code in the solution validator, record the validator version per unit, and classify every exit as ok, content failure, limit or unknown.

What generalized and what was domain-specific. The loop's mechanics generalized: defects found by a check the producer did not write, sorted by class, fixed at the station that let them through, with the work list rebuilt from disk so a fix can be re-run over old units. What was domain-specific is the cheap oracle. An algorithm problem has one right answer that running code can confirm, which is why 86 defects could be found by a machine at all; most enterprise work has no such oracle, and the loop there leans on contract tests, golden cases and people. WatchAlgo is one content factory, not an enterprise delivery process. The enterprise blueprint holds the full table of where the analogy holds and where it fails.

Enterprise translation#

In an organization the loop crosses team lines. A domain team finds the defect; the platform team owns the pattern that produced it; other domain teams receive the fix. Three practices keep that safe. Defect records carry the pattern version and the station, so the platform team sees classes across products, not one team's complaint. Every pattern release states its kind, computes its blast radius from the catalog of which product pins which version, and runs the selective suite plus consumers' contract tests before anyone upgrades. And owners of the products on an older major line get a dated deprecation, because "not affected" today is a migration later.

For the insurer's intake services, a change to the audit record of PAT-INTAKE-API is the expensive case: the field is read by compliance tooling, the component is governed, and the approvals multiply. What dominates the timeline is rarely writing the change; it is the approvals and the consumers' contract tests. Enterprise governance owns the approval roles, and Platform foundations owns the catalog that makes the blast radius computable.

Failure test#

The failure: a fix to the instance. A defect is repaired where it was seen, the class stays open, and the next member of the class fails the same way. WatchAlgo's limit phrases are the recorded case.

How the factory catches it: by classifying with a safe default. The script below applies the phrase list recorded in docs/evidence/git-history.md to the three limit messages the worker logs ended with, once as an instance fix (unmatched means content failure) and once as a class fix (an exit that produced nothing and matches no class is unknown, which pauses the line). It was written for this page and run here.

import re

# The phrase list as recorded in docs/evidence/git-history.md section 6 (after commit 971863f2).
LIMIT = re.compile(r"usage limit|rate limit|rate limited|out of extra usage|spend limit|session limit"
                   r"|429|overloaded|capacity|quota|temporarily limiting", re.I)


def classify_instance(text):
    return "limit" if LIMIT.search(text) else "content_fail"


def classify_class(text, produced_file):
    if LIMIT.search(text):
        return "limit"
    return "content_fail" if produced_file else "unknown"


for m in ["You've hit your monthly spend limit", "You've hit your session limit", "You've hit your weekly limit"]:
    print(f"{m:38} instance fix: {classify_instance(m):13} class fix: {classify_class(m, produced_file=False)}")
You've hit your monthly spend limit    instance fix: limit         class fix: limit
You've hit your session limit          instance fix: limit         class fix: limit
You've hit your weekly limit           instance fix: content_fail  class fix: unknown

What to run or toggle: run the script, then add a fourth message the list has never seen. In the lab, pick the test harness bump and read the suite: five products get zero selected tests, because no test is tagged with the harness. The folded block "When selective regression selects nothing" below explains why that change must run the whole suite.

Where to go deeper#

Checklist for Monday#

Exercises#

  1. A validator gains a rule that every test case must state its category. Which version bump, and what happens to first-pass yield on the next batch?
Answer 1Major for the check, and yield falls

Units that passed before can now fail, so for anyone relying on "accepted under this pattern" it is incompatible: a major version, or a minor one with the rule in warning mode first. Yield falls on the next batch because the line rejects what it used to accept; measure escaped defects instead.

  1. In the lab, why is loss-run-request-intake never affected?
Answer 2It pins an older major line

It pins PAT-INTAKE-API 1.4.0 while the current line is 2.x. A release on the current line does not reach it until it upgrades, and then every change made since reaches it at once.

  1. Batch 15 used a cheaper model and its first-pass yield matched batch 14. Is quality identical?
Answer 3Not shown yet

Not from that number. Compare per category, check that the validator was the same version, and count escaped defects found later by an independent check in both batches.

Advanced perspective#

When selective regression selects nothingTags pick tests; the harness runs them all

Selective regression in the lab picks tests by tag: a test tagged with the changed component is selected. For the test harness bump that rule selects zero tests in five products, because no test is tagged "harness": every test depends on it. The scratch run printed affected broker-submission-intake (2.1.0): 0 of 50 tests for that change.

The general lesson is that tag-based selection works for components that some tests exercise and fails for components that all tests depend on: the test runner, shared fixtures, the base image, the clock. Those changes need the whole suite, run twice: once on the old harness and once on the new, with the results compared test by test. Any test whose verdict changed is a harness defect, not a product defect. Build tools that select tests by dependency graph rather than by tag avoid part of this gap, but only when the harness itself appears in the graph.

Older major lines are deferred blast radiusNot affected today, all at once later

Pinning an older major line protects a product from a breaking change, and it accumulates every change that happens on the current line. When loss-run-request-intake upgrades from 1.4.0, it receives the idempotency key, event emission, the trace id, the logging change and the audit rename in one step, and the export job that reads its audit records breaks on that day, not on the day of the rename.

Two governance habits keep this manageable. Give every older major line a support window with a date, published in the catalog, so its owners plan the upgrade. And keep consumers' contract tests running against every line still in use, so the export job fails in CI when it meets a principal field from one producer and an actor field from another, not in production.

Class fix or rework: an economic choiceRepair cost per unit against one release with approvals

Reworking units one by one costs a little per defect, forever. A class fix costs a release: design, a version, the selective suite, approvals and possibly a re-run over old units. The class fix wins when the class keeps recurring and the release is cheap; it can lose when the class is rare and the component is governed, since an audit change needed eight approvals in the lab and a logging change three.

A practical rule, offered as this site's recommendation: a class that appears in more than one batch gets a class fix proposal; a class whose instances escaped to users gets one at once, whatever its count. Record the decision either way, so the next person who meets the class sees that it was considered.

How big must a batch be to see an effect?Small batches swing by points on their own

A batch of 30 hashing problems with a true first-pass yield of 0.6 will show anything from about 0.42 to 0.78 in most batches, purely by chance (two standard deviations of a binomial share: 0.6 plus or minus 2 times the square root of 0.6 times 0.4 divided by 30, about 0.18). The numbers are illustrative arithmetic, not a measured batch. An improvement of 3 points is invisible in one such batch. Either aggregate several batches per pattern version, or run a held-out replay: rebuild a fixed sample of old units under the new version and compare unit by unit, which removes the mix confound entirely and halves the noise.

A replay has its own trap in a factory with a model at a station: the model's output varies between runs, so a replay compares one draw against one draw. Replay each unit more than once, or compare at the class level rather than per unit.

In one breath#

A factory learns when defects become records with a class, a station and a cause, and when a recurring class is fixed in the pattern rather than in the artifact or the one message in front of you. The fix ships as a semantically versioned release whose blast radius, selective regression suite and approvals are computed from a catalog of who pins which version. Whether the next batch is better is a fair comparison, per category or on a fixed mix, with the checker's version held constant; WatchAlgo's logs show the defects, an instance fix that failed the next night, and validator drift that no version record could explain.

This closes Part II and the algorithm factory. Part III turns to an enterprise, where the first problem is not a defect but a request nobody has fully described yet. Classify the enterprise demand shows what the factory must learn about users, data, systems, security, compliance and maintainers before it chooses a pattern, and where the critical path is not code generation at all.