Example B's oracle checks a generated lower_bound on 30 cases: 10 hand-written fixtures and 20 seeded random ones. AI enters the factory ran a third implementation against it, a variant written for the verify lab that returns any index holding the target instead of the first one. The oracle caught it with 28 of 30 passing. The two failures were both hand-written fixtures, duplicates and negatives; none of the 20 random cases happened to contain a repeated target.
Now imagine an ordinary edit. Someone tidies fixtures.json and removes the two cases that look redundant. This page's writer ran exactly that in a scratch copy of Example B:
0. Example B's oracle (10 fixtures + 20 seeded random cases), then without two fixtures
dupes, all fixtures 28/30 passed failed=['duplicates', 'negatives']
dupes, minus duplicates, negatives 28/28 passed failed=[]
A wrong function, a perfect score. Nothing about the oracle's code changed; its evidence did. A test suite made of examples is only as strong as the examples someone thought to write, and a factory that produces thousands of units cannot rely on someone thinking of the right ones each time.
The learner's question: what counts as evidence that a generated artifact is correct, and how does a factory get evidence that depends neither on the producer's word nor on the luck of which examples were written down?
Evidence, not a report#
Plain English. A test is evidence when someone other than the producer wrote it and it can fail. The more ways a test can fail on a wrong answer, the stronger the evidence when it passes.
Beginner definition: evidence of correctness is the result of a check, independent of the producer, that a wrong artifact would fail.
AI enters the factory set the rule that frames this page: a self-report of "done" is not evidence; a test the producer did not write is. This chapter asks the next question: which tests, and how strong.
A test oracle is whatever decides the right answer for a test. It can be a list of known outputs, a slow implementation that is easy to trust, or a rule that any right answer must satisfy. Example B's oracle is oracle.py : brute_force, a linear scan that returns the first index whose value is at least the target. The analogy is the answer key a teacher grades with. The consequence is that every test is as good as its oracle: an oracle built by the same producer, from the same misunderstanding, agrees with the producer's mistakes. The trade-off is cost: a good oracle can take as much thought as the solution.
Before AI. Compiler teams have long kept test suites of programs with known output, and some test a new compiler against an older one on generated programs. Database teams compare query results across engines. Payment teams replay recorded traffic against a new version and diff the responses. Every technique on this page predates language models; what changed is the volume of artifacts that need them and the confidence with which wrong ones arrive.
Seven kinds of evidence#
The table names each technique this page demonstrates or describes, and what it needs. The sections that follow define each one with a tiny example.
| Technique | Needs a full oracle? | Catches | Misses | Who writes it |
|---|---|---|---|---|
| Example tests | yes, per example | the cases someone thought of | everything else | spec author or pattern team |
| Property test | no: a rule any right answer obeys | violations on many inputs | errors the rule does not describe | pattern team |
| Randomized test | depends on what checks each input | cases nobody thought of, if the generator reaches them | regions the generator never visits | pattern team |
| Differential test | a second implementation | any disagreement | errors both share | an independent producer |
| Metamorphic test | no: a relation between two runs | broken relations, without knowing either answer | errors that preserve the relation | pattern team |
| Trace consistency | invariants on intermediate states | a wrong path to a right answer | errors in what is not traced | pattern team |
| Human review | a person's judgment | meaning, teaching value, fit to purpose | whatever the reviewer did not sample | a named reviewer |
A property test checks a rule that every correct answer must satisfy, on many inputs, without computing the answer itself. For lower bound the rule is complete: the result i lies between 0 and the length of the list, every value before i is smaller than the target, and every value from i on is at least the target. A randomized test generates its inputs from a seeded random generator, so it reaches cases nobody wrote down while staying repeatable. A differential test runs two independent implementations on the same inputs and treats any disagreement as a defect in one of them. A metamorphic test checks a relation between two runs: change the input in a known way, and the output must change in a known way, even when neither output is known. For lower bound: put a value smaller than the target at the front of the list, and the answer must grow by exactly one.
The executable demonstration: five checks on one function#
The script below was written for this page and run with Python 3.14 next to a scratch copy of Example B on 2026-11-20. It uses Example B's two stand-in sources (generator.py), its oracle (oracle.py) and its trace checker (visualize.py : check_invariants), plus the verify lab's dupes variant, which is not part of Example B. The agent here is a scripted stand-in, so these results show how the factory's checks work, not how good any model is.
"""evidence_demo.py: five kinds of evidence about one lower_bound implementation, standard library only.
Runs next to a copy of Example B. The buggy and correct sources are Example B's stand-in outputs
(generator.py); DUPES is the verify lab's own variant (returns any matching index, not the first).
The agent is a scripted stand-in, so this shows how checks work, not how good any model is.
"""
import json
import random
import generator
import oracle
import visualize
DUPES = '''
def lower_bound(values, target):
lo, hi = 0, len(values)
while lo < hi:
mid = (lo + hi) // 2
if values[mid] == target:
return mid
if values[mid] < target:
lo = mid + 1
else:
hi = mid
return lo
'''
def load(source):
ns = {}
exec(compile(source, "candidate.py", "exec"), ns)
return ns
VARIANTS = {name: load(src)["lower_bound"] for name, src in
[("buggy", generator.BUGGY_SOURCE), ("dupes", DUPES), ("correct", generator.CORRECT_SOURCE)]}
def random_inputs(seed, n, lo=-5, hi=5, max_len=8):
"""Sorted lists drawn from a SMALL value range, so duplicates and misses are common."""
rng = random.Random(seed)
for _ in range(n):
values = sorted(rng.randint(lo, hi) for _ in range(rng.randint(0, max_len)))
yield values, rng.randint(lo - 1, hi + 1)
def holds(values, target, i):
"""The two properties that define lower bound, checked without knowing the answer."""
return 0 <= i <= len(values) and all(v < target for v in values[:i]) and all(v >= target for v in values[i:])
def shortest(found):
return min(found, key=lambda c: (len(c[0]), c)) if found else None
fixtures = json.load(open("fixtures.json"))
print("0. Example B's oracle (10 fixtures + 20 seeded random cases), then without two fixtures")
for label, fx in [("all fixtures", fixtures),
("minus duplicates, negatives", {"cases": [c for c in fixtures["cases"]
if c["name"] not in ("duplicates", "negatives")]})]:
ev = oracle.evaluate(VARIANTS["dupes"], oracle.build_cases(fx))
print(f" dupes, {label:28} {ev['passed']}/{ev['total']} passed failed={ev['failed_names']}")
print("1. property test: 0 <= i <= n, values[:i] < target, values[i:] >= target (500 inputs, seed 11)")
for name, f in VARIANTS.items():
bad = [(v, t, f(list(v), t)) for v, t in random_inputs(11, 500) if not holds(v, t, f(list(v), t))]
print(f" {name:8} violations={len(bad):3} shortest={shortest(bad)}")
print("2. differential test against oracle.brute_force (500 inputs, seed 12)")
for name, f in VARIANTS.items():
bad = [(v, t, f(list(v), t), oracle.brute_force(v, t)) for v, t in random_inputs(12, 500)
if f(list(v), t) != oracle.brute_force(v, t)]
print(f" {name:8} disagreements={len(bad):3} shortest={shortest(bad)}")
print("3. metamorphic test: put a value smaller than target in front; the answer must grow by 1 (500 inputs, seed 13)")
for name, f in VARIANTS.items():
bad = []
for v, t in random_inputs(13, 500):
smaller = min(v + [t]) - 1
before, after = f(list(v), t), f([smaller] + v, t)
if after != before + 1:
bad.append((v, t, before, after))
print(f" {name:8} broken relations={len(bad):3} shortest={shortest(bad)}")
print("4. trace consistency: Example B's correct trace vs the oracle answer (500 inputs, seed 14)")
trace = load(generator.CORRECT_SOURCE)["lower_bound_trace"]
problems = 0
for v, t in random_inputs(14, 500):
problems += len(visualize.check_invariants(trace(v, t), v, t, oracle.brute_force(v, t)))
print(f" correct trace: {problems} invariant violations")
buggy_trace = load(generator.BUGGY_SOURCE)["lower_bound_trace"]
print(" buggy trace on [1, 2, 3], target 9:", visualize.check_invariants(buggy_trace([1, 2, 3], 9), [1, 2, 3], 9, 3))
The output, after the first block shown at the top of the page:
1. property test: 0 <= i <= n, values[:i] < target, values[i:] >= target (500 inputs, seed 11)
buggy violations=100 shortest=([-5], -4, 0)
dupes violations= 17 shortest=([-5, -5, 1, 4], -5, 1)
correct violations= 0 shortest=None
2. differential test against oracle.brute_force (500 inputs, seed 12)
buggy disagreements=104 shortest=([-5], -3, 0, 1)
dupes disagreements= 15 shortest=([2, 2, 5], 2, 1, 0)
correct disagreements= 0 shortest=None
3. metamorphic test: put a value smaller than target in front; the answer must grow by 1 (500 inputs, seed 13)
buggy broken relations= 52 shortest=([], -6, 0, 0)
dupes broken relations= 5 shortest=([-5, -2, 0, 0, 1, 5], 0, 3, 3)
correct broken relations= 0 shortest=None
4. trace consistency: Example B's correct trace vs the oracle answer (500 inputs, seed 14)
correct trace: 0 invariant violations
buggy trace on [1, 2, 3], target 9: ['step 0: answer 3 outside [lo, hi] = [0, 2]', 'step 1: answer 3 outside [lo, hi] = [2, 2]', 'final state lo=2 hi=2 does not equal oracle answer 3']
Read the four results in order, because each one adds something the previous lacked.
The property test found the dupes variant 17 times in 500 inputs, with no answer key at all. Its shortest failure is [-5, -5, 1, 4] with target -5: the variant returns 1, and the value before index 1 is not smaller than the target. The random generator draws values from -5 to 5, so repeated targets are common. Example B's own random cases draw from -20 to 20 with up to 12 values, which is why none of them hit a duplicate target. A randomized test is only as good as its generator's reach: the inputs that matter must be likely, not merely possible.
The differential test compares each variant with brute_force on 500 other inputs: 104 disagreements for the off-by-one, 15 for dupes, none for the correct version. Its strength is that it needs no rule, only a second implementation; its weakness is that both implementations must not share the mistake. A linear scan and a binary search share very little, which is what makes this pair a good one.
The metamorphic test needs neither a rule about the answer nor a second implementation, only a relation between two runs. It caught the off-by-one on the empty list: lower_bound([], -6) returns 0, and after -7 is put in front, the variant still returns 0 instead of 1. It caught dupes 5 times. Metamorphic relations matter most where no oracle exists at all, which is most enterprise logic.
The trace check asks whether the path to the answer was right, not only the answer. Example B's correct trace held its invariants on 500 random inputs. The off-by-one's trace on [1, 2, 3] with target 9 fails at every step: its search window [0, 2] never contained the answer 3. A visualization built from that trace would show a learner a search that could never succeed. The visualizer factory owns the trace contract; this page uses its invariants as evidence.
Diagram: Evidence, from weakest to strongest · how many ways a wrong artifact can fail
Left to right, each column adds a way for a wrong artifact to fail. A self-report fails nothing. Example tests fail the wrong artifacts someone anticipated; delete two fixtures and a wrong lower bound scores 28 of 28. Properties over a well-aimed random generator reach cases nobody wrote down. Differential and metamorphic tests check without an answer key, from an independent implementation or a relation between two runs. Trace checks look inside the computation. A recorded human sample judges what no code can. Every failure found becomes a regression case that stays.
- Claims
- self-report: The producer's own statement; identical for right and wrong work.
- example tests: Known inputs with known outputs; blind outside them.
- Reach
- property test: A rule every right answer obeys, on many inputs.
- randomized inputs: Inputs nobody wrote down; only as good as the generator's reach.
- Without an answer key
- differential test: Two implementations, any disagreement is a defect in one.
- metamorphic test: Change the input in a known way; the output must change in a known way.
- Inside and beyond code
- trace consistency: The path to the answer must be right, not only the answer.
- human review: A person judges meaning and fit; recorded with a rate.
- regression case: Each defect found becomes a shared fixture.
The diagram ranks evidence by how many ways a wrong artifact can fail it. A self-report fails no wrong artifact. Example tests fail the wrong artifacts someone anticipated. Properties and randomized inputs reach cases nobody anticipated. Differential and metamorphic tests check without an answer key, from a second implementation or a relation. Trace checks reach inside the computation. Human review, sampled and recorded, judges what no code can: whether the artifact serves its purpose.
Regression cases: every failure becomes a fixture#
A regression case is a test added because a defect was found, so the same defect can never pass again unnoticed. The shortest failing inputs from the run above are ideal: small enough to read, specific enough to fail the wrong variant for one reason. Added to Example B's fixture file, in its own format, they would look like this (the expected values come from brute_force; this page did not change the example):
{"cases": [
{"name": "regression-dupes-leading-pair", "values": [-5, -5, 1, 4], "target": -5, "expected": 0},
{"name": "regression-dupes-middle-pair", "values": [2, 2, 5], "target": 2, "expected": 0},
{"name": "regression-off-by-one-single", "values": [-5], "target": -3, "expected": 1}
]}
Two rules keep regression cases useful. They are never deleted to make a suite pass, which is the opening's failure in one sentence. And they belong to the pattern, not to one unit: a defect class found in one generated function will appear in the next one, so the case goes into the shared fixtures every unit of the pattern is tested against. The factory learning loop is the home of turning a defect into a pattern change.
Common confusion. Three replacements are easy to mix up. A mock replaces a dependency of the code under test, such as a provider that returns a canned roof assessment, so the test can run without it. A stand-in, on this site, replaces the producer, the model that writes the code. An oracle replaces knowing the answer. A factory needs all three, and the oracle is the only one that makes a test evidence.
Human review: where a person looks#
Human review is a person checking an artifact and recording a verdict, for what code cannot judge: whether an explanation teaches, whether a visualization misleads, whether a rule matches the business intent. The analogy is an editor reading a manuscript that has already passed the spell checker. The consequence is cost and variability: people are slow, they disagree, and a review nobody records cannot be counted. The trade-off is that without review, a factory measures only what its code can check, and calls that quality.
Two rules make review evidence rather than ceremony. It must be recorded: who reviewed which artifact, against what criteria, with what verdict. And in a factory of thousands of units it is a sample: a recorded rate, drawn at random plus targeted at risk, so the result says something about the units nobody looked at. This site's recommendation: sample every batch, oversample new pattern versions and failed-case queue survivors, and treat a sample with zero findings as a reason to check the sampling, not to stop.
The evidence a factory station should demand#
Not every artifact can afford every technique. The question for each station is what a wrong artifact would cost and what is cheap to check.
| If the artifact is | Cheapest strong evidence | Add when the stakes rise |
|---|---|---|
| A pure function with a known definition (lower bound) | property test over a duplicate-heavy random generator | differential test against a brute force; regression cases |
| A function with no simple definition | differential test against an independent implementation | metamorphic relations; a human sample |
| A trace or visualization | invariants on every step; final state equals the oracle's answer | a renderer contract test; a human sample for clarity |
| Business logic with rules (an eligibility check) | one test per rule; metamorphic relations across irrelevant fields | replay of real, masked cases; golden cases; review by the rule's owner |
| Prose (an explanation, a flavor) | structure checks only | a recorded human or evaluated sample |
What this is not. This page names the kinds of evidence a factory station can demand. Rubrics, golden sets, graders, model-as-judge and release-quality evaluation belong to Eval Engineering; the factory invokes those as gates, it does not design them here. Typed contracts and schema validation belong to Schema Engineering and, for this site's products, to Data and integration engineering.
In a coding agent. A coding agent such as Claude Code or Codex, running headless as a station worker, will often write tests along with its code, and run them. Those tests are useful feedback for the agent and weak evidence for the factory, because the producer wrote them. The station's gate runs the factory's own properties, differential tests and regression cases, which live outside the agent's scoped checkout so it cannot edit them; a person with merge rights reviews the result.
Try it#
The verify lab sets the stand-in's self-report beside the oracle's evidence for three implementations. The first embed opens on Example B's off-by-one attempt; the second on the dupes variant this page tested above.
| Variant | Self-report | Oracle | Failing cases (expected, got) | Verdict |
|---|---|---|---|---|
buggy | done, all tests pass | 21 of 30 | all-smaller (3, 2), single-above (1, 0), random-06 (6, 5), random-09 (8, 7), random-10 (2, 1), random-14 (1, 0), random-17 (1, 0), random-18 (6, 5), random-20 (3, 2) | rejected |
correct | done, all tests pass | 30 of 30 | none | accepted |
dupes | done, all tests pass | 28 of 30 | duplicates (1, 2), negatives (1, 2) | rejected |
The trace lab plays Example B's lower-bound trace step by step and checks each step's invariants. With data-preset="corrupt" one step's window is moved off the answer, and the check fails at exactly that step.
| Step | lo | hi | mid | Comparison (target 2, answer 1) | Checks |
|---|---|---|---|---|---|
| 0 | 0 | 5 | none | start | pass |
| 1 | 0 | 2 | 2 | values[2]=2 >= 2: hi = 2 | pass |
| 2 | 0 | 1 | 1 | values[1]=2 >= 2: hi = 1 | with corrupt: window [2, 2], "step 2: answer 1 outside [lo, hi] = [2, 2]" |
| 3 | 1 | 1 | 0 | values[0]=1 < 2: lo = 1 | pass: final lo = hi = 1 |
The fixture is duplicates, values [1, 2, 2, 2, 3]. The corruption is the lab's own; the trace and the check messages are Example B's.
WatchAlgo in practice#
Observed in the repository (zenalgo-frontend, scripts/viz/generate.py, commit d08ba4f0; read-only by this page's writer): the September visualization station ran a differential test. verify_with_node runs the unit's JavaScript solution in Node on every test case input and compares its output with the output of the Python tracer a worker wrote; any difference fails the unit, and only a unit that passes gets its visualization.json written. The two implementations come from different workers in different languages, which is what gives the comparison its strength. The same file shows two ways the check can be skipped: a tracer may set SKIP_NODE_VERIFY, and a solution without a top-level JavaScript function gets a printed note and no comparison. A skip is not a failure, so a skipped unit is accepted on its other checks.
Diagram: WatchAlgo's visualization gate · a differential test between two workers' code
A solution worker wrote the JavaScript solution; a different worker wrote a Python tracer that records the algorithm's states. The gate runs the tracer to produce states and outputs, checks the state contract, and runs the JavaScript in Node on every test input. If the outputs differ, the unit fails and nothing is written. If they agree, code writes the visualization file. Two paths skip the comparison: a tracer flag and a solution without a top-level function. Skipped units are accepted on their other checks, which is why skips need their own count.
- Two producers
- JavaScript solution: The executed reference implementation in solution.json.
- Python tracer: A program that records the algorithm's states and outputs.
- Gate (code)
- state contract: Required keys per state, at least two states per case.
- Node cross-check: Runs the JavaScript on every test input and compares with the tracer's output.
- skip paths: SKIP_NODE_VERIFY, or no top-level function found: no comparison runs.
- Outcome
- visualization written: Only a passing unit gets visualization.json.
- unit fails: A disagreement exposes a wrong solution or a wrong tracer.
The diagram shows the gate as the repository describes it. Two workers produce two pieces of code for the same problem. Code checks the tracer's states against the state contract, then runs the JavaScript in Node on every test input and compares outputs. Agreement lets code write the visualization; a difference fails the unit and exposes a defect in one of the two. The skip path is drawn separately because it ends in the same place as agreement.
Observed (.ai-context/VIZ_SOLUTION_BUGS_FOUND.md, commit 4426f119): this cross-check found 86 defects in solution.json files that had already passed the solution validator, as AI enters the factory and the WatchAlgo story tell. Observed on scratch copies (docs/EVIDENCE.md, section 10.2): the evidence study re-ran the JavaScript solutions of a copied sample against the stored visualization outputs: 649 of 672 test cases agree. All 23 mismatches are explained by documented opt-outs, not by wrong content: seven units have SKIP_NODE_VERIFY = True with a written reason (in-place mutation, class-based or n-ary-tree inputs, node references), and wa-000161, an April-era visualization with no tracer, mutates its input and returns nothing. Checking those eight would need a runner that compares the mutated argument. The evidence notes count 191 opt-outs in all, and no validator checks a solution's expected outputs against running code.
Observed (docs/EVIDENCE.md, matrix row 17): there is no human review record. The reviews in reviews/ are model-written (one names "Reviewer: Claude Opus 4.6"; others are by Gemini). The worker brief mentions browser spot checks, and the session notes say one category was "spot-checked in browser", without saying by whom. Reported by the author: the corpus was assembled with no manual review (watchalgo-architecture/ARCHITECTURE.md, AGENTS.md). The two agree. Teaching reconstruction: a recorded human sample per batch, with a sampling rate, is what would turn "no manual review" into a measured quality rate (docs/EVIDENCE.md, section 12).
Enterprise translation#
Algorithm problems are unusual: they have a single right answer and a cheap oracle. Most of what the insurance factory of Part III builds has neither, and docs/EVIDENCE.md section 11 lists that as the first place the WatchAlgo analogy fails. The techniques still apply, in a different mix. The policy rules are properties: for R-ROOF-01, any submission in wind zone C with a roof older than 25 years must come back not eligible, whatever else it contains. Metamorphic relations cover what no rule states: adding an irrelevant field to a submission must not change its eligibility, and changing the broker's name must not change the recommendation. Differential tests compare a new intake service with the one it replaces on replayed, masked submissions; Data and integration engineering owns contract and replay tests. Golden cases such as UW-004, where the application says the roof is 9 years old and the provider says 22, pin the behaviors that matter most; Insurance factory, part 3 installs them as gates.
Human review changes role. In the factory it reviews generated code, rules and tests before release, by the people who own the rules. In the product it is not a review at all but the decision itself: the underwriting agent recommends, and an underwriter decides. This site's recommendation: tier the evidence by risk. A low-risk internal report may need one property test per rule; a change to eligibility logic needs rule tests, metamorphic relations, golden cases, a replay against the previous version and a recorded review by the rule owner. What dominates the critical path is rarely writing these tests; it is agreeing on what the rules mean, which is a specification problem, and Specification as executable contract is where it starts.
Failure test#
The failure: a fixture file loses the two cases that exercised duplicate targets, and a wrong lower_bound that returns any matching index passes the oracle with 28 of 28.
How the factory catches it: a property test that needs no fixture, run over a generator that makes duplicates common, fails the variant 17 times in 500 inputs; a differential test against brute_force fails it 15 times; the metamorphic relation fails it 5 times. Any one of them restores the gate. The shortest failing inputs then become regression cases in the shared fixtures, so the same edit cannot open the hole again.
What to run or toggle: in a scratch copy of examples/b-algorithm-factory/, save the script above as evidence_demo.py and run /opt/homebrew/bin/python3 evidence_demo.py. Change the generator's value range from -5..5 to -20..20 and watch how many violations the property test still finds. In the verify lab, pick dupes and read which cases catch it.
Where to go deeper#
- The factory learning loop (next): from a defect to a pattern change and a selective regression suite.
- AI enters the factory: self-reports, the stand-in and the oracle that tells them apart.
- The visualizer factory: the trace contract whose invariants this page checks.
- Data and integration engineering: contract tests and replay across services.
- Eval Engineering: rubrics, golden sets and graders that the factory invokes as gates.
- Schema Engineering: typed contracts and validation.
Checklist for Monday#
- For each generated artifact, write down its oracle: known outputs, a reference implementation, or a rule.
- Add at least one property test per function with a precise definition.
- Design the random generator so the risky inputs (duplicates, empty, boundaries) are common, and fix its seed.
- Pair every generated implementation with an independent one where you can, and diff them.
- Find one metamorphic relation per piece of logic that has no oracle.
- Turn every defect found into a regression case in the shared fixtures; never delete one to pass.
- Count skipped checks separately from passed ones, and review the skip list.
- Record human review: who, which artifact, which criteria, which verdict, and the sampling rate.
Exercises#
- Why did Example B's 20 random cases never catch the
dupesvariant?
Answer 1The generator rarely repeats a target
Values come from -20..20 with at most 12 per list, so a repeated value equal to the target is rare; the generator did not reach the risky region.
- Name a metamorphic relation for lower bound other than the one on this page.
Answer 2Shift everything by a constant
Adding the same constant to every value and to the target must leave the answer unchanged.
- Two model-written implementations agree on every input. Is that strong evidence?
Answer 3Weaker than it looks
Agreement shows only that they do not differ; if both come from similar producers with similar briefs, they can share a mistake. Prefer a second implementation that is independent by construction, such as a brute force.
Advanced perspective#
Skipped is not passedOpt-outs need a list, a reason and an owner
Every real verification system has cases it cannot check: inputs it cannot build, outputs it cannot compare, environments it cannot reach. WatchAlgo's cross-check has 191 opt-outs, each with a written reason in the tracer, plus a silent path when no top-level JavaScript function is found. That is honest engineering, and also a gap: an opted-out unit is accepted on weaker evidence than its neighbours, and nothing in the final file says so.
Three practices close the gap. Record, per artifact, which checks ran, which were skipped and why, so "accepted" carries its evidence level. Report the skip rate per batch as its own number, beside the pass rate. And give the skip list an owner whose job is to shrink it: the evidence study's suggestion of a runner that compares mutated arguments would turn most of WatchAlgo's opt-outs into real checks.
Testing the testsWrong variants as a measure of a suite's strength
The buggy and dupes implementations on this page are useful beyond their own verdicts: they measure the oracle. A suite that accepts a known-wrong variant is weak in a way a pass rate can never show. This is the idea behind mutation testing: make small deliberate changes to a correct implementation (flip a comparison, change a bound, drop a branch) and count how many of them the tests reject.
For a factory, the practical form is a small library of known-wrong variants per pattern, run against the oracle whenever the oracle or its fixtures change. The opening of this page is the failure it prevents: removing two fixtures would have dropped the oracle's rejection of dupes from 2 failing cases to 0, and a variant library would have flagged that change before any generated unit was judged by the weakened suite.
How large a human sample needs to beZero findings in a small sample proves little
A reviewer who checks 20 units out of 2,000 and finds nothing has learned less than it seems. If 5% of the units had a problem only a person could see, a random sample of 20 would contain none of them a little more than a third of the time (0.95 to the power 20 is about 0.36). A clean sample of that size is weak evidence that the defect rate is low; a sample of a few dozen, repeated per batch, says much more.
The statistics of sampling plans belong to evaluation and quality engineering, and Eval Engineering treats sampled grading in depth. For a factory, three habits matter: record the sample size and the rule that picked it; combine a random sample, which estimates the rate, with a targeted one, which looks where risk is highest (new pattern versions, opt-outs, failed-case queue survivors); and publish the result as a rate with its sample size, never as "reviewed".
In one breath#
Evidence of correctness is a check, independent of the producer, that a wrong artifact would fail, and its strength is the number of ways a wrong artifact can fail it. Example tests catch what someone anticipated; property tests, randomized inputs from a well-aimed generator, differential tests against an independent implementation, metamorphic relations and trace invariants catch what nobody anticipated, often without an answer key, as five checks on one function showed after two deleted fixtures let a wrong variant score 28 of 28. Every defect found becomes a regression case, skipped checks are counted as skipped, and human review, recorded and sampled, judges what code cannot; WatchAlgo has strong executed cross-checks and no human review record.
Evidence finds defects one unit at a time. A factory that keeps finding the same defect in unit after unit should change the pattern that produces them. The next chapter, The factory learning loop, turns defects into a taxonomy, a versioned pattern change, a measured blast radius and a selective regression suite, and then measures whether the next batch is better.