A report needs a running total. Someone writes a helper in two minutes: for each position, add the previous total to the current value. They try it on [1, 2, 3, 4] and get [1, 3, 6, 10]. The numbers are right, the helper is merged, the report ships.
A week later a different column of the same report is wrong. The daily values now look like totals. Nothing crashed and no test failed, because there was no test. The helper had written the totals back into the list it was given, and the next part of the report read that same list.
The bug was not hard. What was missing was a written statement of what "correct" means ("return a new list; do not change the caller's list") and something other than the author's eye that checks it every time. That pair, a statement of what is wanted and a check that does not trust the producer, is the smallest idea on this site.
The learner's question: what is the smallest thing that deserves the name "factory", and how would anyone know that what it produced is right?
Four parts, older than any AI#
Plain English. A factory takes a description of what is wanted, turns it into a product by fixed steps, checks the product against the description, and keeps the product together with the result of the check.
Every part of that sentence existed long before language models. A bakery has an order (two dozen rolls, no sesame), a recipe it follows, an inspection before the box is closed, and a delivery note. A build tool has source files, a compiler, a test run and a packaged artifact. This site calls the four parts input, transformation, acceptance check and output, and it uses one small function to show them.
Beginner definition: a software factory, at its smallest, is a repeatable procedure that turns a written specification into a software artifact and refuses to call the artifact done until an independent check has passed.
Before AI. A Make build already has this shape: targets (what is wanted), rules (how to produce each one), a test target that fails the build, and files on disk as output. A manufacturing line adds inspection stations between steps. Neither needed a model, and neither trusted the worker's own word that the part was good.
The transformation: the step that produces something#
A transformation is the step that turns the input into the product. In the opening story it is the running-sum helper: a list goes in, a list of totals comes out.
The non-AI analogy is the oven: dough in, bread out. The minimal example is the body of the helper, five lines that keep a total and append it. The operational consequence is that a transformation can be swapped. The oven can be replaced, the helper can be rewritten, and later on this site a station worker (a script, a person, or an agent) can take over the job. The trade-off follows at once: if the transformation can change, the definition of a good product must not live inside it. It has to live somewhere the transformation cannot edit.
The acceptance check: the step that does not trust the producer#
An acceptance check is a test, written from the specification, that the product must pass before anyone calls it done. For the running sum it has two parts: every example must give the expected answer, and the caller's list must be the same after the call as before.
The analogy is the inspector at the end of the bakery line, who works from the order, not from the baker's opinion. The consequence is the most important sentence of this page: the check decides, so the check must be written from the specification and run by something other than the producer. The trade-off is that a check only sees what it was written to see. Three examples prove three cases, not every list.
Diagram: The primitive · input, transformation, acceptance check, output
Four lanes, left to right. The input is the running-sum request written down: three examples and one rule. The transformation is any candidate that produces a list. The acceptance check runs every example and compares the caller's list before and after the call; it was written from the input, not by the producer. The output lane holds either an accepted artifact with its evidence or a rejection that names what failed. A rejection is an output too.
- Input
- examples: Concrete inputs with the expected outputs, written before any code.
- one rule: A requirement that no single returned value can show.
- Transformation
- candidate function: The step that produces the product. It can be swapped, so the definition of good must live outside it.
- Acceptance check
- run every example: Each example becomes one comparison.
- compare input before and after: A deep copy before the call, an equality check after it.
- Output
- accepted artifact + evidence: The product kept together with the checks it passed.
- rejection that names the failure: A failed check produces a record a person can act on.
Read the diagram left to right. The input carries examples and one rule. The transformation produces a candidate. The acceptance check runs every example and compares the caller's list before and after. Only then does the output lane hold something: an accepted artifact with its evidence, or a rejection that names what failed. The rejection is an output too, never silence.
The smallest primitive: one function, one check#
The script below was written for this page and run in a scratch folder with Python 3.14. It holds two candidate transformations for the running-sum request and one acceptance check. A successful run proves that the check can tell the two candidates apart. It does not prove that either candidate is correct for every possible list.
import copy
SPEC = {
"examples": [([1, 2, 3, 4], [1, 3, 6, 10]), ([], []), ([5, -2, 7], [5, 3, 10])],
"rule": "do not change the caller's list",
}
def running_sum(xs):
"""Candidate A: builds a new list."""
total, out = 0, []
for x in xs:
total += x
out.append(total)
return out
def running_sum_in_place(xs):
"""Candidate B: quicker to write, rewrites the caller's list."""
for i in range(1, len(xs)):
xs[i] += xs[i - 1]
return xs
def acceptance_check(fn, spec):
"""Run every example and the one rule; return a list of failures (empty = accepted)."""
failures = []
for given, expected in spec["examples"]:
data = copy.deepcopy(given)
got = fn(data)
if got != expected:
failures.append(f"{given} -> {got}, expected {expected}")
if data != given:
failures.append(f"{given} was changed to {data}")
return failures
for fn in (running_sum, running_sum_in_place):
found = acceptance_check(fn, SPEC)
print(f"{fn.__name__}: {'accepted' if not found else 'rejected'}")
for f in found:
print(" -", f)
running_sum: accepted
running_sum_in_place: rejected
- [1, 2, 3, 4] was changed to [1, 3, 6, 10]
- [5, -2, 7] was changed to [5, 3, 10]
Notice what the in-place candidate got right: every returned value matches. A reviewer who only looks at outputs would accept it, which is exactly what happened in the opening story. The rule "do not change the caller's list" is what catches it, and it catches it only because someone wrote the rule down and put it in the check.
Python
In Python the rule is a test. A list passed to a function can be changed by that function, so the only protection is a check that compares the list before and after, as acceptance_check does above.
Java
static List<Integer> runningSum(List<Integer> xs) {
List<Integer> out = new ArrayList<>();
int total = 0;
for (int x : xs) {
total += x;
out.add(total);
}
return out;
}
runningSum: accepted
runningSumInPlace: rejected at run time, UnsupportedOperationException
The full file (RunningSum.java, with the same three examples and an in-place variant) was compiled and run with javac and java 23 in a scratch folder; the output above is real. The test inputs were built with List.of, which returned lists that refuse changes, so the in-place variant failed with an exception instead of quietly rewriting them.
Before AI. The Java run shows the same rule moved into the type of the input. That only protects callers who pass an unmodifiable list; a caller with an ArrayList gets the quiet rewrite back. Types and tests share the job, and neither alone covers it.
Written once versus a reusable operation#
The helper in the story was written once. Someone typed it, looked at one output, and moved on. If a second report needs a slightly different helper, someone types again, and the knowledge of what "correct" meant lives in a memory, not in a file.
A reusable operation is the same work turned into a procedure that takes the request as data and can run again: read a specification, produce the artifact from fixed templates, run the checks, write down what happened. The analogy is the difference between a cook who improvises a dish and a kitchen that works from a recipe card and an order ticket. The minimal example is Example A on this site, shown below. The consequence is that a second request becomes a second input file, not a second act of improvisation. The trade-off is up-front cost: a reusable operation needs a specification format, templates and checks before it produces its first artifact, and for a request that will never repeat that cost is waste.
Diagram: Written once against a reusable operation
The left lane is the one-off path from the opening story: the definition of correct lives in the author's head, one output is eyeballed, and nothing can re-check the helper later. The right lane is the reusable operation of Example A: the specification, the template and the check are files, every run writes a manifest, and the same input run again gives the same bytes. Read across the two lanes step by step to see what each path keeps.
- Written once
- request in someone's head: The meaning of correct is never written down.
- type the helper: A person writes the function directly.
- eyeball one output: The only check is the author's opinion of one result.
- merged, nothing kept: No record says what was checked, so nobody can repeat it.
- Reusable operation (Example A)
- spec file: The request as data: examples, rules, acceptance criteria.
- render from template: Same spec and template give the same files every time.
- compile and test: Tests generated from the spec, not by the body's author.
- manifest: A record of what was built from what; a rerun gives identical bytes.
The left lane is the one-off path: the definition of correct stays in the author's head, so nothing can re-check it later. The right lane is the reusable operation: the specification, the template and the check are files, so the same request run again gives the same artifact and the same evidence. The next page asks when the right lane is worth building.
Two properties make the bottom lane trustworthy, and both have names.
A step is deterministic when the same input always gives the same output. Template rendering is deterministic; a person retyping the helper is not; a language model, later on this site, is not either. In practice determinism has to be engineered: the time of day, the order of keys in a file, or an absolute path can leak into an output and make two runs differ for no business reason.
Reproducibility is the result of that engineering: anyone can run the procedure again on the same inputs and get the same artifact, byte for byte, which proves that nothing hidden changed. It is the property that lets a second person trust a run they did not watch.
The same function, produced by Example A#
Example A (examples/a-assembly-line/) is the tiny factory this part of the site follows. It runs the running-sum request, and a second one (clamp-list), through four stations: validate the spec, render the artifact, compile and test, write a manifest. Here only the first and third matter; Build the smallest factory walks all four. There is no model anywhere in it: station 2 renders from templates.
The specification is a JSON file, specs/running-sum.json. Its examples and its rule are the same as in the primitive above:
{
"schema_version": "1.0",
"spec_id": "SPEC-ALGO-101",
"problem_name": "running-sum",
"function_name": "running_sum",
"params": ["xs"],
"pattern": {"id": "PAT-ALGO-UNIT", "version": "1.3.0"},
"body_template": "prefix_sum",
"requirements": [
"Return a new list where item i is the sum of xs[0] through xs[i].",
"An empty list returns an empty list.",
"Do not change the caller's list."
],
"examples": [
{"input": [[1, 2, 3, 4]], "output": [1, 3, 6, 10]},
{"input": [[]], "output": []},
{"input": [[5, -2, 7]], "output": [5, 3, 10]}
],
"acceptance_criteria": [
{"id": "AC-1", "text": "Every example in the spec passes.", "check": "all_examples_pass"},
{"id": "AC-2", "text": "The input list is unchanged after the call.", "check": "input_not_mutated"}
]
}
The line turns each example and each criterion into one test method (factory.py : _test_methods). This is the generated test file from a run in a scratch copy, out/FR-0001/running-sum/test_running_sum.py, quoted as written:
import copy
import unittest
from running_sum import running_sum
class TestRunningSum(unittest.TestCase):
def test_example_1(self):
self.assertEqual(running_sum([1, 2, 3, 4]), [1, 3, 6, 10])
def test_example_2(self):
self.assertEqual(running_sum([]), [])
def test_example_3(self):
self.assertEqual(running_sum([5, -2, 7]), [5, 3, 10])
def test_ac_2_input_not_mutated(self):
data = [1, 2, 3, 4]
before = copy.deepcopy(data)
running_sum(data)
self.assertEqual(data, before)
if __name__ == "__main__":
unittest.main()
The test file is written from the specification, not by whoever wrote the function body. Station 3 (factory.py : compile_and_test) compiles both files and runs the tests in a separate process. Then the example's own runner changes one expected value in the spec and runs the line again as factory run FR-0008. Output of /opt/homebrew/bin/python3 run.py --verbose in a scratch copy, the two relevant blocks:
FR-0001 running-sum.json
S1-validate-spec: pass (0 errors)
S2-render-artifact: pass (running_sum.py, test_running_sum.py)
S3-compile-test: pass (tests_run=4, failed=[])
S4-write-manifest: pass (manifest.json, status=accepted)
== FR-0008: the running-sum spec with one example changed ([5, -2, 7] now expects [5, 3, 9])
FR-0008 running-sum-changed.json
S1-validate-spec: pass (0 errors)
S2-render-artifact: pass (running_sum.py, test_running_sum.py)
S3-compile-test: fail (tests_run=4, failed=['test_example_3'])
S4-write-manifest: pass (manifest.json, status=rejected_at_S3)
spec_hash before 1deb0c4b830f after df066a312212 differ=True
Two things to notice. First, four tests ran: three examples and the mutation rule from the primitive, now produced from a file instead of typed by hand. Second, when the spec changed, the code did not, and the check refused the unit and named the failing test. The line does not decide whether the code or the example is wrong. That is a question for a person, and the line makes sure the person gets it.
Quality you can measure#
Measured quality means a claim about an artifact that points at evidence: which checks ran, on which input, with what result. "The helper is good" is an opinion. "Four tests generated from SPEC-ALGO-101, run by station 3, zero failures, spec hash 1deb0c4b830f" is a measurement, and anyone can repeat it.
This is why the word "factory" proves nothing by itself. A pipeline with many steps and no independent check is a fast way to produce unchecked work. A single function with a written rule and a check that runs every time is already a small factory. The rest of this site adds stations, models, parallel work and governance, and every addition is judged the same way: does it produce artifacts that pass checks the producer did not write, at a cost worth paying?
Common confusion. Many tests passing is not the same as correct. Example A's tests are only as strong as its examples. A wrong expected value in the spec produces a wrong test that a wrong function can pass. Evidence of correctness adds properties, randomized and differential tests for exactly that reason.
In a coding agent. When a coding agent such as Claude Code runs headless (non-interactive, started with claude -p) as a worker on one request, the same four parts apply. The agent is the transformation. The acceptance check is the factory's own test suite, not tests the agent wrote for itself, and a person with merge rights reviews the result. The runtime controls around the agent belong to Harness Engineering.
What this is not. This page is not about the controls around one coding agent's run (permissions, hooks, context): that is Harness Engineering. It is not about one change's path through review and CI to production either: that is Diff to Deploy. It is about the smallest unit both of those rely on: a statement of what is wanted and a check that holds the product to it.
Try it#
The lab runs Example A's validator and renderer in the browser. Edit the spec of the running-sum or clamp-list request, and watch the four stations run on every change: the error list at station 1, the rendered text at station 2, expected versus actual for every example at station 3, and the short hashes at station 4. The preset broken loads the rejected third spec; changed edits one example so station 3 fails. Nothing here calls a model.
The table holds the same results for a reader without JavaScript. They are real outputs of examples/a-assembly-line/run.py, run on 2026-11-20 in a scratch copy.
| Preset | Spec | Station 1 (validate) | Station 3 (compile and test) | Status | Spec hash (first 12) |
|---|---|---|---|---|---|
running-sum | SPEC-ALGO-101 | pass, 0 errors | 4 tests run, 0 failed | accepted | 1deb0c4b830f |
clamp-list | SPEC-ALGO-102 | pass, 0 errors | 5 tests run, 0 failed (one example expects ValueError) | accepted | 7af02f7aef73 |
broken | SPEC-ALGO-103 (median, incomplete) | fail, 6 errors | skipped | rejected_at_S1 | 32c1e1899da6 |
changed | SPEC-ALGO-101, one example now expects [5, 3, 9] | pass, 0 errors | 4 run, test_example_3 failed | rejected_at_S3 | df066a312212 |
To run it yourself, copy examples/a-assembly-line to a scratch folder and run /opt/homebrew/bin/python3 run.py --verbose there. Example A has no agent at all. Where later examples put one at a station, the agent is a scripted stand-in, so those results show how the factory mechanics work, not how good any model is.
WatchAlgo in practice#
WatchAlgo, the author's algorithm-learning product, is the case study of Part II (the WatchAlgo story). Its relevance here is one event that shows the acceptance check doing its job at scale.
Observed in the repository (zenalgo-frontend, commit a0a0cbd6): in September 2026 a deterministic visualization generator was added. It ran each problem's JavaScript solution in Node and cross-checked its own state tracer against that run. The commit message records "86 solution.json defects caught by the cross-check", and .ai-context/VIZ_SOLUTION_BUGS_FOUND.md (commit 4426f119) lists them: about 80 wrong expected outputs and about 20 wrong algorithms in one category, plus one solution copied from another problem. These solution files had already passed every earlier step.
The lesson is the one from the opening story, at a larger size: a check that runs the product, written by something other than the producer, found defects that review had not. The evidence notes behind these lines are docs/evidence/inventory.md (section 6) and docs/evidence/git-history.md (section 3).
Reported by the author: WatchAlgo holds 3,246 implemented problems with descriptions, solutions and visualizations in three flavors. The repository shows 3,246 problem statements, with fewer solutions and visualizations; the WatchAlgo story reconciles the numbers.
Enterprise translation#
The same four parts carry over to the enterprise case of Part III, where the factory builds an insurance intake service for a synthetic insurer. In that example (examples/c-insurance-service/) the input is a product specification, SPEC-INTAKE-001, with eleven submission fields, four roles and five routes. The transformation assembles the service from a reusable pattern plus that specification. The acceptance check is a set of gates: a request without a token must get 401, a broker who tries to record a decision must get 403, a malformed submission must get 422 with field errors. The output is the service code, its contracts, and a record of which gates passed.
What changes is the size of the acceptance check. A running sum has three examples; an intake service has authorization rules, schema versions, retries and audit fields, and some of its checks need people (a security reviewer, a release approver). What does not change is the rule from this page: the factory that builds the service does not write the checks it is judged by, and it never calls its own output done. The factory also never makes an underwriting decision; building an insurance service and operating one are different runs (Insurance factory: orchestration keeps them apart).
Failure test#
The failure: a transformation returns the right values and quietly rewrites its input, as in the opening story.
How the factory catches it: acceptance criterion AC-2 (input_not_mutated) becomes a generated test that copies the input, calls the function and compares. The primitive above shows the catch: running_sum_in_place is rejected with the two lists it changed. Without AC-2 in the spec, nothing catches it, because every returned value is right.
What to run or toggle: run the primitive script above, or in the lab remove AC-2 from the running-sum spec and notice that the test count at station 3 drops from 4 to 3. A missing rule is not a failing check; it is a check that does not exist. The second failure on this page, FR-0008, shows the other direction: the spec changed, the code did not, and station 3 refused the unit with status rejected_at_S3.
Where to go deeper#
- From repetition to a pattern (next): what stays the same between two requests, and what must not be shared.
- The specification as an executable contract: how the input is validated before any station runs.
- Build the smallest factory: all four stations of Example A, the manifest and the byte-identical rerun.
- Evidence of correctness: checks stronger than three examples.
- Harness Engineering: the controls around one coding agent's run.
- Eval Engineering: rubrics, golden sets and graders, which a factory invokes as gates when a check cannot be a plain test.
Checklist for Monday#
- Pick one helper your team writes again and again. Write its rule in one sentence that a test can check.
- For every generated or delegated artifact, name the check that decides, and who wrote it. If the producer wrote it, it is not an acceptance check.
- Keep the evidence with the artifact: which checks ran, on what input, with what result.
- Remove time, absolute paths and random ordering from anything you want to compare between runs.
- When a check fails, route it to a person who owns the specification; do not let the pipeline guess which side is wrong.
Exercises#
- The in-place candidate passes every value check. Which single sentence in the specification makes it fail?
Answer 1The mutation rule
"Do not change the caller's list" (AC-2, check input_not_mutated). Without it, the in-place version is accepted.
FR-0008failed at station 3. Is the code wrong or the spec?
Answer 2The line cannot know
The changed example expects [5, 3, 9], but 5 + (-2) + 7 is 10, so here the spec is wrong. The line only knows that the two disagree; a person who owns the spec decides.
- Your Java version uses
int total. Name one input the three examples cannot catch.
Answer 3Overflow
Values whose sum passes 2,147,483,647: a Java int wraps around. None of the examples comes near it, so a passing check says nothing about it.
Advanced perspective#
What determinism costs to engineerClocks, key order, hash seeds, paths
Example A is byte-identical between runs because it removes every source of accidental variation, and the code shows each one. The clock is injected (factory.py : FixedClock, starting at 2026-11-20T09:00:00Z and advancing one second per reading). JSON is written with sorted keys and fixed separators (canonical_json). The test subprocess runs with PYTHONHASHSEED="0" and PYTHONDONTWRITEBYTECODE="1" (compile_and_test). The manifest keeps test counts and failing test names, never timings or absolute paths. The README states the limit honestly: byte-identical output is claimed for the same Python version.
Each of these is a design decision with a cost. An injected clock means real timestamps must be recorded elsewhere, in the run record rather than the artifact. Excluding timings from the manifest means performance evidence needs its own, non-reproducible store. Teams that skip this work discover it later, when two "identical" builds produce different digests and nobody can tell whether anything meaningful changed.
How strong is an example-based check?Examples are a weak oracle; say so
An acceptance check built from a handful of examples is an oracle (the source of the expected answer) with very low coverage. It is strong against regressions on those exact inputs and against obvious misreadings of the request. It is weak against off-by-one errors on lengths the examples skip, overflow, duplicates, very large inputs and anything the spec author did not think of.
Three cheaper upgrades come before any model: a property that must hold for every input (the last running total equals the sum of the list; the output has the same length as the input), a second independent implementation compared on random inputs (differential testing), and a relation between runs (reversing the input of a clamp reverses its output). Evidence of correctness is the home of these techniques. The design rule for now: write down what the check covers, so nobody reads "4 tests passed" as "correct for every list".
When a reusable operation pays for itselfUp-front cost against repeated cost and repeated risk
Turning a one-off helper into a reusable operation costs a specification format, templates, checks and a record, before the first artifact. It pays back through three channels: each later request is cheaper to produce, each later request is checked the same way (so defects like the in-place rewrite stop recurring), and every artifact carries evidence that a reviewer can trust without re-deriving it.
The arithmetic depends on repetition and on the cost of a defect, not on the size of the code. A helper requested once a year, where a defect is cheap to spot, may never pay back. A shape that is requested weekly, where defects reach customers, pays back quickly even if each instance is tiny. The next page turns this into a decision tree, and Factory quality engineering measures cost per accepted unit instead of per generated one.
Computational and inferential checksWhere this page's checks sit in a wider vocabulary
Every check on this page runs code and gives the same answer each time. Birgitta Böckeler's article on harness engineering for coding agent users (martinfowler.com, 2 April 2026, read 2026-11-20) separates checks of that kind, which it calls computational (tests, linters, type checkers: deterministic and fast), from inferential ones (an AI code review, a model used as a judge), which are semantic, probabilistic, slower and more costly. The article places them around one coding agent; a factory places them at stations.
This site's recommendation follows from the distinction: use a computational check wherever the rule can be written as code, and reserve inferential checks for rules that cannot, with their own evidence and their own error rate. A model's judgement that a function "looks correct" is not an acceptance check in the sense of this page.
In one breath#
A factory, at its smallest, is four parts: a written input, a transformation, an acceptance check the producer did not write, and an output that keeps the product together with the result of the check. Example A shows the four on one function, and catches both a rewritten input and a changed specification. Quality is what the check measured, recorded so anyone can repeat it; the word "factory" adds nothing until that is true.
The running-sum request now has a specification, a check and a reproducible artifact. The next request, a clamp on a list, looks almost the same. From repetition to a pattern asks what the two share, what belongs to each request alone, and which similarities are traps.