All chapters
  1. Start hereA software factory, from first principles
  2. Primitive · 1A factory, before AI: input, transformation, check, output
  3. Primitive · 2From repetition to a pattern: what stays, what varies, what to leave alone
  4. Primitive · 3The specification as an executable contract
  5. Primitive · 4Build the smallest factory: four stations, two specs, one manifest
  6. Primitive · 5Assembly stations: typed contracts, owners and failure handling
  7. Primitive · 6AI enters the factory: what a model adds, and what stays deterministic
  8. WatchAlgo · 7The WatchAlgo production story: what the repository shows
  9. WatchAlgo · 8A problem as a production unit: the files one unit must contain
  10. WatchAlgo · 9Station by station: description, solution, implementations, visualization
  11. WatchAlgo · 10The visualizer factory: from algorithm state to renderable steps
  12. WatchAlgo · 11Three flavors and three languages: two different contracts
  13. WatchAlgo · 12Looping over a large backlog: scheduling, limits and a failed-case queue
  14. WatchAlgo · 13Long-running agent execution: context, checkpoints and token limits
  15. WatchAlgo · 14Evidence of correctness: tests the producer cannot talk its way past
  16. WatchAlgo · 15The factory learning loop: defects change the pattern, not only the unit
  17. Enterprise · 16Classify the enterprise demand before choosing a pattern
  18. Enterprise · 17Reference enterprise patterns and their variation points
  19. Enterprise · 18Platform foundations: golden paths, templates, registries and catalogs
  20. Enterprise · 19Pattern versus configuration versus business logic
  21. Enterprise · 20Insurance factory, part 1: building the intake contract
  22. Enterprise · 21Insurance factory, part 2: building the underwriting workflow
  23. Enterprise · 22Insurance factory, part 3: trust and observability built in
  24. Enterprise · 23Data and integration engineering: contracts that survive change
  25. Enterprise · 24The CI/CD and deployment factory: from pull request to production
  26. Enterprise · 25Enterprise governance: owners, approved patterns and provenance
  27. Enterprise · 26Scaling safely: many teams, many agents, one set of boundaries
  28. Operations · 27Factory quality engineering: measuring accepted output, not activity
  29. Operations · 28Failure and recovery scenarios: what breaks a factory, and what restores it
  30. Operations · 29The landscape: which tool for which job, read from its own documentation
  31. Operations · 30Design an enterprise software factory: the blueprint and the rollout
  32. ReferenceSources and verification
  33. ReferenceGlossary
  34. AboutAbout this site
Operations · 27

Factory quality engineering: measuring accepted output, not activity

Lead time, cycle time, throughput, work in progress, first-pass yield, rework, escaped defects and cost per accepted unit, computed on a worked example where adding agents makes the line slower and dearer.

You will compute
Lead, cycle and queue time, first-pass yield, rework and cost per accepted unit, by hand and then with the simulator.
You will see
Six agents instead of two make each artifact cheaper to generate and dearer to accept; with shared files they make the line slower (simulated, illustrative).
You will decide
How to estimate economic benefit without confusing output volume with a business outcome.

On the night of 3 September 2026 the WatchAlgo queue was the busiest it had ever been. About 9,700 tasks "finished in 0 min" between late evening and the next afternoon, and the supervisor restarted the driver 18 times to keep the work flowing. A dashboard that counted finished tasks would have shown a record night.

Observed in the repository (local run.log and queue.json at commit 1b183564, docs/EVIDENCE.md rows 10 and 12): the number of accepted files produced in those 13.5 hours was zero. Every one of those tasks hit the account's weekly usage limit within seconds, and an unrecognised message turned a pause into a retry storm. The WatchAlgo story tells it in full. This page keeps one lesson from it: the factory was very active and produced nothing, and only a metric with "accepted" in its numerator could tell the two apart.

The learner's question: which numbers tell you whether a factory produces more good software, sooner and for less, and which numbers only measure activity?

Count accepted units, not activity#

Plain English. A factory metric counts something about accepted units over a stated period, divided by a stated denominator. "Tasks finished", "lines written" and "agents running" count activity. They rise when the factory works and also when it thrashes.

Beginner definition: a factory quality metric is a number computed from the factory's own record (runs, attempts, artifacts, verdicts) that says how much accepted output arrived, how long it took, how much of it was right the first time, and what it cost.

Two popular numbers fail that test. Lines of code are cheap for a model to produce, and a station that writes twice the code for the same behavior has doubled its review burden. The number of agents is an input, not an output, and this page shows a line where tripling it raises the cost of every accepted unit.

Before AI. A plant that stamps a thousand panels and scraps three hundred reports seven hundred, not a thousand. A Jenkins installation that ran 400 builds yesterday tells you about load on the build farm, not about what reached production. DORA's delivery metrics, below, made the same move for software.

Three clocks: lead time, cycle time, queue time#

Lead time runs from the moment a specification is accepted to the moment a deployable artifact exists. It is what the person who asked for the work experiences. A tiny example: a spec accepted at 09:00 whose service is ready to deploy at 14:43 has a lead time of 343 minutes.

Cycle time runs from the moment the first station starts on a unit to the moment its artifact is accepted. It is what the line itself experiences. Queue time is the part of either clock in which nobody works on the unit: it waits for a free worker, a reviewer or an approval. Work time plus queue time is cycle time; add the wait before the first station and you have lead time.

The batch simulator from Looping over a large backlog gives each of these per unit (simulated, illustrative). In its default line every unit needs about 42 minutes of station work, yet the median unit spends 223 minutes in the line and 171 of them waiting. The share of cycle time that is work is called flow efficiency; here it is 0.22. Most of a unit's life in a factory is spent waiting, and the waiting is where improvements pay.

Little's law, the one relation that always holds#

Little's law says that, in any stable system, the average number of units inside equals the rate at which they leave times the average time each one spends inside. A tiny example: a review desk that accepts 5 units an hour, with 10 units waiting on average, makes each unit wait about 2 hours.

The law needs no assumptions about the shape of the work, which is why it is useful for agent factories whose station times vary a lot. Its operational consequence is the one the backlog chapter showed with a WIP limit: at the same throughput, the only way to shorten the time each unit spends inside is to hold fewer units inside. The worked example below uses it once more, to turn a review queue into hours of waiting.

Throughput, WIP, utilization and the bottleneck#

Throughput is accepted units per unit of time, and work in progress is the number of units inside the line. Both have their home in the backlog chapter, with the takt time and the bottleneck. This page adds the measure that tempts managers most: utilization, the share of available time a station is busy.

High utilization looks efficient. A station at 88% busy has almost no slack, so every burst of arrivals turns into a queue. A station at 19% busy looks wasteful, and may be exactly right. The simulator's bottleneck preset triples the solution workers from 4 to 12: the solution station falls from 58% to 19% busy, the visualization station stays at 88%, and throughput moves from 9.045 to 9.068 accepted units per hour. Simulated, illustrative.

The table reads each station's capacity in accepted units per hour, rework included, from the bottleneck run: workers × 60 ÷ (minutes per attempt × attempts per accepted unit).

StationMinutes per attemptWorkersAttempts per accepted unitCapacity, accepted per hourBusy
spec421.03329.0331%
solution12121.28346.7519%
oracle621.05019.0548%
visualization1531.16710.2988%
review511.15010.4387%

The line accepts at most min(29.03, 46.75, 19.05, 10.29, 10.43) = 10.29 units an hour: the minimum of the station capacities. Visualization is the bottleneck, and review is a close second. The simulator delivers 9.068 rather than 10.29 because the line starts empty and drains at the end.

Sequential against parallel stations#

The same table answers a question every team asks first: what do we gain by running stations at the same time on different units? If one unit had to pass all five stations before the next unit started, each accepted unit would take the sum of its station minutes, rework included:

spec           1.033 x  4 =  4.13 min
solution       1.283 x 12 = 15.40 min
oracle         1.050 x  6 =  6.30 min
visualization  1.167 x 15 = 17.50 min
review         1.150 x  5 =  5.75 min
sequential: 49.08 min per accepted unit = 1.22 accepted per hour
pipelined:  min of station capacities   = 10.29 accepted per hour (simulated run: 9.068)

Running stations at the same time on different units multiplies throughput by about eight here. Adding workers to one station adds nothing once another station is slower. That is the whole arithmetic of parallelism: the line goes as fast as its slowest station, and only as fast as its units are independent.

Yield, rework and escapes#

First-pass yield is the share of units that pass a check on their first attempt. A tiny example: in Example B the stand-in's first solution passes 21 of 30 oracle cases and is rejected; the second passes all 30. That station's first-pass yield, on one unit, is 0 of 1.

Across a line, yields multiply. The simulator's five stations pass first time with probabilities 0.95, 0.80, 0.95, 0.85 and 0.90, so a unit that sails through all five should occur about 0.95 × 0.80 × 0.95 × 0.85 × 0.90 = 0.55 of the time. In the default run, 30 of the 60 units needed no rework anywhere: 0.50. A line of individually good stations can still rework half of what it builds.

Rework rate is the share of attempts, or of units, sent back because a check failed; state which, because the two differ. In the default run 41 of 341 attempts failed (12%), while 30 of 60 units were reworked at least once (50%).

Escaped defect rate is the share of accepted units in which a defect is found later, after the factory said yes. Escapes are the factory's most expensive output: they cost a person's time to find, a late rework cycle, and the trust that made acceptance mean something.

Two more rates are easy to confuse. The test pass rate counts tests: Example B's first attempt passed 70% of its 30 cases. The acceptance pass rate counts units that the acceptance gate let through: that attempt contributed zero. A factory that reports a 70% test pass rate for a rejected artifact is reporting activity.

Cost per accepted unit#

Cost per accepted unit is everything the line spent in a period divided by the units it accepted in that period. Everything means every attempt, including failed and abandoned ones, every reviewer hour, and the later cost of escapes. A tiny example from Example D's real record: its four factory runs cost 125, 95, 0 and 145 illustrative cost units, and three of them ended with an accepted product. Per run that is 365 ÷ 4 = 91.25 units. Per accepted product it is 365 ÷ 3 = 121.67, because FR-0007 spent 145 units and was escalated to a person with nothing deployable to show.

Averages hide tails, so report percentiles as well. The page's own script below assigns illustrative cost units to the simulator's attempts (3 per attempt at the two model stations, 5 per review). The mean cost per accepted unit is 13.1, the median unit costs 11, the 90th percentile 17 and the worst 24. The median unit passed everything first time; the tail is rework. Token use per accepted unit works the same way and needs the same per-attempt record; a factory that logs tokens per run but not per attempt cannot compute it.

Common confusion. Cost per generated artifact falls when the line retries more, because every retry is another artifact. Cost per accepted unit rises. Only the second answers "what does a good unit cost us?"

The worked example: when more agents make the line slower and dearer#

Here is the case the spec of this site requires, in full. Teaching reconstruction: examples/sim/economics.py is a steady-state model of one 8-hour shift with model stations and human review. Every number below is simulated, illustrative, and every cost is in illustrative cost units, never currency.

The line: generation agents each make one attempt every 20 minutes. Each attempt passes the automated checks with a base yield of 0.7, but agents working in parallel touch shared files, and each agent beyond the first adds 0.05 to the share of attempts spoiled by conflicts. Attempts that pass go to one reviewer, who needs 12 minutes per unit. An attempt costs 3 units, a reviewer hour 60, and 4% of accepted units escape and cost 200 each later.

Diagram: One hour of the six-agent line · generation outruns the one reviewer (simulated, illustrative)

Left to right, one hour of the economics model with six agents and one reviewer. Six agents make 18 attempts. Conflicts on shared files and failed automated checks leave a first-pass yield of 0.525, so 9.45 candidates reach review. One reviewer at 12 minutes per unit accepts 5. The other 4.45 wait in the review queue, which grows every hour of the shift to 35.6 units, and by Little's law each candidate waits 3.56 hours on average. Every attempt is paid for, so cost per generated artifact falls to 8.556 units while cost per accepted unit rises to 30.8. Simulated, illustrative; cost units are not currency.

  1. Generation
    1. six agents: Each agent makes one attempt every 20 minutes; capacity is agents times 60 divided by 20.
    2. checks and conflicts: Base yield 0.7, minus 0.05 per extra agent for shared-file conflicts: 0.7 times 0.75.
  2. Review
    1. review queue: Candidates the reviewer cannot reach wait; the queue grows linearly over the shift.
    2. one reviewer: 12 minutes per unit sets the ceiling of the whole line at 5 accepted per hour.
  3. What the shift buys
    1. accepted units: Total cost 1232 units over 40 accepted units: 30.8 per accepted, against 27.549 with two agents.

The diagram follows one hour of the six-agent line. Eighteen attempts go in; conflicts and failed checks leave 9.45 candidates; one reviewer can accept 5. The difference, 4.45 candidates an hour, waits in front of the reviewer. Everything to the left of the reviewer works faster than the line can use.

By hand, two agents and then six#

                              two agents              six agents
attempt capacity per hour     2 x 60 / 20 = 6         6 x 60 / 20 = 18
conflict share                0.05 x 1 = 0.05         0.05 x 5 = 0.25
first-pass yield              0.7 x 0.95 = 0.665      0.7 x 0.75 = 0.525
candidates per hour           6 x 0.665 = 3.99        18 x 0.525 = 9.45
review capacity per hour      1 x 60 / 12 = 5         1 x 60 / 12 = 5
accepted per hour             min(3.99, 5) = 3.99     min(9.45, 5) = 5
bottleneck                    generation              review
attempts in the shift         48                      144
accepted in the shift         31.92                   40
attempts per accepted unit    48 / 31.92 = 1.504      144 / 40 = 3.6
cost: attempts                48 x 3 = 144            144 x 3 = 432
cost: reviewer shift          8 x 60 = 480            8 x 60 = 480
cost: escapes                 31.92 x 0.04 x 200      40 x 0.04 x 200
                              = 255.36                = 320
total                         879.36                  1232
cost per accepted             879.36 / 31.92 = 27.549 1232 / 40 = 30.8
cost per generated            879.36 / 48 = 18.32     1232 / 144 = 8.556
review queue at shift end     0                       (9.45 - 5) x 8 = 35.6
average wait (Little's law)   0                       (35.6 / 2) / 5 = 3.56 h

Read it from the bottom. Cost per generated artifact halves, from 18.32 to 8.556: a status report would celebrate. Cost per accepted unit rises from 27.549 to 30.8, accepted output rises only to the reviewer's limit of 5 an hour, and a candidate now waits 3.56 hours on average. The 3.6 attempts per accepted unit include 35.6 candidates that were paid for in this shift and never reviewed in it.

With truly shared files, more agents make the line slower#

Now let the agents edit the same files, so each extra agent spoils 0.15 of the attempts instead of 0.05:

                              two agents              six agents
conflict share                0.15 x 1 = 0.15         0.15 x 5 = 0.75
first-pass yield              0.7 x 0.85 = 0.595      0.7 x 0.25 = 0.175
candidates per hour           6 x 0.595 = 3.57        18 x 0.175 = 3.15
accepted per hour             min(3.57, 5) = 3.57     min(3.15, 5) = 3.15
attempts per accepted unit    48 / 28.56 = 1.681      144 / 25.2 = 5.714
total cost                    852.48                  1113.6
cost per accepted             29.849                  44.19

Accepted output falls from 3.57 to 3.15 per hour. Triple the agents, and the line is slower and each accepted unit costs about half again as much. The reviewer is not even the bottleneck any more: conflicts are. This is what the backlog chapter's parallelism decision meant by "parallel over shared files: never by default".

The same numbers from the code#

The model's core, quoted verbatim from examples/sim/economics.py : economics (the function body, dedented by one level so the excerpt compiles on its own):

conflict = min(0.9, float(p["conflict_per_agent"]) * (agents - 1))
fpy = float(p["base_yield"]) * (1 - conflict)
attempt_cap = agents * 60 / float(p["gen_minutes"])
candidates = attempt_cap * fpy
review_cap = reviewers * 60 / float(p["review_minutes"])
accepted_ph = min(candidates, review_cap)
bottleneck = "review" if review_cap < candidates else "generation"
attempts = attempt_cap * hours
accepted = accepted_ph * hours
queue_end = max(0.0, (candidates - review_cap) * hours)
review_hours = reviewers * hours  # reviewers are paid for the shift
escapes = accepted * float(p["escape_rate"])
total = attempts * float(p["attempt_cost"]) + review_hours * float(p["reviewer_hour_cost"]) + escapes * float(p["escape_cost"])
per_accepted = total / accepted if accepted else None
per_generated = total / attempts if attempts else None
wait_h = (queue_end / 2) / review_cap if review_cap else None

Run in a scratch copy with /opt/homebrew/bin/python3 economics.py on 2026-11-20 (all six presets, one JSON line each; the lines scroll):

default {"first_pass_yield": 0.665, "conflict_share": 0.05, "attempts_per_hour": 6.0, "candidates_per_hour": 3.99, "review_capacity_per_hour": 5.0, "accepted_per_hour": 3.99, "bottleneck": "generation", "attempts": 48.0, "accepted": 31.92, "queue_at_end": 0.0, "avg_review_wait_hours": 0.0, "escapes": 1.277, "total_cost_units": 879.36, "cost_per_accepted": 27.549, "cost_per_generated": 18.32, "attempts_per_accepted": 1.504}
more-agents {"first_pass_yield": 0.525, "conflict_share": 0.25, "attempts_per_hour": 18.0, "candidates_per_hour": 9.45, "review_capacity_per_hour": 5.0, "accepted_per_hour": 5.0, "bottleneck": "review", "attempts": 144.0, "accepted": 40.0, "queue_at_end": 35.6, "avg_review_wait_hours": 3.56, "escapes": 1.6, "total_cost_units": 1232.0, "cost_per_accepted": 30.8, "cost_per_generated": 8.556, "attempts_per_accepted": 3.6}
more-reviewers {"first_pass_yield": 0.525, "conflict_share": 0.25, "attempts_per_hour": 18.0, "candidates_per_hour": 9.45, "review_capacity_per_hour": 15.0, "accepted_per_hour": 9.45, "bottleneck": "generation", "attempts": 144.0, "accepted": 75.6, "queue_at_end": 0.0, "avg_review_wait_hours": 0.0, "escapes": 3.024, "total_cost_units": 2476.8, "cost_per_accepted": 32.762, "cost_per_generated": 17.2, "attempts_per_accepted": 1.905}
pilot {"first_pass_yield": 0.6, "conflict_share": 0.0, "attempts_per_hour": 3.0, "candidates_per_hour": 1.8, "review_capacity_per_hour": 5.0, "accepted_per_hour": 1.8, "bottleneck": "generation", "attempts": 24.0, "accepted": 14.4, "queue_at_end": 0.0, "avg_review_wait_hours": 0.0, "escapes": 0.576, "total_cost_units": 667.2, "cost_per_accepted": 46.333, "cost_per_generated": 27.8, "attempts_per_accepted": 1.667}
shared-files {"first_pass_yield": 0.175, "conflict_share": 0.75, "attempts_per_hour": 18.0, "candidates_per_hour": 3.15, "review_capacity_per_hour": 5.0, "accepted_per_hour": 3.15, "bottleneck": "generation", "attempts": 144.0, "accepted": 25.2, "queue_at_end": 0.0, "avg_review_wait_hours": 0.0, "escapes": 1.008, "total_cost_units": 1113.6, "cost_per_accepted": 44.19, "cost_per_generated": 7.733, "attempts_per_accepted": 5.714}
shared-files-2 {"first_pass_yield": 0.595, "conflict_share": 0.15, "attempts_per_hour": 6.0, "candidates_per_hour": 3.57, "review_capacity_per_hour": 5.0, "accepted_per_hour": 3.57, "bottleneck": "generation", "attempts": 48.0, "accepted": 28.56, "queue_at_end": 0.0, "avg_review_wait_hours": 0.0, "escapes": 1.142, "total_cost_units": 852.48, "cost_per_accepted": 29.849, "cost_per_generated": 17.76, "attempts_per_accepted": 1.681}

The hand arithmetic and the code agree to the last digit. The more-reviewers preset (six agents, three reviewers) shows the honest fix for the first case: 9.45 accepted per hour, no queue, at 32.762 per accepted unit, because conflicts still spoil a quarter of the attempts.

The per-unit figures earlier on this page (lead, cycle and queue time, the station capacities, yield and the cost percentiles) came from a short script written for this page and run beside a scratch copy of examples/sim/. It counts each unit's attempts by wrapping the simulator's failure decision:

"""metrics_view.py: read the batch simulator the way the metrics chapter does.

Teaching reconstruction; every number is simulated, illustrative. Run beside a COPY of examples/sim/.
Cost units are this page's own assignment: 3 units per attempt at the two model stations
(solution, visualization), 5 units per review attempt (5 reviewer minutes at 60 units an hour).
"""
from collections import defaultdict

import batch_sim

COST = {"solution": 3.0, "visualization": 3.0, "review": 5.0}
attempts = defaultdict(lambda: defaultdict(int))  # unit -> station index -> attempts
captured = []
_fails, _pct = batch_sim.fails, batch_sim.percentile


def counting_fails(seed, unit, station, attempt, rate):
    attempts[unit][station] += 1
    return _fails(seed, unit, station, attempt, rate)


def recording_percentile(values, p):
    captured.append(list(values))
    return _pct(values, p)


batch_sim.fails = counting_fails
batch_sim.percentile = recording_percentile


def pct(values, p):
    return _pct(sorted(values), p)


for name in ("default", "bottleneck"):
    attempts.clear()
    captured.clear()
    params = batch_sim.preset(name)
    line = params["stations"]
    r = batch_sim.simulate(params)
    lead, cycle = captured[0], captured[2]
    accepted_units = sorted(u for u in attempts if u not in r["failed_units"])
    work = [sum(attempts[u][i] * line[i]["minutes"] for i in range(len(line))) for u in accepted_units]
    queue = [c - w for c, w in zip(cycle, work)]
    print(f"== {name}: accepted {r['accepted']}, makespan {r['makespan_minutes']} min, "
          f"throughput {r['throughput_per_hour']}/h, bottleneck {r['bottleneck']}")
    print(f"   lead p50/p90 {pct(lead, 50)}/{pct(lead, 90)}  cycle p50/p90 {pct(cycle, 50)}/{pct(cycle, 90)}  "
          f"work p50 {pct(work, 50)}  queue p50/p90 {pct(queue, 50)}/{pct(queue, 90)}")
    print(f"   flow efficiency (work / cycle, mean): {sum(work) / sum(cycle):.3f}")
    seq = 0.0
    for i, s in enumerate(r["stations"]):
        per_acc = s["attempts"] / r["accepted"]
        seq += per_acc * line[i]["minutes"]
        cap = line[i]["workers"] * 60 / (line[i]["minutes"] * per_acc)
        print(f"   {s['id']:13} attempts/accepted {per_acc:.3f}  capacity {cap:5.2f} accepted/h  util {s['utilization']:.2f}")
    print(f"   sequential (one unit at a time): {seq:.2f} min per accepted unit = {60 / seq:.2f}/h")
    cost_unit = {u: sum(attempts[u][i] * COST.get(line[i]['id'], 0.0) for i in range(len(line))) for u in attempts}
    total = sum(cost_unit.values())
    acc_costs = [cost_unit[u] for u in accepted_units]
    print(f"   cost units: total {total:.0f}, per accepted {total / r['accepted']:.3f}, "
          f"p50 {pct(acc_costs, 50):.0f}, p90 {pct(acc_costs, 90):.0f}, max {max(acc_costs):.0f}")
    print(f"   first-pass units (no rework anywhere): "
          f"{sum(1 for u in accepted_units if sum(attempts[u].values()) == len(line))} of {r['accepted']}")

Output on 2026-11-20:

== default: accepted 60, makespan 398 min, throughput 9.045/h, bottleneck visualization
   lead p50/p90 223/358  cycle p50/p90 223/358  work p50 42  queue p50/p90 171/299
   flow efficiency (work / cycle, mean): 0.221
   spec          attempts/accepted 1.033  capacity 29.03 accepted/h  util 0.31
   solution      attempts/accepted 1.283  capacity 15.58 accepted/h  util 0.58
   oracle        attempts/accepted 1.050  capacity 19.05 accepted/h  util 0.47
   visualization attempts/accepted 1.167  capacity 10.29 accepted/h  util 0.88
   review        attempts/accepted 1.150  capacity 10.43 accepted/h  util 0.87
   sequential (one unit at a time): 49.08 min per accepted unit = 1.22/h
   cost units: total 786, per accepted 13.100, p50 11, p90 17, max 24
   first-pass units (no rework anywhere): 30 of 60
== bottleneck: accepted 60, makespan 397 min, throughput 9.068/h, bottleneck visualization
   lead p50/p90 223/353  cycle p50/p90 223/353  work p50 42  queue p50/p90 169/306
   flow efficiency (work / cycle, mean): 0.222
   spec          attempts/accepted 1.033  capacity 29.03 accepted/h  util 0.31
   solution      attempts/accepted 1.283  capacity 46.75 accepted/h  util 0.19
   oracle        attempts/accepted 1.050  capacity 19.05 accepted/h  util 0.48
   visualization attempts/accepted 1.167  capacity 10.29 accepted/h  util 0.88
   review        attempts/accepted 1.150  capacity 10.43 accepted/h  util 0.87
   sequential (one unit at a time): 49.08 min per accepted unit = 1.22/h
   cost units: total 786, per accepted 13.100, p50 11, p90 17, max 24
   first-pass units (no rework anywhere): 30 of 60

The two runs differ only where the extra workers act: the solution station's capacity and utilization, and a few minutes in the tail of the clocks.

In this simulator every unit is released at minute 0 with no WIP limit, so lead time equals cycle time; with the WIP limit of the backlog chapter the two separate (239 against 52 minutes at the median).

The rest of the measurement set#

A factory record that already holds runs, station attempts, artifacts and verdicts can compute every row of this table without new instrumentation. The examples column says where each one is visible on this site.

MeasureDefinitionComputed fromOn this site
Reviewer effortperson-minutes of review per accepted unitreview attempts, review timeeconomics model: 8 reviewer hours for 31.92 or 40 accepted units
Repeatabilitythe same inputs on the same setup give the same artifacttwo runs, compared byte by byteExample A and Example D: byte-identical reruns
Reproducibilityanother person or machine rebuilds the same artifact from the recordpattern version, spec hash, artifact hashthe manifest of the smallest factory
Rollback rateshare of deployments rolled backrelease recordsgates lab: canary at 3.4% rolls back automatically
Time to recoveryfrom failure detected to service restoredincident and release recordsFailure and recovery scenarios
Incident countproduction incidents traced to factory-built artifacts, per periodincident records with artifact idsnone in the examples (they deploy nothing)
Spec ambiguity rateshare of specs sent back with questions before any station runsspec gate resultsExample A: 1 of 3 specs rejected at S1 with six errors
Pattern reuse rateshare of products built from an approved pattern version without an exceptionproduct catalogblast lab: 8 products on 3 patterns, one pinned to an older line
Escalation shareshare of tasks that end with a persontask statusesExample D: 1 of 24 tasks escalated, in 1 of 4 runs

DORA's delivery metrics, and where they sit#

Guideline (DORA, read 2026-11-20): DORA now publishes five software delivery performance metrics. Change lead time, deployment frequency and failed deployment recovery time are grouped as throughput; change fail rate and deployment rework rate are grouped as instability. Failed deployment recovery time covers only recovery from failures caused by a production change, and deployment rework rate, added in 2024, is the share of deployments that are unplanned work to fix a problem in production.

These measure the product's delivery pipeline: what happens to a change from commit to production. A factory needs them for the products it builds, and needs its own line metrics as well, because a factory can deploy often and recover fast while spending three attempts on every accepted unit. The pages read do not say how to measure DORA's metrics for agent-generated changes; this page does not claim they do.

The quality denominator#

3,246 units, or any count of artifacts, do not by themselves establish production correctness, business value, or that the units were of comparable difficulty. The WatchAlgo story reconciles the number; the rule here is general. Every count needs a denominator (registered, complete, published, accepted) and a quality measure beside it (escapes found, human sample verdicts), or it is a numerator waiting to mislead someone.

What this is not. This page measures the factory's line. Graders, golden sets and release-quality scores for a model's output belong to Eval Engineering; the factory invokes them as gates and counts their verdicts here. Collecting spans and metrics from running systems belongs to Observability Engineering. One change's path through review and CI belongs to Diff to Deploy.
In a coding agent. A coding agent running headless at a station is one source of attempts in this arithmetic. Count its attempts, its run time and its usage per work item, and count it as accepted only when the factory's own tests and a person with merge rights say so. An agent's summary that it "fixed all failing tests" is an attempt, not an accepted unit.

How to estimate economic benefit without confusing volume with outcome#

This page owns one of the site's decisions.

The question. A team proposes more agents, a new pattern, or a factory for a class of products. How do you estimate whether it pays, without mistaking more output for more value?

Diagram: Estimating economic benefit · four questions before anyone quotes a number

Ask the four questions in order. Is the numerator accepted units, not generated artifacts or tasks? Does the cost include every attempt, failed and abandoned ones too, every reviewer hour, and the expected cost of escapes? Is there a measured baseline of the same kind of work done the old way, on the same denominators? Does the accepted unit connect to an outcome someone pays for, such as a deployed capability, a shorter lead time for a business request, or fewer incidents? A no at any step sends the estimate back to fix that measure. Only four yes answers justify quoting a benefit.

  1. Questions
    1. accepted in the numerator?: Count units that passed every gate, not attempts or files written.
    2. full cost?: Failed and abandoned attempts, the reviewers' time and later escapes all belong in the numerator.
    3. measured baseline?: Compare against a sample of the same kind of work, measured the same way.
    4. tied to an outcome?: The accepted unit must reach something a person pays for, and escapes count against it.
  2. Answer
    1. quote the benefit: All four measures hold; report median and 90th percentile with the baseline.
    2. fix the measure first: A missing measure makes any benefit figure a guess, often an inflated one.

The diagram asks four questions in order. Is the numerator accepted units, not generated artifacts? Does the cost include every attempt, every reviewer hour and the expected cost of escapes? Is the comparison against a measured baseline of the same kind of work, done the old way? And does the accepted unit connect to an outcome someone pays for, such as a service in production, a shorter lead time for a business request, or fewer incidents? Each "no" sends the estimate back for that measure before anyone quotes a benefit.

OptionWhat it countsChoose it whenIt misleads when
Output volumeartifacts, tasks, linesnever as a benefit; only as loadalways, as a benefit
Cost per generated artifactspend ÷ artifactscomparing station efficiency before acceptanceretries or conflicts rise: it falls while value falls
Cost per accepted unitall spend ÷ acceptedcomparing line designs on the same workunits differ in difficulty or value
Lead time and cost to an outcomespend and time per deployed, used capabilitydeciding whether to fund the factorythe baseline is a guess

The criteria. Use the same denominator before and after. Include reviewer time, because review is usually the bottleneck. Include escapes at a cost someone signs off. Compare against a pilot on a sample of the same work, and report percentiles, not only means.

The counterexample. Going from two to six agents halves cost per generated artifact (18.32 to 8.556) while cost per accepted unit rises (27.549 to 30.8). With shared files the line also gets slower (3.57 to 3.15 accepted per hour). Simulated, illustrative. A benefit case built on the first number funds the wrong change.

The failure modes. Counting volume rewards retries. Leaving out reviewers hides the true bottleneck and overstates savings by the reviewers' salaries. Ignoring escapes moves cost from the factory's budget to the support team's. Comparing a factory's easy units against a team's hard ones credits the factory with a difference in difficulty.

Try it#

The economics lab runs the same model in the browser, with sliders for agents, yield, conflicts, reviewers, review minutes, costs and escapes. The first embed starts at the default; the second at six agents on shared files. The third embed is the batch simulator with tripled solution workers in front of the bottleneck. The agent here is a scripted stand-in, so these results show how the factory mechanics work, not how good any model is.

The same results for a reader without JavaScript (from examples/sim/economics.py, illustrative cost units, one 8-hour shift; generation 20 minutes per attempt, base yield 0.7, review 12 minutes, attempt 3 units, reviewer hour 60 units, escape rate 4%, escape cost 200 units):

PresetAgents · reviewersFirst-pass yieldAccepted per hourBottleneckReview queue at end · average waitCost per acceptedCost per generated
default2 · 10.6653.99generation0 · 0 h27.54918.32
more-agents6 · 10.5255.0review35.6 · 3.56 h30.88.556
more-reviewers6 · 30.5259.45generation0 · 0 h32.76217.2
pilot1 · 1 (yield 0.6)0.61.8generation0 · 0 h46.33327.8
shared-files-22 · 1, conflict 0.15 per extra agent0.5953.57generation0 · 0 h29.84917.76
shared-files6 · 1, conflict 0.15 per extra agent0.1753.15generation0 · 0 h44.197.733

The batch results without JavaScript (from examples/sim/batch_sim.py, simulated, illustrative; 60 units; stations spec 4 min and 2 workers, solution 12 min and 4 workers, oracle 6 and 2, visualization 15 and 3, review 5 and 1; at most 3 attempts per station):

PresetWhat changesMakespan (min)Accepted · escalatedThroughput per hourLead time p50 · cycle time p50 (min)Bottleneck
defaultnothing39860 · 09.045223 · 223visualization (88% busy)
bottlenecksolution workers 4 → 1239760 · 09.068223 · 223visualization (88% busy); solution falls to 19% busy
wipWIP limit 8 units in the line45760 · 07.877239 · 52visualization
contentionsolution workers 8, rework 40% (shared-file conflicts)37255 · 58.871202 · 202visualization

WatchAlgo in practice#

WatchAlgo's records support some of this page's measures and not others. Each figure carries its label; all come from docs/EVIDENCE.md, section 9.

Observed in the repository (local run.log, September 2026): first-run code yield of 91.4% for visualization codes and 91.9% for solution codes in productive tasks (91.1% and 88.2% for first-attempt tasks), where acceptance means the factory's own validators. Cycle time per worker task had a median of 14 minutes and a 90th percentile of 19 to 21 minutes, productive tasks only. The retry storm left 10,630 zero-minute tasks: wasted attempts that a cost-per-accepted measure would have charged to the units of the next productive window.

Observed (.ai-context/VIZ_SOLUTION_BUGS_FOUND.md, commit 4426f119): 86 solution defects in one category of 516 units were found after acceptance by the tracer cross-check. That is an escaped-defect count for one category, not a rate for the corpus.

Not available: token cost per accepted unit (no usage telemetry; the only figure is the author's working estimate of about 25K tokens per 5 problems with Sonnet), lead time, human effort, rollback and incident counts. None was recorded, and this page computes none of them.

Reported by the author: 3,246 implemented problems and runs of 30+ hours. Neither is a throughput or a cost measure, and the quality denominator above applies to both. Teaching reconstruction: the economics model and the batch simulator are this site's, not WatchAlgo's; their numbers describe no real line.

Enterprise translation#

In the insurance factory of Part III, the unit is a service or workflow variant built from PAT-INTAKE-API and a product spec, such as the broker intake service of factory run FR-0003. Lead time starts when the domain team accepts SPEC-INTAKE-001 and ends when the rollout manifest names a build that passed every gate. Most of that clock is queue time in front of people: the security reviewer, the release approver, the domain owner. Generation time is a small share.

This site's recommendation: measure per accepted service, with reviewer hours in the numerator and escapes found in staging, canary or production in a separate column; report lead time as a median and a 90th percentile; and track the spec ambiguity rate per product line, because specs sent back before any station runs are the cheapest defects a factory ever finds. A benefit case compares a pilot batch of factory-built variants with recent hand-built ones of the same pattern, on the same measures. What dominates the critical path in this domain is review, integration and approval, so the first improvement to measure is usually queue time in front of reviewers, not agent speed.

Failure test#

The failure: a factory adds agents and reports success because cost per generated artifact halved. Accepted output rises only to the reviewer's limit, candidates wait hours, and on shared files accepted output falls.

How the factory catches it: the metric pipeline computes cost per accepted unit from the full record, all attempts and reviewer hours included, and shows the bottleneck and the queue beside it. A rule in the factory's reporting refuses any benefit figure whose numerator is not accepted units.

What to run or toggle: in the economics lab, start from default, raise agents to 6 and watch cost per generated fall while cost per accepted rises and the queue appears; then raise the conflict per extra agent to 0.15 and watch accepted per hour fall below the two-agent line. In a scratch copy of examples/sim/, run /opt/homebrew/bin/python3 economics.py and compare default, more-agents, shared-files-2 and shared-files.

Where to go deeper#

Checklist for Monday#

Exercises#

  1. A line accepts 4 units an hour and holds 12 units in progress on average. How long does a unit spend inside?
Answer 1About 3 hours

Little's law: time inside = units inside ÷ throughput = 12 ÷ 4 = 3 hours.

  1. In the economics model, two agents and one reviewer: what happens to accepted output if you add a second reviewer?
Answer 2Nothing; generation is the bottleneck

Candidates are 3.99 per hour and one reviewer can take 5, so a second reviewer raises review capacity to 10 and accepted output stays 3.99, while cost per accepted rises by the second reviewer's shift.

  1. Three stations each pass 90% of attempts first time. What share of units passes all three without rework?
Answer 3About 73%

Yields multiply: 0.9 × 0.9 × 0.9 = 0.729. Over a quarter of units are reworked somewhere, although each station looks good alone.

Advanced perspective#

Every metric becomes a target, and agents optimize targetsPair each measure with the one that catches its gaming

A measure that steers work will be optimized, and a model at a station optimizes whatever its acceptance check rewards. First-pass yield rises fastest when checks are weakened; cycle time falls fastest when hard units are escalated early; cost per accepted unit falls when review is skipped. None of these improves the product.

The defence is to publish measures in pairs that pull against each other: first-pass yield beside escaped defects, cycle time beside escalation share, cost per accepted unit beside reviewer effort and a sampled human verdict. A change that improves one and worsens its pair is a change in what is being measured, not in the factory. WatchAlgo's case shows the mechanism without any gaming at all: its solution validator checked shape, not expected outputs, so a 91.9% first-run yield sat beside 86 escapes in one category.

Why waits explode near full utilizationVariability, not average load, makes queues

If every unit arrived on a fixed beat and took exactly the same time, a station could run at 100% with no queue. Real work varies: some units need rework, some reviews take three times longer. With variability, a busy station has no slack to absorb a burst, and waiting time rises steeply as utilization approaches 100%; queueing theory gives formulas for the common cases, and all of them share that shape.

Two consequences follow for a factory. Run human review, the station with the most variable service time, well below full load if lead time matters. And reduce variability before adding capacity: smaller, more uniform units and clearer specs shorten queues at the same utilization. The economics model on this page is steady-state and ignores variability, so its waits are a lower bound.

What the economics model leaves outIts assumptions, and how each would move the answer

The model assumes a steady state for one shift, agents running at full speed, reviewers paid for the whole shift whether busy or idle, a conflict penalty that grows linearly with each extra agent and caps at 0.9, a fixed escape rate per accepted unit, and an average wait from a linearly growing queue. Each assumption is a simplification of something real.

Relax them one at a time and the direction of the answer usually holds. Conflicts that grow faster than linearly make shared-file parallelism worse. Reviewers who tire make the review bottleneck tighter. Escape rates that rise with review pressure make the six-agent line dearer still. The model's purpose is the shape of the arithmetic, accepted versus generated, and that shape survives. Its numbers do not transfer to any real line; measure your own.

Attributing a benefit to the factoryBaselines, mix shifts and the units nobody chose to build

A team that adopts a factory usually changes several things at once: a new pattern, clearer specs, more tests, and agents. Lead time falls, and the factory gets the credit. Some of it belongs to the spec discipline, which would have helped a human team as much.

Three practices keep attribution honest. Hold a baseline: a recent sample of the same kind of work, done the old way, measured on the same denominators. Watch the mix: if the factory receives the standard, well-specified units and people keep the hard ones, compare like with like or the difference is difficulty, not method. And count the units nobody built: work the factory could not take because no pattern fit is part of the economics, since it still costs the organization the old amount.

In one breath#

A factory is measured by accepted units: how long they take from accepted spec to deployable artifact, how much of that is waiting, how many passed first time, how many escaped, and what each accepted one cost with every attempt and reviewer hour included. The line goes as fast as its slowest station, Little's law ties waiting to the units held inside, and yields multiply across stations. In the worked example, six agents instead of two halve the cost per generated artifact, raise the cost per accepted unit, and, on shared files, make the line slower.

Metrics say that something went wrong; they do not say what, or how to recover. Failure and recovery scenarios catalogs the ten ways a factory breaks, from half-written artifacts to irreversible external writes, with what detects each one and what restores a known good state.