All chapters
  1. Start hereA software factory, from first principles
  2. Primitive · 1A factory, before AI: input, transformation, check, output
  3. Primitive · 2From repetition to a pattern: what stays, what varies, what to leave alone
  4. Primitive · 3The specification as an executable contract
  5. Primitive · 4Build the smallest factory: four stations, two specs, one manifest
  6. Primitive · 5Assembly stations: typed contracts, owners and failure handling
  7. Primitive · 6AI enters the factory: what a model adds, and what stays deterministic
  8. WatchAlgo · 7The WatchAlgo production story: what the repository shows
  9. WatchAlgo · 8A problem as a production unit: the files one unit must contain
  10. WatchAlgo · 9Station by station: description, solution, implementations, visualization
  11. WatchAlgo · 10The visualizer factory: from algorithm state to renderable steps
  12. WatchAlgo · 11Three flavors and three languages: two different contracts
  13. WatchAlgo · 12Looping over a large backlog: scheduling, limits and a failed-case queue
  14. WatchAlgo · 13Long-running agent execution: context, checkpoints and token limits
  15. WatchAlgo · 14Evidence of correctness: tests the producer cannot talk its way past
  16. WatchAlgo · 15The factory learning loop: defects change the pattern, not only the unit
  17. Enterprise · 16Classify the enterprise demand before choosing a pattern
  18. Enterprise · 17Reference enterprise patterns and their variation points
  19. Enterprise · 18Platform foundations: golden paths, templates, registries and catalogs
  20. Enterprise · 19Pattern versus configuration versus business logic
  21. Enterprise · 20Insurance factory, part 1: building the intake contract
  22. Enterprise · 21Insurance factory, part 2: building the underwriting workflow
  23. Enterprise · 22Insurance factory, part 3: trust and observability built in
  24. Enterprise · 23Data and integration engineering: contracts that survive change
  25. Enterprise · 24The CI/CD and deployment factory: from pull request to production
  26. Enterprise · 25Enterprise governance: owners, approved patterns and provenance
  27. Enterprise · 26Scaling safely: many teams, many agents, one set of boundaries
  28. Operations · 27Factory quality engineering: measuring accepted output, not activity
  29. Operations · 28Failure and recovery scenarios: what breaks a factory, and what restores it
  30. Operations · 29The landscape: which tool for which job, read from its own documentation
  31. Operations · 30Design an enterprise software factory: the blueprint and the rollout
  32. ReferenceSources and verification
  33. ReferenceGlossary
  34. AboutAbout this site
WatchAlgo · 12

Looping over a large backlog: scheduling, limits and a failed-case queue

Independent work units, work in progress, concurrency limits, retries, deduplication and budgets for thousands of units, with a batch simulator that shows where the queue forms.

You will schedule
Thousands of independent units with a queue, a WIP limit and a concurrency limit, and say what each one protects.
You will see
Why tripling the workers in front of a bottleneck moves throughput by less than one percent (simulated, illustrative).
You will decide
Parallel or not, and one long local loop or a queue of distributed workers, with a counterexample for each.

On the evening of 2 September 2026 the WatchAlgo pipeline had a long backlog and an obvious way to clear it faster: more workers. A background driver ran 8 workers, and 14 more workers ran inside an interactive session, all taking tasks from the same queue. That made 22 Claude sessions on one account at the same time. Within seconds the provider throttled the whole account, and every one of the 14 session workers died.

Observed in the repository (zenalgo-frontend, .ai-context/SESSION_STATE.md, commit 0309048f): the note written that night reads "keep total concurrent Claude sessions ≤ ~8". The supervisor's default became 8 workers (scripts/viz/supervise.sh). This is a recorded observation from the author's sessions, not a measured capacity curve, and the evidence notes treat it that way (docs/evidence/git-history.md, section 7).

Nothing was wrong with the workers, the queue or the checks. The line had a limit that lived outside the machine, and nobody had written it into the schedule.

The learner's question: when a factory has thousands of independent units to build, how does it keep the line full, never build the same unit twice, keep one failure from blocking the rest, and never run more at once than the slowest shared resource allows?

A backlog is work that is not done yet#

Plain English. A backlog is everything the factory still has to build. A queue is the short, ordered list of tasks that are ready for a worker to pick up right now. Workers take tasks from the queue, not from the backlog.

Beginner definition: scheduling a backlog means deciding which units enter the line, in what order, how many at a time, and what happens to each one that fails.

A backlog is the set of units the factory has not finished yet. In WatchAlgo it was every problem folder without a solution.json or a visualization.json. The non-AI analogy is a warehouse's open orders. The consequence that matters most: the backlog should be computed from what exists, not typed by hand, so that a crash or a restart cannot make it wrong. The trade-off is a scan of the disk or the database every time the queue is rebuilt.

A queue is the ordered list of tasks ready to be claimed, each with a status such as pending, running, done or failed. A task may hold several units. WatchAlgo grouped 5 problems into a solution task and 8 into a visualization task. The consequence is that the queue, not the worker, holds the state of the work, so a worker can die without taking the plan with it.

Before AI. A Kafka consumer group spreads the partitions of a topic across consumers, and a consumer that dies has its partitions handed to another. A Jenkins build queue holds jobs until an executor is free. A print spooler holds documents until the printer is ready. All three separate "what must be done" from "who is doing it now", which is the first thing a factory scheduler does.

The smallest primitive: a queue a worker can claim from#

The least machinery that works on one machine is a JSON file, a lock and a rule for rebuilding the file from disk. WatchAlgo's September pipeline is exactly that. Observed in the repository (zenalgo-frontend, scripts/viz/pipeline.py, commit 971863f2; read-only, quoted verbatim by this page's writer):

class Locked:
    def __enter__(self):
        PIPE.mkdir(parents=True, exist_ok=True)
        self.f = open(LOCK, "w")
        fcntl.flock(self.f, fcntl.LOCK_EX)
        return self

    def __exit__(self, *a):
        fcntl.flock(self.f, fcntl.LOCK_UN)
        self.f.close()


def save_queue(q: dict) -> None:
    tmp = QUEUE.with_suffix(".tmp")
    json.dump(q, open(tmp, "w"), indent=1)
    os.replace(tmp, QUEUE)


def build(reset: bool) -> None:
    with Locked():
        q = load_queue() if not reset else {"tasks": [], "next_id": 1}
        existing = {(t["type"], n) for t in q["tasks"] for n in t["codes"] if t["status"] != "failed"}
        added = 0
        for cat in CATEGORY_ORDER:
            lo, hi = next((a, b) for a, b, nm in CATEGORIES if nm == cat)
            sol_missing = [n for n in range(lo, hi + 1) if not has_solution(n) and ("solution", n) not in existing]
            for i in range(0, len(sol_missing), SOL_BATCH):
                new_task(q, "solution", sol_missing[i:i + SOL_BATCH]); added += 1
            viz_missing = [n for n in range(lo, hi + 1) if not has_viz(n) and n not in BLOCKED_VIZ and ("viz", n) not in existing]
            for i in range(0, len(viz_missing), VIZ_BATCH):
                new_task(q, "viz", viz_missing[i:i + VIZ_BATCH]); added += 1
        save_queue(q)
    log(f"build: {added} tasks added → {QUEUE}")
    status()

Read it as three rules. Locked takes an exclusive file lock (flock), so only one process edits the queue at a time; every claim, finish and rebuild goes through it. save_queue writes a temporary file and renames it over the real one, so a crash never leaves half a queue under the real name. And build derives the backlog from the disk: a problem is missing work if its file does not exist, and it is queued only if the same (task type, problem) pair is not already in the queue.

That last rule is the queue's deduplication, and its one exception teaches a lot. Pairs that belong to failed tasks are left out of existing, so a rebuild queues failed problems again. That is the right behavior after a fixed cause. During the night the WatchAlgo story tells, it is also why every restart of the driver added the same 272 tasks back: the rebuild kept re-admitting work that failed for a reason nobody had classified.

WIP, WIP limits and concurrency limits#

Work in progress (WIP) is the number of units that have entered the line and are not yet finished: started, waiting between stations, or being reworked. The analogy is the number of orders on the kitchen pass at a restaurant. The consequence is that every unit in progress holds something: a partial artifact, a record, a place in someone's attention. The trade-off of holding many is the subject of the next section.

A WIP limit caps how many units may be inside the line at once; a new unit enters only when one leaves. A Kanban board's column limit is the classic form. A concurrency limit is different: it caps how many workers act at the same time against a shared resource, such as model sessions on one account, connections to one database, or reviewers on one team. A database connection pool size is the pre-AI version.

The two limits protect different things. A WIP limit protects lead time and focus: fewer half-built units, each finishing sooner. A concurrency limit protects the shared resource: above it, the resource throttles, rejects or slows everyone. WatchAlgo's ceiling of about 8 sessions is a concurrency limit set by the account, not by the CPU of the machine. This site's recommendation: set the concurrency limit from the slowest shared resource you can name, and write down which resource it is.

Little's law, in plain words#

There is one relation between these quantities that holds for any stable line, with or without AI. In plain words: the average number of units inside the line equals the rate at which units leave it, times the average time each unit spends inside. It is known as Little's law, and Factory quality engineering is its home on this site.

The batch simulator below makes it concrete. In its default line, with no WIP limit, accepted units leave at 0.1508 per minute and spend 222.6 minutes inside on average, so about 33.6 units are inside at any moment. With a WIP limit of 8, units leave at 0.1313 per minute and spend 56.6 minutes inside, so about 7.4 are inside, under the cap. Simulated, illustrative.

The operational consequence is the one teams find surprising: at the same throughput, the only way to make each unit spend less time in the line is to hold fewer units in it. Releasing the whole backlog at once does not make anything finish sooner. It makes every unit wait longer.

Throughput, takt time and the bottleneck#

Throughput is accepted units per unit of time: not generated, not attempted, accepted. Takt time is the pace the demand sets: available time divided by the units required. If 60 units must be ready within an 8-hour window of 480 minutes, the takt time is 8 minutes per unit. A line keeps up when every station can finish one accepted unit faster than that. The bottleneck is the station with the slowest pace per accepted unit; it sets the ceiling on throughput for the whole line, and every station in front of it only fills its queue.

Pace per accepted unit counts rework. A station that fails 1 attempt in 7 does more than one attempt per accepted unit, so its pace is its minutes per attempt, times attempts per accepted unit, divided by its parallel workers. The table below comes from the simulator's default line (simulated, illustrative):

StationMinutes per attemptWorkersAttempts for 60 acceptedMinutes per accepted unitBusy
spec42622.0731%
solution124773.8558%
oracle62633.1547%
visualization153705.8388%
review51695.7587%

The visualization station is the bottleneck at 5.83 minutes per accepted unit, so this line cannot accept more than about 10 units an hour, however many solution workers it has. That pace is under the 8-minute takt time, so the line keeps up with the demand of 60 units in 480 minutes; the simulator finishes in 398. Review is close behind at 5.75: speed up visualization alone and review becomes the bottleneck.

The executable demonstration: the batch simulator#

examples/sim/batch_sim.py is a deterministic simulation of units flowing through stations. Teaching reconstruction: it is not a model of WatchAlgo and measures nothing; every number it prints is simulated, illustrative. Each station has minutes per attempt, parallel workers and a failure rate; a failed attempt goes back to the end of the same station's queue, and after the third failure the unit leaves the line for the failed-case queue. Failure is decided by a hash, not by chance, so the Python and the JavaScript lab give identical results.

Run in a scratch copy with /opt/homebrew/bin/python3 batch_sim.py on 2026-11-20:

default {"makespan_minutes": 398, "accepted": 60, "failed": 0, "throughput_per_hour": 9.045, "lead_time_p50": 223, "cycle_time_p50": 223, "bottleneck": "visualization"}
bottleneck {"makespan_minutes": 397, "accepted": 60, "failed": 0, "throughput_per_hour": 9.068, "lead_time_p50": 223, "cycle_time_p50": 223, "bottleneck": "visualization"}
wip {"makespan_minutes": 457, "accepted": 60, "failed": 0, "throughput_per_hour": 7.877, "lead_time_p50": 239, "cycle_time_p50": 52, "bottleneck": "visualization"}
contention {"makespan_minutes": 372, "accepted": 55, "failed": 5, "throughput_per_hour": 8.871, "lead_time_p50": 202, "cycle_time_p50": 202, "bottleneck": "visualization"}

Four presets, four lessons. bottleneck triples the solution workers from 4 to 12: throughput moves from 9.045 to 9.068 per hour, and the solution station falls to 19% busy. wip admits at most 8 units at a time: the time a unit spends inside the line (cycle time, p50) drops from 223 to 52 minutes, while the whole batch takes 457 minutes instead of 398. contention returns in the parallelism decision below.

To read the per-station figures and check Little's law, this page used a short script written for it, run beside the copy (it wraps the simulator's percentile function to capture each unit's cycle time):

"""backlog_view.py: run the batch simulator's presets and print what the backlog chapter reads from them.

Teaching reconstruction; every number is simulated, illustrative.
Little's law check: average units inside the line = throughput x average time inside the line.
"""
import batch_sim

captured = []
_orig = batch_sim.percentile


def recording_percentile(values, p):
    captured.append(list(values))
    return _orig(values, p)


batch_sim.percentile = recording_percentile

for name in ("default", "bottleneck", "wip", "contention"):
    captured.clear()
    r = batch_sim.simulate(batch_sim.preset(name))
    cycle = captured[2]  # order in simulate(): lead p50, lead p90, cycle p50, cycle p90
    mean_cycle = sum(cycle) / len(cycle)
    per_min = r["accepted"] / r["makespan_minutes"]
    print(f"{name:10} makespan={r['makespan_minutes']} accepted={r['accepted']} failed={r['failed']} "
          f"throughput/h={r['throughput_per_hour']} cycle_mean={mean_cycle:.1f}")
    print(f"{'':10} little: throughput/min x mean cycle = {per_min:.4f} x {mean_cycle:.1f} = "
          f"{per_min * mean_cycle:.1f} units in the line on average")
    line = batch_sim.preset(name)["stations"]
    for s, cfg in zip(r["stations"], line):
        # station minutes needed per accepted unit, counting rework, divided by parallel workers
        pace = s["attempts"] * cfg["minutes"] / cfg["workers"] / r["accepted"]
        print(f"{'':10}   {s['id']:13} util={s['utilization']:.2f} max_queue={s['max_queue']:2} "
              f"attempts={s['attempts']} failures={s['failures']} min_per_accepted_unit={pace:.2f}")
    if r["failed_units"]:
        print(f"{'':10}   failed-case queue: units {r['failed_units']}")

The output for the two presets this page reads most closely (the full run prints all four):

default    makespan=398 accepted=60 failed=0 throughput/h=9.045 cycle_mean=222.6
           little: throughput/min x mean cycle = 0.1508 x 222.6 = 33.6 units in the line on average
             spec          util=0.31 max_queue=58 attempts=62 failures=2 min_per_accepted_unit=2.07
             solution      util=0.58 max_queue=28 attempts=77 failures=17 min_per_accepted_unit=3.85
             oracle        util=0.47 max_queue= 2 attempts=63 failures=3 min_per_accepted_unit=3.15
             visualization util=0.88 max_queue=20 attempts=70 failures=10 min_per_accepted_unit=5.83
             review        util=0.87 max_queue= 2 attempts=69 failures=9 min_per_accepted_unit=5.75
wip        makespan=457 accepted=60 failed=0 throughput/h=7.877 cycle_mean=56.6
           little: throughput/min x mean cycle = 0.1313 x 56.6 = 7.4 units in the line on average
             spec          util=0.27 max_queue= 6 attempts=62 failures=2 min_per_accepted_unit=2.07
             solution      util=0.51 max_queue= 2 attempts=77 failures=17 min_per_accepted_unit=3.85
             oracle        util=0.41 max_queue= 1 attempts=63 failures=3 min_per_accepted_unit=3.15
             visualization util=0.77 max_queue= 3 attempts=70 failures=10 min_per_accepted_unit=5.83
             review        util=0.75 max_queue= 3 attempts=69 failures=9 min_per_accepted_unit=5.75

Look at max_queue. Without a WIP limit, 58 units wait in front of the first station and 20 in front of the bottleneck. With the limit, no queue is longer than 6. The work is the same; only where it waits has changed.

Diagram: From backlog to one of two exits · a release gate, station queues, a failed-case queue

Left to right. The backlog is derived from what exists on disk. A release gate admits a unit only while fewer units than the WIP limit are inside the line. Each station takes work from its own queue, with at most its share of the concurrency limit acting at once. A failed check sends the unit back to the same station as rework; after the third failure the unit leaves for the failed-case queue, where a person decides. Every other unit leaves as an accepted artifact. The longest queue forms in front of the bottleneck, the station with the slowest pace per accepted unit.

  1. Backlog
    1. backlog: Every unit not yet built, computed from what exists, never typed by hand.
    2. release gate: Admits a unit only while the line holds fewer units than the limit.
  2. Stations and their queues
    1. station queue: Tasks ready to claim, each with a status; the queue holds the plan, not the worker.
    2. workers: At most as many workers act at once as the slowest shared resource allows.
    3. bottleneck: The station with the most minutes per accepted unit; it sets the line's throughput.
  3. Exits
    1. accepted artifact: Only accepted units count, not attempts or generated files.
    2. failed-case queue: Units that used up their attempts, parked with their evidence; the line keeps moving.

The diagram follows a unit from the backlog to one of two exits. A release gate admits a unit only while the line holds fewer than the WIP limit. Each station takes work from its own queue with at most its share of the concurrency limit. A failed check sends the unit back to the same station as rework; a third failure sends it to the failed-case queue, where a person looks at it. Every other unit leaves as an accepted artifact. The thickest queue forms in front of the bottleneck, whatever happens upstream.

Retries, the failed-case queue and duplicates#

A unit that fails a check must not loop forever, and it must not hold up the others. Two rules achieve both. Attempts are bounded per unit (three in the simulator, MAX_ATTEMPTS = 3 in WatchAlgo). And a unit that uses up its attempts leaves the line for the failed-case queue: a separate list of units that the line gave up on, each with its failure evidence, waiting for a person. In the contention run that queue holds units 9, 20, 30, 36 and 43; the other 55 units were accepted around them. The failed-case queue is a backlog of its own, with an owner, and an empty one is only good news if something could have put units there. The difference between a retry and rework, and the classification that must come before either, is the subject of Assembly stations.

Duplicates come in two kinds, and a factory needs a defence against each. A queue duplicate is the same unit queued twice for the same work, which doubles cost and can produce two artifacts that disagree. WatchAlgo's build keys its queue by (task type, problem) to prevent it, and the session notes record the fallback when retries fragmented anyway: rebuild the queue from disk with build --reset. A content duplicate is two units that are really the same product under two ids. Observed in the repository (.ai-context/SESSION_STATE.md, commit 0309048f): wa-000379 is identical to wa-000378; both are "Two Sum". The writer of this page checked the folders read-only: both hold all three files, so the line built the same problem twice. A key on the id could never have caught it, because the ids differ.

Deduplication, then, is choosing the key that says "this is the same work". Identity keys (the id) stop queue duplicates. Content keys (a hash of the normalized problem, or the spec hash of chapter 4) stop content duplicates. Intent duplicates, two different wordings of the same requirement, need a person or a similarity check with a person behind it.

Token budgets as a scheduling input#

A token budget is a cap on how much model usage one unit, or one run, may consume before the line stops spending on it. It belongs in the schedule because cost and usage limits are shared resources too, and it should be counted per accepted unit, because a unit that needs three attempts costs three times as much as one that passes first time. Long-running agent execution shows Example B's budget guard, which checks before each call and stops with budget_exhausted.

Token budgets in the schedule, and what WatchAlgo recordedPer accepted unit; shared budgets behave like ceilings

Two consequences matter for scheduling. A budget counted per generated artifact looks better the more the line retries, which is backwards; counted per accepted unit, it rises with rework and shows the real cost of a weak station. And a budget shared by every worker on an account behaves like the concurrency ceiling: when it runs out, it runs out for everyone in the same minute, so the scheduler must pause the whole line rather than let each worker discover the limit on its own.

Observed in the repository: WatchAlgo kept no per-task token log; the only figures are the author's working estimates in SESSION_STATE.md of about 25K tokens per 5 problems with Sonnet against about 170K with Opus (docs/evidence/git-history.md, section 8). Token cost per accepted unit is therefore not available for this case study, and this page does not compute one.

Should these units run in parallel?#

This page owns one of the site's decisions: parallelism, weighed as independence against shared-file conflicts and integration overhead.

The question. Two units, or two stations, could run at the same time. Should they?

Diagram: Should these tasks run in parallel? · three questions, in order

Ask about one pair of tasks. First: do they share anything both may write, a file, a row, a shared resource? If yes, run them in sequence or partition the shared state first. Second: with one more worker, is the slowest shared resource still below its ceiling? If not, more parallel workers only buy throttling. Third: can the integration and review step absorb what the parallel workers produce? If not, parallel work only lengthens the queue in front of it. Three safe answers make parallelism pay.

  1. Questions
    1. disjoint writes?: Each task writes only its own unit's files or rows.
    2. below the ceiling?: The shared limit (account, quota, team) has room for another worker.
    3. downstream keeps up?: The step after the parallel work is faster than its combined output.
  2. Answer
    1. run in parallel: Independent units, room under the ceiling, a fast enough downstream.
    2. run in sequence: Shared writes, a full ceiling or a slow downstream: parallel work becomes rework or waiting.

The diagram asks three questions about a pair of tasks. Do they share any output file, row or resource that both may write? Is the shared limit (an account, a quota, a review team) still below its ceiling with one more worker? And is the work after them, integration and review, fast enough to absorb what parallel work produces? Only three answers in the safe direction make parallelism pay; otherwise run them in sequence, or partition the shared state first.

OptionChoose it whenIt buysIt costs
SequentialTasks write the same files, or the bottleneck is downstreamNo conflicts; one simple logThroughput capped by one worker
Parallel over independent unitsEach unit has its own folder or rows (WatchAlgo's one folder per problem)Throughput up to the shared limitA queue, a lock, a concurrency limit to maintain
Parallel over shared filesNever by default; only with partitioning, locks or merge checksSpeed on paperConflicts become rework, and rework lands at the integration station

The criteria, in order: is each task's write set disjoint from every other running task's? Is the slowest shared resource below its ceiling? Is the integration or review station faster than the combined output of the parallel workers? Does a failure in one task leave the others valid?

The counterexample, simulated, illustrative: in the contention preset the solution station gets 8 workers instead of 4, and because they now touch shared files, its rework rate rises from 20% to 40%. Its attempts climb from 77 to 97, five units end in the failed-case queue, and throughput falls from 9.045 to 8.871 accepted units per hour. More parallel workers made the line slower. Factory quality engineering works the same effect out in cost units.

The failure modes. Parallel work on shared state turns into merge conflicts and silent overwrites. Parallel work above a shared ceiling turns into throttling, which on 2 September 2026 killed 14 workers at once. Parallel work in front of a slow integration step only lengthens its queue.

One long local loop, or a queue of distributed workers?#

The second decision this page owns: a long-running local CLI loop against queue-based distributed workers.

The question. Should the factory run as one process on one machine, starting command-line workers in a loop, or as a queue service with workers on many machines?

OptionChoose it whenIt buysIt costs
Local loop, file queue, CLI workers (WatchAlgo, September)One operator, one machine, independent units, a shared limit far below the machine's capacityAlmost no infrastructure; the disk is the source of truthOne machine is a single point of failure; a file lock does not span machines; logs live on one laptop
Queue service with distributed workersSeveral teams, more work than one machine, steps that wait hours for people, audit needsDurability, workers that come and go, central recordsA service to run, credentials per worker, a second place where state can disagree

The criteria: how many machines the work needs; whether any step waits for a person; who must read the records besides the operator; and, before anything else, what the real ceiling is. Product (Temporal, as documented on 2026-11-20): a worker process polls a task queue, runs the code for each task and reports results to the service, which schedules work and records history but does not run the code itself. That is the shape of the second option, with the history kept for you.

The counterexample. Distribution does not raise a ceiling that lives outside the machines. WatchAlgo's limit was the account: moving its 8 workers to eight servers would have changed nothing about the throttling at 22 sessions. A queue service is worth its cost when the constraint is durability, coordination or audit, not when it is a shared quota.

The failure modes. A local loop keeps its records on one machine: the history of WatchAlgo's 2,163 September visualizations exists only on the author's machine, on a branch with no upstream (docs/EVIDENCE.md, section 13). A distributed fleet adds partial failures between machines and needs the durable record of the next chapter to say what is done.

What this is not. This page schedules units through one factory. The control plane that schedules many agents in general belongs to Agent Orchestration Engineering. Quotas, tenant boundaries, locks and scoped credentials across many teams belong to Scaling safely. The full metric set, including Little's law worked as arithmetic, belongs to Factory quality engineering.
In a coding agent. A coding agent such as Claude Code or Codex, running headless as a station worker, is one slot of the concurrency limit. The queue gives it one work item and a scoped checkout; when it exits, the factory's own tests, not its summary, decide whether the unit is accepted, and the queue records the verdict. Two such agents should never hold work items that write the same files, and a person with merge rights reviews what is accepted.

Try it#

The batch lab runs the same simulator in the browser. Change units, the WIP limit, the maximum attempts, or any station's minutes, workers and failure rate, and watch the queues, the bottleneck and the failed-case queue move. The second embed starts with the WIP limit of 8.

The same results for a reader without JavaScript (from examples/sim/batch_sim.py, simulated, illustrative; 60 units; stations spec, solution, oracle, visualization and review as in the table above; at most 3 attempts per station):

PresetWhat changesMakespan (min)Accepted · escalatedThroughput per hourLead time p50 · cycle time p50 (min)Bottleneck
defaultnothing39860 · 09.045223 · 223visualization (88% busy)
bottlenecksolution workers 4 → 1239760 · 09.068223 · 223visualization (88% busy); solution falls to 19% busy
wipWIP limit 8 units in the line45760 · 07.877239 · 52visualization
contentionsolution workers 8, rework 40% (shared-file conflicts)37255 · 58.871202 · 202visualization

WatchAlgo in practice#

Observed in the repository (zenalgo-frontend, scripts/viz/pipeline.py, commit 971863f2; scripts/viz/supervise.sh, commit 017ed6b9): the September backlog was derived from disk by build, in a fixed category order. Two task types shared one queue: solution tasks of 5 problems (SOL_BATCH) and viz tasks of 8 (VIZ_BATCH), and a viz task could not be claimed until every problem in it had a solution.json. A problem that failed verification got a new retry task, up to MAX_ATTEMPTS = 3. Output matching the rate-limit pattern paused all workers for 12 minutes and re-queued the task without spending an attempt. The supervisor started 8 workers by default, 25 seconds apart.

Observed (SESSION_STATE.md, commit 0309048f; commit message of 971863f2): the concurrency ceiling of about 8 sessions; 22 sessions at once triggered account-wide throttling; one run used 24 workers. These are recorded observations, not measurements. Observed (pipeline.py, BLOCKED_VIZ): 13 problems were excluded from the visualization queue because their reference code was broken beyond what a test case could route around; they wait for a person, which makes BLOCKED_VIZ a hand-kept failed-case queue. Observed (SESSION_STATE.md): the earlier March batches ran 10 agents of 5 problems each; the notes record "some timeout/truncation" with 10 problems per agent and settle on 5.

Reported by the author: a repeatable process ran for 30+ hours until token limits required intervention; the WatchAlgo story reads that report against the logs and tells the retry storm in full. Teaching reconstruction: the WIP limit and the explicit failed-case queue with an owner are this site's additions; the September pipeline had a concurrency limit and bounded retries, but no cap on units in progress beyond the worker count.

Enterprise translation#

In the insurance factory of Part III, the backlog is not 3,246 algorithm problems but a few dozen services and workflow variants built from PAT-INTAKE-API, each for a different product or region. The scheduling vocabulary carries over unchanged; what changes is where the limits live. The concurrency limit is set by shared CI runners, the model quota of the platform account and, above all, the number of reviewers with authority to approve a change. The WIP limit should be set by the review team: units generated faster than people can review them only wait, and Little's law says they wait longer the more of them are released.

This site's recommendation: give the failed-case queue to the domain team that owns each product's spec; give the concurrency ceilings to the platform team, each written down with the resource it protects; and key deduplication on the specification hash. Two services that share a schema or a library are one unit for the purpose of parallelism. What dominates the critical path is the queue in front of human review and integration, not the generation time of any station.

Failure test#

The failure: a scheduler releases the whole backlog and starts as many workers as the machine allows. The shared limit, an account or a quota, throttles every worker at once; meanwhile the queues in front of the bottleneck grow, and every unit takes longer to finish.

How the factory catches it: a concurrency limit set from the named shared resource keeps the worker count below the ceiling; a pause shared by all workers handles the limit when it is hit anyway; a WIP limit keeps the queue in front of the bottleneck short. The simulator shows the WIP half: the default line holds 58 units in its first queue and 20 in front of visualization; the wip preset holds at most 6 anywhere.

What to run or toggle: in the lab, start from default, raise the solution workers to 12 and watch throughput stay flat; then set the WIP limit to 8 and watch the cycle time fall. In a scratch copy of examples/sim/, run /opt/homebrew/bin/python3 batch_sim.py and compare the four presets.

Where to go deeper#

Checklist for Monday#

Exercises#

  1. A line accepts 6 units an hour and each unit spends 2 hours inside it. About how many units are in progress?
Answer 1About 12

Little's law: 6 per hour times 2 hours gives about 12 units inside the line on average.

  1. In the simulator's default line, why does tripling the solution workers change almost nothing?
Answer 2Solution is not the bottleneck

Visualization needs 5.83 minutes per accepted unit and sets the ceiling; extra solution workers only move the queue from one station to the next.

  1. Two problems have different ids and identical statements. Which deduplication key would have caught them?
Answer 3A content key

A hash of the normalized statement (or the spec hash). An id key cannot, which is how wa-000379 and wa-000378 were both built.

Advanced perspective#

Why a WIP limit lengthens the batch and still winsMakespan against time in the line

In the simulator a WIP limit of 8 stretched the batch from 398 to 457 minutes and cut the median time a unit spends inside the line from 223 to 52 minutes. Both numbers are real outputs of the model and both are simulated, illustrative. The longer makespan comes from starving the bottleneck at moments when its queue is empty because the release gate was closed. The shorter cycle time comes from units no longer waiting behind 50 others.

Which number matters depends on what the factory is for. A one-off batch that nobody reads until the end may prefer the shorter makespan. A factory whose output is reviewed as it arrives, or whose specs can change while units wait, prefers short cycle times, because a unit that waited four hours was built against a pattern that may already be stale, and a reviewer facing 58 queued units reviews worse than one facing 6. The usual refinement is a limit slightly above the bottleneck's capacity, so it never starves, with the rest of the backlog held outside the line.

Retry caps per unit and per hourA per-unit cap alone does not stop a storm

A cap of three attempts per unit bounds the cost of one bad unit. It does not bound the cost of a fleet-wide failure: if every unit fails for the same external reason, the line spends three attempts on every unit in the backlog, and a rebuild that re-admits failed units multiplies that again. WatchAlgo's queue ended with 11,896 tasks, 11,503 of them failed, while no content was produced or lost.

Two more controls close the gap. A cap per hour, or a circuit breaker, that pauses the whole line when the failure rate jumps or when accepted output stays flat while attempts rise. And a rule that a rebuild re-admits failed units only after their failure class has been reviewed. The per-unit cap is a station's rule; the per-hour cap is the scheduler's, and it needs an owner who is alerted when it trips.

How many units per taskFewer process starts against longer tasks

WatchAlgo grouped 5 solution problems or 8 visualization problems into one worker task, and its March notes record trouble with 10 problems per agent and settle on 5. The trade-off is general. Larger tasks start fewer worker processes, share setup, and spend less of the concurrency limit on start-up. Smaller tasks keep each worker's context small, lose less work when a worker dies mid-task, and make a failure point at fewer units.

Verification per unit, not per task, removes most of the risk of larger tasks: WatchAlgo's driver checked every problem in a task from disk and credited the ones that passed, so a task that finished 4 of 5 lost only one. Without per-unit verification, the task becomes the unit of failure, and every retry repeats work already done.

Release order is a decision tooCategory order, risk order, value order

WatchAlgo released work in a fixed category order, so whole categories were finished one after another; that made a defect pocket, such as the 86 solution defects found in the Hashing and Prefix category, visible as a cluster. Other orders buy other things. Releasing the riskiest units first finds pattern defects while the backlog is still large enough to benefit from a fix. Releasing a random sample first estimates the failure rate of the whole batch before most of the budget is spent. Releasing by value first delivers the units people are waiting for.

This site's recommendation: start every large backlog with a small, deliberately mixed pilot batch, measure its first-pass yield and failure classes, fix the pattern, and only then release the rest in whatever order the product needs. The factory learning loop is the home of measuring one batch to improve the next.

In one breath#

A backlog is work not yet done, derived from what exists; a queue holds the tasks ready to claim, and deduplication decides what counts as the same work, by id and by content. A concurrency limit protects the slowest shared resource, which for WatchAlgo was an account that throttled at 22 sessions, and a WIP limit protects the time each unit spends in the line, because Little's law ties that time to how many units are inside. Throughput is set by the bottleneck, so workers added in front of it only lengthen its queue, and parallelism pays only for units that share nothing.

A line that runs for hours over thousands of units will be interrupted: a usage limit, a crash, a full context. The next chapter, Long-running agent execution, shows how a factory checkpoints before risky work, resumes without redoing finished units, and makes sure that a write repeated after a crash happens only once.