All chapters
  1. Start hereA software factory, from first principles
  2. Primitive · 1A factory, before AI: input, transformation, check, output
  3. Primitive · 2From repetition to a pattern: what stays, what varies, what to leave alone
  4. Primitive · 3The specification as an executable contract
  5. Primitive · 4Build the smallest factory: four stations, two specs, one manifest
  6. Primitive · 5Assembly stations: typed contracts, owners and failure handling
  7. Primitive · 6AI enters the factory: what a model adds, and what stays deterministic
  8. WatchAlgo · 7The WatchAlgo production story: what the repository shows
  9. WatchAlgo · 8A problem as a production unit: the files one unit must contain
  10. WatchAlgo · 9Station by station: description, solution, implementations, visualization
  11. WatchAlgo · 10The visualizer factory: from algorithm state to renderable steps
  12. WatchAlgo · 11Three flavors and three languages: two different contracts
  13. WatchAlgo · 12Looping over a large backlog: scheduling, limits and a failed-case queue
  14. WatchAlgo · 13Long-running agent execution: context, checkpoints and token limits
  15. WatchAlgo · 14Evidence of correctness: tests the producer cannot talk its way past
  16. WatchAlgo · 15The factory learning loop: defects change the pattern, not only the unit
  17. Enterprise · 16Classify the enterprise demand before choosing a pattern
  18. Enterprise · 17Reference enterprise patterns and their variation points
  19. Enterprise · 18Platform foundations: golden paths, templates, registries and catalogs
  20. Enterprise · 19Pattern versus configuration versus business logic
  21. Enterprise · 20Insurance factory, part 1: building the intake contract
  22. Enterprise · 21Insurance factory, part 2: building the underwriting workflow
  23. Enterprise · 22Insurance factory, part 3: trust and observability built in
  24. Enterprise · 23Data and integration engineering: contracts that survive change
  25. Enterprise · 24The CI/CD and deployment factory: from pull request to production
  26. Enterprise · 25Enterprise governance: owners, approved patterns and provenance
  27. Enterprise · 26Scaling safely: many teams, many agents, one set of boundaries
  28. Operations · 27Factory quality engineering: measuring accepted output, not activity
  29. Operations · 28Failure and recovery scenarios: what breaks a factory, and what restores it
  30. Operations · 29The landscape: which tool for which job, read from its own documentation
  31. Operations · 30Design an enterprise software factory: the blueprint and the rollout
  32. ReferenceSources and verification
  33. ReferenceGlossary
  34. AboutAbout this site
Enterprise · 26

Scaling safely: many teams, many agents, one set of boundaries

Repository layouts, team isolation, rate and cost quotas, tenant boundaries, concurrent agents, queues, locks on shared state, dependency scheduling, secrets and scoped permissions.

You will draw
Four boundaries a shared factory needs: repository, tenant, quota, credential, and what each one stops.
You will run
Example D's scheduler with 1, 2 and 4 workers: same record every time, and the graph, not the pool, sets the parallelism.
You will decide
One repository or many, with integration contracts, and a counterexample for each.

The insurance factory of Part III has had a good quarter. The submissions team built the broker intake service with it; now the claims team and the policy servicing team want the same thing for their own intake products, and the platform team says yes. Teaching reconstruction: the following week is this page's illustrative scenario, not a recorded incident.

On Tuesday night the claims team regenerates all four of its products after a pattern patch. The run is correct. It also takes every shared CI runner and most of the model quota on the platform account until morning. On Wednesday the submissions team has an urgent fix for a production defect and waits three hours for a runner. On Thursday a worker in the claims run, which held the factory's one broad credential, updates a shared schema file in the submissions team's repository because its task said "align the submission schema". The submissions pipeline catches the change. Nobody can say which claims run made it until someone reads the logs line by line.

No single run did anything wrong by its own rules. Every problem came from sharing: one queue of runners, one quota, one credential, one set of files, and no record of whose work was whose.

The learner's question: when many teams and many agents use one factory, which boundaries keep their work apart, which shared resources need explicit limits, and how small can each worker's permissions be?

What changes when a factory is shared#

Plain English. A factory that one team uses can trust everything inside it. A factory five teams use has to assume that any run can be wrong, greedy or compromised, and limit what each run can reach, spend and change.

Looping over a large backlog scheduled thousands of independent units through one factory for one owner, with a concurrency limit set by the slowest shared resource. This page takes the next step: several owners. The vocabulary of queues and WIP carries over unchanged. What is new is that limits now have to be fair between teams, and that a mistake in one team's run must not reach another team's products.

Before AI. Multi-tenant systems solved this long before factories. A shared Jenkins installation gave each team its own folder of jobs and its own executors. A software-as-a-service product kept each customer's rows behind a tenant id that every query had to carry. A cloud account gave every service its own role instead of one administrator key. A shared factory needs the same three ideas: separate places, separate limits and separate identities.

Where the code lives: one repository or many#

A monorepo keeps many products and shared libraries in one version-controlled repository, built with one build graph. A polyrepo layout gives each product (and often each shared library) its own repository, connected by published versions and integration contracts. The tiny example: the three teams' intake services and the shared PAT-INTAKE-API skeleton in one repository, or in four.

The consequence for a factory is about blast radius and conflicts. In a monorepo, one change can update the pattern and every product in a single atomic commit, and one build graph knows exactly which products to rebuild; but every factory run writes into the same repository, so parallel runs compete for the same branch and the same files. In a polyrepo layout, each run writes only to its own product's repository, so runs cannot collide, but a pattern change reaches each product as a separate upgrade, and products drift onto different versions. Platform foundations showed how a catalog keeps that drift visible.

This page owns one of the site's eleven decisions.

The question. Should the factory's products live in one repository or in many, and what holds them together?

OptionChoose it whenIt gives youFailure mode
Monorepoone organization owns the products and the pattern; changes often span both; the build tool understands the graphatomic cross-product changes; one dependency version; one place to searchparallel runs conflict on shared files; a broken pattern breaks every build at once
Polyrepo with integration contractsteams release independently; products have different owners, cadences or access rulessmall blast radius per run; per-repository permissions; no merge contention between teamsversion drift; a breaking change needs many coordinated upgrades
Hybrid: shared pattern repository, product repositoriesthe pattern is a platform product with its own release, the products belong to domain teamspattern changes reviewed once, adopted per product at its own paceproducts pinned to old pattern lines, which Factory learning loop calls deferred blast radius

The criteria, in order: who owns the products (one team or many); whether they release together or apart; whether access must differ between them (a product handling regulated data may need its own repository rules); and whether the build tooling can compute what a change affects.

The counterexample. Example C's factory could put the claims team's intake service into the submissions team's repository, since both come from the same pattern. That shared home is exactly what let Thursday's worker touch the other team's schema file. The opposite counterexample: putting the shared skeleton, its tests and the products in separate repositories with no integration contract means a skeleton change is "released" with nothing that tells the products it broke them.

The failure modes. A monorepo without a lock or merge queue on shared files turns parallel factory runs into merge conflicts. A polyrepo layout without contracts turns every shared change into a silent break. Either layout without a catalog loses track of which product is on which pattern version.

Isolation between teams and tenants#

A tenant boundary is the line that keeps one tenant's data, runs, artifacts and credentials apart from another's, enforced by the system rather than by good behavior. Here a tenant is a team using the factory, or a business unit whose products must not mix. The tiny example: claims runs read and write only claims repositories, claims artifacts in a claims registry namespace, and claims records in the run store under a claims tenant id. The consequence is that a mistake or a compromise in one team's run stays in that team's products.

A quota is a budget per tenant on a shared resource over a period: model usage per day, CI runner minutes per hour, factory runs in flight. A rate limit caps how fast requests may arrive, for example calls per minute to a model endpoint. They protect different things. The quota protects fairness between teams over a day; the rate limit protects the resource from bursts in a minute. Both are set below the shared ceiling, which the backlog chapter taught to name explicitly: WatchAlgo's recorded ceiling was about 8 concurrent sessions on one account, and 22 at once triggered account-wide throttling.

The scenario's Tuesday night is a missing quota. The claims run was entitled to the runners by the queue's rules (first come, first served) and took all of them. This site's recommendation: give each tenant a guaranteed share and a maximum share of every shared pool, let an idle tenant's share be borrowed, and let an urgent production fix jump the queue only through a recorded priority class, never by someone killing another team's run.

Concurrent agents and shared state#

When two workers run at the same time, the question from the backlog chapter returns: do their write sets overlap? Independent units, such as WatchAlgo's one folder per problem, need no coordination beyond a queue. Shared state needs a rule.

A shared-state lock is a mechanism that lets only one worker change a given piece of shared state at a time: a file lock on one machine, a row lock or lease in a database, a branch protection rule or merge queue in a repository. WatchAlgo's queue file was protected by an exclusive file lock (flock), shown verbatim in the backlog chapter. The consequence of a lock is safety; its cost is waiting, and a lock held by a crashed worker blocks everyone until something releases it.

The better design, where possible, is to need no lock at all: give each task a disjoint write set. A shared schema file that two teams' runs might both change is a sign that the file has two owners. Either it becomes one team's artifact, published with a version that the other consumes through a contract, or every change to it goes through one serialized station with its own owner. Simulated, illustrative: the batch lab's contention preset shows what happens without either: doubling the solution workers onto shared files raises their rework from 20 to 40 percent, sends 5 of 60 units to the failed-case queue, and lowers throughput from 9.045 to 8.871 accepted units per hour.

The smallest primitive: a scheduler that knows what is independent#

Before any team-level boundary, a factory needs to know which of its own tasks may run at once. Dependency scheduling answers that from the station graph alone. Example D (examples/d-supervisor/) plans its station graph in levels with Kahn's algorithm. Quoted verbatim from dag.py : topo_levels:

def topo_levels(tasks) -> list:
    """Kahn's algorithm by levels. Each level is sorted by station id, so the plan is deterministic."""
    by_station = {t.station_id: t for t in tasks}
    for t in tasks:
        for d in t.deps:
            if d not in by_station:
                raise CycleError(f"{t.station_id} depends on unknown station {d}")
    remaining = {t.station_id: set(t.deps) for t in tasks}
    levels = []
    done = set()
    while remaining:
        ready = sorted(s for s, deps in remaining.items() if deps <= done)
        if not ready:
            raise CycleError("cycle among: " + ", ".join(sorted(remaining)))
        levels.append([by_station[s] for s in ready])
        done.update(ready)
        for s in ready:
            del remaining[s]
    return levels

A station is ready when everything it depends on is done. Every ready station forms one level, and stations in one level have no path between them, so they may run at the same time. A graph with an unknown dependency or a cycle is refused before anything runs, instead of hanging halfway.

The supervisor then runs each level on a bounded pool and keeps every write to the record on one thread. From scheduler.py : Supervisor.run_round (an excerpt, verbatim, with the method's indentation removed):

self.conn.commit()  # 'running' rows are durable BEFORE any work starts

with ThreadPoolExecutor(max_workers=self.workers) as pool:
    futures = {t.station_id: pool.submit(self.work, spec, t, ih, inputs, failed)
               for (t, ih, inputs, attempt, failed) in to_run}
    results = {sid: f.result() for sid, f in futures.items()}

max_workers is the bound: no level runs more stations at once than the pool allows, however wide it is. The worker threads only compute and write content-addressed files; the main thread records results afterwards, in sorted station order. That separation is a scaling rule in miniature: parallel workers produce, one owner records. Long-running agent execution is the home of the record and of resume after a crash; this page uses the same scheduler for its parallel half.

Diagram: Example D plans the station graph in levels; the widest level sets the parallelism

Four levels, left to right. Level 1 validates the spec. Level 2 runs the solution and the oracle at the same time because neither reads the other. Level 3 runs the visualization and the tests. Level 4 packages the manifest. With a pool of 4 workers, at most 2 stations ever run at once. In FR-0007 the visualization fails three times and is escalated to a person; the manifest is blocked, the tests still finish, and the run ends incomplete.

  1. Level 1
    1. S1-validate-spec: Checks the spec before anything else runs.
  2. Level 2: in parallel
    1. S2-generate-solution: Reads the spec; does not read the oracle.
    2. S2-generate-oracle: Reads the spec; does not read the solution.
  3. Level 3: in parallel
    1. S3-build-visualization: Needs the solution. Fails three times with invariant_violation and goes to a person.
    2. S3-run-tests: Needs solution and oracle; finishes even though its neighbor failed.
  4. Level 4
    1. S4-package-manifest: Needs both level-3 stations; blocked, so the run ends incomplete, never succeeded.

The diagram shows the four levels of the pattern PAT-ALGO-UNIT 1.3.0. Level 1 validates the spec. Level 2 runs the solution and the oracle side by side, since neither reads the other. Level 3 runs the visualization (which needs the solution) and the tests (which need both). Level 4 packages the manifest. The widest level has two stations, so a third worker would sit idle. When the visualization station fails three times, it is escalated to a person and the manifest is blocked; the tests station, which does not depend on it, still finishes.

The executable demonstration: one record, whatever the worker count#

What this demonstrates: the scheduler runs independent stations in parallel up to its bound, keeps dependent ones in order, ends a run with a stuck station as incomplete with an escalation, and writes the same record whether it has one worker or four. What it does not demonstrate: speed. The stations are scripted stand-ins; no timing here is a benchmark.

A script written for this page (scaling_demo.py) ran Example D's escalation case, FR-0007 on SPEC-ALGO-003, three times in a scratch copy on 2026-11-20, with 1, 2 and 4 workers. It wraps the scripted stand-in station with a counter of how many stations are inside it at once, and compares the canonical dump of the record across the three runs:

plan (levels of PAT-ALGO-UNIT 1.3.0):
  level 1: S1-validate-spec
  level 2: S2-generate-oracle, S2-generate-solution
  level 3: S3-build-visualization, S3-run-tests
  level 4: S4-package-manifest
workers=1: run incomplete; peak stations running at once: 1
workers=2: run incomplete; peak stations running at once: 2
workers=4: run incomplete; peak stations running at once: 2
tasks: S1-validate-spec=succeeded, S2-generate-solution=succeeded, S2-generate-oracle=succeeded, S3-build-visualization=escalated, S3-run-tests=succeeded, S4-package-manifest=blocked
record identical for 1, 2 and 4 workers: True

Three lines carry the lesson. With 4 workers the peak is still 2: the graph's widest level, not the pool, sets the parallelism, so buying more workers for this pattern buys nothing. The record is identical for 1, 2 and 4 workers, so a team can change its worker bound (to fit a quota) without changing what the run means. And the run ends incomplete, with one station escalated and its dependent blocked, while the independent tests station succeeded.

The supervisor's own event log for the same run, from run.py --verbose in the scratch copy, shows the bounded failure step by step:

FR-0007 started  spec=SPEC-ALGO-003 hash=ce4f9c120e37 pattern=PAT-ALGO-UNIT 1.3.0
FR-0007 S1-validate-spec         attempt 1 succeeded artifact://FR-0007/S1-validate-spec/367a9daba2a1 (wrote)
FR-0007 S2-generate-oracle       attempt 1 succeeded artifact://FR-0007/S2-generate-oracle/d74b7cef763b (wrote)
FR-0007 S2-generate-solution     attempt 1 succeeded artifact://FR-0007/S2-generate-solution/47cabbdf3454 (wrote)
FR-0007 S3-build-visualization   attempt 1 failed invariant_violation; retry (1/3 failures)
FR-0007 S3-run-tests             attempt 1 succeeded artifact://FR-0007/S3-run-tests/ae596d71be22 (wrote)
FR-0007 S3-build-visualization   attempt 2 failed invariant_violation; retry (2/3 failures)
FR-0007 S3-build-visualization   attempt 3 failed invariant_violation; 3/3 failures: ESCALATED to a person
FR-0007 S4-package-manifest      blocked by S3-build-visualization
FR-0007 finished status=incomplete cost_units=145

At scale this behavior is what keeps one stuck product from consuming a shared pool: three attempts, then a person, and the run's cost (145 illustrative units, failed attempts included) is on the record for its tenant. The agent here is a scripted stand-in, so these results show how the factory mechanics work, not how good any model is.

Credentials, sandboxes and budgets for every worker#

Thursday's problem was not the worker's task. It was what the worker could reach. This site's safety rules say it plainly: every automated worker runs with minimal scoped credentials, an isolated filesystem and network where feasible, a command allowlist or approval, resource, time and cost budgets, and no secrets in prompts or traces.

A scoped credential is a credential that grants only what one task needs, for one target, for a short time: write to one repository for the length of one run, not to every repository forever. A sandbox is an execution environment that limits what a worker process can touch: which files it can write, which network addresses it can reach, which commands it can run. Together they turn "this worker should only change the claims repository" from an instruction the worker may ignore into a boundary it cannot cross.

ControlWhat it limitsIn the scenarioWho owns it
Scoped credential per runwhich repositories, registries and services the worker can writethe claims run cannot write the submissions repository at allplatform team issues; security reviews
Filesystem sandboxwhich paths the worker process can modifythe worker sees one checkout, its ownplatform team
Network allowlistwhich hosts the worker can reachthe model endpoint and the package mirror, nothing elseplatform and security
Command allowlist or approvalwhich commands run without a persontests and builds yes; pushing tags or deleting branches noplatform team, per station
Budgetstime, cost and attempts per run and per tenantthe claims batch stops at its quota, not at the account's ceilingplatform sets, domain team spends
No secrets in prompts or traceswhat a model or a log can leakcredentials come from the runtime, never from the spec or the tracesecurity

Several of these exist as product features, and the documentation is specific. Product (Codex, as documented on 2026-11-20): sandbox modes are read-only, workspace-write and danger-full-access; codex exec defaults to a read-only sandbox; enforcement uses Seatbelt on macOS and bubblewrap on Linux; for CI the documentation says to set CODEX_API_KEY only on the invocation that needs it. Product (Claude Code, as documented on 2026-11-20): --allowedTools auto-approves tools using permission rule syntax such as Bash(git diff *); without --bare, a -p session runs a project's hook settings and .mcp.json servers even in an untrusted folder, with no trust dialog. Product (GitHub Actions, as documented on 2026-11-20): a workflow can obtain a short-lived cloud token through OIDC, with the id-token: write permission, instead of storing long-lived cloud credentials as secrets. Product (GitLab CI/CD, as documented on 2026-11-20): id_tokens creates OIDC tokens for a job, with a required audience, that expire at the job timeout (5 minutes if none is set).

Security note. The Claude Code behavior above matters for a factory: a station that runs a coding agent headless inside a checkout it did not author will execute that repository's configured hooks unless it uses --bare or an equivalent control. A repository is input to a station, and input is untrusted. The station runner, not the repository, must decide what runs.

Diagram: One worker's boundaries: what it can reach, spend and change

Read each lane as one boundary around the same worker. The sandbox exposes one checkout. The credential is scoped to one tenant's repository for one run and expires with it. The network reaches the model endpoint and the package mirror only. The budget counts against the tenant's quota. The only exit is an artifact handed to the factory's gates and recorded under tenant and run id. The worker cannot reach another team's repository, the signing identity or the shared record.

  1. What it can reach
    1. sandbox: one checkout: The worker process can modify only its own checkout.
    2. credential per run: Issued at start through OIDC federation; grants write to one repository.
    3. two hosts, allowlisted commands: Model endpoint and package mirror; tests and builds run, tag pushes need a person.
  2. What it can spend
    1. tenant quota: Model usage and runner minutes count against the team's share, below the account ceiling.
    2. rate limit: Caps bursts to the shared model endpoint.
  3. What leaves
    1. artifact to gates: The only way out: an artifact recorded under its tenant, checked by the factory's gates.

The diagram shows one worker's boundaries from the inside out. The worker process sits in a sandbox that exposes one checkout. It holds a credential scoped to one repository and one run, issued at start and expiring at the end. Its network reaches two hosts. Its budget counts against its tenant's quota. Everything it produces leaves through one exit: an artifact handed to the factory's gates, recorded under its tenant and run id. Nothing in the worker can reach the signing identity of Enterprise governance, another team's repository or the shared record directly.

What this is not. This page owns boundaries between teams, tenants and workers of a shared factory: repository layout, quotas, locks, scoped credentials, sandboxes. The control plane that schedules many agents in general is Agent Orchestration Engineering. The controls around one coding agent's run (permissions, hooks, context) are Harness Engineering. Identity, delegation and token exchange are Auth Engineering. Retries and recovery as a discipline are Agent Reliability Engineering.
In a coding agent. A coding agent such as Claude Code or Codex, running headless as a station worker, is one slot of a tenant's quota and one holder of a scoped credential. It receives one work item and a checkout in a sandbox, with a command allowlist for that station; two agents never hold work items that write the same files; when it exits, the factory's own tests decide whether the work advances, and a person with merge rights reviews. The agent's instructions saying "only touch the claims repository" are not the boundary; the credential and the sandbox are.

Try it#

The batch lab opens on the contention preset: the solution station doubled to 8 workers that now touch shared files, with rework at 40 percent. Lower the workers back to 4 and the failure percentage to 20 to see the line without contention; then raise workers again and watch the failed-case queue fill.

The same results for a reader without JavaScript (from examples/sim/batch_sim.py via docs/LABS.md; simulated, illustrative; 60 units; stations spec 4 min 2 workers 5%, solution 12 min 4 workers 20%, oracle 6 min 2 workers 5%, visualization 15 min 3 workers 15%, review 5 min 1 worker 10%; at most 3 attempts per station):

PresetWhat changesMakespan (min)Accepted · escalatedThroughput per hourLead time p50 · cycle time p50 (min)Bottleneck
defaultnothing39860 · 09.045223 · 223visualization (88% busy)
contentionsolution workers 8, rework 40% (shared-file conflicts)37255 · 58.871202 · 202visualization

Reading: the contention run finishes sooner (372 against 398 minutes) while accepting five fewer units, and per hour it accepts fewer. More parallel workers on shared state produced less accepted work and a failed-case queue that a person must clear.

WatchAlgo in practice#

Observed in the repository (docs/EVIDENCE.md rows 11 and 14, section 5): WatchAlgo ran one factory for one owner on one machine. Its shared state was protected by a file-locked queue (queue.json under fcntl.flock, written atomically, in scripts/viz/pipeline.py, 971863f2), and its units were independent folders, so workers never wrote the same file. Its concurrency was bounded by the account: the recorded ceiling of about 8 concurrent sessions, after 22 sessions triggered account-wide throttling (SESSION_STATE.md, 0309048f), and a supervisor default of 8 workers. These are the author's and the agent's recorded observations, not measurements; the backlog chapter tells that night in full.

Observed (section 5): the workers ran as claude -p --model sonnet --dangerously-skip-permissions, with a 60-minute timeout per task. There was no per-worker credential scoping, sandbox or command allowlist beyond what the machine and the account provided; the boundary was the task and the deterministic acceptance check that ran after the worker exited. Observed (row 13): usage limits were account-wide (weekly, monthly spend, session), so one limit stopped every worker at once.

Reported by the author: a repeatable process ran for 30+ hours until token limits required intervention; The WatchAlgo story reconciles that with the logs. Teaching reconstruction: for one author on one machine, a file lock and a worker count were the least machinery that worked, and the single account was the right ceiling to schedule against. Every boundary on this page appears only when a second owner arrives: a second team would have needed its own quota inside the account's ceiling, and a skip-permissions flag would no longer be acceptable on a machine whose checkouts belong to more than one team.

Enterprise translation#

At the insurer, the platform team runs the shared factory as a multi-tenant service. Each domain team is a tenant with its own repositories, registry namespace, run records and quota. Factory runs get credentials issued per run through the CI system's OIDC federation, scoped to the tenant's repositories and expiring with the job. Station workers run in sandboxes with per-station command allowlists; the model endpoint and the package mirror are the only network destinations. The shared pattern lives in its own repository, released by the platform team and adopted per product.

What dominates the critical path when scaling is not adding workers. It is the security review of the factory's identity model (one identity per tenant, or per run), agreeing quotas between teams that have never shared a budget before, and untangling files that two teams both believe they own. The fastest scaling step is often the least technical: assign every shared file one owner.

Failure test#

The failure: two factory runs from different teams are scheduled at the same time with tasks that both write the shared schema file, and the factory holds one broad credential.

How the factory catches it: in layers, and the lesson is that the earliest layer should win. A per-run credential scoped to the tenant's repositories refuses the second team's write outright. Without it, a lock or merge queue on the shared file serializes the two writes, and the product pipeline's contract and schema gates catch an incompatible result. Without those, the conflict lands as rework in the line, which the contention preset shows as a failed-case queue. And whatever happens, a station that keeps failing stops after three attempts and is escalated, as FR-0007 shows, instead of retrying into the shared quota.

What to run or toggle: in the lab, start from contention and compare with default. In a scratch copy of Example D, run /opt/homebrew/bin/python3 run.py --verbose and read the FR-0007 lines; then run /opt/homebrew/bin/python3 -m unittest tests.test_scheduler to see the tests that prove parallel stations really meet and that one worker cannot.

Where to go deeper#

Checklist for Monday#

Exercises#

  1. A pattern's widest level has three stations. A team asks for eight workers per run. What do the extra five buy?
Answer 1Nothing within one run

Within one run, at most three stations are ever ready at once, as Example D's peak of 2 with 4 workers showed for its two-wide graph. Extra workers help only across several runs at once, and then the tenant's quota, not the pool, should decide.

  1. Why is "only modify files under claims/" in a worker's instructions not a tenant boundary?
Answer 2Instructions are not enforcement

A worker can misread, be misled by injected text, or make a plain mistake. A boundary is enforced by the system: a credential that cannot write elsewhere, or a sandbox that does not expose other paths. Ownership and authorization must be enforced independently of agent instructions.

  1. Monorepo, three teams, nightly factory runs that each regenerate their products. What do you add first?
Answer 3Disjoint write sets, then a merge queue

First make sure each run writes only its own products' paths (owners per path, a credential or check that enforces it). Then serialize merges of shared paths through a queue with gates, so two runs never land conflicting changes.

Advanced perspective#

Locks that outlive their holder: leases and fencingA crashed worker must not block everyone, or write after it lost the lock

A file lock taken with flock is released by the operating system when the process exits, which is why WatchAlgo's single-machine queue never stayed locked after a worker died. Across machines there is no such help. A distributed lock is usually a lease: a lock with an expiry that the holder must renew. If the holder crashes, the lease expires and another worker can proceed.

Leases bring their own failure: a worker that paused (a long garbage collection, a suspended laptop, a slow network) may still believe it holds the lock after it expired and someone else took it. If it then writes, two holders have written. The standard defence is a fencing token: every grant of the lock carries a number that only increases, and the shared store refuses a write that carries a smaller number than the last one it accepted. Example D avoids the whole problem by design: one main thread writes the record, workers only write content-addressed files that are safe to write twice. This site's recommendation: prefer designs where shared writes have a single owner or are idempotent, and use leases with fencing only where neither is possible. Agent Reliability Engineering treats coordination failures as a discipline.

Fair-share quotas and priority classesGuaranteed share, borrowable idle capacity, and an audited fast lane

A first-come queue is fair to tasks and unfair to teams: the team that submits the biggest batch first takes the pool. Fair-share scheduling gives each tenant a weight; when the pool is contended, each gets its weighted share; when a tenant is idle, others may borrow its share and give it back when it returns. A maximum share stops one tenant from borrowing everything.

Priority needs a separate mechanism, because "urgent" is the word every team will use. A small number of priority classes (production fix, scheduled release, bulk regeneration), each with a quota of its own and a record of who used it, keeps the fast lane honest. CI products offer building blocks. Product (GitHub Actions, as documented on 2026-11-20): concurrency groups with cancel-in-progress, and a concurrency.queue setting where max allows up to 100 pending runs per group; strategy.max-parallel caps parallel matrix jobs. Product (GitLab CI/CD, as documented on 2026-11-20): parallel accepts 1 to 200 and parallel:matrix at most 200 permutations. These limit parallelism per pipeline or group; a tenant-level fair share across many pipelines is something the factory or the platform has to add.

Model quotas behave like the account ceiling WatchAlgo recorded: shared, invisible until hit, and hit by everyone at once. A factory that splits one account among tenants must enforce the split itself, before requests leave, because the provider sees only one account.

No secrets in prompts or tracesCredentials from the runtime; masking at the source; scans as a gate

A model's context and an agent's trace are copied far more than code: into logs, telemetry, evaluation datasets, bug reports and other agents' context. A credential that appears once in a prompt can appear in a dozen stores. The rule therefore applies at the source: no credential in a spec, a template, a generated file, a prompt or a trace.

Three practices make it enforceable. Credentials are injected by the runtime into the process environment of the step that needs them, scoped to it (Codex's documentation describes setting CODEX_API_KEY only on the invocation that needs it). Logs mask sensitive fields where they are written, as Example C's logger masks broker_email. And a secret scan runs as a gate on every artifact, as Example C's forbidden-secrets gate scans every file for credential shapes and allowlists only tokens with the DEMO-NOT-A-SECRET- prefix. Observability Engineering owns telemetry design; Insurance factory, part 3 owns masking inside the product.

What isolation costs, and when it is worth itPer-run identities, sandboxes and namespaces are work; shared state is debt

Every boundary on this page has a price. Per-run credentials need an identity federation and a policy per tenant. Sandboxes need images, allowlists and someone who answers when a legitimate command is blocked. Separate registry namespaces and run stores need provisioning per tenant. A single team on a single machine pays all of that for nothing, which is why WatchAlgo's file lock and worker count were the right amount of machinery for it.

The price is worth paying at a specific moment: when a second owner shares any resource whose misuse is not reversible by that owner alone. Shared files, shared credentials and shared quotas are debts that come due as incidents between teams, and those incidents are expensive in the way Factory quality engineering measures: rework, waiting and escaped defects, not compute. A useful order is credentials first (they bound the worst case), quotas second (they bound the common case), sandboxes third, and repository restructuring last, because it is the slowest to change and the other three reduce its urgency.

In one breath#

A shared factory needs four boundaries that a single-owner factory does not: where each team's code lives, which tenant each run belongs to, how much of each shared pool each tenant may use, and what each worker can reach. Inside one run, the station graph decides what may run in parallel, a bounded pool caps it, one owner writes the record, and a station that keeps failing is escalated instead of consuming the pool. Across runs, scoped per-run credentials and sandboxes, not instructions, keep one team's worker out of another team's files.

The factory can now serve several teams without one team's night run becoming another team's morning incident. Whether all this machinery makes the line produce more accepted work, or only more activity, is a question of measurement. Factory quality engineering counts accepted units, cost per accepted unit and the case where adding agents makes the line slower and dearer.