A developer on the submissions team of the synthetic insurer has a bad week. The roof-data provider times out often, and the broker intake service gives up after three attempts and answers 503 provider_unavailable. The fix looks like one line. The developer opens the service's code, finds the retry limit, changes 3 to 5, and pushes the branch.
The pull request fails. Not because someone reviewed it, but because the service's own unit tests say the retry limit is 3 and the code now says 5. The developer is annoyed: the change was deliberate. Was the pipeline wrong?
It was not. The retry limit is product behavior, written in the product specification (max_attempts: 3 in SPEC-INTAKE-001), and the code is an artifact the factory generated from that spec. A change in the generated code that the spec does not ask for is drift, and the next factory run would overwrite it anyway. The right change goes somewhere else: into the spec, through the factory, and only then through the service's own pipeline. Two pipelines were involved, and the developer knew only one of them.
The learner's question: which checks run where, who owns each pipeline, what has to be true before a build moves from one environment to the next, and what record proves it afterwards?
Two pipelines, two different products#
Plain English. A pipeline is a fixed sequence of automatic checks that every change passes on its way somewhere. The factory has one for itself, because the factory is software too. Every product the factory builds has its own.
A CI pipeline (continuous integration pipeline) is the sequence of builds and checks that runs automatically on every change to a repository and reports pass or fail. A tiny example: on every pull request to the intake service, run its 50 tests, check its API contract and scan it for secrets. The operational consequence is that "the build is green" becomes a statement anyone can check, not an opinion. The trade-off is that a pipeline is only as strong as its weakest check, and a green result says nothing about what no check looked at.
The factory pipeline is the pipeline of the factory itself. Its input is a change to the factory: a new pattern version, a changed station, a new product spec. Its output is generated products, each checked against its own gates. The product pipeline (often called the application pipeline) ships one product: it builds that product's code, runs its tests, and promotes it through environments to production. The factory pipeline answers "does the factory still make correct products?". The product pipeline answers "is this build of this product safe to release?".
Before AI. A team that maintained a Maven archetype for its company had two kinds of builds. One built and released the archetype itself, usually by generating a sample project from it and building that. The other was the ordinary Jenkins pipeline of each service someone had generated. A broken archetype release could break every new service at once; a broken service build broke one service. The split here is the same, with a factory in place of the archetype.
The opening story crossed the line between them. The developer changed an artifact inside the product pipeline when the decision belonged to the factory pipeline: change the spec, let a factory run regenerate the service, and let both pipelines check the result. One change's path from a branch through review and CI to production is the subject of Diff to Deploy; this page is about what a factory adds around that path when it builds many products from one pattern.
Diagram: The factory pipeline builds and verifies products; the product pipeline ships one
Two lanes. The factory pipeline starts from a change to the pattern or a product spec, runs the factory, runs the factory's own tests, and rebuilds golden products and checks them. Its output is a generated, hashed artifact set. The product pipeline takes that artifact set, runs the pull request gates, promotes one digest through environments, runs a canary and reaches production. If the product needs other behavior, the change goes back to the spec at the start of the factory lane; nothing in the product lane edits a generated file.
- Factory pipeline
- pattern or spec change: A new pattern version or a changed product spec: the only legitimate way to change generated behavior.
- factory run: The stations assemble contracts and service configuration from pattern plus spec.
- factory tests + golden products: The factory's own tests, plus a rebuild of fixed sample products compared by digest and run through their gates.
- hashed artifacts: Generated files with their sha256 in a manifest: the only thing handed to the product lane.
- Product pipeline
- pull request: A change waiting to merge into the product's main branch.
- PR gates: The fast tier: every check returns an exit code and writes a record.
- promote one digest: The same build moves through environments; it is never rebuilt on the way.
- canary, then production: A small share of traffic for a fixed window; above the threshold, roll back to the previous digest.
Read the two lanes from left to right. The top lane is the factory pipeline: a change to the pattern or a spec, a factory run, the factory's own tests, then sample and real products built and checked. Its last step hands each generated product to the bottom lane. The bottom lane is one product's pipeline: pull request gates, promotion through environments, canary, production. The only arrow between the lanes is a generated, hashed artifact. Nothing in the bottom lane edits what the top lane produced; if the product needs a different behavior, the change goes back to the spec at the start of the top lane.
What the product pipeline must prove, tier by tier#
A test tier is a group of checks with the same cost and the same question: fast checks of one unit, slower checks of a contract between services, slower still checks with real dependencies or real traffic. A pipeline runs cheap tiers first, so most failures cost seconds, and runs the expensive tiers only on builds that passed the cheap ones.
A PR gate (pull request gate) is the set of checks that must pass before a change may merge into the main branch. It is a quality gate in the sense of the index: an independently checked condition, not a test the author of the change wrote for themselves and ran once.
Example C (examples/c-insurance-service/) gives the intake service four gates, run by ci_checks.py. The gates lab adds three later stages that a real release would have. The table puts them in order, with the tier each belongs to and where it would run:
| Stage | Tier | What it proves | In Example C |
|---|---|---|---|
| unit tests | fast, every change | the service's behavior: 401, 403, 422, retries, idempotency, workflow | real: 50 tests |
| contract check | fast, every change | the generated contract matches the spec; the decision route is underwriter-only | real |
| schema compatibility | fast, every change | every submission schema version reads the one before | real: v1 to v2 |
| secret scan | fast, every change | no credential-shaped string in any file | real: demo tokens allowlisted |
| evals | slower, before release | the recommender gives the expected recommendation on golden cases | real in the lab: 4 cases |
| human approval | before production | a person with the release-approver role signs off | simulated in the lab |
| canary | in production, small share | the error rate stays under the threshold for the canary window | simulated in the lab |
The first four are PR gates. The evals belong to Evaluation Engineering, which owns how golden cases and graders are designed; the factory's job is to invoke them as a gate and record the result. The last two are release stages, explained further down.
The smallest primitive: a gate is an exit code and a record#
A gate needs two things to be a gate and not a printout: a verdict a machine can act on, and a record a person can read later. ci_checks.py has both. This is its main, quoted verbatim from examples/c-insurance-service/ci_checks.py:
def main(argv=None) -> int:
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("--skip-unit-tests", action="store_true")
ap.add_argument("--verbose", action="store_true")
args = ap.parse_args(argv)
rows, ok = run_gates(args.skip_unit_tests, args.verbose)
print(f"{'gate':<18} {'status':<6} detail")
for g, s, d in rows:
print(f"{g:<18} {s:<6} {d}")
print("ALL GATES PASS" if ok else "GATES FAILED")
(OUT / "gate_report.json").write_text(json.dumps({"gates": [{"gate": g, "status": s, "detail": d}
for g, s, d in rows], "pass": ok},
sort_keys=True, indent=2) + "\n")
return 0 if ok else 1
The return value is the verdict: exit code 0 only when every gate passes. Any CI system reads that and stops the pipeline. The gate_report.json file is the record: one row per gate with its status and detail. One design choice deserves notice: run_gates runs every gate even after one fails, so a single run reports all problems at once. A release pipeline usually does the opposite and stops at the first failing stage, which is what the gates lab shows. Both are reasonable; reporting everything suits pull requests, stopping early suits release stages that cost real resources.
The executable demonstration: green, then a planted regression#
What this demonstrates: the same four gates on two builds of the intake service, one clean and one with the opening story's change. What it does not demonstrate: that the gates are complete. A gate catches what it checks.
Run in a scratch copy of Example C on 2026-11-20, first as it is:
$ /opt/homebrew/bin/python3 ci_checks.py
gate status detail
unit-tests PASS 50 tests, 0 failures
contract-check PASS 5 routes, 6 event types, 0 problems
schema-compat PASS submission v1 -> v2: 0 breaking changes
forbidden-secrets PASS 34 files scanned, 0 findings, 10 demo-token mentions allowlisted
ALL GATES PASS
exit=0
Then with one file added to the scratch copy's tests/, standing in for the developer's change: a test that holds the retry limit after the change (5) against the spec's 3. This file exists only in the scratch copy; the example itself is unchanged.
import unittest
class PlantedRegression(unittest.TestCase):
def test_retry_limit_is_three(self):
retry_limit_after_the_change = 5 # someone raised it in code, not in the spec
self.assertEqual(retry_limit_after_the_change, 3)
$ /opt/homebrew/bin/python3 ci_checks.py
gate status detail
unit-tests FAIL 51 tests, FAILED (failures=1)
contract-check PASS 5 routes, 6 event types, 0 problems
schema-compat PASS submission v1 -> v2: 0 breaking changes
forbidden-secrets PASS 35 files scanned, 0 findings, 10 demo-token mentions allowlisted
GATES FAILED
exit=1
Three things to notice. The exit code changed from 0 to 1, which is all a CI system needs to block the merge. The other three gates still ran and passed, so the record says exactly which property broke. And the file count of the secret scan moved from 34 to 35, because the scan covers every file, including the new test. After the planted file was removed, the gates passed again with exit code 0.
The second half of the demonstration is the record. A script written for this page (cicd_demo.py, run on the same scratch copy) joins the gate result with the factory manifest of FR-0003 and the rollout manifest, which is what a reviewer needs in one place:
{
"factory_run_id": "FR-0003",
"pattern": "PAT-INTAKE-API 2.1.0",
"specification_hash": "sha256:ecb151ab7f7f...",
"openapi_digest": "sha256:d79646305fc5...",
"gates": {
"unit-tests": "PASS",
"contract-check": "PASS",
"schema-compat": "PASS",
"forbidden-secrets": "PASS"
},
"all_gates_pass": true,
"promotion_requires": [
"all ci_checks gates PASS",
"a person with the release-approver role signs off"
]
}
canary error rate 0.4% at 5% of traffic for 60 min -> promote
canary error rate 3.4% at 5% of traffic for 60 min -> roll back to the previous version
The record ties a gate verdict to an exact build: the factory run, the pattern version, the hash of the spec and the digest of the generated contract. The last two lines apply the first rollback condition of the rollout manifest (error rate above 2 percent) to two canary error rates. Simulated, illustrative: the rates 0.4 and 3.4 percent are the gates lab's presets, not measurements, and the rule is this page's teaching reconstruction of a sentence in the manifest. The agent here is a scripted stand-in, so these results show how the factory mechanics work, not how good any model is.
Promotion, infrastructure as code, canary and rollback#
Passing the PR gates makes a build safe to merge. It does not make it safe to run in production. Between the two sit environments, and the rules for moving between them.
An environment promotion moves one exact build, identified by its digest, from one environment to the next (development, staging, production) after the checks for that step pass. The rule that makes promotion meaningful: the build is never rebuilt on the way. If staging tested digest d796… and production runs a fresh build of the same commit, staging tested something else. Example C's rollout manifest names the three environments and the conditions for promotion:
{
"artifact": "broker-submission-intake@1.0.0",
"canary_duration_minutes": 60,
"canary_percentage": 5,
"documentation_only": true,
"environments": ["dev", "staging", "production"],
"note": "Documentation only. Nothing in this example deploys anything.",
"promotion_requires": [
"all ci_checks gates PASS",
"a person with the release-approver role signs off"
],
"rollback_conditions": [
"error rate on any route above 2 percent for 10 minutes (illustrative threshold)",
"any 5xx on record_decision",
"any log line containing an unmasked email address"
]
}
(The environments array is reflowed onto one line; the values are as printed by run.py.) The factory wrote this file from the spec's rollout section in FR-0003. It is documentation only: nothing in the example deploys. A real factory hands the same facts to the delivery platform, which enforces them.
Infrastructure as code is the practice of describing environments (networks, databases, queues, permissions, the service's runtime settings) in versioned files that a tool applies, instead of configuring them by hand. Pattern: the environment definition goes through review and a pipeline like any other code, so "what does staging look like" has one answer, in the repository, with history. For a factory the consequence is direct: the pattern can ship infrastructure definitions with its templates, and every generated product starts from a reviewed environment. Example C has none; its environment names are labels in a manifest. Platform foundations separates environment names, which are promotion stages, from environment values, which differ per deployment.
A canary release sends a small share of real traffic to the new build for a fixed window while the old build serves the rest, and compares error rates before promoting further. Example C's manifest says 5 percent for 60 minutes. A rollback returns production to the previous known-good build. It is fast and safe only if two things hold: the previous build still exists, unchanged, by digest; and nothing the new build did is irreversible. A schema migration that dropped a column, a message sent to a broker, a recommendation written to a partner system: none of these is undone by switching builds back. That is why Data and integration engineering insists on backward-compatible schema changes, and why irreversible deployments are on this site's list of changes that require a human reviewer.
Diagram: One build from merge to production, with the two failure exits
The top lane is the normal path: pull request gates, staging, the release approval, the canary, production. The bottom lane holds the exits. A failed gate or a declined approval stops the build and sends the change back to its author. A canary above its threshold rolls production back to the previous digest automatically. Recovery after a rollback is always a new build and a new canary.
- Normal path
- PR gates pass: Unit tests, contract, schema compatibility, secret scan.
- staging: The merged build runs in staging; evals run on golden cases.
- release approval: A person with the release-approver role signs off, with the evidence in front of them.
- canary: A small share of traffic on the new build; the error rate is compared with the threshold.
- production: The promoted digest, its gate results, approver and canary result are stored together.
- Failure exits
- stop: back to the author: A failed gate or a declined approval ends this build; the record says which check failed.
- automatic rollback: Production returns to the previous known-good build; a fix comes as a new build and a new canary.
The diagram follows one build from merge to production. Each box is a stage with the check that gates it. Normal flow goes right. The two failure exits are the important part: a failed gate before production stops the build and sends the change back to its author, while a canary that crosses its threshold rolls production back to the previous digest automatically. Recovery after a rollback is always a new build and a new canary, never a retry of the same one.
Evidence capture: the record that outlives the terminal#
Evidence capture means storing, for every build that moves, the facts that justify the move: which build (digest), from which inputs (pattern version, spec hash, commit), which checks ran with which result, who approved, and what the canary saw. The opening reviewer's questions, "is this the build in production, and who approved it?", are answered from that record and from nothing else. A gate table that scrolled past in a terminal is not evidence.
This site's recommendation: capture evidence as an immutable record keyed by artifact digest, written by the pipeline and not by the people it judges. The minimum fields are the ones cicd_demo.py printed, plus the approver's identity and time, the canary's result, and the deployment's target. Enterprise governance turns the same record into signed provenance a reviewer can verify without trusting the producer.
The same gates in three CI products#
Example C's gates run as a script. In an organization they run in a CI system, and every major one can express the two release controls this page needs: dependent stages, and a person's approval before a protected environment. These facts come from each product's own documentation, read 2026-11-20 (docs/facts/cicd.md).
| Capability | GitHub Actions (product) | GitLab CI/CD (product) | Jenkins Pipeline (open-source project) |
|---|---|---|---|
| Where the pipeline lives | YAML files under .github/workflows | .gitlab-ci.yml at the project root | a Jenkinsfile in source control |
| Ordering | needs between jobs; dependents skipped if a needed job fails | stages, or needs to start as soon as listed jobs finish | stages, with parallel for concurrent branches |
| Human approval before production | environment protection rules with required reviewers (up to 6; one approval needed) | protected environments plus deployment approvals (Premium or Ultimate) | the input step pauses for a person; submitter limits who |
| Self-approval | an option to prevent self-review | by default the triggering user cannot approve their own deployment | the documentation read here does not say |
| Short-lived cloud credentials | OIDC token with id-token: write | id_tokens with a required aud | the documentation read here does not say |
Product (GitLab CI/CD, as documented on 2026-11-20): the shape below is assembled from documented keywords for the intake service: stage, script ("Shell script that is executed by a runner"), environment with name and deployment_tier, and rules with when: manual. Documented shape; not run here. The page also says a manual job is allowed to fail by default and that allow_failure: false makes it a blocking manual job; later stages wait until someone starts it.
stages: [test, deploy]
gates:
stage: test
script:
- python3 ci_checks.py
deploy-production:
stage: deploy
script:
- ./promote.sh production
environment:
name: production
deployment_tier: production
rules:
- when: manual
allow_failure: false
What the CI product does not give you is the factory's half. None of them knows that openapi.json was generated from SPEC-INTAKE-001 by PAT-INTAKE-API 2.1.0, or that a hand edit to it is drift. That knowledge lives in the factory's manifest and in a contract gate like Example C's, which the CI product runs like any other script.
What this is not. This page owns release gates, promotion, canary, rollback and evidence for products a factory builds. One change's path through review, CI and production is Diff to Deploy. How golden cases and graders are designed is Evaluation Engineering. The trust controls the factory installs in the product (authorization, audit, masking) are Insurance factory, part 3. Retries and recovery as a discipline are Agent Reliability Engineering.
Manual approval or automated gate? Decide by risk#
This page owns one of the site's eleven decisions. Teams tend to fall into one of two habits: a person approves everything, which turns approval into a rubber stamp, or nothing needs a person, which leaves irreversible changes to a script.
The question. For a given stage of a given product, should a person approve, or is an automated quality gate enough?
Diagram: Manual approval or automated gate: three questions, asked in order
Ask the questions on the left in order. A change that cannot be undone by switching builds needs a person. A change in a high-risk class (authorization, insurance decision logic, schema destruction, credential handling) needs a person. A property no machine can decide needs a person. Only when all three answers point to automation does a recorded automated gate stand alone; in the other cases the automated gates still run first, and the person decides with their evidence.
- Ask in order
- can switching back to the old build undo it?: Code changes usually yes; dropped columns, sent messages and partner writes no.
- is it in a high-risk class?: Authorization, insurance decision logic, schema destruction, credential handling.
- can a machine decide the property?: A contract match is machine-checkable; the business intent of a threshold is not.
- Who decides
- automated gate, recorded: Exit code plus record; no person in the loop.
- gates first, then a person: The person reviews what machines cannot decide, with the gate evidence in front of them.
Read the diagram top to bottom. The first question is whether the change can be undone by switching back to the previous build. If not, a person approves, whatever the tests say. The second is whether the change touches a risk class this site always sends to a person: authorization, insurance decision logic, schema destruction, credential handling. The third is whether an automated check can decide the property at all; a check can decide "the contract matches the spec", it cannot decide "this new rollback threshold is the business's intent". Only when all three answers point to automation does an automated gate stand alone.
| Option | Choose it when | It gives you | Failure mode |
|---|---|---|---|
| Automated gate only | reversible change, no high-risk class, the property is machine-checkable | speed; the same verdict every time | a gate that checks the wrong thing passes everything |
| Automated gate, then a person | high-risk class or irreversible step, with machine-checkable parts | the person reviews what machines cannot decide, with the evidence in front of them | approval becomes a click without reading |
| Person only | the property cannot be checked by a machine yet | judgment where no oracle exists | slow, inconsistent, and unrecorded unless forced |
The criteria, in order: reversibility, risk class, machine-checkability, and the cost of delay. Risk tier comes from demand classification, done before the factory runs.
The counterexample. Imagine a change to Example C's spec that raises the rollback threshold from 2 to 20 percent, with every test that pins the old value updated in the same change. Every gate can be green, and the change quietly removes the safety net. No automated gate can know that 20 percent is wrong for this business. That is a release-policy change, and a person must approve it. The opposite counterexample: a person asked to approve every regenerated service after a logging-format patch approves forty of them in an afternoon, and the approval no longer means anything. A reversible patch with full gate evidence should promote without a person.
The failure modes. Manual approval without evidence in front of the approver becomes a rubber stamp. Automated gates without a list of what they do not check become false confidence. And an approval that is not recorded is, a month later, indistinguishable from no approval.
In a coding agent. A coding agent such as Claude Code or Codex, running headless as a station worker, produces a change; it never passes its own gates. The factory pipeline runs the factory's tests on its output, the product pipeline runs the product's gates, and a person with merge rights reviews before merge. Product (Claude Code GitHub Actions, as documented on 2026-11-20): the action anthropics/claude-code-action@v1 can run in a workflow, and the triggering user needs write access on issue and pull request events. Running an agent inside CI does not make it a gate. The runtime controls around its run belong to Harness Engineering.
Try it#
The lab walks Example C's build from pull request to production through seven gates. Inject a failure at any gate and see what stops and what evidence remains; move the canary error-rate slider above 2 percent to trigger the automatic rollback. The second embed opens on the rollback case.
The same information for a reader without JavaScript (from docs/LABS.md; unit tests, contract check, security scan and evals are real outputs of Example C; the approver's answer and the canary error rate are simulated, illustrative; nothing deploys):
| Gate | Passing evidence | Failing evidence when injected |
|---|---|---|
| unit tests | 50 tests, 0 failures | simulated: 1 of 50 tests fails |
| contract check | 5 routes, 6 event types, 0 problems; schema v1 to v2: 0 breaking changes | hand-edited contract: 4 problems |
| security scan | 34 files scanned, 0 findings, 10 demo-token mentions allowlisted | planted secrets: 4 findings; the demo token allowlisted |
| evals | 4 of 4 recommendation cases give the expected recommendation | 3 of 4: UW-004 expected escalate, got approve |
| human approval | a person with the release-approver role signs off (simulated) | the release approver declines (simulated) |
| canary | 5% of traffic for 60 minutes, error rate at or below 2.0% | above 2.0%: automatic rollback to the previous version |
| promote | promoted to production (documentation only) | not reached |
| Preset | Outcome |
|---|---|
default | promoted: all seven gates pass, canary 0.4% |
rollback | the first five gates pass; canary 3.4% is above 2.0%, automatic rollback; promote not run |
WatchAlgo in practice#
WatchAlgo had a factory pipeline in all but name, and no product pipeline at all.
Observed in the repository (zenalgo-frontend, docs/EVIDENCE.md rows 7, 15 and 16): the app's package.json has no test suite (its scripts are dev, build, start and lint). The checks that existed belonged to the factory: verify_solution and verify_viz in scripts/viz/pipeline.py (971863f2) accepted or rejected each generated file before it was written, and vizlib.validate plus a Node cross-check ran the JavaScript solution against the visualization's test cases (d08ba4f0). Those are station acceptance checks, run per unit. Nothing ran when the site itself was built or deployed.
Observed, with an inference (docs/EVIDENCE.md row 15): the remote holds only main (ea250663, 217 visualizations) while the live site serves content of the 2,162-visualization era, and the sitemap generator is untracked. The evidence notes call this consistent with a command-line deploy of the local working tree; no deployment record was inspected. In this page's terms: no promotion of a recorded build, and no evidence capture tying the live site to a commit.
Observed (row 11): the supervisor made a checkpoint commit every 30 minutes. That is a record of what the factory produced, by time, which is a form of evidence; it is not a release record, because no check gated a commit and nothing says which commit went live.
Reported by the author: the corpus of 3,246 problems came from a repeatable process over long sessions; The WatchAlgo story reconciles that report with the repository. Teaching reconstruction: for a solo content site, the missing product pipeline cost little. The one addition worth making is the row EVIDENCE.md marks as missing: publish from a tagged commit and record the commit in the deployment.
Enterprise translation#
For the insurer, the split of ownership follows the split of pipelines. The platform team owns the factory pipeline: the pattern PAT-INTAKE-API, the factory's own tests, and the set of sample products built on every pattern change. Each domain team owns its product's pipeline and its release approval; the submissions team decides when broker-submission-intake reaches production. The security reviewer owns the secret scan and the list of governed components. The pipeline definitions live in the repositories and change through review like code.
What dominates the critical path is rarely the build. It is the approval of the release stages that touch regulated behavior, the availability of a staging environment that behaves like production for the provider and the identity system, and the evidence an auditor will ask for. A factory can generate a service and its gates in minutes; the canary window, the approval and the change-management record take as long as they took before. A pipeline design that ignores this makes the generator faster and the line no faster.
Failure test#
The failure: a developer edits generated code in the product repository to change product behavior (the retry limit from 3 to 5), bypassing the spec and the factory.
How the factory catches it: the product pipeline's unit-test gate fails, because the tests encode the spec's behavior, and the exit code blocks the merge. If the edit had touched the API contract instead, the contract gate would have caught it (test_contract_gate_catches_a_hand_edited_contract). The fix is routed back to the spec: change max_attempts there, run the factory, update the acceptance tests on purpose, and let both pipelines check the regenerated service.
What to run or toggle: in a scratch copy of Example C, add the planted test shown above to tests/, run /opt/homebrew/bin/python3 ci_checks.py, and read out/gate_report.json; remove it and run again. In the lab, pick "unit tests" in the failure select and see which later gates never run.
Where to go deeper#
- Enterprise governance (next): who approves what, approved patterns, policy as code, and signed provenance.
- Diff to Deploy: one change's path through review, CI and production.
- Evaluation Engineering: designing the golden cases and graders the factory runs as a gate.
- Data and integration engineering: the schema compatibility gate and why migrations must stay reversible.
- Insurance factory, part 3: the security gates and trust controls installed in every product.
- Platform foundations: the registry and environment configuration a promotion depends on.
- Failure and recovery scenarios: broken deployments and irreversible writes, in the full failure catalog.
Checklist for Monday#
- Write down, for each product, which pipeline is the factory's and which is the product's, and who owns each.
- Make every gate return an exit code and write a record; a gate that only prints is not a gate.
- Promote one build by digest through every environment; never rebuild between staging and production.
- Keep the previous build available by digest, and list the changes a rollback cannot undo.
- Store an evidence record per promoted build: digest, pattern version, spec hash, gate results, approver, canary result.
- For each release stage, decide manual or automated with the three questions: reversible, high-risk class, machine-checkable.
- Treat a change to the rollout policy (canary share, thresholds) as a reviewed change to the spec, never as a code edit.
Exercises#
- A pattern change passes the factory's own unit tests. Is it ready to reach products?
Answer 1Not until products built with it pass their gates
The factory's unit tests check the factory's code. The factory pipeline must also build sample products (and ideally the real ones) with the new pattern and run each product's gates on them, because the pattern's output is what ships.
- Staging passed with a build made on Monday. Production deploys a build of the same commit made on Tuesday. What is wrong?
Answer 2Production runs an untested artifact
Two builds of one commit can differ (dependencies, toolchain, generated files). Promotion must move the same digest staging tested; the evidence record should make the mismatch visible.
- A canary rolled back. The fix is ready. Do you resume the same canary?
Answer 3No, a new build and a new canary
The fix is a different artifact with a different digest. It passes the gates again and starts a fresh canary window, so the evidence for the new build stands on its own.
Advanced perspective#
Testing the factory: golden products and byte-identical rebuildsWhat the factory pipeline checks that no product pipeline can
A factory's own tests have an unusual subject: their output is other software. Example C shows three kinds. Reproducibility: two assemblies from the same pattern and spec give the same six files with the same sha256 digests (test_two_assemblies_are_byte_identical). Refusal: the factory refuses a spec with an unfilled extension point, a wrong pattern version, or a decision route opened to the agent (test_factory_gates.py). Sensitivity: one changed spec value changes the spec hash and exactly the artifacts that depend on it.
The scratch run for this page checked the last one directly. Changing canary_percentage from 5 to 10 in a copy of the spec and rerunning the factory changed the specification hash (ecb151ab7f7f… to 443b418e1bb9…) and the rollout manifest's digest, and left openapi.json byte-identical. That is how a factory pipeline knows which product gates must rerun after a spec change: only those whose artifacts changed.
The same run showed a quieter safety net. Example C's unit-test gate then failed with two failures. One test pins the canary at 5 percent, so a release-policy change cannot pass without a visible test edit that a reviewer sees. The other is a test that itself raises the canary to 10 to prove the hash changes, and with the spec already at 10 it saw no change. The second is a property of that test's design, not a product defect. Both are worth knowing about: a policy value pinned in a test turns a silent configuration change into a reviewable code change.
A sound factory pipeline therefore runs, on every pattern change: the factory's unit tests, a rebuild of a fixed set of golden products with digest comparison, and the full product gates on each rebuilt product.
A flaky gate is worse than no gateRetry policies, quarantine, and what a rerun is allowed to prove
A test that fails sometimes for reasons unrelated to the change teaches the team to rerun until green. After a few weeks, a red result means "run it again", and a real regression passes on the second try. The gate still exists on paper, and no longer decides anything.
Two disciplines help. First, record every gate run, not only the last one; a build that passed on its third attempt is different evidence from one that passed first time, and the evidence record should say so. Second, quarantine a flaky test with an owner and an expiry, the same way Platform foundations handles a golden-path exception, instead of letting reruns hide it.
CI products offer retry settings at job level. Product (GitLab CI/CD, as documented on 2026-11-20): the retry:when values stuck_or_timeout_failure and job_execution_timeout were deprecated in GitLab 19.1 for more specific reasons such as runner_interrupted. The direction is the right one for any system: retry infrastructure failures automatically, never a failing assertion. Example C's ci_checks.py retries nothing, which is the correct default for a gate. Failure and recovery scenarios treats flaky tests in the full catalog.
Rollback is a promise about the pastExpand and contract, roll forward, and the writes no rollback undoes
Switching traffic back to the previous build undoes code. It does not undo data. If the new build wrote records in a new shape, the old build must still read them; if it removed a column, the old build may fail on every request. The common pattern is expand and contract: first deploy a build that writes both shapes and reads either, then migrate, then remove the old shape in a later release once no build that needs it can come back. Example C's schema gate enforces the first half for submissions: every version reads the version before, and a breaking v3 that removes wind_zone fails the gate (test_breaking_v3_is_rejected_by_the_gate).
Some effects have no rollback at all: a recommendation written to a partner system, a message on a shared event bus, an email to a broker. For those the design work happens before release: idempotency keys so a retried write is not a second write, an outbox so a message is sent only once its transaction commits, and a human approval on any deployment that turns such a write on. When rollback is impossible, the remaining option is to roll forward with a fix, which is slower and riskier, and is the reason irreversible deployments require a person on this site.
DORA's current delivery metrics include a measure that counts these situations. Guideline (DORA, read 2026-11-20): DORA publishes five metrics, with change fail rate and deployment rework rate grouped as instability and failed deployment recovery time grouped with throughput; the recovery measure covers only failures caused by a production change. Factory quality engineering is the home of delivery metrics on this site.
The economics of gate orderCheap and decisive first; expensive and slow last; people at the end
Every gate has a cost per run and a probability of catching something. Putting the cheapest, most frequently failing gates first means most bad builds are stopped in seconds, before they consume a staging slot, an eval budget or a reviewer's attention. Example C's four PR gates are local, in-process checks with no external calls; the evals call the recommender on golden cases, which in a real system costs model usage; the canary costs an hour of a production window; the approval costs a person.
Two consequences follow for a factory. First, the factory generates more builds than a team of people did, so expensive late gates are hit more often; a human approval that takes a few minutes per release becomes the bottleneck when the factory regenerates forty services after a pattern patch (illustrative numbers). The answer is not to remove the approval but to scope it: approve the pattern change once, with its blast radius in front of the approver, and let reversible regenerations promote on their gate evidence. Second, a gate that rarely fails but is expensive (a long integration suite) should run where it pays: on the release candidate, not on every commit. Enterprise governance covers approving a pattern change once for many products.
In one breath#
A factory has two pipelines. The factory pipeline checks that a change to the pattern or a spec still produces correct products; the product pipeline checks that one build of one product is safe to release, promotes the same digest through environments, watches a canary and rolls back to the previous digest when the threshold is crossed. Every gate is an exit code plus a record, and the records joined to the build's digest, pattern version and spec hash are the evidence. A person approves where the change is irreversible, high-risk or not machine-checkable; everywhere else, a recorded automated gate is enough.
The intake service now ships through gates that leave a record. The record says what passed and who signed off, but not who was allowed to sign off, which patterns the organization accepts, or how a reviewer can trust a record the factory wrote about itself. Enterprise governance assigns those roles and turns the evidence into provenance that can be checked.