All chapters
  1. Start hereA software factory, from first principles
  2. Primitive · 1A factory, before AI: input, transformation, check, output
  3. Primitive · 2From repetition to a pattern: what stays, what varies, what to leave alone
  4. Primitive · 3The specification as an executable contract
  5. Primitive · 4Build the smallest factory: four stations, two specs, one manifest
  6. Primitive · 5Assembly stations: typed contracts, owners and failure handling
  7. Primitive · 6AI enters the factory: what a model adds, and what stays deterministic
  8. WatchAlgo · 7The WatchAlgo production story: what the repository shows
  9. WatchAlgo · 8A problem as a production unit: the files one unit must contain
  10. WatchAlgo · 9Station by station: description, solution, implementations, visualization
  11. WatchAlgo · 10The visualizer factory: from algorithm state to renderable steps
  12. WatchAlgo · 11Three flavors and three languages: two different contracts
  13. WatchAlgo · 12Looping over a large backlog: scheduling, limits and a failed-case queue
  14. WatchAlgo · 13Long-running agent execution: context, checkpoints and token limits
  15. WatchAlgo · 14Evidence of correctness: tests the producer cannot talk its way past
  16. WatchAlgo · 15The factory learning loop: defects change the pattern, not only the unit
  17. Enterprise · 16Classify the enterprise demand before choosing a pattern
  18. Enterprise · 17Reference enterprise patterns and their variation points
  19. Enterprise · 18Platform foundations: golden paths, templates, registries and catalogs
  20. Enterprise · 19Pattern versus configuration versus business logic
  21. Enterprise · 20Insurance factory, part 1: building the intake contract
  22. Enterprise · 21Insurance factory, part 2: building the underwriting workflow
  23. Enterprise · 22Insurance factory, part 3: trust and observability built in
  24. Enterprise · 23Data and integration engineering: contracts that survive change
  25. Enterprise · 24The CI/CD and deployment factory: from pull request to production
  26. Enterprise · 25Enterprise governance: owners, approved patterns and provenance
  27. Enterprise · 26Scaling safely: many teams, many agents, one set of boundaries
  28. Operations · 27Factory quality engineering: measuring accepted output, not activity
  29. Operations · 28Failure and recovery scenarios: what breaks a factory, and what restores it
  30. Operations · 29The landscape: which tool for which job, read from its own documentation
  31. Operations · 30Design an enterprise software factory: the blueprint and the rollout
  32. ReferenceSources and verification
  33. ReferenceGlossary
  34. AboutAbout this site
Enterprise · 23

Data and integration engineering: contracts that survive change

API contracts, schema evolution, events, database migrations, change data capture, legacy adapters, fakes versus real dependencies, service virtualization and contract tests.

You will check
Every boundary the factory does not own with a contract and a test: callers, event consumers, a provider, a database, a legacy system.
You will run
Example C's v1 to v2 migration, a breaking v3 the gate refuses, and a stubbed provider that times out.
You will decide
One repository or several, and what integration contracts must hold the pieces together either way.

On a Friday afternoon the team that runs the roof-data provider ships what its changelog calls a small improvement: roof_age_years now comes back as "22 years" instead of 22, "for readability". Nobody told the insurer's intake team, because nobody thought of the intake service as a consumer.

On Monday morning the underwriting agent assesses SUB-1004. A scratch run of Example C with exactly that change in the stubbed provider's answer printed this:

unhandled: TypeError - unsupported operand type(s) for -: 'int' and 'str'
state: assessed | writes: {'recommendation': 0, 'decision': 0} | audit records: 1
events: ['submission.received.v1', 'submission.validated.v1', 'submission.assessed.v1']
retry, same key: 409 {'error': 'invalid_transition', 'state': 'assessed'}

Every check the factory installed passed when the service was built. The case is now stuck: the service moved it to assessed and announced that to every consumer, then crashed before writing a recommendation, left no audit record of the attempt, and refuses the retry because the case is no longer in validated. The failure did not start in the insurer's code. It started at a boundary nobody had written down as a contract.

The learner's question: at every boundary the factory builds against but does not own, what must it check, with what kind of contract, against what kind of stand-in, and what does a factory do differently from a team that integrates by hand?

A contract is a promise at a boundary#

Plain English. Two systems that talk to each other each believe something about the other: which fields exist, what type each has, what happens on error. A contract writes that belief down, and a contract test checks it before release instead of in production.

An API contract is the written description of what a service accepts and returns at one boundary: its routes, fields, types, error responses and required headers. A tiny example: Example C's factory writes openapi.json from the spec, and in it the assessment route requires an Idempotency-Key header and documents 401, 403, 409 and 503. This site's own artifact: the file is OpenAPI-shaped JSON; the README states it is not validated against the OpenAPI specification. The operational consequence is that a reviewer, a consumer and a gate can all read the same promise. The trade-off is that a contract describes only what someone wrote into it; the provider's answer in the opening scene had no contract at all.

Before AI. Integration teams have long shared an interface description (a WSDL file, an IDL file, a shared library of request classes) and broken each other anyway, because the description lived in one repository and the change in another. A factory adds one thing that hand integration rarely had: it generates the contract from the spec on every run and fails the build when the generated contract and the spec disagree.

The insurer's service has five boundaries. The table names each one, its contract, and what in Example C stands for it.

BoundaryWho is on the other sideContractWhat tends to changeIn Example C
Callersbrokers, the cockpit, the agentthe API contractnew fields, new error codesopenapi.json, checked by the contract gate
Event consumerssystems that read submissions and decisionsversioned event schemaspayload fields, meaning of a fieldevent_schemas.json, six types, names ending in .v1
Outside providerthe roof-data providerits response formattypes, fields, timing, outagesStubRoofProvider; no response contract (the opening scene)
Databasethe service's own storethe table schema and its migrationscolumns added, renamed, droppedan in-memory dictionary; no database
Legacy systeman older intake path that still feeds some caseswhatever it does todaybehavior nobody wrote downnot modelled

Diagram: Five boundaries of the insurer's service · the contract and the test at each

Read the middle lane as the built service and the lanes around it as its neighbors. Callers and event consumers are boundaries Example C checks: the contract gate compares the generated API contract with the spec, and the schema-compatibility gate proves new readers can read old submission payloads. The roof-data provider is played by a stub, and nothing checks the real provider's response format: the boundary that broke in the opening scene. The database and the legacy system are drawn for the enterprise case; Example C keeps its state in memory and has no legacy feed.

  1. Inbound and outbound contracts Example C checks
    1. callers: Routes, roles, error codes and required headers, generated from the spec and compared with it.
    2. event consumers: Six event types with versioned names; submission payloads gated for backward compatibility.
  2. The built service
    1. intake + underwriting review service: The product: the pattern's skeleton configured by the spec.
  3. Boundaries without a checked contract
    1. roof-data provider: A scripted stub with timeouts; the real provider's format is never checked.
    2. database: In an enterprise service, versioned migrations run on a copy before release.
    3. legacy intake system: An older system whose behavior is recorded and translated by an adapter.

The diagram puts the built service in the center and its five neighbors around it. Each edge carries the contract that holds at that boundary and the test that checks it. Two edges are solid in Example C: callers, checked by the contract gate, and event consumers, whose submission payload is checked by the schema-compatibility gate. The provider edge is dashed: a stub plays the provider, and nothing checks the provider's real format. The database and legacy edges are shown for the enterprise case and are not modelled.

Schema evolution: change without breaking readers#

Schema evolution is changing a data format over time while records and messages written in the older format remain readable. A tiny example: Example C's submission schema v2 adds one optional field, roof_material, with the default "unknown". A schema change is backward compatible when readers that use the new schema can still read everything written under the old one. Adding an optional field with a default is backward compatible; removing a field, changing a type, taking away null, narrowing an enum, or adding a required field with no default is not.

That list is not prose in Example C; it is the gate. From examples/c-insurance-service/schemas.py, function compat_problems, verbatim:

def compat_problems(old: dict, new: dict) -> list[str]:
    """Backward-compatibility check between two schema versions. Empty list = compatible.

    Breaking: a field removed, a type changed, nullable taken away, an enum narrowed, an optional field
    made required, or a new required field without a default (old records could not be read).
    """
    problems = []
    of, nf = old["fields"], new["fields"]
    for name, f in of.items():
        if name not in nf:
            problems.append(f"{name}: field removed")
            continue
        g = nf[name]
        if g["type"] != f["type"]:
            problems.append(f"{name}: type changed {f['type']} -> {g['type']}")
        if f.get("nullable") and not g.get("nullable"):
            problems.append(f"{name}: no longer nullable")
        if "enum" in f and "enum" in g:
            lost = sorted(set(f["enum"]) - set(g["enum"]))
            if lost:
                problems.append(f"{name}: enum narrowed, removed {', '.join(lost)}")
        if not f.get("required") and g.get("required") and "default" not in g:
            problems.append(f"{name}: became required without a default")
    for name, g in nf.items():
        if name not in of and g.get("required") and "default" not in g:
            problems.append(f"{name}: new required field without a default")
    return sorted(problems)

The second half of evolution is reading old data under the new schema. Example C never rewrites an old record or event; it upgrades on read. From schemas.py, function upgrade, verbatim:

def upgrade(data: dict, from_schema: dict, to_schema: dict) -> dict:
    """Read an old record under a newer schema: fields added later get their declared default."""
    out = dict(data)
    for name, f in to_schema["fields"].items():
        if name not in from_schema["fields"] and name not in out and "default" in f:
            out[name] = f["default"]
    return out

The executable demonstration#

The script below was written for this page and run on a scratch copy of examples/c-insurance-service/, from inside that folder after one run.py (which writes out/factory/). It reads an old event under the new schema, offers the gate a breaking v3, and hands the contract gate a contract with one header removed.

"""Chapter 23 scratch demo: an old v1 event read under v2, a breaking v3 refused by the gate,
and a hand-edited contract refused by the contract check."""
import copy

import ci_checks
import events
import factory
from clock import FixedClock
from run import req

pattern, spec, policy, cases = factory.load_inputs()
v = spec["submission_schemas"]
config, _ = factory.assemble(pattern, spec)
svc, _ = factory.build_service(config, cases, policy, FixedClock())
svc.handle(req("POST", "/submissions", "broker", cases["cases"]["UW-004"]["body"]))

old = svc.bus.events[0]                        # submission.received.v1, written with schema v1
print("event on the bus:", old["event_type"], "schema", old["data"]["submission_schema_version"],
      "| roof_material in it:", "roof_material" in old["data"]["submission"])
print("read under v2:   roof_material =", events.read_submission(old, v, 2)["roof_material"])
print("v1 -> v2 gate:  ", ci_checks.gate_schema_compat(v) or "0 breaking changes")

v3 = copy.deepcopy(v["2"])
v3["fields"]["roof_age_years"]["type"] = "string"              # "22 years" instead of 22
v3["fields"]["inspection_date"] = {"type": "string", "required": True}
print("v2 -> v3 gate:  ", ci_checks.gate_schema_compat(dict(v, **{"3": v3})))

openapi = factory.load_json("out/factory/openapi.json")
ev = factory.load_json("out/factory/event_schemas.json")
edited = copy.deepcopy(openapi)
edited["paths"]["/submissions/{submission_id}/assessment"]["post"].pop("parameters")   # drop Idempotency-Key
print("contract gate:  ", ci_checks.gate_contract(edited, spec, pattern, ev))
event on the bus: submission.received.v1 schema 1 | roof_material in it: False
read under v2:   roof_material = unknown
v1 -> v2 gate:   0 breaking changes
v2 -> v3 gate:   ['v2 -> v3: inspection_date: new required field without a default', 'v2 -> v3: roof_age_years: type changed integer -> string']
contract gate:   ['assess_submission: write route without Idempotency-Key']

Line by line: the event for SUB-1004 was written with schema v1 and stays that way on the bus; read under v2 it gains roof_material = "unknown", and nothing was rewritten. The real v1 to v2 change passes the gate. The v3 that a hurried team might write, the same type change the provider made in the opening scene plus a new required field, is refused with two named problems before it can ship. And a contract with the Idempotency-Key parameter removed from one write route is refused by the contract gate in one line. Example C's own suite covers the larger breaking case: test_breaking_v3_is_rejected_by_the_gate removes wind_zone, takes away null from roof_age_years, drops tile from the enum and adds a required field, and expects exactly four problems; it passed in today's run with the other 49 tests.

The schema format is Example C's own small shape, "JSON-Schema-like, not JSON Schema" in its docstring; the checks carry over to any schema language.

Events carry two version numbers#

An event is a message one service publishes to say something happened, for any number of consumers to read later. Example C's pattern requires every event name to carry a version (submission.received.v1), and every envelope carries event_schema_version, the trace id, the correlation id and the run id. Look closely at SUB-1101, submitted with schema v2: its event is still submission.received.v1, and inside the data sits "submission_schema_version": 2. Two versions travel together. The event version describes the envelope and its keys; the payload's schema version describes the submission inside. A consumer that reads only the event name will meet a v2 payload under a v1 name, and must read the inner version to know which schema applies.

That matters because the gate on this page checks one direction only. It proves that new readers can read old data. It does not prove that old readers can read new data, which is called forward compatibility. Example C's suite shows the forward direction failing on purpose: test_v2_body_keeps_its_value_and_v1_rejects_the_new_field sends a v2 body labelled v1, and v1's strict validation answers 422 with roof_material: unknown field. A consumer written last year that validates the same way would reject the first v2 event it receives.

Eventual consistency is the property that two parts of a system will agree after a delay, not at every instant. A tiny example: when the underwriter records refer for SUB-1004, the service's state is decided at once, and decision.recorded.v1 goes onto the bus; a reporting consumer that reads the bus a few seconds later shows awaiting_underwriter until then. Nothing is wrong during those seconds; the systems are not yet in step. The consequence for design is that a consumer must never treat its copy as the authority for a decision, and every event must say which run and case it belongs to, so a late reader can put it in order. The trade-off is that the producing service stays fast and independent of its consumers, at the cost of every reader handling delay and duplicates.

Databases, change data capture and legacy systems#

Example C keeps its state in memory, so this part is described, not run. Pattern: each of the three practices below is common in enterprise integration and none is mandated.

A database migration is a versioned script that changes a database's structure, run in order and recorded, so every environment reaches the same schema the same way. A tiny example: adding a roof_material column with a default. The safe shape, often called expand and contract, mirrors Example C's upgrade on read: add the new column so old and new code both work, move reads and writes to it, and remove the old structure only in a later release. A one-step rename breaks every running instance of the old code. The factory generates the migration from the spec change, runs it on a copy before release, and refuses a destructive step without a named approval.

Change data capture (CDC) turns each committed change in a database into an event, by reading the database's own change log rather than asking the application to publish. A tiny example: the row for SUB-1004 changes to decided, and a capture process publishes that change for consumers. It is the usual way to get events out of a system that was never written to publish them, such as an older intake system. Example C publishes from its handlers instead, which leaves a gap between writing and publishing; the folded block on dual writes below covers it.

A legacy adapter is a component that translates between an older system's formats and behavior and the contract the new service expects, so the new service never depends on the legacy system's quirks. A tiny example: a legacy record that stores roof age as two characters with "??" for unknown, mapped to roof_age_years: null. The adapter's contract is the hard part, because the legacy system's behavior was never written down; Classify the enterprise demand explains recording real traffic and replaying it, and Enterprise reference patterns calls this shape the modernization adapter.

Fake or real: what stands on the other side of the boundary#

A fake dependency is a stand-in for an outside system that the tests control, so they can reproduce its answers and its failures without the real thing. A tiny example: Example C's roof provider plays back a script per submission. From provider.py, the class StubRoofProvider, verbatim:

class StubRoofProvider:
    """get_roof_assessment(submission_id) from a script; counts every call per submission."""

    def __init__(self, script: dict, clock, timeout_ms: int = 2000) -> None:
        self.script = {k: list(v) for k, v in script.items()}
        self.clock = clock
        self.timeout_ms = timeout_ms
        self.calls: dict[str, int] = {}

    def get_roof_assessment(self, submission_id: str) -> dict:
        n = self.calls.get(submission_id, 0)
        self.calls[submission_id] = n + 1
        outcomes = self.script.get(submission_id) or [{"status": "no_record"}]
        outcome = outcomes[min(n, len(outcomes) - 1)]
        if outcome == "timeout":
            self.clock.advance(self.timeout_ms)
            raise ProviderTimeout(f"roof provider timed out after {self.timeout_ms} ms")
        result = dict(outcome)
        result["source"] = "roof-data provider (stand-in)"
        return result

The stub never sleeps: a timeout moves the injected clock forward by 2,000 ms and raises. The bounded retry around it, call_with_retry in the same file, tries at most max_attempts times with backoff from the spec ([200, 400] ms) and then returns None, so the caller must report the failure. The README run shows both outcomes, and run.py --verbose shows the clock moving:

14 UW-F02  underwriting-agent  200    agent assesses SUB-1102 (2 timeouts): recommendation=approve citations=R-HUMAN-01, provider_attempts=3
17 UW-F03  underwriting-agent  503    agent assesses SUB-1103 (always times out): provider_unavailable after 3 attempts
  {"attempt": 1, "level": "warn", "msg": "provider attempt", "outcome": "timeout", "submission_id": "SUB-1102", "trace_id": "f76bfc80bbfa968f2d86a7b2de69db6c", "ts": "2026-11-20T09:00:50.000Z"}
  {"attempt": 2, "level": "warn", "msg": "provider attempt", "outcome": "timeout", "submission_id": "SUB-1102", "trace_id": "f76bfc80bbfa968f2d86a7b2de69db6c", "ts": "2026-11-20T09:00:53.200Z"}
  {"attempt": 3, "level": "info", "msg": "provider attempt", "outcome": "ok", "submission_id": "SUB-1102", "trace_id": "f76bfc80bbfa968f2d86a7b2de69db6c", "ts": "2026-11-20T09:00:54.600Z"}

The series case UW-006 (provider timeout) is this situation; Example C's own cases UW-F02 (two timeouts, then an answer) and UW-F03 (never answers: 503 after 3 attempts, nothing written, assessment.failed.v1 emitted) are its two endings. The consequence of a fake is speed and control: these runs take milliseconds and fail the same way every time. The trade-off is the opening scene: a fake behaves the way its author believed the real system behaves, and when the real system changes, the fake does not.

Service virtualization is the next step up: a stand-in that runs as a separate service, answers over the network like the real one, and replays recorded or configured behavior, including latency and errors. Between a fake and the real thing sits a range of choices, each catching a different class of failure:

Stand-inWhat it isCatchesMisses
Stub or fake, in processa class that returns scripted answers (StubRoofProvider)your own retry, timeout and error logicany change in the real system
Virtualized servicea separate process replaying recorded traffic over the networkserialization, network errors, latencychanges since the recording
Real dependency in a containerthe real software started for the test, then removedreal behavior of that versionthe version or configuration production runs
Shared test environmentthe other team's real deployment for testingcurrent behavior, integration end to endnothing locally reproducible; shared and often unstable

Product (Testcontainers, as documented on 2026-11-20): Testcontainers is an open-source library that runs throwaway instances of databases, message brokers and other services that run in Docker during tests; containers are started before tests and removed afterwards whether tests pass or fail, and a Docker-API-compatible container runtime is required. Its documentation shows this Python usage (documented shape; not run here):

pip install testcontainers[postgres]
with PostgresContainer("postgres:16") as postgres:
    postgres.get_connection_url()

Contract tests: checking both sides before release#

Contract testing checks, before release, that each side of a boundary sends and accepts messages matching a shared contract, without starting the whole system. Example C runs the provider-side half for its own API: the contract gate compares the generated openapi.json with the spec. What it cannot run is the consumer side: no test written by the insurer records what the intake service expects from the roof-data provider, and no test at the provider replays those expectations. That missing pair is the opening scene.

Product (Pact, as documented on 2026-11-20): Pact is an open-source, code-first tool for contract tests of HTTP and message integrations. Its contracts are consumer-driven: the consumer's tests run against a mock provider and write a pact file describing each interaction, and provider verification replays each recorded request against the real provider and compares the response with the minimal expected one, so extra fields do not fail. Its documentation calls this "contract by example", unlike a static schema such as OpenAPI. The Pact Broker stores contracts and verification results, and can-i-deploy answers whether a version is compatible with what is already in an environment, exiting 0 when safe and 1 when not. The documented command shape (not run here):

pact-broker can-i-deploy --pacticipant Foo --version 23 --to-environment production

Applied to the opening scene, the intake service's test would record "for SUB-1004, the provider returns roof_age_years as an integer", and the provider's pipeline would replay that before its Friday release and fail.

Static contract languages carry the other half. Standard (OpenAPI Specification 3.2.1, read 2026-11-20): the latest version is 3.2.1, dated 10 September 2026; an OpenAPI document requires openapi and info, and at least one of components, paths or webhooks; its Schema Object is by default a superset of JSON Schema Draft 2020-12. Standard (JSON Schema 2020-12, read 2026-11-20): 2020-12 is the current version, declared with "$schema": "https://json-schema.org/draft/2020-12/schema", and its validation vocabulary includes type, enum, minimum, maximum, pattern and required. Two of Example C's v2 fields, rewritten as JSON Schema with those keywords (documented shape; not run here):

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "properties": {
    "area_sq_ft": {"type": "integer", "minimum": 1},
    "roof_material": {"type": "string", "enum": ["asphalt_shingle", "membrane", "metal", "tile", "unknown"]}
  },
  "required": ["area_sq_ft"]
}

Example C's default and nullable are left out, because the source pack read for this page does not list the keywords for them.

The schema says what is allowed; a consumer contract says what one consumer actually uses. A factory with many consumers usually wants both.

One repository or several#

This page owns one of the site's decisions, because the answer is mostly about contracts.

The question. Should the intake service, its event consumers and the adapters around it live in one repository (a monorepo) or in several repositories joined by integration contracts (multi-repo or polyrepo)? Scaling safely owns the operational side of both shapes: team isolation, locks, scoped credentials.

Diagram: One repository or several · decide from ownership, test every boundary

Ask the questions top to bottom. If the pieces release together and one team owns them, one repository lets a change and its consumers' updates land in one commit. If any piece has another owner or another release schedule, that boundary is a separate repository whatever you choose, held together only by a versioned contract with a contract test on both sides. If building everything on every change is too slow, one repository needs selective builds. If teams need different access to the code, separate repositories or enforced ownership rules. The roof-data provider is the counterexample: its boundary is a contract under any layout.

  1. Questions, in order
    1. same owner, same release train?: Pieces that always change and release together can share a repository.
    2. another owner or schedule on one side?: That boundary is a contract whatever the layout, tested on both sides.
    3. whole build fast on every change?: If not, one repository needs selective builds; different access rights favor separate repositories.
  2. Outcomes
    1. one repository: One commit crosses a boundary; risk: boundaries blur because crossing is cheap.
    2. several repositories + integration contracts: Versioned contracts and consumer contract tests before promotion; risk: drift.
    3. hybrid: Atomic where you own both sides; contracts where you do not.

The decision tree asks its questions in order. Pieces that one team owns and releases together can share one repository, where a change and its consumers' updates land in one commit. A piece with another owner or another release schedule sits behind a versioned contract with tests on both sides, whatever you choose for the rest. A build too slow to run whole on every change needs selective builds, and teams that need different access rights need separate repositories or enforced ownership rules.

OptionChoose it whenIt buysIt risks
One repositoryone team, one release train, pieces change togetheratomic changes across a boundary; one place for the factory's gatesslow builds without selective testing; boundaries blur because crossing them is cheap
Several repositories with integration contractsdifferent owners, schedules or access rightsindependent releases; explicit, versioned boundariescontract drift; a change across a boundary needs coordinated releases
Hybridthe product's own pieces together; outside parties apartatomic changes where you own both sides; contracts where you do nottwo sets of rules to explain

The criteria, in order: who owns each side, whether they release together, whether the whole build can run on every change, and who may read and change which code. The counterexample: the roof-data provider. Even a team that puts every one of its own services in one repository cannot put the provider's code there; that boundary is a contract whatever the repository layout, and in the opening scene it was the one that broke. The failure modes: a single repository where teams change each other's internal code because nothing stops them, so the service boundary exists only on the architecture diagram; and several repositories with no contract tests, where every release is an integration test run in production. This site's recommendation: decide the layout from ownership and release schedules, and treat every boundary between different owners as a versioned contract with a test on both sides, whichever layout holds the code.

In a coding agent. A coding agent working as a station worker sees one checkout. In a single repository it can edit a consumer while changing a producer, which is how boundaries erode; in several, only the contract test tells it a change breaks someone. Either way the factory's gates decide, not the agent's summary. The controls around its run are Harness Engineering.
What this is not. This page checks the boundaries of a product the factory builds. Designing typed schemas, validating model output against them and repairing it is Schema Engineering; retries and recovery as a discipline is Agent Reliability Engineering; the release gates that run these checks in order, with canary and rollback, are the next chapter, CI/CD and deployment.

Try it#

The lab runs Example C's release gates from pull request to production with a failure injected at the contract check: a hand-edited contract that opens the decision route to the broker role, drops its 403 response and adds a path the spec does not have. Change the injected gate, or press the recovery button to fix the contract and run again.

GatePassing evidence (real unless marked)Failing evidence when injected
unit tests50 tests, 0 failures (Example C's suite)simulated: 1 of 50 tests fails
contract check5 routes, 6 event types, 0 problems; schema v1 to v2: 0 breaking changeshand-edited contract: 4 problems (decision route open to broker, no 403, an extra GET /admin, roles differ)
security scan34 files scanned, 0 findings, 10 demo-token mentions allowlisted (counts move as files are added)planted secrets: 4 findings (aws-access-key-id, private-key-block, bearer-token, password-assignment); the demo token allowlisted
evals4 of 4 recommendation cases give the expected recommendation3 of 4: UW-004 expected escalate, got approve
human approvala person with the release-approver role signs off (simulated)the release approver declines (simulated)
canary5% of traffic for 60 minutes, error rate at or below 2.0%error rate above 2.0%: automatic rollback to the previous version
promotepromoted to production (documentation only)not reached

With the contract preset the line stops at the contract check and the 5 later gates do not run. Nothing deploys; the canary error rate and the approver's answer are simulated, illustrative, and the agent in Example C is a scripted stand-in, so these results show how the factory mechanics work, not how good any model is.

WatchAlgo in practice#

WatchAlgo's line shows what happens to data contracts that are never versioned. Observed in the repository (docs/EVIDENCE.md, matrix row 7): the solution validator gained rules after many files were written, and 70 of 281 sampled pre-September solutions fail today's validate_payload. The files did not change; the contract did, with no version to say which rules a file was written under. Observed (matrix row 4): one unit, wa-001609, stores its solution code as one string instead of a map of three languages, a shape the renderer was not written for. A problem as a production unit calls this schema drift and owns the unit file contract.

Observed (matrix row 15): the publication path includes a manifest sync to a backend endpoint, content-status/sync, from generate_content_manifest.mjs, an integration between two systems by the same author whose contract lives only in the code on both sides. Teaching reconstruction (matrix row 5, third column): the fix this site proposes is a versioned unit manifest that lists each facet's schema version and the validator version that accepted it, which is Example C's submission_schema_version applied to content units.

Enterprise translation#

In an enterprise, integration is usually where the critical path runs, and a factory makes that visible rather than shorter. Each boundary gets an owner on both sides, a contract in a registry the factory reads, a version, and a deprecation window agreed between the owners: how long the old version stays supported after the new one ships. The platform team owns the compatibility rules and the gates; each domain team owns its contracts and answers for breaking them; the owner of an outside provider is a party to a written agreement, not a repository the factory can gate.

The factory's part is mechanical and therefore reliable: generate contracts from the spec, refuse a breaking change without a new major version and an approval, run consumer contracts before promotion, run migrations on a copy. It cannot make another team ship a compatible change. A spec for an integration service is not complete until it names every boundary, its contract and its owner.

Failure test#

The failure: the opening scene. The roof-data provider returns roof_age_years as a string. Example C has no contract on the provider's response, so the value reaches the stand-in recommender, which subtracts it from an integer and raises. Read the scratch output again with the code in mind: _assess_submission moves the case to assessed and emits submission.assessed.v1 before calling the recommender, so the crash leaves a case whose state and event say "assessed" with no recommendation behind them. The exception escapes Service.handle before its audit line, so the attempt leaves no audit record. The idempotency store keeps only completed responses, so the retry with the same key runs again, finds the case in assessed, and gets 409. The case is stuck until a person intervenes.

How the factory catches it: in Example C as it stands, it does not, and all 50 tests pass. Three changes would catch it, and each belongs in the pattern rather than in one product. Validate the provider's response against a response contract in the adapter, and treat a mismatch like a timeout: classify it, retry within the bound, then report provider_unavailable with nothing written. Move the case's state only after the step's work has succeeded, and emit the event after that. And wrap the handler so that any unexpected error still returns a 500 with an audit record. Then add the scratch script's provider change as a test, so the pattern proves the case can never again be left half-moved.

What to run or toggle: in a scratch copy of Example C, change the provider_script of UW-004 in cases.json so that roof_age_years is "22 years", run the assessment for SUB-1004, and read the state, the events and the audit. Then run /opt/homebrew/bin/python3 -m unittest tests.test_schema_migration and see that the schema gate covers only the schemas the factory owns.

Where to go deeper#

Checklist for Monday#

Exercises#

  1. The underwriting team wants to rename roof_age_years to roof_age in v3. What does the gate say, and how would you ship the rename?
Answer 1Removed field; expand, then contract

The gate reports roof_age_years: field removed (and, if roof_age is required with no default, a new required field). Ship it in steps: add roof_age as optional while keeping the old field, have writers fill both, move readers to the new one, and remove the old field only in a later major version after consumers confirm.

  1. A consumer team says "we only read submission.received.v1, so we are safe from v2". Are they?
Answer 2No: the payload version moved

No. The event name stays .v1 while the payload's submission_schema_version can be 2. If their reader validates strictly, it will reject roof_material as an unknown field, as v1 validation does in Example C's own test.

  1. Why did the schema-compatibility gate not catch the provider's type change in the opening scene?
Answer 3It only checks schemas the factory owns

The gate compares versions of the submission schema in the spec. The provider's response has no schema in the spec at all, so there was nothing to compare. A boundary without a contract cannot be gated.

Advanced perspective#

Backward, forward, and transitive compatibilityKnow which promise your gate makes

Backward compatibility (new readers read old data) is what Example C's gate checks, and it protects the producer's own readers after an upgrade. Forward compatibility (old readers read new data) protects consumers that upgrade later; it requires readers that ignore unknown fields, which Example C's strict validation deliberately does not do. Full compatibility is both.

There is a third dimension: Example C's gate compares adjacent versions only (v1 to v2, v2 to v3), and it does not look at defaults. Suppose v3 kept roof_material but changed its default from "unknown" to "not_stated". The gate would pass. A v1 record upgraded on read straight to v3 would get "not_stated", while the same record upgraded to v2 last month and stored would hold "unknown": one submission, two answers, depending on the path it took. A gate that compares every version still in storage with the current one, defaults included, costs little and removes the question. Which promise a contract makes belongs in the contract, not in a team's memory; Schema Engineering goes deeper.

Bounded per request is not bounded per caseA retry storm through a polite caller

Example C bounds retries inside one request: at most 3 provider calls, then 503. Its idempotency store deliberately does not keep a 503, so the caller may retry with the same key, and test_503_is_not_cached_so_the_same_key_may_retry shows 6 provider calls after two requests. Each request is bounded; the case is not. A caller that retries every minute on 503 turns a provider outage into a steady load on the provider at the moment it can least take it.

WatchAlgo's line met the same shape at a larger scale. Observed in the repository (docs/EVIDENCE.md, matrix row 12): an unrecognized limit message was counted as a content failure and created 11,503 failed tasks. The fix in both cases is a budget above the single attempt, per case and per dependency per hour, plus a circuit that stops calling a dependency that keeps failing and sends the cases to a queue for later. Agent Reliability Engineering owns that discipline.

Write, then publish: the dual-write gapThe failure test, generalized

Example C's handlers change the in-memory store and publish an event in the same function. In a real service those are two systems, a database and a broker, and no single operation covers both. Publish first and crash before the write, and consumers hear about a change that never happened. Write first and crash before publishing, and consumers never hear about a change that did. The failure test on this page is the first ordering inside one process: the event said assessed and the work behind it never completed.

Pattern: the usual answers are an outbox (write the event as a row in the same transaction as the change, publish it afterwards, and let consumers drop duplicates by event id) or change data capture from the database's log. Both make the event a consequence of a committed change rather than a separate promise. Both make delivery at-least-once, so every consumer must be idempotent, which is the same rule Long-running agent execution applies to station writes.

Who may change a contractGenerated is not the same as owned

Example C generates openapi.json and event_schemas.json from the spec, which keeps them consistent with the service. It also makes a spec edit a contract edit, with no consumer in the loop. In an enterprise that is the gap where breaking changes slip through with every gate green: the spec author changes a field, the factory regenerates the contract, the contract gate compares contract to spec and finds them equal.

Two controls close it. Compare each generated contract with the previously released one, not only with the spec, and treat a difference as a change needing the compatibility rules. And give consumers a voice: either their contracts run before promotion (the Pact model) or a change to a published contract needs an approval from the contract's registered consumers. Both are governance as much as engineering; Enterprise governance owns who approves.

In one breath#

Every product the factory builds has boundaries it does not own, and each needs a written contract, an owner on both sides and a test before release. Example C generates its API contract and event schemas from the spec and gates them: the v1 to v2 migration adds roof_material with a default and passes, old events are upgraded on read and never rewritten, a breaking v3 is refused with named problems, and a contract missing an idempotency header is refused in one line. Fakes, virtualized services and real dependencies in containers each catch a different class of failure, contract tests check what each consumer actually uses, and the repository layout follows ownership, never the reverse. The provider whose answer had no contract is the lesson: it broke the case with every gate green.

Example C now has contracts at its boundaries and gates that refuse breaking changes, but so far the gates have run only when someone typed a command. Releasing the service means running them in a fixed order on every change, promoting through environments, watching a canary and rolling back without a person racing a dashboard. CI/CD and deployment builds that pipeline and keeps it apart from the factory's own.