All chapters
  1. Start hereA software factory, from first principles
  2. Primitive · 1A factory, before AI: input, transformation, check, output
  3. Primitive · 2From repetition to a pattern: what stays, what varies, what to leave alone
  4. Primitive · 3The specification as an executable contract
  5. Primitive · 4Build the smallest factory: four stations, two specs, one manifest
  6. Primitive · 5Assembly stations: typed contracts, owners and failure handling
  7. Primitive · 6AI enters the factory: what a model adds, and what stays deterministic
  8. WatchAlgo · 7The WatchAlgo production story: what the repository shows
  9. WatchAlgo · 8A problem as a production unit: the files one unit must contain
  10. WatchAlgo · 9Station by station: description, solution, implementations, visualization
  11. WatchAlgo · 10The visualizer factory: from algorithm state to renderable steps
  12. WatchAlgo · 11Three flavors and three languages: two different contracts
  13. WatchAlgo · 12Looping over a large backlog: scheduling, limits and a failed-case queue
  14. WatchAlgo · 13Long-running agent execution: context, checkpoints and token limits
  15. WatchAlgo · 14Evidence of correctness: tests the producer cannot talk its way past
  16. WatchAlgo · 15The factory learning loop: defects change the pattern, not only the unit
  17. Enterprise · 16Classify the enterprise demand before choosing a pattern
  18. Enterprise · 17Reference enterprise patterns and their variation points
  19. Enterprise · 18Platform foundations: golden paths, templates, registries and catalogs
  20. Enterprise · 19Pattern versus configuration versus business logic
  21. Enterprise · 20Insurance factory, part 1: building the intake contract
  22. Enterprise · 21Insurance factory, part 2: building the underwriting workflow
  23. Enterprise · 22Insurance factory, part 3: trust and observability built in
  24. Enterprise · 23Data and integration engineering: contracts that survive change
  25. Enterprise · 24The CI/CD and deployment factory: from pull request to production
  26. Enterprise · 25Enterprise governance: owners, approved patterns and provenance
  27. Enterprise · 26Scaling safely: many teams, many agents, one set of boundaries
  28. Operations · 27Factory quality engineering: measuring accepted output, not activity
  29. Operations · 28Failure and recovery scenarios: what breaks a factory, and what restores it
  30. Operations · 29The landscape: which tool for which job, read from its own documentation
  31. Operations · 30Design an enterprise software factory: the blueprint and the rollout
  32. ReferenceSources and verification
  33. ReferenceGlossary
  34. AboutAbout this site
Operations · 29

The landscape: which tool for which job, read from its own documentation

Concepts, standards, open-source projects and products kept apart: platform templates, generators, CI systems, durable workflow engines, agent frameworks, coding agents, contract tools and provenance standards, each with when not to use it.

You will sort
Thirty-one named things into concepts, standards, open-source projects and products, and never blur one into another.
You will place
Each family at one part of the factory: pattern, orchestration, station worker, contracts, release, provenance and operations.
You will decide
Script or engine, SDK or CLI, workflow or agent loop, with the evidence that would justify more machinery.

The insurer's platform team holds an architecture review for the factory that builds the broker intake service. Two proposals arrive on the same morning. The first says "use Temporal for orchestration". The second says "use LangGraph for orchestration". After twenty minutes the reviewer notices that "orchestration" means different things in each. One proposal wants a record that survives a crashed process and a week-long wait for a security approval. The other wants a model to choose the next tool call inside one station.

Observed in the repository (docs/EVIDENCE.md rows 8 and 11): the WatchAlgo pipeline that produced 2,163 visualizations and 639 solutions in September 2026 used no workflow engine at all: a file-locked JSON queue, claude -p worker processes and a shell supervisor that committed a checkpoint every 30 minutes. And this site's Example D gives durable resume and bounded retries in about 650 lines of standard-library Python.

The learner's question: for each job in a software factory, which kind of tool does that job, what does it cost to adopt, and when is none of them needed?

Four kinds of thing, kept apart#

Plain English. Before comparing tools, say what kind of thing each one is. An idea, a written rule, free code you run, and a service someone sells are four different commitments.

This page uses five labels from the site's convention (section C2 of the series reference). A concept is an idea or practice you can apply with any tool (golden paths, product lines, decision records). A standard or specification is a published document that says what a conforming artifact must contain (OpenAPI, SLSA). A guideline is advice from a recognized body that you may follow partly (NIST SSDF, DORA). An open-source project is code you can run and change yourself (Temporal, Pact). A product is offered and changed by a vendor (GitHub Actions, Claude Code, Codex). Some projects have a managed product beside them (Temporal Cloud, Prefect Cloud, Dagster+, PactFlow); the page names both and keeps them apart.

Before AI. A team choosing between Maven archetypes, Jenkins and a cron job faced the same confusion: "build tool", "CI server" and "scheduler" all ran things in order. The useful question was "which job is this, and who owns its record?"

Every fact below comes from the product's or standard's own documentation, read on 2026-11-20, and is logged in docs/facts/landscape.md. Where the documentation read here does not say something, the page says so. The cells named Trade-offs, Lock-in and When not to use it are this site's assessment unless they cite the documentation. There is no ranking, no price and no "best": the right choice depends on the job.

Where each family sits in the factory#

The factory's reference architecture (spec section 7, taught on the blueprint) has a pattern registry and templates, a planner and run record, workcells where station workers run, verification, release, and operations with governance. The tools on this page each serve one or two of those parts. Put them on the map first, then read the items.

Diagram: Where each family sits: kinds of thing (lanes) against parts of the factory (columns)

Lanes are kinds of thing: concepts and guidelines, standards and specifications, open-source projects, products. Columns are parts of the factory from left to right: pattern and scaffold, orchestrating the run, the station worker, contracts and tests, gates and release, provenance and operations. One artifact's path crosses lanes several times: a scaffold from a generator, a run coordinated by an engine or a run record, one work item done by a coding agent, a check by a contract tool, a gate in a CI product, provenance under a standard. No single tool spans more than two columns.

  1. Concepts and guidelines
    1. platform engineering, product lines, ADRs: Ideas for organizing reusable patterns, templates and decisions; no tool required.
    2. NIST SSDF, OWASP, DORA: What to protect and what to measure across the factory's life cycle.
  2. Standards and specifications
    1. OpenAPI, JSON Schema, Jakarta Validation: Machine-readable rules an artifact must satisfy, checked by code the producer did not write.
    2. SLSA, in-toto, SPDX, CycloneDX: Formats that let someone else verify how an artifact was built and what it contains.
  3. Open-source projects
    1. Backstage templates, Cookiecutter, Yeoman: Write a project's first files from variables; a scaffold, not a verified product.
    2. Temporal, Argo, Prefect, Dagster: Record an application-defined process step by step so it survives crashes, retries and long waits.
    3. LangGraph: A graph in which a model's output can choose the next node, with checkpoints per thread.
    4. Pydantic, Pact, Testcontainers: Libraries that run the checks: typed models, consumer contracts, real dependencies in containers.
    5. Jenkins: Pipelines as reviewed code on infrastructure you operate.
    6. Sigstore, OpenTelemetry: Keyless signing with a transparency log; traces and spans for every station attempt.
  4. Products
    1. managed engines: The same engines operated by their vendors; the workflow code stays yours.
    2. Claude Code, Codex, Claude Agent SDK: Station workers that read a repository, edit files and run commands; run headless by the factory.
    3. GitHub Actions, GitLab CI/CD: Gates on every change and approvals before protected environments.

Lanes are kinds of thing; columns are parts of the factory. A run starts with a scaffold, is coordinated by a run record or a workflow engine, hands one work item to a station worker (often a coding agent), checks the artifact against contracts and tests the worker did not write, passes CI gates, and ends with provenance and telemetry. Each arrow crossing a lane line is a hand-off between kinds of tool. No tool covers more than two columns, which is why "which tool for the factory?" has no one-word answer.

The questions this chapter answers#

SDK or CLI?#

A coding agent can staff a station as a command-line process or as a library inside your own program. Product (Claude Code, as documented on 2026-11-20): claude -p runs Claude Code non-interactively, and the documentation now presents it as the Agent SDK used through the CLI. Product (Claude Agent SDK, as documented on 2026-11-20): the SDK offers the same tools, agent loop and context management in Python and TypeScript only; for other languages the documentation says to run claude -p as a subprocess with --output-format json. Product (Codex, as documented on 2026-11-20): codex exec is the non-interactive CLI, and the Codex SDK exists for TypeScript and Python.

The CLI suits a station that a script starts and judges by the files it leaves, as WatchAlgo's workers were. The SDK suits a station whose program must intercept tool calls or decide approvals in code. This site's recommendation: start with the CLI and a strict acceptance test; move to the SDK when the factory needs to act during the run, not only after it.

A script or an orchestration engine?#

Open-source project (Temporal, as documented on 2026-11-20): the documentation states the problem it removes: without it, surviving crashes needs a state table, checkpoint columns, a reconciliation job and retry logic at every call site. That list is exactly what Example D builds by hand. The question is therefore not whether you need durable state, but whether you want to own its code.

Build system or workflow queue?#

A CI system runs a graph of jobs when a repository event happens, usually for minutes, and records artifacts per run. A workflow queue or engine runs a long business process with retries and waits that may last days. They overlap in shape (both have dependent steps) and differ in purpose. Product (GitHub Actions, as documented on 2026-11-20): a matrix can generate at most 256 jobs per workflow run. Product (GitLab CI/CD, as documented on 2026-11-20): parallel accepts 1 to 200. A backlog of thousands of units is a queue's job, not a CI matrix's; the queue design itself belongs to Backlog scheduling.

What does a coding-agent runtime give, and what does the factory add?#

A coding agent is a program that reads a repository, edits files and runs commands in a loop driven by a model. Its runtime gives the loop and many controls; the factory adds everything that makes the output a product.

A coding-agent runtime gives (as documented)The factory adds (this site's design)
tools, the agent loop, context managementthe product spec and the versioned pattern
permission modes, sandboxes (Codex), hooks (Claude Code)the station contract: inputs, outputs, owner
subagents, sessions and resumea durable run record keyed by input hash
structured final output (--json-schema, --output-schema)acceptance tests the agent did not write
project instructions (CLAUDE.md, AGENTS.md)queue, WIP limits, budgets per accepted unit
a GitHub Action for each agentprovenance, escalation to a person, metrics
In a coding agent. A coding agent such as Claude Code or Codex, running headless as a station worker, receives one work item and a scoped checkout and writes code. The factory's own tests, not the agent's, decide whether the artifact advances, and a person with merge rights reviews. The runtime controls around that run (permissions, hooks, context, the loop) are Harness Engineering.

Durable workflow engines or agent orchestration?#

The difference is who decides the next step. In a workflow engine the application's code defines the steps and their order, and the engine makes that sequence survive crashes and waits. In an agent framework a model's output chooses part of the control flow: which tool next, which branch, when to stop.

Diagram: Who decides the next step: the workflow, the model, or the factory's check

The top lane is a workflow the factory defines: validate the spec, build, wait for a person, promote. Its steps and order are code, which is what a durable workflow engine runs. One of its steps, the semantic station, contains the middle lane: an agent loop in which the model reads the work item, picks a tool, reads the result and repeats until it stops. The bottom lane is the factory's acceptance check: tests the agent did not write decide whether the artifact advances. A failure goes back as rework with a bounded attempt count; it never lets the agent grade itself.

  1. Workflow: the application decides
    1. validate spec: A plain function; the engine records that it ran and its result.
    2. semantic station: The one step where business logic must be interpreted; it hands a work item to an agent.
    3. wait for a person: The run is a record, not a live process; it continues when the approval arrives.
    4. promote: Moves the checked artifact on; no model involved.
  2. Agent loop: the model decides
    1. model reads the work item: A coding agent, run headless, receives one work item and a scoped checkout.
    2. model picks a tool: Which tool next is chosen by the model's output, within permission rules.
    3. model stops: The loop ends when the model says it is done; that is a self-report, not evidence.
  3. Factory check: tests decide
    1. acceptance tests the agent did not write: The factory's own tests judge the artifact; exit code plus record.
    2. rework, bounded: A failed check sends the item back, at most a fixed number of attempts, then escalates to a person.

The diagram shows the two kinds nested, not competing. The top lane is a workflow the factory defines: validate, build, wait for a person, promote. One step of it, the station that interprets business logic, contains an agent loop in the middle lane, where the model picks tools until it stops. The bottom lane is the factory's acceptance check, which judges the agent's artifact without asking the agent. Specification (OpenTelemetry GenAI conventions, Development status, read 2026-11-20): the conventions name the same split, invoke_agent when the model drives control flow and invoke_workflow when the application defines it. Open-source project (Temporal): workflow code must be deterministic, and model calls belong in Activities. Open-source project (LangGraph, as documented on 2026-11-20): it has its own checkpointers and durability modes, so "durable" alone does not separate the two families.

What this is not. This page names tools and places them; it is not a buying guide and it ranks nothing. Release gates in CI are CI/CD and deployment; provenance as governance is Enterprise governance; contract tests are Data and integration. The control plane for many agents is Agent Orchestration Engineering, retries as a discipline are Agent Reliability Engineering, and typed contracts and repair are Schema Engineering.

The smallest primitive: what an engine replaces#

Before adopting an engine, see the primitive it replaces running. Example D (examples/d-supervisor/) keeps every factory run, task and station attempt in SQLite, schedules a station DAG, and resumes after a crash. Its README is authoritative for the table and column names. The agent in every example here is a scripted stand-in, so these results show how the factory mechanics work, not how good any model is. Run in a scratch copy on 2026-11-20 with /opt/homebrew/bin/python3 run.py, the crash-and-resume part printed:

== FR-0005 SPEC-ALGO-002: process dies after S2-generate-solution writes its artifact
   run status: CRASHED at S2-generate-solution; physical artifact writes by this process: 3
  station                  att  status     cost  artifact / error
  S1-validate-spec           1  succeeded     0  artifact://FR-0005/S1-validate-spec/6b1ff6027d36
  S2-generate-oracle         1  succeeded    30  artifact://FR-0005/S2-generate-oracle/17a636457aeb
  S2-generate-solution       1  running       0  
  tasks: T1=succeeded T2=running T3=succeeded T4=pending T5=pending T6=pending

== FR-0005 SPEC-ALGO-002: resume run (new supervisor, same database)
   run status: succeeded; physical artifact writes by this process: 3
  station                  att  status     cost  artifact / error
  S1-validate-spec           1  succeeded     0  artifact://FR-0005/S1-validate-spec/6b1ff6027d36
  S2-generate-oracle         1  succeeded    30  artifact://FR-0005/S2-generate-oracle/17a636457aeb
  S2-generate-solution       1  abandoned     0  interrupted
  S2-generate-solution       2  succeeded    40  artifact://FR-0005/S2-generate-solution/b03036903131
  S3-build-visualization     1  succeeded    25  artifact://FR-0005/S3-build-visualization/f176f95de42b
  S3-run-tests               1  succeeded     0  artifact://FR-0005/S3-run-tests/192abe0c57d5
  S4-package-manifest        1  succeeded     0  artifact://FR-0005/S4-package-manifest/0212923625ed
  tasks: T1=succeeded T2=succeeded T3=succeeded T4=succeeded T5=succeeded T6=succeeded

The new supervisor marked the interrupted attempt abandoned, ran only the unfinished station again, and wrote no artifact twice; costs are illustrative units. The seven Python files total 654 lines. An engine earns its place when you need this across machines, teams, days-long waits and audits, not because a loop needs a checkpoint. The record's design is on Long runs.

A script or an engine? Decide by evidence#

This page owns this decision. The question: should the factory's run coordination be your own script with a durable record, or a workflow engine?

OptionChoose it whenIt gives youFailure mode
Script plus durable record (Example D, WatchAlgo)one machine, one operator, runs of minutes to hoursfull control, no service to operatethe record, retries and resume are your code and your bugs
Workflow engine (Temporal, Argo, Prefect, Dagster)waits of days for people, many machines or teams, audit of every stephistory, retries, timers and workers as a servicea second system of record; engine rules (determinism, caching) surprise the team
Agent framework inside a station (LangGraph, Agent SDK)the work is semantic and the next tool depends on the model's readinga model-driven loop with checkpoints or sessionscontrol flow you cannot predict; retries repeat side effects

The criteria: how long the longest wait is; how many machines and teams touch a run; who must read the record besides the operator; and what the real ceiling is. The counterexample: WatchAlgo's limit was the account's usage quota (docs/EVIDENCE.md rows 13 and 14); an engine would have recorded the stalled tasks more neatly and produced no more output. The failure modes: adopting an engine for a one-machine loop doubles the places where state can disagree; refusing one when runs wait a week for a security sign-off produces a home-made engine without tests.

Concepts and practices#

Group: concepts and practices. Ideas you apply with any tool; each is described by a guideline or a practice community, not by a product.

Platform engineering and golden paths#

Concept (CNCF Platforms White Paper v1.0, a guideline, read 2026-11-20). The paper defines a platform as an integrated collection of capabilities presented according to its users' needs, and a golden path as a bundled workflow offered with a project template and documentation.

FieldPlatform engineering and golden paths
Labelconcept, described in a CNCF TAG App Delivery guideline
Problem solvedevery team rebuilding the same build, scan, test, deploy and observe chain
Without iteach product team chooses its own tools and scripts; the factory has nothing standard to target
Where it sitsunder the factory: pattern registry, templates, environments (Platform foundations)
Trade-offsa platform is a product with users and a roadmap; the paper stresses product managers and leadership support
Lock-innone in the concept; the paper says a platform can be as thin as a wiki of standard procedures
When not to use itone team, one product: a repository template is enough
Version verifiedVersion 1 (v1.0); the page shows no date; read 2026-11-20
Official sourcetag-app-delivery.cncf.io/whitepapers/platforms
Platform engineering: minimal sampleA golden path is a workflow plus a template plus documentation

The paper gives no file format. Its description of a golden path, written as a shape (illustration, not a format):

golden path "web service"
  workflow:  build -> scan -> test -> deploy -> observe
  template:  starting files for a new project
  docs:      how to use the path and when to leave it

The paper lists thirteen capability domains, from web portals and golden path templates to artifact storage, and does not use the term "internal developer platform". Its DORA references use the classic four names, which DORA has since replaced (see DORA).

Software product lines#

Concept (Carnegie Mellon SEI, a guideline, read 2026-11-20). A product line is a set of systems sharing a managed set of features, built from common core assets in a prescribed way.

FieldSoftware product lines
Labelconcept, described by SEI guidelines
Problem solvedbuilding many similar products without copying and drifting
Without itclone-and-own, which the SEI explicitly says a product line is not
Where it sitsthe pattern registry: the pattern is a core asset, each spec a product (Pattern)
Trade-offsthe proactive approach needs investment up front; reactive and incremental approaches lower entry cost
Lock-innone; it is a way of organizing work
When not to use itproducts that only look alike; abstracting them early is the over-abstraction Pattern warns about
Version verifiedA Framework for Software Product Line Practice, Version 5.0, dated 31 December 2012; the SEI collection spans 1996 to 2014
Official sourcesei.cmu.edu/library/software-product-lines-collection
Software product lines: minimal sampleThree essential activities, with their inputs and outputs
Core Asset Development   in: product and production constraints, strategy, existing assets
                         out: product line scope, core asset base, production plan
Product Development      in: a product description + scope + core assets + production plan
                         out: products, feedback, new assets, new constraints
Management               coordinates both; the SEI calls each activity essential

The mapping to this site: the pattern is a core asset, the product spec is the product description, and a factory run is product development. The SEI slides also give alternate terms: domain engineering for core asset development, application engineering for product development.

ADRs and reference architectures#

Concept (Architecture Decision Records, a practice, read 2026-11-20). An ADR captures one justified design choice with its context and consequences; a project's ADRs form its decision log.

FieldADRs and reference architectures
Labelconcept (practice); MADR is an open template
Problem solveda later reader, or a later agent, not knowing why the pattern is the way it is
Without itdecisions live in chat threads and leave with the people
Where it sitsthe pattern registry: each pattern version carries its decisions; governance reviews them
Trade-offsone or two pages per decision, kept even when superseded
Lock-innone; Markdown files in the repository
When not to use ittrivial or easily reversed choices
Version verifiedNygard post 15 November 2011; MADR 4.0.0, released 2024-09-17
Official sourceadr.github.io
ADRs: minimal sampleNygard's five sections; MADR's folder and status list differ

Nygard's sections, with his title example:

ADR 9: LDAP for Multitenant Integration
Status: proposed | accepted | deprecated | superseded
Context: the forces at play, stated value-neutrally
Decision: "We will ..." in full sentences
Consequences: positive, negative and neutral results

The two canonical sources disagree on where files live: Nygard uses doc/arch/adr-NNN.md, MADR 4.0.0 uses docs/decisions/NNNN-title-with-dashes.md, and MADR adds a "rejected" status. Reference architectures: the documentation read here does not define them; this site's own are on Enterprise patterns and the blueprint.

Generators and templates#

Group: generators and templates. Tools that write a project's first files from variables. They produce a scaffold, not a verified product.

Backstage software templates#

Open-source project (Backstage, as documented on 2026-11-20). Software Templates load code skeletons, fill in variables and publish the result to places such as GitHub or GitLab, inside a Backstage app.

FieldBackstage software templates
Labelopen-source project (self-hosted app)
Problem solveda self-service form that creates a repository and registers it in the software catalog
Without ita wiki page of manual steps, or a script each team copies
Where it sitsthe first commit of a product; templates and catalog (Platform foundations)
Trade-offsyou run a Backstage app; provider actions come from separately installed modules
Lock-intemplates use Backstage's entity format and action names
When not to use itno portal or catalog in the organization; a CLI generator is enough
Version verifieddocs "Stable (v1.57.0)"; template apiVersion: scaffolder.backstage.io/v1beta3
Official sourcebackstage.io/docs/features/software-templates
Backstage: minimal sampleThe head of a template; the full one is on Platform foundations

A full template for the intake service is on Platform foundations. Its head (documented shape; not run here):

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: intake-service
  title: Intake service
spec:
  owner: platform-team
  type: service

SURPRISING in the source pack: the descriptor-format page still gives backstage.io/v1beta2 for templates and heads spec.owner "optional" while its text says it is required. Follow the writing-templates page. Step ids should be camelCase, because dashes can make expressions return NaN. Modules exist for running Cookiecutter and Yeoman from the scaffolder.

Cookiecutter#

Open-source project (Cookiecutter, as documented on 2026-11-20). Cookiecutter generates a project from a template whose fields and defaults are declared in cookiecutter.json.

FieldCookiecutter
Labelopen-source project (Python CLI)
Problem solvedthe same starting files for every new project, with variables filled in
Without itcopy an old repository and search-and-replace
Where it sitsthe scaffold step of a pattern; can run inside a factory station
Trade-offshooks run code at generation time; a nonzero hook exit stops generation and cleans up
Lock-inlow; templates are files with {{ cookiecutter.<variable> }} placeholders
When not to use itproducts whose differences are behavior, not file names and settings
Version verifiedv2.7.1 (latest GitHub release; the page shows no year)
Official sourcecookiecutter.readthedocs.io
Cookiecutter: minimal sampleFields, a derived slug, a run without prompts

cookiecutter.json (the derived slug is the tutorial's own expression; documented shape, not run here):

{
  "project_name": "Broker Intake",
  "project_slug": "{{ cookiecutter.project_name.lower().replace(' ', '_') }}"
}
cookiecutter --no-input ./intake-template

--no-input uses the defaults and cannot be combined with --replay. Replay stores the answers in ~/.cookiecutter_replay/, so a project can be regenerated from an updated template: useful when a pattern version changes.

Yeoman#

Open-source project (Yeoman, as documented on 2026-11-20). Yeoman runs generators, which are plugins named generator-<name>, with the yo command.

FieldYeoman
Labelopen-source project (Node.js)
Problem solvedcomposable scaffolding: one generator can run others alongside it
Without itone large template that tries to cover every variant
Where it sitsthe scaffold step of a pattern
Trade-offsgenerators are code with a fixed run loop; composed generators share priority groups
Lock-ingenerators are JavaScript classes against the yeoman-generator package
When not to use ita template with a few variables; Cookiecutter-style files are simpler
Version verifiedyeoman-generator v10.3.1, 16 November 2026; v9.0.0 dropped Node 20
Official sourceyeoman.io/authoring
Yeoman: minimal sampleThe run loop's fixed order
initializing -> prompting -> configuring -> default -> writing -> conflicts -> install -> end
a method named after a priority joins it; any other method runs in "default"
this.composeWith(<generator path>, options) runs another generator alongside

SURPRISING in the source pack: the authoring guide shows "yeoman-generator": "^1.0.0" and CommonJS, while the package is at v10.3.1 with three releases between 20 October and 16 November 2026. Check API details against the package, not only the guide.

CI systems#

Group: CI systems. They run a pipeline of jobs on repository events and gate merges and releases. The capability comparison of the three is on CI/CD and deployment; this page adds what matters for a factory.

GitHub Actions#

Product (GitHub Actions, as documented on 2026-11-20). Workflows are YAML files under .github/workflows, triggered by events, made of jobs that run steps on runners.

FieldGitHub Actions
Labelproduct (hosted service; self-hosted runners possible)
Problem solvedgates on every change, approvals before protected environments, short-lived cloud credentials
Without ita CI server you operate, or gates nobody runs
Where it sitsthe product pipeline and the factory pipeline (CI/CD)
Trade-offsa matrix generates at most 256 jobs per run; environments on private repositories need GitHub Pro or Team
Lock-inworkflow syntax and marketplace actions are GitHub-specific
When not to use itas the factory's backlog queue for thousands of units, or for waits of days
Version verifiedhosted, no version shown; artifact actions upload-artifact@v4, download-artifact@v5
Official sourcedocs.github.com/en/actions
GitHub Actions: minimal sampleThe keys a factory uses; runner and steps omitted

A fragment with only the keys this page's facts cover (documented shape; runner selection and steps omitted, so it is not a complete workflow):

on: pull_request
permissions:
  id-token: write
concurrency:
  group: intake-service
  queue: max
jobs:
  gates: {}
  promote:
    needs: gates
    environment: production

SURPRISING in the source pack: repositories created after July 15, 2026 use a default OIDC sub claim that includes owner and repository IDs (not on GitHub Enterprise Server), so cloud trust policies written for the old shape may not match. concurrency.queue: max allows up to 100 pending runs per group; single stays the default.

GitLab CI/CD#

Product (GitLab CI/CD, as documented on 2026-11-20). Pipelines live in .gitlab-ci.yml; runners run jobs grouped in stages, and needs lets a job start as soon as its listed jobs finish.

FieldGitLab CI/CD
Labelproduct (GitLab.com, Self-Managed, Dedicated)
Problem solvedgates, environments, approvals and reusable components in one product
Without itthe same gates scripted and run by hand
Where it sitsthe product pipeline and the factory pipeline (CI/CD)
Trade-offsprotected environments and deployment approvals need Premium or Ultimate; parallel caps at 200
Lock-in.gitlab-ci.yml keywords and CI/CD Catalog components
When not to use itthe same as GitHub Actions: not as a long-running work queue
Version verifiedthe pages state no version; the newest history notes mention GitLab 19.7
Official sourcedocs.gitlab.com/ci
GitLab CI/CD: minimal sampleA shared gate as a versioned component

A factory's standard gates can be one CI/CD component that every generated product includes by version (documented shape; the project path is invented; not run here):

include:
  - component: $CI_SERVER_FQDN/platform/factory-gates/run-gates@1.4.0

Components live in a top-level templates/ directory, need semantic-version tags for releases, and became generally available in GitLab 17.0. SURPRISING: $CI_JOB_JWT_V2 is deprecated in favor of id_tokens, and retry:when values stuck_or_timeout_failure and job_execution_timeout were deprecated in 19.1. The manual-approval job for production is on CI/CD.

Jenkins#

Open-source project (Jenkins Pipeline, as documented on 2026-11-20). A Jenkinsfile in source control defines a pipeline in Declarative or Scripted syntax.

FieldJenkins
Labelopen-source project (self-hosted controller and agents)
Problem solvedpipelines as reviewed code on infrastructure you control
Without itjobs configured by hand in a web form, with no review or audit trail
Where it sitsthe product pipeline; shared libraries can carry the factory's standard gates
Trade-offsyou operate the controller; trusted libraries run any code, untrusted ones run in the Groovy sandbox
Lock-inGroovy pipelines and shared libraries
When not to use itas a durable store of artifacts: archiveArtifacts does not replace a repository manager
Version verifiedLTS 2.580.2, Weekly 2.591 (download page; no dates)
Official sourcejenkins.io/doc/book/pipeline
Jenkins: minimal sampleRequired blocks, a person's approval, a cleanup step

Shape only (angle brackets are placeholders, not syntax):

pipeline {
  agent <where it runs>
  stages {
    stage('Gates')   { steps { <run ci_checks.py> } }
    stage('Promote') {
      input { message 'Promote to production?'  submitter 'release-approvers' }
      steps { <run promote> }
    }
  }
  post { always { junit 'reports/*.xml' } }
}

agent and stages are required; a stage holds exactly one of steps, stages, parallel or matrix; post conditions include always, failure and cleanup, which runs last. A pinned shared library is loaded with @Library('my-shared-library@1.0') _.

Durable workflow engines#

Group: durable workflow engines. They record a long process step by step so it survives crashes, retries failed steps and waits for timers or people. Durable execution means a started process runs to completion even if the machine running it fails.

Temporal#

Open-source project (Temporal, as documented on 2026-11-20). A Workflow runs to completion, from seconds to a year; if its process crashes, another worker replays the Event History and continues.

FieldTemporal
Labelopen-source project; Temporal Cloud is the managed product
Problem solvedthe docs' own list: state table, checkpoint columns, reconciliation job, retries at every call site
Without itExample D's code, written and tested by you
Where it sitsthe planner and run record: one factory run per Workflow, stations as Activities
Trade-offsworkflow code must be deterministic; reordering Activity calls needs versioning APIs
Lock-inorchestration code is written against Temporal SDKs and their replay rules
When not to use itone machine, runs of minutes, one operator
Version verifiedTemporal Server v1.32.2 (releases page, "02 Nov", no year shown)
Official sourcedocs.temporal.io
Temporal: minimal sampleA factory run as a Workflow; defaults that matter

The documented rules, as a shape for the intake build (not code; the SDK calls are not shown because the pages read here describe concepts, not one SDK's API):

Workflow build-intake-service    deterministic: no clock, no random numbers, no model call
  Activity validate_spec         Start-To-Close timeout set (strongly recommended)
  Activity generate_contract     the model call lives here
  wait for Signal release_approved   a person; may take days
  Activity promote
Default Activity retry: 1 s initial, backoff 2.0, max interval 100 s, unlimited attempts

Workers run on your infrastructure; the service schedules and records history but never runs your code. Event data is readable at rest by default unless a Data Converter encrypts payloads. SDKs listed: .NET, Go, Java, PHP, Python, Ruby, Rust, TypeScript. SURPRISING: v1.32 is the last release with the deprecated Worker Versioning APIs; removal is planned for v1.33.

Argo Workflows#

Open-source project (Argo Workflows, as documented on 2026-11-20). A Workflow is a Kubernetes custom resource that is both the definition and the live instance holding state.

FieldArgo Workflows
Labelopen-source project (runs in a Kubernetes cluster)
Problem solveda DAG of containers with retries, artifacts and approval pauses, on Kubernetes
Without itKubernetes Jobs chained by hand-written scripts
Where it sitsthe planner and run record, with each station as a container template
Trade-offsneeds a cluster and a configured artifact repository; DAGs fail fast by default
Lock-inKubernetes and the argoproj.io/v1alpha1 resource format
When not to use itno Kubernetes in the organization
Version verifiedv4.1.6, 22 October 2026 (v4 is the current major line)
Official sourceargo-workflows.readthedocs.io
Argo Workflows: minimal sampleTwo stations in parallel, then a suspend gate

Documented shape; not run here:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: fr-0007
spec:
  entrypoint: line
  templates:
    - name: line
      dag:
        tasks:
          - name: solution
            template: station
          - name: oracle
            template: station
          - name: approve
            template: approval
            dependencies: [solution, oracle]
    - name: station
      retryStrategy:
        limit: 3
        retryPolicy: OnFailure
      container:
        image: factory-station:1.0
        command: [python3, station.py]
    - name: approval
      suspend: {}

argo resume WORKFLOW continues after the suspend. SURPRISING: v4.0 removed the Python SDK (the docs recommend Hera) and removed schedule in CronWorkflows plus mutex, semaphore and podPriority; release v4.0.14 is flagged as failed, use 4.0.15.

Prefect#

Open-source project (Prefect, as documented on 2026-11-20). @flow turns a Python function into a tracked, retryable flow; @task marks retryable, cacheable units inside it.

FieldPrefect
Labelopen-source project; Prefect Cloud is the managed product
Problem solvedrun tracking, retries and scheduling for Python workflows with little ceremony
Without itcron plus logs plus hand-written retry loops
Where it sitsthe planner and run record for Python-based factories
Trade-offscaching and failure rules changed in Prefect 3 (below); RBAC, audit logs and SSO are listed as Cloud-only
Lock-inlow in code (decorators); higher in deployments and work pools
When not to use itnon-Python stations, or when audit records must be self-hosted with RBAC
Version verified3.9.4 (GitHub releases, 19 November)
Official sourceprefect.io/docs/v3
Prefect: minimal sampleA station with side effects must opt out of caching

Shape (decorator arguments as documented; imports omitted):

@task(retries=3, retry_delay_seconds=[1, 2, 4], cache_policy=None)   a station attempt that writes files
@flow                                                                 one factory run

SURPRISING: in Prefect 3 a failed task no longer fails its flow by itself; the flow's final state comes from its return value and uncaught exceptions. Prefect 3 also caches repeated same-input task calls in a flow run automatically, which can skip a side effect; cache_policy=None turns it off. The caching page adds that caching requires result persistence, which is off by default. docs.prefect.io now redirects (308) to prefect.io/docs.

Dagster#

Open-source project (Dagster, as documented on 2026-11-20). Dagster centers on assets: objects in storage defined by code that says how to produce them, with dependencies tracked.

FieldDagster
Labelopen-source project; Dagster+ is the managed product
Problem solvedknowing which stored artifacts exist, what they depend on, and rebuilding them
Without ita folder of outputs and a README saying which script made which
Where it sitsartifacts and state, plus the planner, for data-shaped factories
Trade-offsschedules and sensors need a long-running dagster-daemon; run retries are off in OSS until enabled
Lock-inasset definitions and Definitions objects are Dagster-specific
When not to use itfactories whose units are code changes in repositories rather than stored data assets
Version verified1.13.31 core, 0.29.31 libraries (19 November)
Official sourcedagster.io/docs
Dagster: minimal sampleTwo assets and a dependency; one storage trap

The documentation's minimal shape:

@dg.asset
def daily_sales(): ...

@dg.asset(deps=[daily_sales], group_name="sales")
def weekly_sales(): ...

The run-retry strategy FROM_FAILURE (default) skips successful ops and reuses their stored outputs: the same idea as Example D's input-hash reuse. Trap: with DAGSTER_HOME unset, instance storage uses a temporary directory cleaned up when the process exits. Materializations count toward Dagster+ credit usage.

Agent frameworks and coding agents#

Group: agent frameworks and coding agents. An agent framework is a library for building programs in which a model's output decides part of the control flow. A coding agent is such a program packaged for software work. In a factory both are station workers or the inside of one station; neither is the factory.

LangGraph#

Open-source project (LangGraph, as documented on 2026-11-20). A StateGraph of nodes and edges over a typed state, compiled with an optional checkpointer that saves state per thread.

FieldLangGraph
Labelopen-source project (Python library)
Problem solvedan agent loop with explicit nodes, conditional routing, pauses for a person and checkpoints
Without ita while-loop around model calls with ad hoc state
Where it sitsinside one semantic station (a workcell), not as the factory's run record
Trade-offsretries re-run the whole node (at-least-once), so side effects can repeat
Lock-ingraph code and checkpointer formats are LangGraph-specific
When not to use itdeterministic stations; the factory's cross-station record
Version verifiedlanggraph 1.2.24 (20 November)
Official sourcedocs.langchain.com/oss/python/langgraph
LangGraph: minimal sampleTwo nodes, a pause for a reviewer, a thread id

Imports and calls as documented; compiled by the site's checker, not run here:

import operator
from typing import Annotated, TypedDict

from langgraph.checkpoint.memory import InMemorySaver
from langgraph.graph import END, START, StateGraph
from langgraph.types import interrupt


class State(TypedDict):
    notes: Annotated[list[str], operator.add]


def draft(state: State) -> dict:
    return {"notes": ["draft written"]}


def review(state: State) -> dict:
    answer = interrupt({"question": "accept the draft?"})
    return {"notes": [f"reviewer answered {answer}"]}


builder = StateGraph(State)
builder.add_node("draft", draft)
builder.add_node("review", review)
builder.add_edge(START, "draft")
builder.add_edge("draft", "review")
builder.add_edge("review", END)
graph = builder.compile(checkpointer=InMemorySaver())
config = {"configurable": {"thread_id": "FR-0007"}}

Resuming with Command(resume=value) makes value the return of interrupt(), and the interrupted node restarts from its beginning. InMemorySaver is lost on restart; SqliteSaver and PostgresSaver are separate packages. SURPRISING: the old docs site says "Documentation has moved"; create_react_agent is deprecated in favor of LangChain's create_agent.

Claude Agent SDK#

Product (Claude Agent SDK, as documented on 2026-11-20). A library that runs the Claude Code binary in a process you operate, with the same tools, agent loop and context management, in Python and TypeScript.

FieldClaude Agent SDK
Labelproduct (library; use governed by Anthropic's Commercial Terms)
Problem solveda station worker your program controls: permission callbacks, hooks, sessions, subagents
Without itclaude -p as a subprocess and judging only the files it leaves
Where it sitsthe workcell: one station worker per work item
Trade-offsa subagent spawn depth default of 3 and 20 concurrent subagents; max_budget_usd has no default limit
Lock-inone model provider's agent runtime
When not to use itdeterministic stations; languages other than Python and TypeScript (use the CLI)
Version verifiedTypeScript 0.3.329, Python 0.2.178 (changelogs, no dates)
Official sourcecode.claude.com/docs/en/agent-sdk/overview
Claude Agent SDK: minimal sampleEntry point, isolation options, packages

Shape (option names as documented for Python; not run here):

query(<prompt>, options) -> async iterator: reasoning, tool calls, tool results, final result
options: permission_mode="default", setting_sources=[], allowed_tools=[...], max_budget_usd=<cap>
install: pip install claude-agent-sdk (import claude_agent_sdk)
         npm install @anthropic-ai/claude-agent-sdk

setting_sources=[] isolates the run from user and project settings, which the docs recommend for CI and multi-tenant systems. Permission evaluation order: hooks, deny rules, ask rules, mode, allow rules, then the canUseTool callback; deny rules apply even in bypassPermissions. SURPRISING: renamed from the Claude Code SDK (ClaudeCodeOptions became ClaudeAgentOptions); since v0.1.0 it no longer uses Claude Code's system prompt by default; with Claude Code v2.1.302 or later an app with no mode set starts in default and denies calls that need approval.

Claude Code#

Product (Claude Code, as documented on 2026-11-20). An agentic coding tool that reads a codebase, edits files and runs commands, in the terminal, IDEs, a desktop app and the web.

FieldClaude Code
Labelproduct (CLI and other surfaces)
Problem solveda coding agent that can run headless, without a person at the keyboard, through claude -p
Without ita person implementing each work item
Where it sitsthe workcell: a station worker started by the queue or CI
Trade-offswithout --bare, a -p session runs a project's hooks and .mcp.json servers even in an untrusted folder
Lock-inone model provider; instructions in CLAUDE.md (AGENTS.md is read when no CLAUDE.md exists)
When not to use itas its own acceptance gate; as the run record
Version verified2.1.329, November 19, 2026 (changelog)
Official sourcecode.claude.com/docs/en/headless
Claude Code: minimal sampleA station worker command with an explicit permission mode

Documented flags; the prompt and tool rule are this site's example; not run here:

claude -p "Implement work item T2 of SPEC-ALGO-001; write only src/solution.py" \
  --bare --output-format json \
  --permission-mode acceptEdits \
  --allowedTools "Bash(git diff *)"

json output carries the result and session_id; --resume <session-id> continues a session; --json-schema puts conforming output in structured_output. SURPRISING: from v2.1.302, claude -p with no mode set starts in default (Manual) and denies calls needing approval, where earlier it could start in auto mode; --bare will become the -p default in a future release; anthropics/claude-code-action@beta users must move to @v1.

Codex#

Product (OpenAI Codex, as documented on 2026-11-20). Codex runs in the ChatGPT apps, a CLI, an IDE extension and Codex Cloud; the CLI's non-interactive mode is codex exec.

FieldCodex
Labelproduct (CLI, IDE extension, cloud, GitHub integration)
Problem solveda headless coding agent with OS-level sandboxing: Seatbelt on macOS, bubblewrap on Linux
Without ita person implementing each work item
Where it sitsthe workcell: a station worker started by the queue or CI
Trade-offscodex exec defaults to a read-only sandbox; writes need --sandbox workspace-write
Lock-inone model provider; instructions in AGENTS.md
When not to use itas its own acceptance gate; as the run record
Version verifiedthe documentation read here shows no release version (a mockup shows v0.143.0)
Official sourcelearn.chatgpt.com/docs/non-interactive-mode
Codex: minimal sampleA station worker command with a sandbox and a result schema

Documented flags; the prompt and file names are this site's example; not run here:

codex exec --sandbox workspace-write --json \
  --output-schema station-result.schema.json \
  -o out/last-message.txt \
  "Implement work item T2 of SPEC-ALGO-001; write only src/solution.py"

Progress streams to stderr and only the final message goes to stdout; CI authenticates with CODEX_API_KEY; codex exec resume --last continues. Approval policies are on-request and never. SURPRISING: the docs moved from developers.openai.com/codex to learn.chatgpt.com/docs (some old paths 404); --full-auto is no longer supported for codex exec (use --sandbox workspace-write or --approve-for-me); the untrusted approval policy is retired; codex mcp-server was removed in favor of the app server.

Contracts and tests#

Group: contracts and tests. They let a check the producer did not write decide whether an artifact is acceptable. The factory's use of them is on Data and integration.

OpenAPI#

Standard (OpenAPI Specification 3.2.1, read 2026-11-20). A formal standard for describing HTTP APIs; its Schema Object is by default a superset of JSON Schema Draft 2020-12.

FieldOpenAPI
Labelspecification (OpenAPI Initiative)
Problem solvedone machine-readable contract for an HTTP API that code, tests and people read
Without itthe API is whatever the code does this week
Where it sitsspec compiler and verification: Example C's contract gate compares the generated contract with the spec
Trade-offsdescribes what is allowed, not what a consumer uses
Lock-innone; an open specification
When not to use itevent-only or non-HTTP interfaces need another description
Version verified3.2.1, dated 10 September 2026; Overlay 1.2.0; Arazzo 1.1.0
Official sourcespec.openapis.org/oas/latest.html
OpenAPI: minimal sampleThe required top-level fields

openapi and info (with title and version) are required, plus at least one of components, paths or webhooks (documented shape; the operation is left empty because its fields are not in the pages read here):

openapi: 3.2.1
info:
  title: Broker submission intake
  version: 2.1.0
paths:
  /submissions: {}

SURPRISING: 3.2.0 is already followed by 3.2.1, the 3.2 schema iterations were re-issued as recently as 2026-11-12, and 3.2 adds a top-level $self field.

JSON Schema#

Standard (JSON Schema 2020-12, read 2026-11-20). Core defines the foundation; Validation defines keywords such as type, enum, minimum and required.

FieldJSON Schema
Labelspecification (IETF Internet-Drafts)
Problem solvedone language-neutral check for the shape of a JSON document
Without itevery station validates input its own way
Where it sitsthe spec validator and every station contract that exchanges JSON
Trade-offsformat is an annotation by default; asserting it needs the optional format-assertion vocabulary
Lock-innone
When not to use itrules that span documents or need business context; those are tests
Version verified2020-12 (drafts dated 16 June 2022); dialect URI https://json-schema.org/draft/2020-12/schema
Official sourcejson-schema.org/specification
JSON Schema: minimal sampleA spec fragment with a declared dialect

Documented keywords; field names are Example C's style; parsed by the site's checker:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "properties": {
    "submission_id": {"type": "string", "maxLength": 16},
    "roof_age_years": {"type": "integer", "minimum": 0}
  },
  "required": ["submission_id"]
}

SURPRISING: the "current" version is still the June 2022 Internet-Draft revision; the next version is a work-in-progress snapshot and a "Draft 2021-NN" row is still marked TBD.

Pydantic#

Open-source project (Pydantic, as documented on 2026-11-20). Python models that inherit from BaseModel; creating an instance parses and validates the input, or raises one ValidationError listing every problem.

FieldPydantic
Labelopen-source project (Python; core logic in Rust)
Problem solvedtyped validation of a model's draft output before any station trusts it
Without itdictionary access that fails three stations later
Where it sitsstation contracts in Python workers; can emit JSON Schema 2020-12
Trade-offslax mode by default converts '123' to 123; strict mode does not
Lock-inPython only
When not to use ita contract other languages must share as the source of truth: keep JSON Schema as source
Version verified2.14.4 (major version 2)
Official sourcepydantic.dev/docs/validation/latest
Pydantic: minimal sampleA strict model for one station's output

Compiled by the site's checker, not run here:

from pydantic import BaseModel, ConfigDict, Field


class StationResult(BaseModel):
    model_config = ConfigDict(strict=True)
    submission_id: str = Field(max_length=16)
    area_sq_ft: int = Field(gt=0)


# StationResult.model_validate_json(raw) raises one ValidationError listing every problem

The documentation's own caveat: Pydantic guarantees the types and constraints of the output, not the input data. SURPRISING: docs.pydantic.dev now redirects (301) to pydantic.dev/docs/validation.

Jakarta Bean Validation#

Standard (Jakarta Validation 3.1, read 2026-11-20). Constraint annotations on Java fields and records, checked by a Validator; @Valid cascades into referenced objects.

FieldJakarta Bean Validation
Labelspecification (Jakarta EE); Hibernate Validator is the compatible implementation listed
Problem solveddeclarative input validation in Java services the factory generates
Without ithand-written null and range checks in every controller
Where it sitsthe generated product's input boundary; Java station contracts
Trade-offsneeds Java SE 17 or higher for 3.1
Lock-inlow; a specification with implementations
When not to use itcross-field business rules that need context; write them as rule modules (Pattern vs logic)
Version verified3.1 final (Jakarta EE 11); 4.0 in development for Jakarta EE 12
Official sourcejakarta.ee/specifications/bean-validation
Jakarta Validation: minimal sampleA record with built-in constraints

Documented annotations from jakarta.validation.constraints; not compiled here:

import jakarta.validation.Valid;
import jakarta.validation.constraints.NotBlank;
import jakarta.validation.constraints.Positive;
import jakarta.validation.constraints.Size;
import java.util.List;

public record Submission(
        @NotBlank @Size(max = 16) String submissionId,
        @Positive int areaSqFt,
        List<@Valid Building> buildings) {
}

record Building(@Positive int storeys) {
}

SURPRISING: the spec is now titled "Jakarta Validation" (2.0 and 3.0 were "Jakarta Bean Validation"); 3.0 moved javax.validation to jakarta.validation; 4.0 had its first milestone on 12 November 2025.

Pact#

Open-source project (Pact, as documented on 2026-11-20). Consumer-driven contract tests: the consumer's tests write a pact file, and provider verification replays it against the real provider.

FieldPact
Labelopen-source project; PactFlow is a commercial managed Broker
Problem solvedknowing before release that a provider still serves what each consumer uses
Without itfull integration environments, or breakage found in production
Where it sitsverification, between products the factory builds and the systems they call
Trade-offsneeds consumers who write tests and a Broker to share results
Lock-inlow; implementations in more than 10 languages
When not to use itone team owns both sides and deploys them together
Version verifiedno single release version stated; the mock server supports Pact specification V2 to V4
Official sourcedocs.pact.io
Pact: minimal sampleThe release question as an exit code
pact-broker can-i-deploy --pacticipant Foo --version 23 --to-environment production

Exit code 0 means safe, 1 means not: a gate a CI system can run. Provider verification ignores extra fields. The full treatment, applied to the roof-data provider, is on Data and integration.

Testcontainers#

Open-source project (Testcontainers, as documented on 2026-11-20). A library that runs throwaway instances of databases, brokers and other services in containers during tests.

FieldTestcontainers
Labelopen-source project; Testcontainers Cloud is a product
Problem solvedtests against the real dependency version, not a mock or in-memory substitute
Without itshared test databases, or mocks that drift from the real thing
Where it sitsverification: integration test tier of generated products
Trade-offsneeds a Docker-API-compatible runtime; container start adds time (Java waits up to 60 s by default)
Lock-inlow; libraries in many languages
When not to use itthe fast tier on every change; pure functions
Version verifiedTestcontainers for Java 2.0.5
Official sourcetestcontainers.com
Testcontainers: minimal sampleThe documented Python usage
pip install testcontainers[postgres]
with PostgresContainer("postgres:16") as postgres:
    postgres.get_connection_url()

Containers are removed whether tests pass or fail; a sidecar called Ryuk cleans up even if the test process is killed. SURPRISING: testcontainers-python dropped the separate testcontainers-* packages from 4.0.0; Java is on a new major line, 2.x.

Supply chain#

Group: supply chain. Standards and tools that let someone else verify where an artifact came from. The factory's use of them is on Enterprise governance.

SLSA#

Standard (SLSA v1.2, read 2026-11-20). Levels of assurance for how an artifact was built and, new in v1.2, for how its source was controlled.

FieldSLSA
Labelspecification, status Approved
Problem solveda consumer checking that an artifact came from the expected build of the expected source
Without ittrust in whoever uploaded the file
Where it sitsrelease: the factory's manifest becomes provenance a reviewer verifies
Trade-offsBuild L1 provenance may be unsigned and easy to forge; L2 needs a hosted signing platform; L3 hardened builds
Lock-innone
When not to use itinternal throwaway builds (Build L0 is meant for development builds)
Version verifiedv1.2 (Build Track and new Source Track L1 to L4)
Official sourceslsa.dev/spec/v1.2
SLSA: minimal sampleThe provenance statement's skeleton

Field names as documented; values are placeholders; parsed by the site's checker:

{
  "_type": "https://in-toto.io/Statement/v1",
  "subject": [{"name": "openapi.json", "digest": {"sha256": "<hex digest>"}}],
  "predicateType": "https://slsa.dev/provenance/v1",
  "predicate": {
    "buildDefinition": {"buildType": "<build type URI>", "externalParameters": {}},
    "runDetails": {"builder": {"id": "<builder id>"}}
  }
}

Verification compares provenance with expectations; unrecognized externalParameters should cause verification to fail. The Source Track levels are listed on the source-requirements page. A guessed /spec/v1.2/levels page returns 404.

in-toto#

Standard (in-toto, read 2026-11-20). A framework that records which steps of a supply chain ran, who performed them and in what order; its Attestation Framework defines signed statements about artifacts.

Fieldin-toto
Labelspecification plus Apache-licensed libraries; CNCF Graduated (2025-02-10)
Problem solvedproof that each station ran, by an allowed functionary, on the expected materials
Without ita log the factory wrote about itself
Where it sitsrelease and governance: SLSA provenance is an in-toto Statement
Trade-offslayouts and functionary keys must be managed
Lock-innone
When not to use itbefore the factory has a stable station list to put in a layout
Version verifiedin-toto spec 1.0.0 (2 June 2023); Attestation Framework: GitHub README v1.2, in-toto.io "Stable (v1.0)"
Official sourcein-toto.io/docs/specs
in-toto: minimal sampleThe four attestation layers and the envelope
Predicate   typed metadata (for example SLSA provenance, or a CycloneDX BOM)
Statement   _type https://in-toto.io/Statement/v1, subject (with digest), predicateType
Envelope    DSSE v1.0 recommended: payloadType application/vnd.in-toto+json,
            payload = base64 Statement, signatures required
Bundle      a group of attestations

A layout is signed by the project owner and lists steps and the functionaries allowed to perform each; a link is the signed record a functionary writes while running a step. The two version labels above disagree in the sources (SURPRISING).

Sigstore#

Open-source project (Sigstore, as documented on 2026-11-20). Three parts: cosign signs, Fulcio issues short-lived certificates bound to an OIDC identity, Rekor records each signing in a transparency log.

FieldSigstore
Labelopen-source project; a public-good instance is run by the community
Problem solvedsigning without long-lived keys to store, rotate and leak
Without ita signing key on a build machine
Where it sitsrelease: sign artifacts and attestations; verify at upload or install
Trade-offsFulcio certificates are valid for 10 minutes; verification needs the expected identity and issuer
Lock-inlow; open protocols and several client libraries
When not to use itair-gapped builds that cannot reach an identity provider or log (the docs read here do not cover that setup)
Version verifiedcosign v3.1.4 (20 October); Rekor v2 GA, but the public-good default log stays Rekor v1
Official sourcedocs.sigstore.dev
Sigstore: minimal sampleKeyless sign and verify

Documented commands; variables are placeholders; not run here:

cosign sign "$IMAGE"
cosign verify "$IMAGE" --certificate-identity="$ID" --certificate-oidc-issuer="$ISSUER"

SURPRISING: cosign is on the v3 line; Rekor v2 keeps only hashedrekord and dsse entries and needs cosign v2.6.0+ or v3.0.1+; a 2026-06-28 post says the public-good instance keeps Rekor v1 as default "for the foreseeable future" and flags a post-quantum migration with breaking client changes; --use-signing-config is deprecated.

SPDX and CycloneDX#

Standards (SPDX and CycloneDX, read 2026-11-20). Two formats for a software bill of materials (SBOM): the list of components inside an artifact.

FieldSPDX and CycloneDX
Labelstandards: SPDX is ISO/IEC 5962:2021 per spdx.dev; CycloneDX is ECMA-424
Problem solvedanswering "which artifacts contain this vulnerable component?" in minutes
Without itgrepping lock files across every repository
Where it sitsrelease: one SBOM per artifact, attached as an attestation
Trade-offsan SBOM lists components; it does not say they are safe
Lock-innone; converting between the two is possible but lossy where scopes differ (inference)
When not to use itthere is always a use once artifacts ship; the choice is which format consumers read
Version verifiedSPDX 3.0 on spdx.dev, spec site 3.0.1; CycloneDX 1.7 (2025-10-21), ECMA-424 2nd edition
Official sourcespdx.dev, cyclonedx.org
SPDX and CycloneDX: minimal sampleMedia types and the attestation predicate
CycloneDX JSON   application/vnd.cyclonedx+json   (IANA-registered)
CycloneDX XML    application/vnd.cyclonedx+xml    (IANA-registered)
in-toto predicate type for any CycloneDX BOM: https://cyclonedx.org/bom
SPDX 3.0.1 profiles: Core, Software, Security, Licensing, Dataset, AI, Build, Lite, Extension

Which SPDX version ISO/IEC 5962:2021 matches: the documentation read here does not say (iso.org refused automated reading).

Security and operations references#

Group: security and operations references. Guidelines and conventions that tell a factory what to protect and what to measure.

NIST SSDF#

Guideline (NIST SP 800-218, read 2026-11-20). The Secure Software Development Framework: practices in four groups, Prepare the Organization, Protect the Software, Produce Well-Secured Software, Respond to Vulnerabilities.

FieldNIST SSDF
Labelguideline (NIST Special Publication)
Problem solveda common list of secure-development practices that auditors and buyers recognize
Without iteach team's own idea of "secure enough"
Where it sitsgovernance: map factory stations and gates to SSDF tasks
Trade-offspractices, not tools; mapping takes work
Lock-innone
When not to use itas a checklist that replaces threat modeling of the factory itself
Version verifiedSSDF 1.1 final (February 2022); SSDF 1.2 (SP 800-218 Rev. 1) Initial Public Draft 2025-12-17; SP 800-218A final (July 2024)
Official sourcecsrc.nist.gov/projects/ssdf
NIST SSDF: minimal sampleThe four groups, mapped to factory parts

The groups are NIST's; the mapping is this site's recommendation:

PO  Prepare the Organization       roles, approved patterns, training      -> governance
PS  Protect the Software           signed artifacts, protected branches   -> release, provenance
PW  Produce Well-Secured Software  gates, scans, independent tests        -> verification
RV  Respond to Vulnerabilities     SBOM lookup, pattern patch, blast radius -> learning loop

SP 800-218A adds practices for generative AI and dual-use foundation models, for model producers, system producers and acquirers. SURPRISING: 1.2 is still a draft (comments closed 2026-01-30, no final), so cite 1.1 as final.

OWASP#

Guideline (OWASP GenAI Security Project and OWASP SAMM, read 2026-11-20). Ranked risk lists for LLM and agentic applications, and a software assurance maturity model.

FieldOWASP
Labelguidelines (community project)
Problem solveda shared vocabulary for agent risks: prompt injection, excessive agency, tool misuse
Without iteach review invents its own threat list
Where it sitsverification and governance: threats a station worker's permissions must answer
Trade-offslists name risks; they do not configure controls
Lock-innone
When not to use itas proof of security; a ranked list is a starting point for a threat model
Version verifiedLLM Top 10 2026 (announced 2026-09-01); Top 10 for Agentic Applications for 2026 (2025-12-09); SAMM v2
Official sourcegenai.owasp.org
OWASP: minimal sampleThe items that bite a factory first
LLM01:2026 Prompt Injection       untrusted spec text or repository docs steering a worker
LLM03:2026 Excessive Agency       a worker allowed to merge, deploy or delete
ASI02 Tool Misuse                 a permitted tool used for an unintended effect
ASI04 Agentic Supply Chain Vulnerabilities   a malicious dependency or plugin pulled in by a worker
ASI08 Cascading Failures          one bad artifact consumed by every downstream station

The item names are OWASP's; the factory readings are this site's. SURPRISING: the /llm-top-10/ page still shows the 2025 list; the 2026 ranks come from the 2026-09-01 announcement and a later roundup, and items 7 to 10 for 2026 were not seen in the pages read.

OpenTelemetry#

Open-source project and specification (OpenTelemetry, read 2026-11-20). Traces, metrics and logs with a vendor-neutral protocol (OTLP), a Collector, and semantic conventions that name attributes consistently.

FieldOpenTelemetry
Labelopen-source project and specification; CNCF graduated (May 2026)
Problem solvedone trace per factory run, one span per station attempt, readable by any backend
Without ittext logs per script, joined by hand
Where it sitsoperations: the run record's live view
Trade-offsGenAI and agent span conventions are still Development, so names may change
Lock-inlow; vendor-neutral by design
When not to use itas the run record itself; telemetry is sampled and expires
Version verifiedsemantic conventions 1.45.0; OTLP 1.11.1; GenAI conventions Development
Official sourceopentelemetry.io/docs/specs/status
OpenTelemetry: minimal sampleSpan names for workflows and agents; content off by default
invoke_workflow <name>       the application defines the control flow (a factory run)
invoke_agent <agent name>    the model drives the control flow (inside a semantic station)
execute_tool                 one tool call
gen_ai.input.messages / gen_ai.output.messages   opt-in; default NO_CONTENT
OTLP HTTP default port 4318, path /v1/traces

Do not fabricate gen_ai.conversation.id from a trace id. SURPRISING: the GenAI conventions moved to the repository open-telemetry/semantic-conventions-genai, and the old opentelemetry.io page is no longer maintained.

DORA#

Guideline (DORA software delivery performance metrics, read 2026-11-20). Five metrics in two groups: throughput (change lead time, deployment frequency, failed deployment recovery time) and instability (change fail rate, deployment rework rate).

FieldDORA
Labelguideline (research program)
Problem solvedcomparing delivery performance over time with stable definitions
Without itcounting commits or generated files, which says nothing about delivery
Where it sitsoperations: the product pipelines a factory feeds; factory metrics are on Metrics
Trade-offsdelivery measures, not product correctness; the docs read here do not say how to measure agent-made changes
Lock-innone
When not to use itas the only factory metric; cost per accepted unit and first-pass yield are separate
Version verifiedno version number; guide last updated January 5, 2026
Official sourcedora.dev/guides/dora-metrics
DORA: minimal sampleThe five metrics and what changed
throughput:   change lead time, deployment frequency, failed deployment recovery time
instability:  change fail rate, deployment rework rate

SURPRISING: the classic four keys are now five; "deployment rework rate" was added in 2024; MTTR was renamed failed deployment recovery time in 2023 and narrowed to failures caused by a production change; the old four-keys URL redirects to the five-metric guide.

Capability comparison#

The tables compare by capability only, from the documentation read on 2026-11-20. A blank answer is written as "not stated": the documentation read here does not say. No column ranks anything.

Tool (label)One unit of workOrderingPause for a personRetriesState after a crashWhere it runs
GitHub Actions (product)job of stepsneedsenvironment reviewersnot statednot statedhosted or self-hosted runners
GitLab CI/CD (product)job in a stagestages, needsdeployment approvals (Premium, Ultimate)retrynot statedGitLab.com, Self-Managed, Dedicated
Jenkins (open-source project)stage of stepsstages, parallelinput stepnot statednot statedyour controller and agents
Temporal (open-source project)Activity in a Workflowworkflow codewait on a Signal or UpdateActivities by defaultEvent History replayyour workers; self-hosted or Temporal Cloud service
Argo Workflows (open-source project)container templateDAG dependenciessuspend, then argo resumeretryStrategystored in the Workflow resourceKubernetes
Prefect (open-source project)task in a flowPython callsnot statedretries on tasks and flowsrun states; results persisted only if enabledself-hosted server or Prefect Cloud
Dagster (open-source project)asset or opasset depsnot statedRetryPolicy, run retriesSQLite, Postgres or MySQL storageyour daemon; Dagster+ Serverless or Hybrid
LangGraph (open-source project)nodeedges, routing functionsinterrupt()node re-run (at-least-once)checkpointer per thread_ida library in your process
Claude Agent SDK (product)tool call chosen by the modelthe model, inside limitspermission modes, canUseToolhandled by the SDKsession conversation, not filesa library in your process
Claude Code (product)headless run, claude -pthe modelpermission modes, hooksnot stated--resume a sessionyour machine or CI
Codex (product)headless run, codex execthe modelapproval policy on-requestnot statedcodex exec resumeyour machine, CI or Codex Cloud
Item (label)What it checks or recordsForm
OpenAPI (specification)the HTTP contractYAML or JSON document
JSON Schema (specification)the shape of one JSON documentJSON
Pydantic (open-source project)typed Python models, one error listPython classes
Jakarta Validation (specification)Java field and record constraintsannotations
Pact (open-source project)what each consumer uses, replayed at the providerpact files, Broker
Testcontainers (open-source project)behavior against a real dependencytest library
SLSA (specification)how and from what source an artifact was builtprovenance predicate
in-toto (specification)which steps ran, by whom, in ordersigned statements
Sigstore (open-source project)who signed, when, loggedcertificates, transparency log
SPDX, CycloneDX (standards)what an artifact containsSBOM documents

Try it#

There is no lab on this page; the exercise is a table. Copy it, add one row per station or coordination job in a factory you know, and fill each column from the questions in this chapter. The first two rows are filled: one from WatchAlgo's observed pipeline, one from this site's Example C (a teaching reconstruction).

JobLongest waitMachines or teamsWho decides the next stepLeast machinery that would do
WatchAlgo visualization line (observed, 2026-09)minutes per task; hours of rate-limit pausesone machine, one operatorcode (queue), model inside each taskscript + file-locked queue + checkpoint commits (what it used)
Example C intake build (teaching reconstruction)days, at release approvalplatform and domain teamscode; model only at semantic stationsrun record with a human gate; an engine if runs span teams
your job here

If no row waits longer than a working session and one person runs everything, a script with a durable record is enough. Any row with "model" in the fourth column needs acceptance tests the model did not write, whatever runs it.

WatchAlgo in practice#

Observed in the repository (docs/EVIDENCE.md rows 8, 9, 11 and 14, section 5): the September 2026 line ran on four ingredients. scripts/viz/pipeline.py (971863f2) held the queue in .ai-context/pipeline/queue.json under fcntl.flock, claimed tasks of 5 or 8 codes, and accepted each result with deterministic checks. Workers were claude -p --model sonnet --dangerously-skip-permissions processes with a 60-minute timeout; the Claude Code documentation equates that flag with the bypassPermissions mode. scripts/viz/supervise.sh (017ed6b9) restarted the driver and made a git checkpoint commit every 30 minutes. There was no workflow engine, no agent framework and no CI test suite (package.json has dev, build, start and lint).

Observed: earlier drivers included an hourly cron watchdog relaunching a claude -p session and Codex workers with Gemini reviewing (row 9). A June orchestration engine in the shared backend (engine.py, 93e2a0b1) defines planner, worker, gate and reviewer roles, with no evidence it produced published content.

Reported by the author: a repeatable process ran for 30+ hours with specialized agents per station until token limits required intervention; The WatchAlgo story reconciles that report with the logs. Teaching reconstruction: for one operator on one machine, the script was the right amount of machinery. What an engine would have added is a classified failure history; what it would not have added is output, because the account quota was the ceiling.

Enterprise translation#

For the insurer, the landscape becomes an approved-tool list with an owner per row. The platform team owns the scaffold (Backstage templates or Cookiecutter), the CI system and its shared gate component, and the factory's run record, which may be a workflow engine because intake builds wait days for security review. Each domain team owns its product pipeline. The security team owns the coding-agent configuration: which permission mode, which sandbox, which tools, pinned to which version. Governance maps stations to SSDF practices and requires SLSA provenance for anything that ships.

What dominates the critical path is not the tool choice. It is getting an engine or a coding agent through security approval, connecting it to identity and secrets, and agreeing who reads its records. Ask those questions during demand classification, before picking the tool.

Failure test#

The failure: the insurer upgrades Claude Code in the workcell image. Its station worker command, claude -p "..." --output-format json, sets no permission mode. Product (Claude Code, as documented on 2026-11-20): from v2.1.302, a -p run with no mode set starts in default and denies calls that need approval; before, it could start in auto mode. Overnight every worker exits without writing its file.

How the factory catches it: the station's acceptance check finds no artifact, so the attempt fails; the run record shows every attempt of every semantic station failing with the same error class from the hour of the upgrade; the bounded retry stops after the attempt limit and escalates to a person instead of looping. That is what Example D does in FR-0007: three failed attempts, escalated, its dependent blocked, run incomplete. The fix is to set the mode and tool rules explicitly in the worker command, pin the CLI version in the workcell image, and run one sample work item through the factory pipeline on every version bump. The same class of failure hit WatchAlgo differently: an unrecognised "weekly limit" message became a 13.5-hour retry storm (docs/EVIDENCE.md section 8), because the error was not classified. The Codex equivalent is a worker script still passing --full-auto to codex exec.

What to run or toggle: in a scratch copy of Example D, run /opt/homebrew/bin/python3 run.py and read the FR-0007 block; then search your own worker launch lines for a missing --permission-mode or --sandbox.

Where to go deeper#

Checklist for Monday#

Exercises#

  1. A team proposes LangGraph as the factory's run record for a build that waits a week for approval. What is the gap?
Answer 1A thread checkpoint is not the factory's cross-station record

LangGraph checkpoints a graph's state per thread inside a station. The factory needs a record across stations, products and teams, with artifacts, input hashes and approvals. Use LangGraph inside the semantic station and a run record (or a workflow engine) around it.

  1. A Prefect flow re-runs a station that writes a file, but the file does not change. Why might that be?
Answer 2Prefect 3 caches same-input task calls in a flow run

Prefect 3 caches repeated same-input task calls within a flow run automatically. A task with side effects should set cache_policy=None, or the factory should key reuse on its own input hash on purpose.

  1. Your GitHub Actions matrix should run 3,246 units. What breaks first?
Answer 3The 256-job matrix limit, and the wrong tool for a backlog

A matrix generates at most 256 jobs per workflow run. More importantly, a backlog of thousands of units needs a queue with WIP limits, retries and a durable record, which is Backlog scheduling's subject.

Advanced perspective#

Durable is not idempotentEvery engine retries; only your design stops a retry from writing twice

Each durable tool on this page re-executes work after a failure, and each documents a different unit of re-execution. Temporal retries Activities by default with unlimited attempts. LangGraph re-runs a whole node on retry (at-least-once) and restarts an interrupted node from its beginning, so its docs say to put non-idempotent side effects after interrupt() and derive idempotency keys from durable state. Prefect 3 caches same-input task calls in a run. Dagster's FROM_FAILURE reuses stored outputs of successful ops.

None of them knows whether a station's side effect (a pull request opened, a record written to a partner system) already happened. That knowledge belongs to the factory: an input hash per attempt and an idempotency key per external write, as in Example D, whose resumed FR-0005 wrote six distinct artifacts and never a duplicate. An engine makes retries reliable; it does not make them safe.

Version churn is an operating costOne reading day found moves, renames, removals and changed defaults across the pack

The source pack read for this page on one day found, among others: moved documentation (Claude Agent SDK to code.claude.com, Codex to learn.chatgpt.com, Pydantic, Prefect and Dagster to new hosts); renames (Claude Code SDK to Claude Agent SDK, Jakarta Bean Validation to Jakarta Validation); removals (Argo's Python SDK, Codex's --full-auto for exec and codex mcp-server, a TypeScript session API); and changed defaults (claude -p permission mode, GitHub's OIDC sub format for new repositories, Prefect 3's failure rule).

For a factory this is not trivia. A station worker command is code that depends on a vendor's flags and defaults, and it breaks silently when they change. Three habits contain it: pin versions in the workcell image; run a canary work item through the factory pipeline on every upgrade; and keep a dated fact log for every claim in the factory's own documentation, so a reader can tell which statements were true on which day. This page's log is docs/facts/landscape.md.

Where lock-in actually livesHosting is the easy part to move; semantics are the hard part

Several projects have a managed product beside them: Temporal Cloud, Prefect Cloud (where RBAC, audit logs and SSO are listed as Cloud-only), Dagster+ (with credit-based use of materializations), PactFlow and Testcontainers Cloud. Moving between self-hosted and managed is usually a configuration change, and the documentation describes both.

The deeper lock-in is semantic. Temporal workflow code obeys replay rules; Argo workflows are Kubernetes resources; Dagster code is shaped around assets; LangGraph code is a graph with reducers; a coding-agent worker depends on one vendor's permission modes and output formats. This site's recommendation: keep the factory's own contracts (spec, station contract, run record, artifact hashes, acceptance tests) outside every tool, and let tools implement them. Then replacing a tool rewrites an adapter, not the factory.

One trace across workflow and agentUse the conventions' split, and keep content capture off

OpenTelemetry's GenAI conventions, still Development, give the factory a ready vocabulary: an invoke_workflow span for the factory run whose steps the application defines, invoke_agent spans inside semantic stations where the model drives control flow, and execute_tool spans for tool calls. That maps onto this site's identifiers: one trace per factory run, one span per station attempt.

Two documented defaults help governance. Message content (gen_ai.input.messages, gen_ai.output.messages) is opt-in, default NO_CONTENT, which keeps specs and generated code out of telemetry unless someone decides otherwise. And the conventions say not to fabricate gen_ai.conversation.id. Because the status is Development, pin the conventions version your dashboards depend on. The depth of agent telemetry is Observability Engineering.

In one breath#

The landscape sorts into concepts, standards, guidelines, open-source projects and products, and each tool serves one or two parts of the factory: scaffold, run coordination, station worker, contracts, release gates, provenance or operations. A workflow engine makes an application-defined process survive crashes and long waits; an agent framework lets a model choose steps inside one station; a coding agent is a station worker whose output the factory's own tests judge. Choose the least machinery that the longest wait, the number of teams and the audit needs require, and set every agent default explicitly.

The tools now have places on the map. Design an enterprise software factory assembles the whole: the reference blueprint, the decision trees, four maturity levels, a pilot and the rollout, with the points where the WatchAlgo analogy stops holding.