All chapters
  1. Start hereA software factory, from first principles
  2. Primitive · 1A factory, before AI: input, transformation, check, output
  3. Primitive · 2From repetition to a pattern: what stays, what varies, what to leave alone
  4. Primitive · 3The specification as an executable contract
  5. Primitive · 4Build the smallest factory: four stations, two specs, one manifest
  6. Primitive · 5Assembly stations: typed contracts, owners and failure handling
  7. Primitive · 6AI enters the factory: what a model adds, and what stays deterministic
  8. WatchAlgo · 7The WatchAlgo production story: what the repository shows
  9. WatchAlgo · 8A problem as a production unit: the files one unit must contain
  10. WatchAlgo · 9Station by station: description, solution, implementations, visualization
  11. WatchAlgo · 10The visualizer factory: from algorithm state to renderable steps
  12. WatchAlgo · 11Three flavors and three languages: two different contracts
  13. WatchAlgo · 12Looping over a large backlog: scheduling, limits and a failed-case queue
  14. WatchAlgo · 13Long-running agent execution: context, checkpoints and token limits
  15. WatchAlgo · 14Evidence of correctness: tests the producer cannot talk its way past
  16. WatchAlgo · 15The factory learning loop: defects change the pattern, not only the unit
  17. Enterprise · 16Classify the enterprise demand before choosing a pattern
  18. Enterprise · 17Reference enterprise patterns and their variation points
  19. Enterprise · 18Platform foundations: golden paths, templates, registries and catalogs
  20. Enterprise · 19Pattern versus configuration versus business logic
  21. Enterprise · 20Insurance factory, part 1: building the intake contract
  22. Enterprise · 21Insurance factory, part 2: building the underwriting workflow
  23. Enterprise · 22Insurance factory, part 3: trust and observability built in
  24. Enterprise · 23Data and integration engineering: contracts that survive change
  25. Enterprise · 24The CI/CD and deployment factory: from pull request to production
  26. Enterprise · 25Enterprise governance: owners, approved patterns and provenance
  27. Enterprise · 26Scaling safely: many teams, many agents, one set of boundaries
  28. Operations · 27Factory quality engineering: measuring accepted output, not activity
  29. Operations · 28Failure and recovery scenarios: what breaks a factory, and what restores it
  30. Operations · 29The landscape: which tool for which job, read from its own documentation
  31. Operations · 30Design an enterprise software factory: the blueprint and the rollout
  32. ReferenceSources and verification
  33. ReferenceGlossary
  34. AboutAbout this site
WatchAlgo · 10

The visualizer factory: from algorithm state to renderable steps

A visualization is a sequence of algorithm states with a data contract. Semantic correctness, step invariants, timing, edge cases and accessibility, checked on a trace you can play.

You will see
A visualization as a recorded run: a list of algorithm states with a data contract, not a picture.
You will check
Four kinds of correctness: semantic, step invariants, timing and edge cases, plus accessibility.
You will break
Example B's lower-bound trace, and see which corruption an invariant catches and which only a replay catches.

Open WatchAlgo's unit wa-001689, "Number of Connected Components in an Undirected Graph", and play its first test case. At state 2 the animation says find(0) = 0, find(1) = 1, and the JavaScript tab highlights the line inside find that compresses the path. Now switch the code tab to Java and stay on the same state. The highlighted Java line is for (int[] edge : edges) union(edge[0], edge[1]);, the loop over the edges. The picture says one thing and the code says another.

Nothing in the factory complained. Observed in the repository (copy of wa-001689/visualization.json, commit d08ba4f0, read on 2026-11-20): the state's line id is L2, and L2 exists in all three languages, which is what WatchAlgo's visualization validator checks. It does not check that L2 means the same step in each language. In this unit's Java code the numbered steps come in a different order, so L2 is the edge loop there.

The learner's question: a visualization looks correct when it plays smoothly. What does it mean for it to be correct, and which of those meanings can a factory check without a person watching every animation?

A visualization is a recorded run#

Plain English. An algorithm animation is a list of snapshots. Each snapshot says what the algorithm's variables and data structure hold at one moment, and which line of code produced it. The renderer draws the snapshots one after another. If the snapshots are wrong, the drawing is wrong, however good it looks.

This page calls that list a state trace: an ordered list of states recorded while an algorithm runs on one input, one state per interesting step. A tiny example from Example B, lower bound in a sorted list (find the first index whose value is at least the target): for values = [1, 2, 2, 2, 3] and target = 2, the trace has four states, with the search window [lo, hi] shrinking from [0, 5] to [0, 2] to [0, 1] to [1, 1]. The answer is index 1.

The operational consequence is that a visualization becomes data, and data can be checked. A factory cannot judge whether an animation is pleasant. It can check whether every state is one the algorithm really passes through, whether each state obeys rules that hold at every step, and whether the last state agrees with the answer.

Before AI. Backend engineers already trust this idea in two places. A debugger's step-through shows real program state because the debugger reads it from the running process; nobody types it. And an event-sourced system rebuilds a screen from its event log, so a support agent's timeline is derived from the record, not written next to it. A visualization factory should work like the debugger: the states come from executing the algorithm.
Common confusion. "The animation ends on the right answer" is a much weaker claim than "every state is correct". A trace can start right and end right and show an impossible window in the middle.

The smallest primitive: a trace and its invariants#

A step invariant is a condition that must hold at every state of a trace, not only at the end. For lower bound there are two, plus one rule for the last state. The search window stays inside the list (0 <= lo <= hi <= n). The window always contains the answer (lo <= answer <= hi). And the final state has lo == hi == answer. A trace that breaks any of them would show a learner a window that never held the answer.

Example B checks exactly these, quoted from examples/b-algorithm-factory/visualize.py : check_invariants:

def check_invariants(states, values, target, answer):
    """Return a list of violation strings; empty means the trace is valid."""
    n = len(values)
    problems = []
    if not states:
        return ["trace is empty"]
    for s in states:
        lo, hi = s["lo"], s["hi"]
        if not (0 <= lo <= hi <= n):
            problems.append("step %d: 0 <= lo <= hi <= n fails (lo=%d hi=%d n=%d)" % (s["step"], lo, hi, n))
        if not (lo <= answer <= hi):
            problems.append("step %d: answer %d outside [lo, hi] = [%d, %d]" % (s["step"], answer, lo, hi))
    last = states[-1]
    if not (last["lo"] == last["hi"] == answer):
        problems.append("final state lo=%d hi=%d does not equal oracle answer %d" % (last["lo"], last["hi"], answer))
    return problems

Notice where answer comes from. The factory passes in the oracle's answer, computed by brute force, not the solution's own return value. The check therefore ties the trace to an independent source, which is the site's third refrain in miniature: a self-report of "done" is not evidence; a test the producer did not write is.

The trace itself is produced by running the accepted solution's code with recording added, quoted from the correct attempt in generator.py (CORRECT_SOURCE, function lower_bound_trace):

def lower_bound_trace(values, target):
    lo, hi = 0, len(values)
    states = [{"step": 0, "lo": lo, "hi": hi, "mid": None, "comparison": "start"}]
    step = 0
    while lo < hi:
        step += 1
        mid = (lo + hi) // 2
        if values[mid] < target:
            comparison = "values[%d]=%d < %d: lo = %d" % (mid, values[mid], target, mid + 1)
            lo = mid + 1
        else:
            comparison = "values[%d]=%d >= %d: hi = %d" % (mid, values[mid], target, mid)
            hi = mid
        states.append({"step": step, "lo": lo, "hi": hi, "mid": mid, "comparison": comparison})
    return states

Each state has a step number, the window, the probe position and a sentence that explains the comparison. That sentence matters later, for accessibility.

The data contract: what one state must hold#

Example B's states have five fields. WatchAlgo's have nine, because its renderer draws more than a window: arrays, trees, graphs, grids, hash maps, pointers and a calculation panel. Observed in the repository (REQUIRED_STATE_KEYS and validate in scripts/viz/vizlib.py, commit d08ba4f0):

FieldWhat it carriesShown by the renderer?
stepthe state's position; the validator requires it to equal its indexno
codeLineIdthe id of the code line this state highlights, such as L2yes, in the code panel
phasea short label of the stage, such as init, find, union, returnno
descriptionone line of what happenedno
variablesnamed values, each with changed and displayValueyes, as a grid
dataStructureStatethe primary structure (and an optional secondary one) with per-element statusyes, as the drawing
pointersnamed positions with a colour and labelyes
annotationa sentence for the learneryes, as "Step N:" text
calculationlabelled expressions with values and a highlight colouryes

The "shown" column comes from the renderer, read on 2026-11-20 at HEAD 1b183564: src/components/visualization/VisualizationView.tsx and AnnotatedCodeView.tsx (last changed in df3a297c) and StructureVisualization.tsx (d08ba4f0). The evidence matrix already records that step, phase and description are not displayed.

Two fields link a state to code. codeLineId names a line in annotatedCode, which stores the code per language as a list of lines with ids. A value slot is a small template placed after a code line, filled from the current state's variables while the highlighted line is near it. Observed (copy of wa-001689/visualization.json): its one slot sits after line L2 with the template → find({u}) = {rootX}, find({v}) = {rootY} → {compareResult}, so the learner sees the live values next to the code that produced them. vizlib.validate checks that every variable a slot names exists in the states shown near the slot.

The animation phase is the stage of the algorithm a state belongs to, the phase label. WatchAlgo stores it and does not display it. It is the natural key for timing (pause longer when the phase changes) and for a text summary ("initialize, then 3 unions, then return"), which is why a contract should carry it even if today's renderer ignores it.

Diagram: From algorithm to renderable steps: the visualization contract and its checks

Left to right. A structured test input and the algorithm go into a tracer, which records one state per interesting step. Three checks stand between the trace and the reader: the contract's shape and link checks, the step invariants of the algorithm, and an executed comparison with the real solution. Only a trace that passes all three is written as the artifact. The renderer plays it, with step controls, a text line per step, and motion only if the reader allows it. Examples come from Example B and from WatchAlgo's September design.

  1. Inputs
    1. structured test input: Inputs stored as data, not display strings, so code can execute them.
    2. algorithm: The code whose run the visualization shows; in Example B the accepted solution itself.
  2. Record
    1. tracer: Runs the algorithm and appends one state per interesting step; no state is typed by hand.
  3. Checks
    1. contract checks: Every state has its required fields, a step equal to its index, a code-line id present in every language, slot variables that resolve.
    2. step invariants: Rules that hold at every step, checked against the oracle's answer, not the solution's.
    3. executed comparison: Run the real solution and compare: every state (replay) or only the final output (WatchAlgo's Node cross-check).
  4. Artifact and renderer
    1. trace artifact: The accepted trace, with a hash; nothing is written if a check fails.
    2. renderer: Plays the states with step controls and a text line per step; animation only if the reader allows motion.

Read the diagram left to right. The algorithm runs on a structured test input inside a tracer, and the tracer records states. Three checks sit between the trace and the reader: the shape and link checks of the contract, the invariants, and an executed comparison with the real solution. Only a trace that passes all of them is written as the artifact the renderer plays.

Four kinds of correctness#

A visualization can be wrong in four distinct ways, and each needs its own check.

KindThe questionA check a factory can runWhere WatchAlgo stands
SemanticIs each state what the algorithm really has at that step?produce states by executing the algorithm; replay and comparetracers execute a Python re-implementation; Node compares the final output only
Step invariantsDoes every state obey the rules that hold at every step?per-algorithm invariant functionsgeneral rules only: statuses, pointers inside the structure, edges between existing nodes
TimingCan the reader follow the steps at their own pace?a renderer test: step controls, interval, pause, reduced motionfixed 1,000 ms play interval; step and pause buttons
Edge casesDoes the trace stay valid on empty, single and boundary inputs?fixtures that include them; at least one state per casethree cases per unit (basic, edge, complex); at least two states per case

Semantic correctness is the hard one, and the honest description of WatchAlgo's check needs care. Observed (scripts/viz/generate.py, vizlib.py, commit d08ba4f0): the September tracer is a Python module, written by a model, that re-implements the JavaScript algorithm and records states. The Node cross-check then runs the real JavaScript solution on each test input and compares its output with the tracer's output. That proves the tracer computes the same answer. It does not prove that every intermediate state matches what the JavaScript would have held, because only the final value is compared. Within those limits it is strong evidence, and it found real defects: 86 solution defects in one category, logged in .ai-context/VIZ_SOLUTION_BUGS_FOUND.md (commit 4426f119).

Edge cases have a quieter failure: a trace with one state. A learner pressing Play sees nothing move. Observed (generate.py --check): WatchAlgo refuses any test case with fewer than two states. Example B's empty-list fixture produces one state, [0, 0], and is still valid, because its invariants say what a single state must satisfy. The rule to take away: decide per algorithm what the minimum honest trace is, and test the empty and single-element inputs explicitly.

The demonstration: Example B's trace, its tests, and a corruption#

Copied to a scratch folder with cp -R examples/b-algorithm-factory, then run with /opt/homebrew/bin/python3 run.py --verbose on 2026-11-20. The visualization station's part of the output:

[run ] S3-trace-visualization visualization: 10 traces, 28 states, invariants hold, sha256 2bf5ff26c1a1
       trace for duplicates values=[1, 2, 2, 2, 3] target=2:
         step 0 lo=0 hi=5 mid=None  start
         step 1 lo=0 hi=2 mid=2  values[2]=2 >= 2: hi = 2
         step 2 lo=0 hi=1 mid=1  values[1]=2 >= 2: hi = 1
         step 3 lo=1 hi=1 mid=0  values[0]=1 < 2: lo = 1
end FR-0002 status=succeeded cost=800 units generator_calls=2

The station S3-trace-visualization runs only after the solution was accepted by the oracle, traces all ten shared fixtures, and checks every state. The agent here is a scripted stand-in, so these results show how the factory mechanics work, not how good any model is.

The example's tests prove the invariants on more than the fixtures. tests/test_visualization.py checks every fixture trace, 50 seeded random traces, and five corruptions that must be rejected. Run in the same copy:

$ /opt/homebrew/bin/python3 -m unittest tests.test_visualization
.......
----------------------------------------------------------------------
Ran 7 tests in 0.000s

OK

The seven tests are two valid-trace tests (test_every_fixture_trace_satisfies_invariants, test_seeded_random_traces_satisfy_invariants) and five rejection tests: hi beyond n, lo above hi, a window that excludes the answer, a wrong final state, and an empty trace.

Now the question the tests cannot answer by themselves: which wrong states slip through? The script below was written for this page and run in the same scratch copy, next to visualize.py. It corrupts one state in two ways.

"""Corrupt one state of Example B's lower-bound trace and let check_invariants find it."""
import copy

import factory
import oracle
import visualize
from generator import CORRECT_SOURCE

trace = factory.load_module(CORRECT_SOURCE)["lower_bound_trace"]
values, target = [1, 2, 2, 2, 3], 2
answer = oracle.brute_force(values, target)
states = trace(values, target)
print("valid trace:", visualize.check_invariants(states, values, target, answer))

bad = copy.deepcopy(states)
bad[2]["lo"] = 2  # a renderer bug or a hand-typed state: the window skips the answer
for problem in visualize.check_invariants(bad, values, target, answer):
    print("corrupted:", problem)

subtle = copy.deepcopy(states)
subtle[1]["hi"] = 3  # wrong, but the window [0, 3] still holds the answer
print("subtle, invariants:", visualize.check_invariants(subtle, values, target, answer))
replay = trace(values, target)  # re-execute the algorithm and compare state by state
print("subtle, replay:", [s["step"] for s, r in zip(subtle, replay) if (s["lo"], s["hi"]) != (r["lo"], r["hi"])])
valid trace: []
corrupted: step 2: 0 <= lo <= hi <= n fails (lo=2 hi=1 n=5)
corrupted: step 2: answer 1 outside [lo, hi] = [2, 1]
subtle, invariants: []
subtle, replay: [1]

The first corruption breaks both invariants at step 2, and the check names the step. The second is the instructive one. Setting step 1's hi to 3 is wrong (the algorithm set it to 2), yet the window [0, 3] is inside the list and still holds the answer, so every invariant passes. Only a replay, running the algorithm again and comparing state by state, finds step 1. Invariants are necessary and cheap; they are not sufficient. Semantic correctness needs the states to come from execution, and a replay is the test that proves they did.

Timing and accessibility#

A trace that is semantically perfect can still fail its reader. Three properties belong in the contract.

Timing. The renderer decides how long each state stays on screen. Observed (VisualizationView.tsx, df3a297c): WatchAlgo plays at a fixed interval of 1,000 milliseconds (useState(1000) with no setter), with Step Back, Play or Pause, Step Forward and Reset buttons, and the step counter "N/total". A fixed interval treats a pointer move and a merge of two components alike; a contract that carries the animation phase lets a renderer slow down at phase changes.

Reduced motion. Reduced motion is the reader's operating-system setting asking for less non-essential movement. Standard (CSS media feature, as documented by MDN, read 2026-11-20): the CSS media feature prefers-reduced-motion detects "if a user has enabled a setting on their device to minimize the amount of non-essential motion", with the values no-preference and reduce. Observed (read-only search of src/ at 1b183564): WatchAlgo's source contains no prefers-reduced-motion query, and the code panel's active-line arrow uses a pulsing animation. This site's labs follow the opposite rule: in the trace lab, Play under reduced motion jumps to the last step instead of animating.

A text alternative and keyboard stepping. Standard (WCAG 2.2, W3C Recommendation of 12 December 2024, read 2026-11-20): success criterion 1.1.1 asks that non-text content have "a text alternative that serves the equivalent purpose"; 2.1.1 asks that all functionality be "operable through a keyboard interface"; 2.2.2 asks for "a mechanism for the user to pause, stop, or hide" moving content. Observed: each WatchAlgo state's annotation is shown as text ("Step N: …"), which is a text alternative for the step; the control buttons are native buttons with title attributes, so they are reachable by Tab; no key handler for arrow-key stepping exists in the visualization components. Whether the whole page meets any WCAG level was not assessed here; these are observations of three files.

This site's recommendation. Put accessibility in the data contract, not only in the renderer: every state carries a one-sentence text alternative (Example B's comparison, WatchAlgo's annotation), and the trace carries a summary built from phases. Then a renderer contract test can assert three things on every unit: stepping works from the keyboard, Play stops when motion is reduced, and every state has non-empty text.
In a coding agent. When a coding agent staffs the visualization station, give it the job WatchAlgo's September brief gave its workers: write the tracer, not the JSON. The agent receives one work item (one unit) and a scoped checkout; it writes a tracer module; the factory, not the agent, runs the tracer, validates the states, and replays the real solution. The agent's message that the trace "looks right" carries no weight. Harness Engineering owns the controls around that agent's run.
What this is not. This page is not a guide to animation libraries or front-end performance, and not an accessibility audit method. It treats a visualization as a factory artifact with a checkable contract. Typed validation of such contracts in general is the subject of Schema Engineering.

Try it#

The trace lab is a browser port of Example B's lower_bound_trace and check_invariants. Pick a fixture, change the target, and step with Prev, Next or Play. Each step shows lo, hi, mid, the oracle's answer and the three checks.

Every fixture's trace for its own target, for a reader without JavaScript (generated on 2026-11-20 from examples/labcore/a_lowerbound.py, which wraps Example B's code; this table was written for this page):

FixtureValuesTargetAnswerWindows [lo, hi] per stateViolations
empty[]50[0,0]0
all-smaller[1, 2, 3]93[0,3] [2,3] [3,3]0
all-larger[5, 6, 7]10[0,3] [0,1] [0,0]0
duplicates (default)[1, 2, 2, 2, 3]21[0,5] [0,2] [0,1] [1,1]0
duplicates-at-end[1, 1, 4, 4, 4]42[0,5] [0,2] [2,2]0
single-below[4]30[0,1] [0,0]0
single-equal[4]40[0,1] [0,0]0
single-above[4]51[0,1] [1,1]0
between[1, 3, 5, 7, 9, 11]63[0,6] [0,3] [2,3] [3,3]0
negatives[-5, -3, -3, 0, 2]-31[0,5] [0,2] [0,1] [1,1]0

The 28 states in this table are the "10 traces, 28 states" of the run above. Now corrupt one step:

The corrupt preset on the default fixture moves the window of the middle step off the answer (from the same generator):

SteplohimidComparisonCheck
005nonestartpass
1022values[2]=2 >= 2: hi = 2pass
2221corrupted: window moved to [2, 2]fails: answer 1 outside [lo, hi] = [2, 2]
3110values[0]=1 < 2: lo = 1pass

The final state is still correct. A check that looked only at the end would accept this trace.

WatchAlgo in practice#

Observed in the repository (docs/evidence/station-contracts.md, S2a and S2b): the visualization station had two designs. In April, one agent per problem wrote visualization.json directly, and the checks after it exited were shape checks: the file parses, has at least three test cases, three languages and three flavors. Nothing tied the states to the algorithm. In September, the worker wrote a tracer module per problem (scripts/viz/problems/wa_NNNNNN.py), and code produced and checked the visualization. 2,144 current visualizations have a tracer; 236 come from the earlier era without one. Station decomposition maps that change; this page is about the contract it enforces.

Observed (vizlib.validate, d08ba4f0): for every state, all nine keys present, step equal to its index, codeLineId present in every language, non-empty calculation and annotation, allowed statuses for elements, nodes, edges, cells and map entries, pointers inside the structure, edges between existing nodes; every value slot's variables resolvable near the slot. Then at least two states per test case, then the Node cross-check. 191 tracers opt out of the cross-check with SKIP_NODE_VERIFY = True and a written reason. Across the corpus there are 7,142 test cases with 1 to 101 states each (median 11).

Observed, verified read-only for this page (copies of the four explorer units, 2026-11-20): the line ids are shared by label, and labels come from each language's own numbered step comments. In wa-000001 and wa-001830 the labels line up. In wa-001689 and wa-001378 the Java code numbers its steps in a different order, so the same codeLineId highlights a different step in Java: in wa-001378 the state for "record this node's value" (L5) highlights traverse(node.left, result); in Java instead of result.add(node.val);. The validator passes both units because it checks that the id exists, not what it points to. The renderer highlights the line whose id equals the state's codeLineId in the selected language (AnnotatedCodeView.tsx), so a learner on the Java tab sees the mismatch.

Diagram: One state, three code tabs: a line id that exists everywhere and means different steps

One state of wa-001689 carries the label L2 and the phase find. Each language's annotated code assigns its line ids from its own numbered step comments. In JavaScript and Python, L2 is the path-compression line inside find. In Java the steps are numbered in another order, so L2 is the loop over the edges. The validator only asks whether L2 exists in every language, and the answer is yes. Observed in a copy of the unit, commit d08ba4f0.

  1. State
    1. state 2, phase find: The state says find(0) = 0, find(1) = 1 and names line L2.
  2. Annotated code per language
    1. JavaScript L2: if (parent[x] !== x) parent[x] = find(parent[x]);
    2. Python L2: if parent[x] != x:
    3. Java L2: for (int[] edge : edges) union(edge[0], edge[1]);
  3. Validator
    1. id exists in every language?: vizlib.validate checks presence of the id, not that it names the same step.

The diagram follows one state of wa-001689 into the three code tabs. The state carries only the label L2. Each language's annotated code assigns L2 from its own numbered comments, so JavaScript and Python point at the path-compression line and Java at the edge loop. The validator's question, "does L2 exist in every language?", has the answer yes in all three.

Reported by the author: visualizations exist for the units, in three flavors. Observed: the animation (testCases[].states) is one per test case, shared by all three flavors; only the short thinkingContent text differs by flavor. The next chapter takes that split apart.

Enterprise translation#

Enterprise systems are full of state traces shown to people: an order's status timeline, a claim's history, a deployment's progress page. In the insurance factory of Part III, the factory builds the underwriting-review workflow, and the underwriter cockpit shows an underwriter the steps of one runtime execution, such as run-7f3a for submission SUB-1004: evidence gathered, rules applied, recommendation recorded, waiting for a decision. That is operating the workflow, not building it. The factory's job at build time is to make that timeline trustworthy before any submission arrives.

The rules of this page carry over directly. Each displayed step must be derived from the recorded workflow transitions, never written separately (semantic correctness). Each step must satisfy invariants of the workflow (a recommendation cannot appear before the evidence it cites; requires_human stays true until an underwriter decides). The timeline must handle edge cases: a provider timeout (UW-006) shows a retry and not a gap. And each step needs a text alternative and keyboard navigation, because the underwriter's decision depends on reading it. This site's recommendation: the factory installs a contract test that replays recorded transitions from fixtures through the timeline builder and compares step by step, the enterprise version of Example B's replay. Building vs operating an insurance agent keeps the two sides apart.

Failure test#

The failure: a state's line id points at different code in different languages, as in wa-001689 and wa-001378. Every existing check passes. A learner studying the Java tab connects the "find" step to the edge loop.

How a factory catches it: a check that the line id means the same step in every language, not only that it exists. The cheapest version compares the step label each language attaches to the id (the numbered comment, such as "Find root with path compression") and fails when they differ. The script below was written for this page and run read-only over every visualization file. It is a heuristic: it compares the words of the step comments, so it also flags pairs that say the same thing in other words.

$ /opt/homebrew/bin/python3 lineid_audit.py <copy of the four explorer units> 0.2
visualizations read: 4 (threshold 0.20)
units where a used line id's Java step comment differs from JavaScript: 2
units where a used line id's Python step comment differs from JavaScript: 0
  wa-001378 L3 js='Base case: null node contributes nothing' java='Recursive helper: left → value → right'
  wa-001689 L2 js='Find root with path compression' java='Process all edges'

On the four explorer units it flags exactly the two confirmed cases. What to run or toggle: the stronger fix is upstream: require the same step numbering in all three languages (the solution prompt already asks for "same step numbers in all three languages"), and make the validator compare step labels. To see the semantic failure that invariants miss, run the corruption script above in a copy of Example B and change which field it corrupts.

Decision: who writes the visualization?#

OptionWhat it looks likeChoose it whenCounterexampleFailure mode
A model writes the statesthe worker emits the visualization JSONnever for a trace of execution; acceptable for explanatory textWatchAlgo's April design: shape checks onlyplausible states the algorithm never had
A model writes a tracer; code runs itthe worker writes a program; the factory executes, validates and comparesthe algorithm varies per unit and an executable oracle existsWatchAlgo's September designtracer re-implements a different algorithm; opted-out cross-checks
Code instruments the real solutionone recorder per language hooks the reference codethe reference code is trusted and few languages are supportedExample B traces the accepted solution itselfinstrumentation changes behavior; one recorder per language to maintain

The criterion is how far the states are from execution. This site's recommendation: never accept states that no program produced; prefer instrumenting the real solution where you can, and where a model must write the tracer, compare more than the final output.

Where to go deeper#

Checklist for Monday#

Exercises#

  1. Why does check_invariants receive the oracle's answer instead of the solution's return value?
Answer 1Independence

A buggy solution would produce a trace that agrees with its own wrong answer. Passing the brute-force answer ties the trace to a source the producer did not write.

  1. In the corrupt preset, the final state is correct. Which check fails, and what would a check of only the final output say?
Answer 2The step invariant at step 2

answer 1 outside [lo, hi] = [2, 2] at step 2. A final-output check, like WatchAlgo's Node comparison, would accept the trace.

  1. Name one corruption of the lower-bound trace that passes every invariant.
Answer 3A wider, still valid window

Step 1's hi set to 3 instead of 2, as in the demonstration; or a wrong mid or comparison sentence, which no invariant reads. A replay finds both.

Advanced perspective#

Replay, invariants and output checksThree strengths of semantic check, and what each costs

An output check compares the trace's final answer with an executed solution. It is cheap and language-neutral, and it is what WatchAlgo's Node cross-check does. Invariants check every state against rules of the algorithm; they catch impossible states at low cost but must be written per algorithm, which in a corpus of thousands of units means per algorithm family (binary search, union-find, flood fill, traversal) rather than per unit. A replay compares every state with a second execution; it catches everything the recorder can see, but it needs a second producer of states that is independent of the first, or it compares a tracer with itself.

The practical design is layered. Run the output check and the general shape checks on every unit. Write invariants once per algorithm family and attach them by algorithmMeta.type. Replay a sample against an instrumented reference implementation, and raise the sample rate for families where defects were found. WatchAlgo's corpus makes the family approach realistic: the inventory counts types such as hash-map (199 units) and union-find (47), so a few dozen invariant functions would cover most units.

The renderer is a station tooContract tests per primitive, not screenshots

A correct trace can be drawn wrongly: a pointer outside the array, an edge to a node the renderer skipped, a status colour that means "found" on one structure and "visited" on another. WatchAlgo's validator pushes some of this upstream (pointers inside the structure, edges between existing nodes, allowed status values), which is the right direction: the data contract encodes what the renderer can draw. The remaining risk lives in the renderer code, which has no test suite in the repository (package.json scripts: dev, build, start, lint).

This site's recommendation: one contract test per rendering primitive (array, tree, graph, grid, map), fed with small fixture states, asserting the rendered elements and their accessible text rather than comparing screenshots. Screenshot tests break on every style change and say nothing about meaning. A primitive test fails only when the drawing stops matching the data, which is the defect that reaches learners.

Governing opt-outs191 skipped cross-checks and how to keep the number honest

Every factory check grows an escape hatch. WatchAlgo's is SKIP_NODE_VERIFY = True with a written reason, used by 191 of 2,160 tracers on disk, for inputs the harness cannot build: in-place mutation that returns nothing, class-based solutions, n-ary trees, node references. The written reason is good practice. What is missing is a record that makes the opt-outs visible as a population: how many per family, how many added per week, which ones a later harness could check.

The evidence notes show the cost of not closing the gap: a runner that compared the mutated argument would verify the in-place cases, and the corpus-wide JavaScript re-run on copies found that all eight failing units in a sample were opt-outs or in-place mutation, not wrong content. This site's recommendation: treat each opt-out as an open defect against the harness, report the count next to the acceptance rate, and fail the line if the count grows faster than an agreed threshold. An escape hatch that nobody counts becomes the default path.

How big is the line-id problem?A heuristic upper bound, not a defect count

The same heuristic audit, run read-only over all 2,380 visualization files on 2026-11-20, flags 379 units where a used line id's Java step comment shares less than a fifth of its words with the JavaScript one, and 206 for Python. The flagged set includes false positives: "Sort a copy, leave original unchanged" against "Clone and sort" names the same step in other words. So 379 is an upper bound for a manual or model-assisted review, not a count of wrong highlights; the two confirmed cases among the four explorer units show that the true count is not zero.

The better check needs no heuristic: compare the numbered step each language attaches to the id (the emoji step number the annotated lines already carry) and the order of steps, and make "same step numbers in all three languages" a validated property rather than a prompt instruction.

In one breath#

A visualization is a state trace with a data contract, and its correctness has four parts: every state is what the algorithm really holds, every state obeys the algorithm's invariants, the reader can follow it at their own pace and without motion or a mouse, and edge-case inputs produce honest traces. Example B shows that invariants catch impossible windows and miss plausible wrong ones, which only a replay finds. WatchAlgo's September design, a model-written tracer run and checked by code, closed the April gap, and its remaining weak link is visible in its own files: a line id that exists in every language but points at different steps.

The trace is one per test case and shared by every flavor, while the code beside it comes in three languages. Three flavors and three languages separates those two axes, gives each its own contract, and runs one real unit's nine programs on the same fixtures to see what "the same solution" can honestly mean.