The reliability bottleneck
In mid-2026, the AI narrative shifted from scaling models to improving harnesses:
- Alex Karp (Palantir CEO) critiques "tokenmaxxing," where enterprises pay for massive context windows that return little value. He argues for structured data layers over replaceable models.
- Aravind Srinivas (Perplexity CEO) frames AI orchestration as a team-composition problem: "You don't build a homogenous group where everyone has the same skills. You build a team with diverse strengths. We're applying that same logic to AI workflows."
- Andrej Karpathy (Anthropic) frames agents as large language models in a loop, noting that reliability takes exponentially more work. His agent experiment, autoresearch, edits one file and tests it against a deterministic fitness signal to keep the surface small.
- Sakana AI recently shipped Fugu, a coordinator model that dynamically routes subtasks to specialized Thinker, Worker, and Verifier models. Like a structured harness, this constrains each participant's decision space rather than asking a single model to do everything.
The Experiment
This is an experiment to see if I can build a minimal agentic harness using Ollama and LangChain to see if I can improve small model reliability on coding tasks.
If this works, it means that investing in better agent harnesses is a good bet.
Test Setup
I built several harnesses for coding: One raw prompt, one using a standard agent (Aider), and 3 variants of my own harnesses to test different assumptions.
I evaluated each harness on larger and larger files (100, 1000, 10000 lines) on simple coding tasks to measure when and how they failed.
The model presented here is Gemma4:latest, but I also had similar results with Ornith, Qwen2.5-Coder and Llama3.1.
Raw Prompt (Control)
A naive baseline harness that pastes the entire source file into the context and expects the rewritten file in a single turn. No tools, no loop.
# Entire file sent in a single query
prompt = f"Code:\n{source_code}\nTask: {task}\nReturn the entire rewritten file."
new_source = llm.invoke(prompt).content
Standard Agent (Aider)
An off-the-shelf multi-turn harness that receives raw source files and edits them using search-and-replace blocks. This is how the Aider CLI and other coding agents work under the hood.
# Multi-turn loop using search-and-replace tools on raw text
@tool
def replace_block(search_block: str, replace_block: str):
"""Replace search_block with replace_block in the file."""
agent = create_react_agent(llm, tools=[replace_block])
One-Shot Structured (My Custom Test)
My first test was seeing if passing a function signature in addition to the source code helped.
# Single-turn query using structured abstract syntax tree signatures and target source
signatures = [f"{n}: {fn['sig']}" for n, fn in index.items()]
prompt = f"Signatures:\n{signatures}\nTarget Source:\n{index[target]['source']}\nTask: {task}"
new_function = llm.invoke(prompt).content
Raw Line-Editing Loop (My Custom Test)
My second test was seeing if passing a line range to the agent helped. This agent was given the ability to run for multiple turns to control for loops, but not any function metadata.
# Multi-turn loop using raw line-number range edits
@tool
def edit_lines(start: int, end: int, new_lines: str):
"""Replace lines from start to end with new_lines."""
agent = create_react_agent(llm, tools=[edit_lines])
Structured Specialist Harness (My Proposed Solution)
Finally, my proposed solution: a multi-turn agent which is given function signatures as well as deterministic helpers to find and edit methods. This makes the model singly responsible for code editing and offloads the grunt work of file navigation and syntax checking to the harness.
# Multi-turn loop with abstract syntax tree signatures, target source, and abstract syntax tree syntax validation gate
@tool
def edit_function(name: str, new_source: str) -> str:
"""Replace function 'name' with 'new_source'. Enforces syntax compile check."""
ast.parse(new_source) # syntax gate
# updates file source and rebuilds function map index
agent = create_react_agent(llm, tools=[list_functions, get_function, edit_function])
Detailed Experimental Results
Single-Function Failure at Scale
I evaluated single-function edits across varying file sizes (100, 1,000, and 10,000 lines) and found that while raw-text setups degrade at 1,000 lines and collapse entirely at 10,000 lines, the structured harnesses maintain a success rate of 75% or higher.
Setup Performance Breakdown
By mapping correctness rates across all setups, I confirmed that the degradation pattern generalizes: at 10,000 lines, raw-text setups fell to 0% correctness, while the Structured Specialist Harness held strong at 75%.
Statistical Validity
The Wilson score intervals is a statistical tool used to calculate a "confidence interval" for a percentage or proportion.
Because these two ranges do not overlap at all, we can definitively say that the Structured Harness is better.
Failure Taxonomy
I evaluate performance using Python's ast module and file comparisons to measure correctness, categorize errors, and detect hallucinations. Correctness is verified by ensuring the intended logic—such as adding guard clauses or deleting functions—is applied accurately while keeping all other AST nodes untouched.
Failures are grouped into specific buckets: syntax errors (broken_syntax), out-of-bounds modifications (wrong_function or scope_creep), JSON formatting issues, turn-limit timeouts, and hallucinations. Hallucinations are strictly measured by extracting all function calls from the transcript and verifying them against a whitelist of builtins, local identifiers, and tool names, counting any unrecognized symbol as an invented one.
As the file size scales up, I noticed failures shift from minor formatting slips into complete syntactic and logical collapse. At 10,000 lines, the naive setups completely fall apart, failing primarily due to syntax errors and hallucinations. In contrast, the Structured Specialist Harness completely avoids these syntax, scope-creep, and wrong-function errors, only occasionally struggling with naming precision.
Latency and Token Costs
I found that the structured setup was much faster, but it burns through a lot more tokens. For example, just listing out all the functions in a large 10,000-line file used up 16,000 tokens in a single turn. Because of this, as your files get larger, this method ends up needing just as much context space as Aider.
Conclusion
Is it possible that a better harness can make small models reliable? Yes it is.
Get Started Prompt
Show full prompt
# Representation vs Reliability: Experiment Design
Version: 2.0 (2026-07-06)
Author: Nicholas Ventimiglia
## 1. Context
Goal: Build a minimal, reproducible Python experiment that measures how structured code representations affect small-model reliability on code editing tasks at three file sizes, and produces the charts used in the blog post.
### Product goals:
- A single-model sweep across 5 conditions × 3 sizes × 4 tasks × 3 repeats completes in under 3 hours on consumer hardware.
- All blog charts are produced by a single `python report.py` command.
- Corpus is deterministic (seeded, Secure Hash Algorithm (SHA-256) verified) - any researcher gets identical inputs.
- Runner is resumable: re-running skips scenario IDs already in `JSON Lines (JSONL) results file`.
- Scoring is deterministic and model-free (stdlib `ast` only - no large language model (LLM) calls after the run).
- All tunables live in `config.json`; no hardcoded paths, no CLI flags needed.
### Non-product goals:
- No Phase 2 (cross-method propagation, dependency corpus, `harness_cg`).
- No multi-model orchestration; one model per run.
- No cloud inference; Ollama local only.
- No real codebases; synthetic corpus only.
- No database or web user interface (UI).
---
## 2. General Design
| Module | Responsibility |
|--------|----------------|
| `config.json` | All tunables: model, conditions, sizes, repeats, paths. |
| `gen_corpus.py` | Generate and freeze the synthetic Python corpus. |
| `tasks.json` | Four instruction templates (find, insert, edit, delete). |
| `tools.py` | standard library abstract syntax tree (AST) tools: `list_functions`, `get_function`, `apply_edit`; raw line tools: `find_text`, `read_lines`, `replace_lines`. |
| `runner.py` | Sequential, resumable sweep; dispatches to condition handlers. |
| `score.py` | Deterministic AST scorer; pure function of disk artifacts. |
| `report.py` | `JSON Lines (JSONL) results file` → `report.md` + all blog charts. |
The 2×2 condition grid is the experiment's core claim:
| | No tool loop | Tool loop |
|-------------------|-------------------|------------------------------------|
| **No structure** | Raw Prompt | Raw Line-Editing Loop |
| **Structure** | One-Shot Structured | Structured Specialist Harness |
Standard Agent (Aider) is a fifth condition (off-the-shelf baseline). It is included by default - set `"aider_exe": null` in `config.json` to skip it.
### Config Design
```jsonc
{
"model": "gemma4:latest",
"ollama_url": "http://localhost:11434",
"conditions": [
"control", // Raw Prompt
"structure", // One-Shot Structured
"tools_raw", // Raw Line-Editing Loop
"aider", // Standard Agent (Aider)
"harness" // Structured Specialist Harness
],
"sizes": [100, 1000, 10000],
"repeats": 3,
"max_turns": 10,
"aider_exe": "path/to/aider",
"num_ctx": {"100": 8192, "1000": 32768, "10000": 131072}
}
```
### Multi-model generalization
The blog also reports results for a second model (Ornith 1.0 9B Dense) run under a subset of conditions (Raw Prompt, Aider, Harness only) to test whether the collapse-vs-hold patterns generalize. To reproduce this, run `runner.py` twice with different `model` values in `config.json` (changing `conditions` to the subset for the second run). `report.py` aggregates all records in `JSON Lines (JSONL) results file` and splits charts by model automatically.
---
## 3. Implementation Plan
```
/
config.json # All tunables. Edit this; do not edit scripts.
tasks.json # T1 to T4 instruction templates.
gen_corpus.py # Run once to produce and freeze the corpus.
tools.py # AST and raw-line tool implementations.
runner.py # Entry point: reads config.json, runs sweep.
score.py # Deterministic scorer; no model calls.
report.py # Generates report.md + all blog charts.
requirements.txt
corpus/
f100.py # 100-line seeded synthetic file (frozen).
f1000.py # 1,000-line seeded synthetic file (frozen).
f10000.py # 10,000-line seeded synthetic file (frozen).
manifest.json # SHA-256 hashes; runner aborts on mismatch.
runs/
JSON Lines (JSONL) results file # One JSON record per scenario (append-only).
report.md # Generated summary tables.
transcripts/ # Full conversation logs per scenario.
charts/ # Generated PNG charts.
```
---
## 4. Module Detail
### gen_corpus.py
Generates three synthetic Python files (100, 1,000, 10,000 lines) using a fixed random seed. All functions are top-level, no imports, no external calls. After generation, writes `corpus/manifest.json` with SHA-256 hashes and `corpus/f{size}.symbols.json` with ground-truth metadata. `runner.py` verifies the manifest before every sweep and aborts on mismatch. Idempotent.
**Key decisions:**
- **Seed per size**: `seed = 42 + size` (i.e., 142, 1042, 10042). Different seeds so each file has a distinct function set.
- **Target signature** is fixed across all files: `process_records_{size}(records: list, limit: int, strict: bool) -> dict`. The scorer and tasks rely on this exact shape.
- **Target placement**: inserted at a random index between 40% and 60% through the function list - not always last. This tests that the model navigates by name, not by position.
- **`corpus/f{size}.symbols.json`**: written alongside each `.py`. Contains `{functions, identifiers, target, sha256}`. `score.py` reads this for ground-truth line numbers and the hallucination whitelist.
```python
def generate(size: int, seed: int) -> str:
# random.seed(seed); build function pool from VERBS x NOUNS
# inject process_records_{size} at a random index 40-60% through the list
return source
def write_manifest(corpus_dir: Path) -> None:
# sha256 each .py -> corpus/manifest.json
# also writes corpus/f{size}.symbols.json (ground truth for scorer)
```
### tools.py
All functions are pure over a file path. The file is re-read and re-parsed on every call; no line numbers are ever cached between calls.
**Structural tools** (used by `harness`):
```python
def list_functions(file: str) -> str:
# Shape header + JSON array of {name, signature, start_line, end_line}
# for every top-level FunctionDef.
def get_function(file: str, name: str) -> str:
# Exact source of the named function.
def apply_edit(file: str, op: str, name: str, body: str | None) -> str:
# op: "replace" | "insert_after" | "delete"
# Name-anchored: re-parses on every call, resolves name -> lines at apply time.
# Two-stage syntax gate before writing:
# 1. ast.parse(body): body must parse and be exactly one FunctionDef.
# 2. ast.parse(new_source): full file must parse after the splice.
# Either failure returns error JSON without writing; model receives it and retries.
# Returns: {"ok": bool, "error": str|null, "new_start_line": int, "new_end_line": int}
```
**Raw line tools** (used by `tools_raw`):
```python
def find_text(file: str, pattern: str) -> str:
# Line numbers and text of lines containing the substring.
def read_lines(file: str, start: int, end: int) -> str:
# Lines [start, end] (1-based, inclusive), each prefixed with its number.
def replace_lines(file: str, start: int, end: int, text: str) -> str:
# Overwrites lines [start, end] with text. No AST validation.
# Returns: {"ok": bool, "error": str|null}
```
### runner.py
Reads `config.json`. Builds a flat scenario list (task × size × condition × repeat), sorted alphabetically by ID. Copies the corpus file to a fresh per-scenario workdir before each run so conditions are isolated. Appends one JSONL record to `runs/JSON Lines (JSONL) results file` after scoring. Skips scenario IDs already present.
**Scenario ID format**: `{task}-{size}-{condition}-r{repeat}`, for example: `T3-10000-harness-r1`. Alphabetical sort gives a deterministic, human-readable run order.
**Workdir**: `runs/work/{scenario_id}/` (wiped and recreated for each scenario). Contains `target.py` (a fresh copy of the corpus file). Preserved after the run for post-hoc inspection.
**Condition handlers:**
| Setup | Code name | Turns | Tools | Representation |
|-------|-----------|-------|-------|----------------|
| Raw Prompt | `control` | 1 | none | Full raw source in user message |
| One-Shot Structured | `structure` | 1 | none | AST map + target source; one JSON edit spec applied once, no retry |
| Raw Line-Editing Loop | `tools_raw` | multi (`max_turns`) | `find_text`, `read_lines`, `replace_lines` | None: raw line numbers |
| Structured Specialist Harness | `harness` | multi (`max_turns`) | `list_functions`, `get_function`, `apply_edit` | AST function map + syntax gate on write |
| Standard Agent (Aider) | `aider` | multi (subprocess) | aider search-replace | Raw text (silently returns SKIPPED record when `aider_exe` is null) |
**Raw Prompt output repair**: if the model wraps its entire response in a single ``` fence pair, that pair is stripped before writing. Only a single outer fence is removed (no recursive stripping). This is the only post-processing applied to model output.
**Scenario record** (appended to `JSON Lines (JSONL) results file` after scoring):
```jsonc
{
"scenario_id": "T3-10000-harness-r1",
"task": "T3",
"file_lines": 10000,
"condition": "harness",
"repeat": 1,
"model": "gemma4:latest",
"target": "process_records_10000",
"correct": true,
"localized": true,
"hallucinated": false,
"failure_bucket": "none", // "none" = correct AND localized AND not hallucinated
"latency_ms": 38200,
"tokens_in": 16400,
"tokens_out": 420,
"tool_calls": 3,
"ops_valid": 2, // edit tool calls that passed schema validation
"ops_total": 2 // total edit tool calls attempted
}
```
### score.py
Pure function of disk artifacts. Reads `corpus/f{size}.py` (original) and `corpus/f{size}.symbols.json` (ground truth + whitelist). Reads `runs/work/{scenario_id}/target.py` (edited file). No model calls.
**Correctness oracle per task:**
- `T1` (find): JSON extracted from `final_text` must match `name`, `signature` (whitespace-normalized), `start_line`, and `end_line` from AST ground truth. All four fields must match.
- `T2` (insert): `validate_{target}_input` must exist immediately after target in the function list, and must contain `if param is None: raise ValueError(...)` anywhere in its body (`ast.walk` - not just the first statement).
- `T3` (edit): first statement after any docstring must be `if param is None: raise ValueError(...)`, AND the rest of the body must be unchanged (`ast.unparse` comparison). Adding the guard while also modifying other lines is a failure.
- `T4` (delete): target absent; all pre-existing function names (minus target) still present - extras are allowed.
**Localization:** `difflib` line diff on the raw source. Any changed line outside the target's original range ±1 is flagged. For T2, the allowed range extends to cover the inserted function's line count plus one blank separator line.
**Hallucination:** call-shaped regex `\b([a-z_][a-z0-9_]{3,})\(` applied to `transcript + final_text` combined (not just the edited file; tool call arguments are also scanned). Any match not in the whitelist is an invented symbol. Whitelist = builtins + original file identifiers (from `symbols.json`) + tool names + `validate_{target}_input` + names the model itself defined in the edited file.
**`clean_success`** (`failure_bucket == "none"`) = correct AND localized AND not hallucinated. This is stricter than `correct` alone: a run that edits the right function but also touches other code is `correct=true` but `failure_bucket=wrong_function`.
**Failure buckets** (mutually exclusive, first-match wins):
`wrong_function` → `invented_symbol` → `broken_syntax` → `scope_creep` → `format_error` → `gave_up` → `none`
`wrong_function` is checked first: a non-target modification is a structural failure even if the file parses correctly. `format_error` is the catch-all for "parseable and localized but the task predicate is unmet."
### report.py
Reads `runs/JSON Lines (JSONL) results file`. Produces `runs/report.md` and all blog charts in a single run. Derives model name(s) from records, avoiding hardcoded model names. When multiple models are present, splits per-model charts and auto-generates the aggregate. Uses matplotlib with a minimal style (no spines, `#fcfcfb` background, light grid).
**Model slug**: `:`, `/`, `.` → `_` (e.g. `gemma4:latest` → `gemma4_latest`). Used in all output filenames.
**`failures_{model}.png`** uses only 10,000-line records (the collapse pattern is most pronounced at scale and all five setups are always present there.
**Wilson CI** is implemented inline with z = 1.96 (95%). No external stats library required.
| File | Description |
|------|-------------|
| `report.md` | Markdown tables: correct / localized / clean success / hallucination rates by setup and file size |
| `charts/model_{model}.png` | Line chart: correctness vs file size (log scale), one line per setup |
| `charts/model_Aggregate.png` | Same, all models combined (only generated when >1 model present) |
| `charts/setup_{condition}.png` | Per-setup slope chart (one file per condition) |
| `clean_success_{model}.png` | Grouped bar: clean success rate at each file size, one group per setup |
| `failures_{model}.png` | Grouped bar: failure bucket counts at 10,000 lines, one group per setup |
| `ci_10k_{model}.png` | Horizontal bar: Wilson 95% CI for clean success at 10,000 lines |
| `latency_{model}.png` | Two-panel: median latency (s) and median total tokens, by setup and file size |
---
## 5. CLI / Entry Points
```bash
# Step 1: install dependencies
pip install langchain>=0.3,<0.4 langchain-ollama>=0.2,<0.3 matplotlib>=3.8
# To install Aider support: (only needed if aider_exe is set in config.json)
# Step 2: pull the model
ollama pull gemma4:latest
# Step 3: generate and freeze corpus (run once)
python gen_corpus.py
# Step 4: run sweep (resumable, re-running skips completed scenarios)
python runner.py
# Step 5: generate report and all charts
python report.py
# To run a second model for generalization (e.g., Ornith):
# 1. Edit config.json: set "model" and narrow "conditions"
# 2. Re-run runner.py (new records append to JSON Lines (JSONL) results file)
# 3. Re-run report.py (splits charts by model, adds Aggregate chart)
```
All tunables are in `config.json`. No flags are needed.
---
## 6. Decisions
| Question | Decision |
|---|---|
| Config file vs CLI flags | **Config file**: all tunables in `config.json`; scripts read it at startup. Makes sharing reproducible without re-reading script internals. |
| Corpus format | Synthetic Python, no imports, no external calls. Removes real-codebase dependency; makes scoring fully deterministic. |
| Scoring oracle | `stdlib ast` only. No LLM calls post-run; results are reproducible without re-running the model. |
| Line number stability | `apply_edit` re-parses on every call; never stores coordinates across calls. Prevents stale-line bugs on multi-edit runs. |
| Two-stage syntax gate | `apply_edit` checks `ast.parse(body)` (body is a single FunctionDef) then `ast.parse(new_source)` (whole file). Either failure returns an error for retry rather than silently writing broken code. |
| All charts in `report.py` | Single command produces everything. No separate supplemental scripts. Chart names are model-derived, not hardcoded. Aggregate chart auto-generated when multiple models are present. |
| Aider | Included as a condition by default. Set `"aider_exe": null` to skip. Aider requires an external binary and `git init` per workdir. |
| Repeatability | 3 repeats per cell (n=12 at 4 tasks per model). Wilson CI at 10k lines is ±22pp. Sufficient to distinguish structured vs. raw-text tiers; insufficient to rank models within a tier. |
Open:
- Whether to include T1 (find, no edit): it measures navigation accuracy independently but has no chart impact. Can be excluded by removing T1 from `tasks.json`.
- Whether to ship a `Dockerfile` with Ollama + Gemma 4 pre-loaded to eliminate "works on my machine" setup problems.
---
## 7. Appendix: AI Build Prompt
Paste the following into an AI coding assistant to build this experiment from scratch.
```
Build a Python experiment measuring how code representation affects small-model
reliability on code editing tasks. Use LangChain with ChatOllama as the LLM backend.
Read all tunables from config.json at startup. No hardcoded paths. No command-line interface (CLI) flags.
## config.json
{
"model": "gemma4:latest",
"ollama_url": "http://localhost:11434",
"conditions": ["control", "structure", "tools_raw", "aider", "harness"],
"sizes": [100, 1000, 10000],
"repeats": 3,
"max_turns": 10,
"aider_exe": "path/to/aider",
"num_ctx": {"100": 8192, "1000": 32768, "10000": 131072}
}
## tasks.json
{
"T1": "Report name, signature, and inclusive line range of '{target}'. Answer as JSON: {\"name\":..., \"signature\":..., \"start_line\":..., \"end_line\":...}",
"T2": "Add validate_{target}_input immediately after '{target}'. Same params, raises ValueError if first param is None, returns True otherwise, one-line docstring.",
"T3": "In '{target}', add a guard at the top that raises ValueError if its first parameter is None.",
"T4": "Remove '{target}' entirely."
}
Target: process_records_{size} where size is 100, 1000, or 10000.
## gen_corpus.py
Generate three synthetic Python files of exactly 100, 1,000, and 10,000 lines.
Rules: top-level functions only, no imports, no external calls, fixed random seed.
One function per file named process_records_{size}. After generation, write a
SHA-256 manifest to corpus/manifest.json. runner.py verifies it before every sweep
and aborts on mismatch.
## tools.py (stdlib ast only)
Structural tools:
list_functions(file): shape header + JSON array of {name, signature, start_line, end_line}
get_function(file, name): exact source of the named function
apply_edit(file, op, name, body): name-anchored edit with two-stage syntax gate:
1. ast.parse(body) and verify body is exactly one FunctionDef.
2. ast.parse(new_source) after applying the splice.
Either failure returns error JSON without writing; model retries.
op: "replace" | "insert_after" | "delete"
Returns {"ok": bool, "error": str|null, "new_start_line": int, "new_end_line": int}
Raw line tools (tools_raw condition):
find_text(file, pattern): line numbers + text of matching lines
read_lines(file, start, end): lines [start, end] prefixed with numbers
replace_lines(file, start, end, text): replace range, no AST validation
## runner.py
Load config.json. Build scenario list: task x size x condition x repeat. Sort model-major.
Copy corpus file to temp workdir before each run. Append one JSONL record to
runs/JSON Lines (JSONL) results file after scoring. Skip scenario IDs already in the file.
Condition handlers:
control: single turn; full source in user message; write response back as file.
structure: single turn; AST map + target source; parse one JSON {op, name, body}
and call apply_edit once; no retry.
tools_raw: multi-turn loop (max_turns); tools: find_text, read_lines, replace_lines.
No AST validation on write.
harness: multi-turn loop; tools: list_functions, get_function, apply_edit.
apply_edit error JSON is returned as tool result; model retries.
aider: subprocess (skip if aider_exe is null in config).
## score.py
Pure function of disk artifacts; no model calls.
correct: AST predicate per task:
T1: JSON answer matches actual name/sig/lines.
T2: validate_{target}_input exists after target; raises ValueError on None first param.
T3: first statement after any docstring is "if param is None: raise ValueError(...)".
T4: target absent; all other pre-existing names still present.
localized: no line outside target original range (+-1) changed (difflib). For T2,
range extends to include the inserted function.
hallucinated: call-shaped token not in builtins + original file symbols + tool names.
failure_bucket (first match wins):
wrong_function -> invented_symbol -> broken_syntax -> scope_creep -> format_error -> gave_up -> none
## report.py
Load config.json for model name. Read runs/JSON Lines (JSONL) results file. Produce all outputs in one run.
Derive model name(s) from records (do not hardcode). When multiple models are present,
split per-model charts and produce a combined Aggregate chart.
Use matplotlib, #fcfcfb background, no spines, light grid.
Outputs:
report.md: markdown tables
charts/model_{model}.png: correctness vs file size, one line per condition
charts/model_Aggregate.png: same, all models combined
charts/setup_{cond}.png: per-condition slope chart
clean_success_{model}.png: grouped bar: clean success rate vs file size
failures_{model}.png: failure buckets at 10,000 lines
ci_10k_{model}.png: Wilson 95% CI for clean success at 10,000 lines
latency_{model}.png: median latency and total tokens by condition/size
```
All views are my own, and I do not represent any employer. All ownership of attached open source code is waved.