Benchmarks
The benchmarks page shows what the shipped models and the pipeline score. It computes nothing: every number was written to disk by the run that measured it.
soldermask/app/benchmarks.py reads those files and hands them to the page. A
page load never recomputes a score, so the number cannot drift from the run
that produced it.
| Source | Written by |
|---|---|
| The routability model's scores | routability.train(), into the model file |
| Pipeline eval runs | soldermask.evaluate, into eval*/summary.json |
| Human rubric | tools/rubric.py --write, into rubric/summary.json |
| Mendicant | tools/mendicant_summary.py --write, into .cache/mendicant_summary.json |
| Graph judge | tools/gnn_summary.py --write, into .cache/gnn_summary.json |
| Second router | tools/router_ab.py --report --save, into .cache/router_ab_summary.json |

What is on it
| Card | What it shows |
|---|---|
| Routability model | Each candidate model on the same split: the shipped CNN, a logistic fit on the scalar features, and RUDY as the baseline. Routes, boards, clean rate, AUC with its interval, Brier, ECE, accuracy and unrouted error. |
| Picking a clean placement | Of the boards where at least one placement routed clean, how often the top-ranked one was clean. Sorting by wirelength is the comparison. |
| Calibration | Predicted chance of a clean route against the actual clean rate, with Brier, ECE and the Platt temperature |
| Ensemble seeds | Each ensemble member scored alone on held-out boards: the spread the average hides. It appears only when the model file stores its members' scores, and the shipped model's does not. |
| Heat head | The copper the network expects in each cell, and the pins it expects to miss, against RUDY's guess |
| Pipeline eval · LLM board generation | One row per backend per run: pass@1, parts resolved, ERC errors, gaps, routed, fab-readable and seconds |
| Human rubric · the placer against a person | The placer against the people who drew the boards, on wirelength and decoupling distance |
| Mendicant · Physarum and the fold | Two unfitted instruments against three baselines. Between-board AUC with a bootstrap interval, computed from the held-out rows, and within-board AUC beside it. The label-ceiling and payoff figures are quoted from paper/mendicant.tex. |
| A graph judge · GNN vs the CNN | The GNN and the shipped CNN on the same 182 boards, within each board, and whether Mendicant's features move the GNN. Nothing is trained on this page. |
| Second router · freerouting vs KiCadRoutingTools | Two independent routes of the same boards, one DRC judging both. Each router's clean rate, and how many of freerouting's failures the other router closed on its own. It measures the corpus's labels, not builds, which route with Lightcycle. Routers has the comparison behind that choice. |
Read the two questions separately
A model can be asked two different questions:
- Between boards: can it tell an easy board from a hard one?
- Within a board: can it order one board's own anneals?
The placer asks the second question. On this corpus the two have repeatedly had different answers. A model that cannot order one board's anneals cannot help the placer, whatever its held-out AUC.
Every split in the selector above the model table answers the first question. The held-out, annealed only split pools the annealed placements of many boards, so it is still a between-board AUC.
The model file also stores a within-board AUC, which the page does not show. For the shipped model it is 0.593: one AUC for each of 498 held-out boards, over the placements the placer made for it, averaged. It was computed on 14 Sep 2026, when the model was fitted. Routability quotes 0.586 for the same model.
The numbers are from training day
The model card shows the metrics computed when the model was fitted. They are
stored in soldermask/assets/routability.json and read back as they are.
Not a fresh evaluation
The stored numbers do not move when the corpus does. Before quoting one,
compare meta.trained and meta.n_train with what soldermask learn status
reports now. Then read the correction block at the top of
Routability, where a figure gets withdrawn.
The shipped model was fitted on 14 Sep 2026 on 41,777 placements. Its stored AUC is 0.916 held out, 0.917 among annealed placements, and 0.914 on PCBench. PCBench is held out by content, so its copies in the training bank were kept out.
Reading the pipeline runs
- The per-tier line. Under each pass@1 figure, the card breaks the score down by prompt tier, from
by_tierinsummary.json. A pooled figure mostly reflects how the suite is weighted, so read the tiers. - No per-tier line means the run predates a fix to the evaluator. Those runs used an older build path with no pour, no board growth and no precedent. They are not comparable to newer runs.
was … · rescoredunder a figure means the run was scored again under the current pass rule. Hover it to see which boards changed.python -m soldermask.evaluate --rescoredoes this from the gaps each run recorded.- A replay (
--designs <run>/designs) builds the same saved designs with no model call. A change between two replays is a change in the code. - Two pass@1 figures can differ in their denominator. A prompt that never produced a design counts as a failure in the pool of 60.
summary.jsondivides by the designs it could score.
The run-by-run numbers, and what each fix moved, are in the CHANGELOG.
An empty card is normal
Most sources are missing from most checkouts. A card without its data says so, rather than showing a zero.
| Card | Needs |
|---|---|
| Pipeline eval | eval* directories at the repository root. They are gitignored, so a deployed checkout usually has none. |
| Human rubric | The boards it grades, which are other people's to distribute |
| Mendicant, second router | Large research caches in .cache/, which are not shipped |
| Graph judge | A trained GNN (torch) and the graph corpus, from one experiment |
From a script
GET /api/benchmarks returns the same data as JSON: routability,
pipeline.runs, rubric, router_ab, mendicant and gnn. Each report except
the runs carries an available flag. See the HTTP API.