Benchmarks
/benchmarks is the page that says what the shipped models actually score, and it computes nothing. Every number on it was written to disk by the run that measured it — routability.train() into the model file, soldermask.evaluate into an eval*/summary.json, tools/rubric.py --write into rubric/summary.json — and soldermask/app/benchmarks.py parses those files and hands them to the page. Nothing is recomputed on a page load, which is the point: a benchmark you can re-derive on request is a benchmark you can accidentally re-derive differently.
What is on it
| Card | What it shows |
|---|---|
| Routability model | Every candidate model — the shipped CNN (ISOnet), a logistic fit on all 23 scalars, and RUDY on four congestion scalars as the baseline — scored on the same split. Routes, boards, clean rate, AUC with its interval, Brier, ECE, accuracy and unrouted MAE. |
| Picking a clean placement | The payoff in the units the pipeline cares about: of the boards where at least one candidate placement routed clean, how often the top-ranked one was clean, against sorting by wirelength. |
| Calibration | Predicted p(clean) against actual clean rate, with Brier, ECE and the Platt temperature. |
| Ensemble seeds | Each ensemble member scored alone, held out — the spread the average smooths over. |
| Heat head | The copper the network expects per cell and the pins it expects to be unreached, against RUDY's guess at the same picture. |
| Pipeline eval · LLM board generation | One row per backend per run: pass@1, parts resolved, ERC errors, gaps, routed, fab-readable, seconds. Since 11 Sep 2026 the pass@1 cell carries a per-tier breakdown underneath it (by_tier in summary.json) — a pooled figure over a suite weighted toward small boards is a number about the weighting, and the 11 Sep run is the one where that mattered: 0.317 pooled is A 9/12, B 8/16, C 1/18, D 1/14. Two more runs replay the same 59 saved designs with no model call (eval-2026-09-13, eval-2026-09-14 in the CHANGELOG) after fixing, in turn, the resolver's pin-name and ERC rules and then the poured-copper writer and the A* router's escalation policy: 0.467 (A 9/11, B 12/16, C 5/18, D 2/14) then 0.517 (A 11/11, B 12/16, C 6/18, D 2/14), read against the same 60-prompt pool the 11 Sep run scored -- one prompt never got a saved design (Gemini returned unparseable JSON for it on the original run) and counts as failing in every replay; summary.json's own pass@1 divides by the 59 it could score instead, 0.475 then 0.525, which is the number the card shows. |
| Human rubric · the placer against a person | The placer against the people who drew the boards, on wirelength and decoupling distance. |
| Mendicant · Physarum and the fold | Two unfitted instruments (a slime-mould negotiation and a placement folded as RNA) against three baselines, between-board AUC with a bootstrap CI computed live off the held-out rows, and within-board AUC beside it -- the paper's own headline is that within-board every reading is weak and even the label's own ceiling is only ~0.84. Read from .cache/mendicant_summary.json, written by python tools/mendicant_summary.py --write; the label-ceiling and payoff figures are quoted from paper/mendicant.tex rather than recomputed. |
| A graph judge · GNN vs the CNN | The GNN and the shipped CNN scored on the identical 182-board within-board population -- a paired comparison, not the CNN's own stored split -- plus whether adding Mendicant's features to the graph moves it (three training objectives, three seeds, all close to zero). Read from .cache/gnn_summary.json, written by python tools/gnn_summary.py --write; nothing is trained or re-inferred by this page or that script, both only collect what tools/gnn_train.py already wrote. |
| Second router · freerouting vs KiCadRoutingTools | One kicad-cli DRC judging two independent routes of the same board: each router's clean rate, and the rescue rate -- of the boards freerouting could not close, how many the second router closed on its own. Not the same question astar.second_opinion() answers in a build (which only tries to finish a net freerouting left open); this is the fuller comparison behind that decision. Read from .cache/router_ab_summary.json, written by python tools/router_ab.py --report --save ... [--also FILE...]. |
The split selector above the model table is the load-bearing control. Held-out is a between-board question — can the model tell an easy board from a hard one — and held-out, annealed only is the within-board question, which is the one the placer actually asks: can it order one board's own anneals. They have repeatedly had different answers on this corpus, and a score that cannot rank a single board's anneals cannot help the placer whatever its AUC. Read the two separately; the page shows them separately for that reason.
The numbers are training-day numbers
The model card's metrics are the ones computed when the model was fitted, stored in soldermask/assets/routability.json and read straight back. They do not move when the corpus does, and they are not a fresh evaluation. The shipped model's stored 0.954 held-out was measured against labels that have since been corrected; judged against the corrected labels it scores 0.937 held-out and 0.923 among anneals. Before quoting anything from this page, check meta.trained and meta.n_train against what soldermask learn status reports for the corpus now, and read the correction block at the top of Routability — that page is where a figure gets withdrawn.
The external, PCBench split carries its own caveat: an audit of board identity found 866 copies of that test set inside the training directory, because freerouting vendors PCBench in its own benchmark fixtures. The external AUC on this page was measured with roughly a twelfth of its test set seen during training.
An empty card is a normal state
Most of these sources are absent from most checkouts, and that is not an error.
- Eval runs are directories
soldermask.evaluatewrote at the repository root. They are gitignored, so a deployed checkout typically has none. - The rubric needs the boards it grades, and those are somebody else's to distribute. Most checkouts will never have scored any.
- Mendicant and the second router need
.cache/physarum_ab_heldout_geo.jsonand.cache/router_ab*.jsonrespectively -- the raw measurement files behind the paper and the corpus's router comparison. Both are large, machine-specific research caches, not shipped with the repository; a checkout without them shows the card's empty state rather than a zero. - The graph judge needs
tools/gnn_train.py's own report files, which need a trained GNN (soldermask/gnn.py,torch) and the graph corpus (tools/graph_corpus.py) -- a research artifact from one afternoon's experiment, not something most checkouts will have run.
The page says so in place rather than showing an empty table or a zero.
Correction, 16 Sep 2026: every pass@1 since 11 Sep was one to three boards high
The pass rule counted a net the design asked for that was not on the board, and not a connection dropped from a net that survived: a net that kept two of its three pins was still a net, routed clean, and passed. Three boards did this in nearly every run -- the rotary encoder breakout (043, its push-switch's two pins never placed), the DRV8825 breakout (036, the driver's motor-supply pin), the mixed-signal ADC (056, whose ADC lost VDD, Vin+, Vin- and SDA and sat on the board wired to nothing) -- and the MIDI board (041, the optocoupler's VE) once its DIN connectors resolved. python -m soldermask.evaluate --rescore read every stored run's rows again under today's rule, from the gaps each row had already recorded, and the page shows the first number under the corrected one (was … · rescored). Corrected: 11 Sep 19 -> 18 of 60; 13 Sep 28 -> 26 of 59; 14 Sep 31 -> 28; 15 Sep control 33 -> 32, wire 40 -> 40 (the wiring turn places the switch's pins by their real names, so its arm was never hollow, and its lead is 8 boards, not 7); 15 Sep evening replay 37 -> 36; 16 Sep catalogue replay 39 -> 37 against a control 37 -> 36; 16 Sep spacing 0.0 38 -> 37 and 0.5 42 -> 40, three gained and none lost, not four. No conclusion below reverses. The figures in the sections below are the ones first reported and are left as written; the table above them is corrected.
A run without a per-tier breakdown predates the fix, not the pipeline
Before 11 Sep 2026 soldermask.evaluate scored boards through its own copy of the build sequence, which had drifted from workspace.realise() on 9 Sep: no pour, no board grown to its real footprints, no precedent placement. Every eval-* directory made before that date shows pass@1 between 0.5 and 0.75 on the eight-prompt suite — a number about the easier, unshipped path, not about the pipeline. The Repair loop and evaluation page and the CHANGELOG carry the fix and the first honest measurement (pass@1 0.317 pooled, by tier A 9/12, B 8/16, C 1/18, D 1/14, on the full sixty prompts). The page marks this itself: a run with no per-tier breakdown under its pass@1 figure is one of the old ones, and the two are not the same question answered twice — they are different rulers, and only the newer one measures the board a user is actually handed.
The 13 and 14 Sep runs are the same ruler, not a new one
The 11 Sep run was the first honest measurement; it was also the one that found, for the first time, exactly what a shipped pipeline drops. Two deterministic replays — soldermask.evaluate --designs eval-2026-09-11/designs, the same 59 saved designs through realise(), no model call, so a change in the number is a change in the code and nothing else — moved it in two steps. 13 Sep, the front half: vendor-decorated pin names (PB0 in (PCINT0/CLKO/ICP1)PB0), inverted spellings, one supply tied across several pads, USB-C positions, a two-pad switch or crystal taking 1/2, and two ERC rules (ic-no-gnd, undriven-rail) that were reading a relay or a switched rail as broken — pass@1 0.317 -> 0.467 (0.475 of the 59 scored), designs surviving resolve and ERC 22 -> 38 of 59. That run also moved the ceiling: eight boards reached the router carrying every net for the first time and failed in the poured copper, one with 294 clearance violations. 14 Sep, the back half: a round pad longer than it is wide had been written as circle, which KiCad reads as a dot the DRC judge, the pour and the A* router (all three read the .kicad_pcb) never saw the real shape of; the A* router's default escalation policy was dropping a via into a pad under the fab's drill floor while this pipeline's reader wrote it back at the rule's size; and a USB-C footprint's own unconnected touching pads were counted as a clearance violation and a solder-mask bridge on every board that carried one. Fixed, the same 59 designs replay clean by two more: pass@1 0.467 -> 0.517 (0.525 of 59), poured-copper DRC errors 868 -> 135, boards routed clean back to the 11 Sep figure (42) while carrying the nets the front half now keeps. Both runs are in the CHANGELOG in full, with the DRC kinds counted and what is left named: freerouting narrowing a fine-pitch entry stub under the fab's track floor, and the A* router leaving a net open on 0.5 mm pitch rather than closing it with an illegal via.
15 Sep has three rows, and they are two questions. eval-2026-09-15-control and eval-2026-09-15-wire are the morning's pair: the same 59 designs replayed with and without the wiring turn (--wire, one model call a board, the only call on a replay), 33 against 40 of 59, the turn's effect on the pipeline as it stood at nine that morning. eval-2026-09-15-control4 is the evening's replay of the same designs with no model call, after the catalogue and router work of that day: pass@1 0.559 -> 0.627 (37 of 59), four boards gained and none lost, the run 58 minutes to 30, the median board 16 s to 7, the three-attempt boards a median 148 s to 68. The two questions are separable and were not added: the wiring turn has not been replayed on the evening's pipeline. Three replays sit between the morning's control and this row in .cache/ and are not shown here: the first (34 of 59) failed two boards the walk of the session file called short of a pin that KiCad's connectivity finds reached, the second (39) passed two boards with a ground pin the pour never reached, which nothing had ever counted, and each was a rule fixed before the next; the number here is the one where every passing board was then judged again by an independent DRC and found clean, with no copper error and nothing unconnected. The seconds_mean column across days is not one ruler either: the evaluator's anneals ran in the loop until 15 Sep and one a core after, and the morning's builds waited up to 30 s on a parts search that never answered.
Getting at it from a script
GET /api/benchmarks returns the same three reports as JSON — routability, pipeline.runs and rubric, the first and last carrying an available flag. See the HTTP API.