soldermaskdocs

Benchmarks

The benchmarks page shows what the shipped models and the pipeline score. It computes nothing: every number was written to disk by the run that measured it.

soldermask/app/benchmarks.py reads those files and hands them to the page. A page load never recomputes a score, so the number cannot drift from the run that produced it.

SourceWritten by
The routability model's scoresroutability.train(), into the model file
Pipeline eval runssoldermask.evaluate, into eval*/summary.json
Human rubrictools/rubric.py --write, into rubric/summary.json
Mendicanttools/mendicant_summary.py --write, into .cache/mendicant_summary.json
Graph judgetools/gnn_summary.py --write, into .cache/gnn_summary.json
Second routertools/router_ab.py --report --save, into .cache/router_ab_summary.json

Benchmarks

What is on it

CardWhat it shows
Routability modelEach candidate model on the same split: the shipped CNN, a logistic fit on the scalar features, and RUDY as the baseline. Routes, boards, clean rate, AUC with its interval, Brier, ECE, accuracy and unrouted error.
Picking a clean placementOf the boards where at least one placement routed clean, how often the top-ranked one was clean. Sorting by wirelength is the comparison.
CalibrationPredicted chance of a clean route against the actual clean rate, with Brier, ECE and the Platt temperature
Ensemble seedsEach ensemble member scored alone on held-out boards: the spread the average hides. It appears only when the model file stores its members' scores, and the shipped model's does not.
Heat headThe copper the network expects in each cell, and the pins it expects to miss, against RUDY's guess
Pipeline eval · LLM board generationOne row per backend per run: pass@1, parts resolved, ERC errors, gaps, routed, fab-readable and seconds
Human rubric · the placer against a personThe placer against the people who drew the boards, on wirelength and decoupling distance
Mendicant · Physarum and the foldTwo unfitted instruments against three baselines. Between-board AUC with a bootstrap interval, computed from the held-out rows, and within-board AUC beside it. The label-ceiling and payoff figures are quoted from paper/mendicant.tex.
A graph judge · GNN vs the CNNThe GNN and the shipped CNN on the same 182 boards, within each board, and whether Mendicant's features move the GNN. Nothing is trained on this page.
Second router · freerouting vs KiCadRoutingToolsTwo independent routes of the same boards, one DRC judging both. Each router's clean rate, and how many of freerouting's failures the other router closed on its own. It measures the corpus's labels, not builds, which route with Lightcycle. Routers has the comparison behind that choice.

Read the two questions separately

A model can be asked two different questions:

  • Between boards: can it tell an easy board from a hard one?
  • Within a board: can it order one board's own anneals?

The placer asks the second question. On this corpus the two have repeatedly had different answers. A model that cannot order one board's anneals cannot help the placer, whatever its held-out AUC.

Every split in the selector above the model table answers the first question. The held-out, annealed only split pools the annealed placements of many boards, so it is still a between-board AUC.

The model file also stores a within-board AUC, which the page does not show. For the shipped model it is 0.593: one AUC for each of 498 held-out boards, over the placements the placer made for it, averaged. It was computed on 14 Sep 2026, when the model was fitted. Routability quotes 0.586 for the same model.

The numbers are from training day

The model card shows the metrics computed when the model was fitted. They are stored in soldermask/assets/routability.json and read back as they are.

Not a fresh evaluation

The stored numbers do not move when the corpus does. Before quoting one, compare meta.trained and meta.n_train with what soldermask learn status reports now. Then read the correction block at the top of Routability, where a figure gets withdrawn.

The shipped model was fitted on 14 Sep 2026 on 41,777 placements. Its stored AUC is 0.916 held out, 0.917 among annealed placements, and 0.914 on PCBench. PCBench is held out by content, so its copies in the training bank were kept out.

Reading the pipeline runs

  • The per-tier line. Under each pass@1 figure, the card breaks the score down by prompt tier, from by_tier in summary.json. A pooled figure mostly reflects how the suite is weighted, so read the tiers.
  • No per-tier line means the run predates a fix to the evaluator. Those runs used an older build path with no pour, no board growth and no precedent. They are not comparable to newer runs.
  • was … · rescored under a figure means the run was scored again under the current pass rule. Hover it to see which boards changed. python -m soldermask.evaluate --rescore does this from the gaps each run recorded.
  • A replay (--designs <run>/designs) builds the same saved designs with no model call. A change between two replays is a change in the code.
  • Two pass@1 figures can differ in their denominator. A prompt that never produced a design counts as a failure in the pool of 60. summary.json divides by the designs it could score.

The run-by-run numbers, and what each fix moved, are in the CHANGELOG.

An empty card is normal

Most sources are missing from most checkouts. A card without its data says so, rather than showing a zero.

CardNeeds
Pipeline evaleval* directories at the repository root. They are gitignored, so a deployed checkout usually has none.
Human rubricThe boards it grades, which are other people's to distribute
Mendicant, second routerLarge research caches in .cache/, which are not shipped
Graph judgeA trained GNN (torch) and the graph corpus, from one experiment

From a script

GET /api/benchmarks returns the same data as JSON: routability, pipeline.runs, rubric, router_ab, mendicant and gnn. Each report except the runs carries an available flag. See the HTTP API.

On this page