Repair loop and evaluation

Repair: feeding the findings backCopy Anchor Link

Every stage after the model produces machine-readable ground truth — which parts are not real, which pins do not exist, which nets are wrong — and a single-shot pipeline throws it all away. --repair N sends it back: the design as it stands plus a brief of the gaps and ERC findings, asking the model for a revised design, up to N times. The loop stops before the router (the expensive stage the pipeline already passes) and stops early once the kept board is clean — every part resolved, no ERC errors, nothing dropped.

The thing to be careful about is that repair does not become laundering. The cheapest way to clear R3 did not resolve is to delete R3, and that scores better on every naive metric while producing a worse board. So the score weights a dropped part above an unresolved one:

Lower is better; each round revises the current best board rather than the latest one, so a bad round is a dead end rather than something to dig out of; ties go to the earlier round. The kept board, the gain over the single shot, tokens spent, and the parts lost are recorded (repair.json next to the build), and the command line's honest section says repair: n part(s) present in the first design are gone from this one — check they were not needed. A network outage in the parts search is deliberately not shown to the model: it is our failure, not the design's, and it must never cost a good part.

The wiring turn: the model wires the parts it can seeCopy Anchor Link

Repair sends the findings back and asks for the whole design again. The wiring turn (wire.py, --wire) is narrower and comes first: the model's draft names an IC's pins from memory and never sees the part, so it writes VM for a DRV8825 whose pads are VMA and VMB, or IO20 for a module that does not bring it out, and the resolver can only refuse the connection. After the parts resolve, the model is shown each part as it will arrive — the catalogue row and the exact names its pins answer to, the same list the parts palette shows a person — and asked for the nets against those names. The output schema has one property a part and one a pin, with nothing else allowed, so a pin the part does not have cannot be written; a backend that will not take a schema that size gets a flat list and the same rule is applied here, with every refused name said in unsupported. A part the model says is not what the design needs (a shift register ordered for a level shifter) goes back with a better search, the rejected code is kept off that part's candidates, and the turn is taken once more, which is the bound. A net the turn can only put one pin on is kept rather than dropped, so the build reports it and the verdict counts it: the turn cannot tidy a defect out of the question. A failed call leaves the draft's wiring; the board is still built.

Evaluation: pass@1 on this taskCopy Anchor Link

python -m soldermask.evaluate scores a backend on this task rather than a leaderboard. The pipeline grades itself — resolve says whether the parts are real, ERC whether the circuit is sane, freerouting whether it can be built — so pass@1 means a board that resolved completely, kept every net and every connection the design asked for, passed ERC with no errors, routed with no unrouted nets, and passed DRC once the ground was poured. The same passes() rule is shared by the repair loop, the evaluator and the by-hand driver, so a board cannot pass by one definition and fail by another.

python -m soldermask.evaluate --backends gemini            # all sixty, one backend
python -m soldermask.evaluate --backends gemini --n 8      # the original eight
python -m soldermask.evaluate --tier CD                    # the MCU and multi-rail bands
python -m soldermask.evaluate --repair 2                 # paired before/after on the same designs
python -m soldermask.evaluate --designs eval/gemini/designs --repair 0   # replay: deterministic
python -m soldermask.evaluate --designs eval/gemini/designs --wire       # replay + the wiring turn: one model call a board
python -m soldermask.evaluate --rescore eval-2026-09-14 .cache/eval-*     # a stored run re-read under today's rule; builds nothing

--rescore is for the day the rule changes: every row of a stored run is read again under passes() as it is now, from the fields and gaps the row recorded, and the two files are rewritten with the first kept beside them (*.before-rescore) and a rescored: {when, rule, was, moved} record on the summary, so the number first reported stays visible under the corrected one. It rebuilds nothing; a change in the pipeline needs a replay, not a rescore. The first use was 16 Sep 2026, when the rule gained dropped_connections and every run since 11 Sep lost one to three boards.

Sixty fixed prompts in four tiers by part count — A basic (12), B easy (16), C medium (18), D hard (14) — of which the first eight are the original suite in their original order, so --n 8 still runs exactly what the early revisions ran; --tier CD runs a band. summary.json reports per tier as well as pooled, because a pooled number over a suite weighted toward small boards is a number about the weighting. results.json holds every measured field per run — parts asked and resolved, ERC counts, gaps with their text, routing and unrouted nets, BOM cost, seconds, token usage, the Gerber read-back result, and with --repair the base-round figures beside the repaired ones. summary.json aggregates per backend: pass@1, generated, parts_resolved, erc_errors_mean, gaps_mean, routed, seconds_mean, fab_readable, and when repair ran pass@1_base and parts_dropped_total — if that is not about zero, the gain is partly laundering.

Every board is built by workspace.realise(), once — the same call the app and the command line make — and the row is the verdict realise() stamped on it (results.json carries that dict whole under verdict, and a row whose own rule disagrees with it is refused as a harness fault rather than counted). So pass@1 measures the product as shipped: the outline grown to the real footprints, the precedent start, the ground pour and its DRC, the second router, six candidate placements in the routability model's order with the router shown up to three. Until 11 Sep 2026 the evaluator routed through its own copy of the sequence, which had stopped matching on 9 Sep — no pour, no grown outline, no precedent — so pour_errors could never be non-zero there and a pass@1 from it was a number about a pipeline nobody was shipped; tests/test_evaluate.py now counts the realise() calls the way tests/test_cli_build.py does for the CLI. A net whose copper misses a pin is incomplete, not routed, and results.json records how many placements each board needed (route_attempts) and what the model gave the first (p_clean). The Gerbers handed to the independent reader (fab_readable) are the build's own, copper and fills included; they used to be plotted from the placement alone.

Saved designs are named NNN-<board>.json by the prompt's position in the whole suite, so --designs pairs each with the prompt and tier it was written for whatever the names sort to; a directory from before the index is paired by sorted position and says so. Every attempt is written to the corpus like any build's — point SOLDERMASK_CORPUS elsewhere for a run that should not be.

--no-route skips the router and drops routability from the rule rather than scoring every board as unroutable: a harness that quietly reports zero for a run nobody routed is worse than no harness. Two such instrument faults have cost real time on this project, and tests/test_harness.py exists to keep the instrument honest.