Repair loop and evaluation
The pipeline can send its own findings back to the model, and a harness scores the whole pipeline. All of them judge a board by the same pass rule.
The pass rule
repair.passes() is the one rule. A board passes when:
- every part resolved to a real part the pipeline can draw
- no net and no connection the design asked for was dropped
- the ERC ran and found no errors
- every net routed, with no pin short
- KiCad's DRC judged the routed copper and found no errors, including after the pour
- no run of copper narrower than its net asks reaches a 100 °C rise at the net's declared current
ERC warnings do not fail a board. A module's unused pins are floating, and that is normal. An out-of-stock part still resolves; stock is reported as a note.
Datasheet layout rules do not fail a board either. The verdict lists their counts as advice, and passes() does not read them. See Datasheet layout rules.
workspace.verdict() reads the build's own report and calls the same passes(). So the verdict a user sees and the evaluator's pass@1 are one number. The repair loop's clean is the same rule without the routing and copper clauses, because that loop stops before the router.
passed | Command line | App | Means |
|---|---|---|---|
true | this board passes | GO | Every clause passed |
false | this board DID NOT PASS | NO-GO | A check that ran found a failure. The verdict names each failed clause. |
null | this board is NOT JUDGED | UNJUDGED | Nothing that ran failed, but a check the rule needs did not run, such as the ERC or KiCad's DRC |
The app shows the verdict as a poll with one station for each part of the rule: Parts, Nets, ERC, Route and Copper.
Repair: feeding the findings back
--repair N sends the pipeline's findings back to the model and asks for a revised design, up to N times. Off by default.
- The model writes a design, and the pipeline resolves its parts and runs the ERC.
- The design goes back with a brief of the gaps and ERC findings.
- The model returns a revised design, which is resolved and checked again.
- The loop stops after N rounds, or as soon as the kept board is clean: every part resolved, no ERC errors, nothing dropped.
The loop stops before the router. Routing is the slow stage, and every failure worth a model call shows up in the parts and the ERC.
Repair must not become laundering. The cheapest way to clear R3 did not resolve is to delete R3. So the score weights a dropped part above an unresolved one:
| Finding | Weight |
|---|---|
| ERC error | 10 |
| Part dropped since the first design | 9 |
| Unrouted net (when routing is measured) | 7 |
| Part that did not resolve | 6 |
| Gap | 0.4 |
| ERC warning | 0.3 |
- Lower is better.
- Each round revises the best board so far, not the latest, so a bad round is a dead end. Ties go to the earlier round.
repair.json, next to the build, records the kept board, the gain over the single shot, the tokens spent and the parts lost.- If parts were lost, the command line says so: repair: n part(s) present in the first design are gone from this one. Check they were not needed.
- A network outage in the parts search is never shown to the model. It is the pipeline's failure, not the design's, and must never cost a good part.
The wiring turn: the model wires the parts it can see
The model's first draft names pins from memory, before it has seen the parts. So it writes VM for a DRV8825 whose pads are VMA and VMB, and the resolver can only refuse the connection. The wiring turn (wire.py, --wire) fixes that after the parts resolve:
- The model is shown each part as it will arrive: the catalogue row, and the exact names its pins answer to. This is the list the parts palette shows a person.
- It returns the nets against those names. The output schema has one property per part and one per pin, so it cannot write a pin the part does not have.
- A part the model says is wrong for the design goes back with a better search. The rejected part is kept off that part's candidates.
- The turn is taken once more after a replacement, and no more.
- A backend that will not take a schema that size gets a flat list, and the same rule is applied afterwards. Every refused name is listed in
unsupported. - A net the turn can put only one pin on is kept, so the build reports it and the verdict counts it.
- If the call fails, the draft's wiring stays and the board is still built.
Evaluation: pass@1 on this task
python -m soldermask.evaluate scores a backend on this task. pass@1 is the share of boards that pass the pass rule on the first try.
python -m soldermask.evaluate --backends gemini # all sixty, one backend
python -m soldermask.evaluate --backends gemini --n 8 # the original eight
python -m soldermask.evaluate --tier CD # the MCU and multi-rail bands
python -m soldermask.evaluate --repair 2 # paired before/after on the same designs
python -m soldermask.evaluate --designs eval/gemini/designs --repair 0 # replay: deterministic
python -m soldermask.evaluate --designs eval/gemini/designs --wire # replay + the wiring turn
python -m soldermask.evaluate --rescore eval-2026-09-14 .cache/eval-* # re-read under today's rule| Flag | Does |
|---|---|
--backends | Which model backends to run, comma-separated. The default is gemini,claude; name claude-code to score the app's own designer. |
--n | Run the first N prompts. The first eight are the original suite, in order. |
--tier | Run only these tiers, such as CD |
--repair N | Run the repair loop, and report the scores before and after on the same designs |
--designs DIR | Replay saved designs instead of calling the model, so the run is deterministic |
--wire | Add the wiring turn: one model call per board |
--rescore DIR... | Re-read stored runs under today's rule. Builds nothing. |
--no-route | Skip the router, and drop routability from the rule |
--out | Where to write the results (default eval) |
The suite
Sixty fixed prompts in four tiers, by part count:
| Tier | Band | Prompts |
|---|---|---|
| A | Basic | 12 |
| B | Easy | 16 |
| C | Medium | 18 |
| D | Hard | 14 |
summary.json reports each tier as well as the pooled number. A pooled number over a suite weighted toward small boards says more about the weighting than the pipeline.
The suite is saturated
Replayed on today's pipeline, the saved designs of the sixty prompts pass all sixty. So the suite no longer separates one change from another. Changes are judged on replays of the 305 saved designs and on the hard set (below).
What it records
| File | Holds |
|---|---|
results.json | Every measured field per board. That covers parts asked and resolved, ERC counts, gaps with their text, and routing and unrouted nets. It also holds BOM cost, seconds, tokens, the Gerber read-back, the verdict, route_attempts and p_clean. With --repair, the base round's figures sit beside the repaired ones. |
summary.json | Per backend: pass@1, generated, parts_resolved, erc_errors_mean, gaps_mean, routed, seconds_mean, fab_readable. With --repair, also pass@1_base and parts_dropped_total. |
If parts_dropped_total is not about zero, part of the repair gain is laundering.
How a board is built
Every board is built by one workspace.realise() call, the same call the app and the command line make. So pass@1 measures the product as shipped. tests/test_evaluate.py counts the realise() calls. That build includes:
-
the outline grown or cut to the parts, and the precedent start
-
six candidate placements routed and ranked by Lightcycle, nine when none reaches every pin, then up to three routed in full
-
escape stubs, the pour and its DRC
-
a second build with the power nets poured, when the copper alone fails
-
A row is the verdict
realise()stamped on the board. A row whose own rule disagrees with it is refused as a harness fault. -
A net whose copper misses a pin is incomplete, not routed.
-
The Gerbers handed to the independent reader (
fab_readable) are the build's own, copper and pours included.
Replays and rescores
- Saved designs are named
NNN-<board>.jsonby the prompt's position in the suite, so--designspairs each with its prompt and tier. A directory from before that naming is paired by sorted position, and the run says so. - Every attempt is written to the corpus, like any build. Point
SOLDERMASK_CORPUSelsewhere for a run that should not be. --rescoreis for the day the rule changes. It reads every row of a stored run again underpasses()as it is now. It keeps the first files as*.before-rescoreand adds arescoredrecord to the summary, so the first number stays visible beside the corrected one. A change in the pipeline needs a replay, not a rescore.soldermask projects rejudgedoes the same for saved versions. It runs KiCad's DRC again over each version's own copper and rewrites a verdict that moved. See Command line.
A run nobody routed is not a run that failed
--no-route drops routability from the rule instead of scoring every board as unroutable. tests/test_harness.py holds the harness to that.
Is it the board that was asked for
The pass rule says whether a board can be built. It does not say whether the board does what its prompt asked. A 1 kHz filter built with a 10 kHz corner passes it. soldermask/grade.py is the second judge for that question.
- A task is a file. Each one under
bench/index/tasks/names a prompt, a tier and the clauses its check reads off the board as built. - It reads the parts as ordered. A part's value, rating and type come from the catalogue row of its part number. The design's text is read only where the catalogue says nothing.
- A clause has three outcomes. It passes, fails, or is
unknownwhere the grader lacks what it needs. Unknown never counts as a pass, and the grader reads a part's datasheet record before it settles on unknown. - A failure is declared or silent. It is declared when the design's
unsupportedorassumptionssays so.
tools/index_grade.py grades a run. Its --mutate flag breaks a correct board the ways a wrong one is wrong, to show each check fails what it should.
The hard set
The hard set is 24 prompts for boards text-to-PCB tools fail on: four layers, USB and Ethernet pairs, and rails of 5 to 80 A. It is kept outside the repository, so it stays held out. Each board is judged three ways, from the files a fab gets:
| Judge | What it asks |
|---|---|
bench/referee.py | KiCad's DRC under the bench's fixed rules: no open connection and no copper, hole or courtyard error |
The task's grade.py check | Whether the board does what the prompt asked |
soldermask/copperjudge.py | Whether the stack has its layers, pairs run coupled and matched, and each stated current's path carries it |
Stock and pricing
This stage checks whether you can build the board today, at the quantities people order. It reads stock and price for every part from the parts catalogue on disk.
What is not guaranteed
This page lists what a finished board does not promise. A board that routes is not always a board that works, and this list is the difference.