soldermaskdocs

Repair loop and evaluation

The pipeline can send its own findings back to the model, and a harness scores the whole pipeline. All of them judge a board by the same pass rule.

The pass rule

repair.passes() is the one rule. A board passes when:

  1. every part resolved to a real part the pipeline can draw
  2. no net and no connection the design asked for was dropped
  3. the ERC ran and found no errors
  4. every net routed, with no pin short
  5. KiCad's DRC judged the routed copper and found no errors, including after the pour
  6. no run of copper narrower than its net asks reaches a 100 °C rise at the net's declared current

ERC warnings do not fail a board. A module's unused pins are floating, and that is normal. An out-of-stock part still resolves; stock is reported as a note.

Datasheet layout rules do not fail a board either. The verdict lists their counts as advice, and passes() does not read them. See Datasheet layout rules.

workspace.verdict() reads the build's own report and calls the same passes(). So the verdict a user sees and the evaluator's pass@1 are one number. The repair loop's clean is the same rule without the routing and copper clauses, because that loop stops before the router.

passedCommand lineAppMeans
truethis board passesGOEvery clause passed
falsethis board DID NOT PASSNO-GOA check that ran found a failure. The verdict names each failed clause.
nullthis board is NOT JUDGEDUNJUDGEDNothing that ran failed, but a check the rule needs did not run, such as the ERC or KiCad's DRC

The app shows the verdict as a poll with one station for each part of the rule: Parts, Nets, ERC, Route and Copper.

Repair: feeding the findings back

--repair N sends the pipeline's findings back to the model and asks for a revised design, up to N times. Off by default.

  1. The model writes a design, and the pipeline resolves its parts and runs the ERC.
  2. The design goes back with a brief of the gaps and ERC findings.
  3. The model returns a revised design, which is resolved and checked again.
  4. The loop stops after N rounds, or as soon as the kept board is clean: every part resolved, no ERC errors, nothing dropped.

The loop stops before the router. Routing is the slow stage, and every failure worth a model call shows up in the parts and the ERC.

Repair must not become laundering. The cheapest way to clear R3 did not resolve is to delete R3. So the score weights a dropped part above an unresolved one:

FindingWeight
ERC error10
Part dropped since the first design9
Unrouted net (when routing is measured)7
Part that did not resolve6
Gap0.4
ERC warning0.3
  • Lower is better.
  • Each round revises the best board so far, not the latest, so a bad round is a dead end. Ties go to the earlier round.
  • repair.json, next to the build, records the kept board, the gain over the single shot, the tokens spent and the parts lost.
  • If parts were lost, the command line says so: repair: n part(s) present in the first design are gone from this one. Check they were not needed.
  • A network outage in the parts search is never shown to the model. It is the pipeline's failure, not the design's, and must never cost a good part.

The wiring turn: the model wires the parts it can see

The model's first draft names pins from memory, before it has seen the parts. So it writes VM for a DRV8825 whose pads are VMA and VMB, and the resolver can only refuse the connection. The wiring turn (wire.py, --wire) fixes that after the parts resolve:

  1. The model is shown each part as it will arrive: the catalogue row, and the exact names its pins answer to. This is the list the parts palette shows a person.
  2. It returns the nets against those names. The output schema has one property per part and one per pin, so it cannot write a pin the part does not have.
  3. A part the model says is wrong for the design goes back with a better search. The rejected part is kept off that part's candidates.
  4. The turn is taken once more after a replacement, and no more.
  • A backend that will not take a schema that size gets a flat list, and the same rule is applied afterwards. Every refused name is listed in unsupported.
  • A net the turn can put only one pin on is kept, so the build reports it and the verdict counts it.
  • If the call fails, the draft's wiring stays and the board is still built.

Evaluation: pass@1 on this task

python -m soldermask.evaluate scores a backend on this task. pass@1 is the share of boards that pass the pass rule on the first try.

python -m soldermask.evaluate --backends gemini            # all sixty, one backend
python -m soldermask.evaluate --backends gemini --n 8      # the original eight
python -m soldermask.evaluate --tier CD                    # the MCU and multi-rail bands
python -m soldermask.evaluate --repair 2                   # paired before/after on the same designs
python -m soldermask.evaluate --designs eval/gemini/designs --repair 0   # replay: deterministic
python -m soldermask.evaluate --designs eval/gemini/designs --wire       # replay + the wiring turn
python -m soldermask.evaluate --rescore eval-2026-09-14 .cache/eval-*     # re-read under today's rule
FlagDoes
--backendsWhich model backends to run, comma-separated. The default is gemini,claude; name claude-code to score the app's own designer.
--nRun the first N prompts. The first eight are the original suite, in order.
--tierRun only these tiers, such as CD
--repair NRun the repair loop, and report the scores before and after on the same designs
--designs DIRReplay saved designs instead of calling the model, so the run is deterministic
--wireAdd the wiring turn: one model call per board
--rescore DIR...Re-read stored runs under today's rule. Builds nothing.
--no-routeSkip the router, and drop routability from the rule
--outWhere to write the results (default eval)

The suite

Sixty fixed prompts in four tiers, by part count:

TierBandPrompts
ABasic12
BEasy16
CMedium18
DHard14

summary.json reports each tier as well as the pooled number. A pooled number over a suite weighted toward small boards says more about the weighting than the pipeline.

The suite is saturated

Replayed on today's pipeline, the saved designs of the sixty prompts pass all sixty. So the suite no longer separates one change from another. Changes are judged on replays of the 305 saved designs and on the hard set (below).

What it records

FileHolds
results.jsonEvery measured field per board. That covers parts asked and resolved, ERC counts, gaps with their text, and routing and unrouted nets. It also holds BOM cost, seconds, tokens, the Gerber read-back, the verdict, route_attempts and p_clean. With --repair, the base round's figures sit beside the repaired ones.
summary.jsonPer backend: pass@1, generated, parts_resolved, erc_errors_mean, gaps_mean, routed, seconds_mean, fab_readable. With --repair, also pass@1_base and parts_dropped_total.

If parts_dropped_total is not about zero, part of the repair gain is laundering.

How a board is built

Every board is built by one workspace.realise() call, the same call the app and the command line make. So pass@1 measures the product as shipped. tests/test_evaluate.py counts the realise() calls. That build includes:

  • the outline grown or cut to the parts, and the precedent start

  • six candidate placements routed and ranked by Lightcycle, nine when none reaches every pin, then up to three routed in full

  • escape stubs, the pour and its DRC

  • a second build with the power nets poured, when the copper alone fails

  • A row is the verdict realise() stamped on the board. A row whose own rule disagrees with it is refused as a harness fault.

  • A net whose copper misses a pin is incomplete, not routed.

  • The Gerbers handed to the independent reader (fab_readable) are the build's own, copper and pours included.

Replays and rescores

  • Saved designs are named NNN-<board>.json by the prompt's position in the suite, so --designs pairs each with its prompt and tier. A directory from before that naming is paired by sorted position, and the run says so.
  • Every attempt is written to the corpus, like any build. Point SOLDERMASK_CORPUS elsewhere for a run that should not be.
  • --rescore is for the day the rule changes. It reads every row of a stored run again under passes() as it is now. It keeps the first files as *.before-rescore and adds a rescored record to the summary, so the first number stays visible beside the corrected one. A change in the pipeline needs a replay, not a rescore.
  • soldermask projects rejudge does the same for saved versions. It runs KiCad's DRC again over each version's own copper and rewrites a verdict that moved. See Command line.

A run nobody routed is not a run that failed

--no-route drops routability from the rule instead of scoring every board as unroutable. tests/test_harness.py holds the harness to that.

Is it the board that was asked for

The pass rule says whether a board can be built. It does not say whether the board does what its prompt asked. A 1 kHz filter built with a 10 kHz corner passes it. soldermask/grade.py is the second judge for that question.

  • A task is a file. Each one under bench/index/tasks/ names a prompt, a tier and the clauses its check reads off the board as built.
  • It reads the parts as ordered. A part's value, rating and type come from the catalogue row of its part number. The design's text is read only where the catalogue says nothing.
  • A clause has three outcomes. It passes, fails, or is unknown where the grader lacks what it needs. Unknown never counts as a pass, and the grader reads a part's datasheet record before it settles on unknown.
  • A failure is declared or silent. It is declared when the design's unsupported or assumptions says so.

tools/index_grade.py grades a run. Its --mutate flag breaks a correct board the ways a wrong one is wrong, to show each check fails what it should.

The hard set

The hard set is 24 prompts for boards text-to-PCB tools fail on: four layers, USB and Ethernet pairs, and rails of 5 to 80 A. It is kept outside the repository, so it stays held out. Each board is judged three ways, from the files a fab gets:

JudgeWhat it asks
bench/referee.pyKiCad's DRC under the bench's fixed rules: no open connection and no copper, hole or courtyard error
The task's grade.py checkWhether the board does what the prompt asked
soldermask/copperjudge.pyWhether the stack has its layers, pairs run coupled and matched, and each stated current's path carries it

On this page