Firmware
Real portable C talks over bus/time/GPIO callbacks to independently implemented peripheral emulators.
RealityCheck / AI for the Physical World / October 2026
Hardware agents.
Grounded in tests.
A testing ladder for electronic design — and an early benchmark with room to improve.
Portable CIndependent chip emulatorsngspiceKiCad
Open foundation · scoped simulation results · no measured-silicon claims
01 / The thesis
Learning needs a
fast, trustworthy feedback loop.
Unit tests
→Integration tests
→End-to-end tests
Software agents can iterate against executable checks. Hardware teams have simulation tools, but end-to-end validation often still requires fabrication and measurement.
That delay already hurts human engineers. It also makes reinforcement learning and agent evaluation harder.
Proposal: build a useful, explicitly bounded virtual testing ladder before asking models to build physical hardware.
02 / The environment
One design. Three independent checks.
Analogue behaviour
ngspice measures accuracy, filtering, settling and load response from the submitted circuit.
Physical connectivity
KiCad checks schematic/PCB parity, pin maps, routing, ERC and DRC.
Passing one layer cannot stand in for another. Candidate execution is separated from scoring authority; each attempt gets a fresh container and session.
03 / Tasks
Three optimisation problems.
01
Sensor monitor
Four 0–12 V inputs. Choose converter, scaling, filtering and buffering.
Minimise cost after accuracy, settling and sample-rate gates.
02
Analogue output
Generate a programmable waveform through a real DAC command stream and output network.
Maximise contiguous passing bandwidth under a BOM cap.
03
Fault-tolerant monitor
Eight inputs, equal-address ADC banks, bus switching, GPIO and power recovery.
Minimise cost after correctness and recovery gates.
Published criteria + fixed component catalogue + explicit model limits. Benchmark credits measure cost; these are not distributor prices.
04 / Early evidence
There is substantial
headroom.
Latest budgeted attempts: remaining unrouted connections
| Model | 01 · monitor | 02 · output | 03 · recovery |
|---|---|---|---|
| Luna | No native CAD | No native CAD | No native CAD |
| Sol | 10 | 33* | 125 |
| Astra | 0 | 0 | 4 |
Clean routing is only one gate. Astra’s sensor design earns a positive restricted score under a reviewed v2 adapter; arbitrary-design and full-task coverage remain incomplete.
Exploratory 15-minute attempts, one latest attempt per cell; *Sol 02 is a fresh replacement after infrastructure failure. Raw run data is kept private. Timeouts can retain useful artifacts. This is evidence of headroom, not a statistically valid model leaderboard or a proof of universal failure.
05 / Feasibility anchors
Reference boards give us
a place to start.
01 · 7.21 credits
ADS7924 + calibrated front end.
23.8 mV DC error; 10.18 ms step.
02 · 6.65 credits
DAC80501Z + buffer.
100 Hz passing bandwidth; 5.21 mV DC.
03 · 13.05 credits
Switched ADC banks + mux + GPIO.
27.79 ms recovery in the reference scenario.
Native KiCad boards rendered from public reference sources. Reported calibration measurements use declared, bounded models and reference scenarios; metrics are not measured silicon and do not establish complete evaluator coverage. References are withheld from held-out candidate sandboxes.
06 / What the pilot taught us
Test the reward, too.
For one 100 Hz output attempt, sampling policy changed the measured tracking error:
Request grid18.39 mV
Dense waveform26.60 mV
20 mV threshold. Same waveform, distinct measurement policies. This is valid optimisation of the published objective, and a reason to improve its next version.
Freeze the contract
Declare sample times, corners, startup policies, unsupported behaviour and score version before evaluating submissions.
Probe failure modes
Missing LDAC, stale samples, bad scaling, disconnected traces, missing decoupling and edited prices need independent negative controls.
Do not silently replace historical criteria. More simulator detail is useful only when its assumptions are explicit and validated.
07 / Training contract
Meet the constraints.
Then optimise X.
Tell the agent
Fixed challenge, catalogue, hard gates, budget and primary objective. Give it tool access and a fresh sandbox.
Return useful feedback
Separate feasibility violations, cost, bandwidth, accuracy, recovery, tool failures and unsupported coverage.
Reward valid improvement
Cost tasks: Cref / (Cref + C). Bandwidth: bounded logarithmic reward. Positive scores require supported, passing gates.
Next: broaden trusted grading, run larger repeated comparisons, train against the closed loop, and validate model assumptions on hardware.
An unsaturated starting point for hardware agents — with an honest boundary between simulation and reality.
RealityCheck · github.com/anton-pt/hwdevsbx