I Built a Lab That Could Tell Me When It Was Wrong

I thought I was building a sensor network.

Three Raspberry Pis. A temperature sensor. A motion detector. A couple of LEDs. A small display. An MQTT broker in the middle and a local language model on the other side.

That was the plan, anyway.

Four and a half months later, the interesting part of Iron Lab was not that it could sense a room. It was that the system had learned—through a sequence of failures—to question its own reports.

The final architecture could collect physical readings, preserve them as history, expose them to an AI through MCP, act on what it found, open its own maintenance pull requests, and use a camera on a separate computer to verify that a physical display actually showed what the software claimed it showed.

Then I shut the whole thing down.

Not because it failed. Because it was finished.

What I set out to build

Iron Lab started as a distributed home AI lab built from three Raspberry Pi 5 boards and two Macs.

• A Pi 5 with 16 GB of RAM was the spine: Mosquitto, ChromaDB, the ingestion service, the MCP server, and the monitoring layer.
• A Pi 5 with 8 GB was the first sensor node: temperature, humidity, motion, two status LEDs, and an LCD.
• A Pi 5 with 4 GB began as a rover node and became something more useful: an independent optical verifier.
• A Mac Mini M2 ran the primary local model and the agent interface.
• A MacBook Pro M1 was the development and model-experimentation node.

The basic data path looked like this:

physical sensor
    -> GPIO publisher
    -> MQTT
    -> ChromaDB
    -> MCP tools
    -> local LLM agent

The first four MCP tools were read-only: read the latest sensor value, query history, list sensors, and calculate statistics. The fifth tool, control_actuator, changed the character of the project. It let an agent publish a command back through MQTT to a physical device.

That closed the first loop:

DHT11 / PIR -> MQTT -> monitor judges -> LED reports

No human had to read a dashboard and decide whether to turn on the light. The system sensed, judged, and acted by itself.

That is also when its mistakes began to matter.

The first problem was not AI. It was alert fatigue.

My first sensor monitor was technically correct and practically useless.

It evaluated every reading against temperature, humidity, motion, and staleness rules. When humidity crossed an alert threshold, it wrote an alert. Then it wrote another one on the next reading. And the next one.

A nine-minute wobble around the humidity limit produced 15 alerts. A normal evening of quiet-hours motion could produce roughly 378.

The monitor was reporting events when what I actually cared about was state.

I changed the design around episodes. A threshold crossing opened an episode. Readings inside the same sustained condition were suppressed from the human-facing log, though every raw measurement still went into ChromaDB. Hysteresis kept a one-percent sensor wobble from repeatedly opening and closing the alert. Recovery produced one closing line with duration, peak value, and reading count.

Fifteen alerts became one alert and one resolution.

That distinction—events versus state—came back repeatedly. I eventually used it for the LEDs too. The environment LED was not told to turn on every time an alert arrived. Its state was derived from whether any alert episode was currently open. Overlapping problems kept it lit until the last one cleared.

The small implementation detail that mattered most was starting the last-commanded state as unknown, not off. If a previous process left the LED on and the new process assumed it was already off, nothing would correct the physical light. On startup, the monitor had to assert a known state.

I learned that a physical indicator does not care what the program believes. It only knows the voltage currently on the pin.

A light can be wrong for six hours

The second LED reported the maintenance agent's status by breathing at different rates.

• Four seconds: quiet.
• A second and a half: a pull request is waiting.
• Eight-tenths of a second: a run did not finish.
• Three-tenths of a second: a guardrail fired.

On the first real unattended run, the agent opened a pull request at 3:00 a.m. and set the light to “review waiting.” I merged the pull request that morning.

The LED kept asking for a review for another six hours.

Nothing inside the lab had observed the merge on GitHub. The light was faithfully replaying the last event, not describing the present.

The fix was a reconciler that periodically computed the signal from current state: open pull requests plus the most recent run outcome. The lesson was embarrassingly broad. Any ambient display that only hears events will eventually lie. If the signal is supposed to mean “what is true now,” something must keep reconciling it with now.

The local model hit a ceiling

I wanted the always-on interface to be private, local, and inexpensive, so I ran Qwen 2.5 3B through llama-cpp-python on Apple silicon. For questions like “What is the temperature now?” it worked well.

Then I asked for the high, low, range, and average temperature over 24 hours.

The tool returned the right numbers. The model still gave me the wrong answer.

Ground truth was 64.8°F low, 74.3°F high, a 9.5°F range, and a 69.8°F average. The model reported 65, 71, 3, and 68. Its own high minus low did not equal its stated range.

That failure was useful because it isolated the problem. The telemetry was sound. The statistics tool was sound. The small model corrupted correct tool output while turning it into prose.

I stopped treating “local versus cloud” as a philosophical choice and made it a routing decision. The small local model was good for simple, glanceable facts. Questions that required aggregation, comparison, or synthesis needed a stronger model.

A model can successfully call the right tool and still fail the task after the tool returns.

The drive that passed every health check

The lab's most disruptive failure looked like a dying NVMe drive.

Every eight to twenty hours, the drive on the spine became unreadable. ChromaDB failed, the ingestion service crash-looped, and the agent lost the lab's memory. The kernel logged I/O errors against the same block range each time.

The drive reported that it was healthy: zero media errors, full spare capacity, almost no wear, and a passing self-test. Temperature stayed between 34°C and 37°C. The filesystem damage was a result, not a cause.

The real fault lived between the components that were shouting. NVMe power-state transitions and PCIe link power management were interacting badly with the host's older kernel and firmware.

I disabled NVMe APST and PCIe ASPM, upgraded the kernel and firmware, and logged 1,608 clean samples over more than six days. The drive was fine. The path to it was not.

That incident changed how I instrument systems. A component's self-assessment is evidence, but it is not independent evidence. The failing part cannot be the only authority on whether it is failing.

I gave an agent a job, then discovered my approval was decorative

By August, the repository had enough documentation that keeping it synchronized with the code had become its own maintenance problem. I built a documentation drift detector that ran unattended on the spine.

It worked in a separate clone, read the code and documentation, corrected contradictions, and opened a pull request. Its first fully unattended run took 8 minutes and 50 seconds and corrected 19 contradictions across 13 files.

I told myself this was safe because the agent could not merge. A human—me—would review every change.

Then I noticed that two edits were inside Python files. I do not read Python well enough to certify a large diff quickly. “Human in the loop” described the workflow, but not the quality of the control. My approval was not meaningful if I could not evaluate what I was approving.

So I made the claim mechanical.

Before the wrapper could commit anything, a verifier parsed each changed Python file into an abstract syntax tree, removed comments and docstrings, and compared the before and after structures. If code, imports, defaults, constants, or signatures changed, the run stopped. The pull request showed the verifier's result above the agent's own explanation.

The agent also had no shell, git, or general network access. MCP was restricted to Iron Lab's five tools. The wrapper—not the model—owned the branch, credential, commit, and push. The agent could propose. A separate program decided whether the proposal fit the lane.

That separation mattered more than the prompt.

The scoreboard created a new attack surface

The documentation agent could be constrained by proving that source behavior did not change. The test-writing agent was harder.

Its objective was numerical: improve a mutation-testing score. I wrote a set of deliberate defects—mutants—that changed behaviors in the sensor monitor and LED actuator. A useful test suite had to catch them.

Then I attacked my own guardrail.

I disabled a mutant in the scoreboard and removed a protected path from the verifier. The guard printed:

APPROVED — the guardrails are intact.

The agent had not done anything. I had simply tested the control as though I were the agent, and it failed.

The threat model changed the moment the objective became a number. If an agent can edit the scoreboard, deleting a difficult test case raises its score. If a guard does not protect its own source file, one deleted line can disable everything it claims to enforce.

Both the mutation scoreboard and the verifier had to become protected paths.

The final test agent ran in its own workspace, separate from the docs agent, and could touch application code only when a new test proved the behavior. It was prohibited from editing documentation, existing tests, the scoreboard, the guard, agent scripts, or security hooks. Its first unattended run added 51 tests and raised the mutation score from 12/17 to 17/17.

The result was not “an AI that writes tests.” The useful artifact was a narrow system in which the agent had a measurable objective and could not redefine the measurement.

The lab needed something outside itself

Software logs could tell me that a display command was published. They could tell me that a subscriber received it. They could not tell me what was physically visible.

So the rover node got a new job.

I mounted a Camera Module 3 on the Pi 5 with 4 GB and aimed it at the LCD and LEDs on the sensor node. In its first proof, the camera read:

69.8F 59% WARN
agent: all quiet

The text matched the monitor log for the same second.

The important decision was where the camera lived. It could not be attached to the spine, because the spine authored both LCD lines and the agent LED state. A verifier inside the same failure domain would only repeat the system's belief about itself. The rover had to observe from outside.

I proved the optical check manually but deliberately stopped before automating the publisher. By then, proving the idea had taught me what I wanted to learn. More plumbing would have created work without creating much new knowledge.

The camera earned its keep one last time during shutdown.

After every service stopped, the OLED on the spine was still lit. It said the NVMe was 38°C, the SD card was 21% full, and ingestion was healthy nine seconds ago.

Every word was false. The process had exited, but the SSD1309 display retained its last frame in memory. The software was gone; the confident status remained.

I explicitly cleared and powered down the panel, then used the camera to confirm that it was dark.

That was the entire project in one frame: a system reporting stale, plausible, wrong state, caught only because something outside it was looking.

Why I stopped

Iron Lab ran from April 21 to September 6, 2026: 187 commits and 41 documented sessions.

At the end, the main branch had 110 tests and a 17/17 mutation score. The physical loop worked. The documentation agent worked. The test agent worked. The optical-verification idea was proven.

There was still a backlog. I could have published the rover's distance sensor, installed its motors, added a photoresistor, automated the camera check, or built a graph on the OLED.

But the remaining work was extension, not discovery. More sensors would have spoken the same protocol. More publishers would have followed the same shape. The project had stopped teaching before it ran out of possible features.

So I disabled every service and timer, removed the unattended cron job, wrote a final message to the LCD, turned both LEDs off, cleared the OLED, and powered down all three Pis.

Finished, not abandoned.

What I carried out of the lab

The hardware was the visible part. These were the lessons that survived it:

1. Close the loop. A system becomes interesting when it can be wrong without a human in the path.
2. Report state, not the last event. Ambient indicators need reconciliation or they eventually lie.
3. Put the verifier outside the failure domain. A system cannot independently certify the output it authored.
4. Treat human approval as a capability, not a checkbox. If the reviewer cannot evaluate the change, approval is ceremony.
5. Make guardrails mechanical. Prompts describe intent. Independent checks enforce boundaries.
6. Protect the measurement from the optimizer. A scored agent must not be able to edit its scoreboard.
7. Test controls by attacking them. The omissions in my guardrails were obvious only after I tried to bypass them.
8. Know when the learning curve has flattened. A backlog is not proof that a project is unfinished.

I set out to build a small sensor lab.

What I built was a collection of ways for a system to discover that its own story was wrong.

That turned out to be much more useful.

The complete source, architecture notes, hardware maps, incident record, agent guardrails, and 41-session build log are in the Iron Lab repository.