The Drive That Wasn't Dying

I trusted the drive because the drive told me it was fine.

Every diagnostic I ran came back clean. Zero percent wear. Full spare pool. No media errors. The SSD's own health check returned a single, confident word: PASSED. By every number the drive reported about itself, it was one of the healthiest components in my lab.

And every eight to twenty hours, it took the whole system down.

A note on who was doing the looking. I am not a hardware engineer or a software engineer. I am an IT executive. For twenty years I led the people who build and quietly told myself that was the same as building. It wasn't, and I felt the gap. What changed is that AI tools finally handed someone like me the surface area to build for real. So now I spend my nights in a homelab I made myself, learning by breaking things. This is one of the things that broke.

What was actually at stake

The lab is a small cluster of Raspberry Pi 5 boards and two Macs. Sensors on the edge nodes publish temperature, humidity, and motion over MQTT. One Pi in the middle is the important one. It runs the message broker, the vector database that holds every reading the lab has ever recorded, and the MCP server that lets a local language model ask questions about that history.

That one board is the lab's memory and its nervous system. When its NVMe drive goes unreadable, the database can't be opened, the ingest service crash-loops, and the agent that answers questions goes blind. Everything downstream is fine. The brain just stops.

So when it started failing on a loop, I did what most builders do. I went looking for the broken part.

Three confident wrong answers

The failure had a fingerprint. The kernel logged buffer I/O errors on the exact same block range every time. The filesystem flipped into a "clean with errors" state and refused to write. The drive stopped answering admin commands entirely.

Same signature, every event. That consistency told me it was real and recurring, not random. It did not tell me where to look, so I guessed.

Heat was the obvious suspect. A drive in a small case, running around the clock, throttling itself into errors. Except the temperature logs said 34 to 37 degrees Celsius straight through a failure window. Well inside spec. I even bumped the case fan to its aggressive profile before one of the failures. It failed anyway.

Wear was next. Flash memory dies eventually, and a database doing constant small writes is exactly the workload that grinds it down. Except the drive had written about 69 gigabytes in its entire life and reported three percent of its endurance used. It had barely been touched.

Then the filesystem itself. Maybe the journal was corrupt. Except the metadata was intact between every event. The filesystem only broke in response to something underneath it breaking first. It was a symptom wearing a costume.

Three good hypotheses. Three dead ends. Each one cost me a day.

The suspect I almost framed

Here is where I nearly fooled myself.

When the failures started, I built a small health logger. A script that woke up every five minutes, probed the drive, and wrote the result to a separate card so I would catch the exact moment of the next failure.

Then I noticed something that made my stomach drop. The failing block range was being read every few minutes, on a periodic cadence, by a process that kept retrying the same fetch. A five-minute cadence. The same interval as my logger.

I had built the tool to observe the failure, and now the tool looked like the cause. The tempting move was to kill it, watch the errors stop, and declare victory.

I left it running instead. If the logger were the trigger, disabling it would fix the problem but teach me nothing, and it would blind me the moment the real fault came back. My logger was only the flashlight that happened to be pointed at the intruder. Confusing the flashlight for the intruder would have ended the investigation at exactly the wrong moment.

The vendor told me it wasn't their drive

I filed a warranty claim expecting the usual script: send it back, get a replacement, move on.

The support engineer read my symptom report and pushed back. Their words, roughly: this looks like compatibility between the drive, the firmware, and the host environment, not a confirmed hardware failure. A vendor talking me out of a replacement is not something I expected, and it turned out to be the most useful reply in the whole thread.

They pointed at the host's power management. Two settings that let the drive and the PCIe link drop into low-power states, combined with an outdated kernel that handled those transitions badly. The drive was healthy. The path between the drive and the operating system was not.

The fix

Two kernel parameters to stop the aggressive power transitions, plus a kernel and firmware upgrade to fix how they were handled.

# /boot/firmware/cmdline.txt (append to the single existing line)
nvme_core.default_ps_max_latency_us=0   # disable NVMe APST
pcie_aspm=off                           # disable PCIe link power saving

# then pull a current kernel + firmware
sudo apt full-upgrade                   # 6.12.75 -> 6.18.33 in my case
sudo reboot

The prior failure cadence was every eight to twenty hours under load. After the change I logged 1,608 samples across six-plus days of continuous uptime. Zero I/O errors. Zero filesystem errors. The drive I almost returned is still running today.

What I actually took away

The first lesson is about trust. A status report from a failing component is not evidence, because the thing generating the report is the thing that's broken. The drive said PASSED while it was actively refusing commands. I now treat any component's self-assessment as one input, and I instrument the failure signature separately, from outside the thing being measured.

The second is about your own tools. When you add instrumentation to catch a bug, that instrumentation becomes a suspect, and it is a seductive one because it correlates perfectly with the symptom. Correlation with your own probe usually means your probe is the most visible reader of a fault that was already there. Killing the flashlight does not remove the intruder.

The third is the one I keep relearning. The layer that fails is often not the layer that looks broken. The errors surfaced in the filesystem. The healthy readings came from the drive. The real fault lived in the power-management path between them, a layer that never threw an error of its own and never showed up in a single health check. I spent three days interrogating the two layers that were shouting and none on the quiet one in the middle that was actually lying.

The drive was never dying. I just kept asking the wrong part whether it felt okay.