The Missing Half of AI Governance

Ramon B. Nuez Jr.

Ramon B. Nuez Jr.

— 4 min read
Share
The Missing Half of AI Governance

I lost an evening arguing with myself about Dyson spheres.

It started with a piece by Jean-Stanislas Denain, a researcher at Epoch AI, on what he calls the missing half of AI futurism debates. His argument is simple and it stuck with me. We spend enormous energy asking how smart the machines will get. We spend almost none asking how hard the things we expect them to build actually are. Molecular nanotech, self-replicating interstellar probes, Dyson swarms built to harvest a whole star. In most futurist writing these get treated as a foregone conclusion the moment intelligence crosses some threshold. As he puts it, "we've thought hard about how good the AIs might be, but not about how hard it'll be to develop specific technologies."

I felt the pull toward that debate immediately. Self-replicating probes that mine a new star system and launch copies onward at a fraction of light speed. I could lose a weekend there and enjoy every minute of it.

Then a different sentence caught me, and it had nothing to do with probes.

The tell was in the metric

When we try to reason about any of this, we reach for economic numbers. Gross domestic product (GDP). Growth rates. Output per worker. We reach for them because they are familiar and they touch daily life. Denain flags the quiet problem in a single line: these numbers "might be poor proxies for what we care about." That is the sentence where I stopped reading about probes. You could raise world GDP a hundredfold and still not know whether anyone can build the thing that actually changes the game. Whether we reach the stars. Whether a system can be turned toward something catastrophic.

We measure GDP because GDP is legible. The capability underneath it is the thing that matters, and it is the thing we skip.

I have watched that exact move for twenty years. Not in futurism. In governance.

Governance measures the legible half

Most AI governance today measures the part that is easy to see. Model cards. Risk registers. Audit reports. Policy documents that say the correct things in the correct order. This is the GDP of governance. A set of artifacts everyone agrees to look at because they are easy to produce and easy to check.

The decisive half is a different question entirely. It is whether you can reconstruct why a system made a specific decision, for a specific person, at a specific moment. Why this application was denied. Why this transaction was flagged. Why this case was routed the way it was.

Most programs cannot answer that. Not because the tools do not exist. Because the audit trail was never built into the architecture in the first place. The artifact became the goal. Accountability was quietly never the point.

I spent two decades inside large public-sector technology, where a bad automated decision does not create a poor user experience. It disrupts a service someone depends on. That environment teaches you to tell the difference between a system that documents itself and a system that can actually explain itself. They are not the same system, and the paperwork rarely tells you which one you have.

Watch it become law

This is not abstract. It is being written into statute right now.

Illinois has enacted the Artificial Intelligence Safety Measures Act, known as SB 315. It is a serious law. It targets the largest frontier developers, above roughly half a billion dollars in revenue and a heavy compute threshold. It requires them to hire third-party auditors and to publish documentation on how they measure model capability, how they estimate catastrophic risk, and how they respond to safety incidents. The state attorney general can enforce penalties up to three million dollars per violation. Compliance lands in 2028.

Read the structure carefully and you can see the same bet on the legible half.

A third-party audit verifies what a system was built to reveal. Published documentation describes what the developer chose to describe. If a model's architecture cannot reconstruct why it produced a given output, the audit certifies the documentation, not the behavior. You end up with a clean report about a system that no one can actually explain. Compliance goes up. Accountability stays flat.

That is the symmetry that turned an evening of Dyson-sphere daydreaming into something useful. Futurism forecasts the legible half and calls it a prediction. Governance audits the legible half and calls it accountability. Both mistake the metric they can see for the capability that decides everything.

What this means when the tools get this good

Here is the part that matters for anyone building or answering for these systems.

When the tools get this good, the legible work gets cheap. Anyone can generate a model card. Before long the model generates its own, formatted perfectly, sourced neatly, ready for the auditor. The artifact stops being evidence of anything because producing it costs nothing.

What stays scarce is the capability the artifact was supposed to stand for. A system built so a human can reconstruct its decisions after the fact. And a human who knows which decision to interrogate. When execution collapses to near zero, judgment about what to measure becomes the entire job.

So the question I would put to any team shipping AI right now is not whether they have governance documentation. Everyone will have that soon, generated on demand. The question is smaller and much harder.

Pick one decision your system made last week. A denial, a flag, a routing, a recommendation. Show me, from the system's own records, why it did that, for that person, at that time.

If the answer is a meeting instead of a trace, you have the legible half. You are missing the half that counts.

Denain wants more people asking how hard the future will be to build. I want more people asking a smaller question about the systems already in production. Forget how smart the model is for a moment. Can anyone explain what it just did?

Sources and notes

Read more