When AI Writes the Software, What Do Architects Design?
Governance isn’t the whole story. If behavior, not code, is what gets built and promoted, what’s the actual unit of architecture?
A comment on my last article stopped me mid-scroll. I’d argued that AI becomes the compiler of intent, and that governance sits downstream of it, verifying what the AI proposes before it executes. Suresh Balivada pushed back: in a world where implementation is opaque and probabilistic, governance isn’t a constraint on the architecture. It IS the architecture. Intent, he said, is the easy part.
He’s right, and it’s worth sitting with why. But it isn’t the whole story. Governance is necessary. It isn’t sufficient. Follow that thread far enough and you land somewhere more specific than “governance matters”: what’s the actual unit of architecture once code stops determining behavior?
Code used to be the architecture
For as long as most of us have worked in software, architecture meant something concrete: diagrams, module boundaries, interfaces, coupling, cohesion, the shape of the codebase. An architecture review meant looking at the code, or the structures it would take, and asking whether it would hold up.
That worked because behavior lived in the code. If you wanted to know what a system would do, you could read it, trace it, or step through it in a debugger. The code was the contract between intention and outcome.
AI moves behavior somewhere else
That’s no longer where behavior lives. In an AI-driven system, behavior emerges from the interaction of a prompt, a policy, a retrieval step, available tools, memory, context, and a planner deciding what to do next. No single artifact contains “the application” the way a codebase used to. The application is closer to an emergent property of a system than a fixed thing you can point to.
It’s tempting to place that emergent property entirely inside the model, and that’s not right. Behavior emerges from the continuous interaction between probabilistic reasoning and deterministic enterprise capabilities, neither alone constitutes the application. Take procurement: the LLM never buys anything, it reasons about what should happen. The ERP executes the purchase, the policy engine approves it, the payment gateway moves the money, the audit platform records the trail. What an observer sees, a pump ordered, a budget respected, a vendor notified, is the joint output of all of them, not a decision reasoning made alone.
This is the part of my last article I’d still stand by: the visible artifacts of software shrink. What I underweighted is everything deterministic that fills the space they leave behind.
Intent is not a specification
Here’s a distinction that’s been underneath this whole conversation without quite being named. Intent tells a system what to accomplish, “process invoices,” “order replacement parts within budget.” A specification tells it what’s allowed while accomplishing that: which approvals are required, which vendors are eligible, what counts as an exception, what must never happen no matter how efficient the alternative looks.
Conflating the two is where a lot of the anxiety about AI-native systems comes from. Intent is genuinely easy to express, that’s what makes natural-language interfaces feel powerful. Constraint engineering, deciding what “under what conditions” actually means and making it enforceable, is the hard part. It always was. AI didn’t remove that difficulty, it relocated it from a requirements document to something that has to be executable.
The architecture becomes a loop, not a diagram
If intent and specification are separate, the system connecting them can’t be a static pipeline. It has to be closed:
Intent → Policy → Reasoning → Evaluation → Execution → Observation → Learning → (back to Policy)
Static architecture answers “what does the system consist of.” A closed loop answers “how does the system stay correct as it keeps acting.” That second question barely existed for traditional software, once shipped, behavior was fixed until the next release. Now it’s a permanent, running condition, which is why “governance as a constraint” undersells what’s being built. Governance here isn’t a gate at the end, it’s woven through every stage: what gets reasoned about, what gets evaluated, what gets learned from.
One thing that loop leaves implicit: none of these stages run in a vacuum. They all read from and write to enterprise state, orders, balances, inventory, entitlements, contracts, the facts the enterprise keeps as true. Reasoning consults that state before proposing anything; execution is what changes it; observation is watching the delta. Worth saying plainly: reasoning doesn’t own truth. Enterprise state does. The AI can be wrong about what should happen next; the ledger can’t be wrong about what already happened.
What we version is changing
This is where the hardest, most concrete question lives, the one Suresh raised: if there’s no stable codebase, what does it mean to change the system safely? What do you diff? What tells you something broke?
Every generation of computing has versioned whatever artifact actually determined behavior at the time: source code, then configuration and rule engines as behavior outgrew the code, then infrastructure, then container images as the runtime needed pinning. Each shift versioned a bigger unit than the last, but each was still enumerated, a fixed set of branches a human had written down, however large the set got. Enterprise behavior is the first unit that breaks the pattern: intent, policy, tool permissions, model and prompt versions, and the evaluation results that gated the last change, versioned as a bundle, not because it’s a bigger rulebook but because it can produce a plan nobody wrote down, only made possible by the boundaries around it.
This isn’t hypothetical. Eval-driven development, writing the evaluation suite as the actual release gate, is close to standard practice now for teams shipping agents on frontier models, precisely because model behavior can shift under a prompt that hasn’t changed. Agent platforms like LangSmith and Braintrust already treat prompt versions, evaluation suites, and tool permissions as versioned deployment artifacts, pairing this with anomaly detection and kill switches so drift gets caught and halted, not just logged.
Picture a commit log for that bundle instead of for code:
Change 145: procurement agent now escalates purchases above $250,000 to a human approver.
No function changed. No module was touched. But the enterprise’s behavior changed, in a way that’s recorded, reviewable, and reversible. That’s a meaningfully different thing to version than code ever was, and it’s the more honest answer to “does every changed intent regenerate the application.” No: changing intent changes a governed, evaluated, promotable unit of behavior. The application underneath may not be touched at all.
Where I have to be careful not to oversell this
This is the place it gets genuinely hard, exactly the gap Suresh pointed at with the compiler analogy, and it’d be dishonest to wave past it. A code test suite checks a deterministic system, same input, same output, every time, so a passing suite is a real guarantee. An evaluation suite checks a probabilistic one: agent evaluation practice this year shows systems that pass final-output scoring can still hide real regressions inside the reasoning trajectory, a change at one planning step corrupting a downstream step in a way a single-turn score never catches.
So “if evaluation passes, behavior is promoted” is the right shape of an answer, but not yet the clean guarantee CI/CD gives you over code. Getting evaluation suites to the point where passing them means what a passing code test means is, I’d argue, the real unsolved engineering problem of this era, more than anything about the models themselves. That’s not a footnote. It’s the thing worth building toward.
A new SDLC, not a faster old one
If behavior, not code, is the unit that gets built, tested, and promoted, the lifecycle around it looks different in kind, not just in speed.
Intent Modeling → Capability Modeling → Policy Design → Evaluation Design → Agent Assembly → Simulation → Observability → Continuous Alignment → Governance Evolution
None of this replaces the engineering that builds and maintains the underlying systems, the tools, APIs, and data platforms agents actually call still need to be built and operated the conventional way. What’s new is the layer above it, one that never really existed as a discipline before, because nothing used to change as continuously as a reasoning system’s behavior does.
The architect’s job
Which brings me back to where this started. Suresh said governance is the architecture. I think it’s the most mature part of a bigger discipline that doesn’t have a settled name yet, one that also includes intent modeling, evaluation design, and the trust and escalation structures that decide when a human needs to be in the loop. Whether that adds up to a genuinely new pillar alongside structural and behavioral architecture is a real question for its own piece. Thanks for the pushback, Suresh, this one exists because of it.
What I’m confident of is this: the architect’s job doesn’t shrink as code becomes less central. It moves. Less time spent designing what the system consists of. More time spent designing how it decides, how it proves that decision was sound, and how it gets safely corrected when it wasn’t.
For decades, architects designed systems whose behavior was explicitly encoded. Increasingly, they’ll design systems whose behavior is continuously negotiated between reasoning, policy, enterprise state, and human oversight. That changes the question we start with. It’s no longer “how should this application be built.” It’s “how should this enterprise behave.” That question, more than any particular model or programming language, may end up defining the architecture work of the next decade.
If you’re working out what your own version of “the bundle” should be, intent, policy, tool permissions, and evaluation results versioned together, that’s usually a concrete design exercise once you trace one real agent end to end. Happy to walk through it — get in touch.