You Probably Don’t Need to Build an LLM. You Need to Build a System.
Almost every technical leader I talk to is having some version of the same conversation: should we build our own model?
It’s the wrong question — and answering it wrong is expensive.
The $10M lesson
Here’s the story I keep coming back to. Bloomberg spent around $10M training BloombergGPT, a 50-billion-parameter model purpose-built on decades of financial data. Serious engineering. Then GPT-4 shipped with zero finance-specific training and beat it on nearly every public financial benchmark. The lesson wasn’t “Bloomberg did it wrong.” It was: scale beats specialization when the frontier moves faster than you can train. That dynamic hasn’t slowed down. It’s accelerated.
So when a team tells me they want to build a domain LLM, I ask what they’re actually trying to do. Nine times out of ten the goal isn’t a model — it’s a system that answers domain questions accurately and takes action. And you don’t get there by baking knowledge into the weights. You get there by keeping the model deliberately dumb about facts.
One principle does most of the work
The LLM reasons; it doesn’t remember facts. Anything that has to be exact — a number, a rule, a price, a citation — comes from a tool or a retrieved document, never the model’s memory.
That single rule dissolves most of the “we need a custom model” argument. Wrong dosage, wrong torque value, wrong contract clause, wrong balance — those aren’t fixed by more training. They’re fixed by not asking the model to be a database.
The three paths, and when each is right
- Train from scratch — tens of millions of dollars, a research team, a year, and a result the frontier laps before you ship. Almost never the answer.
- Fine-tune — genuinely useful, but for format, tone, and behavior, not for injecting facts. A few thousand examples, not a training run.
- Retrieval + tools on a frontier model — grounded answers with citations, live data, real actions, and a human approving anything that commits. Weeks to months, thousands to low-tens-of-thousands of dollars. This is the answer far more often than people expect.
The real work is the plumbing
The uncomfortable part for engineers is that the interesting work isn’t the model at all. It’s the plumbing: the integrations to your real systems, the retrieval layer over your own documents, the memory, the guardrails, and the human-in-the-loop that keeps a person accountable for anything irreversible. That’s 80% of the effort and 100% of the differentiation — and it’s the part a competitor can’t replicate by renting the same model you did.
It’s also the smarter strategic bet
There’s a strategic upside too. Because your value lives in tools and data, not weights, the model becomes a swappable component. The frontier improves, you upgrade, and everything you built still works. The team that poured a year into custom weights is left maintaining a depreciating asset.
The bottom line
So before you scope a model, scope the system. Ask two questions: what has to be exact, and where does that truth actually live? Wire the model to those sources, put a human on the commit button, and ship.
You’ll be in production while everyone else is still buying GPUs.