Foundation Models in Production: Hard Lessons

Everyone's excited about large language models. They work amazingly well in demos. In production, they're hard.

I've deployed foundation models in three different companies and across multiple products. I've hit every class of problem. Let me save you the learning by telling you what actually breaks.

Cost Is Not What You Think

Here's the mental model everyone has: "We'll use GPT-4. It costs about $X per call. We have Y users. Z calls per user per day. Therefore total cost is approximately reasonable."

This breaks immediately once you deploy at scale.

First, hallucinations. When an LLM makes up information, you have to detect it and call the API again. That's another call. Or you ask it more carefully to re-examine. Another call. Or you ask it to cite its sources. Another call. The actual cost per "correct answer" is 2-3x the stated cost because you're often asking twice.

Second, context. You're not calling an LLM with "answer this question." You're calling it with "here's my business context, here's my data, here's my requirements, here's what I tried before, now answer this question." That context is expensive. 50,000 tokens of context to ask a 100-token question. You're paying for that context.

Third, batch vs. real-time. Real-time LLM calls are expensive. If you batch them—collect requests, send them in bulk, process in off-hours—you can save 50% on cost. But that costs engineering complexity and doesn't work for all use cases.

Fourth, caching. GPT-4 Turbo supports caching now. So does Claude. If you cache your system prompt and common context, you save significantly. But setting this up requires rearchitecting your API calls.

By the time you've built all this, your "cost per inference" is maybe 40% what you thought based on the public API pricing.

Latency Is Not What You Think Either

Everyone assumes: "LLM calls take 3-5 seconds. That's slow but acceptable for async operations."

In production, it's worse.

Sometimes an LLM call takes 30 seconds. Sometimes it times out and you retry. Your retry logic interacts with rate limits. You hit a rate limit. You back off. Your users see a 60-second delay.

Latency also isn't consistent. Peak times are slower. Weekend traffic might be faster. In 2024, there were spikes where OpenAI's API was notably slower on certain days.

If you're building a feature where users see the latency directly—typing and waiting for an LLM response—you need sub-second latency. You won't get that from a remote API. You're looking at on-device models or a massive caching/batch system.

Most features people want to add don't actually need real-time LLM responses. "Suggest next action" can be async. "Categorize this transaction" can run in the background. But product designers always want real-time. And then you're fighting latency.

Hallucinations Don't Go Away

Everyone knows LLMs hallucinate. Everyone thinks they'll mitigate it. They don't.

Instruction-based mitigation helps. "If you're not sure, say so." Helps a bit. Your LLM will be more honest about uncertainty. It won't eliminate false confidence.

Retrieval-based mitigation helps more. Ground the LLM in actual data. "Here's the customer's account. Answer based on what you see here." Still doesn't eliminate hallucinations, but they're less about facts and more about reasoning.

The problem with hallucinations in production is that they're not random. They're systematic. An LLM hallucinated about "vendor ABC" five times this week? There's a pattern. You need to detect it, report it, and figure out why. Is it bad data? Is it a gap in your context? Is it the LLM's training data?

I've spent hours in production debugging situations where an LLM confidently gave the wrong answer. Not because it didn't understand the question. Because it learned a pattern from training data that doesn't apply to your specific situation.

The only real mitigation: human oversight. Every output the LLM produces that affects a real decision needs human review, at least initially.

Guardrails Are Fragile

You think you'll tell the LLM "only do X, never do Y." Guardrails are harder than you think.

Simple guardrails work: "never output credit card numbers." LLMs respect that most of the time.

Complex guardrails break: "suggest a vendor, but only from our approved list, and prefer vendors in this geographic region, and consider lead time, and factor in recent price changes." That's a complex set of constraints. The LLM will do its best, but it'll miss some of them. You're back to human review.

The issue is that guardrails compete with the primary task. "Suggest the best vendor" vs. "only suggest from this list." When the best vendor isn't on the list, the LLM has to balance the constraints. It's not algorithmic. There's no "right" answer to conflicting objectives. The LLM does its best, which sometimes isn't what you wanted.

Data Privacy Is a Constraint

If you're calling OpenAI's API with customer data, you're sending that data to OpenAI. You have to trust that they won't train on it. You have to trust your legal compliance on data residency. In many jurisdictions, you can't send customer data to US-based APIs.

This pushes toward on-prem or private endpoints. Your own instances of LLMs. That costs more and requires more engineering.

I've seen projects completely rearchitected because data privacy requirements made cloud LLM APIs impossible. The workaround is running your own model, your own infrastructure. It's doable, but it changes the budget equation.

Quality Degrades With Scale

I've deployed models that worked great with 100 users and degraded with 10,000 users.

Not because the model itself changed. Because the data distribution changed. When you had 100 users, they were all from your pilot customer. They used the product the way you expected. At 10,000 users across different industries, edge cases appear.

Vendors' edge cases. Your edge cases. The LLM's edge cases. They compound.

The quality you measure in closed testing isn't the quality you'll see in production at scale.

What Actually Works

The projects I've seen succeed with foundation models follow this pattern:

Narrow scope. Not "use LLMs for all analysis." Use LLMs for specific tasks where hallucination is low-cost (categorization, tagging) or where human review is baked in.

Heavy guardrailing. Not hoping the LLM will behave. Actively constraining its behavior. Structured outputs. Few-shot prompts. Grounding in real data.

Human-in-the-loop. Not autonomous. Human review on outputs that matter.

Measurement. Logging what the LLM suggested, what the human approved, where they diverged. Using that to improve prompts, retrain, or add guardrails.

Cost-conscious. Caching aggressively. Batching where possible. Using smaller models for tasks that don't need the largest model.

These aren't the flashy way to deploy LLMs. But they're the way that works.

Where We're Headed

I think 2026 will see foundation models become more boring. Less "revolutionize everything," more "use this specific tool for this specific problem." The era of trying to use one LLM for all problems is ending.

The companies shipping real value with LLMs now are the ones that treat them as a tool, not a magic solution. They measure costs. They accept human oversight. They iterate on quality. It's engineering work. Not exciting. It works.

That's actually where we are with LLMs in production. Not the revolution. The work.