The LLM Wave: Separating Signal From Noise

By June 2023, the noise around LLMs is deafening. Every company has an "AI strategy." Every product has an "AI feature." Every founder is positioning their tool as "ChatGPT for X." Most of it is nonsense.

I've been paying attention to what actually works and what's hype. It's useful to have a framework for this because the noise is only going to get louder. Here's how I think about it.

The Real Constraint: Data and Context

The most important variable for LLM usefulness is data. Not the model. Not the hype. Data.

A LLM is useful when it has access to data about your specific problem. Generic LLMs like ChatGPT are trained on the internet. They know general knowledge. But they don't know your customer data, your orders, your workflows. That's where they fall apart.

So the question isn't "is GPT-4 good?" It's "does GPT-4 have access to the data it needs to solve this problem?" If yes, it might work. If no, it won't, no matter how good the model is.

This immediately filters out a lot of noise. A "ChatGPT for HR" that doesn't have access to your employee data, your policies, your history—that's not useful. A tool that pulls your HR data and uses LLMs to answer questions about compensation, benefits, policy—that's potentially useful.

The companies doing this right are not talking about "AI" or "LLMs." They're talking about their data. They're saying "we have your employee data, we've built connectors to pull it, and we're using LLMs to ask questions about it." That's the real innovation.

The Pattern I'm Seeing

The useful LLM applications follow this pattern:

1. They solve a specific, bounded problem. Not "general intelligence." Not "AI assistant." Something concrete. "Extract key dates from legal documents." "Find inconsistencies in financial statements." "Categorize support tickets."

2. The problem has a clear measure of success. If you can't measure whether it's working, it probably isn't. For document extraction: did we get the dates right? For categorization: did we sort the tickets correctly? For anomaly detection: did we catch the actual problems?

3. Humans need to review the output. LLMs still make mistakes. The good applications build that in. "The AI categorized these 100 support tickets, here are the ones it's uncertain about, please review." Not "the AI will do this automatically."

4. The alternative is clearly worse. If you have to choose between "manual work by a human" and "AI-assisted work with human review," the AI usually wins on time and cost. If you're choosing between "simple rule-based system" and "LLM," the rule-based system often wins on speed and cost.

The companies I'm excited about are using LLMs to improve existing workflows by 20-40%. Not transforming the world. Not replacing workers. Incrementally better processes.

The Hype Patterns to Watch Out For

"ChatGPT for [your industry]": This is almost always not thought through. Yes, you can wrap OpenAI's API and add a UI. That's not a product. If all you've done is add a UI on top of ChatGPT, you have no defensibility and no unique value.

"AI-powered forecasting": Forecasting is hard. LLMs are not magic. If your forecasting model is just "ask GPT-4 what will happen," you're in trouble. Forecasting requires data, methodology, and usually specialized models. LLMs can help with feature engineering or summarizing predictions. They can't replace the actual work.

"Autonomous AI agents": Every company is talking about agents. The idea that AI can operate independently in your system. The reality: autonomous systems are brittle. They break when assumptions change. They're useful in narrow, controlled environments. For most business processes, you still need humans in the loop.

"AI will make you 10x more efficient": Nope. Maybe 20-40% if you're lucky. Usually 10-15%. That's still worth building, but don't believe the hype.

"We're using cutting-edge AI": Most useful AI applications are using existing models. GPT-4. Llama. Claude. The innovation isn't in the model. It's in how the model is integrated into your workflow. The companies talking about "cutting-edge" are usually just marketing.

My Framework for Evaluating LLM Applications

When someone pitches me on an LLM application, I ask:

1. What's the specific task? If I can't describe it in one sentence, it probably doesn't work.
2. What data does it need? How do you get that data into the system? How current is it?
3. How do you measure success? What are the metrics? What's the baseline? What's the improvement?
4. What's the human role? Are people reviewing every output? Some outputs? None?
5. What happens when it fails? It will fail. How does the system recover? What does the human do?
6. What's the cost? API calls, hosting, labor. Total cost per transaction or per unit of value delivered.

If a company can answer these clearly, they might have something. If they can't, they're still in the pitch phase, not the product phase.

What I'm Watching

The most interesting LLM applications I'm seeing are on the structural integration side. Not "AI chatbot," but "AI integrated into my ERP system." Pulling data from multiple systems, understanding context, making recommendations within existing workflows.

I'm also watching open-source. Meta's Llama 2 came out in July, but by June we knew it was coming. Open-source models are approaching GPT-4 capability at a fraction of the cost. That changes the economics. Companies that were worried about API costs can now self-host.

The biggest winner will probably be whoever builds the best infrastructure for connecting LLMs to business data. Not the model itself. The plumbing. The company that makes it easy to say "use this LLM on our data and give me usable output with uncertainty quantified" wins.

The Uncomfortable Take

Most LLM startups shipping today will not survive 2025. The barrier to entry is low. Everyone has access to the same models. The only differentiation is data, integration, and UX. Most teams have one of those. Few have all three.

The LLM wave is real. But the real opportunity isn't in the LLM itself. It's in boring, hard work: getting your data clean, understanding your workflows, building integrations that actually work. That's why so many companies will fail and a few will win.

The ones talking about AI the least are usually the ones doing the most with it.