Small Language Models: The Quiet Shift

In September, something interesting happened. Nobody wrote headline articles about it. But Microsoft, Google, and Meta all shipped impressive small language models in the same month.

Phi-3 from Microsoft (3.8B parameters). Gemma 2 from Google (2B to 27B). Llama 3.1 from Meta (up to 405B, but critically, also 8B and 70B versions).

Everyone's obsessed with the big models: Claude's massive Opus tier, GPT-4's power. But the real shift is happening at the other end of the curve.

Small language models are becoming genuinely capable. And that changes everything.

The Economics Problem

For the past 18 months, the economics of LLM deployment pushed toward massive models.

You needed GPT-4 or Claude 3 Opus for good reasoning. These models cost money to run. For a startup, for a small company, for a feature buried deep in an app, the cost was prohibitive.

So you either accepted worse quality with cheaper models, or you built your feature for users who could afford expensive API calls.

That was the tradeoff.

In September, that tradeoff started to vanish.

Phi-3 runs on a laptop. Not slowly. Actually fast. And it performs remarkably well on reasoning tasks. Not GPT-4 level, but genuinely useful.

Gemma 2 7B—seven billion parameters—can be quantized down to run on mid-range phones. And it's competitive with models five times its size on many benchmarks.

Llama 3.1 8B is... actually really good. Like, I ran a coding task through it and got a better solution than I've gotten from some proprietary models.

Why This Matters For Enterprise

Here's the insight: enterprise software doesn't need GPT-4 for most tasks.

Think about the actual use cases:

Classification (is this invoice valid? is this email spam?) - a 3B parameter model crushed this.

Entity extraction (pull the vendor name and amount from this document) - a 7B parameter model is plenty.

Summarization (summarize this customer conversation) - a 7B parameter model does this well.

Reasoning about structured data (given these business rules, what's the right decision?) - sometimes you need Claude 3, but often a 70B parameter model is overkill.

For the tasks that actually require deep reasoning, you're usually combining them with retrieval (RAG). The retrieval grounds the model. A smaller model can handle the reasoning because it's not reasoning in a vacuum.

So the question becomes: why pay for Opus when Phi-3 does the job?

Three Ways Small Models Change the Game

First, cost: Phi-3 costs 1/50th what GPT-4 costs. Gemma is free (open source). Llama 3.1 8B is effectively free if you run it yourself.

For features where you're running inference thousands of times a day, this is the difference between profitability and loss.

Second, latency: small models are fast. Phi-3 responds in 200ms on a laptop. Opus responds in 1-2 seconds. For real-time features, that matters.

Third, deployment flexibility: you can run small models anywhere. On-device, in a containerized service, in a serverless function, on an edge server. You're not locked into cloud APIs.

For enterprise, this third thing is huge. You can guarantee data stays on-premises. You don't depend on an external service. You own the model.

What We're Rebuilding

We have an ERP classification system that was running on Claude 3 Sonnet via API. Every transaction got classified (is this capital expense? is this travel expense? who should approve it?).

We were running 50,000 classifications a day. Cost was $75/day on the model alone.

In September, we rebuilt it on Llama 3.1 8B running locally on our servers.

Same accuracy. Faster responses. Cost: essentially zero (just the server running cost).

But here's what we really got: now we can offer classification as a feature to clients who want to run it on-premises. Clients in regulated industries (finance, healthcare) who won't send transaction data to cloud APIs.

That feature was impossible with Opus. Now it's trivial.

The Quality Tradeoff

I need to be honest: small models aren't magic. They're not as capable as large models.

Phi-3 hallucinates more than Claude 3. Gemma makes reasoning mistakes that larger models don't. Llama 3.1 8B is great, but ask it a truly hard reasoning question and it struggles.

But here's the key insight: you usually don't need that performance.

The 99% of tasks that are straightforward? Small models crush. The 1% that require deep reasoning? Use a bigger model for those specific tasks.

So the new pattern is: start with small models. Profile where the big models are actually needed. Only use expensive models for those specific cases.

Result: 90% savings on inference cost, 95% of quality.

The Open Source Angle

Something else shifted in September: Llama 3.1 405B. That's a massive, competitive open-source model.

Meta didn't have to open-source it. They could have built a service around it. But they released the weights.

Why? Because they're betting that in a world of on-device AI, of private enterprise deployments, of "we don't want to depend on a single API provider," the moat isn't the model. The moat is being the company that makes building with models easiest.

Google's doing the same with Gemma. Releasing models so developers use them, build on them, get locked in through familiarity.

This is a seismic shift. For the first time, there are legitimately competitive open-source models competing with proprietary APIs.

What I Think Happens

By end of 2024, I expect most enterprise AI workloads to run on small models. Not because they're better, but because they're better enough at 90% of the cost.

Enterprises will keep using Claude 3 and GPT-4, but only for complex reasoning that actually needs them.

The companies that move fast will do what we're doing: measure where the large models are adding value. For 90% of the features, they're not. Migrate to small models, pocket the savings.

The companies that don't will keep paying 10x for infrastructure they don't need.

Building For Small Models

One thing I'm noticing: developers have to think differently when building for small models.

With Opus, you can throw complex prompts at it. The model handles nuance. With small models, you have to be more explicit. You have to think about what information the model needs and structure your prompts carefully.

It's more work. But it's the good kind of work. It forces you to think clearly about what the model needs to succeed.

By the end of 2024, small models won't be a compromise. They'll be the default, and large models will be the exception you reach for when you specifically need them.

The companies building that way right now are the ones that are going to have massive cost advantages next year.