The GPT-3 Hangover: What Happened Next

By September, the GPT-3 narrative had shifted. In June, everyone was talking about the singularity. By September, people were talking about why their GPT-3 project wasn't working.

I want to be precise about what changed, because it's not that GPT-3 got worse. Our understanding of what it actually is got more accurate.

The Reliability Problem Became Apparent

In June, you'd see tweets like "I built an AI legal advisor" or "I built an AI therapist." The demos were genuinely impressive. By September, the follow-ups were "here's why it gave someone terrible advice."

The issue is that GPT-3 isn't good at reasoning. It's good at pattern matching. Those are different things. Ask it "what's 8 times 7" and it'll probably say 56. Ask it "if a farmer has 17 apples and gives 4 to his daughter and 3 to his son, how many does he have left?" and sometimes it gets 10, sometimes 11, sometimes nonsense.

The model is interpolating from billions of examples, and sometimes that interpolation leads you astray. There's no way to ask it "are you sure?" or "show your working." It just outputs what comes next in the pattern.

This seems obvious in retrospect. The model was trained on internet text, which includes correct and incorrect information in roughly equal measure. The model learned to produce text that sounds authoritative, which is different from being correct.

By September, serious deployments were all building guardrails. Not "use GPT-3 to solve this problem." Instead, "use GPT-3 to generate possible approaches, then validate them through other means."

The Overconfidence Pattern

I spent time in September looking at GPT-3 projects that failed and seeing a pattern. They all involved trusting GPT-3 to do something it couldn't quite do, but that it would confidently attempt anyway.

"Build an AI that reads contracts and identifies risky clauses." The AI would read a contract, identify clauses, claim they were risky. An 80% of the time it was reasonable. 20% of the time it was completely wrong. But it was confidently wrong, which is worse than uncertain.

"Generate legal advice based on customer questions." It would generate text that sounded like legal advice and was sometimes correct and sometimes dangerously incorrect. And a non-technical user couldn't tell the difference.

This is the fundamental problem with large language models in production: they're bullshitters in the technical sense. They generate plausible-sounding output even when they don't know the answer. For creative tasks, that's fine. For anything where accuracy matters, it's a liability.

The Practical Limitations

The more I thought about it, the clearer it became: GPT-3 is genuinely useful for specific tasks and mostly not useful for everything else.

Useful: Brainstorming variations. Generating explanations for technical concepts. Writing marketing copy. Tasks where you want plausible-sounding output and will review it yourself. The model is great at these.

Useful but requires guardrails: Classification and tagging. Q&A over documents. Tasks where you can validate the answer. The model is often right, but you can't trust it blindly.

Not useful: Reasoning about novel problems. Making decisions with consequences. Tasks where you need certainty. The model produces confident nonsense.

By September, the successful GPT-3 deployments were all in the first category or deeply embedded in the second with heavy validation. The failures were mostly in the third category or second category without guardrails.

The Cost Equation

Something else became clear: GPT-3 is expensive for heavy-duty use.

The API is metered. You pay per token. If you're processing thousands of documents a day, the costs add up. Early experiments were cheap. Scaling was expensive. Some teams found their GPT-3 costs exceeded their entire engineering budget.

This created a weird dynamic where GPT-3 was cheaper than hiring humans for low-volume tasks but more expensive than hiring humans or building custom ML for high-volume tasks.

So the sustainable business models were either "use GPT-3 for high-margin tasks where we can afford the API cost" or "use GPT-3 as one component of a larger system where it's not the only cost."

What Actually Changed About AI

Here's what September taught me about where AI actually is:

GPT-3 didn't invent anything new. It's an incremental improvement on the scaling of language models. A really significant one, but not a category change.

What it did was make the gap between "easy" and "hard" in AI crystal clear. Tasks where you just need plausible-sounding text? Solved. Language understanding? Pretty good actually. Reasoning? Impossible.

This is different from the narrative in June, which was "AI can do anything, look at these demos." The September narrative was more honest: "AI can do specific things very well, and does them in ways that are hard to predict and validate."

The Companies That Won

The companies that did well with GPT-3 by September all had a pattern: they understood the specific problem it was good at solving and built it into a larger system that handled validation.

A company using GPT-3 to generate chat support responses but with a human in the loop. A company using GPT-3 to classify documents but with a QA process that checks 10% of the output. These worked.

Companies trying to build "an AI that does X" without thinking through what happens when the AI confidently makes mistakes were still struggling.

The Recalibration

By September, the conversations had changed. Instead of "what can we build with GPT-3?" it was "what specific problems does GPT-3 actually solve well?"

Instead of "is this the singularity?" it was "what do we need to build around this to make it production-ready?"

The hype didn't die. But it got more grounded. The technology is genuinely significant. But it's not magic, and it doesn't work the way a lot of June tweets implied.

What I'm Watching Now

I'm watching to see if anyone can build a genuinely useful product on top of GPT-3 that justifies the API costs, delivers real value, and doesn't require an unreasonable amount of human validation.

The early answers suggest the answer is "yes, but in specific domains." Not "yes, broadly." Which is probably healthier for the technology anyway.

The companies that succeed won't be the ones that hype GPT-3. They'll be the ones that integrate it quietly into their product as one component of a larger system and don't make a big deal about it.

By December, I expect GPT-3 will have become less of a story and more of a tool. Which is when it'll probably be most useful.