Llama and the Open-Source AI Moment

Meta released Llama 2 in July 2023, and I've spent the last month experimenting with it. The timing feels significant. By August, it's clear that open-source LLMs have reached a capability threshold where they're genuinely competitive with commercial models for many applications. That changes the calculus for how you build AI products.

Let me be specific: Llama 2 is very good. It's not better than GPT-4. It's not better than Claude. But it's nearly as good as GPT-3.5 for many tasks, and it costs next to nothing to run. That's the shift that matters.

The Old Game: API Costs

Until July, the question for builders was simple: OpenAI's API or build it yourself? OpenAI was fast to integrate, reliable, and powerful. Building your own required ML expertise. Most companies chose OpenAI.

The economics were: GPT-3.5 at scale gets expensive. If you're processing thousands of documents daily, API costs add up. You're paying per token. Companies were doing math like "this LLM feature will cost us $50K per year in API calls." That's real money. Some features got killed because the unit economics didn't work.

Now you can run Llama 2 on your own hardware. Or on cheaper cloud infrastructure. The cost per inference drops from cents to fractions of a cent. That changes what's economically viable.

What Changed Practically

I've started deploying Llama 2 in parallel with GPT-3.5 for various tasks. Here's what I'm seeing:

For structured task like text classification, sentiment analysis, entity extraction—Llama 2 is nearly indistinguishable from GPT-3.5. Accuracy is the same. Speed is comparable. Cost is dramatically lower.

For unstructured reasoning or edge cases, GPT-3.5 is still noticeably better. Ask Llama a tricky question, and it's more likely to hallucinate or miss nuance. But for 80% of real-world tasks, it's fine.

Here's what matters: you can now deploy Llama 2 on your infrastructure and maintain full control. No API calls. No rate limiting. No surprise pricing changes. You own the model.

The Implications

For businesses self-hosting is now viable. You can run Llama 2 on standard cloud infrastructure. If you're already hosting your application, adding a Llama 2 instance isn't a huge lift. You get to avoid API dependency.

That's huge for regulated industries. Healthcare, finance, government. They were nervous about sending data to OpenAI's servers. Now they can run open-source models internally. Data never leaves your infrastructure.

It's also huge for cost-sensitive applications. If you're building a product for emerging markets or price-sensitive customers, API-based AI is a non-starter. Open-source models let you offer AI features at lower prices.

The third implication is reduced vendor lock-in. With OpenAI, you're locked into their model, their pricing, their availability. With open-source, you can switch models. Llama 2, Falcon, Mistral—they're all available. If one stops working, you can swap for another.

What This Means for Enterprise

For large enterprises building internal tools, this is transformative. You can now add AI to your ERP, your CRM, your internal tools without worrying about API costs or data privacy. You build once, deploy internally, and scale for pennies.

I'm already thinking about this for Arvension. Document processing in ERP is expensive with GPT-3.5 API calls. With Llama 2, we can offer this as a feature without blowing up unit economics. That changes what's possible.

The other change: enterprises can fine-tune models. GPT-4 API doesn't support fine-tuning yet. But you can fine-tune Llama 2 on your specific data. Your ERP workflows, your document formats, your language. A model tuned to your business is more useful than a generic model.

The Caveats

Open-source isn't a free lunch. Someone needs to deploy and maintain the infrastructure. If you're not infrastructure-heavy, OpenAI's API is still easier. They handle updates, scaling, availability. Open-source requires ops work.

Also, Llama 2 requires bigger hardware than GPT-3.5 API calls. The 70B parameter model is good but requires multiple GPUs or significant CPU. If you need instant on-demand availability, API calls might still be cheaper than maintaining infrastructure.

And the models move fast. By the time you've integrated Llama 2 deeply into your product, there might be a better model. You have to stay current. That requires investment.

The Strategic Question

The question I'm asking now: what's the right model for our use case? GPT-4 API for high-value, reasoning-heavy tasks? Llama 2 self-hosted for high-volume, lower-cost tasks? A mix?

For Arvension, I'm leaning toward a hybrid. High-judgment workflows use GPT-4 via API for accuracy. High-volume workflows like document classification use Llama 2 self-hosted for cost. Both routed through the same interface so customers don't care which model handles their request.

This flexibility is new. In 2022, you had one choice: commercial APIs or build it yourself. Now you have options. You can mix and match.

What's Coming Next

By the end of 2023, there will be more open-source models, probably as good or better than Llama 2. Llama 3 is coming. Mistral is improving. The pace of improvement in open-source AI is accelerating.

What that means: the "build vs. buy" decision for AI becomes more nuanced. You're not choosing between OpenAI or in-house. You're choosing between different models, different architectures, different deployment patterns. You evaluate on accuracy, cost, latency, privacy, and control.

That's better for builders. Competition is good. Open-source AI is real now. It's not the future—it's August 2023, and the future is here.