Switching LLM Providers Isn't Your Cost Lever
A pre-seed startup wanted to self-host Gemma 3 to save 5x before their raise. The math said otherwise. The real AI cost levers are not the provider: they are metering, caching, and cascading.
"We'll save 5x by self-hosting." I heard that last week. The math said otherwise.
I ran a cost-transparency memo on a pre-seed AI startup that wanted to self-host Gemma 3 before their next raise. The cost lever was never the provider. Here is what those workloads actually cost at published rates, all figures from public pricing right after the Gemini 3.5 Flash release on May 19th 2026.
The self-host math does not hold at pre-MVP scale
Self-hosted Gemma 3 27B on a rented H100 at $1.99 an hour only beats hosted Gemini Flash when you keep the GPU pinned at high utilization. At pre-MVP scale, 1 to 5 percent utilization on a live path, the effective cost lands at $2 to $10 per million tokens. That is worse than the hosted model they already call. The same pattern repeats across every workload they wanted to move in-house.
| Workload | The 'save money' move | What it actually costs |
|---|---|---|
| LLM tokens | Self-host Gemma 3 27B on an H100 | $2 to $10 / M tokens at 1 to 5% utilization, worse than hosted Flash |
| Audio | Whisper API, the 2023 ops-blog default | $0.006 / min vs $0.002 for Gemini Flash, 3x too expensive |
| Moderation | Rent a dedicated moderation API | 3 to 100x over the existing thumbnail call |
Break-even for self-hosting sits near 1,200 active users at their usage median, and that is after a 2x multiplier for ops, high availability, and engineering time. For a pre-seed team, that is many quarters of growth away. You would be paying more, today, for the privilege of running infrastructure you do not need yet.
The three levers that actually move the bill
The savings were real. They were just in a different place. Three things the team had not built.
Where the cost actually lives
-
Metering
A cost per user and per feature, instead of one number nobody owns. You cannot cut what you cannot see.
-
Content-hash caching
A verdict gets paid for once and served to everyone who hits the same input. Free on the second call.
-
Cascading
A cheap classifier first, the expensive model only on flagged content. Published research puts this at 20 to 60 percent savings with no quality loss.
None of these depend on which provider you use. They are properties of how you architect the workload, and they compound. Metering tells you which feature is expensive. Caching removes the repeated work. Cascading keeps the expensive model off the 80 percent of inputs that never needed it. Switching providers, by contrast, is a one-time discount that a price change erases and that costs you a migration to capture.
What a fund should actually hear
The slide
We'll switch providers and save 5x.
Collapses the moment someone asks for the eval harness.
The defensible answer
We measure what we spend, and we cut where the math says cut.
A team that has done the work.
For a fund doing diligence, the reading is simple. A team that leads with "we'll self-host and save" is showing you a slide. A team that can hand you a per-feature cost breakdown and point at the caching and cascading they already shipped is showing you defensibility. One is a claim. The other is a habit. The habit is the one that keeps paying off as they scale, because the provider will keep changing and the discipline will not.
Same shape, a different bill: The Claude Code Router Pattern is about the token cost hiding in your agent tooling. This one is about the token cost hiding in your product.
If your portfolio company is about to self-host to save money, that is exactly the memo I write as a fractional CTO: the real numbers, the actual levers, before the migration. Work with me and I will tell you where the cost really is.
Gabe Giro
Next read
The Claude Code Router PatternRelated service
Work with meStay in the loop
Practical thoughts on engineering leadership, Android, and AI. No spam, unsubscribe anytime.