Make money doing the work you believe in
OpenAI says it has discovered fresh inference optimizations that slash the expense of running its models by over half.
Per The Information, engineers said this month the techniques let them serve ChatGPT to logged-out visitors on just a couple hundred Nvidia GPUs at one point.
The exact method isn’t confirmed. Could be quantization, KV caching, batching, routing easy queries to cheaper models, or some combination.
But the business story beats the technical one. OpenAI closed Q1 at a 39% gross margin and is targeting 52% by year-end. Cheaper inference gives it room to lift margins, raise ChatGPT usage limits, or ease API pricing pressure on devs.
Increasingly, OpenAI’s moat is inference efficiency and cost, especially up against Anthropic.
Jun 30
at
4:11 PM