I know this story has been misinterpreted widely. But the overall lesson is to never bet on an LLM that is sitting behind an obscure API, where you don't know whether you're using the same AI model at every request you send.

Open weight LLMs might not match the performance of closed frontier AI at every task, but at least you get better predictability and don't expect wild fluctuations in performance.

Overall, when it comes to production, robustness is more valuable than benchmark results. See less