New blog post coming out tomorrow morning on LLM benchmarking.

The best way to understand how LLM benchmarks are created—and how we can create a useful benchmark for our own task of interest—is to simply study details of the most popular and effective LLM benchmarks.

This post will study the following properties of a wide variety of b…