Nathan Lambert (@natolambert): "In the latest open model recap, @Florian Brand (eval researcher) also broke down why he thinks the open-closed model gap presented by CAISI has methodological quirks that weren’t properly explained or addressed."

Make money doing the work you believe in

In the latest open model recap, Florian Brand (eval researcher) also broke down why he thinks the open-closed model gap presented by CAISI has methodological quirks that weren’t properly explained or addressed.

The huge Elo difference is explained by DeepSeek V4s bad score in CTF-Archive-Diamond (which was only run with a subset of the benchmark and extrapolated with IRT for V4), PortBench (a CAISI-private benchmark) and ARC-AGI-2 (with a different scoring method than the public leaderboards). The differences in these benchmark have a huge impact on the overall Elo, which can exacerbate the difference in capabilities. When using Epoch AI’s ECI, which also uses IRT over a set of different benchmarks, we see that the gap roughly stays between 3-7 months since R1:

Florian Brand and Nathan Lambert

May 16

5:02 PM

Make money doing the work you believe in

Log in or sign up