METR’s work is illuminating, but I’ve been disappointed by the tendency to use it to jump to conclusions about superhuman AI being imminent. As Joel Becker and others at METR themselves point out, it’s very hard to say what it teaches us. We need a much more serious debate about METR’s methodology, from researchers spanning AI, social science, and philosophy of science. The most central chart in AI must be properly studied.