Excellent analysis of why benchmarking in AI has so often go awry, by Melanie Mitchell.

I fully agree with the subtext too, which is that machine learning has not paid enough attention to cognitive science.