I enjoyed this podcast from Stuart Ritchie and Tom Chivers (they mention me!)

At the end they talk about whether what they try to do on the podcast is “critical thinking”. EG apply domain knowledge about good research, bad research, p-hacking, so on and so forth.

I think there is valuable domain knowledge about good research practices which can be applied to different domains.

But I also think that you still need knowledge of the domainbeing researched.

This was the point I was trying to make in a recent tangent about whether Don Bradman is the best sportsman ever. I remember being a teenager and realising someone had used the standard deviation to compare sportspeople from different eras and sports and I was blown away by it.

So yes, concepts like the standard deviation and effect sizes are incredibly powerful (and are domain knowledge in their own right). They are so powerful because they are abstract.

But that is what also makes them dangerous. People can apply them with zero understanding of the underlying domain being researched and come up with ludicrous conclusions.

This is the problem with a lot of really bad meta-analyses. They yoke together totally different studies with totally different structures and spit out a number at the end and pretend it is definitive.

(Tangent on Bradman and SDs here substack.nomoremarking.…)