Not breaking news: many scientific studies are ultimately proved wrong!

When a theory is shown to be incorrect or a publication in error, it is all too easy to think that the scientist who came up with this theory is a liar or a dishonest fraudster intent on misleading the public for personal gain. Or as Richard Smith, former editor of the British Medical Journal, puts it:

Most scientific studies are wrong, and they are wrong because scientists are interested in funding and careers rather than truth.

Smith goes on to talk about this in the context of a talk given by the "brilliant" and "cuddly" John Ioannidis, professor of medicine at Stanford (while Smith assures us that he would never describe a woman as cuddly, even if she were). Ioannidis has published a report in PLOS Medicine entitled "Why most published research findings are false". Ioannidis' theory is that most scientific studies are wrong as a result of bias and random error, based on "simulations that show for most study designs and settings, it is more likely for a research claim to be false than true".

As a research scientist, albeit in biophysics rather than medicine, it seems to me that Ioannidis' claim describes scientific business as usual. That most scientific studies are ultimately wrong is normal for science. There are more theories in the graveyard of science than theories that stand the test of time. Why? Because new data is always emerging and theories have to be adjusted. Theories are only as good as theories are, until new data comes along and ruins them. Theories give a best guess at what is going on based on things we observe (data), but they are not immutable. If you only have a few data points, then your working theory is more likely to turn out to be wrong. This is not news to science, this is science. Ioannidis goes on to say:

Published research findings are sometimes refuted by subsequent evidence, with ensuing confusion and disappointment.

I am not sure who exactly is confused and disappointed, because this is how science works. The great thing about science is its ability to be flexible enough to change when it needs to. In science, if all the sudden "black" really does become "white", theories must be adjusted. Individual scientists themselves, being human beings, might have trouble adjusting and be disappointed; but if they know how science works they shouldn't be confused. In science if you can't adjust your theories to new data, the body of science will just, eventually, leave you behind. Einstein famously refused to believe in some aspects of quantum mechanics, and the field moved on without him.

Ioannidis has a theory based on data that these incorrect or "false" investigations are due to bias and random error, and there is "plenty of both". This may or may not turn out to be the case. This theory is, exactly like all of the studies that Ioannidis investigates in his paper, a hypothesis based on some data. When there is new data his theory may prove to be robust, or he may too be disappointed and confused when new data shows that his findings were false.

While Ioannidis does have some good suggestions for eliminating bias and relying more on large studies (more data), and reproducibility by separate laboratories to form proper theories, it is hard to see what is new about this. Science has always worked this way. The more data, the more reliable the theories. Given that, as Smith puts it, linking genes with diseases is a huge area of growth, it is unsurprising that in a newly emerging growth area many, if not most, of the initial working theories may turn out to be wrong.

Leaving aside the details of how Ioannidis quantifies bias, which is somewhat opaque in his article, it is not clear that this "bias" is due to scientists being predominantly "careerists", as Smith proposes. I am not saying there aren't scientists who are careerists, or who even possibly do it "for the love of beautiful women", but the data doesn't seem to support the theory that the majority of scientists are more interested in funding and fame than "truth". What the data shows is business as usual: that scientists can be biased (not news), and that most scientific theories, in the end, are thrown on the garbage heap.