Since starting this blog I’ve posted a few times (here, here, and here) on the credibility or lack thereof in medical and scientific journal articles. Today’s New York Times features a long article on this very subject that is well worth reading in its entirety. It’s one of the few articles from the New York Times with which I agree completely.
I’ll excerpt the article and let it speak for itself. Click here, though, and read it completely.
Recent disclosures of fraudulent or flawed studies in medical and scientific journals have called into question as never before the merits of their peer-review system.
Because findings published in peer-reviewed journals affect patient care, public policy and the authors’ academic promotions, journal editors contend that new scientific information should be published in a peer-reviewed journal before it is presented to doctors and the public.
That message, however, has created a widespread misimpression that passing peer review is the scientific equivalent of the Good Housekeeping seal of approval.
The release of news about scientific and medical findings is among the most tightly managed in country. Journals control when the public learns about findings from taxpayer-supported research by setting dates when the research can be published.
Here the author points out the same case I have made many times:
Increasingly, journals and authors’ institutions also send out news releases ahead of time about a peer-reviewed discovery so that reports from news organizations coincide with a journal’s date of issue.
A barrage of news reports can follow. But often the news release is sent without the full paper, so reports may be based only on the spin created by a journal or an institution.
Even the system’s most ardent supporters acknowledge that peer review does not eliminate mediocre and inferior papers and has never passed the very test for which it is used. Studies have found that journals publish findings based on sloppy statistics. If peer review were a drug, it would never be marketed, say critics, including journal editors.
A widespread belief among nonscientists is that journal editors and their reviewers check authors’ research firsthand and even repeat the research. In fact, journal editors do not routinely examine authors’ scientific notebooks. Instead, they rely on peer reviewers’ criticisms, which are based on the information submitted by the authors.
Editors say they have accepted a number of papers that reviewers have harshly criticized as unworthy of publication and have rejected many that received high plaudits.
I have found this last to be half true in my own situation. I have served as a ‘peer’ in the review process for a scientific journal, and of all the papers I have savaged, none were published. But several that I thought were terrific or needed only slight changes never saw the light of day.
Many nonscientists perceive reviewers to be impartial. But the reviewers, called independent experts, in fact are often competitors of the authors of the papers they scrutinize, raising potential conflicts of interest.
The analogy is often made of Ford being in the position of ”reviewing’ GM’s cars and deciding if they should be allowed to be made and sold.
Journals have devolved into information-laundering operations for the pharmaceutical industry, say Dr. Richard Smith, the former editor of BMJ, the British medical journal, and Dr. Richard Horton, the editor of The Lancet, also based in Britain.
The journals rely on revenues from industry advertisements. But because journals also profit handsomely by selling drug companies reprints of articles reporting findings from large clinical trials involving their products, editors may “face a frighteningly stark conflict of interest” in deciding whether to publish such a study, Dr. Smith said.
These last charges are the most serious as far as I’m concerned. I posted earlier about how many medical journals had become publishing arms of the pharmaceutical industry.
All the above makes me look at each article I see with a jaundiced eye and makes me pour over the thing hypercritically. I read scientific/medical articles constantly, and I can tell you, well written, well argued papers that include decent, statistically relavent data are scarce as hen’s teeth. This terrific New York Times piece tells us why.
I wonder how many people have suffered because of this…
Probably a lot.
Great post Mike. As you know, exercise ‘scientists’ are among the worst offerders. But at least they don’t kill people.
I have been pointing out the junk science status of mainstream science for some time, and indeed peer review is one of the two biggest problems I tend to cite, as well as the most systematic.
The main problem is that it is a censorship board which must, inherently, tend to (consciously or otherwise) be an advocate for the status quo. The bar tends to be set higher for unconventional ideas, and lower for accepted ones. This results in a system where scientific progress is retarded, and errant conventional wisdom is retained overlong, or even promoted because it is politically correct but starts with no scientific support at all.
When I was a consultant in Washington, DC, I had a friend who was head scientist on a key weather-related scientific project. He complained to me, regularly, that he was finding specific evidence against “global warming” and human-caused ozone depletion. We would discuss, in detail, why these two things were false, from a scientific standpoint.
And yet nothing he published contained any of this information. In fact, he was forced to focus his attention on the data which, taken in isolation from the parts he excluded, could be broadly interpreted as circumstantial support for human-caused global warming.
Why?
Peer review. If he published the data without massaging it to support global warming and/or ozone depletion– nevermind actually presenting the logical conclusions to which they led — his work would be rejected by peer review, and as a result his funding cut, and probably his job endangered.
The implication of the article, and presumably most critics of peer review, is that it should be fixed. That authoritarian censorship is somehow necessary, even good, and it’s simply not been done right so far.
But, in reality, authoritarian censorship is always wrong and harmful.
As Thomas Jefferson once pointed out, “Error of opinion may be tolerated where reason is left free to combat it.”
Peer review undermines both aspects of this fact. It attempts to ban “error”, claiming to be intolerant of it…and yet of course it also leaves reason unfree to combat error which is already favored by the system.
But peer review is, in effect, what got Galileo condemned for claiming the sun centered our system instead of the earth.
One need only examine ideas which currently have evidence as significant as the conventional wisdom, yet which are nearly banned from peer-reviewed publication, to see how the system undermines progress. These ideas may, or may not, actually be correct. But they are at least arguable, and yet are — or were until recently — effectively banned by peer review.
Examples, off the top of my head, include nanobes and nanobacteria — completely dismissed by the establishment because of a purely arbitrary decision that 200nm was the minimum size of cellular life. It took a decade of snowballing evidence for a few publications to finally allow any counter-argument into print, and yet the evidence is indeed snowballing, and it may well turn out that the largest source of biomass on the planet was ignored by conventional wisdom for decades. Anyone presenting evidence of cellular life below 200 nanometers was told by peer review “we’re sorry, this is not possible; we all know it” and, in effect, censored.
Likewise, in anthropology, for years anyone who even simply said “I looked beneath the 12,000 year stratum and found human artifacts” was interrupted with “but why did would you look below 12,000? We all know that humans did not live in the Americas before that time”. And it didn’t matter what they’d found; it was assumed to be bad science. Yet now evidence is snowballing that European and Afro-australian humans settled the Americas as many as 20,000 years ago, and were simply displaced 12,000 years ago by humans from Asia.
There are other cases, without as much evidence in their favor /yet/, but which have enough evidence to not be rejected, or at least an utter lack of counter-evidence to justify dismissing out of hand. The influence of semi-aquatic lifestyle on human evolution, for example. There is not actually any evidence for the famous trees-down theory of human evolution purely on dry plains — neither hard evidence nor even circumstantial, but because it was accepted first, no alternative is even considered. You absolutely could not get an article seriously considering the aquatic ape hypothesis published, period. Your methodology and logic, good or bad, would be irrelevent. Yet any assumption of a plains origin in an article is simply accepted without examination, because it’s the conventional wisdom.
Peer review is censorship. It doesn’t matter if the motive is good, same as any other censorship.
—
Words of the Sentient:
The Constitution on which our Union rests [should] be administered…
according to the safe and honest meaning contemplated by the plain
understanding of the people of the United States at the time of its adoption
— a meaning to be found in the explanations of those who advocated, not
those who opposed it, and who opposed it merely lest the construction should
be applied which they denounced as possible.
— Thomas Jefferson: Reply to Address, 1801.
E-Mail: KazVorpal@yahoo.com
Yahoo Messenger/AIM/AOL: KazVorpal
MSN Messenger: KazVorpal@yahoo.com
ICQ: 1912557
http://360.yahoo.com/kazvorpal