For those of you who missed it, Gary Taubes wrote the cover story for the New York Times Magazine this past Sunday on the foibles of observational or epidemiological studies. He made all of the points that I have made in past posts about the inability of these studies to prove anything. And he elaborated on how the press misrepresents these studies and makes much more of them than they are worth.
Gary uses the hormone replacement therapy (HRT) fiasco as a case in point of how observational studies are misused. In the 1960s women began using HRT to relieve the many symptoms of menopause. Then in 1985 the authors of the Harvard Nurses’ Health Study released a report that women using HRT had only a third as many heart attacks as women who didn’t. This paper electrified the press, which reported it widely. Doctors began prescribing estrogen to women as a preventative against cardiovascular disease. Almost 20 years later studies began to emerge showing that HRT actually increased the risk for heart disease, blod clots and stroke. Now, many women entering menopause are avoiding using HRT for the one thing for which it is an appropriate treatment: the short term (several years) relief of menopausal symptoms.
Many explanations have been offered to make sense of the here-today-gone-tomorrow nature of medical wisdom — what we are advised with confidence one year is reversed the next — but the simplest one is that it is the natural rhythm of science. An observation leads to a hypothesis. The hypothesis (last year’s advice) is tested, and it fails this year’s test, which is always the most likely outcome in any scientific endeavor. There are, after all, an infinite number of wrong hypotheses for every right one, and so the odds are always against any particular hypothesis being true, no matter how obvious or vitally important it might seem.
As discussed many times on this blog, basically what happens is this: the researchers release the results of an observational study. The press picks up on the conclusions and makes hay with them. A later double-blind, placebo-controlled study shows that the hypothesis generated by the observational study was wrong. Then the press picks up that study and makes hay with it. It’s like the old (very true) saying that no matter what the outcome of a legal dispute, the lawyers always win. In the case of medical studies, the press always wins.
How many times have you seen articles with the headline saying something along the lines of Substance A shown to increase the risk of cancer? Everyone rushes to avoid Substance A, manufacturers remove it from their products, and its name becomes mud. Then a few years later comes the headline: Researchers at such and such university show that Substance A doesn’t cause cancer.
Why the flip flop? Because the first study, the one showing that Substance A causes cancer, was an observational study. The second, the one absolving Substance A, was a double-blind, placebo-controlled study.
It’s axiomatic in the research community that observational studies can lead only to the development of hypotheses. In our example above, the observational study led to the hypothesis that Substance A causes cancer. But when that hypothesis was tested, it was found wanting.
The authors of these observational studies and the journals that publish them love to issue press releases implying that there is causation when they know better. And the press loves to pick up on these releases and publish them as fact.
The catch with observational studies like the Nurses’ Health Study, no matter how well designed and how many tens of thousands of subjects they might include, is that they have a fundamental limitation. They can distinguish associations between two events — that women who take H.R.T. have less heart disease, for instance, than women who don’t. But they cannot inherently determine causation — the conclusion that one event causes the other; that H.R.T. protects against heart disease. As a result, observational studies can only provide what researchers call hypothesis-generating evidence — what a defense attorney would call circumstantial evidence.
Testing these hypotheses in any definitive way requires a randomized-controlled trial — an experiment, not an observational study — and these clinical trials typically provide the flop to the flip-flop rhythm of medical wisdom. Until August 1998, the faith that H.R.T. prevented heart disease was based primarily on observational evidence, from the Nurses’ Health Study most prominently. Since then, the conventional wisdom has been based on clinical trials — first HERS, which tested H.R.T. against a placebo in 2,700 women with heart disease, and then the Women’s Health Initiative, which tested the therapy against a placebo in 16,500 healthy women. When the Women’s Health Initiative concluded in 2002 that H.R.T. caused far more harm than good, the lesson to be learned, wrote Sackett in The Canadian Medical Association Journal, was about the “disastrous inadequacy of lesser evidence” for shaping medical and public-health policy. The contentious wisdom circa mid-2007 — that estrogen benefits women who begin taking it around the time of menopause but not women who begin substantially later — is an attempt to reconcile the discordance between the observational studies and the experimental ones. And it may be right. It may not. The only way to tell for sure would be to do yet another randomized trial, one that now focused exclusively on women given H.R.T. when they begin their menopause.
The take-home lesson from all this is that correlation does not mean causation. Just because obese people wear bigger belts doesn’t mean that bigger belts cause obesity. Just because there are more cops in high-crime areas doesn’t mean that cops cause crime. Just because you see more umbrellas when it’s raining doesn’t mean that umbrellas cause rain. But that’s the way these things would be reported in observational studies.
I’m probably beating this issue to death, but it’s really important that the limitations of observational studies be understood. And sometimes showing the farcical results of a ridiculous study allows people to understand that the results of a not-so-ridiculous-seeming observational study are still as farcical.
Let’s say I want to look at the belt sizes of obese people and correlate them to weight. I get a couple of hundred (or a couple of thousand; the numbers make no difference in observational studies) obese volunteers. I write down the belt sizes and the weights of all my subjects. I run this data through the statistical program on my laptop, and I discover that there is a direct correlation between belt size and weight. And not only is there a correlation, but the correlation is highly statistically significant. I publish my results and issue a press release pointing out my results and implying that belt size is not just correlated with obesity, but that belt size causes obesity. The press picks up on the story and publishes it under the headline: Belt size may cause fatness. Then unwitting overweight people rush out to buy smaller belts in an effort to treat their obesity.
Ridiculous as this sounds, it is no different than the observational study I reported on recently about red meat increasing the risk of death for victims of stage III colon cancer. No difference whatsoever.
As Taubes points out, in observational studies even the way the questions are posed can lead to different outcomes.
Even the way epidemiologists frame the questions they ask can bias a measurement and produce an association that may be particularly misleading. If researchers believe that physical activity protects against chronic disease and they ask their subjects how much leisure-time physical activity they do each week, those who do more will tend to be wealthier and healthier, and so the result the researchers get will support their preconceptions. If the questionnaire asks how much physical activity a subject’s job entails, the researchers might discover that the poor tend to be more physically active, because their jobs entail more manual labor, and they tend to have more chronic diseases. That would appear to refute the hypothesis.
So what should you do when confronted with an observational study appearing to show a correlation that is applicable to you? Let’s say you follow my advice and don’t worry about your intake of saturated fat, and then you read the press report of an observational study implying that eating saturated fat will increase your risk for colon cancer, what do you do?
So how should we respond the next time we’re asked to believe that an association implies a cause and effect, that some medication or some facet of our diet or lifestyle is either killing us or making us healthier? We can fall back on several guiding principles, these skeptical epidemiologists say. One is to assume that the first report of an association is incorrect or meaningless, no matter how big that association might be. After all, it’s the first claim in any scientific endeavor that is most likely to be wrong. Only after that report is made public will the authors have the opportunity to be informed by their peers of all the many ways that they might have simply misinterpreted what they saw. The regrettable reality, of course, is that it’s this first report that is most newsworthy. So be skeptical.
If the association appears consistently in study after study, population after population, but is small — in the range of tens of percent — then doubt it. For the individual, such small associations, even if real, will have only minor effects or no effect on overall health or risk of disease. They can have enormous public-health implications, but they’re also small enough to be treated with suspicion until a clinical trial demonstrates their validity.
If the association involves some aspect of human behavior, which is, of course, the case with the great majority of the epidemiology that attracts our attention, then question its validity. If taking a pill, eating a diet or living in proximity to some potentially noxious aspect of the environment is associated with a particular risk of disease, then other factors of socioeconomic status, education, medical care and the whole gamut of healthy-user effects are as well. These will make the association, for all practical purposes, impossible to interpret reliably.
This entire long, well-written article is worth taking the time to read. The part about the healthy-user effect is especially interesting. And the next time you read a report in the paper about the findings of an observational study, ignore it.
“The take-home lesson from all this is that correlation does not mean causation.”
I love this video, have you seen it?
http://www.mercola.com/townofallopath/index.htm
Hi kar-kar–
Yes, I’ve seen the video. It is excellent. Thanks for providing the link so that others can watch it.
Cheers–
MRE
Dr. Eades,
I might go one step beyond Gary Taubes’ article, and your post–which is that the medical profession should refrain from making public-health recommendations based on observational data.
This habit of medicine accounts, I think, for the mess we have with cholesterol and statins, and for the bad advice we get from government on diet. I know that you have a thing about government messing up all the time, but in this case, the root of it is the medical profession, which should know better.
Thanks for a wonderful blog!
Art D
Hi Art–
Thanks for the compliment. It’s not just the medical profession as in physicians but the scientific research community that is to blame. These articles get published and noted, then their authors gain prestige. Then Uncle Sam steps in and looks for prestigious researchers to form the panels that come up with the lousy nutritional recommendations.
Cheers–
MRE
Hello- I’ve never posted here before but have been reading through your blog- very interesting. Can you comment on South Beach Diet? Perhaps you ahve already but I couldn’t find the post. Thanks!
Hi Greg–
There’s not much to comment about re: the South Beach Diet. It’s basically a low-carb diet, despite its authors protestations to the contrary. The author has bought into all the anti-saturated-fat hype, which is understandable since he’s a cardiologist and cardiologists are eaten up with an anti-sat-fat bias. Other than his admonitions to avoid saturated fat it is basically a regular garden-variety low-carb diet.
Cheers–
MRE
Actually, what reading all of this says to me is that there usually *is* causation somewhere around when there is a strong correlation. You just can’t figure out what factor is causing what. The correlation between fewer heart attacks and using HRT did come from causation, it’s just that both the fewer heart attacks and the HRT were (apparently) caused by something else – women who were of a higher socio-economic class and went to the doctor more had both fewer heart attacks and used HRT more.
Your belt example is even easier. Higher weight does cause bigger belts. The causation was even between the right two factors; it was just backward.–
Hi Emma Anne–
You’re right. There is always a cause. It’s just difficult to figure out.
Cheers–
MRE
Whenever talking about causation vs. correlation on the internet, one picture often comes up. It’s one I’ve come to love.
http://doggrinnit.typepad.com/photos/uncategorized/2007/03/17/776pxpchart.jpg
From the picture we can tell that global warming has been caused by the shrinking number of pirates in the world.
Fitting that today is National Talk Like A Pirate Day…
Great picture. Every researcher should have it up in his/her lab.
Thanks–
MRE
As you’ve stated before, the researchers and the media like to play with numbers. If a control group of 10,000 has three heart attacks and a medicated group of 10,000 have two heart attacks over the same length of time the researchers and media like to report it as a 33% reduction. In reality the difference represents .005% of the 20,000 participants or .01% of the 10,000.
Precisely.
Nice post, Dr. M.
I read Taubes’ article earlier and forwarded it on to a few friends and family who can’t quite believe their doctors could possibly be wrong. I’m still waiting for my copy of his new book to arrive and in the meantime will continue eating my butter.
Hi Mike,
While I see no reason to exonerate the press, I believe that the process by which a questionable hypothesis becomes a received idea is more complicated than the press’ slaking its appetite for sensationalism. Why is it that people who should know better say nothing when the press touts hypotheses as truths? Many of these studies are not designed to unearth testable hypotheses but rather to flatter tenure committees and to pry funds loose from granting institutions. In neither case will the flatterers or the flatterees find any objections in the press’ running away with the story. There is much to blame here in the structure of the science biz.
Hi Chuck–
I don’t really blame the press. Other than bright science reporters most of them don’t have the knowledge to evaluate the studies as to whether they are valid or not. It’s the scientists who issue press releases about studies that they know to be observational and, consequently, worthless as being meaningful in terms of causation. These scientists know that the press will grab at these studies and proclaim them as something they aren’t. In my view it is the scientists and even the editors of the journals in which these papers are published who are to blame.
Best–
MRE
Thanks for the reply on South Beach Diet.
I am currently on a statin and trying to get”good” lipid numbers, CRP number, etc. I would like to get off the statin. I don’t have the aches buy it seems my memory is worse. I am curretly 6’2″ and 225 lbs so I think getting down to around 195 would feel better and help my lipids too. I am exercising – mostly weights 2 x per week and walking at my farm 3 mi per day.
Can you recommend a good book to help me out, one with recipes. I don’t know if South Beach will help.
My physician is no help, and there is so much confusing info out for the general public that it gives me a headache trying to understand it all and sort out personal agendas, political agendas, or just trying to sell a book to make money.
Thanks!
Hi Greg–
This is probably going to sound self serving, but why don’t you get a copy of Protein Power or the quick and easy version The 30-Day Low-Carb Solution. Both will tell you how to go on and follow a low-carb diet.
You should be trying to achieve health not get ‘good’ lipid numbers. ‘Bad’ lipid numbers aren’t a disease, much though the sellers of statins would have you believe otherwise.
Keep me posted.
MRE
Hi Dr. Mike:
I’ve commented about this before but I’m still confused on your position—you have no concerns with the inflammatory potential in saturated fat? Didn’t you agree that its tendency to produce cytokines could interfere with overall metabolic health?
I’m not heckling–just trying to understand your exact position on saturated fat. Thanks for clarifying.
Hi Jennifer–
I wouldn’t have thought you were heckling. I’m not like some self-styled experts who immediately ban anyone from their sites who asks a reasonable question of them.
As far as I know, saturated fats don’t have much ‘inflammatory potential.’ Saturated fats are neutral fats by their very makeup. The definition of a saturated fat is a fat that has hydrogens attached to all the carbons in its chain so that all the carbon-carbon bonds are single, stable bonds. When a fat doesn’t have all its carbons ‘saturated’ with hydrogens it is forced to have double bonds between one or more of the carbons. These carbon-carbon double bonds are unstable and tend to make the fat behave differently than a saturated fat. Depending on where these carbon-carbon double bonds lie on the chain, the fat can have inflammatory or anti-inflammatory activity. But since saturated fats don’t have these double bonds, they are stable and neutral.
Hope this clears things up a little.
Best–
MRE
Great post! Thanks for pointing the way to the article as I had missed it.
I spent some time in your neck of the woods last week down the road in Santa Clarita. The low-fat meme is indeed alive and well out in California as my hosts spent a lot of time talking about diet and how healthy things like potatoes are and how meat is so bad for you. It’s amazing that my tongue didn’t end up flopping around on the table, I was biting it so hard. My hostess was very enamored with Jamba Juice. I had never been in one before and spent some time flipping through the book of nutritional values while she was waiting for her order. The carb counts on those things are eye-popping, most are more than a 100 grams of carb per serving. I did get to enjoy my most favorite omelet though, as the California is far more readily available on the menu out there than it is here. 🙂
Hi Esther–
California is like a separate country because it is so huge and so diverse. And although there is plenty of low-fattery to go around, there are also pockets of good sense here and there. Hope you enjoyed your trip. Whatever else one might want to say about it, California is a beautiful place.
Cheers–
MRE
A bit off topic, but is there a way to subscribe to Gary Taubes’ articles? Is he a regular writer on the staff of the New York Times?
Another aside. It’s really too bad that it takes a JOURNALIST writing about the benefits of low-carb eating to get people to pay attention, when doctors and other experts like you, your wife, and Dr. Atkins have been saying the say thing for years. Sheeesh.
Hi Kathy–
Gary is a free-lance science journalist who writes in depth articles for a number of different papers and journals. I’ve been encouraging him to write a blog – but I don’t know whether he will or not.
The thinking is – wrongly or rightly – that journalists are seekers of the truth. They have no diet regimen to sell; they are not vested in any particular point of view. They take an unbiased look at the evidence and report what they find, whereas MD and I have a point of view, then twist the evidence to or include only evidence that does support our point of view. The truth is that most low-carb ‘experts’ became that way because they went through the journalistic search for truth on their own, which lead them to believe that low-carb diets are the most effective way to treat obesity and a host of other disorders.
In our case, MD and I didn’t just wake up one morning, look at one another across the bed, and say, hey, let’s write a book on low-carb dieting today. We had many years of practice fiddling and refining the diet with the help of thousands of patients. Our books were an outgrowth of that experience. But, somehow, people tend to think, Oh, they’re just trying to sell a diet book.
Cheers–
MRE
Forgive me for posting again, but I’m in the midst of reading the entire article by Taubes and the following caught my eye:
Again, it’s a bit off topic, but what do you think about his statement regarding sun exposure and skin cancer as a lifestyle factor that has been convincingly established as a cause of a chronic disease?
I’ve read your opinions on sun exposure (specifically in reference to sunscreens) and agree with them whole-heartedly. Perhaps the causal relationship between the sun and skin cancer isn’t quite so convincingly established? What about a possible relationship between sunSCREEN and skin cancer?
Also a bit curious about his reference to alcohol and cancers.
Thanks!
Hi Kathy–
He’s right in that sun exposure does cause a type of skin cancer that is fairly benign. And it can cause melanoma. But, strangely, it also protects against melanoma. Sun exposure is a two-edged sword in that it has both beneficial and detrimental effects. The task is to balance those so that you get the beneficial effects while minimizing the detrimental effects.
As to alcohol and cancers, there is no question that cirrhosis leads to liver cancer. And there is no question that overconsumption of alcohol leads to cirrhosis. So, alcohol is indeed a cause of much, but not all, liver cancer.
Best–
MRE
One of those pockets of good sense being in Santa Barbara, I’m sure. 🙂
Yes, I had a wonderful time! And you are right about California being beautiful. We drove down the PCH one day and stopped in at Geoffrey’s in Malibu for lunch. A lovely place and I highly recommend it if you and MD are ever in the area. The oceanside view from the dining area is spectacular.
There is probably a small pocket in Santa Barbara, though that pocket has migrated to Tahoe right now.
MD and I love Geoffrey’s and stop there just about every time we drive down to the LA area. It’s pricey but the setting is worth it. The food is pretty good, too.
Best–
MRE
Sir..is not also twoo that yr old Mucka Ricardo Veech says that ketones actually have anti-inflam properties ?
Thanks much as always yr enlightening blog
Yep, Veech has said same.
Cheers–
MRE
A different take on the global warming Vs pirates. I think the chart proves that it isn’t the loss of pirates that causes global warming but that global warming is deadly to pirates.
I wondered what happened to them all. The chart now makes it all clear.
Cheers–
MRE
Thanks for the clarification–it was very helpful.
So I suppose the studies like this one:
http://content.onlinejacc.org/cgi/content/full/j.jacc.2006.04.080v1
…provide a good example of an over-reported observational study? The power of the study is very weak using only 14 subjects.
This site is a great resource for me, Dr. Mike. I’m glad you don’t mind my questions.
Hi Jennifer–
I wrote an entire post on this very study about a year ago. (Click here to read it.)
Best–
MRE
I wonder. After reading Taubes brilliant article in the Times I checked out the debate on Charlie Rose on U-tube.Taubes was heckled badgered and scorned by the AHA dupe and Dean Ornish. Several times Ornish stated that he would be the first to change his mind when he was shown the evidence. Seems that Taubes is giving Ornish that chance at last. I am not holding my breath. But I am going to buy lots of copies of Ornish´s book for Christmas this year. Especially for all the dietitians I know.
I wish Taubes did have a biog so I could tell him thanks in person. As I again thank you for your courage and persistence in bucking the vested interests that are killing us and destroying one of the simple joys of life: eating good food.
Hi Marilyn–
Didn’t you mean that you were going to buy lots of copies of Taubes’ book for Christmas…? If so, sounds like a great gift, but don’t hold your breath that it will be read. Most people have their minds made up, and when confronted with such a book will make inane remarks along the lines of Oh, it’s just the same old discredited low-carb propaganda. You can buy them the books and send them to school, but you can’t make them open their minds.
I’ll pass along your thanks to Gary.
Cheers–
MRE
“You can buy them the books and send them to school, but you can’t make them open their minds.”
My favorite quote, along those lines, is this:
“I have provided you an explanation. I am unable to furnish an understanding.”
– John
Hey John–
That’s perfect. Who said it?
Best–
MRE
That’s perfect. Who said it?
It’s a paraphrase of a Samuel Johnson quote.
Another retort I like to pull out now an then: “Well, I’ll say this much for you – you sure don’t
embarrass easy.”
John
Thanks, John–
I’ll have to look up the real SJ quote in my small library of SJ’s writings.
Cheers–
MRE
Dr. Mike,
Another excellent, penetrating post. Thanks to your blog, I now know how to look at these types of studies and pick apart what’s wrong with them. You’ve really opened my eyes to a great many things.
The other day I came across an assessment of another observational study, evidently published recently in the journal “Obesity”. Said study looked at the correlation between alcohol consumption and “beer bellies”. Part of the findings indicate “there was a very “robust” association between alcohol intake, waist circumference and waist to hip ratio. They pointed out that a high alcohol intake, especially hard liquor, was closely associated with abdominal body fat, not just overall body mass.” Can you say “duh”?
But one paragraph caught my attention: “Another thing that confounds the reports on whether alcohol contributes to weight gain is the fact that the game changes in heavy drinkers. We know that alcohol contains 7.1 calories per gram and these calories always count as part of the energy balance equation… or do they? With chronic excessive alcohol consumption, it’s possible that not all of these calories are available for energy. Due to changes in liver function and something called the microsomal ethanol oxidizing system (MEOS), alcoholism may be a real case of where some calories don’t count.”
Huh? I already knew that while drinking the body will burn the alcohol preferentially as fuel ((to purge the toxins?), while inhibiting fat oxidation (and this is indeed mentioned in this fellow’s assessment). But why would this “calories don’t count” only apply to heavy, chronic drinkers. Or is his science, uh… impaired? Just curious, cuz I enjoy lowcarb beer in moderation (occassionally somewhat more than moderation,lol). Do I not metabolize/process the alcohol in the same way as a heavy chronic drinker? Many thanks, and again, great job!
thomas
Hi Thomas–
Heavy drinkers have more liver enzymes and can metabolize alcohol faster than you, assuming you’re not a heavy drinker. But, with time, these heavy drinkers destroy their livers and reach the point where they can’t much metabolize alcohol at all. Manny of them continue to drink, however, and it kills them.
Best–
MRE
I’ll have to look up the real SJ quote in my small library of SJ’s writings.
No need. Here it is:
“Sir, I have found you an explanation, but I am not obliged to find you an understanding.”
I think my own paraphrase is a bit of an improvement, realizing, though, that I too stand on the shoulders of giants.
John
Hey John–
I agree – I think your modernization is an improvement.
Cheers–
MRE