We’ve all seen the headlines.  Statins improve bone health.  Statins prevent cancer.  Statins make us smarter.  Low-fat diets improve longevity.  All these headlines and others like them are followed by articles describing studies seeming to show that subjects taking certain medications (usually statin drugs, it seems) or following a particular diet have improvements in health and/or longevity.  The promise of these articles is that if we all take the medication or follow the lifestyle choice, we, too, will reduce our risk of [fill in the blank] or live longer.  But will we?

Maybe so.  But not for the reason most people think.

The adherer effect demonstrates that people who adhere to medical or lifestyle regimens end up with better outcomes than those who don’t…even if the regimens are nothing but placebo.

I mentioned this phenomenon in an earlier post.

Almost thirty years ago a study was published in the New England Journal of Medicine looking at this very idea. [The adherer effect]  The study that inspired the article didn’t start out looking at this idea, but one of the investigators noted a key piece of the data and published on it.  The study was looking at clofibrate, a pre-statin cholesterol lowering drug,  and all cause mortality.  Subjects were randomized into two groups – those in one group got the drug, those in the other got the placebo.  After the subjects were on either the drug or the placebo for five years, researchers calculated the mortality from the number of deaths in each group.  Turned out that the five-year mortality of those on clofibrate was 20.0 percent whereas the five-year mortality of those on the placebo was 20.9 percent, or essentially the same.  Taking the drug was no different than taking the placebo, i.e., the drug was worthless. Had one of the researchers not looked a little closer, that would have been the end of the story.

When the data were looked at from the perspective of how many people actually took the drug as prescribed, the researcher discovered that those subjects who took at least 80 percent or more of their clofibrate had a five year mortality of only 15.0 percent, substantially less than the overall five-year mortality.  Those who took their clofibrate sporadically had a five-year mortality of 24.6 percent, significantly higher than those who took it as directed, a piece of data that would seem to confirm the efficacy of clofibrate.  Right?  Not necessarily.  Let’s look at compliance with the placebo.

Turns out that those subjects on the placebo who regularly took their placebo had a five-year mortality of 15.1 percent while those who took their placebo sporadically had a five-year mortality of 28.3 percent.  What this study really showed was that there is something intrinsic to people who religiously take their medicine that makes them live longer.  There was no difference between the drug and placebo in either those who took them regularly or those who took them sporadically, but there was a huge difference in mortality between those who took either drug or placebo on schedule and those who didn’t.

Gary Taubes discussed this same study and the adherer effect in  a long article he wrote for the New York Times Magazine a few years ago:

A still more subtle component of healthy-user bias has to be confronted. This is the compliance or adherer effect. Quite simply, people who comply with their doctors’ orders when given a prescription are different and healthier than people who don’t. This difference may be ultimately unquantifiable. The compliance effect is another plausible explanation for many of the beneficial associations that epidemiologists commonly report, which means this alone is a reason to wonder if much of what we hear about what constitutes a healthful diet and lifestyle is misconceived.

The lesson comes from an ambitious clinical trial called the Coronary Drug Project that set out in the 1970s to test whether any of five different drugs might prevent heart attacks. The subjects were some 8,500 middle-aged men with established heart problems. Two-thirds of them were randomly assigned to take one of the five drugs and the other third a placebo. Because one of the drugs, clofibrate, lowered cholesterol levels, the researchers had high hopes that it would ward off heart disease. But when the results were tabulated after five years, clofibrate showed no beneficial effect. The researchers then considered the possibility that clofibrate appeared to fail only because the subjects failed to faithfully take their prescriptions.

As it turned out, those men who said they took more than 80 percent of the pills prescribed fared substantially better than those who didn’t. Only 15 percent of these faithful “adherers” died, compared with almost 25 percent of what the project researchers called “poor adherers.” This might have been taken as reason to believe that clofibrate actually did cut heart-disease deaths almost by half, but then the researchers looked at those men who faithfully took their placebos. And those men, too, seemed to benefit from adhering closely to their prescription: only 15 percent of them died compared with 28 percent who were less conscientious. “So faithfully taking the placebo cuts the death rate by a factor of two,” says David Freedman, a professor of statistics at the University of California, Berkeley. “How can this be? Well, people who take their placebo regularly are just different than the others. The rest is a little speculative. Maybe they take better care of themselves in general. But this compliance effect is quite a big effect.”

In the same blog post of mine I linked to above, I wrote about another study showing the adherer effect, showing graphically how potent the phenomenon is.

Previously, the study of the adherer effect has been a secondary finding in studies of various drug regimens, but now comes a paper in which the adherer effect is the primary focus of the investigation.  Based on the data in this recent paper, the effect is robust and should be accounted for in the analysis of any data generated when subjects following a particular treatment are compared to those who don’t.

The authors lay out the problem:

The healthy-user effect [the adherer effect] is a hypothetical source of confounding bias that is thought to affect observational studies of drugs, diets, screening procedures, and other health-related behaviors. This bias presumes that patients who initiate and adhere to preventive therapies are more likely to engage in behaviors consistent with a healthy lifestyle than are patients who do not initiate or adhere to such treatments. Aspects of a healthy lifestyle could include diet, exercise, moderation of alcohol, and avoidance of risky behaviors. These characteristics, which are unmeasured in typical pharmacoepidemiological databases, may be associated with morbidity and mortality outcomes in observational studies. Thus, failure to adjust for them can lead to bias in studies of preventive therapies.

The healthy-user bias has been suggested as an explanation for the discrepancy between several experimental and observational studies, including studies of the effects of long-term use of estrogen therapy and vitamin E. It has also been discussed as a potential source of bias in observational studies of the effectiveness of influenza vaccines in the elderly  and the association between use of 3-hydroxy-3-methylglutaryl coenzyme A reductase inhibitors (statins) and reduced risk of hip fracture,  Alzheimer disease,  sepsis,  cancer,  and mortality.  This bias has also been observed in randomized controlled trials in which adherence to placebo was found to be associated with decreased mortality.  Although long suspected as a source of bias, a paucity of empirical data exists on the healthy-user effect.

Here’s how the study was set up.

It was really pretty simple.  The researchers looked at a group of patients who had been prescribed one of a variety of statin drugs and followed their compliance by looking at how many times these subjects picked up their medicines in the year following their prescription.  The typical statin prescription was for 60 days worth of the medication, and all subjects had available to them a full year’s worth of medicines.  The researchers grouped subjects into two groups: one group who took the trouble to go get over 120 day’s worth of the medication (the “more adherent”) and one group of subjects who were dispensed under 120 days of meds (the “less adherent”).

All subjects entered into the study were evaluated after the one year baseline study period during which their effort to follow their prescribed statin regimen sorted them into the categories of more adherent or less adherent.  The researchers were looking to see which subjects – the adherers or the non-adherers – would develop problems that had nothing to do with the statin drugs.

We evaluated a spectrum of events after the 1-year baseline period to assess the healthy-adherer bias. The outcomes were grouped into 4 broad categories: accident events, screening events, other events not expected to be associated with statin exposure, and other events for which a possible association with statin exposure could be expected. We included inpatient and outpatient events as well as primary and secondary diagnoses.

When the data on these 141,086 subjects was crunched, it turned out that the more adherent subjects had significantly fewer accidents, especially motor vehicle and workplace accidents.  The more adherent also had a lower likelihood of developing other disorders that were not likely to be attributed to the effect of the statin drugs.

In other words, whatever characteristic it was that made subjects hang in there with their statin prescriptions also made them less likely to indulge in risky behaviors and less likely to develop all kinds of medical problems.  Why?  Probably because these people were simply more health conscious, kept themselves in better shape, and didn’t act impulsively.

The real take-home message from this study is that the adherer effect significantly affects the outcome of drug and lifestyle intervention studies.  If you see a study that says those subjects using statin drugs developed 20 percent fewer problems (of whatever kind are being studied) than those who don’t use statins, you can be sure that the adherer effect is at work.  This adherer effect is why randomized, double-blind, placebo-controlled studies are needed to determine the efficacy of any drug, and even then the adherer effect should be controlled for.

There is a big note, enclosed in a box and titled Clinical Perspective, at the end of this study that exhorts doctors to consider this adherer effect when looking at data from observational studies.  Here is the note in full.

Clinicians need to read observational studies reporting surprising benefits of drug therapy with a healthy skepticism. Observational studies of preventive medications and health behaviors are susceptible to various sources of bias, including the so-called healthy-user and healthy-adherer biases. In this article, evidence of the healthy-adherer effect is demonstrated by showing that adherence to statins is associated with a reduction in the risk of accidents (eg, workplace or motor vehicle), outcomes that would not be expected to be affected by a statin. The approximate magnitude of the adherer effect was a 15% relative risk reduction. The most likely explanation for this association is that good adherence to statin therapy is a marker for other healthy behaviors, most of which cannot be accounted for in this type of study. In keeping with this explanation, the study also shows that adherence predicts a 7% to 17% increased incidence of medical screening procedures (eg, fecal occult blood testing, mammography). Risk of myocardial infarction, which has been demonstrated to be reduced by statin therapy in randomized placebo-controlled trials, was found in this study to be reduced by 28%. This observed relative reduction must be interpreted as reflecting a combination of the healthy-adherer effect and the drug effect. Clinicians can also learn from this study that patients who follow their advice are also likely to have other healthy behaviors and a lower risk of adverse events.

It is unfortunate, but I doubt that many doctors (or researchers, for that matter) will consider the adherer effect when they read these studies.  I would bet that we will continue to see studies reported as if the positive effects found were a function of the drug or lifestyle regimen studied and not the adherer effect.

To me the saddest part of this study was the statistic that of the 141,086 subjects in this study, 49 percent were women.  The randomized, double-blind, placebo-controlled studies of statins have never shown a benefit in terms of decreased all-cause mortality in women of any age.  Which means that over 70,000 women in this study took a drug that would do them no good, but which could well cause them significant and harmful side effects.

In this study, those who dropped out of their statin regimen because of intolerable side effects would be considered to be less adherent or non adherers.  My guess is that many of these ‘non adherers’ who dropped out because of side effects were really ‘adherers’ by nature.  Had these drop outs due to side effects been controlled for, I would bet that the difference between the less adherent and the more adherent would have been much larger than the data showed.

The news is abuzz with reports of the latest study to come out showing that eating meat, especially red meat, kills us off before our time.  (You can read some of the reporting here, here, here and here.)  Google shows 547 new articles about this study.

Although this study is totally worthless from a causality perspective because it is an observational study, it does serve to confirm the biases of those non-critical thinkers who have already bought into the idea that meat is bad.  To give you an example of such a soft thinker, here is the second comment on the blog post about this study in the New York Times.

I could have told you that 30 years ago. I been a vegetarian for 47 years and I have never seen vegetarians die from heart disease or cancer. They died from basic infectious diseases and malnutrition. Make no mistake it is harder to be a vegetarian than a carnivour but your body does not expel everying [sic] that is in the meat especially red meat.

Red meat is the major culprin [sic] in colon cancer. I actually know people who have colon cancer gene that only eat a no red meat diet and have no issues with their colon. Of course they also do not smoke or drink too much alcohol.

Red meat lobby is very powerful in America – Let them pay for this!

Ah, yes, an enlightened cogitator indeed.

The study published in the Archives of Internal Medicine (free full text here) is a typical epidemiological or observational study.  The reports have it tarted up with a lot of fancy clothes, but it is really nothing but an observational study.  And, as we’ve gone over ad nauseum in these pages, observational studies can’t be used to prove causation.

Even if they could, this study would be questionable at best because the relative risk (RR) is slightly over 1.0.  Because of the nature of the difficulty in doing these kinds of studies with any kind of accuracy it takes a RR of over at least 2.0 to get the serious attention of anyone who doesn’t have a built-in bias.

What I found more interesting than this study (which isn’t interesting or important at all) was the press coverage of it.  And I found especially interesting that which the press didn’t report.

Scientific journals have a couple of ways of getting their articles out there ahead of publication, so that the press can do stories on them.  If it works out right, the reports all hit the media on the same day that the article itself is published.  Doctors who read the journal often find out in their morning newspaper about a new paper before they even get their journal in the mail later that same day.  Many of the larger journals, Archives of Internal Medicine, for example, will issue press releases the week before on those papers coming out that the editors feel are important.  These press releases go to anyone with press credentials (I even get them), and are embargoed until the date of publication of the journal.  Reporters get advanced copies of the papers and get the editor’s (and maybe even the author’s) take on the paper.  Journalists can then write their stories to be timed with publication of the paper.

Another way followed by a number of journals is to publish papers online in advance of their actual publication date.  Reporters troll these advanced online articles looking for material for stories and often write them up for publication before the paper in question makes it into actual publication in the journal.

At the same time that this paper appeared, showing increased red meat consumption to be tied to a slight increased risk of death (and showing that those subjects eating white meat had less risk), a couple of other papers came out in the online pre-publication section of the American Journal of Clinical Nutrition (AJCN), arguably the world’s most prestigious nutritional scientific journal.  These two AJCN papers saw the light of day at around the same time as this highly-publicized study on meat and mortality, but demonstrated the opposite results.  They got no press coverage whatsoever.  Which proves what I’ve been saying all along: the press is biased against meat in general, and especially against red meat.  Knowing this, careful readers should take anything negative thing the media reports about red meat with an enormous grain of salt.

Let’s look at the other two studies published in AJCN.

The first is titled Meta-analysis of animal fat or animal protein intake and colorectal cancer.  One of the constant themes anti meat people like to hammer out is that meat intake, especially red meat intake, causes colon or colorectal cancer.  This is heard so often that most people take it for granted, assuming that there must be a ton of research backing it up.  As this paper points out, there isn’t.

The association between total dietary fat, including fat constituents such as saturated fat, monounsaturated fat, polyunsaturated fat, and cholesterol, and risk of colorectal cancer has been evaluated in numerous epidemiologic [observational] studies.  Results from these analytic investigations have generally been mixed.  Whereas some studies have reported positive associations, several studies have observed null and inverse associations.  In a pooled analysis of data from 13 case-controlled studies, risk of colorectal cancer was found to increase significantly with increasing categories of total daily energy intake.  In the same analysis, and after adjustment for total energy intake, the authors observed no evidence of an energy-independent effect of total dietary fat or specific fat components other than cholesterol.  In fact, many of the associations among men and women were in the inverse direction [i.e., more animal fat equals greater longevity].

Animal foods and meat products contain both saturated and unsaturated fats; however, similar to analyses of total fat intake, several studies have not observed any consistent epidemiologic evidence of an association between saturated fat or polyunsaturated fat intake and risk of colorectal cancer.  Although some studies reported positive associations for consumption of saturated fat, nonsignificant associations at or near the null value [no association] or inverse associations have been observed in numerous cohort studies and case-control studies.

This paper goes on to discuss how the hypothesis that fat and meat intake are a bad thing healthwise got kicked off way back in the 1960s from a presentation at a symposium. In shades of Ancel Keys and his discredited Seven Countries Study, a researcher named Ernst Wynder used the international food and cancer mortality data to demonstrate an increase in colorectal cancer as a correlate of increasing oil and fat consumption.  The hypothesis, although never proven, has been with us since.  The authors of this paper set out to study it once again.

Here is what they did:

To clarify the potential association between animal fat intake and colorectal cancer, we conducted a meta-analysis of prospective cohort studies in which data for animal fat were available.  In addition, we identified case-control studies that reported results for animal fat intake and combined data from these studies with the prospective cohort data in separate analyses.  Because the primary macronutrients in the consumption of animals include protein and fat, we also conducted a separate meta-analysis of prospective cohort studies in which data categorized as animal protein or meat were available.

After sifting through all this data, what did the authors find?  Absolutely nothing.  No correlation between meat and/or fat intake and colorectal cancer.

In this meta-analysis, no consistent evidence of a positive association between  consumption of animal fat and colorectal cancer was observed.  Specifically, we found no association  between the highest animal fat intake category and colorectal cancer.  Furthermore, none of the subgroup analysis (i.e., sex, anatomic tumor site, and study design) indicated positive patterns of associations.

And their conclusion:

On the basis of the results of this quantitative assessment, the available epidemiologic evidence does not appear to support an independent association between animal fat intake or animal protein intake and colorectal cancer.

Like the study above showing the slight correlation between red meat intake and decreased longevity, this study is an observational study, and, as such, doesn’t demonstrate any kind of definitive proof.  But what I find galling is that the meat and mortality study hit all the airwaves and this study – made available to the media at the same time – received zero press.

Yet another study in the advanced online section of AJCN titled Mortality in British vegetarians: results from the European Prospective Investigation in Cancer and Nutrition (EPIC-Oxford) shows that things ain’t always as they seem.  Yet the press refuses to pick up and report this man-bites-dog story.

If you ask the man on the street (who has been fed a load of bunkum over the years by the press) if vegetarians or non-vegetarians are healthier and live longer, you will almost assuredly be told that vegetarians are the healthiest.  Most people believe this, but they just don’t want to make the sacrifice to follow the vegetarian lifestyle.  They are willing to give up a couple of years of life to not have to live on a steady diet of beans, tofu, vegetables, fruits and dry bread.  You would think that if a study came out from a prestigious institution (Oxford) published in a top-line scientific journal showing that vegetarians don’t live any longer than non-vegetarians and actually have a higher incidence of some particularly nasty cancers (but slightly lower rates of death from heart disease) it would be newsworthy.  But the press has totally ignored this study just like they did the last one.

This vegetarian study was interesting on a couple of levels.  Not only did it not show a difference in longevity between vegetarians and nonvegetarians, it showed major increases in longevity just from being in the study.  Not long ago I wrote a post about a statin study in which I discussed the adherer verses the non-adherer effect.  A number of studies have shown that subjects who take all their medicines as directed – even the placebos – live longer and/or do better than those who takes their medications irregularly.  There is something about people who go the extra mile that makes them live longer than those who don’t.

In this Oxford University vegetarian study, vegetarian subjects were recruited by all sorts of methods.  Those in the study cast out their nets for other vegetarians and recruitment was done through all kinds of advertising venues.  Those accepted into the study -both vegetarians and non vegetarians – had to jump through a fair number of hoops to get accepted and stay in the study.  And to stay in the study for the ten plus years that it went on.  After the study period, the numbers of deaths in the two groups was tallied, and it was found that vegetarians didn’t live any longer than non-vegetarians.  As a percentage, the number of deaths in each group was the same.

What’s more interesting to me, however, is the difference between the rate of deaths in both the vegetarian and non-vegetarian subjects as compared to their neighbors who weren’t in the study.  The researchers calculated the standard mortality ratios (SMRs) for vegetarians and non-vegetarians from deaths before the age of 90 years old as compared to the mortality rate for non-study subjects living in the same area.

The SMR is the ratio of the observed number of deaths to the number of deaths expected from the national rates, standardized for sex and age, and expressed as a percentage.

In other words, if the observed number of deaths in the study group had been three quarters of that expected in a similar population from the area, the SMR would have been 75 percent.  And would have been a striking finding to boot.  It would have meant that just being in the study reduced one’s risk of death.

When all the data was tallied, the SMR for all causes of death among study subjects was only 52 percent, and was identical in vegetarians and nonvegetarians!  It didn’t matter if you were a vegetarian or a nonvegetarian, as long as you were in this study you were about half as likely to die as your neighbor who wasn’t in the study.  Now that’s an adherer effect in spades.  And I would think pretty newsworthy.  But, like the study above on meat and colorectal cancer, it was completely ignored by the press.

The point of this post is that you shouldn’t get wound up about a study that gets reported throughout the media because there are more than likely other studies that are just as well done and just as important showing exactly the opposite findings that the press chooses to ignore.  You’re not seeing the science as it is, you’re seeing the science as the press wants you to see it, which, typically, is the way that confirms the bias of members of the press.

As a journalist friend of ours once remarked:  what is news?  News is whatever the reporter decides it is.  In my opinion, they decided wrongly in this case.

I’ve had a number of people email me about a new study appearing in the Archives of Internal Medicine purportedly showing that statins really do provide benefit to those who take them regularly.  As you can see from the heading of an email piece I pasted above, even Medscape is all over this article and blasting it out to physicians all over the world.

I’m sad to say that this is the same kind of paper I would have been taken in by 20 years ago before I really understood how to read the scientific literature critically.  In fact, I would have used it myself to justify giving statins to all kinds of people, and I’m sure other physicians are doing so right now.  But I would have been in error to base my prescribing on this paper, and all the other docs out there giving statins like they were candy are in error as well.

If you don’t want to read a dissection of this study, let me just tell you up front that it doesn’t really mean a thing.  It certainly doesn’t prove that you should rush out and get started on statins.  If, however, you do want to learn about how perniciously deceptive these kinds of studies are and how to analyze them, read on.

Here’s the deal.  Researchers went back and combed through the records of a large HMO in Israel and pulled those of patients who had been prescribed statins from 1998-2006.  Since the HMO provided the statin prescriptions, there were records of how many of these people who were prescribed statins actually filled their prescriptions (and, one would assume, took the medications).  Then the researchers figured out how many of those people prescribed statins died.  The final step was to compare the list of those who died with the list of those who took their statin prescriptions (or, more accurately, those who filled their statin prescriptions).  After crunching all this data, it turns out that those patients who filled over 90 percent of their prescriptions were 45 percent less likely to die than those who filled under 10 percent of their prescriptions.  Which, to the uncritical reader (including, obviously the Medscape writers and the peers who reviewed this piece for the journal in which it was published), this appears to be pretty persuasive evidence that statins confer some kind of benefit in terms of preventing death.  After all, those that took them lived while those who didn’t died.

As I say, these kinds of studies are pretty beguiling.  But do they really mean anything?

Before we get to the specifics of this study, let’s contemplate this type of study in general to see why the data they generate is often misleading.

The gold standard for scientific studies is the randomized, double-blind, placebo-controlled trial.  In this type of study, researchers randomize the study population into two similar groups and give the members of one group the drug being studied and the other a placebo.  Double blinded means that neither the researchers nor the subjects know who got what.  At the end of the trial, the data are analyzed to determine if the study drug really showed any difference in efficacy as compared to the placebo.  If it did, then it can be said that the drug works to treat whatever condition was being studied.  Or that it decreases all-cause mortality, if that is the end point of the study.

It’s impossible to do these gold standard studies with diet and/or exercise because a) they involve lifestyle changes and b) they can’t be double blinded.  When it comes to diet and exercise, there are basically two ways studies can be done.  Researchers can allow subjects to self-select which arm of the study they want to be in.  Or researchers can put subjects into one arm or the other.  Neither of these choices is optimal, but they are all that are available.

If I decide that I’m going to compare a very-low-carb diet to a very-low-fat diet, I can recruit volunteers and ask them which diet they would prefer.  If readers of this blog were recruited into such a study, I would assume most would opt for the very-low-carb diet.  Those who are fans of Dean Ornish would opt for the other.  What you end up with is people in each arm of the study who are already believers in the diet they will be following, and they will be more likely to remain on the diet until the end of the study.  At the end, the data will be a little polluted because it really doesn’t prove that one diet is superior to the other – it only proves that people who self-select into that diet do better on that diet than people who self-select into the other.  The last it an important point, especially when applied to exercise.  More about which in a moment.

The other way to study diet is to gather a group of people together and randomize them into one diet group or the other.  That takes the self-selection bias out of the equation.  But it creates other problems.  If a person committed ideologically to a low-carb diet gets randomized into the low-fat group (or vice verse) there are problems with compliance.  Most nutritional studies randomized this way end up with large numbers of dropouts.  If you do an intention-to-treat analysis of the data (which includes the drop outs), you usually find little difference between the two diets.  If you look at only those subjects who hung in there for the duration on whichever diet they were randomized onto, it raises the issue of whether these subjects may have been the same ones who would have self-selected themselves into this same diet if given the chance, which then creates the same problems as self-selection.  These issues make diet studies difficult to do and difficult to interpret validly.  It’s even worse with exercise.

I get a ton of email and comments from people who can’t come to grips with the idea that there is no proof that exercise brings about weight loss.  I say this because it is difficult to come by this proof.  Even those who are adamant that exercise brings about weight loss agree that pretty intensive exercise is required to do so.  The typical prescription to just get out and move a little more virtually everyone realizes is worthless.  Most people believe that it’s intensive exercise that does the trick.  Maybe so, but how do you prove it?

If you randomize people into an intensive exercise group and another into a no exercise group to see which loses the most weight (assuming diet is held constant), how many of those sedentary people are going to stick with the intensive exercise for any length of time.  They will be the dropouts.  If you allow people to self select, all the people who enjoy exercise will put themselves into the exercise group while those who hate it will put themselves into the sedentary group.  Then if those in the exercise group do lose weight, how can you tell it’s the exercise and not due to some other component of a person who will commit to an intensive exercise program that brings about the weight loss?  The answer is that you can’t tell.  Which is why the notion that exercise brings about weight loss is similar to a particular religious belief: it is accepted as an article of faith, not as a product of scientific investigation.

You can send me a comment (as several people have done) telling me how you were stuck in your weight loss efforts at 220 pounds and then you decided to start high intensity interval training.  After a couple of months of this, you lost 25 more pounds.  Therefore that’s proof that exercise brings about weight loss.  Wrong!  That’s proof that in you exercise brought about weight loss.  There may be something different about you that allows you to commit to such a regimen that others might have difficulty following AND allows you to lose weight.  This sounds ridiculous, but it is true.  And it is the key to understanding why this statin study is bogus in terms of whether or not taking statins makes people live longer.

Almost thirty years ago a study was published in the New England Journal of Medicine looking at this very idea.  The study that inspired the article didn’t start out looking at this idea, but one of the investigators noted a key piece of the data and published on it.  The study was looking at clofibrate, a pre-statin cholesterol lowering drug and all cause mortality.  Subjects were randomized into two groups – those in one group got the drug, those in the other got the placebo.  After the subjects were on either the drug or the placebo for five years, researchers calculated the mortality from the number of deaths in each group.  Turned out that the five-year mortality of those on clofibrate was 20.0 percent whereas the five-year mortality of those on the placebo was 20.9 percent, or essentially the same.  Taking the drug was no different than taking the placebo, i.e., the drug was worthless. Had one of the researchers not looked a little closer, that would have been the end of the story.

When the data were looked at from the perspective of how many people actually took the drug as prescribed, the researcher discovered that those subjects who took at least 80 percent or more of their clofibrate had a five year mortality of only 15.0 percent, substantially less than the overall five-year mortality.  Those who took their clofibrate sporadically had a five-year mortality of 24.6 percent, significantly higher than those who took it as directed, a piece of data that would seem to confirm the efficacy of clofibrate.  Right?  Not necessarily.  Let’s look at compliance with the placebo.

Turns out that those subjects on the placebo who regularly took their placebo had a five-year mortality of 15.1 percent while those who took their placebo sporadically had a five-year mortality of 28.3 percent.  What this study really showed was that there is something intrinsic to people who religiously take their medicine that makes them live longer.  There was no difference between the drug and placebo in either those who took them regularly or those who took them sporadically, but there was a huge difference in mortality between those who took either drug or placebo on schedule and those who didn’t.

Lest you think this was a bizarre one-of-a-kind study, another study published a few years ago in The Lancet showed a virtually identical outcome.  Patients taking a medication for congestive heart failure were compared to those taking placebo.  Those taking the drug (Candesartan) showed no difference in mortality compared to those taking placebo.  But when compliance was evaluated, those taking either the drug or the placebo as directed had much lower mortality than those taking either one sporadically.  In fact, as you can see from the graph below, the mortality curves were almost identical.

From Lancet (2005); 366(9502):2005-2011

So there is something about adherers to a drug regimen that promotes longevity as compared to non-adherers.

Getting back to our statin study, how do we know that the decreased risk of death in those who religiously stuck with their statin prescriptions as compared to those who didn’t came about because they were adherers and not because of the statins?  We don’t.  In fact, based on the two studies I detailed above, it’s much likelier that the decreased mortality in those who took all their statins came about not because of the statins, but because those who stuck with them are adherers and have what ever quality it is that adherers have that makes them live longer.  And, if this is the case in this study as in the others, the statins don’t really do anything at all.

Despite its not really proving that statins confer greater longevity, the study does provide some interesting admissions and entertaining confabulations.

First, the study authors admit that there is no gold standard, randomized controlled study data showing that statins are of benefit in preventing death except for one group of people (and they even get that wrong).

The beneficial effects on cardiovascular mortality of treatment with statins to decrease levels of low-density lipoprotein cholesterol (LDL-C) have been established in several long-term, placebo-controlled trials.

The value of primary prevention with statin therapy in the reduction of overall mortality has recently been questioned.

A pooled analysis of 8 randomized trials in primary prevention populations showed that statins did not reduce overall mortality, indicating that lipid-lowering therapy with statins should not be prescribed for true primary prevention in women of any age or in men older than 69 years.

What they’re saying here is that statins have been shown to reduce mortality from heart disease in those who have elevated LDL, which is true.  But this decrease in deaths from heart disease is compensated for by an increase in deaths from cancer and other causes, so there really isn’t a gain.  You’re still dead.  Just maybe not from heart disease, but what difference does it make.  Are you going to spend $200 per month for the rest of your life and stay on medications that may make you feel lousy and lose your memory just so you can die of something other than heart disease?

In the last paragraph in the quote above, the authors confess that the data from actual randomized control trials show that statins confer no all-cause mortality benefits to women of any age and to men over 69.  They are playing a little fast and loose with the truth here because as I have posted before, the gold standard trials have shown no benefit for women and no benefit to men over 65 or to men under 65 who have never had heart disease.  The only improvement in all-cause mortality has been in men under 65 who have been diagnosed with heart disease, and even that benefit is so small that many people question if the extra cost and side effects of the statins are worth it.

So the authors of this study acknowledge that there has never been a randomized control trial that has shown any benefit to taking statins, but that doesn’t stop them.  They forge ahead trying to figure a reason that all these clinical trials haven’t shown an advantage.

Because clinical trials do not usually include individuals with multiple comorbid conditions or those receiving an extensive list of medications, there are considerable concerns regarding the applicability of findings from randomized clinical trials to the general population of patients seen in routine clinical practice.

Aha! They are saying that because the randomized controlled trial didn’t show what they wanted them to show – that statins worked for everyone all the time (thus the “considerable concerns”) -  that they need to figure out a better way to study them, one that involves patients with a lot of problems so that they don’t have to randomize them and confront failure yet again.

In light of the controversy surrounding lipid-lowering treatment for reduction of mortality among primary prevention populations, we undertook the present study to evaluate the effect of statin therapy in a large and diverse cohort of patients treated for dyslipidemia in a single health maintenance organization.

Interesting take.  There is no controversy.  The randomized controlled studies clearly show very little benefit to statin therapy in terms of decreasing all-cause mortality, the one statistic that really counts.  The controversy arises because the statinators simply don’t want to believe what these carefully performed trials tell them.  They by God want statins to work.  And they’re going to keep looking and fiddling with the data until they get a study that tells them what they want to hear whether the data is valid or not.

It’s pitiful that they are so desperate.

Don’t fall for the false promise of this or any other version of an observational study.  These kinds of studies do not prove causality.  Nor do they prove that a drug regimen works.  The patients in this study who religiously took their statins had better all-cause mortality than those who didn’t.  But, as we saw above, adherers always have better all-cause mortality than non-adherers.  In this case, was it that the adherers lived longer or was it that statins conferred some sort of benefit.  We can’t tell.  But we do know that in the real studies, the randomized control trials, statins didn’t do squat, so my vote would be that what we’re seeing here is an adherer effect and not a statin effect.

My advice is to continue to regard statins with a jaundiced eye.  So far, we haven’t seen any evidence that justifies the expense and the side effects of these drugs.

One view of the value of epidemiology

A day almost never passes without someone sending a comment my way about some recent study, plucked by the media from the hundreds published that same day, showing that low-carb diets cause brain fog or decreased longevity or cancer of some type or any number of conditions any of us would rather not have.  These comments always  end with the plaintive request, is there any truth to this?

My answer follows: This data comes from an observational study, and, as such, can’t possibly indicate causality.

Since I get these comments so often and answer them the same equally often, I figured it was about time to write a post on what an observational study really is so that I can link to it when I give my standard reply.

I can then add this post to the ones on the glycemic index and relative risk, both of which serve the same purpose.  I can simply link instead of explaining what these terms mean each time I have to use them.

Observational studies – also called prospective or cohort studies and sometimes even epidemiological studies – are the kind most often reported in the media simply because there are so many of them.  These are the studies in which researchers look for disease disparities between large populations of people with different diets, lifestyles, medications, incomes, etc.  If disease disparities are found to exist between groups, then researchers try to make the case that the difference in diet, lifestyle, medication, etc. is the driving force behind the disparity.

We’ve all seen these studies by the score.  We read that a large study population of people is separated into two groups based on blood levels of vitamin C.  One group of subjects has high blood levels, the other group has lower blood levels.  And since every one seems to believe that vitamin C protects against the common cold, the researchers decide to monitor these two groups for a year and find that the group with the highest blood levels of vitamin C has the fewest colds.   These findings are rushed into publication, and soon we read everywhere that vitamin C prevents the common cold.  It all seems so reasonable and so scientific, but the truth is that these studies don’t mean squat.  And the researchers who do them know it, or at least should know it.  The fact that they do know is evident in the weasel words they use in describing their findings.  You’ll read that these data ’suggest’ or that they ‘imply’ or that this ‘may cause’ that.  The non-technically trained public, however, read these to say that vitamin C prevents the common cold.  And usually the media helps to sway opinion by slanting the story in the same direction.

But, you may ask, why aren’t these studies sound?  If the one group with the greater blood levels of vitamin C had significantly fewer colds, why is it such a stretch to say that vitamin C prevents colds?

I can explain by way of a game I used to play with myself as a child.  I’ve never been one to sleep much even when I was a kid.  I always stayed up late and I always woke up early.  My brain never seemed to slow down.  I was always ruminating on something.  My way of trying to get to sleep was to try to think of everything that could be thought of.  My mind would race, and I would think of my brothers sleeping in the room with me, their beds, my bed, the closet, the tree outside, my dad’s car, the rug on the floor, the moon, and on and on and on.   As I thought faster and faster, continuing to compile things that could be thought of, I would finally hit a quitting point.  Then I would try to figure if there was anything I hadn’t thought of.  Of course, immediately I would think of something. I hadn’t thought of the pigs on my grandfather’s farm.  Or I hadn’t thought of the fire hydrant out front.  Or my father’s shoes.  Or whatever.  Then I would start the game again, this time, of course, starting with the pigs on my grandfather’s farm and going from there.  I would always fall asleep before I had ever thought of everything there was to think of.

Researchers doing observational studies have much the same problem.  They try to think of all the differences between two large populations of subjects so that they can statistically negate them so that only the observation in question – the vitamin C level in the example above – is different between the groups.  Problem is they can never possibly think of all the differences between the groups.  As a consequence, they never have a perfect study with exactly the same number, sex, age, lifestyle, etc. on both sides with the only difference being the study parameter. And so they don’t really ever prove anything.  In fact, we would all probably be a lot better off if all the researchers doing observational studies had followed my lead and fallen asleep mid study.

But I’m being too harsh.  These studies do have some value.  Their value is in generating hypotheses.

The observational study demonstrates a correlation.  In our example above, the correlation is that higher vitamin C levels correlate (in this particular study) with lower rates of colds.  So, from this data, we could hypothesize that vitamin C prevents the common cold.  But at this stage that would be just an hypothesis – not a fact.

Once we have the hypothesis, we can then do a randomize, placebo-controlled trial.   We can recruit subjects, randomize them into two groups that are as equal as possible, especially as vitamin C levels are concerned.  Then we give one group of subjects vitamin C and the other a placebo and watch them for a year.  At the end of the year (or whatever the study period is), we break the codes, see who is on vitamin C and who is on placebo.  We already know how many got colds, so now we compare that to vitamin C intake.  We may find that those who took the vitamin C got significantly fewer colds, so we can say that our study demonstrates that vitamin C prevents the common cold.  If this same study is repeated a number of times with the same outcome, then it can be said to be proven that vitamin C prevents colds. (This study is, of course, hypothetical.)

But these studies are randomized trials, not observational studies.  Observational studies only show correlation, not causation, a fact that everyone doing research and reading about research should have tattooed on their foreheads.

CORRELATION IS NOT CAUSATION

More often than not observational studies are chock full of all kinds of technical-looking graphs, charts and tables.  Many even have complicated equations.  And long statistical analyses of the data derived.  They are like zombies, however.  They give the appearance of scientific life, but they are really scientifically dead.  Irrespective of how many scientific baubles are strewn through them, they are nothing but observational studies, worthwhile only as generators of hypotheses.  They demonstrate only correlation, not causation.

If you want to bear with me, I’ll show you a bizarre observational study that was actually performed that demonstrates everything you need to know about observational studies.

The study was published in 2003 in the prestigious American Journal of Epidemiology.  The title of the study is Shaving, Coronary Heart Disease, and Stroke. (Click here for free full text) This study purports to show that the frequency of shaving correlates with risk for developing heart disease, with those men shaving less having a greater risk.

Here’s the finding that initiated this study.

A case-control study comparing the frequency of shaving in 21 men under 43 years of age who had suffered a myocardial infarction and 21 controls found that nine of the cases but none of the controls shaved only every 2 or 3 days.

Someone noticed that about half of the men in a small group of subjects who had a heart attack shaved once every two or three days.  Another group of men of similar age who hadn’t had a heart attack were designated as controls.  Upon questioning it was discovered that all of the men in the control group shaved every day.  Thus the first hypothesis was born:  Infrequent shaving correlates with heart attack.

The researchers had access to a large population of subjects from another ongoing study called the Caerphilly Study.  Researchers recruited 2,513 men aged 45-59 from this study and gave them comprehensive medical workups including extensive laboratory testing.

Men were asked about their frequency of shaving by a medical interviewer during phase I. Responses were classified into categories ranging from twice daily to once daily, every other day, or less frequently. The 34 men with beards were not classified. These categories were dichotomized into once or twice per day and less frequently.

The men in the study were followed for the next 20 years with follow-up exams periodically to monitor for history of chest pain, heart attack and/or stroke.

Of the 521 men who shaved less frequently than daily, 45.1 percent died during the follow-up period, as compared with 31.3 percent of men who shaved at least daily.

When the data were further refined it was determined that

The age-adjusted hazard ratios demonstrate increased risks of all-cause, cardiovascular disease, and non-cardiovascular-disease mortality and all stroke events among men who shaved less frequently.

So there you have it.  Proof that shaving daily prevents heart disease. Or is it?

The researchers doing this study aren’t so stupid that they really think that the act of shaving itself has anything to do with a man’s risk for developing heart disease.  In fact, they went to great lengths to show that shaving was merely a marker for other things going on that may well have something to do with risk for developing heart disease or increased all-cause mortality.

The one fifth (n = 521, 21.4%) of men who shaved less frequently than daily were shorter, were less likely to be married, had a lower frequency of orgasm, and were more likely to smoke, to have angina, and to work in manual occupations than other men.

And these are just the differences the researchers found.  Had they looked harder, I’m sure they would have found more, just like I did when I played my ‘think of everything that can be thought about’  game with myself as a kid.

But if these researchers had really believed that the data showed that the lack of frequent shaving itself may have been the driving force behind the development of heart disease, they may have designed a randomized clinical trial to show causality.  They could have recruited men without heart disease, randomized them into two groups, instructed the men in one group to shave daily and the men in the other to shave every third day.  Then after 20 years the researchers could tell whether or not shaving protects against heart disease.

But the idea that shaving itself has anything to do with heart disease is so ludicrous that no one would ever do such a study.  We can all see that.  It’s a ridiculous idea.  It should be obvious that the shaving or lack thereof has nothing to do with heart disease or early death; the lack of shaving is merely a marker for all the other conditions that are risk factors for heart disease, i.e., small stature, unmarried, smoking, lower socioeconomic class, etc.  It’s all so easy to see.

But let’s just suppose that we take this same study and substitute the term ‘elevated cholesterol’ for ‘infrequent shaving.’  Now what do we see?  Let’s change one of the quotes from above to reflect this change.  What then?

Of the 521 men who had elevated cholesterol, 45.1 percent died during the follow-up period, as compared with 31.3 percent of men who had low or normal cholesterol.

We nod our heads sagely.  Suddenly we have a study that seems to make sense.  But – and this is important – it doesn’t make any more sense than the shaving study.  Both are observational studies.  We are programmed to think cholesterol is bad and causes heart disease, so this second study appears reasonable to us.  It triggers our confirmation bias.  We don’t believe for a second that shaving has anything to do with heart disease, so we can easily dismiss those findings.  But we are more than ready to believe that the elevated cholesterol caused those men who had it to have heart attacks. But the reality is that both studies are exactly the same – and neither proves anything.

If you’re interested in a longer, more in-depth article on observational studies, take a look at Gary Taubes long piece in the New York Times a few years ago.  I’ve tried to take a little different slant than he did so that my post and his article would cover all the bases.

Cartoon above from: Smith, G. D. et al. Int. J. Epidemiol. 2001 30:1-11

`I quite agree with you,’ said the Duchess; `and the moral of that is–Be what you would seem to be–or if you’d like it put more simply–Never imagine yourself not to be otherwise than what it might appear to others that what you were or might have been was not otherwise than what you had been would have appeared to them to be otherwise.’

`I think I should understand that better,’ Alice said very politely, `if I had it written down: but I can’t quite follow it as you say it.’

Lewis Carroll

If you tell a lie big enough and keep repeating it, people will eventually come to believe it.

Dr. Joseph Goebbels, Nazi minister of propaganda

I’m starting this post with two apropos quotes.  The first, from Alice in Wonderland, because the post will be a little difficult to understand; the second because I just read for the umpteenth time the Big Lie about low-carb diets and wanted to blog about it but couldn’t until I wrote this post first.

Intention-to-treat analysis (ITT) has become the de rigueur way of looking at experimental results that more often than not gives erroneous results.  These erroneous results are then reported as gospel, when in reality they are simply erroneous.  When unbiased, intelligent people (the readers of this blog, for example) consider ITT, they cannot understand how it can be used by scientist trying to make sense out of their data, but, unfortunately, it is in almost every experiment.  Here is how it works.

Let’s say were going to do an experiment comparing two different diets.  We round up 100 subjects and randomize them into two groups of 50.  We put one group, Group A, on one diet, Diet A, and we put the other, Group B, on a different diet, Diet B.  We keep both groups on their respective diets for 8 weeks to see what happens.

At the end of the 8 weeks we find that 30 members of Group A dropped out, but those who hung in there lost an average of 3 pounds per week for a total of 24 pounds each over the course of the study.  We look at Group B and find that no one dropped out of the study and that all the subjects lost an average of 1.2 pounds per week.

What does this data tell us?  It’s pretty simple.  It tells us that Diet A is much more effective, but is more difficult to follow.  It tells us that Diet B is less effective but easier to follow.  Right?  All intelligent people could agree on that.  So that’s how this study would be presented if it were published in a journal, right?  Uh, no.

No?

No.  If published, the conclusion would be that both diets are exactly the same.

Say what?!?!?

Yep.  That’s what the authors would conclude.  Why?  Because they would use an intention-to-treat analysis.  In fact, the peer-review process would probably demand it.

An intention-to-treat analysis demands that all subjects remain in the data pool, even if some have dropped out.  The intention was to treat all the subjects, so the analysis should contain all the subjects, even if some left the study after the first day.  In an ITT, researchers pretend that subjects who chose to abandon the study really didn’t and include them in their final data.  Sounds like something from Through the Looking Glass, doesn’t it?

Let’s look at how this would work in our dietary study above.  The 20 subjects in Group A who followed Diet A lost 24 pounds each.  Multiply this 24 pounds times the 20 subjects who stayed in the study and you find that the group lost 480 pounds over the course of the 8 weeks.  Now divide this 480 pounds by the 50 subjects who started the study, and you get a weight loss of 9.6 pounds for the 8 weeks.  Dividing by 8 gives us an average weight loss of 1.2 pounds per week for all 50 subjects in Group A.  Which is exactly the same as the weight loss in the subjects in Group B.  So, according to the dictates of ITT, the study would show that both diets were equally effective. But, as we’ve seen, they’re not.

If a doctor were recommending a diet to his/her patients based on the actual findings of the study, he/she could reasonably say:  Diet A is very effective but tough to follow, so if you think you can do it, Diet A is definitely the fastest way to lose weight.  If you want something that will help you lose a little weight and is easy to stick to, then try Diet B.

If the same doctor recommends a diet to his/her patients based on the ITT results, he/she would say: Follow whichever diet you want – they’re both the same.

Why, you may ask, could seemingly intelligent people do something so stupid as use ITT to evaluate data? There is a reason, although it has its own problems.

We all know from experience and from talking to a lot of people who have lost weight that a lot of different diets work.  People lose weight on the Ornish diet and they lose weight on the infinitely preferable Protein Power diet.  And many other diets as well.  So, we can reasonably assume that almost any diet will help some people lose weight.  But we want to compare two diets to see which one is really the best.  So, let’s do another experiment.

Let’s take another 100 people and randomize them into two groups of 50, Group C and Group D.  Those subjects in Group C go on Diet C and all of them do well.  They lose an average of 2 pounds per week and all of them stay on the diet. The subjects in Group D go on Diet D, and most don’t do very well. As we all know from experience, it’s tough to stay motivated to stay on a diet if you’re not losing weight.  So, 30 of the subjects in Group D drop out because they’re not losing.  We know that any diet will work for some people, and Diet D is no different.  The 20 who stay in the study are those who are losing on Diet D.  And those 20 Group D subjects lose an average of 2 pounds per week.

In analyzing our data, if we remove from the pool of subjects all those who dropped out of the study, we are left with all 50 people in Group C, who lost an average of 2 pounds per week and only 20 people in Group D, who lost an average of 2 pounds per week.  We would then find that both diets are exactly the same.  Subjects in both groups lost 2 pounds per week.  Therefore both diets are equally effective.

But is that true?  Clearly not.  And that is the problem that ITT was designed to deal with.  But, as we’ve seen above, it brings its own errors.

So, how do we deal with the issue honestly and effectively?  Easy. By explaining the data in two ways.  Most people – researchers included – want to boil an issue down to a single answer, when two answers are required. ITT allows one answer – often incorrect – to two different questions.  ITT is like the old TV show in which the clown Bozo always asked the little kids he interviewed something like this:

So, Bobby, tell me: Do you walk to school or carry your lunch?

Were Bozo adamant on an ITT-type analysis of the question, he could get only one answer.

Going back to our Group A/Group B diet study we can look at the data in two ways:

1. Diet A is extremely effective for those who stick with it. (Called the adherence effect.)

2. Only 40 percent of those attempting Diet A achieve the desired effect. (Called the assignment effect.)

Both of these statements are true.  Both contain valuable information.  But they answer two different questions.  The first answers the question: what happens to people who stick to the diet?  The second answers the question: What happens to people who are placed on the diet?

As Dr. Gerard Dallal writes about ITT

The fraud occurs when the answer to the question of assignment is given as though it were the answer to the question of adherence!

Instead of the conclusion that both Diet A and Diet B show the same results (when, clearly, they don’t), which would be the way it would be presented in a scientific paper demanding ITT, why not present it this way?:

The adherence effect: Subjects following Diet A for 8 weeks lost an average of 3 pounds per week whereas those following Diet B lost 1.2 pounds per week.

The assignment effect: 40 percent of those attempting Diet A remained in the study whereas 100 percent of those following Diet B remained in the study.

Conclusion: Diet A is significantly more effective (3 pounds per week vs 1.2 pounds per week) for those able to remain on the diet.  Diet B is less effective but significantly less difficult to follow than Diet A. (100 percent of subjects on Diet B remained on the diet throughout the study whereas 60 percent of those on Diet A dropped out).

It just ain’t that hard to present it that way.  It provides much more information than the ITT, which attempts to answer two questions with one answer.

Now, let’s look at the big low-carb lie that launched me into this post.  I was reading a book that I intended to review for this blog and came across the following statement:

There is evidence from a variety of sources that [low-carb diets] work for short-term weight loss.  One year after starting a diet, however, there appears to be no significant difference in success rate than that seen on any other common diet plan.

Have you heard that one before?  It’s a specific variant of the old: Studies show that while effective in the short term low-carb diets show no difference in weight loss after one year than do low-fat diets. It’s the Big Lie.

It’s the last refuge argument of low-fat advocates who are getting hammered with all the data showing low-carb diets to be more effective.  Yeah, well, they say, Protein Power may work in the short term, but over a year studies show it’s no better than low-fat.  It’s like a cross thrust in a vampire’s face.

But is it true?  It is if you believe in intention-to-treat analysis.  But what if you believe in a more accurate way of presenting the data?

Let’s briefly look at a few studies published that confirm the idea that there is no difference between low-carb diets and low-fat diets after one year.

The first was published in the Annals of Internal Medicine in 2004.  The conclusion of the authors was that after one year subjects

had more favorable triglyceride and high-density lipoprotein cholesterol levels on the low-carbohydrate diet than on the conventional diet. However, weight loss and the other metabolic parameters were similar in the 2 diet groups.

In the body of the paper, however, one can read the following:

The final 1-year weight change (mean ± SD) was -5.1 ± 8.7 kg in the low-carbohydrate group and -3.1 ± 8.4 kg in the conventional diet group (Figure). The difference in weight loss between the 2 diet groups was not significant (-2.0 kg [CI, -4.9 kg to 1.0 kg]; P = 0.195 before and P > 0.2 after adjustment for baseline variables). The difference in weight loss between the 2 diet groups between 6 months and 1 year was not statistically significant (P = 0.063).

But that’s all ITT blather.  Let’s read the next couple of sentences:

Persons on the low-carbohydrate diet who dropped out lost less weight than those who completed the study (change, -0.2 ± 7.6 kg vs. -7.3 ± 8.3 kg, respectively; mean difference, -7.1 kg [CI, -11.6 kg to -2.8 kg]; P = 0.003). In contrast, weight loss was not significantly different for those on the conventional diet, whether they dropped out or completed the study (change, -2.2 ± 9.5 kg vs. -3.7 ± 7.7, respectively; mean difference, -1.5 kg [CI, -5.7 kg to 2.7 kg]; P > 0.2).

Let’s translate.  Those who dropped out of the low-carb diet but were counted as if they hadn’t lost 0.2 kg (about 0.4 pounds) whereas those who completed the study lost 7.3 kg (about 16 pounds).  Do you think the dropouts skewed the numbers?  I guess so.  And look at the next astounding sentence.  “In contrast, weight loss was not significantly different for those on the conventional diet, whether they dropped out or completed the study…”  So, there was no difference in the results of those following the low-fat diet whether they dropped out or stayed in.  Had the subjects who dropped from the low-fat arm not been included, the results for that diet would have been the same.  Including the subjects who dropped from the low-carb arm, however, dramatically lowered the overall weight loss of the subjects as a group, making them equal to those in the low-fat arm.

It could be accurately stated that those who remained on the low-carb diet for one year lost significantly more weight than those who remained on the low-fat diet. which, of course, refutes the Big Lie that low-carb and low-fat diets provide equal weight loss at one year.

The two other studies used to perpetrate the Big Lie that low-carb diets show no difference in weight loss after one year are the ones by Foster et al and Samaha et al in the May 2003 New England Journal of Medicine.

When analyzed by ITT, both of these studies show no significant difference between low-carb and low-fat diets after a year.  But when looked at from the perspective of those subjects remaining in the study, we see a big difference between the low-carb and the low-fat arms.

In the Foster et al study using a modified version of the Atkins diet, we find a statistically insignificant 1.9 kg difference in weight loss between the two groups by ITT.  But when we eliminate the drop outs and look instead at the data from those subjects who remained on the diets for the entire one year, we find a statistically significant 2.8 kg (over 6 pounds) greater weight loss in those following the low-carb diet.

In the Samaha et al study using the diet from the Protein Power LifePlan, those following the low-carb diet lost a statistically insignificant 2 kg more weight than those following the low-fat diet by ITT.  Eliminating the dropouts, however, gives us a statistically significant 3.6 kg (almost 8 pounds) greater weight loss on the low-carb verses the high-carb diet after one year.**

Intention-to-treat analysis gives us the Big Lie:  Low-carb diets are no more effective than low-fat diets after one year. Dr. Goebbels would have been proud.

The truth, however, is a little different and can be stated thus:

Those who follow low-carb diets for a year lose significantly greater weight than those who follow low-fat diets for a year.

After reading this post you should know more about intention-to-treat analysis than 99.9 percent of the physicians and dietitians practicing in the world today.  Don’t let this knowledge go to waste.  Next time you hear the Big Lie, point out the truth.

** Thanks to Richard Feinman, Ph.D. for the tabulation of these data and for our many conversations on this subject.

]]> http://www.proteinpower.com/drmike/bogus-studies/the-fraud-of-intention-to-treat-analysis/feed/ 32 http://www.proteinpower.com/drmike/weight-loss/average-doesn-tell-the-whole-story/ http://www.proteinpower.com/drmike/weight-loss/average-doesn-tell-the-whole-story/#comments Tue, 25 Mar 2008 22:20:10 +0000 mreades http://www.proteinpower.com/drmike/weight-loss/average-doesn-tell-the-whole-story/ Virtually all of the results presented in medical studies are displayed as ‘average’ or ‘mean’ values. I’m sure everyone knows how to come up with an average or mean (the two are synonymous) value for a group of data points is to add them and divide the sum by the number of data points analyzed. For example, if you are a teacher, and you want to find out the average score on a test you gave to 30 students, you would add all the test scores together and divide by 30. You would then have the ‘mean’ or ‘average’ score of the students in your class.

Most medical papers list the mean values of whatever is being studied. If the researchers are trying to determine whether or not an experimental weight-loss therapy works, they add the weight lost by all the subjects participating in the study then divide by the number of subjects. The number they get is the ‘mean’ or ‘average’ weight loss brought about by the therapy being tested. It all sounds pretty reasonable and scientific, but is it really?

It would be realistic if we were all average people. But we’re not. And averages don’t represent us all that well. In fact, if you think about it, the average American would have one breast and one testicle.

Averages don’t always represent the true findings in a scientific experiment, either. Let’s look at an example to demonstrate. Let’s say we are testing a new weight loss regimen on 10 people. We start these people on the program, keep them on it for three months, then evaluate. When we look at the numbers we find the following results:

• Subject #1 -4 lbs
• Subject #2 -5 lbs
• Subject #3 – 7 lbs
• Subject #4 – 4 lbs
• Subject #5 – 2 lbs
• Subject #6 -6 lbs
• Subject #7 +12 lbs
• Subject #8 – 1 lb
• Subject #9 + 4 lbs
• Subject #10 -3 lbs

If you add these numbers up and divide by 10 you find that the average or mean weight loss for the group is 1.6 pounds, which doesn’t seem like a lot. But if you look at the data itself instead of the average, you see that most of the people lost around 4-5 pounds. In fact, assuming these numbers to be accurate, if you went on this same regimen, the odds are that you would lose 4-5 pounds instead of the 1.6 pounds that the ‘mean’ of the data would predict. You could also gain 12 pounds, but that would be unlikely (a 1 in 10 chance). This is the problem in simply looking only at average values and not the data as a whole.

Another way to look at this data is to calculate the median, which is basically the midpoint within the data set, i.e., the point at which half of the subjects are above and the other half below. You can do this by arranging the data in ascending or descending order and finding the midpoint by lopping off from both the top and the bottom until you get to the middle.

If we do this to our data, it looks like this:

• Subject #7 +12 lbs
• Subject #9 + 4 lbs
• Subject #8 – 1 lb
• Subject #5 – 2 lbs
• Subject #10 -3 lbs
• Subject #1 -4 lbs
• Subject #4 -4 lbs
• Subject #2 -5 lbs
• Subject #6 -6 lbs
• Subject #3 -7 lbs

We can see that the median falls between 3 and 4 or 3.5. So in this experiment the mean (average) weight loss is 1.6 pounds, the median is 3.5 pounds, about twice what the mean is. But if you look at the actual weights lost you can see that most clustered around the 4-5 pound level. You can see that the results of our experimental weight-loss regimen look different depending upon how they’re reported.

It works this way in the real medial literature as well.

Let’s look at a study from a couple of years ago (full text pdf) that many people have used to ‘prove’ there is no metabolic advantage and to ‘prove’ that a calorie is simply a calorie irrespective of its macronutrient composition.

Boden et al studied 10 overweight patients with type II diabetes in metabolic ward for 21 days. During the first 7 days the subjects were allowed to follow their regular diet (the control) and were switched to a low-carbohydrate diet (21 g carb/day) for the next 14 days. During the course of the study, numerous parameters were evaluated, including fasting glucose levels, insulin sensitivity, HbA1c, weight change, caloric intake and energy expenditure.

After the 14 days on the low-carb diet subjects lost weight and markedly improved in all parameters measured. Blood glucose levels normalized, HgbA1c decreased from 7.3% to 6.8%, insulin sensitivity improved by about 75%, triglycerides dropped by 35% and total cholesterol fell by 15%. Pretty dramatic results for only 14 days on the low-carb regimen, I would say.

The subjects lost weight as well. They lost an average (mean) of 3.63 lbs (1.65 kg) while decreasing their food intake from 3111 kcal/day to 2164 kcal/day, a decrease of 947 kcal/day. Multiplying 947 X 14 gives us a total caloric deficit of 13,258 kcal. If we divide this number by 3500, the kcal in a pound of fat, we get 3.79, which is the amount of fat that should be lost simply from the caloric deficit. And which is pretty close to the actual 3.63 lbs actually lost. The difference is insignificant, so it really is, as they say, close enough for government work.

The authors of the study conclude that irrespective of all the other markedly positive benefits of the low-carb diet, the “weight loss…was completely accounted for by reduced caloric intake.” In other words, there is no metabolic advantage to a low-carb diet. Weight lost simply occurs because the satiating effects of the low-carb diet bring about a spontaneous reduction in calories.

But is that all the story here? Not really. Let’s see why.

Below is a chart from the study showing graphically what happened to caloric intake and weight during the course of the experiment.

Look at the lines in the upper half of this chart that represent the body weight ranges of all the subjects and the lines representing the caloric intake of all the subjects. Notice that the lines for caloric intake are small while the lines representing the amount of weight loss are large. If you compare the dimensions of these lines to the scale, you find that these subjects varied their caloric intake by about only 200-250 kcal/day. But the variation in weight loss is much, much larger, which means that some subjects lost considerably more weight than would be expected from the caloric deficit while others didn’t lose as much or may have even gained a little. What this chart shows us is that there is indeed a metabolic advantage for some of these people even though on average there wasn’t for the group. And remember the post on Carl Popper: if the metabolic advantage can be shown to be present, that means the hypothesis that a metabolic advantage doesn’t exist is false.

Unfortunately, most medical articles don’t show the array of data as this one has, so all we have to go by is the average, which doesn’t always tell the whole story. One of the things I like about my favorite journal, Nutrition & Metabolism, is that the editors almost always require the raw data to be shown along with the averages, which truly allows astute readers of the medical literature to come to meaningful conclusions about the data. Only after they report it in this way, can you really decide what happened.

Let’s say we are studying the rate at which people taking a particular medication get cancer. If we randomize our study population into two groups and give one group the drug and the other group a placebo, we can wait around for 5 years (or 10 years or 20 or whatever our protocol is) to see how many develop cancer in each group. Then we can compare.

If after 5 years we find that 10 of the subjects taking the drug develop cancer and only 5 of the subjects taking the placebo get cancer, we can say (assuming there were equal numbers of people in both study groups to start) that twice as many people taking the drug got cancer. We can then say that the relative risk (usually written as RR) of getting cancer from taking the drug is 2.

The relative risk is the number of subjects who develop the problem divided by the number who didn’t. In our example above it would be 10/5 = 2.

Another way to present this data is to say that the relative risk of not getting cancer by not taking the drug is 0.5. In other words, people who didn’t take the drug had half the risk of getting cancer compared to those who did.

In our simple example we assumed that there were exactly the same number of subjects in both groups. In reality this usually isn’t the case. There will be a difference in group sizes so instead of using numbers of subjects, researchers will use percentages. So in our case above, let’s say that 4 percent of the subjects taking the drug got cancer and 2 percent of those taking the placebo got cancer. We have the same relative risk of 2. (4 divided by 2 = 2)

It all seems pretty straight forward. But it often isn’t as it seems. There can be two studies that show the same relative risk but there is a world of different meaning for the relative risk. This idea is what we want to focus on because this is the one that researchers use all the time to scare us needlessly.

I’m going to create an example loosely based on the facts to show you what I mean. Let’s compare the safety records of two airlines over the past 25 years. In 1985 Delta had a crash in Dallas (in which one of my best friends was killed) and in 2001 American had three crashes. The two from 9/11 and one going from New York to San Juan, Puerto Rico. Let say that there were 200 people killed in the Dallas Delta crash and 600 people killed in the three American crashes.

Let’s assume that both airlines fly about the same number of miles each year. American has about 4000 flights per day, so if we assume an average distance (probably too short) of 500 miles per flight, we get 730 million miles flown per year. If we multiply by 25 to get the miles flown in 25 years we come up with 18.25 billion miles flown. Let’s assume the same for Delta. If we divide the number of victims of crashes from the two airlines by the number of miles flown, we can come up with a risk of dying per mile flown on each of the airlines. If we do this for Delta we discover that the risk of dying comes out to be 200 divided by 18.25 billion or about 0.00000001096. This tiny number represents the number of people who have died per mile flown by Delta. If we do the same calculation for American we find that the number is 0.00000003288 deaths per mile flown.

Now if we want to calculate the relative risk of flying on American verses flying on Delta we divide the American risk by the Delta risk (0.00000003288 divided by 0.00000001096) and we get 3. So, the relative risk of flying on American is 3 compared to flying on Delta based on the last 25 years of crash history. This means that it is 3 times more risky to fly American than it is to fly Delta. Based on these figures you are 3 times more likely to die if you fly American than if you fly Delta. It’s true, and these figures prove it.

We all know that it’s hogwash. There is no difference in risk between flying these airlines because the numbers are so low as to be ridiculous, which is the precise reason I used this example. The total numbers make a difference.

What if there were an airline that had 90 deaths for every 100,000 miles flown and another that had 30 deaths for each 100,000 miles flown. The relative risk for flying the first as compared to the second would be 3 just as it is with American verses Delta, but the real risk would be much, much different. In fact, you wouldn’t want to fly on either of these airlines because they are both too risky. But if you had to fly on one, you would much prefer the one with only 30 deaths per 100,000 miles.

Many drug companies use the relative risk in their propaganda. They might do a study with two groups of 40,000 people with one group taking their drug and the other taking a placebo and monitor them for 5 years. If the drug is supposed to prevent heart disease, then the researchers would tabulate the number of cases of heart disease that arose over the 5 years, then see how many cases developed in the subjects taking the drugs compared to the ones taking the placebo.

If there were 43 people who developed heart disease while taking the placebo and 31 who developed heart disease while taking the drug, then the relative risk of not taking the drug would be 43 divided by 31 which is 1.39. The drug company then proclaims that people not taking their drug had a 30 percent greater chance of developing heart disease, which sounds pretty significant. But it’s a lot like the Delta/American example: the numbers aren’t all that large.

What it really means is that out of the 40,000 people who took the placebo for 5 years only 12 more developed heart disease than in the group of 40,000 people who took the drug. Thats 2.4 people out of 40,000 per year. (12 people divided by 5 years = 2.4 people per year) If the drug costs a lot of money and has unpleasant side effects associated with it (think statins), would you be willing to pay the bucks and put up with the side effect hassle on the chance that you might be one of the 2.4 out of 40,000 who might be saved from developing heart disease? Or would you just take your chances? I would definitely take my chances.

2.4 people out of 40,000 per year sounds a lot less risky than a 39 percent increased risk that the relative risk implies.

One other way the relative risk can be used to flim flam people is when it is too small to be meaningful yet is touted as having great merit.

Without going into all the statistical detail (which if you’re interested in you can read about here and here from previous posts) suffice it to say that relative risk ratios below 2 are pretty much meaningless. So when you see a relative risk of 1.28 and hear someone say that represents a 28 percent increased risk, you can blow it off. Once the relative risk gets to the 2 or greater level, then it starts to have some meaning. But, once again, only if the numbers aren’t so small as to be meaningless. Remember, the relative risk of flying American is three times that of flying Delta if you simply look at the relative risk. In reality, however, the numbers are so small that the differences are meaningless.

In brief the JAMA study was done with data pulled from the monster-sized Chicago Heart Association Detection Project in Industry that was begun in 1967. Subjects who were at least 31 years old were evaluated for a number of parameters including BMI, blood pressure, elevated cholesterol, and history of smoking. The researchers re-evaluated these subjects over the next several decades.

Researchers divided the subjects into five groups: low risk, moderate risk, intermediate risk, elevated risk, and highest risk. It’s not important to the point of this post to get into what specifically constituted these varying levels of risk, but in general, the greater the number or the more severe the risk factors, i.e., elevated cholesterol, high blood pressure, etc., the more high-risk the category. The researchers then divided the subjects into three other groups based solely on BMI: normal weight, overweight, and obese. Within each of these three weight-related groups were spread subjects with varying degrees of risk. In other words, the normal weight group contained subjects who were low risk, moderate risk, intermediate risk, elevated risk and highest risk as a function of their cholesterol levels, blood pressures, smoking history, etc. It was the same for all the groups. The obese group was composed of obese subjects who ranged from low-risk to highest risk. The object of the study was to follow these subjects for many years to see if obesity was truly a risk factor for death from heart disease or if obesity led to elevated cholesterol, high blood pressure, and all the rest, which in turn caused the heart disease mortality.

If the subjects in the normal weight, low-risk group had no more heart disease than those in the obese, low-risk group, then it could be inferred that obesity by itself may not cause heart disease. If, on the other hand, the subjects in the obese, low-risk group had a much higher death rate from heart disease than did those subjects in the normal weight, low-risk group, then at least some of the heart disease could be attributed to the excess fat.

(There is really much, much more under the surface of this paper worthy of exploration, but it will have to wait.)

So what did the study show? It depends upon where you get your information.

According to a statistical analysis of the data, the odds for death by heart disease in the obese, low-risk subjects was 1.43 times that of the normal weight, low-risk subjects implying an almost half again greater risk simply for being obese. And that’s how it was reported in the lay press.

WebMD allowed as to how

The researchers found that the risk of dying from heart disease was 43% higher for study participants who were obese but also met these qualifications for low cardiovascular risk than for normal-weight, low-risk participants.

It appears to be a pretty clear indictment against obesity.

But not if the statistics are analyzed correctly.

Before I get into that I want to produce a quote from Judge Samuel Alito that he uttered during his confirmation hearing before the Senate Judiciary Committee last week. Said he

Well, the analogy went to the issue of statistics and the use and misuse of statistics and the fact that statistics can be quite misleading. … And that’s what that was referring to. There’s a whole – I mean, statistics is a branch of mathematics, and there are ways to analyze statistics so that you draw sound conclusions from them and avoid erroneous conclusions from them.

Truer words were never spoken. But you’ve got to analyze the statistics, not take them at face value.

So let’s analyze the statistics used in our study under discussion to see if and how anyone went wrong.

Here is how the 1.43 ratio was written in the paper:

the odds ratio (95% confidence interval) for CHD [coronary heart disease] death for obese participants compared with those of normal weight in the same risk category was 1.43 (0.33-6.25).

What does this really mean?

First, it means that 143 people who were in the obese, low-risk group died from CHD for each 100 people who were in the normal weight, low-risk group, giving the risk ratio of 143/100 or 1.43. It seems reasonable that if that were really the finding, then the risk of dying if you are obese with low-risk (as these researchers define low risk) is 1.43 times greater than if you aren’t obese. Right? Not necessarily, and here’s why.

If you flip a coin 10,000 times the odds are that you will get about 5000 heads and 5000 tails since the odds are 50-50 of the coin landing on either side. But what about if you only flip it 40 times? Are you going to get exactly 20 heads and 20 tails? Probably not. What about if you only flip it six times? Will you get three and three? Maybe, but probably not. In fact I just flipped a coin 10 times and got five heads in a row, two tails, one head, and one tail, giving me seven heads and three tails. Now if this were a study on coin flipping I could confidently predict based on my data that if I flipped this same coin 10,000 times I would get 7000 heads and 3000 tails. But we all know this isn’t really true because the sample size I used (ten) was too small to be used to accurately predict the outcome for 10,000 flips. The fact that I went 7 heads and 3 tails came about strictly by chance, which plays a smaller and smaller role as the sample size gets larger.

Statisticians realize that virtually any outcome can be influenced by chance and have developed equations to quantify just how much chance is involved. One of the terms they have come up with (after some pretty complex mathematical maneuvers) is the confidence interval. Pretty much the gold standard for confidence intervals is what’s called the 95 percent confidence interval. What this means is that once a confidence interval has been established (a range between two numbers) you can be confident that your result will fall into that range 95 percent of the time. Since chance can’t be totally eliminated, there will still be a 5 percent chance that our result will fall outside the range.

Let me digress a little here to define what we mean by our result falling into or outside of this range. In our study the data showed that 43 percent more people died in the obese, low-risk group than in the normal weight, low-risk group. But so what? As callous as it seems, we don’t care about those people; they’re already dead. What we care about is how the data provided by all these dead people affects you and me and our loved ones and all the other people who aren’t dead yet. We all want to know if this 43 percent is just a chance finding like my flipping 7 out of 10 heads, in which case it’s meaningless, or do I really have a 43 percent greater chance of dying of heart disease if I’m obese even though I don’t have any other risk factors? Those are the questions the confidence interval addresses.

In this study the 95 percent confidence interval is (0.33-6.25). It’s usually stated like it is in this case 1.43 (0.33-6.25). This means that the risk ratio as applied to the population at large should come in 95 percent of the time between 0.33 and 6.25. So this means that the risk of dying if you are obese with no other risk factors could be anywhere from 1/3 as much to 6.25 times as much.

Say what? 1/3 as much?

Yep. Even though the middle of this range is around 1.43 the actual risk is just as likely to be 0.5, which would mean you have half the chance of dying as someone who is normal weight without risk factors. In other words, you would be better off being fat.

These numbers in the parenthesis are critical. If you’ve got a positive number in front of the parenthesis as we do with the 1.43, then you want to make sure that both numbers within the parenthesis are above 1. If the first number is below one that means that the risk is actually greater the other way, which negates your analysis.

In this case since the first number is less than 1, indicating that the risk ratio is meaningless and can be ignored. What it means in this specific case is that it makes no difference whether or not you’re overweight early in life as long as your other risk factors (as identified by the researchers in this study) are normal in terms of your risk of dying of heart disease later in life. Not the 43 percent that the authors and the press that picked the story up proclaimed. Too bad the authors and all the medical press people weren’t a little more statistically honest.

But to tell you the truth, I suspect that the authors of this paper (and I know that the medical writers) don’t have the same understanding of the confidence limits and what they really mean that you do after reading this post. Most researchers run their data through a computerized statistical program and simply look at the risk ratio (the 1.43 in this case) without really having a clue what the numbers inside the parenthesis mean.

I swear that over the next few weeks I’ll post in as simple a way as I can a basic (very basic) primer on statistics. If I don’t do this in a timely fashion you may write and cancel your subscription to this blog, and the unused portion of your subscription fee will be cheerfully returned.