..man, proud man,
Dress’d in a little brief authority,
Most ignorant of what he’s most assur’d…

From Measure for Measure by Wm Shakespeare

The web has been alive with commentary the past few weeks since Denise Minger lobbed her first cannonball of a critique across the bow of The China Study, the vessel T. Colin Campbell, Ph.D. rode to fame and bestsellerdom.  Seems like everyone is now jumping into the fray and gunning for poor Dr. Campbell, who early on in the fracas made a few halfhearted attempts to fight back but has now fled the scene.  I’ve been laying low watching it all play out, and so now figured it’s about time I add my two cents worth to the debate. But first a little history.

I met Dr. Campbell about ten years ago (five or so years before the publication of the popular book The China Study) when we both spoke at the same conference.  He was a nice enough man who spoke about the work he and his team had done in China gathering the data published in the massive 894 page monograph Diet, Life-style and Mortality in China (pictured above left).  As Dr. Campbell presented his data ‘demonstrating’ the superiority of a plant-based diet and demonizing protein of animal origin, I didn’t think much about it because the data was all in the form of observational studies, which, as all readers of this blog should know by now, despite often showing correlation don’t prove causation.  My lecture, which followed Dr. Campbell’s, was, as you might imagine, a lecture of a different sort.  Then we both sat on a panel after our talks and fielded questions.  And were both cordial to one another.

A few years ago, I became vaguely aware that Dr. Campbell had written a popular book on his work in China titled, appropriately enough, The China Study.  I assumed it pretty much mirrored his presentation I had watched, so didn’t rush out and grab a copy.  Over the past few years a number of people have asked about The China Study through the comments section of this blog, and I’ve typically answered that the data are all observational and so not really meaningful in terms of causation.

(Note: Throughout this post whenever I refer to the popular book Dr. Campbell wrote, I’ll call it by it’s title The China Study, and when I refer to the large study Dr. Campbell was involved with in China and was the basis for the monograph Diet, Life-style and Mortality in China, I’ll call it the China study.)

About a year ago, I wrote a guest post for Tim Ferriss’s The Four Hour Workweek blog.  It actually wasn’t a guest post as much as it was an excerpt of a chapter from our book The Six-Week Cure for the Middle-aged Middle extolling the virtues of saturated fat.  It was a popular post that has garnered to date 520 plus comments, many of them fairly spirited.  I agreed to answer a number of the comments and did so.  I noticed as I sifted through them that a handful were absolutely fawning of Dr. Campbell and The China Study.  Here is a sampling:

The number one study of diet and disease is the China Study. All other data points are slivers compared to the volume of data and statistical correlations that came from the China Study.

Have you read The China Study? Dr. Campbell points out repeatedly that none of the weight loss studies such as Atkins or South Beach diet follow any type of peer reviewed scientific method.

Tim…and to think I was such a big fan of yours. This is by far the weakest (and least cited) argument I have ever read on diet–especially increasing saturated fats. Half knowledge is a scary thing in the hands of influential people. Maybe it’s another genius marketing ploy (like the myth riddled protein Atkin’s diet)–people love to feel good about their personal yet poor decision making–and diet is very personal. Check out researchers that actually meant to study nutrition–like Dr. T. Colin Campbell’s The China Study comes to mind.

It was pretty apparent that the disease of non-critical thinking was at epidemic proportions.

After reading a number of these, I decided I had better take a look to see what Dr. Campbell had going on that had attracted such devotees.  I pulled up his book on Amazon and read through a few comments, most of which were even more nauseatingly gushing than the above.  I ordered a copy of The China Study.

I knew that both Anthony Colpo and Chris Masterjohn had done their own critiques of the original data, so I figured, what the hell, I’ll take a look at the ‘real’ China study (as opposed to the popular book of that name) and do one too.  And I’ll critique the popular book, which I figured was a rehash of the China project, while I’m at it.

I tracked down a copy of the 894 page book in a bookstore in the UK and forked over $240 to purchase it and have it shipped.  As I was awaiting its arrival I told Gary Taubes what I had done, and he replied that he had done the same thing himself a few years earlier.  And that I could have borrowed his.  And, even worse, that most of the data was available online for free.

When the book arrived, I was amazed at the size of it.  Not only was it the 894 pages as advertised, it was in a large format.  Much larger than a volume of the Enclyclopaedia Britannica.  It wasn’t at all what I thought it would be.

Here are a couple of photographs shamelessly using our own book to show the size of this behemoth

Of the 894 pages, the first 82 are a study overview, description of methodology and author commentary.  It is written in the form of a scientific paper with half the page in English and half in Chinese (which, presumably, is a translation of the English half).  The remainder of the 894 pages are raw data and correlations.  Page after page after page of correlations.  I didn’t bother counting them, but Dr. Campbell says there are 367 variables, each of which is compared with every other variable.  I don’t doubt him.  This study was a massive undertaking, requiring thousands upon thousands of man hours and God only knows how much money.  No one can possibly accuse the team members of not giving it their all.

Here is one page of correlations.  This one between stearic acid and all the other variables studied.

But in the end it is still only an observational study.  And even though – again, according to Dr. Campbell – there are over 8000 statistically significant correlations, correlations are not causation.  Any scientist worth his/her salt will tell you that all you can do with data from observational studies is use them to form hypotheses that can be rigorously tested in randomized, controlled trials.  Then and only then (assuming the study results show it) can you even begin to talk about causation. The whole enterprise, costly and time consuming though it was, was described perfectly by Shakespeare in the words of MacBeth:

…it is a tale
Told by an idiot, full of sound and fury,
Signifying nothing.

Once I saw that the original China study was nothing but a huge number of correlations, I quickly lost interest.  What is the point in going through the brain damage of ferreting around in these to see if Dr. Campbell interpreted them correctly when he tries to make his case that a plant-based diet is optimal.  It doesn’t really matter whether he interprets them correctly or not, they are only correlations.  Repeat after me one last time: Correlation is not causation, correlation is not causation, correlation is not causation…

I wondered why Dr. Campbell and his group didn’t spend a fraction of the time and money they spent on this behemoth of a spreadsheet full of correlations on a real study that could provide hard evidence.  Why not randomize subjects into two groups and provide one a plant-based diet and the other a meat-based diet or something similar.  Lock them down as Ancel Keys did if they had to.  Surely the money spent on the China study could’ve covered that.  Get some real data.  I discovered later that I wasn’t the only one who wondered that.  Even some of Dr. Campbell’s own colleagues abandoned him to this study and told him it would be worthless.  More about this later.

So enough for me.  I stuck my copy of the $240 book of correlations in my library and forgot about it.  Until Denise Minger’s critique hit the net.

Upon reading her blog post, my first reaction was This is great; someone took the time to do what I was going to do. I figured Dr. Campbell had cherry picked his correlations to  make the case he wanted to make, and I had seen Colpo and Masterjohn catch him on it.  Ms. Minger went even further and really caught Dr. Campbell with his pants down, correlation-misinterpretation speaking.  I continued to read with mounting glee Ms Minger’s successive critiques and a few other bloggers who had critiques of their own.  (Believe me, there is no dearth of material here for people to attack without any two attacking the same data twice.)

After this went on for a while, I had my second reaction to the whole affair.

Which was that I had fallen victim to the confirmation bias.  My bias was that Dr. Campbell was wrong, so I was more than happy to uncritically accept evidence confirming his error without lifting a finger to double check said evidence myself.  I knew that if a blogger somewhere had come out with a long post describing an analysis of the China study demonstrating the validity of all of Dr. Campbell’s notions of the superiority of the plant-based diet, I would’ve been all over it looking for analytical errors.  But since Ms. Minger’s work accorded with my own beliefs, my confirmation bias ensured that I accepted it at face value.

Once the fact that I had succumbed to my confirmation bias settled in around me, I became suffused with angst.  I had tweeted and retweeted Ms. Minger’s analysis a number of times, giving the impression that I had at least minimally checked it out and had approved it.  I had emailed it to a number of people, many of whom, I’m sure, had forwarded it on.  I’m sure I played a fairly large role in the rapid dissemination of the anti Campbell/China study info.

(It didn’t really make me feel better to know I wasn’t alone in falling into the confirmation bias quicksand.  Take a look at this post from Richard Nikoley’s Free the Animal blog.  I doubt that all these people checked Ms. Minger’s calculations before posting.)

My angst wasn’t because I had possibly fed the flames of a misinformation wildfire – I wasn’t particularly worried about that because mountains of other data (including first hand data from my own clinical practice) have persuaded me that Dr. Campbell is dead wrong in his ideas about the superiority of a plant-based diet.  No, my angst arose for two other reasons: first, because I was distressed that I so easily fell prey to the confirmation bias, and, second, because I felt I needed to go through all the calculations  myself to make sure Ms. Minger and others whose work I had circulated were truly correct in their analyses.

As I was wallowing in self pity over all this, I didn’t realize that salvation was at hand. And that my savior was none other than Dr. T. Colin Campbell himself.

Yep, his first response to Denise Minger’s critique of his work appeared on the Tynan.net website and rescued me from my pit of self-loathing.  In it, Dr. Campbell wrote:

But she suffers one major flaw that seeps into her entire analysis by focusing on the selection of univariate correlations to make her arguments (univariate correlations in a study like this means, for example, comparing 2 variables–like dietary fat and breast cancer–within a very large database where there will undoubtedly be many factors that could incorrectly negate or enhance a possible correlation). She acknowledges this problem in several places but still turns around and displays data sets of univariate correlations.

In other words, the China study is an observational study comparing one variable to another (univariate correlations) and, as such, meaningless.  And this from the man’s own pen.

Since these observational studies are meaningless in terms of causality, it doesn’t really matter how one slices and dices the data because meaningless correlations by any other names are still just as meaningless.  All this falderal over whether or not Dr. Campbell had his interpretations right was tantamount to the medieval theological argument over how many angels could stand on the head of a pin.  And my participation certainly wasn’t required.

I’d known this all before, of course, but somehow had lost my focus on it.

I was ready to wash my hands of the whole affair when I came across another statement Dr. Campbell made in his response to Ms. Minger’s critique.  Writing of her, he said:

One further flaw…is her assumption that it was the China project itself, almost standing alone, that determined my conclusions for the book (it was only one chapter!). She, and others like her, ignore much of the rest of the book.

Only one chapter? As I mentioned above, I always figured The China Study was simply Dr. Campbell’s tale of the China study and the conclusions he had drawn from it.  Now he says that only one chapter is about the China study, leaving me to conclude that the rest must be about something else.  I found the book, which I hadn’t yet taken from the pack it came in from Amazon, opened it and started reading.

Wow!

In 1976 author Mary McCarthy famously said live on the Dick Cavett show of her rival Lillian Hellman:

Every word she writes is a lie, including ‘and’ and ‘the’.**

I feel much the same way about The China Study.  Except it’s not really a lie, it’s an obfuscation.

In fact, in my studied opinion, The China Study is a masterpiece of obfuscation.

It is obfuscatory in so many ways it could truly qualify as a work of obfuscatory genius. It would be difficult for a mere mortal to pen so much confusion, ambiguity, distortion and misunderstanding in what is basically a book-length argument for a personal opinion masquerading as hard science.

Let me take just one tiny section of the book, one that is in no way atypical, and show you what I mean.

In Chapter 3 titled Turning Off Cancer, Dr. Campbell is starting to hit his stride in his anti animal protein jihad.  He has described the three stages of cancer – initiation, promotion and progression – and is setting the stage for his description of his laboratory work implicating animal protein in all three stages.

Here is his setup paragraph starting on page 50:

At the start of our research, the stages of cancer formation were known only in vague outline.  But we knew enough about these stages of cancer to be able to structure our research more intelligently.  We had no shortage of questions. Could we confirm the findings from India that a low-protein diet represses tumor formation?  More importantly, why does protein affect the cancer process?  What are the mechanisms; that is, how does protein work?  With plenty of questions to be answered, we went about our experimental studies meticulously and in depth in order to obtain results that would withstand the harshest of scrutiny.

The “findings from India that a low-protein diet represses tumor formation” were the results of a rodent study published in the Archives of Pathology in 1968 that Dr. Campbell wrote about 14 pages earlier in the book.  He mysteriously refers to the Archives of Pathology as an obscure journal when it is anything but.  It was published then by the American Medical Association and still is today under the new name Archives of Pathology & Laboratory Medicine.  But the notion of the paper initiating his quest being discovered by Dr. Campbell in an “obscure medical journal” fosters the impression of him as a leave-no-stone-unturned kind of guy.  Even the little throw away but incorrect phrase “obscure medical journal” is part of the greater picture of obfuscation that maintains throughout the book.

The study from India showed that rats given aflatoxin along with a high-protein diet got liver cancer while rats given the same amount of aflatoxin while consuming a low-protein diet didn’t.  Aflatoxin is a substance released from a fungus often found in peanuts, corn, other grains and even hay. It is converted in the liver to a much more toxic compound and is often used in laboratory experiments with animals to induce cancer and other problems.

Moving on, here is what Dr. Campbell has to say about protein and cancer initiation:

[I] How does protein intake affect cancer initiation?  Our first test was to see whether protein intake affected the enzyme principally responsible for aflatoxin metabolism, the mixed function oxidase (MFO).  This enzyme is very complex because it also metabolizes pharmaceuticals and other chemicals, friend or foe to the body.  Paradoxically this enzyme both detoxifies and activates aflatoxin.  It is an extraordinary transformation substance.

[II] At the time we started our research, we hypothesized that the protein we consume alters tumor growth by changing how aflatoxin is detoxified by enzymes present in the liver.

[III] We initially determined whether the amount of protein that we eat could change this enzyme activity.  After a series of experiments, the answer was clear (Chart 3.2).  Enzyme activity could be easily modified simply be changing the level of protein intake.

[IV] Decreasing protein intake like that done in the original research in India (20% to 5%) not only greatly decreased enzyme activity, but did so very quickly.  What does this mean?  Decreasing enzyme activity via low-protein diets implied that less aflatoxin was being transformed into the dangerous aflatoxin metabolite that had the potential to bind and to mutate DNA.

These four little paragraphs and accompanying chart take up less than a page in space, and are tiny glittering gems of obfuscation.  Let’s deconstruct.

First, take a look at how subtly these four paragraphs are written, especially II.  Note how he writes “the protein we consume”?  I’m sure many people took these paragraphs to mean that the studies were done on humans.  That’s almost the implication.  Reread them to see if they indicate anywhere that the author is talking about rat studies.

As Dr. Campbell progresses through this chapter, he does ultimately tell the reader he is talking about rat studies and not human studies, but he doesn’t mention the word rat for another two pages after the above paragraphs. By this time it’s probably implanted in the minds of many readers that he’s talking about human studies.

He describes experiments showing that rats getting diets high in casein (a milk/animal protein) develop more cancer at the same dose of aflatoxin than do rats getting a lower-casein diet.  The implication: animal protein causes cancer.

Dr. Campbell then gave his rats diets of varying amounts of plant protein (wheat gluten) and found that they did not get cancer after exposure to aflatoxin irrespective of protein dose.  Same thing happened with soy.  Implication: plant protein protects against cancer.

If you’re worried about cancer – and who isn’t – you’re now starting to look at animal protein a little differently.  Which is what Dr. Campbell wants.  But he hasn’t told you the complete story.

As I’ve written often in these pages, rodents aren’t just furry little humans.  They are a distinct species separate and apart from humans.  The rodents usually used in lab experiments are Sprague-Dawley rats, and inbred strain that has a tendency to develop cancer easily. (See Abelson, PH. (1992) Diet and Cancer in Humans and Rodents, Science 255(5041); Jan 10: 141)  In fact, these rats can develop cancer just from a change in diet.  I ran quick checks on a bunch of the studies referenced in The China Study, and all checked used Sprague-Dawley rats.

And think about this.  If you were to visit a farm and search for rodents, where do you think you would be most likely to find them?  In the grain or in the milking area?  Like Dr. Campbell, I grew up in a rural area and spent a lot of time on a farm.  Rats and mice are in the hay and in the grain.  You have a helluva time keeping them out of the animal feed, which they eat, too.  Grain and hay are common places for growth of the fungus that produces aflatoxin.  Since rodents spend most of their days in this stuff (grains), and since they eat it as well, I would bet that most have adapted over the generations to  the combination of plant protein and aflatoxin.  If this did them in regularly, there wouldn’t be the rodent problem on farms that there is.  So, in my opinion, making a huge issue of the fact that rats didn’t get cancer after dosing with aflatoxin irrespective of how much plant protein they ate is pretty disingenuous.

Most disingenuous of all in the above four paragraphs and chart is the lack of full disclosure in these paragraphs of the very study Chart 3.2 is made from. Let me explain.

Certain enzymes in the liver convert aflatoxin into a more toxic substance that Dr. Campbell claims can initiate the formation of cancer.  He demonstrates in rat studies that giving the rats a lower protein diet decreases the activity of this enzyme, meaning that the lower the protein intake, the less conversion of the aflatoxin into the really nasty stuff.  Chart 3.2 above and on page 52 of his book shows this graphically.

When I pulled the study from which this chart was adapted (Mgbodile MUK and Campbell TC. (1972) Effect of protein deprivation of male weanling rats on the kinetics of hepatic microsomal enzyme activity, J Nutr, 102: 53-60.) and read it, I found a little disclaimer Dr. Campbell didn’t bother to mention in The China Study.  You can read the last paragraph of the study (highlighted in yellow) below:

Nice, eh?  He hits the nail on the head.  Protein utilization may be influenced by what is eaten along with the protein.  Sucrose (table sugar) was eaten along with the protein used in this experiment.  In other experiments corn starch was used instead of sugar and the effect of the protein on the enzyme was diminished, meaning that the protein along with starch did not have nearly the same effect as protein with the sugar.  Who knows whether or not it’s even the protein that has the effect and not the sugar?  It can’t be shown from this study. That caveat certainly didn’t make in into The China Study.

See what I mean about a masterpiece of obfuscation?

I could go on and on, but I’ll quit after I give you just a couple more examples.

On page 107 of The China Study Dr. Campbell writes:

At the end of the day, the strength and consistency of the majority of the evidence is enough to draw valid conclusions.  Namely, whole plant-based foods are beneficial, and animal-based foods are not.

Then one inch below (literally) he writes the following:

The China Study was an important milestone in my thinking.  Standing alone, it does not prove that diet causes disease. [Italics in the original]

So, the China study produces valid conclusions as to causality, i.e., “whole plant-based foods are beneficial, and animal-based foods are not.”  Yet the China study “does not prove that diet causes disease.”  Say what?

Don’t believe me, take a look at a scan of my copy:

On page 73 Dr. Campbell dons the mantle of prestige conferred by one of America’s most august newspapers.  Writes he referring to the China study:

We had a study that was unmatched in terms of it’s comprehensiveness, quality and uniqueness.  We had what the New York Times termed “the Grand Prix of epidemiology.”

A quick search of that phrase in the online version of the NY Times reveals that it came from an opinion piece by none other than Jane Brody, a kindred spirit to Dr. Campbell.  Brody, a lipophobe of the deepest hue, has written a number of low-fat cookbooks and is a believer in the plant-based diet. So she hardly qualifies as an unbiased commenter.

And speaking of the so-called plant-based diet, when Dr. Campbell responded to Ms. Minger’s critique, he took her to task for mentioning the words ‘vegan’ and ‘vegetarian’ as it applied to his work.

One final note: she repeatedly uses the ‘V’ words (vegan, vegetarian) in a way that disingenuously suggests that this was my main motive. I am not aware that I used either of these words in the book, not once. I wanted to focus on the science, not on these ideologies.

Just for grins, I turned to the index of The China Study to see if ‘vegan’ or ‘vegetarian’ were indexed.  Here’s what I found on page 417:

vegetarianism or veganism. See plant-based diet

When I flipped over  to ‘plant-based diet’ on page 414, I found a long grocery list of references.

Even in his online response to his opponents, Dr. Campbell apparently can’t resist obfuscating.

Okay, just one more, then I’ve got to draw this to a close.  Let’s go back to the bottom of page 52, the page the paragraphs above and Chart 3.2 appear on.  Dr. Campbell shows in Chart 3.2 how protein is involved in stimulating the liver to convert aflatoxin to the toxic product that he implies is involved in cancer initiation.  He then reports how he wanted to see if animal-based protein was involved in the other phases of the cancer progression cascade.  So he and his grad students started to look.  He writes:

As time passed, we were to learn something quite remarkable. Almost every time we searched for a way, or mechanism, by which protein works to produce its effects [on cancer formation and progression], we found one!

That, my friends, is almost the dictionary definition of the confirmation bias summed up in one sentence.

This tiny bit of the book that I’ve chosen to lay bare is truly the tip of the iceberg.  I could go on and on and on, but I’m sure you get the picture.

Before I finish, I want to get back to something I mentioned earlier about how one of Dr. Campbell’s own colleagues bailed out from the China study because he recognized it for what it was: a giant observational study that was meaningless.  Here is how Dr. Campbell describes it on page 105-106:

When we first started this project we encountered significant resistance from some people.  One of my colleagues at Cornell, who had been involved in the early planning of the China Study, got quite heated in one of our meetings.  I had put forth the idea of investigating how lots of dietary factors, some known but many unknown, work together to cause disease.  Thus we had to measure lots of factors, regardless of whether or not they were justified by prior research.  If that was what we intended to do, he said he wanted nothing to do with such a “shotgun” approach. [i.e., a big, meaningless observational study]

This colleague was expressing a view that was more in line with mainstream scientific thought than with my idea [i.e., a randomized, controlled trial that might demonstrate causality would be a better use of the funds.] He and like-minded colleagues think that science is best done when investigating single – mostly known – factors in isolation. [He and like-minded colleagues are correct.] An array of largely unspecified factors doesn’t show anything, they say. [They are right.] It’s okay to measure the specific effect of, say, selenium on breast cancer, but it’s not okay to measure multiple nutritional conditions in the same study, in the hope of identifying important dietary patterns.

I prefer the broader picture, for we are investigating the incredible complexities and subtleties of nature itself…

So I say we need more, not less, of the “shotgun approach.” We need more thought about overall dietary patterns and whole foods.  Does this mean that I think the shotgun approach is the only way to do research?  Of course not.  Do I think that the China Study findings constitute absolute scientific proof?  Of course not.  Does it provide enough information to inform some practical decision-making? [No.] Absolutely.

Dr. Campbell uses an impassioned written speech to persuade the scientifically untrained that the China study carries vastly more scientific value than it actually does. Once again, it’s a large observational study, but an observational study nonetheless.  And as such, it is useful only in developing hypotheses to be tested with randomized, controlled trials.  The entire 894 page study proves not a shred of causality.

What saddens me about all this is that hundreds of thousands (probably millions) of people who can’t (or won’t) read critically have fallen for the premise of The China Study without even thinking about it.  Believing that the entire book is based on the greatest and most important nutritional study ever completed.  What happened to the ability to read critically?  Has it vanished from the populace?  Based on the comments on The China Study on Amazon it would seem so.

In my opinion, there really isn’t much of substance in the entire 400 plus page book.  But I encourage you to buy it and read it to test your own critical reading skills.  If you don’t want to test your critical reading skills, you’ll at least enjoy coming across some real howlers such as this one believed only by the vegetarian/vegan zealots out there (oh, sorry, plant-based diet followers):

As you will see in this book, there is a mountain of scientific evidence to show that the healthiest diet you can possibly consume is a high-carbohydrate diet. [italics in the original]

I wonder if Gary Taubes, who wrote a vastly more scientific book, would agree?

Lest you think I’m being too hard on poor Dr. Campbell, let me tell you a few things.  First, as I mentioned earlier, the few sections of The China Study I dissected are just a tiny fraction of the whole.  I could go on and on. Second, Dr. Campbell mentions Protein Power by name on page 19 and labels it a modern protein fad diet that “continue[s] to inflict a great variety of dangerous health disorders.”  Third, he is absolutely and unnecessarily brutal in his treatment of Dr. Robert Atkins.  He has an entire section on Dr. Atkins starting on page 95 that runs for almost three pages.  After quoting from one of Dr. Atkins’ books, he writes the following about the deceased diet doctor:

There are snake oil salesman, who have no professional research, professional training or professional publications in the field of nutrition, and there are scientists, who have formal training, have conducted research and have reported on their findings in professional forums. Perhaps it is a testament to the poser of modern marketing savvy that an obese man with heart disease and high blood pressure [here he inserts a citation for an article discussing Dr. Atkins’ death] became one of the richest snake oil salesmen ever to live, selling a diet that promises to help you lose weight, to keep your heart healthy and to normalize your blood pressure.

A way below-the-belt commentary when you consider that Dr.Atkins was a trained cardiologist who took care of thousands of real, live patients throughout his career – he wasn’t, like Dr. Campbell, a bench scientist doing rat studies in a lab.  Bob Atkins and I have had our differences, but were he still alive, I would vastly prefer to put my own care in his hands than I would those of Dr. Campbell, who has never treated a patient in his life.

You may ask if I took anything of value from my reading of this book?  I did.  On page 107 Dr. Campbell writes the following:

The results of this study…convinced me to turn my dietary lifestyle around. I stopped eating meat fifteen years ago, and I stopped eating almost all animal-based foods, including dairy, within the past six to eight years, except on very rare occasions,  MY cholesterol has dropped, even as I’ve aged; I am more physically fit now than when I was twenty-five; and I am forty-five pounds lighter now than  was when I was thirty years old.  I am now at an ideal weight for my height.

I have no reason to doubt Dr. Campbell’s own medical and dietary history (except maybe for the part about being more physically fit than he was at age 25 – that’s a tough act for someone who is 73), so I’ll assume it’s all true.  As I recall, he had a trim physique when I met him 10 years ago, which, assuming nothing has changed, is probably the same.  And I’m going to take Dr. Campbell at his word about what he eats.

Granted, I’m younger than Dr. Campbell, but I follow almost the opposite diet as he does yet I, too, have low cholesterol, very low blood pressure and am ideal weight for my height.  What this all tells me is how wonderfully adaptive the human species is where diet is concerned.  It’s no wonder we took over the earth.

** Lillian Hellmann was predictably furious over McCarthy’s comment and adopted the typical American response: she sued.  In one of those turns in which the law of unintended consequences jumps up and bites one, many of her untruths came to light in the courtroom as McCarthy was forced to defend her statement.  Hellmann disengaged by dying during the proceedings.

As if trying to pull meaning out of the medical literature weren’t difficult enough as it is, a new study demonstrates yet another obstacle to easy understanding: the adherer effect.

We’ve all seen the headlines.  Statins improve bone health.  Statins prevent cancer.  Statins make us smarter.  Low-fat diets improve longevity.  All these headlines and others like them are followed by articles describing studies seeming to show that subjects taking certain medications (usually statin drugs, it seems) or following a particular diet have improvements in health and/or longevity.  The promise of these articles is that if we all take the medication or follow the lifestyle choice, we, too, will reduce our risk of [fill in the blank] or live longer.  But will we?

Maybe so.  But not for the reason most people think.

The adherer effect demonstrates that people who adhere to medical or lifestyle regimens end up with better outcomes than those who don’t…even if the regimens are nothing but placebo.

I mentioned this phenomenon in an earlier post.

Almost thirty years ago a study was published in the New England Journal of Medicine looking at this very idea. [The adherer effect]  The study that inspired the article didn’t start out looking at this idea, but one of the investigators noted a key piece of the data and published on it.  The study was looking at clofibrate, a pre-statin cholesterol lowering drug,  and all cause mortality.  Subjects were randomized into two groups – those in one group got the drug, those in the other got the placebo.  After the subjects were on either the drug or the placebo for five years, researchers calculated the mortality from the number of deaths in each group.  Turned out that the five-year mortality of those on clofibrate was 20.0 percent whereas the five-year mortality of those on the placebo was 20.9 percent, or essentially the same.  Taking the drug was no different than taking the placebo, i.e., the drug was worthless. Had one of the researchers not looked a little closer, that would have been the end of the story.

When the data were looked at from the perspective of how many people actually took the drug as prescribed, the researcher discovered that those subjects who took at least 80 percent or more of their clofibrate had a five year mortality of only 15.0 percent, substantially less than the overall five-year mortality.  Those who took their clofibrate sporadically had a five-year mortality of 24.6 percent, significantly higher than those who took it as directed, a piece of data that would seem to confirm the efficacy of clofibrate.  Right?  Not necessarily.  Let’s look at compliance with the placebo.

Turns out that those subjects on the placebo who regularly took their placebo had a five-year mortality of 15.1 percent while those who took their placebo sporadically had a five-year mortality of 28.3 percent.  What this study really showed was that there is something intrinsic to people who religiously take their medicine that makes them live longer.  There was no difference between the drug and placebo in either those who took them regularly or those who took them sporadically, but there was a huge difference in mortality between those who took either drug or placebo on schedule and those who didn’t.

Gary Taubes discussed this same study and the adherer effect in  a long article he wrote for the New York Times Magazine a few years ago:

A still more subtle component of healthy-user bias has to be confronted. This is the compliance or adherer effect. Quite simply, people who comply with their doctors’ orders when given a prescription are different and healthier than people who don’t. This difference may be ultimately unquantifiable. The compliance effect is another plausible explanation for many of the beneficial associations that epidemiologists commonly report, which means this alone is a reason to wonder if much of what we hear about what constitutes a healthful diet and lifestyle is misconceived.

The lesson comes from an ambitious clinical trial called the Coronary Drug Project that set out in the 1970s to test whether any of five different drugs might prevent heart attacks. The subjects were some 8,500 middle-aged men with established heart problems. Two-thirds of them were randomly assigned to take one of the five drugs and the other third a placebo. Because one of the drugs, clofibrate, lowered cholesterol levels, the researchers had high hopes that it would ward off heart disease. But when the results were tabulated after five years, clofibrate showed no beneficial effect. The researchers then considered the possibility that clofibrate appeared to fail only because the subjects failed to faithfully take their prescriptions.

As it turned out, those men who said they took more than 80 percent of the pills prescribed fared substantially better than those who didn’t. Only 15 percent of these faithful “adherers” died, compared with almost 25 percent of what the project researchers called “poor adherers.” This might have been taken as reason to believe that clofibrate actually did cut heart-disease deaths almost by half, but then the researchers looked at those men who faithfully took their placebos. And those men, too, seemed to benefit from adhering closely to their prescription: only 15 percent of them died compared with 28 percent who were less conscientious. “So faithfully taking the placebo cuts the death rate by a factor of two,” says David Freedman, a professor of statistics at the University of California, Berkeley. “How can this be? Well, people who take their placebo regularly are just different than the others. The rest is a little speculative. Maybe they take better care of themselves in general. But this compliance effect is quite a big effect.”

In the same blog post of mine I linked to above, I wrote about another study showing the adherer effect, showing graphically how potent the phenomenon is.

Previously, the study of the adherer effect has been a secondary finding in studies of various drug regimens, but now comes a paper in which the adherer effect is the primary focus of the investigation.  Based on the data in this recent paper, the effect is robust and should be accounted for in the analysis of any data generated when subjects following a particular treatment are compared to those who don’t.

The authors lay out the problem:

The healthy-user effect [the adherer effect] is a hypothetical source of confounding bias that is thought to affect observational studies of drugs, diets, screening procedures, and other health-related behaviors. This bias presumes that patients who initiate and adhere to preventive therapies are more likely to engage in behaviors consistent with a healthy lifestyle than are patients who do not initiate or adhere to such treatments. Aspects of a healthy lifestyle could include diet, exercise, moderation of alcohol, and avoidance of risky behaviors. These characteristics, which are unmeasured in typical pharmacoepidemiological databases, may be associated with morbidity and mortality outcomes in observational studies. Thus, failure to adjust for them can lead to bias in studies of preventive therapies.

The healthy-user bias has been suggested as an explanation for the discrepancy between several experimental and observational studies, including studies of the effects of long-term use of estrogen therapy and vitamin E. It has also been discussed as a potential source of bias in observational studies of the effectiveness of influenza vaccines in the elderly  and the association between use of 3-hydroxy-3-methylglutaryl coenzyme A reductase inhibitors (statins) and reduced risk of hip fracture,  Alzheimer disease,  sepsis,  cancer,  and mortality.  This bias has also been observed in randomized controlled trials in which adherence to placebo was found to be associated with decreased mortality.  Although long suspected as a source of bias, a paucity of empirical data exists on the healthy-user effect.

Here’s how the study was set up.

It was really pretty simple.  The researchers looked at a group of patients who had been prescribed one of a variety of statin drugs and followed their compliance by looking at how many times these subjects picked up their medicines in the year following their prescription.  The typical statin prescription was for 60 days worth of the medication, and all subjects had available to them a full year’s worth of medicines.  The researchers grouped subjects into two groups: one group who took the trouble to go get over 120 day’s worth of the medication (the “more adherent”) and one group of subjects who were dispensed under 120 days of meds (the “less adherent”).

All subjects entered into the study were evaluated after the one year baseline study period during which their effort to follow their prescribed statin regimen sorted them into the categories of more adherent or less adherent.  The researchers were looking to see which subjects – the adherers or the non-adherers – would develop problems that had nothing to do with the statin drugs.

We evaluated a spectrum of events after the 1-year baseline period to assess the healthy-adherer bias. The outcomes were grouped into 4 broad categories: accident events, screening events, other events not expected to be associated with statin exposure, and other events for which a possible association with statin exposure could be expected. We included inpatient and outpatient events as well as primary and secondary diagnoses.

When the data on these 141,086 subjects was crunched, it turned out that the more adherent subjects had significantly fewer accidents, especially motor vehicle and workplace accidents.  The more adherent also had a lower likelihood of developing other disorders that were not likely to be attributed to the effect of the statin drugs.

In other words, whatever characteristic it was that made subjects hang in there with their statin prescriptions also made them less likely to indulge in risky behaviors and less likely to develop all kinds of medical problems.  Why?  Probably because these people were simply more health conscious, kept themselves in better shape, and didn’t act impulsively.

The real take-home message from this study is that the adherer effect significantly affects the outcome of drug and lifestyle intervention studies.  If you see a study that says those subjects using statin drugs developed 20 percent fewer problems (of whatever kind are being studied) than those who don’t use statins, you can be sure that the adherer effect is at work.  This adherer effect is why randomized, double-blind, placebo-controlled studies are needed to determine the efficacy of any drug, and even then the adherer effect should be controlled for.

There is a big note, enclosed in a box and titled Clinical Perspective, at the end of this study that exhorts doctors to consider this adherer effect when looking at data from observational studies.  Here is the note in full.

Clinicians need to read observational studies reporting surprising benefits of drug therapy with a healthy skepticism. Observational studies of preventive medications and health behaviors are susceptible to various sources of bias, including the so-called healthy-user and healthy-adherer biases. In this article, evidence of the healthy-adherer effect is demonstrated by showing that adherence to statins is associated with a reduction in the risk of accidents (eg, workplace or motor vehicle), outcomes that would not be expected to be affected by a statin. The approximate magnitude of the adherer effect was a 15% relative risk reduction. The most likely explanation for this association is that good adherence to statin therapy is a marker for other healthy behaviors, most of which cannot be accounted for in this type of study. In keeping with this explanation, the study also shows that adherence predicts a 7% to 17% increased incidence of medical screening procedures (eg, fecal occult blood testing, mammography). Risk of myocardial infarction, which has been demonstrated to be reduced by statin therapy in randomized placebo-controlled trials, was found in this study to be reduced by 28%. This observed relative reduction must be interpreted as reflecting a combination of the healthy-adherer effect and the drug effect. Clinicians can also learn from this study that patients who follow their advice are also likely to have other healthy behaviors and a lower risk of adverse events.

It is unfortunate, but I doubt that many doctors (or researchers, for that matter) will consider the adherer effect when they read these studies.  I would bet that we will continue to see studies reported as if the positive effects found were a function of the drug or lifestyle regimen studied and not the adherer effect.

To me the saddest part of this study was the statistic that of the 141,086 subjects in this study, 49 percent were women.  The randomized, double-blind, placebo-controlled studies of statins have never shown a benefit in terms of decreased all-cause mortality in women of any age.  Which means that over 70,000 women in this study took a drug that would do them no good, but which could well cause them significant and harmful side effects.

In this study, those who dropped out of their statin regimen because of intolerable side effects would be considered to be less adherent or non adherers.  My guess is that many of these ‘non adherers’ who dropped out because of side effects were really ‘adherers’ by nature.  Had these drop outs due to side effects been controlled for, I would bet that the difference between the less adherent and the more adherent would have been much larger than the data showed.

The news is abuzz with reports of the latest study to come out showing that eating meat, especially red meat, kills us off before our time.  (You can read some of the reporting here, here, here and here.)  Google shows 547 new articles about this study.

Although this study is totally worthless from a causality perspective because it is an observational study, it does serve to confirm the biases of those non-critical thinkers who have already bought into the idea that meat is bad.  To give you an example of such a soft thinker, here is the second comment on the blog post about this study in the New York Times.

I could have told you that 30 years ago. I been a vegetarian for 47 years and I have never seen vegetarians die from heart disease or cancer. They died from basic infectious diseases and malnutrition. Make no mistake it is harder to be a vegetarian than a carnivour but your body does not expel everying [sic] that is in the meat especially red meat.

Red meat is the major culprin [sic] in colon cancer. I actually know people who have colon cancer gene that only eat a no red meat diet and have no issues with their colon. Of course they also do not smoke or drink too much alcohol.

Red meat lobby is very powerful in America – Let them pay for this!

Ah, yes, an enlightened cogitator indeed.

The study published in the Archives of Internal Medicine (free full text here) is a typical epidemiological or observational study.  The reports have it tarted up with a lot of fancy clothes, but it is really nothing but an observational study.  And, as we’ve gone over ad nauseum in these pages, observational studies can’t be used to prove causation.

Even if they could, this study would be questionable at best because the relative risk (RR) is slightly over 1.0.  Because of the nature of the difficulty in doing these kinds of studies with any kind of accuracy it takes a RR of over at least 2.0 to get the serious attention of anyone who doesn’t have a built-in bias.

What I found more interesting than this study (which isn’t interesting or important at all) was the press coverage of it.  And I found especially interesting that which the press didn’t report.

Scientific journals have a couple of ways of getting their articles out there ahead of publication, so that the press can do stories on them.  If it works out right, the reports all hit the media on the same day that the article itself is published.  Doctors who read the journal often find out in their morning newspaper about a new paper before they even get their journal in the mail later that same day.  Many of the larger journals, Archives of Internal Medicine, for example, will issue press releases the week before on those papers coming out that the editors feel are important.  These press releases go to anyone with press credentials (I even get them), and are embargoed until the date of publication of the journal.  Reporters get advanced copies of the papers and get the editor’s (and maybe even the author’s) take on the paper.  Journalists can then write their stories to be timed with publication of the paper.

Another way followed by a number of journals is to publish papers online in advance of their actual publication date.  Reporters troll these advanced online articles looking for material for stories and often write them up for publication before the paper in question makes it into actual publication in the journal.

At the same time that this paper appeared, showing increased red meat consumption to be tied to a slight increased risk of death (and showing that those subjects eating white meat had less risk), a couple of other papers came out in the online pre-publication section of the American Journal of Clinical Nutrition (AJCN), arguably the world’s most prestigious nutritional scientific journal.  These two AJCN papers saw the light of day at around the same time as this highly-publicized study on meat and mortality, but demonstrated the opposite results.  They got no press coverage whatsoever.  Which proves what I’ve been saying all along: the press is biased against meat in general, and especially against red meat.  Knowing this, careful readers should take anything negative thing the media reports about red meat with an enormous grain of salt.

Let’s look at the other two studies published in AJCN.

The first is titled Meta-analysis of animal fat or animal protein intake and colorectal cancer.  One of the constant themes anti meat people like to hammer out is that meat intake, especially red meat intake, causes colon or colorectal cancer.  This is heard so often that most people take it for granted, assuming that there must be a ton of research backing it up.  As this paper points out, there isn’t.

The association between total dietary fat, including fat constituents such as saturated fat, monounsaturated fat, polyunsaturated fat, and cholesterol, and risk of colorectal cancer has been evaluated in numerous epidemiologic [observational] studies.  Results from these analytic investigations have generally been mixed.  Whereas some studies have reported positive associations, several studies have observed null and inverse associations.  In a pooled analysis of data from 13 case-controlled studies, risk of colorectal cancer was found to increase significantly with increasing categories of total daily energy intake.  In the same analysis, and after adjustment for total energy intake, the authors observed no evidence of an energy-independent effect of total dietary fat or specific fat components other than cholesterol.  In fact, many of the associations among men and women were in the inverse direction [i.e., more animal fat equals greater longevity].

Animal foods and meat products contain both saturated and unsaturated fats; however, similar to analyses of total fat intake, several studies have not observed any consistent epidemiologic evidence of an association between saturated fat or polyunsaturated fat intake and risk of colorectal cancer.  Although some studies reported positive associations for consumption of saturated fat, nonsignificant associations at or near the null value [no association] or inverse associations have been observed in numerous cohort studies and case-control studies.

This paper goes on to discuss how the hypothesis that fat and meat intake are a bad thing healthwise got kicked off way back in the 1960s from a presentation at a symposium. In shades of Ancel Keys and his discredited Seven Countries Study, a researcher named Ernst Wynder used the international food and cancer mortality data to demonstrate an increase in colorectal cancer as a correlate of increasing oil and fat consumption.  The hypothesis, although never proven, has been with us since.  The authors of this paper set out to study it once again.

Here is what they did:

To clarify the potential association between animal fat intake and colorectal cancer, we conducted a meta-analysis of prospective cohort studies in which data for animal fat were available.  In addition, we identified case-control studies that reported results for animal fat intake and combined data from these studies with the prospective cohort data in separate analyses.  Because the primary macronutrients in the consumption of animals include protein and fat, we also conducted a separate meta-analysis of prospective cohort studies in which data categorized as animal protein or meat were available.

After sifting through all this data, what did the authors find?  Absolutely nothing.  No correlation between meat and/or fat intake and colorectal cancer.

In this meta-analysis, no consistent evidence of a positive association between  consumption of animal fat and colorectal cancer was observed.  Specifically, we found no association  between the highest animal fat intake category and colorectal cancer.  Furthermore, none of the subgroup analysis (i.e., sex, anatomic tumor site, and study design) indicated positive patterns of associations.

And their conclusion:

On the basis of the results of this quantitative assessment, the available epidemiologic evidence does not appear to support an independent association between animal fat intake or animal protein intake and colorectal cancer.

Like the study above showing the slight correlation between red meat intake and decreased longevity, this study is an observational study, and, as such, doesn’t demonstrate any kind of definitive proof.  But what I find galling is that the meat and mortality study hit all the airwaves and this study – made available to the media at the same time – received zero press.

Yet another study in the advanced online section of AJCN titled Mortality in British vegetarians: results from the European Prospective Investigation in Cancer and Nutrition (EPIC-Oxford) shows that things ain’t always as they seem.  Yet the press refuses to pick up and report this man-bites-dog story.

If you ask the man on the street (who has been fed a load of bunkum over the years by the press) if vegetarians or non-vegetarians are healthier and live longer, you will almost assuredly be told that vegetarians are the healthiest.  Most people believe this, but they just don’t want to make the sacrifice to follow the vegetarian lifestyle.  They are willing to give up a couple of years of life to not have to live on a steady diet of beans, tofu, vegetables, fruits and dry bread.  You would think that if a study came out from a prestigious institution (Oxford) published in a top-line scientific journal showing that vegetarians don’t live any longer than non-vegetarians and actually have a higher incidence of some particularly nasty cancers (but slightly lower rates of death from heart disease) it would be newsworthy.  But the press has totally ignored this study just like they did the last one.

This vegetarian study was interesting on a couple of levels.  Not only did it not show a difference in longevity between vegetarians and nonvegetarians, it showed major increases in longevity just from being in the study.  Not long ago I wrote a post about a statin study in which I discussed the adherer verses the non-adherer effect.  A number of studies have shown that subjects who take all their medicines as directed – even the placebos – live longer and/or do better than those who takes their medications irregularly.  There is something about people who go the extra mile that makes them live longer than those who don’t.

In this Oxford University vegetarian study, vegetarian subjects were recruited by all sorts of methods.  Those in the study cast out their nets for other vegetarians and recruitment was done through all kinds of advertising venues.  Those accepted into the study -both vegetarians and non vegetarians – had to jump through a fair number of hoops to get accepted and stay in the study.  And to stay in the study for the ten plus years that it went on.  After the study period, the numbers of deaths in the two groups was tallied, and it was found that vegetarians didn’t live any longer than non-vegetarians.  As a percentage, the number of deaths in each group was the same.

What’s more interesting to me, however, is the difference between the rate of deaths in both the vegetarian and non-vegetarian subjects as compared to their neighbors who weren’t in the study.  The researchers calculated the standard mortality ratios (SMRs) for vegetarians and non-vegetarians from deaths before the age of 90 years old as compared to the mortality rate for non-study subjects living in the same area.

The SMR is the ratio of the observed number of deaths to the number of deaths expected from the national rates, standardized for sex and age, and expressed as a percentage.

In other words, if the observed number of deaths in the study group had been three quarters of that expected in a similar population from the area, the SMR would have been 75 percent.  And would have been a striking finding to boot.  It would have meant that just being in the study reduced one’s risk of death.

When all the data was tallied, the SMR for all causes of death among study subjects was only 52 percent, and was identical in vegetarians and nonvegetarians!  It didn’t matter if you were a vegetarian or a nonvegetarian, as long as you were in this study you were about half as likely to die as your neighbor who wasn’t in the study.  Now that’s an adherer effect in spades.  And I would think pretty newsworthy.  But, like the study above on meat and colorectal cancer, it was completely ignored by the press.

The point of this post is that you shouldn’t get wound up about a study that gets reported throughout the media because there are more than likely other studies that are just as well done and just as important showing exactly the opposite findings that the press chooses to ignore.  You’re not seeing the science as it is, you’re seeing the science as the press wants you to see it, which, typically, is the way that confirms the bias of members of the press.

As a journalist friend of ours once remarked:  what is news?  News is whatever the reporter decides it is.  In my opinion, they decided wrongly in this case.

I’ve had a number of people email me about a new study appearing in the Archives of Internal Medicine purportedly showing that statins really do provide benefit to those who take them regularly.  As you can see from the heading of an email piece I pasted above, even Medscape is all over this article and blasting it out to physicians all over the world.

I’m sad to say that this is the same kind of paper I would have been taken in by 20 years ago before I really understood how to read the scientific literature critically.  In fact, I would have used it myself to justify giving statins to all kinds of people, and I’m sure other physicians are doing so right now.  But I would have been in error to base my prescribing on this paper, and all the other docs out there giving statins like they were candy are in error as well.

If you don’t want to read a dissection of this study, let me just tell you up front that it doesn’t really mean a thing.  It certainly doesn’t prove that you should rush out and get started on statins.  If, however, you do want to learn about how perniciously deceptive these kinds of studies are and how to analyze them, read on.

Here’s the deal.  Researchers went back and combed through the records of a large HMO in Israel and pulled those of patients who had been prescribed statins from 1998-2006.  Since the HMO provided the statin prescriptions, there were records of how many of these people who were prescribed statins actually filled their prescriptions (and, one would assume, took the medications).  Then the researchers figured out how many of those people prescribed statins died.  The final step was to compare the list of those who died with the list of those who took their statin prescriptions (or, more accurately, those who filled their statin prescriptions).  After crunching all this data, it turns out that those patients who filled over 90 percent of their prescriptions were 45 percent less likely to die than those who filled under 10 percent of their prescriptions.  Which, to the uncritical reader (including, obviously the Medscape writers and the peers who reviewed this piece for the journal in which it was published), this appears to be pretty persuasive evidence that statins confer some kind of benefit in terms of preventing death.  After all, those that took them lived while those who didn’t died.

As I say, these kinds of studies are pretty beguiling.  But do they really mean anything?

Before we get to the specifics of this study, let’s contemplate this type of study in general to see why the data they generate is often misleading.

The gold standard for scientific studies is the randomized, double-blind, placebo-controlled trial.  In this type of study, researchers randomize the study population into two similar groups and give the members of one group the drug being studied and the other a placebo.  Double blinded means that neither the researchers nor the subjects know who got what.  At the end of the trial, the data are analyzed to determine if the study drug really showed any difference in efficacy as compared to the placebo.  If it did, then it can be said that the drug works to treat whatever condition was being studied.  Or that it decreases all-cause mortality, if that is the end point of the study.

It’s impossible to do these gold standard studies with diet and/or exercise because a) they involve lifestyle changes and b) they can’t be double blinded.  When it comes to diet and exercise, there are basically two ways studies can be done.  Researchers can allow subjects to self-select which arm of the study they want to be in.  Or researchers can put subjects into one arm or the other.  Neither of these choices is optimal, but they are all that are available.

If I decide that I’m going to compare a very-low-carb diet to a very-low-fat diet, I can recruit volunteers and ask them which diet they would prefer.  If readers of this blog were recruited into such a study, I would assume most would opt for the very-low-carb diet.  Those who are fans of Dean Ornish would opt for the other.  What you end up with is people in each arm of the study who are already believers in the diet they will be following, and they will be more likely to remain on the diet until the end of the study.  At the end, the data will be a little polluted because it really doesn’t prove that one diet is superior to the other – it only proves that people who self-select into that diet do better on that diet than people who self-select into the other.  The last it an important point, especially when applied to exercise.  More about which in a moment.

The other way to study diet is to gather a group of people together and randomize them into one diet group or the other.  That takes the self-selection bias out of the equation.  But it creates other problems.  If a person committed ideologically to a low-carb diet gets randomized into the low-fat group (or vice verse) there are problems with compliance.  Most nutritional studies randomized this way end up with large numbers of dropouts.  If you do an intention-to-treat analysis of the data (which includes the drop outs), you usually find little difference between the two diets.  If you look at only those subjects who hung in there for the duration on whichever diet they were randomized onto, it raises the issue of whether these subjects may have been the same ones who would have self-selected themselves into this same diet if given the chance, which then creates the same problems as self-selection.  These issues make diet studies difficult to do and difficult to interpret validly.  It’s even worse with exercise.

I get a ton of email and comments from people who can’t come to grips with the idea that there is no proof that exercise brings about weight loss.  I say this because it is difficult to come by this proof.  Even those who are adamant that exercise brings about weight loss agree that pretty intensive exercise is required to do so.  The typical prescription to just get out and move a little more virtually everyone realizes is worthless.  Most people believe that it’s intensive exercise that does the trick.  Maybe so, but how do you prove it?

If you randomize people into an intensive exercise group and another into a no exercise group to see which loses the most weight (assuming diet is held constant), how many of those sedentary people are going to stick with the intensive exercise for any length of time.  They will be the dropouts.  If you allow people to self select, all the people who enjoy exercise will put themselves into the exercise group while those who hate it will put themselves into the sedentary group.  Then if those in the exercise group do lose weight, how can you tell it’s the exercise and not due to some other component of a person who will commit to an intensive exercise program that brings about the weight loss?  The answer is that you can’t tell.  Which is why the notion that exercise brings about weight loss is similar to a particular religious belief: it is accepted as an article of faith, not as a product of scientific investigation.

You can send me a comment (as several people have done) telling me how you were stuck in your weight loss efforts at 220 pounds and then you decided to start high intensity interval training.  After a couple of months of this, you lost 25 more pounds.  Therefore that’s proof that exercise brings about weight loss.  Wrong!  That’s proof that in you exercise brought about weight loss.  There may be something different about you that allows you to commit to such a regimen that others might have difficulty following AND allows you to lose weight.  This sounds ridiculous, but it is true.  And it is the key to understanding why this statin study is bogus in terms of whether or not taking statins makes people live longer.

Almost thirty years ago a study was published in the New England Journal of Medicine looking at this very idea.  The study that inspired the article didn’t start out looking at this idea, but one of the investigators noted a key piece of the data and published on it.  The study was looking at clofibrate, a pre-statin cholesterol lowering drug and all cause mortality.  Subjects were randomized into two groups – those in one group got the drug, those in the other got the placebo.  After the subjects were on either the drug or the placebo for five years, researchers calculated the mortality from the number of deaths in each group.  Turned out that the five-year mortality of those on clofibrate was 20.0 percent whereas the five-year mortality of those on the placebo was 20.9 percent, or essentially the same.  Taking the drug was no different than taking the placebo, i.e., the drug was worthless. Had one of the researchers not looked a little closer, that would have been the end of the story.

When the data were looked at from the perspective of how many people actually took the drug as prescribed, the researcher discovered that those subjects who took at least 80 percent or more of their clofibrate had a five year mortality of only 15.0 percent, substantially less than the overall five-year mortality.  Those who took their clofibrate sporadically had a five-year mortality of 24.6 percent, significantly higher than those who took it as directed, a piece of data that would seem to confirm the efficacy of clofibrate.  Right?  Not necessarily.  Let’s look at compliance with the placebo.

Turns out that those subjects on the placebo who regularly took their placebo had a five-year mortality of 15.1 percent while those who took their placebo sporadically had a five-year mortality of 28.3 percent.  What this study really showed was that there is something intrinsic to people who religiously take their medicine that makes them live longer.  There was no difference between the drug and placebo in either those who took them regularly or those who took them sporadically, but there was a huge difference in mortality between those who took either drug or placebo on schedule and those who didn’t.

Lest you think this was a bizarre one-of-a-kind study, another study published a few years ago in The Lancet showed a virtually identical outcome.  Patients taking a medication for congestive heart failure were compared to those taking placebo.  Those taking the drug (Candesartan) showed no difference in mortality compared to those taking placebo.  But when compliance was evaluated, those taking either the drug or the placebo as directed had much lower mortality than those taking either one sporadically.  In fact, as you can see from the graph below, the mortality curves were almost identical.

From Lancet (2005); 366(9502):2005-2011

So there is something about adherers to a drug regimen that promotes longevity as compared to non-adherers.

Getting back to our statin study, how do we know that the decreased risk of death in those who religiously stuck with their statin prescriptions as compared to those who didn’t came about because they were adherers and not because of the statins?  We don’t.  In fact, based on the two studies I detailed above, it’s much likelier that the decreased mortality in those who took all their statins came about not because of the statins, but because those who stuck with them are adherers and have what ever quality it is that adherers have that makes them live longer.  And, if this is the case in this study as in the others, the statins don’t really do anything at all.

Despite its not really proving that statins confer greater longevity, the study does provide some interesting admissions and entertaining confabulations.

First, the study authors admit that there is no gold standard, randomized controlled study data showing that statins are of benefit in preventing death except for one group of people (and they even get that wrong).

The beneficial effects on cardiovascular mortality of treatment with statins to decrease levels of low-density lipoprotein cholesterol (LDL-C) have been established in several long-term, placebo-controlled trials.

The value of primary prevention with statin therapy in the reduction of overall mortality has recently been questioned.

A pooled analysis of 8 randomized trials in primary prevention populations showed that statins did not reduce overall mortality, indicating that lipid-lowering therapy with statins should not be prescribed for true primary prevention in women of any age or in men older than 69 years.

What they’re saying here is that statins have been shown to reduce mortality from heart disease in those who have elevated LDL, which is true.  But this decrease in deaths from heart disease is compensated for by an increase in deaths from cancer and other causes, so there really isn’t a gain.  You’re still dead.  Just maybe not from heart disease, but what difference does it make.  Are you going to spend $200 per month for the rest of your life and stay on medications that may make you feel lousy and lose your memory just so you can die of something other than heart disease?

In the last paragraph in the quote above, the authors confess that the data from actual randomized control trials show that statins confer no all-cause mortality benefits to women of any age and to men over 69.  They are playing a little fast and loose with the truth here because as I have posted before, the gold standard trials have shown no benefit for women and no benefit to men over 65 or to men under 65 who have never had heart disease.  The only improvement in all-cause mortality has been in men under 65 who have been diagnosed with heart disease, and even that benefit is so small that many people question if the extra cost and side effects of the statins are worth it.

So the authors of this study acknowledge that there has never been a randomized control trial that has shown any benefit to taking statins, but that doesn’t stop them.  They forge ahead trying to figure a reason that all these clinical trials haven’t shown an advantage.

Because clinical trials do not usually include individuals with multiple comorbid conditions or those receiving an extensive list of medications, there are considerable concerns regarding the applicability of findings from randomized clinical trials to the general population of patients seen in routine clinical practice.

Aha! They are saying that because the randomized controlled trial didn’t show what they wanted them to show – that statins worked for everyone all the time (thus the “considerable concerns”) -  that they need to figure out a better way to study them, one that involves patients with a lot of problems so that they don’t have to randomize them and confront failure yet again.

In light of the controversy surrounding lipid-lowering treatment for reduction of mortality among primary prevention populations, we undertook the present study to evaluate the effect of statin therapy in a large and diverse cohort of patients treated for dyslipidemia in a single health maintenance organization.

Interesting take.  There is no controversy.  The randomized controlled studies clearly show very little benefit to statin therapy in terms of decreasing all-cause mortality, the one statistic that really counts.  The controversy arises because the statinators simply don’t want to believe what these carefully performed trials tell them.  They by God want statins to work.  And they’re going to keep looking and fiddling with the data until they get a study that tells them what they want to hear whether the data is valid or not.

It’s pitiful that they are so desperate.

Don’t fall for the false promise of this or any other version of an observational study.  These kinds of studies do not prove causality.  Nor do they prove that a drug regimen works.  The patients in this study who religiously took their statins had better all-cause mortality than those who didn’t.  But, as we saw above, adherers always have better all-cause mortality than non-adherers.  In this case, was it that the adherers lived longer or was it that statins conferred some sort of benefit.  We can’t tell.  But we do know that in the real studies, the randomized control trials, statins didn’t do squat, so my vote would be that what we’re seeing here is an adherer effect and not a statin effect.

My advice is to continue to regard statins with a jaundiced eye.  So far, we haven’t seen any evidence that justifies the expense and the side effects of these drugs.

First I would like the heartily thank everyone who took the time to send me a comment on how to make this blog better both functionally and in content.  I read every single suggestion, and appreciated every one.  I’ll try to incorporate as many of the functional changes as I can within the design framework I have and within the limits of my pocketbook.  To demonstrate my profound gratitude for all the blog topic selections, I’m going to put up a post that absolutely no one asked for. But only because I’ve had it rattling around in my brain for the past week.

One view of the value of epidemiology

A day almost never passes without someone sending a comment my way about some recent study, plucked by the media from the hundreds published that same day, showing that low-carb diets cause brain fog or decreased longevity or cancer of some type or any number of conditions any of us would rather not have.  These comments always  end with the plaintive request, is there any truth to this?

My answer follows: This data comes from an observational study, and, as such, can’t possibly indicate causality.

Since I get these comments so often and answer them the same equally often, I figured it was about time to write a post on what an observational study really is so that I can link to it when I give my standard reply.

I can then add this post to the ones on the glycemic index and relative risk, both of which serve the same purpose.  I can simply link instead of explaining what these terms mean each time I have to use them.

Observational studies – also called prospective or cohort studies and sometimes even epidemiological studies – are the kind most often reported in the media simply because there are so many of them.  These are the studies in which researchers look for disease disparities between large populations of people with different diets, lifestyles, medications, incomes, etc.  If disease disparities are found to exist between groups, then researchers try to make the case that the difference in diet, lifestyle, medication, etc. is the driving force behind the disparity.

We’ve all seen these studies by the score.  We read that a large study population of people is separated into two groups based on blood levels of vitamin C.  One group of subjects has high blood levels, the other group has lower blood levels.  And since every one seems to believe that vitamin C protects against the common cold, the researchers decide to monitor these two groups for a year and find that the group with the highest blood levels of vitamin C has the fewest colds.   These findings are rushed into publication, and soon we read everywhere that vitamin C prevents the common cold.  It all seems so reasonable and so scientific, but the truth is that these studies don’t mean squat.  And the researchers who do them know it, or at least should know it.  The fact that they do know is evident in the weasel words they use in describing their findings.  You’ll read that these data ‘suggest’ or that they ‘imply’ or that this ‘may cause’ that.  The non-technically trained public, however, read these to say that vitamin C prevents the common cold.  And usually the media helps to sway opinion by slanting the story in the same direction.

But, you may ask, why aren’t these studies sound?  If the one group with the greater blood levels of vitamin C had significantly fewer colds, why is it such a stretch to say that vitamin C prevents colds?

I can explain by way of a game I used to play with myself as a child.  I’ve never been one to sleep much even when I was a kid.  I always stayed up late and I always woke up early.  My brain never seemed to slow down.  I was always ruminating on something.  My way of trying to get to sleep was to try to think of everything that could be thought of.  My mind would race, and I would think of my brothers sleeping in the room with me, their beds, my bed, the closet, the tree outside, my dad’s car, the rug on the floor, the moon, and on and on and on.   As I thought faster and faster, continuing to compile things that could be thought of, I would finally hit a quitting point.  Then I would try to figure if there was anything I hadn’t thought of.  Of course, immediately I would think of something. I hadn’t thought of the pigs on my grandfather’s farm.  Or I hadn’t thought of the fire hydrant out front.  Or my father’s shoes.  Or whatever.  Then I would start the game again, this time, of course, starting with the pigs on my grandfather’s farm and going from there.  I would always fall asleep before I had ever thought of everything there was to think of.

Researchers doing observational studies have much the same problem.  They try to think of all the differences between two large populations of subjects so that they can statistically negate them so that only the observation in question – the vitamin C level in the example above – is different between the groups.  Problem is they can never possibly think of all the differences between the groups.  As a consequence, they never have a perfect study with exactly the same number, sex, age, lifestyle, etc. on both sides with the only difference being the study parameter. And so they don’t really ever prove anything.  In fact, we would all probably be a lot better off if all the researchers doing observational studies had followed my lead and fallen asleep mid study.

But I’m being too harsh.  These studies do have some value.  Their value is in generating hypotheses.

The observational study demonstrates a correlation.  In our example above, the correlation is that higher vitamin C levels correlate (in this particular study) with lower rates of colds.  So, from this data, we could hypothesize that vitamin C prevents the common cold.  But at this stage that would be just an hypothesis – not a fact.

Once we have the hypothesis, we can then do a randomize, placebo-controlled trial.   We can recruit subjects, randomize them into two groups that are as equal as possible, especially as vitamin C levels are concerned.  Then we give one group of subjects vitamin C and the other a placebo and watch them for a year.  At the end of the year (or whatever the study period is), we break the codes, see who is on vitamin C and who is on placebo.  We already know how many got colds, so now we compare that to vitamin C intake.  We may find that those who took the vitamin C got significantly fewer colds, so we can say that our study demonstrates that vitamin C prevents the common cold.  If this same study is repeated a number of times with the same outcome, then it can be said to be proven that vitamin C prevents colds. (This study is, of course, hypothetical.)

But these studies are randomized trials, not observational studies.  Observational studies only show correlation, not causation, a fact that everyone doing research and reading about research should have tattooed on their foreheads.

CORRELATION IS NOT CAUSATION

More often than not observational studies are chock full of all kinds of technical-looking graphs, charts and tables.  Many even have complicated equations.  And long statistical analyses of the data derived.  They are like zombies, however.  They give the appearance of scientific life, but they are really scientifically dead.  Irrespective of how many scientific baubles are strewn through them, they are nothing but observational studies, worthwhile only as generators of hypotheses.  They demonstrate only correlation, not causation.

If you want to bear with me, I’ll show you a bizarre observational study that was actually performed that demonstrates everything you need to know about observational studies.

The study was published in 2003 in the prestigious American Journal of Epidemiology.  The title of the study is Shaving, Coronary Heart Disease, and Stroke. (Click here for free full text) This study purports to show that the frequency of shaving correlates with risk for developing heart disease, with those men shaving less having a greater risk.

Here’s the finding that initiated this study.

A case-control study comparing the frequency of shaving in 21 men under 43 years of age who had suffered a myocardial infarction and 21 controls found that nine of the cases but none of the controls shaved only every 2 or 3 days.

Someone noticed that about half of the men in a small group of subjects who had a heart attack shaved once every two or three days.  Another group of men of similar age who hadn’t had a heart attack were designated as controls.  Upon questioning it was discovered that all of the men in the control group shaved every day.  Thus the first hypothesis was born:  Infrequent shaving correlates with heart attack.

The researchers had access to a large population of subjects from another ongoing study called the Caerphilly Study.  Researchers recruited 2,513 men aged 45-59 from this study and gave them comprehensive medical workups including extensive laboratory testing.

Men were asked about their frequency of shaving by a medical interviewer during phase I. Responses were classified into categories ranging from twice daily to once daily, every other day, or less frequently. The 34 men with beards were not classified. These categories were dichotomized into once or twice per day and less frequently.

The men in the study were followed for the next 20 years with follow-up exams periodically to monitor for history of chest pain, heart attack and/or stroke.

Of the 521 men who shaved less frequently than daily, 45.1 percent died during the follow-up period, as compared with 31.3 percent of men who shaved at least daily.

When the data were further refined it was determined that

The age-adjusted hazard ratios demonstrate increased risks of all-cause, cardiovascular disease, and non-cardiovascular-disease mortality and all stroke events among men who shaved less frequently.

So there you have it.  Proof that shaving daily prevents heart disease. Or is it?

The researchers doing this study aren’t so stupid that they really think that the act of shaving itself has anything to do with a man’s risk for developing heart disease.  In fact, they went to great lengths to show that shaving was merely a marker for other things going on that may well have something to do with risk for developing heart disease or increased all-cause mortality.

The one fifth (n = 521, 21.4%) of men who shaved less frequently than daily were shorter, were less likely to be married, had a lower frequency of orgasm, and were more likely to smoke, to have angina, and to work in manual occupations than other men.

And these are just the differences the researchers found.  Had they looked harder, I’m sure they would have found more, just like I did when I played my ‘think of everything that can be thought about’  game with myself as a kid.

But if these researchers had really believed that the data showed that the lack of frequent shaving itself may have been the driving force behind the development of heart disease, they may have designed a randomized clinical trial to show causality.  They could have recruited men without heart disease, randomized them into two groups, instructed the men in one group to shave daily and the men in the other to shave every third day.  Then after 20 years the researchers could tell whether or not shaving protects against heart disease.

But the idea that shaving itself has anything to do with heart disease is so ludicrous that no one would ever do such a study.  We can all see that.  It’s a ridiculous idea.  It should be obvious that the shaving or lack thereof has nothing to do with heart disease or early death; the lack of shaving is merely a marker for all the other conditions that are risk factors for heart disease, i.e., small stature, unmarried, smoking, lower socioeconomic class, etc.  It’s all so easy to see.

But let’s just suppose that we take this same study and substitute the term ‘elevated cholesterol’ for ‘infrequent shaving.’  Now what do we see?  Let’s change one of the quotes from above to reflect this change.  What then?

Of the 521 men who had elevated cholesterol, 45.1 percent died during the follow-up period, as compared with 31.3 percent of men who had low or normal cholesterol.

We nod our heads sagely.  Suddenly we have a study that seems to make sense.  But – and this is important – it doesn’t make any more sense than the shaving study.  Both are observational studies.  We are programmed to think cholesterol is bad and causes heart disease, so this second study appears reasonable to us.  It triggers our confirmation bias.  We don’t believe for a second that shaving has anything to do with heart disease, so we can easily dismiss those findings.  But we are more than ready to believe that the elevated cholesterol caused those men who had it to have heart attacks. But the reality is that both studies are exactly the same – and neither proves anything.

If you’re interested in a longer, more in-depth article on observational studies, take a look at Gary Taubes long piece in the New York Times a few years ago.  I’ve tried to take a little different slant than he did so that my post and his article would cover all the bases.

Cartoon above from: Smith, G. D. et al. Int. J. Epidemiol. 2001 30:1-11

`I quite agree with you,’ said the Duchess; `and the moral of that is–Be what you would seem to be–or if you’d like it put more simply–Never imagine yourself not to be otherwise than what it might appear to others that what you were or might have been was not otherwise than what you had been would have appeared to them to be otherwise.’

`I think I should understand that better,’ Alice said very politely, `if I had it written down: but I can’t quite follow it as you say it.’

Lewis Carroll

If you tell a lie big enough and keep repeating it, people will eventually come to believe it.

Dr. Joseph Goebbels, Nazi minister of propaganda

I’m starting this post with two apropos quotes.  The first, from Alice in Wonderland, because the post will be a little difficult to understand; the second because I just read for the umpteenth time the Big Lie about low-carb diets and wanted to blog about it but couldn’t until I wrote this post first.

Intention-to-treat analysis (ITT) has become the de rigueur way of looking at experimental results that more often than not gives erroneous results.  These erroneous results are then reported as gospel, when in reality they are simply erroneous.  When unbiased, intelligent people (the readers of this blog, for example) consider ITT, they cannot understand how it can be used by scientist trying to make sense out of their data, but, unfortunately, it is in almost every experiment.  Here is how it works.

Let’s say were going to do an experiment comparing two different diets.  We round up 100 subjects and randomize them into two groups of 50.  We put one group, Group A, on one diet, Diet A, and we put the other, Group B, on a different diet, Diet B.  We keep both groups on their respective diets for 8 weeks to see what happens.

At the end of the 8 weeks we find that 30 members of Group A dropped out, but those who hung in there lost an average of 3 pounds per week for a total of 24 pounds each over the course of the study.  We look at Group B and find that no one dropped out of the study and that all the subjects lost an average of 1.2 pounds per week.

What does this data tell us?  It’s pretty simple.  It tells us that Diet A is much more effective, but is more difficult to follow.  It tells us that Diet B is less effective but easier to follow.  Right?  All intelligent people could agree on that.  So that’s how this study would be presented if it were published in a journal, right?  Uh, no.

No?

No.  If published, the conclusion would be that both diets are exactly the same.

Say what?!?!?

Yep.  That’s what the authors would conclude.  Why?  Because they would use an intention-to-treat analysis.  In fact, the peer-review process would probably demand it.

An intention-to-treat analysis demands that all subjects remain in the data pool, even if some have dropped out.  The intention was to treat all the subjects, so the analysis should contain all the subjects, even if some left the study after the first day.  In an ITT, researchers pretend that subjects who chose to abandon the study really didn’t and include them in their final data.  Sounds like something from Through the Looking Glass, doesn’t it?

Let’s look at how this would work in our dietary study above.  The 20 subjects in Group A who followed Diet A lost 24 pounds each.  Multiply this 24 pounds times the 20 subjects who stayed in the study and you find that the group lost 480 pounds over the course of the 8 weeks.  Now divide this 480 pounds by the 50 subjects who started the study, and you get a weight loss of 9.6 pounds for the 8 weeks.  Dividing by 8 gives us an average weight loss of 1.2 pounds per week for all 50 subjects in Group A.  Which is exactly the same as the weight loss in the subjects in Group B.  So, according to the dictates of ITT, the study would show that both diets were equally effective. But, as we’ve seen, they’re not.

If a doctor were recommending a diet to his/her patients based on the actual findings of the study, he/she could reasonably say:  Diet A is very effective but tough to follow, so if you think you can do it, Diet A is definitely the fastest way to lose weight.  If you want something that will help you lose a little weight and is easy to stick to, then try Diet B.

If the same doctor recommends a diet to his/her patients based on the ITT results, he/she would say: Follow whichever diet you want – they’re both the same.

Why, you may ask, could seemingly intelligent people do something so stupid as use ITT to evaluate data? There is a reason, although it has its own problems.

We all know from experience and from talking to a lot of people who have lost weight that a lot of different diets work.  People lose weight on the Ornish diet and they lose weight on the infinitely preferable Protein Power diet.  And many other diets as well.  So, we can reasonably assume that almost any diet will help some people lose weight.  But we want to compare two diets to see which one is really the best.  So, let’s do another experiment.

Let’s take another 100 people and randomize them into two groups of 50, Group C and Group D.  Those subjects in Group C go on Diet C and all of them do well.  They lose an average of 2 pounds per week and all of them stay on the diet. The subjects in Group D go on Diet D, and most don’t do very well. As we all know from experience, it’s tough to stay motivated to stay on a diet if you’re not losing weight.  So, 30 of the subjects in Group D drop out because they’re not losing.  We know that any diet will work for some people, and Diet D is no different.  The 20 who stay in the study are those who are losing on Diet D.  And those 20 Group D subjects lose an average of 2 pounds per week.

In analyzing our data, if we remove from the pool of subjects all those who dropped out of the study, we are left with all 50 people in Group C, who lost an average of 2 pounds per week and only 20 people in Group D, who lost an average of 2 pounds per week.  We would then find that both diets are exactly the same.  Subjects in both groups lost 2 pounds per week.  Therefore both diets are equally effective.

But is that true?  Clearly not.  And that is the problem that ITT was designed to deal with.  But, as we’ve seen above, it brings its own errors.

So, how do we deal with the issue honestly and effectively?  Easy. By explaining the data in two ways.  Most people – researchers included – want to boil an issue down to a single answer, when two answers are required. ITT allows one answer – often incorrect – to two different questions.  ITT is like the old TV show in which the clown Bozo always asked the little kids he interviewed something like this:

So, Bobby, tell me: Do you walk to school or carry your lunch?

Were Bozo adamant on an ITT-type analysis of the question, he could get only one answer.

Going back to our Group A/Group B diet study we can look at the data in two ways:

1. Diet A is extremely effective for those who stick with it. (Called the adherence effect.)

2. Only 40 percent of those attempting Diet A achieve the desired effect. (Called the assignment effect.)

Both of these statements are true.  Both contain valuable information.  But they answer two different questions.  The first answers the question: what happens to people who stick to the diet?  The second answers the question: What happens to people who are placed on the diet?

As Dr. Gerard Dallal writes about ITT

The fraud occurs when the answer to the question of assignment is given as though it were the answer to the question of adherence!

Instead of the conclusion that both Diet A and Diet B show the same results (when, clearly, they don’t), which would be the way it would be presented in a scientific paper demanding ITT, why not present it this way?:

The adherence effect: Subjects following Diet A for 8 weeks lost an average of 3 pounds per week whereas those following Diet B lost 1.2 pounds per week.

The assignment effect: 40 percent of those attempting Diet A remained in the study whereas 100 percent of those following Diet B remained in the study.

Conclusion: Diet A is significantly more effective (3 pounds per week vs 1.2 pounds per week) for those able to remain on the diet.  Diet B is less effective but significantly less difficult to follow than Diet A. (100 percent of subjects on Diet B remained on the diet throughout the study whereas 60 percent of those on Diet A dropped out).

It just ain’t that hard to present it that way.  It provides much more information than the ITT, which attempts to answer two questions with one answer.

Now, let’s look at the big low-carb lie that launched me into this post.  I was reading a book that I intended to review for this blog and came across the following statement:

There is evidence from a variety of sources that [low-carb diets] work for short-term weight loss.  One year after starting a diet, however, there appears to be no significant difference in success rate than that seen on any other common diet plan.

Have you heard that one before?  It’s a specific variant of the old: Studies show that while effective in the short term low-carb diets show no difference in weight loss after one year than do low-fat diets. It’s the Big Lie.

It’s the last refuge argument of low-fat advocates who are getting hammered with all the data showing low-carb diets to be more effective.  Yeah, well, they say, Protein Power may work in the short term, but over a year studies show it’s no better than low-fat.  It’s like a cross thrust in a vampire’s face.

But is it true?  It is if you believe in intention-to-treat analysis.  But what if you believe in a more accurate way of presenting the data?

Let’s briefly look at a few studies published that confirm the idea that there is no difference between low-carb diets and low-fat diets after one year.

The first was published in the Annals of Internal Medicine in 2004.  The conclusion of the authors was that after one year subjects

had more favorable triglyceride and high-density lipoprotein cholesterol levels on the low-carbohydrate diet than on the conventional diet. However, weight loss and the other metabolic parameters were similar in the 2 diet groups.

In the body of the paper, however, one can read the following:

The final 1-year weight change (mean ± SD) was -5.1 ± 8.7 kg in the low-carbohydrate group and -3.1 ± 8.4 kg in the conventional diet group (Figure). The difference in weight loss between the 2 diet groups was not significant (-2.0 kg [CI, -4.9 kg to 1.0 kg]; P = 0.195 before and P > 0.2 after adjustment for baseline variables). The difference in weight loss between the 2 diet groups between 6 months and 1 year was not statistically significant (P = 0.063).

But that’s all ITT blather.  Let’s read the next couple of sentences:

Persons on the low-carbohydrate diet who dropped out lost less weight than those who completed the study (change, -0.2 ± 7.6 kg vs. -7.3 ± 8.3 kg, respectively; mean difference, -7.1 kg [CI, -11.6 kg to -2.8 kg]; P = 0.003). In contrast, weight loss was not significantly different for those on the conventional diet, whether they dropped out or completed the study (change, -2.2 ± 9.5 kg vs. -3.7 ± 7.7, respectively; mean difference, -1.5 kg [CI, -5.7 kg to 2.7 kg]; P > 0.2).

Let’s translate.  Those who dropped out of the low-carb diet but were counted as if they hadn’t lost 0.2 kg (about 0.4 pounds) whereas those who completed the study lost 7.3 kg (about 16 pounds).  Do you think the dropouts skewed the numbers?  I guess so.  And look at the next astounding sentence.  “In contrast, weight loss was not significantly different for those on the conventional diet, whether they dropped out or completed the study…”  So, there was no difference in the results of those following the low-fat diet whether they dropped out or stayed in.  Had the subjects who dropped from the low-fat arm not been included, the results for that diet would have been the same.  Including the subjects who dropped from the low-carb arm, however, dramatically lowered the overall weight loss of the subjects as a group, making them equal to those in the low-fat arm.

It could be accurately stated that those who remained on the low-carb diet for one year lost significantly more weight than those who remained on the low-fat diet. which, of course, refutes the Big Lie that low-carb and low-fat diets provide equal weight loss at one year.

The two other studies used to perpetrate the Big Lie that low-carb diets show no difference in weight loss after one year are the ones by Foster et al and Samaha et al in the May 2003 New England Journal of Medicine.

When analyzed by ITT, both of these studies show no significant difference between low-carb and low-fat diets after a year.  But when looked at from the perspective of those subjects remaining in the study, we see a big difference between the low-carb and the low-fat arms.

In the Foster et al study using a modified version of the Atkins diet, we find a statistically insignificant 1.9 kg difference in weight loss between the two groups by ITT.  But when we eliminate the drop outs and look instead at the data from those subjects who remained on the diets for the entire one year, we find a statistically significant 2.8 kg (over 6 pounds) greater weight loss in those following the low-carb diet.

In the Samaha et al study using the diet from the Protein Power LifePlan, those following the low-carb diet lost a statistically insignificant 2 kg more weight than those following the low-fat diet by ITT.  Eliminating the dropouts, however, gives us a statistically significant 3.6 kg (almost 8 pounds) greater weight loss on the low-carb verses the high-carb diet after one year.**

Intention-to-treat analysis gives us the Big Lie:  Low-carb diets are no more effective than low-fat diets after one year. Dr. Goebbels would have been proud.

The truth, however, is a little different and can be stated thus:

Those who follow low-carb diets for a year lose significantly greater weight than those who follow low-fat diets for a year.

After reading this post you should know more about intention-to-treat analysis than 99.9 percent of the physicians and dietitians practicing in the world today.  Don’t let this knowledge go to waste.  Next time you hear the Big Lie, point out the truth.

** Thanks to Richard Feinman, Ph.D. for the tabulation of these data and for our many conversations on this subject.

Virtually all of the results presented in medical studies are displayed as ‘average’ or ‘mean’ values. I’m sure everyone knows how to come up with an average or mean (the two are synonymous) value for a group of data points is to add them and divide the sum by the number of data points analyzed. For example, if you are a teacher, and you want to find out the average score on a test you gave to 30 students, you would add all the test scores together and divide by 30. You would then have the ‘mean’ or ‘average’ score of the students in your class.

Most medical papers list the mean values of whatever is being studied. If the researchers are trying to determine whether or not an experimental weight-loss therapy works, they add the weight lost by all the subjects participating in the study then divide by the number of subjects. The number they get is the ‘mean’ or ‘average’ weight loss brought about by the therapy being tested. It all sounds pretty reasonable and scientific, but is it really?

It would be realistic if we were all average people. But we’re not. And averages don’t represent us all that well. In fact, if you think about it, the average American would have one breast and one testicle.

Averages don’t always represent the true findings in a scientific experiment, either. Let’s look at an example to demonstrate. Let’s say we are testing a new weight loss regimen on 10 people. We start these people on the program, keep them on it for three months, then evaluate. When we look at the numbers we find the following results:

• Subject #1 -4 lbs
• Subject #2 -5 lbs
• Subject #3 – 7 lbs
• Subject #4 – 4 lbs
• Subject #5 – 2 lbs
• Subject #6 -6 lbs
• Subject #7 +12 lbs
• Subject #8 – 1 lb
• Subject #9 + 4 lbs
• Subject #10 -3 lbs

If you add these numbers up and divide by 10 you find that the average or mean weight loss for the group is 1.6 pounds, which doesn’t seem like a lot. But if you look at the data itself instead of the average, you see that most of the people lost around 4-5 pounds. In fact, assuming these numbers to be accurate, if you went on this same regimen, the odds are that you would lose 4-5 pounds instead of the 1.6 pounds that the ‘mean’ of the data would predict. You could also gain 12 pounds, but that would be unlikely (a 1 in 10 chance). This is the problem in simply looking only at average values and not the data as a whole.

Another way to look at this data is to calculate the median, which is basically the midpoint within the data set, i.e., the point at which half of the subjects are above and the other half below. You can do this by arranging the data in ascending or descending order and finding the midpoint by lopping off from both the top and the bottom until you get to the middle.

If we do this to our data, it looks like this:

• Subject #7 +12 lbs
• Subject #9 + 4 lbs
• Subject #8 – 1 lb
• Subject #5 – 2 lbs
• Subject #10 -3 lbs
• Subject #1 -4 lbs
• Subject #4 -4 lbs
• Subject #2 -5 lbs
• Subject #6 -6 lbs
• Subject #3 -7 lbs

We can see that the median falls between 3 and 4 or 3.5. So in this experiment the mean (average) weight loss is 1.6 pounds, the median is 3.5 pounds, about twice what the mean is. But if you look at the actual weights lost you can see that most clustered around the 4-5 pound level. You can see that the results of our experimental weight-loss regimen look different depending upon how they’re reported.

It works this way in the real medial literature as well.

Let’s look at a study from a couple of years ago (full text pdf) that many people have used to ‘prove’ there is no metabolic advantage and to ‘prove’ that a calorie is simply a calorie irrespective of its macronutrient composition.

Boden et al studied 10 overweight patients with type II diabetes in metabolic ward for 21 days. During the first 7 days the subjects were allowed to follow their regular diet (the control) and were switched to a low-carbohydrate diet (21 g carb/day) for the next 14 days. During the course of the study, numerous parameters were evaluated, including fasting glucose levels, insulin sensitivity, HbA1c, weight change, caloric intake and energy expenditure.

After the 14 days on the low-carb diet subjects lost weight and markedly improved in all parameters measured. Blood glucose levels normalized, HgbA1c decreased from 7.3% to 6.8%, insulin sensitivity improved by about 75%, triglycerides dropped by 35% and total cholesterol fell by 15%. Pretty dramatic results for only 14 days on the low-carb regimen, I would say.

The subjects lost weight as well. They lost an average (mean) of 3.63 lbs (1.65 kg) while decreasing their food intake from 3111 kcal/day to 2164 kcal/day, a decrease of 947 kcal/day. Multiplying 947 X 14 gives us a total caloric deficit of 13,258 kcal. If we divide this number by 3500, the kcal in a pound of fat, we get 3.79, which is the amount of fat that should be lost simply from the caloric deficit. And which is pretty close to the actual 3.63 lbs actually lost. The difference is insignificant, so it really is, as they say, close enough for government work.

The authors of the study conclude that irrespective of all the other markedly positive benefits of the low-carb diet, the “weight loss…was completely accounted for by reduced caloric intake.” In other words, there is no metabolic advantage to a low-carb diet. Weight lost simply occurs because the satiating effects of the low-carb diet bring about a spontaneous reduction in calories.

But is that all the story here? Not really. Let’s see why.

Below is a chart from the study showing graphically what happened to caloric intake and weight during the course of the experiment.

Look at the lines in the upper half of this chart that represent the body weight ranges of all the subjects and the lines representing the caloric intake of all the subjects. Notice that the lines for caloric intake are small while the lines representing the amount of weight loss are large. If you compare the dimensions of these lines to the scale, you find that these subjects varied their caloric intake by about only 200-250 kcal/day. But the variation in weight loss is much, much larger, which means that some subjects lost considerably more weight than would be expected from the caloric deficit while others didn’t lose as much or may have even gained a little. What this chart shows us is that there is indeed a metabolic advantage for some of these people even though on average there wasn’t for the group. And remember the post on Carl Popper: if the metabolic advantage can be shown to be present, that means the hypothesis that a metabolic advantage doesn’t exist is false.

Unfortunately, most medical articles don’t show the array of data as this one has, so all we have to go by is the average, which doesn’t always tell the whole story. One of the things I like about my favorite journal, Nutrition & Metabolism, is that the editors almost always require the raw data to be shown along with the averages, which truly allows astute readers of the medical literature to come to meaningful conclusions about the data. Only after they report it in this way, can you really decide what happened.

I’m throwing up this post on the concept of relative risk as a time saver. I’m going to be doing some posting soon in which relative risk plays a role as it has in many of the postings in the past. Instead of taking time in each post to explain relative risk, I will simply be able to link to this one and get on with it.

Let’s say we are studying the rate at which people taking a particular medication get cancer. If we randomize our study population into two groups and give one group the drug and the other group a placebo, we can wait around for 5 years (or 10 years or 20 or whatever our protocol is) to see how many develop cancer in each group. Then we can compare.

If after 5 years we find that 10 of the subjects taking the drug develop cancer and only 5 of the subjects taking the placebo get cancer, we can say (assuming there were equal numbers of people in both study groups to start) that twice as many people taking the drug got cancer. We can then say that the relative risk (usually written as RR) of getting cancer from taking the drug is 2.

The relative risk is the number of subjects who develop the problem divided by the number who didn’t. In our example above it would be 10/5 = 2.

Another way to present this data is to say that the relative risk of not getting cancer by not taking the drug is 0.5. In other words, people who didn’t take the drug had half the risk of getting cancer compared to those who did.

In our simple example we assumed that there were exactly the same number of subjects in both groups. In reality this usually isn’t the case. There will be a difference in group sizes so instead of using numbers of subjects, researchers will use percentages. So in our case above, let’s say that 4 percent of the subjects taking the drug got cancer and 2 percent of those taking the placebo got cancer. We have the same relative risk of 2. (4 divided by 2 = 2)

It all seems pretty straight forward. But it often isn’t as it seems. There can be two studies that show the same relative risk but there is a world of different meaning for the relative risk. This idea is what we want to focus on because this is the one that researchers use all the time to scare us needlessly.

I’m going to create an example loosely based on the facts to show you what I mean. Let’s compare the safety records of two airlines over the past 25 years. In 1985 Delta had a crash in Dallas (in which one of my best friends was killed) and in 2001 American had three crashes. The two from 9/11 and one going from New York to San Juan, Puerto Rico. Let say that there were 200 people killed in the Dallas Delta crash and 600 people killed in the three American crashes.

Let’s assume that both airlines fly about the same number of miles each year. American has about 4000 flights per day, so if we assume an average distance (probably too short) of 500 miles per flight, we get 730 million miles flown per year. If we multiply by 25 to get the miles flown in 25 years we come up with 18.25 billion miles flown. Let’s assume the same for Delta. If we divide the number of victims of crashes from the two airlines by the number of miles flown, we can come up with a risk of dying per mile flown on each of the airlines. If we do this for Delta we discover that the risk of dying comes out to be 200 divided by 18.25 billion or about 0.00000001096. This tiny number represents the number of people who have died per mile flown by Delta. If we do the same calculation for American we find that the number is 0.00000003288 deaths per mile flown.

Now if we want to calculate the relative risk of flying on American verses flying on Delta we divide the American risk by the Delta risk (0.00000003288 divided by 0.00000001096) and we get 3. So, the relative risk of flying on American is 3 compared to flying on Delta based on the last 25 years of crash history. This means that it is 3 times more risky to fly American than it is to fly Delta. Based on these figures you are 3 times more likely to die if you fly American than if you fly Delta. It’s true, and these figures prove it.

We all know that it’s hogwash. There is no difference in risk between flying these airlines because the numbers are so low as to be ridiculous, which is the precise reason I used this example. The total numbers make a difference.

What if there were an airline that had 90 deaths for every 100,000 miles flown and another that had 30 deaths for each 100,000 miles flown. The relative risk for flying the first as compared to the second would be 3 just as it is with American verses Delta, but the real risk would be much, much different. In fact, you wouldn’t want to fly on either of these airlines because they are both too risky. But if you had to fly on one, you would much prefer the one with only 30 deaths per 100,000 miles.

Many drug companies use the relative risk in their propaganda. They might do a study with two groups of 40,000 people with one group taking their drug and the other taking a placebo and monitor them for 5 years. If the drug is supposed to prevent heart disease, then the researchers would tabulate the number of cases of heart disease that arose over the 5 years, then see how many cases developed in the subjects taking the drugs compared to the ones taking the placebo.

If there were 43 people who developed heart disease while taking the placebo and 31 who developed heart disease while taking the drug, then the relative risk of not taking the drug would be 43 divided by 31 which is 1.39. The drug company then proclaims that people not taking their drug had a 30 percent greater chance of developing heart disease, which sounds pretty significant. But it’s a lot like the Delta/American example: the numbers aren’t all that large.

What it really means is that out of the 40,000 people who took the placebo for 5 years only 12 more developed heart disease than in the group of 40,000 people who took the drug. Thats 2.4 people out of 40,000 per year. (12 people divided by 5 years = 2.4 people per year) If the drug costs a lot of money and has unpleasant side effects associated with it (think statins), would you be willing to pay the bucks and put up with the side effect hassle on the chance that you might be one of the 2.4 out of 40,000 who might be saved from developing heart disease? Or would you just take your chances? I would definitely take my chances.

2.4 people out of 40,000 per year sounds a lot less risky than a 39 percent increased risk that the relative risk implies.

One other way the relative risk can be used to flim flam people is when it is too small to be meaningful yet is touted as having great merit.

Without going into all the statistical detail (which if you’re interested in you can read about here and here from previous posts) suffice it to say that relative risk ratios below 2 are pretty much meaningless. So when you see a relative risk of 1.28 and hear someone say that represents a 28 percent increased risk, you can blow it off. Once the relative risk gets to the 2 or greater level, then it starts to have some meaning. But, once again, only if the numbers aren’t so small as to be meaningless. Remember, the relative risk of flying American is three times that of flying Delta if you simply look at the relative risk. In reality, however, the numbers are so small that the differences are meaningless.

A paper appeared in last weeks JAMA that I just now got around to reading, but had already read about in a number of other publications. The study looked at the effect early-in-life obesity has on death from heart disease decades later. The paper is a real treasure trove of information worthy of a longer, more comprehensive blog later on. For now I want to use it as an example of how statistics can be used to humbug the non-statistically inclined.

In brief the JAMA study was done with data pulled from the monster-sized Chicago Heart Association Detection Project in Industry that was begun in 1967. Subjects who were at least 31 years old were evaluated for a number of parameters including BMI, blood pressure, elevated cholesterol, and history of smoking. The researchers re-evaluated these subjects over the next several decades.

Researchers divided the subjects into five groups: low risk, moderate risk, intermediate risk, elevated risk, and highest risk. It’s not important to the point of this post to get into what specifically constituted these varying levels of risk, but in general, the greater the number or the more severe the risk factors, i.e., elevated cholesterol, high blood pressure, etc., the more high-risk the category. The researchers then divided the subjects into three other groups based solely on BMI: normal weight, overweight, and obese. Within each of these three weight-related groups were spread subjects with varying degrees of risk. In other words, the normal weight group contained subjects who were low risk, moderate risk, intermediate risk, elevated risk and highest risk as a function of their cholesterol levels, blood pressures, smoking history, etc. It was the same for all the groups. The obese group was composed of obese subjects who ranged from low-risk to highest risk. The object of the study was to follow these subjects for many years to see if obesity was truly a risk factor for death from heart disease or if obesity led to elevated cholesterol, high blood pressure, and all the rest, which in turn caused the heart disease mortality.

If the subjects in the normal weight, low-risk group had no more heart disease than those in the obese, low-risk group, then it could be inferred that obesity by itself may not cause heart disease. If, on the other hand, the subjects in the obese, low-risk group had a much higher death rate from heart disease than did those subjects in the normal weight, low-risk group, then at least some of the heart disease could be attributed to the excess fat.

(There is really much, much more under the surface of this paper worthy of exploration, but it will have to wait.)

So what did the study show? It depends upon where you get your information.

According to a statistical analysis of the data, the odds for death by heart disease in the obese, low-risk subjects was 1.43 times that of the normal weight, low-risk subjects implying an almost half again greater risk simply for being obese. And that’s how it was reported in the lay press.

WebMD allowed as to how

The researchers found that the risk of dying from heart disease was 43% higher for study participants who were obese but also met these qualifications for low cardiovascular risk than for normal-weight, low-risk participants.

It appears to be a pretty clear indictment against obesity.

But not if the statistics are analyzed correctly.

Before I get into that I want to produce a quote from Judge Samuel Alito that he uttered during his confirmation hearing before the Senate Judiciary Committee last week. Said he

Well, the analogy went to the issue of statistics and the use and misuse of statistics and the fact that statistics can be quite misleading. … And that’s what that was referring to. There’s a whole – I mean, statistics is a branch of mathematics, and there are ways to analyze statistics so that you draw sound conclusions from them and avoid erroneous conclusions from them.

Truer words were never spoken. But you’ve got to analyze the statistics, not take them at face value.

So let’s analyze the statistics used in our study under discussion to see if and how anyone went wrong.

Here is how the 1.43 ratio was written in the paper:

the odds ratio (95% confidence interval) for CHD [coronary heart disease] death for obese participants compared with those of normal weight in the same risk category was 1.43 (0.33-6.25).

What does this really mean?

First, it means that 143 people who were in the obese, low-risk group died from CHD for each 100 people who were in the normal weight, low-risk group, giving the risk ratio of 143/100 or 1.43. It seems reasonable that if that were really the finding, then the risk of dying if you are obese with low-risk (as these researchers define low risk) is 1.43 times greater than if you aren’t obese. Right? Not necessarily, and here’s why.

If you flip a coin 10,000 times the odds are that you will get about 5000 heads and 5000 tails since the odds are 50-50 of the coin landing on either side. But what about if you only flip it 40 times? Are you going to get exactly 20 heads and 20 tails? Probably not. What about if you only flip it six times? Will you get three and three? Maybe, but probably not. In fact I just flipped a coin 10 times and got five heads in a row, two tails, one head, and one tail, giving me seven heads and three tails. Now if this were a study on coin flipping I could confidently predict based on my data that if I flipped this same coin 10,000 times I would get 7000 heads and 3000 tails. But we all know this isn’t really true because the sample size I used (ten) was too small to be used to accurately predict the outcome for 10,000 flips. The fact that I went 7 heads and 3 tails came about strictly by chance, which plays a smaller and smaller role as the sample size gets larger.

Statisticians realize that virtually any outcome can be influenced by chance and have developed equations to quantify just how much chance is involved. One of the terms they have come up with (after some pretty complex mathematical maneuvers) is the confidence interval. Pretty much the gold standard for confidence intervals is what’s called the 95 percent confidence interval. What this means is that once a confidence interval has been established (a range between two numbers) you can be confident that your result will fall into that range 95 percent of the time. Since chance can’t be totally eliminated, there will still be a 5 percent chance that our result will fall outside the range.

Let me digress a little here to define what we mean by our result falling into or outside of this range. In our study the data showed that 43 percent more people died in the obese, low-risk group than in the normal weight, low-risk group. But so what? As callous as it seems, we don’t care about those people; they’re already dead. What we care about is how the data provided by all these dead people affects you and me and our loved ones and all the other people who aren’t dead yet. We all want to know if this 43 percent is just a chance finding like my flipping 7 out of 10 heads, in which case it’s meaningless, or do I really have a 43 percent greater chance of dying of heart disease if I’m obese even though I don’t have any other risk factors? Those are the questions the confidence interval addresses.

In this study the 95 percent confidence interval is (0.33-6.25). It’s usually stated like it is in this case 1.43 (0.33-6.25). This means that the risk ratio as applied to the population at large should come in 95 percent of the time between 0.33 and 6.25. So this means that the risk of dying if you are obese with no other risk factors could be anywhere from 1/3 as much to 6.25 times as much.

Say what? 1/3 as much?

Yep. Even though the middle of this range is around 1.43 the actual risk is just as likely to be 0.5, which would mean you have half the chance of dying as someone who is normal weight without risk factors. In other words, you would be better off being fat.

These numbers in the parenthesis are critical. If you’ve got a positive number in front of the parenthesis as we do with the 1.43, then you want to make sure that both numbers within the parenthesis are above 1. If the first number is below one that means that the risk is actually greater the other way, which negates your analysis.

In this case since the first number is less than 1, indicating that the risk ratio is meaningless and can be ignored. What it means in this specific case is that it makes no difference whether or not you’re overweight early in life as long as your other risk factors (as identified by the researchers in this study) are normal in terms of your risk of dying of heart disease later in life. Not the 43 percent that the authors and the press that picked the story up proclaimed. Too bad the authors and all the medical press people weren’t a little more statistically honest.

But to tell you the truth, I suspect that the authors of this paper (and I know that the medical writers) don’t have the same understanding of the confidence limits and what they really mean that you do after reading this post. Most researchers run their data through a computerized statistical program and simply look at the risk ratio (the 1.43 in this case) without really having a clue what the numbers inside the parenthesis mean.

I swear that over the next few weeks I’ll post in as simple a way as I can a basic (very basic) primer on statistics. If I don’t do this in a timely fashion you may write and cancel your subscription to this blog, and the unused portion of your subscription fee will be cheerfully returned.