An article appeared in this month’s Metabolism that shows in classic fashion how the medical publication game is played, how data can be manipulated to “prove” a particular point of view, and why abstracts should pretty much be ignored in favor of carefully reading studies from beginning to end.
Very seldom do research scientists set out to do a study just to see what happens. In almost every case they begin with an agenda, or, at the very least, a hypothesis. Clearly, in our study under review, entitled “High-fat meal impairs vascular compliance in a subgroup of young healthy subjects,” the authors set out to prove that dietary fat is bad for the arteries.
Now suppose you are a researcher and you want to come up with a paper showing that dietary fat has a negative effect on the arteries (or anything else, for that matter), what would you do? Well, first you would recruit your group of subjects and feed them the worst kind of fatty foods you could find then see what, if any, changes occur in their arterial function. You would then wait a week or so and feed these same subjects a diet containing little or no fat but made up of otherwise healthful foods and perform the same arterial analysis. If you find a reduction in arterial function when the subjects consumed the high-fat diet, then you could publish a paper showing that dietary fat is really bad stuff that can predispose people to heart attacks. Then Dean Ornish or Arthur Agatston (the South Beach Diet guy) can go on Good Morning America, wave the study around, and proclaim that the low-carb/ high-protein diet is a disaster. Sound familiar?
But what if you recruited all the subjects and fed them the crummiest diet you could imagine and found no difference in terms of arterial function after the high-fat and the low-fat diets? Or worse yet, that the subjects actually did better following the high-fat diet? Do you try to publish a paper reporting that you found no difference and that your hypothesis that a high-fat diet is bad remains unproven? Not a chance. You’ve got to pull this baby out of the fire and get something for all the time and money you’ve spent. What do you do?
As the old saying goes, if you torture the data enough it will confess to anything. So, you take your data and start torturing.
One of the easiest ways to torture data is to plot it out graphically and look for the bell-shaped curve. If the data has been collected properly, what you’ll usually find when you graph it is that most of the results cluster around the middle with a few outliers on each side. You can then pull off all the data from one side of the bell-shaped curve—the negative side, of course—and call the subjects represented by that data a subgroup. Then you term the subjects in this subgroup “responders” or “non-responders” or whatever you want to call them and proceed to make your case that fat is bad because it causes negative effects in the arteries of “responders.” The other side of that same coin, however, is that dietary fat brings about positive effects in the arteries of the “non-responders,” but that doesn’t work with your hypothesis, so you don’t bother to mention it.
With that preface, let’s take a look at our paper.
First, let’s look again at the title of the paper. “High-fat meal impairs vascular compliance in a subgroup of young healthy subjects.” Notice the “subgroup” stuck in there? I’ll bet you didn’t notice it when you first read the title in the second paragraph of this post. Don’t feel bad, I didn’t notice it until I had read the entire paper and then went back and reread the title. The first blush takeaway from the title is that fat impairs vascular compliance, which is precisely what the authors want us to believe.
What did the study set out to do?
This study was conducted to investigate the effects of an oral fat load on vascular compliance in young healthy subjects who were free of confounders that might potentially affect compliance such as drug therapies, smoking, or chronic disease.
The researchers recruited their subjects, who in this case were 7 men and 11 women (ages 18-24) none of whom smoked or had histories of high blood pressure or diabetes. All had normal lipid profiles and none were taking medications or vitamin supplements.
The main parameter of this study was arterial compliance, a measure of how flexible and limber the walls of the arteries are, which was measured non-invasively with a HDI Pulse Wave Research Cardiovascular Profiling Instrument. The more compliant the arterial walls, it is thought, the healthier the artery. A common research technique is to expose subjects to various drugs and/or nutrients (or even stress) and measure the change in arterial compliance that occurs after the exposure.
What kind of high-fat diet did the subjects consume?
Subjects ingested a high-fat meal (Egg McMuffin, Sausage McMuffin, 2 hash brown patties, and a noncaffeinated beverage) containing 3766 J [900 kcal], 110 g of carbohydrates, 50 g of fat, 14 g of saturated fat, and 255 mg of cholesterol.
Huh? This isn’t just a high-fat diet, this is a high everything diet including, according to the nutritional facts on the McDonald’s website, about 5 grams of trans fats.
What did our researchers feed these subjects for a comparison? The subjects ingested
an isoenergetic low-fat/high-carbohydrate meal (Frosted Flakes [Kellogg Company, Battle Creek, Mich], skimmed milk, and orange juice) containing 3682 J [880 kcal], 204 g carbohydrate, 0 g fat, 16 g protein, and 5 mg cholesterol.
Talk about stacking the deck.
Okay, before we go on, you tell me which meal produced decreased arterial compliance? Was it the 900 kcal meal composed of a bunch of trans fat along with 100+ grams of carbohydrate (equivalent to about a half a cup’s worth of sugar) and a “noncaffeinated beverage” that remains unidentified? Or the group who ate 880 kcals of vitamin and mineral fortified cereal, skimmed milk, and orange juice? Go ahead, guess.
Well, as it turns out, there was no difference in the changes in arterial compliance. Some subjects showed no change, some had reduced arterial compliance, and others actually had improved arterial compliance after the high everything meal. Our researchers after stacking the deck were left with without any meaningful data. What did they do? They started torturing the data they did have.
A retrospective analysis of vascular compliance responses in the entire group of subjects revealed 2 subgroups, one in which C2 [C2 is a measure of compliance in small arteries; C1 is a measure in large arteries] fell after the high-fat meal and another in which C2 either remained unchanged or rose. Those in whom C2 declined by 20% or more at 2 or more time points within 4 hours of the ingestion of the high-fat meal were defined as “fat reactors.” In these subjects (n = 8), mean C2 fell from 9.2 ± 0.9 to 7.4 ± 0.9 mL/mm Hg × 100 after the high-fat meal and remained below baseline for 6 hours (P
Let’s look at this paragraph closely to see what it really tells us. According to the study protocol the arterial compliance was measured every hour for six hours after the consumption of the test meal. The researchers looked at their data and selected out those subjects who showed at least a 20% decline in compliance, not over the course of the whole 6 hours, but in at least “2 or more time points within 4 hours of the ingestion of the high-fat meal.” This tells me that even this group wasn’t that bad because the researchers had to nitpick to even select a subgroup to prove their point. Nevertheless, the researchers defined this group who had overall a 19.6% reduction in arterial compliance (but with at least a 20% reduction over at least two time points in the first 4 hours) as “fat reactors.” The other subjects (a majority), defined as “non-fat reactors,” showed a 30% increase in arterial compliance after the high-fat diet. So, actually, a majority of the subjects had improvement in arterial compliance with the high-fat diet as compared to the high-carbohydrate diet. But our researchers remained undeterred by these facts as they went on throughout the rest of the paper describing all the negative findings in the minority of subjects who responded negatively to what could only be described as a horrendous meal by anyone’s standards (except executives of McDonald’s, I suppose).
This article could just as easily been entitled “High-carbohydrate meal impairs vascular compliance in a subgroup of young healthy subjects.” In fact, this title would have been more accurate and appropriate since the majority of the subjects did worse on the high-carb diet.
What amazed me about this study other than the data manipulation shenanigans of the researchers is how reducing carbohydrates is protective against dietary abuse. In this study the subjects consumed about half the carbohydrates in the dreadful high-fat meal than they did in the vitamin/mineral supplemented, orange juice/skimmed milk containing high-carb diet, and the majority were rewarded with improved arterial compliance.
Remember this paper the next time you see someone on Good Morning America talking about how a recent study “proves” the the low-carb diet is bad.