Response by Nina Teicholz, Science Journalist
For The BMJ press release, with links to all the comments, click here.
(This comment is longer than the one published by The BMJ.. Anything in this post that is not in The BMJ version is highlighted in green).
I’m delighted that The BMJ has stood by this article and decided against retraction. Two outside reviewers judged that the criticisms of the piece did not merit its retraction, and in the end, the corrections made by The BMJ do not, in my view, materially undermine any of the article’s key claims. This article therefore stands as one of the most serious ever, peer-reviewed critiques of the expert report for the US Dietary Guidelines for Americans (DGAs).
The importance of the DGAs, and therefore of this article, should not be understated (and indeed was recognized by many in the mainstream media when the article was published). The DGAs have long been considered the “gold standard,” informing the US food supply, military rations, US government feeding assistance programs such as the National School Lunch Program which are, altogether, consumed by 1 in 4 Americans each month, as well as the guidelines of professional societies, governments around the world, and eating habits generally.
Yet rates of obesity began to shoot upwards in the very year, 1980, that the DGAs were introduced, and the diabetes epidemic began soon thereafter. A critically important yet little understood issue is why the DGAs have failed, so spectacularly, to safeguard health from the very nutrition-related diseases that they were supposed to prevent.
In documenting fundamental failures in the science behind the DGAs, this article offers new insights; It establishes that a vast amount of nutrition science funded by the National Institutes of Health and other governments worldwide has, for decades, been systematically ignored or dismissed, and that therefore, that the DGAs are not based on a comprehensive reviews of the most rigorous science. Incorporating this long-ignored relevant science would likely lead to fundamentally different DGAs and could very well be an important step in infusing them with the power to better fight the nutrition-related diseases.
A fundamental question is why 170+ researchers (including all the 2015 DGA committee members, or “DGAC”), organized by the advocacy group, the Center for Science in the Public Interest (CSPI), would sign a letter asking for retraction. After all, in the weeks following publication, any person had the opportunity to submit a “Rapid Response” to the article, and both CSPI and the DGAC did so, alleging many errors. I responded to them all in my Rapid Response. This is the normal post-publication process.
Yet after all this, CSPI returned for a second round of criticisms, recycling two of the issues (CSPI points #3 and #10) that I had already addressed in my Rapid Response (and which had required no correction), adding another 9 (one of which, #4, contained no challenge of fact), and demanding that based on these alleged errors, the article be retracted. CSPI then circulated this letter widely to colleagues and asked them to sign on.
Last spring, a journalist for The Guardian, Ian Leslie, was able to quickly unmask the nature of CSPI’s campaign. Leslie interviewed many of the researchers who had signed CSPI’s petition:
“They were happy to condemn the article in general terms, but when I asked them to name just one of the supposed errors in it, not one of them was able to. One admitted he had not read it. Another told me she had signed the letter because the BMJ should not have published an article that was not peer reviewed (it was peer reviewed). Meir Stampfer, a Harvard epidemiologist, asserted that Teicholz’s work is ‘riddled with errors,’ while declining to discuss them with me.”
Indeed, quite a few sets of emails obtained by a blogger including those by Harvard professor Frank Hu, all obtained via public records requests, reveal researchers passing along the retraction request as if it were a chain letter, agreeing to sign on without asking a single question about the substance of the alleged errors.
This lack of substance in the retraction effort seems to point to the reality that it was first and foremost an act of advocacy—a heavy handed attempt to silence arguments with which CSPI, a longtime supporter of the Dietary Guidelines and its allies disagree.[1] And this applies not just to the retraction letter but to other CSPI efforts to stifle alternative viewpoints. Earlier this year, for example, I was dis-invited from the National Food Policy Conference after CSPI, together with the USDA official in charge of the Dietary Guidelines, threatened to withdraw if I were included, details of which are reported here and which a Spiked columnist called an act of “censorship.”
It’s important to note that I am not the only person disturbed by the lack of rigorous science underpinning our dietary guidelines. Numerous scientists around the world have expressed concern about the science. And indeed, this consternation is shared by no less than the US Congress, which held a hearing on Oct 7, 2015 to address its serious doubts about the DGAs. Such was this concern that last year that Congress mandated the first-ever major peer-review of the DGAs, by the National Academy of Medicine. Congress appropriated $1 million for this review, and it additionally stipulated that all members of the 2015 DGA committee recuse themselves from the process.
What is the dangerous information challenging the DGAs that cannot be heard on a conference panel nor published in a peer-reviewed journal?
The major findings of this article are that:
- The DGAC’s finding that the evidence of a “strong” link between saturated fats and heart disease was not clearly supported by the evidence cited. (Note that as of last year, the Heart and Stroke Foundation of Canada no longer limits saturated fats. Note, also, that Frank Hu, the Harvard epidemiologist in charge of the DGAC review on saturated fats, was an energetic promoter of the retraction letter against my article that critiqued his review, according to emails obtained through FOIA requests);
- Successive DGA committees have for decades ignored or dismissed a large body of rigorous (randomized controlled trial) literature on the low-fat diet, on more than 50K subjects, collectively finding that this diet is ineffective for fighting obesity, diabetes, heart disease or any kind of cancer;
- Although the DGAs have for decades recommended avoiding saturated fats and cholesterol to prevent heart disease, no DGA committee has ever directly reviewed the enormous body of rigorous (government-funded, randomized controlled trials) evidence, testing more than 25,000 people, on this hypothesis. More than a reviews of this data have concluded that saturated fats have no effect on cardiovascular mortality or no association with cardiovascular outcomes;
- The DGAC ignored a large body of scientific literature on low-carbohydrate diets (including several “long term” trials, of 2-years duration) demonstrating that these diets are safe and highly effective for combatting obesity, diabetes, and heart disease;
- Although the DGAC is supposed to consult the NEL to conduct systematic reviews of the science, the 2015 DGAC did so for only 67% of the questions that required systematic reviews;
- For a number of key reviews, the 2015 DGAC relied on work done in part by the American Heart Association and the American College of Cardiology, which are private associations supported by industry and therefore have a potential conflict of interest;
- The DGAs, for the first time, introduce the “vegetarian diet” as one of its three, recommended “Dietary Patterns,” yet a NEL review of this diet concluded that the evidence for this its disease-fighting powers is only “limited,” which is the lowest rank of evidence assigned for available data;
- The DGA’s three recommended “Dietary Patterns” are supported by only “a miniscule quantity of rigorous evidence that only marginally supports claims that these diets can promote better health than alternatives.” The NEL review found only “limited” or “insufficient” evidence that the diets could combat diabetes and only “moderate” evidence that the diets can help people lose weight. The report also gave a strong rating to the evidence that its recommended diets can fight heart disease, yet here, several studies are presented, but none unambiguously supports this claim.
- The DGA process does not require committee members to disclose conflicts of interest and also that, for the first time, the committee chair came not from a university but from industry;
- The DGAC recommended reducing red and processed meats yet did not conduct an analysis of the scientific literature on this subject;
- The DGAC issued a recommendation to reduce salt as much as possible, which is not supported by an authoritative Institute of Medicine study on this subject;
- The 2015 DGAC conducted a number of reviews in ways that were not systematic. This allowed for the potential introduction of bias (e.g., cherry picking of the evidence).
This last claim, on the systematic nature of the DGAC reviews, is the subject of the corrections published in The BMJ this week, and refer to CSPI points #1, #2, #7, and #8 (two of which are statements in the text and two of which are in the supporting tables). I am grateful to have had the opportunity to work with The BMJ on developing this notice.
These corrections address the fact that, for three of the 2015 DGAC reviews, on saturated fats and on the “Dietary Patterns,” I did not note that these reviews had, in fact, included literature searches for the relevant papers. The DGAC conducted two searches: one on saturated fats and one on the Dietary Patterns. I regret that I did not notice these searches, but it is perhaps understandable given that (1) they are not in the printed report, (2) they are not mentioned in the methodology sections of each of these reviews, here, here, and here, (3) they are appended at the end of each relevant appendix, after the reference section. Points (2) and (3) are highly unusual for a scientific paper, and something I’ve rarely, if ever, encountered in the decade-plus I’ve spent reading papers in this field.
Yet the fundamentally important point here is that even with these searches, these reviews and a number of others used for the DGAs were not conducted systematically. This crucial point was confirmed by both of The BMJ’s outside reviewers.
One of the reviewers, Lisa Bero, describes how some of the reviews by the DGAC “circumvent the systematic search process,” and how, “regarding the search strategies, inclusion of studies and quality assessment,” there are aspects of the methods of the DGAC Report that “sometimes lack sufficient detail and may introduce bias.”
The other reviewer, Mark Hefland, writes, “Nothing in my report should be interpreted as a defense of the NEL systematic review process or of the DGAC guideline process…I found nothing to contradict Teicholz’ central concern that the DGAC’ processes to protect against bias are inadequate.”
Bero additionally writes:
“Bias was potentially introduced into the search strategies reported by DGAC by deciding a priori to use existing reports, failing to describe the criteria to determine when a review needs updating, and failing to describe how reviews were selected for inclusion or exclusion when they overlapped.
“This method introduces bias because a decision to conduct a search for reviews should be independent of a committee member’s knowledge of existing reviews. Otherwise, studies in high profile journals, certain fields of research, or from certain groups of researchers may be included while others are not.
“…This process also introduces bias, as the criteria for deciding when a review needs to be updated or not were not described.
“… The Evidence Summaries for each chapter (Appendix 2) give examples of studies that were excluded due to overlapping studies, but it is not clear why one review was selected for inclusion compared to another”
Hefland additionally writes:
“It is clear that further investigation of the composition of the committee, as well as its conflict of interest policies and work group structure, are warranted. The NEL [Nutrition Evidence Library] and DGAC do not appear to have incorporated key developments in methodology and governance of evidence-based guideline development since 2010.”
In my “Rapid Response” last year, when I first responded to criticisms of the article, I made many of these same points.
Regarding the specific reviews on saturated fats and the Dietary Guidelines, I maintain my assertion that these were non-systematic, for a number of reasons:
(1) in each case, a large portion of the papers were selected via “hand search.”[2] This is a method that was commonly used before papers became searchable electronically, but all the references listed in these reviews are available online and should have all been obtainable via standard electronic searches. Moreover, the DGAC reviews do not clearly specify which papers were obtained via hand searches, or what, if any, criteria were used for the selection of these papers, making these searches non-transparent and non-reproducible—a key criteria of systematic methodology.
(2) The appendices do not include the dates on which the electronic searches were conducted, an omission which further diminishes the transparency and reproducibility of these searches.
(3) The review on Dietary Patterns does not identify the electronic data bases searched, thereby making it non-reproducible.
Thus, for these reviews, the DGAC expert report demonstrably falls short of using transparent, reproducible and therefore systematic methods. Additionally, the review on saturated fats was one of those conducted outside the established NEL process.
Other experts have observed that the reviews fall short of systematic standards in other ways: there is no assessment of risk bias, no independent screening of evidence, and in many cases, no meta-analyses of results conducted on outcomes. Furthermore, many reviews searched few (or non-specified) databases and only those in the English language.
The one aspect of the BMJ review that is problematic, I believe, is that it relied heavily on one reviewer, Hefand, who is the director of a center that is both funded by, and solely in service to, the US Department of HHS, which was the lead government agency for the 2015 DGAs, the very document critiqued in this article. I believe this represents a potential conflict of interest. (And this may be why Hefland was so critical of both the article and of me personally, making a number of extraneous allegations in his review. My response to his many criticisms are here .)
Only Hefland recommended the corrections and clarifications that have formed the basis of the The BMJ’s notice. The other reviewer, Bero, in her written review, did not recommend any corrections or clarifications.
Over months of communications with The BMJ, I was able to argue that most of Hefland’s proposed corrections were in error. This resulted in his proposed corrections being downgraded to “clarifications,” some of which, in my view, are not needed:
- I think it is neither relevant nor clarifying to include the fact that the 2015 committee considered review papers that included some of the relevant trials on saturated fats. The central issue here is that the DGAs have, for decades, advised the public to limit saturated fats without ever directly reviewing the relevant, most rigorous science.
- I do not agree that my article was insufficiently clear about the 2010 NEL review. This review excluded a large, NIH-funded study that should have been included. The fact that the DGAC, five years later, cited a review paper that included this trial is irrelevant to the 2010 NEL review being discussed.
- I do not agree that a clarification is needed to note that one trial measured “serum lipids” as a proxy for heart disease. For 50+ years, researchers have used serum lipids, such as total cholesterol, as intermediary outcome measures for heart disease. This is widely known.
- For the reviews on red meat, any confusion about this statement could simply have been clarified by including the point that the reviews conducted no analyses of red meat or total meat on any health outcome.
In my view, only one clarification was truly needed: the addition of the word “directly” in the sentence about the DGAC not reviewing the trial literature on saturated fats. My detailed response to each of CSPI’s allegations and Dr. Hefland’s responses can be found here.
In sum, despite enormous scrutiny of and hostility towards this article, its allegations have stood up. Of the 11 errors alleged by CSPI, only 4 (two of which are in the supporting tables), pertaining to the predefined literature searches, have been confirmed, and none of these has changed the article’s allegation that these reviews were conducted in ways that were non-systematic.
If the science had been rigorously and comprehensively reviewed by the 2015 or any of the previous DGACs, I believe we would not have the grievous health problems that we do today.
[1] CSPI has fought for decades to eliminate saturated fats from the American food supply (so much so, that throughout the late 1980s, CSPI advocated for replacing saturated fats with trans fats and succeeded in driving up consumption of trans fats to historic levels, as described in The Big Fat Surprise, pp.227-228). CSPI has also long advocated for shifting away from animal foods containing saturated fats, towards a plant-based diet based on grains and industrial vegetable oils. The researchers who joined CSPI in signing the letter are largely adherents to this view; many have participated in generating the science that has been used to support the hypothesis that fat and cholesterol cause heart disease, and it is upon this hypothesis that the Guidelines have been based.
[2] Four out of 8, or ½, of the papers for the review on saturated fats, and 3 out of 11, or 1/3, of the papers for the reviews on Dietary Patterns were selected via hand searches (There was only one systematic search conducted for the reviews on the Dietary Patterns and all three outcomes: obesity, diabetes, and heart disease.).
