Critique 157: A mistaken interpretation of data relating alcohol to mortality – 20 February 2015
Knott CS, Coombs N, Stamatakis E, Biddulph JP. All cause mortality and the case for age specific alcohol consumption guidelines: pooled analyses of up to 10 population based cohorts. BMJ 2015;350:h384 doi: 10.1136/bmj.h384.
Authors’ Abstract
Objectives To examine the suitability of age specific limits for alcohol consumption and to explore the association between alcohol consumption and mortality in different age groups.
Design Population based data from Health Survey for
Setting Up to 10 waves of the Health Survey for
Participants The derivation of two analytical samples was based on the availability of comparable alcohol consumption data, covariate data, and linked mortality data among adults aged 50 years or more. Two samples were used, each utilising a different variable for alcohol usage: self reported average weekly consumption over the past year and self reported consumption on the heaviest day in the past week. In fully adjusted analyses, the former sample comprised Health Survey for England years 1998-2002, 18 368 participants, and 4102 deaths over a median follow-up of 9.7 years, whereas the latter comprised Health Survey for England years 1999-2008, 34 523 participants, and 4220 deaths over a median follow-up of 6.5 years.
Main outcome measure All cause mortality, defined as any death recorded between the date of interview and the end of data linkage on 31 March 2011.
Results In unadjusted models, protective effects were identified across a broad range of alcohol usage in all age-sex groups. These effects were attenuated across most use categories on adjustment for a range of personal, socioeconomic, and lifestyle factors. After the exclusion of former drinkers, these effects were further attenuated. Compared with self reported never drinkers, significant protective associations were limited to younger men (50-64 years) and older women (≥65 years). Among younger men, the range of protective effects was minimal, with a significant reduction in hazards present only among those who reported consuming 15.1-20.0 units/average week (hazard ratio 0.49, 95% confidence interval 0.26 to 0.91) or 0.1-1.5 units on the heaviest day (0.43, 0.21 to 0.87). The range of protective effects was broader but lower among older women, with significant reductions in hazards present ≤10.0 units/average week and across all levels of heaviest day use. Supplementary analyses found that most protective effects disappeared where calculated in comparison with various definitions of occasional drinkers.
Conclusions Beneficial associations between low intensity alcohol consumption and all cause mortality may in part be attributable to inappropriate selection of a referent group and weak adjustment for confounders. Compared with never drinkers, age stratified analyses suggest that beneficial dose-response relations between alcohol consumption and all cause mortality may be largely specific to women drinkers aged 65 years or more, with little to no protection present in other age-sex groups. These protective associations may, however, be explained by the effect of selection biases across age-sex strata.
Forum Comments
A group of investigators has carried out a pooled analysis of published data from population-based cohorts and attempted to evaluate the association of alcohol consumption in age-specific categories with the risk of total mortality. The results presented by the authors clearly show reduced risk of total mortality (hazard ratios < 1.0 for mortality in comparison with current non-drinkers and with never drinkers) for subjects in essentially all categories of moderate drinking. However, because of relatively small numbers of subjects and wide confidence intervals for the sub-categories of age, many of the estimates of effect were not statistically significant.
The investigators then appear to have looked only at p-values and ignored the estimates of effect in their own data, coming to the conclusion of no association between moderate alcohol and total mortality for most age groups. Focusing only on significance testing for estimating effects has been strongly condemned by Kenneth Rothman and other leading epidemiologists (Rothman et al 2008; Schmtdt & Rothman 2014).
Forum member Thelle elaborates on this topic: “Type I and II errors force the researcher to decide whether or not to reject the null hypothesis. Valid information may thus be lost, and has led the scientific community into what Stang et al called ‘a tyranny of statistical significance testing.’ The p-value is a continuous measure, but it is often given as a dichotomous variable. By convention in clinical medicine one has ended up with p < 0.05 indicating what is called a statistically significant level, by which the null hypothesis is rejected. This means that we have made a decision regarding the results by setting a limit for when we will accept the alternative hypothesis. This convention is to a large extent ‘non-sense’, as the clinical importance of a difference does not depend on a p-value being above or below 0.05. The p-value is a quantitative finding that is a result of the precision of instruments and the effect size, and contains more information than what is given by drawing a limit at 0.05.”
As pointed out by Forum member Waterhouse: “The paper reflects a misinterpretation of the data, not what I would expect from educated scientists. They have data that cannot demonstrate a difference, which they misinterpret to say it shows there is no difference. In fact, their data can only be interpreted to say they can’t conclude anything about whether or not there is an effect. Their data could not establish that alcohol had ‘no effect’ on mortality, as they loudly proclaimed.” Forum members (and many other scientists, see below) found it upsetting that the authors and the journal did not understand this basic error in their presentation of statistics, or chose to ignore it. (A recent commentary by Tom Jacobs calls attention to “how ideology may trump science” in interpreting scientific data.)
Forum member de Gaetano, with assistance from his colleagues Augusto Di Castelnuovo and Simona Costanzo, agree that “The authors’ conclusions are not backed up by the data. A more appropriate headline based on this paper would be ‘Study supports a moderate protective effect of alcohol.’” Forum member McCormick added: “I am happy to comment on this paper. It scratches one of my perennial itches, i.e., poorly designed studies somehow getting approval from a theoretically sound journal.” And reviewer Ursini: “It is not just a problem of alcohol and mortality, it is matter of ethics in writing and publishing papers. I totally agree with the concise commentary of Professor David Spiegelhalter (see below) which goes directly to the point. In light of the data available, I cannot understand how it went through the review process. Reviewers and handling editors here have a serious responsibility before publishing scientific results and conclusions not based on data presented.”
Reviewer Keil described the danger of such misguided reports: “Last week I attended a continuing education event with my cardiology colleagues. The first lecture was given by a scientist who referred to this paper in the BMJ and wanted to revise the recommendations on alcohol intake. Can it be that more than 30 years of research in this field are undone by one silly paper in the BMJ? Once in a while I reflect on how doctors read papers; they seem to rely more on abstracts and press releases.”
Reviewer Skovenborg had comments as well: “The problem of concluding from a data-set with type-2 errors has been dealt with properly by Professor Spiegelhalter. Further, there is no known biological pathway to explain the decrease in mortality associated with drinking less than once per month (observed in younger and older men and older women in this study) which raises the suspicion of either misclassification, underreporting of alcohol intake, or confounding. Considering the rather meticulous control for known confounding factors, that leaves an unknown confounding factor of considerable magnitude, which is unlikely. Underreporting and/or misclassification seem to be more likely explanations. As stated by Spiegelhalter: ‘This is an exceptionally poor use of statistics.’
“The authors quote two studies on the subject of alcohol and age with this comment: ‘In general data specific to older populations are lacking…’ I do not agree and I have identified more than 25 relevant references. For example, Grønbæk et al reported on 11.5 years follow-up of mortality among 16,304 men and women aged 50 years or more at baseline; they found a “U-shaped curve even among the elderly. In a report from McCaul et al, in people over the age of 65 years, alcohol intake of four standard drinks per day for men and two standard drinks per day for women was associated with lower mortality risk. Halohan et al reported that in their cohort, even after adjusting for all covariates, abstainers and heavy drinkers continued to show increased mortality risks of 51 and 45%, respectively, compared to moderate drinkers. Further, in a pooled analysis of 8 prospective studies from North America and Europe including 192,067 women and 74,919 men, Hvidtfeldt et al found significant inverse associations between alcohol and risk of coronary heart disease in all age groups.”
A letter to BMJ by Forum members Van Velden and Kotze (and an associate, Lückhoff HK), states that “The conclusion of the authors of this paper is simply not supported by the data provided in this study. Rather, their findings are in accordance with those from previous prospective studies showing that moderate alcohol consumption decreases overall mortality risk, across boundaries for age and different categories of intake.”
References from Forum review
Gronbaek M, Deis A, Becker U, et al. Alcohol and mortality: is there a U-shaped relation in elderly people? Age and Ageing 1998;27:739-744.
Holahan CJ et al. Late-life alcohol consumption and 20-year mortality. Alcohol Clin Exp Res 2010;34:1-11.
Hvidtfeldt UA et al. Alcohol intake and risk of coronary heart disease in younger, middleaged, and older adults. Circulation 2010;121:1589-1597.
Jacobs T. “Ideology Often Trumps Science, Especially Among Conservatives. Two carefully couched studies parse how our political views impact the way we respond to scientific findings.” 10 February 2015; http://www.psmag.com/politics-and-law/ideology-often-trumps-science-especially-among-conservatives ?utm_source=feedburner&utm_medium= feed&utm_campaign=Feed:%2Bmiller-mccune%2Fmain_feed%2B(Pacific%2BStandard%2B-%2BMain%2BFeed)
McCaul KA et al. Alcohol use and mortality in older men and women. Addiction 2010;105:1391-1400.
Rothman KJ, Greenland S. Lash TL. Precision and statistics in epidemiologic studies. Modern Epidemiology. 3rd ed; Lippincott Williams & Wilkins, Philadelphia, PA, 2008.
Schmidt M, Rothman KJ. Mistaken inference caused by reliance on and misinterpretation of a significance test. Int J Cardiol 2014;177:1089-1090.
Stang A, Poole C, Kuss O. The ongoing tyranny of statistical significance testing in biomedical research. Eur J Epidemiol 2010;25: 225–230. doi: 10.1007/s10654-010-9440-x.
Van Velden D, Lückhoff HK, Kotze MJ. Misrepresentation of findings from recent alcohol protection study in the media: a cause for alarm. http://www.bmj.com/content/350/bmj.h384/rr-3.
Summary of Forum Review
The authors of this paper have carried out a regression analysis to examine the association of reported alcohol consumption with all-cause mortality, dividing their sample into different age groups. They used data from Health Survey for England 1998-2008, linked to national mortality registration data. Their published results show lower risk of mortality (hazard ratios < 1.0 for mortality in comparison with recent non-drinkers and with never drinkers) for subjects in essentially all categories of moderate drinking. However, the authors have interpreted their analyses as indicating that moderate drinking is not associated with all-cause mortality for the vast majority of the population.
Forum members, and many other scientists, have been surprised by the conclusions of the authors, who apparently did not consider basic statistical principles (such as dealing with type-2 errors) in judging their results. The investigators appear to have looked only at p-values and ignored the estimates of effect in their own data, coming to the conclusion of no association between moderate alcohol and total mortality for most age groups. Focusing only on significance testing for estimating effects has been strongly condemned by epidemiologists and statisticians and can lead to invalid conclusions.
The paper reflects a gross misinterpretation of the data, not what would be expected in a publication in a leading journal. In other words, the authors’ conclusions are not backed up by their data. A more appropriate headline based on this paper would be “Study supports a moderate protective effect of alcohol against all-cause mortality.” Given the wide media coverage of this article, with striking headlines indicating that moderate drinking does not affect mortality, one Forum reviewer wondered “How can more than 30 years of research in this field be undone by one misguided paper in the BMJ? Once in a while I reflect on how some doctors and journalists interpret scientific papers; they seem to rely more on abstracts and press releases.”
The Forum considers that the conclusions of the authors of this paper are simply not supported by the data provided. Rather, their findings are in accordance with those from previous prospective studies showing that, for all age groups, moderate alcohol consumption is associated with a decrease in all-cause mortality risk.
* * *
Contributions to this critique by the International Scientific Forum on Alcohol Research have been provided by the following members:
Giovanni de Gaetano, MD, PhD, Department of Epidemiology and Prevention, IRCCS Istituto Neurologico Mediterraneo NEUROMED, Pozzilli, Italy
Luc Djoussé, MD, DSc, Dept. of Medicine, Division of Aging, Brigham & Women’s Hospital and Harvard Medical School, Boston, MA, USA
R. Curtis Ellison, MD. Section of Preventive Medicine & Epidemiology, Department of Medicine, Boston University School of Medicine, Boston, MA, USA
Harvey Finkel, MD, Hematology/Oncology, Boston University Medical Center, Boston, MA, USA
Ulrich Keil, MD, PhD, Professor Emeritus, Institute of Epidemiology & Social Medicine, University of Muenster, Germany
Maritha J. Kotze, PhD, Human Genetics, Dept of Pathology, University of Stellenbosch, Tygerberg, South Africa.
Fulvio Mattivi, PhD, Head of the Department Food Quality and Nutrition, Research and Innovation Centre, Fondazione Edmund Mach, in San Michele all’Adige, Italy.
Ross McCormick, PhD, MSc, MBChB, Professor Emeritus, The University of Auckland; former Associate Dean, Faculty of Medical and Health Sciences, The University of Auckland, Auckland, New Zealand
Linda McEvoy, PhD, Department of Radiology, University of California at San Diego (UCSD), La Jolla, CA, USA
Erik Skovenborg, MD, specialized in family medicine, member of the Scandinavian Medical Alcohol Board, Aarhus, Denmark
Creina Stockley, PhD, MSc Clinical Pharmacology, MBA; Health and Regulatory Information Manager, Australian Wine Research Institute, Glen Osmond, South Australia, Australia.
Arne Svilaas, MD, PhD, general practice and lipidology, Oslo University Hospital, Oslo, Norway
Pierre-Louis Teissedre, PhD, Faculty of Oenology–ISVV, University Victor Segalen Bordeaux 2, Bordeaux, France
Dag S. Thelle, MD, PhD, Senior Professor of Cardiovascular Epidemiology and Prevention, University of Gothenburg, Sweden; Senior Professor of Quantitative Medicine at the University of Oslo, Norway
Fulvio Ursini, MD, Dept. of Biological Chemistry, Universityof Padova, Padova, Italy
David Vauzour, PhD, Senior Research Associate, Department of Nutrition, Norwich Medical School, University of East Anglia, Norwich, UK
David Van Velden, MD, Dept. of Pathology, Stellenbosch University, Stellenbosch, South Africa
Andrew L. Waterhouse, PhD, Department of Viticulture and Enology, University of California, Davis.
Yuqing Zhang, MD, PhD, Epidemiology, Boston University School of Medicine, Boston, MA, USA
* * *
Published comments on this study by other scientists
A large number of comments related to this article have been published by other scientists. For example, on 10 February 2015, Science Media Centre carried a number of criticisms of this article (available at http://www.sciencemediacentre.org/expert-reaction-to-study-on-health-effects-of-alcohol-across-different-age-groups/). To quote from one such critique:
Prof. Sir David Spiegelhalter, Winton Professor of the Public Understanding of Risk, University of Cambridge, wrote: “The authors’ conclusions are not backed up by the data. All groups consuming less than 20 units a week experienced lower mortality rates than the lifelong teetotalers. But since there are not many teetotalers, there is large uncertainty about what the true underlying relative risks are. All the observed data are compatible with the kind of 15 to 20% protection that has been previously suggested, and the authors are not justified in claiming there is no protection apart from some specific groups. A graphic depiction of their clearly shows the observed hazard ratio (relative risk of dying each year) – curiously such a graph did not appear in the published paper, but can be derived from the data provided in the tables.”
Spiegelhalter continued: “Essentially, the study is grossly underpowered to convincingly detect a plausible protection, and they have committed the cardinal sin of saying that non-significance is the same as ‘no effect’ in a study lacking sufficient events, in this case, deaths in non-drinkers. This is a poor use of statistics, and I am surprised it got past the referees.” His full critique is available at http://understandinguncertainty.org/misleading-conclusions-alcohol-protection-study.
* * *
In addition, Prof. Paul Pharoah, Professor of Cancer Epidemiology, University of Cambridge, said: “Overall the findings of this study are in broad agreement with what has been previously published – despite what is written in the press release. The main findings were that there was a reduction in mortality in almost all categories of alcohol consumption (main number reported in the results table is the relative hazard. A relative hazard of < 1 is a protective effect). For some of the categories this finding was statistically significant and not in others. The investigators make too much of these differences in nominal statistical significance. While some results were statistically significant and others not, the consistency of the findings in the different age/sex groups is more striking. I do not agree with their conclusion that ‘Little to no protection was found in other age-sex groups, regardless of consumption level.’ Because there were 10 different alcohol consumption groups being evaluated the number of deaths in each group was fairly small – particularly in the non-drinker reference group – and the statistical power to detect modest effects will have been small. In short the findings – although not statistically significant in part – were fairly consistent with previously reported research in which moderate alcohol consumption has been associated with a modest reduction in mortality. But the authors’ conclusions are not backed up by the data.”
* * *
A comment entitled, “Don’t worry, drinking is still good for you” by Christopher Snowdon, Researcher, Institute of Economic Affairs, UK concludes: “No reasonable person could conclude from this study that ‘Alcohol has no health benefits after all’, as Britain’s erstwhile newspaper of record claims. On the contrary, it shows much the same as all the other evidence on this subject: that moderate drinkers live longer than both non-drinkers and never-drinkers.” The full comments of Snowdon are available at