Academic publishing was rocked by the news on July 8 that a company called Sage Publications is retracting 60 papers from its Journal of Vibration and Control, about the science of acoustics. The company said a researcher in Taiwan and others had exploited peer review so that certain papers were sure to get a positive review for placement in the journal. In one case, a paper’s author gave glowing reviews to his own work using phony names.
Acoustics is an important field. But in biomedicine faulty research and a dubious peer-review process can have life-or-death consequences. In June, Dr. Francis Collins, director of the National Institutes of Health and responsible for $30 billion in annual government-funded research, held a meeting to discuss ways to ensure that more published scientific studies and results are accurate. According to a 2011 report in the monthly journal Nature Reviews Drug Discovery, the results of two-thirds of 67 key studies analyzed by Bayer researchers from 2008-2010 couldn’t be reproduced.
That finding was a bombshell. Replication is a fundamental tenet of science, and the hallmark of peer review is that other researchers can look at data and methodology and determine the work’s validity. Dr. Collins and co-author Dr. Lawrence Tabak highlighted the problem in a January 2014 article in Nature. “What hope is there that other scientists will be able to build on such work to further biomedical progress,” if no one can check and replicate the research, they wrote.
The authors pointed to several reasons for flawed studies, including “poor training of researchers in experimental design,” an “emphasis on making provocative statements,” and publications that don’t “report basic elements of experimental design.” They also said that “some scientists reputedly use a ‘secret sauce’ to make their experiments workand withhold details from publication or describe them only vaguely to retain a competitive edge.”
Papers with such problems or omissions would never see the light of day if sound peer-review practices were in placeand their absence at many journals is the root of the problem. Peer review involves an anonymous panel of objective experts critiquing a paper on its merits. Obviously, a panel should not contain anyone who agrees in advance to give the paper favorable attention and help it get published. Yet a variety of journals have allowed or overlooked such practices.
Absent rigorous peer review, we get the paper published in June in the Proceedings of the National Academy of Sciences. Titled “Female hurricanes are deadlier than male hurricanes,” it concluded that hurricanes with female names cause more deaths than male-named hurricanesostensibly because implicit sexism makes people take the storms with a woman’s name less seriously. The work was debunked once its methods were examined, but not before it got attention nationwide.
Such a dubious paper made its way into national media outlets because of the imprimatur of the prestigious National Academy of Sciences.
Yet a look at the organization’s own submission guidelines makes clear that if you are a National Academy member today, you can edit a research paper that you wrote yourself and only have to answer a few questions before an editorial board; you can even arrange to be the official reviewer for people you know. The result of such laxity isn’t just the publication of a dubious finding like the hurricane gender-bias claim. Some errors can have serious consequences if bad science leads to bad policy.
In 2002 and 2010, papers published in the Proceedings of the National Academy of Sciences claimed that a pesticide called atrazine was causing sex changes in frogs. As a result the Environmental Protection Agency set up special panels to re-examine the product’s safety. Both papers had the same editor, David Wake of the University of California, Berkeley, who is a colleague of the papers’ lead author, Tyrone Hayes, also of Berkeley.
In keeping with National Academy of Sciences policy, Prof. Hayes preselected Prof. Wake as his editor. Both studies were published without a review of the data used to reach the finding. No one has been able to reproduce the results of either paper, including the EPA, which did expensive, time-consuming reviews of the pesticide brought about by the published claims. As the agency investigated, it couldn’t even use those papers about atrazine’s alleged effects because the research they were based on didn’t meet the criteria for legitimate scientific work. The authors refused to hand over data that led them to their claimed resultswhich meant no one could run the same computer program and match their results.
Earlier this month, Nature retracted two studies it had published in January in which researchers from the Riken Center for Development Biology in Japan asserted that they had found a way to turn some cells into embryonic stem cells by a simple stress process. The studies had passed peer review, the magazine said, despite flaws that included misrepresented information.
Fixing peer review won’t be easy, although exposing its weaknesses is a good place to start. Michael Eisen, a biologist at UC Berkeley, is a co-founder of the Public Library of Science, one of the world’s largest nonprofit science publishers. He told me in an email that, “We need to get away from the notion, proven wrong on a daily basis, that peer review of any kind at any journal means that a work of science is correct. What it means is that a few (1-4) people read it over and didn’t see any major problems. That’s a very low bar in even the best of circumstances.”
But even the most rigorous peer review can be effective only if authors provide the data they used to reach their results, something that many still won’t do and that few journals require for publication. Some publishers have begun to mandate open data. In March the Public Library of Science began requiring that study data be publicly available. That means anyone with the ability to check should be able to reproduce, validate and understand the findings in a published paper. This should also ensure that there is much better scrutiny of flawed claims about sexist weather events and hermaphroditic frogsbefore they appear on every news station in America.
The Corruption of Peer Review Is Harming Scientific Credibility
Hadly Health Manning
Academic publishing was rocked by the news on July 8 that a company called Sage Publications is retracting 60 papers from its Journal of Vibration and Control, about the science of acoustics. The company said a researcher in Taiwan and others had exploited peer review so that certain papers were sure to get a positive review for placement in the journal. In one case, a paper’s author gave glowing reviews to his own work using phony names.
Acoustics is an important field. But in biomedicine faulty research and a dubious peer-review process can have life-or-death consequences. In June, Dr. Francis Collins, director of the National Institutes of Health and responsible for $30 billion in annual government-funded research, held a meeting to discuss ways to ensure that more published scientific studies and results are accurate. According to a 2011 report in the monthly journal Nature Reviews Drug Discovery, the results of two-thirds of 67 key studies analyzed by Bayer researchers from 2008-2010 couldn’t be reproduced.
That finding was a bombshell. Replication is a fundamental tenet of science, and the hallmark of peer review is that other researchers can look at data and methodology and determine the work’s validity. Dr. Collins and co-author Dr. Lawrence Tabak highlighted the problem in a January 2014 article in Nature. “What hope is there that other scientists will be able to build on such work to further biomedical progress,” if no one can check and replicate the research, they wrote.
The authors pointed to several reasons for flawed studies, including “poor training of researchers in experimental design,” an “emphasis on making provocative statements,” and publications that don’t “report basic elements of experimental design.” They also said that “some scientists reputedly use a ‘secret sauce’ to make their experiments workand withhold details from publication or describe them only vaguely to retain a competitive edge.”
Papers with such problems or omissions would never see the light of day if sound peer-review practices were in placeand their absence at many journals is the root of the problem. Peer review involves an anonymous panel of objective experts critiquing a paper on its merits. Obviously, a panel should not contain anyone who agrees in advance to give the paper favorable attention and help it get published. Yet a variety of journals have allowed or overlooked such practices.
Absent rigorous peer review, we get the paper published in June in the Proceedings of the National Academy of Sciences. Titled “Female hurricanes are deadlier than male hurricanes,” it concluded that hurricanes with female names cause more deaths than male-named hurricanesostensibly because implicit sexism makes people take the storms with a woman’s name less seriously. The work was debunked once its methods were examined, but not before it got attention nationwide.
Such a dubious paper made its way into national media outlets because of the imprimatur of the prestigious National Academy of Sciences.
Yet a look at the organization’s own submission guidelines makes clear that if you are a National Academy member today, you can edit a research paper that you wrote yourself and only have to answer a few questions before an editorial board; you can even arrange to be the official reviewer for people you know. The result of such laxity isn’t just the publication of a dubious finding like the hurricane gender-bias claim. Some errors can have serious consequences if bad science leads to bad policy.
In 2002 and 2010, papers published in the Proceedings of the National Academy of Sciences claimed that a pesticide called atrazine was causing sex changes in frogs. As a result the Environmental Protection Agency set up special panels to re-examine the product’s safety. Both papers had the same editor, David Wake of the University of California, Berkeley, who is a colleague of the papers’ lead author, Tyrone Hayes, also of Berkeley.
In keeping with National Academy of Sciences policy, Prof. Hayes preselected Prof. Wake as his editor. Both studies were published without a review of the data used to reach the finding. No one has been able to reproduce the results of either paper, including the EPA, which did expensive, time-consuming reviews of the pesticide brought about by the published claims. As the agency investigated, it couldn’t even use those papers about atrazine’s alleged effects because the research they were based on didn’t meet the criteria for legitimate scientific work. The authors refused to hand over data that led them to their claimed resultswhich meant no one could run the same computer program and match their results.
Earlier this month, Nature retracted two studies it had published in January in which researchers from the Riken Center for Development Biology in Japan asserted that they had found a way to turn some cells into embryonic stem cells by a simple stress process. The studies had passed peer review, the magazine said, despite flaws that included misrepresented information.
Fixing peer review won’t be easy, although exposing its weaknesses is a good place to start. Michael Eisen, a biologist at UC Berkeley, is a co-founder of the Public Library of Science, one of the world’s largest nonprofit science publishers. He told me in an email that, “We need to get away from the notion, proven wrong on a daily basis, that peer review of any kind at any journal means that a work of science is correct. What it means is that a few (1-4) people read it over and didn’t see any major problems. That’s a very low bar in even the best of circumstances.”
But even the most rigorous peer review can be effective only if authors provide the data they used to reach their results, something that many still won’t do and that few journals require for publication. Some publishers have begun to mandate open data. In March the Public Library of Science began requiring that study data be publicly available. That means anyone with the ability to check should be able to reproduce, validate and understand the findings in a published paper. This should also ensure that there is much better scrutiny of flawed claims about sexist weather events and hermaphroditic frogsbefore they appear on every news station in America.
Nothing contained in this blog is to be construed as necessarily reflecting the views of the Pacific Research Institute or as an attempt to thwart or aid the passage of any legislation.