PHI Institute selected to present comment to the Federal Coordinating Committee on Comparative Effectiveness Research “listening session” on behalf of ABQAURP, AHIMA Foundation, CMSA, IHPM, & PHI Institute: 5/13/09
Click here to view webcast (this statement begins about the 32 minute mark, hold for for 15 minutes to hear panel’s comment & question to this statement)
To the Federal Coordinating Council for Comparative Effectiveness Research:
On behalf of the American Board of Quality Assurance and Utilization Review Physicians, Inc., the American Health Information Management Association Foundation, the Case Management Society of America, the Institute of Health and Productivity Management, and the Population Health Impact Institute, we thank the Federal Coordinating Council for Comparative Effectiveness Research, the Department of Health and Human Services (HHS), the Department of Veterans Affairs, the Department of Defense, the Office of Management and Budget and the Agency for Healthcare Research and Quality for today’s opportunity to comment on Comparative Effectiveness Research (CER).
To make Comparative Effectiveness Research (CER) useful and understandable to health care consumers and providers, tax payers, business purchasers and the government, we wish to stress the central role of transparency of methods in reducing bias and rebuilding trust.
The disclosure of possible bias because of a researcher’s potential conflict of interest, common in peer-reviewed journals, is obviously important. But a more serious form of bias is related to the admonition of the 17th century founders of the Royal Society of London, which became its slogan: Nullius in Verba. – “Don’t take anyone’s word for it!”
It is essential that the public understands that biased results can result from the methods employed in the research. Confirmation and reproducibility are the keys to ascertaining the truth.
The methods that can lead to bias include the evaluation design chosen, assumptions behind statistical tests, imperfect comparability of populations being compared, data integrity, choice of cut-off points, mathematical errors, and other internal and external validity issues. The tried-and-true way to resolve this problem is to provide detailed, timely, and clearly written disclosures of the methods used, which will enable the user of CER findings to “trust, but verify.”
Our concern is that the traditional reliance on expert and anonymous peer review to ferret out these problems is not working. The peer-reviewed literature itself has revealed that results in more than one-third of articles in major peer-reviewed journals were later overturned or mitigated [1]. Only eight percent of scientists –one-in twelve — said it “works well as it is.” [2] While this self-criticism shows the eventual positive role of peer-review, we need to take action now to enable the public to trust the system. This newly expanded CER endeavor must signal the start of a new system that should include the following six points:
1) Leaders should state that most, if not all, of science will have strengths as well as weaknesses, biases and shortcomings. We recommend that CER researchers should pledge to reduce bias–it will rarely if ever be totally eliminated –and to prominently state in clear language the usefulness as well limitations of their findings.
2) Researchers should show results prior to “adjustments,” as well as “adjusted” results. If the journal refuses to publish details, then the researchers should make these results easily available to readers on-line.
3) Research papers should disclose, in detail, the methods and metrics used, the assumptions behind statistical tests, the extent to which assumptions are met (or not met), and the comparability of populations compared, as well as the strengths and weaknesses of their results.
4) Funders should allow for an open –even anonymous (and moderated) — peer-review process, in a timely and on-going fashion at the beginning, middle, and end of a research study.
5) The Department of Health and Human Services should consider compensating official, conflict-free peer reviewers at a rate that is competitive with the rates paid to those conducting the CER.
6) A structured approach to method disclosures, such as that developed by the Population Health Impact Institute (www.PHIinstitute.org), should be considered.
We thank the Federal Coordinating Council for Comparative Effectiveness Research for this opportunity to comment.
Respectfully yours, <!–{12428638854085}–><!–{12428638854086}–><!–{12428638854087}–>
Dr. Arthur I. Broder, MD, CHCQM
Chairman of the Board, American Board of Quality Assurance and Utilization Review Physicians, Inc (www.abqaurp.org)
Mary Madison, MPA
Executive Director, American Health Information Management Association Foundation (www.ahimafoundation.org)<!–{12428638854088}–><!–{12428638854089}–>
Jeff Frater, RN, BSN
President, Case Management Society of America (www.cmsa.org)
Sean Sullivan, JD
President and CEO, Institute for Health and Productivity Management (www.ihpm.org)
Thomas W. Wilson, PhD, DrPH [presenter]
Board Chair, Population Health Impact Institute (www.PHIinstitute.org)
End notes:
[1] Ioannidis JP. Contradicted and initially stronger effects in highly cited clinical research. JAMA. 2005 Jul 13;294(2):218-28.
[2] ChubinDE, Hackett EJ. Peerless Science: Peer Review and U.S. Science Policy (S U N Y Series in Science, Technology, and Society, 1990: pg. 192