Performance Characteristics of a Methodology to Quantify Adverse Events over Time in Hospitalized Patients
Paul J. Sharek
Division of General Pediatrics, Department of Pediatrics, Lucile Packard Children's Hospital and Stanford University School of Medicine, 700 Welch Rd., Suite #227, Palo Alto, CA 94304
Address correspondence to Paul J. Sharek, M.D., M.P.H., Division of General Pediatrics, Department of Pediatrics, Lucile Packard Children's Hospital and Stanford University School of Medicine, 700 Welch Rd., Suite #227, Palo Alto, CA 94304; e-mail: [email protected]. Gareth Parry, M.Sc., Ph.D., is with the Institute for Healthcare Improvement, Department of Pediatrics, Harvard Medical School, Cambridge, MA. Donald Goldmann, M.D., is with the Institute for Healthcare Improvement, Department of Pediatrics, Harvard Medical School, Divisions of Immunology, Infectious Diseases, and Epidemiology, Harvard School of Public Health, Cambridge, MA. Kate Bones, Roger Resar, M.D., and Frances A. Griffin, R.R.T., M.P.A., are with the Institute for Healthcare Improvement, Cambridge, MA. Andrew Hackbarth, is with the Pardee RAND Graduate School, RAND Corporation, Santa Monica CA. Dale Rhoda, M.A.S., M.S., M.P.P., and Cathy Murphy, are with the Centers for Public Health Research and Evaluation, Battelle Memorial Institute, Durham NC. Christopher P. Landrigan, M.D., M.P.H., is with the Departments of Medicine, Brigham and Women's Hospital and Children's Hospital Boston, Harvard Medical School, Boston, MA.
Search for more papers by this authorGareth Parry
Institute for Healthcare Improvement, Department of Pediatrics, Harvard Medical School, Cambridge, MA
Search for more papers by this authorDonald Goldmann
Institute for Healthcare Improvement, Department of Pediatrics, Harvard Medical School, Divisions of Immunology, Infectious Diseases, and Epidemiology, Harvard School of Public Health, Cambridge, MA
Search for more papers by this authorAndrew Hackbarth
Pardee RAND Graduate School, RAND Corporation, Santa Monica CA
Search for more papers by this authorRoger Resar
Institute for Healthcare Improvement, Cambridge, MA
Search for more papers by this authorFrances A. Griffin
Institute for Healthcare Improvement, Cambridge, MA
Search for more papers by this authorDale Rhoda
Public Health Research and Evaluation, Battelle Memorial Institute, Durham NC
Search for more papers by this authorCathy Murphy
Public Health Research and Evaluation, Battelle Memorial Institute, Durham NC
Search for more papers by this authorChristopher P. Landrigan
Departments of Medicine, Brigham and Women's Hospital and Children's Hospital Boston, Harvard Medical School, Boston, MA.
Search for more papers by this authorPaul J. Sharek
Division of General Pediatrics, Department of Pediatrics, Lucile Packard Children's Hospital and Stanford University School of Medicine, 700 Welch Rd., Suite #227, Palo Alto, CA 94304
Address correspondence to Paul J. Sharek, M.D., M.P.H., Division of General Pediatrics, Department of Pediatrics, Lucile Packard Children's Hospital and Stanford University School of Medicine, 700 Welch Rd., Suite #227, Palo Alto, CA 94304; e-mail: [email protected]. Gareth Parry, M.Sc., Ph.D., is with the Institute for Healthcare Improvement, Department of Pediatrics, Harvard Medical School, Cambridge, MA. Donald Goldmann, M.D., is with the Institute for Healthcare Improvement, Department of Pediatrics, Harvard Medical School, Divisions of Immunology, Infectious Diseases, and Epidemiology, Harvard School of Public Health, Cambridge, MA. Kate Bones, Roger Resar, M.D., and Frances A. Griffin, R.R.T., M.P.A., are with the Institute for Healthcare Improvement, Cambridge, MA. Andrew Hackbarth, is with the Pardee RAND Graduate School, RAND Corporation, Santa Monica CA. Dale Rhoda, M.A.S., M.S., M.P.P., and Cathy Murphy, are with the Centers for Public Health Research and Evaluation, Battelle Memorial Institute, Durham NC. Christopher P. Landrigan, M.D., M.P.H., is with the Departments of Medicine, Brigham and Women's Hospital and Children's Hospital Boston, Harvard Medical School, Boston, MA.
Search for more papers by this authorGareth Parry
Institute for Healthcare Improvement, Department of Pediatrics, Harvard Medical School, Cambridge, MA
Search for more papers by this authorDonald Goldmann
Institute for Healthcare Improvement, Department of Pediatrics, Harvard Medical School, Divisions of Immunology, Infectious Diseases, and Epidemiology, Harvard School of Public Health, Cambridge, MA
Search for more papers by this authorAndrew Hackbarth
Pardee RAND Graduate School, RAND Corporation, Santa Monica CA
Search for more papers by this authorRoger Resar
Institute for Healthcare Improvement, Cambridge, MA
Search for more papers by this authorFrances A. Griffin
Institute for Healthcare Improvement, Cambridge, MA
Search for more papers by this authorDale Rhoda
Public Health Research and Evaluation, Battelle Memorial Institute, Durham NC
Search for more papers by this authorCathy Murphy
Public Health Research and Evaluation, Battelle Memorial Institute, Durham NC
Search for more papers by this authorChristopher P. Landrigan
Departments of Medicine, Brigham and Women's Hospital and Children's Hospital Boston, Harvard Medical School, Boston, MA.
Search for more papers by this authorAbstract
Objective. To assess the performance characteristics of the Institute for Healthcare Improvement Global Trigger Tool (GTT) to determine its reliability for tracking local and national adverse event rates.
Data Sources. Primary data from 2008 chart reviews.
Study Design. A retrospective study in a stratified random sample of 10 North Carolina hospitals. Hospital-based (internal) and contract research organization–hired (external) reviewers used the GTT to identify adverse events in the same 10 randomly selected medical records per hospital in each quarter from January 2002 through December 2007.
Data Collection/Extraction. Interrater and intrarater reliability was assessed using κ statistics on 10 percent and 5 percent, respectively, of selected medical records. Additionally, experienced GTT users reviewed 10 percent of records to calculate internal and external teams' sensitivity and specificity.
Principal Findings. Eighty-eight to 98 percent of the targeted 2,400 medical records were reviewed. The reliability of the GTT to detect the presence, number, and severity of adverse events varied from κ=0.40 to 0.60. When compared with a team of experienced reviewers, the internal teams' sensitivity (49 percent) and specificity (94 percent) exceeded the external teams' (34 and 93 percent), as did their performance on all other metrics.
Conclusions. The high specificity, moderate sensitivity, and favorable interrater and intrarater reliability of the GTT make it appropriate for tracking local and national adverse event rates. The strong performance of hospital-based reviewers supports their use in future studies.
Supporting Information
Appendix SA1: Author Matrix.
Please note: Wiley-Blackwell is not responsible for the content or functionality of any supporting materials supplied by the authors. Any queries (other than missing material) should be directed to the corresponding author for the article.
| Filename | Description |
|---|---|
| HESR_1156_sm_authormatrix.doc80 KB | Supporting info item |
Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.
REFERENCES
- Agency for Healthcare Research and Quality. 2009. “Healthcare Cost and Utilization Project (HCUP)” [accessed on December 28, 2009]. Available at http://hcupnet.ahrq.gov/HCUPnet.jsp?Id=537A491834DBC7EC&Form=SelDB&JS=Y&Action=%3E%3ENext%3E%3E&_DB=NIS06
- AHA Annual Survey Database for Fiscal Year 2005 and AHA Annual Survey Database for Fiscal Year 2004 [accessed on July 27, 2010]. Available at http://www.ahadata.com/ahadata/html/AHASurvey.html
- Baker, G. R., P. G. Norton, V. Flintoft, R. Blais, A. Brown, J. Cox, E. Etchells, W. A. Ghali, P. Hébert, S. R. Majumdar, M. O'Beirne, L. Palacios-Derflingher, R. J. Reid, S. Sheps, and R. Tamblyn. 2004. “The Canadian Adverse Event Study: The Incidence of Adverse Events among Hospital Patients in Canada. Canadian Medical Association Journal 170 (11): 1678–86.
- Berwick, D. M., D. R. Calkins, C. J. McCannon, and A. D. Hackbarth. 2006. “The 100,000 Lives Campaign: Setting a Goal and a Deadline for Improving Health Care Quality. Journal of the American Medical Association 295 (3): 324–7.
- Brennan, T. A., L. L. Leape, N. M. Laird, L. Hebert, A. R. Localio, A. G. Lawthers, J. P. Newhouse, P. C. Weiler, H. H. Hyatt. 1991. “Incidence of Adverse Events and Negligence in Hospitalized Patients. Results of the Harvard Medical Practice Study I. New England Journal of Medicine 324 (6): 370–7.
- Brennan, T. A., R. J. Localio, and N. L. Laird. 1989. “Reliability and Validity of Judgments Concerning Adverse Events Suffered by Hospitalized Patients. Medical Care 27 (12): 1148–58.
- Catalano, K. 2006. “JCAHO's National Patient Safety Goals 2006. Journal of Perianesthesia Nursing 21 (1): 6–11.
-
Classen, D. C.,
R. C. Lloyd,
L. Provost,
F. A. Griffin, and R. Resar. 2008. “Development and Evaluation of the Institute for Healthcare Improvement Global Trigger Tool.
Journal of Patient Safety
4
(3): 169–77.
10.1097/PTS.0b013e318183a475 Google Scholar
- Fleiss, J. L., and J. Cohen. 1973. “The Equivalence of Weighted Kappa and the Intraclass Correlation Coefficient as Measures of Reliability. Educational and Psychological Measurement 33: 613–9.
- Griffin, F. A., and D. Classen. 2008. “Detection of Adverse Events in Surgical Patients Using the Trigger Tool Approach. Quality and Safety in Health Care 17 (4): 253–8.
- Griffin, F. A., and R. K. Resar. 2007. IHI Global Trigger Tool for Measuring Adverse Events. IHI Innovation Series White Paper. Cambridge, MA: Institute for Healthcare Improvement.
- Jha, A. K., E. J. Orav, A. B. Ridgway, J. Zheng, and A. M. Epstein. 2008. “Does the Leapfrog Program Help Identify High-Quality Hospitals? Joint Commission Journal on Quality and Patient Safety 34 (6): 318–25.
- Joint Commission on the Accreditation of Healthcare Organizations. 2009. Facts about Patient Safety [accessed on July 27, 2010]. Available athttp://www.jointcommission.org/PatientSafety/facts_patient_safety.htm
- L. T. Kohn, J. M. Corrigan, and M. S. Donaldson eds. 2000. To Err Is Human: Building a Safer Health System. Washington, DC: National Academy Press.
- Landis, J. R., and G. G. Koch. 1977. “The Measurement of Observer Agreement for Categorical Data. Biometrics 33 (1): 159–74.
- Leape, L. L., and D. M. Berwick. 2005. “Five Years after To Err Is Human: What Have We Learned? Journal of the American Medical Association 293 (19): 2384–90.
- McCannon, C. J., A. D. Hackbarth, and F. A. Griffin. 2007. “Miles to Go: An Introduction to the 5 Million Lives Campaign. Joint Commission Journal on Quality and Patient Safety 33 (8): 477–84.
- Milstein, A., R. S. Galvin, S. F. Delbanco, P. Salber, and C. R. Buck Jr. 2000. “Improving the Safety of Health Care: The Leapfrog Initiative. Effective Clinical Practice 3 (6): 313–6.
- National Coordinating Council for Medication Error Reporting and Prevention. 2001. “NCC MERP Endex for Categorizing Medication Errors” [accessed on April 6, 2010]. Available athttp://www.nccmerp.org
- Office of the Inspector General. 2010. “Adverse Events in Hospitals: Methods for Identifying Events,” OEI-06-08-00221 [accessed on April 6, 2010]. Available athttp://www.oig.hhs.gov/oei/reports/oei-06-08-00221.pdf
- Ornstein, C. 2007. “Dennis Quaid Files Suit over Drug Mishap. LA Times. December 5 [accessed on April 6, 2010]. Available athttp://www.latimes.com/features/health/la-me-quaid5dec05,1,1883436.story
- Resar, R. K., J. D. Rozich, T. Simmonds, and C. R. Haraden. 2006. “A Trigger Tool to Identify Adverse Events in the Intensive Care Unit. Joint Commission Journal on Quality and Patient Safety 32 (10): 585–90.
- Rozich, J. D., C. R. Haraden, and R. K. Resar. 2003. “Adverse Drug Event Trigger Tool: A Practical Methodology for Measuring Medication Related Harm. Quality and Safety in Health Care 12 (3): 194–200.
- Sari, A. B., T. A. Sheldon, A. Cracknell, and A. Turnbull. 2007. “Sensitivity of Routine System for Reporting for Reporting Patient Safety Incidents in an NHS Hospital: Retrospective Patient Case Note Review. British Medical Journal 334 (7584): 51–4.
- Sharek, P. J., and D. Classen. 2006. “The Incidence of Adverse Events and Medical Error in Pediatrics. Pediatric Clinics of North America 53 (6): 1067–7.
- Sharek, P. J., J. D. Horbar, W. Mason, H. Bisarya, C. W. Thurm, G. Suresh, J. E. Gray, W. H. Edwards, D. Goldmann, and D. Classen. 2006. “Adverse Events in the Neonatal Intensive Care Unit: Development, Testing, and Findings of an NICU-Focused Trigger Tool to Identify Harm in North American NICUs. Pediatrics 118 (4): 1332–40.
- Takata, G. S., W. Mason, C. Taketomo, T. Logsdon, and P. J. Sharek. 2008. “Development, Testing, and Findings of a Pediatric-Focused Trigger Tool to Identify Medication-Related Harm in US Children's Hospitals. Pediatrics 121 (4): e927–35.
- Thomas, E. J., S. R. Lipsitz, D. M. Studdert, and T. A. Brennan. 2002. “The Reliability of Medical Record Review for Estimating Adverse Event Rates. Annals of Internal Medicine 136 (11): 812–6.
-
Vemula, R.,
R. Robyn Assaf, and A. F. Al-Assaf. 2007. “Making the Patient Safety and Quality Improvement Act of 2005 Work.
Journal of Healthcare Quality
29
(4): 6–10.
10.1111/j.1945-1474.2007.tb00199.x Google Scholar
- Vincent, C., P. Aylin, B. D. Franklin, A. Holmes, S. Iskander, A. Jacklin, and K. Moorthy. 2008. “Is Healthcare Getting Safer? British Medical Journal 337: 1205–7.
Citing Literature
