Making the AGREE tool more user-friendly: the feasibility of a user guide based on Boolean operators
N. Ann Scott PhD,
Carmen Moga MD, Christa Harstall MHSA,
Corresponding Author
N. Ann Scott PhD
Research Associate,
Dr N. Ann ScottInstitute of Health Economics#1200, 10405 Jasper AvenueEdmonton, ABCanada T5J 3N4E-mail: [email protected]Search for more papers by this authorChrista Harstall MHSA
Director, Health Technology Assessment Unit, Institute of Health Economics, Edmonton, AB, Canada
Search for more papers by this authorN. Ann Scott PhD,
Carmen Moga MD, Christa Harstall MHSA,
Corresponding Author
N. Ann Scott PhD
Research Associate,
Dr N. Ann ScottInstitute of Health Economics#1200, 10405 Jasper AvenueEdmonton, ABCanada T5J 3N4E-mail: [email protected]Search for more papers by this authorChrista Harstall MHSA
Director, Health Technology Assessment Unit, Institute of Health Economics, Edmonton, AB, Canada
Search for more papers by this authorAbstract
Rationale, aims and objectives The Appraisal of Guidelines Research and Evaluation (AGREE) instrument is a generic tool for assessing guideline quality. This feasibility study aimed to reduce the ambiguity and subjectivity associated with AGREE item scoring, and to augment the tool's capacity to differentiate between good- and poor-quality guidelines.
Methods A literature review was conducted to ascertain what AGREE instrument adjustments had been reported to date. The AGREE User Guide was then modified by:
- 1
constructing a detailed set of instructions, or dictionary, using Boolean operators, and
- 2
overlaying seven criteria to categorize guideline quality.
The feasibility of the Boolean-based dictionary was tested by three appraisers using three randomly selected guidelines on low back pain management. The dictionary was then revised and re-tested.
Results Of the 52 published studies identified, 14% had modified the instrument by adding or deleting items and 35% had adopted strategies, such as using a consensus approach, to overcome inconsistencies and ensure identical item scoring among appraisers. For the feasibility test, Pearson correlation coefficients ranged from 0.27 to 0.81. Revision and re-testing of the dictionary increased the level of agreement (range 0.41 to 0.94). Application of the revised dictionary not only decreased the variability of the domain scores, but also reduced the tool's reliability among inexperienced appraisers.
Conclusion Appraisers found the Boolean-based AGREE User Guide easier to use than the original, which improved their confidence in the tool. Good reliability was achieved in the feasibility test, but the reliability and validity of some of the changes will require further evaluation.
References
- 1 Institute of Medicine Committee to Advise the Public Health Service on Clinical Practice Guidelines (1990) Clinical Practice Guidelines: Directions For a New Program. Washington DC: National Academy Press.
- 2 Bishop, P. B. & Wing, P. C. (2003) Compliance with clinical practice guidelines in family physicians managing worker's compensation board patients with acute lower back pain. Spine Journal, 3 (6), 442–450.
- 3 Hakkennes, S. & Green, S. (2006) Measures for assessing practice change in medical practitioners. Implementation Science, 1, 29 (doi:10.1186/1748-5908-1-29).
- 4 Kohr, R. & Sawhney, M. (2005) Advanced practice nurses' role in the treatment of pain. The Canadian Nurse, 101 (3), 30–34.
- 5 The AGREE Collaboration (2003) Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Quality & Safety in Health Care, 12 (1), 18–23.
- 6 The AGREE Collaboration (2001) AGREE instrument. The AGREE Collaboration. Available at: http://www.agreecollaboration.org/pdf/agreeinstrumentfinal.pdf (last accessed 7 August 2009).
- 7 Burgers, J. S., Fervers, B., Haugh, M., Brouwers, M., Browman, G., Philip, T. & Cluzeau, F. A. (2004) International assessment of the quality of clinical practice guidelines in oncology using the Appraisal of Guidelines and Research and Evaluation Instrument. Journal of Clinical Oncology, 22 (10), 2000–2007.
- 8 Hurdowar, A., Graham, I. D., Bayley, M., Harrison, M., Wood-Dauphinee, S. & Bhogal, S. (2007) Quality of stroke rehabilitation clinical practice guidelines. Journal of Evaluation in Clinical Practice, 13 (4), 657–664.
- 9 Horvath, A. R., Nagy, E. & Watine, J. (2005) Quality of guidelines for the laboratory management of diabetes mellitus. Scandinavian Journal of Clinical and Laboratory Investigation. Supplementum, 240, 41–50.
- 10 Cates, J. R., Young, D. N., Bowerman, D. S. & Porter, R. C. (2006) An independent AGREE evaluation of the Occupational Medicine Practice Guidelines. Spine Journal, 6 (1), 72–77.
- 11 The AGREE Collaboration (2008) The AGREE Collaboration website. The AGREE Collaboration. Available at: http://www.agreecollaboration.org (last accessed 7 August 2009).
- 12 Nuckols, T. K., Lim, Y. W., Wynn, B. O., Mattke, S., MacLean, C. H., Harber, P., Brook, R. H., Wallace, P., Garland, R. H. & Asch, S. (2008) Rigorous development does not ensure that guidelines are acceptable to a panel of knowledgeable providers. Journal of General Internal Medicine, 23 (1), 37–44.
- 13 Vlayen, J., Aertgeerts, B., Hannes, K., Sermeus, W. & Ramaekers, D. (2005) A systematic review of appraisal tools for clinical practice guidelines: multiple similarities and one common deficit. International Journal for Quality in Health Care, 17 (3), 235–242.
- 14
Holohan, V.,
Deenadayalan, Y. &
Grimmer, K. (2006) Evidence-based physiotherapy for acute low back pain: a composite clinical algorithm synthesized from seven recent clinical guidelines.
Physiotherapy Canada, 58 (4), 280–292.
10.3138/ptc.58.4.280 Google Scholar
- 15 Minhas, R. (2007) Eminence-based guidelines: a quality assessment of the second Joint British Societies' guidelines on the prevention of cardiovascular disease. International Journal of Clinical Practice, 61 (7), 1137–1144.
- 16 Boluyt, N., Lincke, C. R. & Offringa, M. (2005) Quality of evidence-based pediatric guidelines. Pediatrics, 115 (5), 1378–1391.
- 17 Harpole, L. H., Kelley, M. J., Schreiber, G., Toloza, E. M., Kolimaga, J. & McCrory, D. C. (2003) Assessment of the scope and quality of clinical practice guidelines in lung cancer. Chest, 123 (Suppl 1), 7S–20S.
- 18 Wimpenny, P. & Van Zelm, R. (2007) Appraising and comparing pressure ulcer guidelines. Worldviews on Evidence-Based Nursing, 4 (1), 40–50.
- 19 Arnau, J. M., Vallano, A., Lopez, A., Pellise, F., Delgado, M. J. & Prat, N. (2006) A critical review of guidelines for low back pain treatment. European Spine Journal, 15 (5), 543–553.
- 20 Institute of Health Economics (2008) The Alberta Ambassador Program website. Institute of Health Economics. Available at: http://www.ihe.ca/research/ambassador-program (last accessed 7 August 2009).
- 21 Rashiq, S., Barton, P., Harstall, C., Schopflocher, D. & Taenzer, P. for the Alberta Ambassador Program Team (2006) The Alberta Ambassador Program: delivering Health Technology Assessment results to rural practitioners. BMC Medical Education, 6, 21. doi:10.1186/1472-6920-6-21
- 22 Hayward, R. S. A., Wilson, M. C., Tunis, S. R., Bass, E. B. & Guyatt, G. for the Evidence-Based Medicine Working Group(1995) Users' guides to the medical literature. VIII. How to use clinical practice guidelines. A. Are the recommendations valid? Journal of the American Medical Association, 274 (7), 570–574.
- 23 Scott, N. A., Moga, C., Barton, P., Rashiq, S., Schopflocher, D., Taenzer, P. & Harstall, C. (2007) Creating clinically relevant knowledge from systematic reviews: the challenges of knowledge translation. Journal of Evaluation in Clinical Practice, 13 (4), 681–688.
- 24 Van Tulder, M., Furlan, A., Bombardier, C., Bouter, L. & the Editorial Board of the Cochrane Collaboration Back Review Group (2003) Updated method guidelines for systematic reviews in the Cochrane Collaboration Back Review Group. Spine, 28 (12), 1290–1299.
- 25 Fervers, B., Burgers, J. S., Haugh, M. C., Brouwers, M., Browman, G., Cluzeau, F. & Philip, T. (2005) Predictors of high quality clinical practice guidelines: examples in oncology. International Journal for Quality in Health Care, 17 (2), 123–132.
- 26 Qaseem, A., Vijan, S., Snow, V., Cross, J. T., Weiss, K. B. & Owens, D. K. (2007) Glycemic control and type 2 diabetes mellitus: the optimal hemoglobin A1c targets. A guidance statement from the American College of Physicians. Annals of International Medicine, 147 (6), 417–422.
- 27 Stiegler, M., Rummel, C., Wahlbeck, K., Kissling, W. & Leucht, S. (2005) European psychiatric treatment guidelines: is the glass half full or half empty? European Psychiatry, 20 (8), 554–558.
- 28 Sinuff, T., Patel, R. V., Adhikari, N. K., Meade, M. O., Schünemann, H. J. & Cook, D. J. (2008) Quality of professional society guidelines and consensus conference statements in critical care. Critical Care Medicine, 36 (4), 1049–1058.
- 29 The AGREE Collaboration (2003) AGREE instrument training manual. The AGREE Collaboration. Available at: http://www.agreecollaboration.org/pdf/aitraining.pdf (last accessed 7 August 2009).
- 30 Wegman, A., Van der Windt, D., Van Tulder, M., Stalman, W. & De Vries, T. (2004) Nonsteroidal antiinflammatory drugs or acetaminophen for osteoarthritis of the hip or knee? A systematic review of evidence and guidelines. Journal of Rheumatology, 31 (2), 344–354.
- 31 Haagen, E. C., Hermens, R. P., Nelen, W. L., Braat, D. D., Grol, R. P. & Kremer, J. A. (2006) Subfertility guidelines in Europe: the quantity and quality of intrauterine insemination guidelines. Human Reproduction, 21 (8), 2103–2109.
- 32
Brosseau, L.,
Graham, I.,
Casimiro, L.,
MacLeay, L.,
Cleaver, S.,
Dumont, A. M.,
Gravel, M.,
March, A. &
McGowan, J. (2004) What is the quality of clinical practice guidelines accessible on the world wide web for the treatment of musculoskeletal conditions in physiotherapy?
Physiotherapy Theory and Practice, 20 (2), 91–105.
10.1080/09593980490453057 Google Scholar
- 33 Hulshof, C. & Hoenen, J. (2007) Evidence-based practice guidelines in OHS: are they agree-able? Industrial Health, 45 (1), 26–31.
- 34 Verwey, B., Van Waarde, J. A., Van Rooij, I. A., Gerritsen, G. & Zitman, F. G. (2006) Availability, content and quality of local guidelines for the assessment of suicide attempters in university and general hospitals in the Netherlands. General Hospital Psychiatry, 28 (4), 336–342.
- 35 Ten Asbroek, A. H., Delnoij, D. M., Niessen, L. W., Scherpbier, R. W., Shrestha, N., Bam, D. S., Gunneberg, C., Van der Hor, C. W. & Klazinga, N. S. (2005) Implementing global knowledge in local practice: a WHO lung health initiative in Nepal. Health Policy and Planning, 20 (5), 290–301.
- 36 Graham, I. D., Harrison, M. B., Brouwers, M., Davies, B. L. & Dunn, S. (2002) Facilitating the use of evidence in practice: evaluating and adapting clinical practice guidelines for local use by health care organizations. Journal of Obstetric, Gynecologic, and Neonatal Nursing, 31 (5), 599–611.
- 37 MacDermid, J. C., Brooks, D., Solway, S., Switzer-McIntyre, S., Brosseau, L. & Graham, I. D. (2005) Reliability and validity of the AGREE instrument used by physical therapists in assessment of clinical practice guidelines. BMC Health Services Research, 5 (1), 18.
- 38 The EQUATOR Network (2008) Introduction to reporting guidelines. The EQUATOR Network. Available at: http://www.equator-network.org/index.aspx?o=1032 (last accessed 7 August 2009).
- 39 Shiffman, R. N., Shekelle, P., Overhage, J. M., Slutsky, J., Grimshaw, J. & Deshpande, A. M. (2003) Standardized reporting of clinical practice guidelines: a proposal from the Conference on Guideline Standardization. Annals of International Medicine, 139 (6), 493–498.
- 40 Fautrel, B., Pham, T. & Mouterde, G., et al. (2007) Recommendations of the French Society for Rheumatology regarding TNFα antagonist therapy in patients with rheumatoid arthritis. Joint Bone Spine, 74 (6), 627–637.
- 41 Green, A. & Kirk, J. on behalf of the Guidelines Development Group (2007) Guidelines for the performance of the sweat test for the diagnosis of cystic fibrosis. Annals of Clinical Biochemistry, 44 (Pt 1), 25–34.
- 42
Berti, E. &
Grilli, R. (2003) Practice guidelines and clinical governance: do the means match with the ends? A quality appraisal of local practice guidelines.
Clinical Governance, 8 (4), 312–317.
10.1108/14777270310499405 Google Scholar
- 43 Devroey, D., Vantomme, K., Betz, W., Vandevoorde, J. & Kartounian, J. (2004) A review of the treatment guidelines on the management of low levels of high-density lipoprotein cholesterol. Cardiology, 102 (2), 61–66.
- 44 Novara, G., Galfano, A., Gardi, M., Ficarra, V., Boccon-Gibod, L. & Artibani, W. (2006) Critical review of guidelines for BPH diagnosis and treatment strategy. European Urology Supplements, 5, 418–429.
- 45 Pencharz, J. N., Grigoriadis, E., Jansz, G. F. & Bombardier, C. (2002) A critical appraisal of clinical practice guidelines for the treatment of lower-limb osteoarthritis. Arthritis Research, 4 (1), 36–44.
- 46 Poitras, S., Avouac, J. & Rossignol, M., et al. (2007) A critical appraisal of guidelines for the management of knee osteoarthritis using Appraisal of Guidelines Research and Evaluation criteria. Arthritis Research & Therapy, 9 (6), R126 (doi:10.1186/ar2339).
- 47 Staal, J. B., Hlobil, H., Van Tulder, M. W., Waddell, G., Burton, A. K., Koes, B. W. & Van Mechelen, W. (2003) Occupational health guidelines for the management of low back pain: an international comparison. Occupational and Environmental Medicine, 60 (9), 618–626.
- 48 Van Tulder, M. W., Tuut, M., Pennick, V., Bombardier, C. & Assendelft, W. J. (2004) Quality of primary care guidelines for acute low back pain. Spine, 29 (17), E357–E362.
- 49 Watine, J., Friedberg, B., Nagy, E., Onody, R., Oosterhuis, W., Bunting, P. S., Charet, J. C. & Horvath, A. R. (2006) Conflict between guideline methodologic quality and recommendation validity: a potential problem for practitioners. Clinical Chemistry, 52 (1), 65–72.
- 50 Watine, J. C. & Bunting, P. S. (2008) Mass colorectal cancer screening: methodological quality of practice guidelines is not related to their content validity. Clinical Biochemistry, 41 (7–8), 459–466.
- 51 Voellinger, R., Berney, A. & Baumann, P., et al. (2003) Major depressive disorder in the general hospital: adaptation of clinical practice guidelines. General Hospital Psychiatry, 25 (3), 185–193.
- 52 Baird, A., Olarinde, O. & Talbot, M. (2007) Evaluation, using two assessment instruments, of the American and British national guidelines for the management of sexually transmissible and genital infections. Sexual Health, 4 (4), 255–260.
- 53 Appleyard, T. L., Mann, C. H. & Khan, K. S. (2006) Guidelines for the management of pelvic pain associated with endometriosis: a systematic appraisal of their quality. International Journal of Obstetrics and Gynaecology, 113 (7), 749–757.
Citing Literature
