Observer Reliability and Agreement

The terms observer reliability and observer agreement represent different concepts. Reliability coefficients express the ability to differentiate between subjects. Agreement parameters determine whether the same value is achieved if a measurement is repeated. Parameters for reliability are ICCs for measurements on an interval scale. Parameters for agreement are kappa for measurements on a nominal scale, weighted kappa for an ordinal scale, and levels of agreement for measurements on an interval scale.

Assessing observer reliability and agreement is essential for the interpretation of clinical observations in research and medical practice. Moreover, only if the sources of measurement errors are known, can they be anticipated, thereby improving the quality of measurements.

References

1 Altman, D. G. (1991). Practical Statistics for Medical Research. Chapman & Hall, London.
10.1136/bmj.328.7441.702
PubMed Google Scholar
2 Bland, J. M. & Altman, D. G. (1996). Statistical methods for assessing agreement between two methods of clinical measurement, Lancet i, 307–310.
Google Scholar
3 Brennan, P. & Silman, A. (1992). Statistical methods for assessing observer variability in clinical measures, British Medical Journal 304, 1491–1494.
10.1136/bmj.304.6840.1491
CAS PubMed Web of Science® Google Scholar
4 Cohen, J. (1960). A coefficient of agreement for nominal scales, Educational and Psychological Measurement 20, 37–46.
10.1177/001316446002000104
PubMed Web of Science® Google Scholar
5 Cohen J. (1968). Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit, Psychological Bulletin 70, 213–220.
10.1037/h0026256
CAS PubMed Web of Science® Google Scholar
6 Cronbach, L. J., Gleser, G. C., Nanda, H. & Rajaratnam, N. (1972). The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles. Wiley, New York.
10.1126/science.176.4036.785
Web of Science® Google Scholar
7 Feinstein, A. R. & Cicchetti, D. (1990). High agreement but low kappa: I. The problems of two paradoxes, Journal of Clinical Epidemiology 43, 543–549.
10.1016/0895-4356(90)90158-L
CAS PubMed Web of Science® Google Scholar
8 Feinstein A. R., Cicchetti D. (1990). High agreement but low kappa: II. Resolving the paradoxes, Journal of Clinical Epidemiology 43, 551–558.
PubMed Web of Science® Google Scholar
9 Fleiss, J. L. (1981). Statistical Methods for Rates and Proportions, 2nd Ed. Wiley, New York.
Google Scholar
10 Fleiss, J. L. & Cohen, J. (1973). The equivalence of weighted kappa and the intra class correlation coefficient as measures of reliability, Educational and Psychological Measurement 33, 613–19.
10.1177/001316447303300309
PubMed Web of Science® Google Scholar
11 McGraw, K. O. & Wong, S. P. (1996). Forming inferences about intraclass correlation coefficients, Psychologists 1, 30–46.
Google Scholar
12 Shrout, P. E. & Fleiss, J. L. (1979). Intraclass correlations: uses in assessing rater reliability, Psychological Bulletin 86, 420–428.
10.1037/0033-2909.86.2.420
CAS PubMed Web of Science® Google Scholar
13 Stratford, P. (1989). Reliability: consistency or differentiating among subjects?, Physical Therapy 69, 299–300.
10.1093/ptj/69.4.299a
CAS PubMed Web of Science® Google Scholar

Citing Literature

Encyclopedia of Biostatistics

Browse other articles of this reference work: