From the Daily Peer Review of Abnormal Pap Test Slides to the Monitoring of Individual and Laboratory Performances: 5 Years of Data Collection and New Potential (Key) Performance Indicators
Funding: The authors received no specific funding for this work.
The Ispro Cytology Working Group members are listed in Appendix.
[Correction added on 09 Jul 2025, after first online publication: The group members have been listed in Appendix.]
ABSTRACT
Objectives
The Peer Review (PR) consists of the daily examination, by all cytologists, of Pap slides that resulted abnormal/difficult, in order to reach a consensus on the final diagnosis (FD). We explore data from 5 years (2017–2021) of PR to: (i) evaluate the agreement (both inter-observer and versus FD) over time; (ii) identify new quality indicators.
Methods
5673 slides were submitted to PR and examined by an average of 8 cytologists (range: 4–13). The agreement between cytologists and between the individual diagnosis with FD were evaluated by Kappa (k) and weighted Kappa (wK) and compared between ‘experts’ and ‘less experienced’ readers.
Results
The inter-observer agreement showed a moderate agreement among readers (whole team k = 0.44; experts k = 0.48). The highest and the lowest agreement was reported in HSIL and ASC-H, respectively. In 2018 and 2021, a significant reduction of kappa was observed, likely attributable to team turnover. The laboratory agreement versus FD resulted in significantly higher scores in experts (wk = 0.73, 95% CI 0.73–0.74) compared to less experienced individuals (wk = 0.65, 95% CI 0.64–0.66), with a general reduction of wk recorded in 2021. The individual agreement versus FD (calculated for 16 cytologists) achieved a moderate/substantial level of agreement (wK range: 0.57–0.80), with a shift toward higher wk in experts.
Conclusion
The levels of agreement are influenced by cytologist experience and team turnover. We propose new potential (key) performance indicators to strictly monitor the occurrence of systematic differences in interpretation criteria among cytologists. The proposed reference values are based on preliminary data and should be validated prospectively over a longer monitoring period.
Graphical Abstract
The study explores data from 5 years of Peer Review to evaluate the trend of agreement level (inter-observer and versus final diagnosis) over time and proposes new potential key performance indicators (KPIs), along with the relative reference values. These KPIs should be used to strictly monitor the occurrence of systematic differences in interpretation criteria among cytologists and to implement prompt corrective actions in case of significant deviations in diagnostic accuracy.
Conflicts of Interest
The authors declare no conflicts of interest.
Open Research
Data Availability Statement
The data that support the findings of this study are available from the corresponding author upon reasonable request.
