Introduction

The potential for radiographers to assist clinical decision making in the emergency department (ED) has evolved significantly over the past 40 years, with abnormality detection systems such as ‘red dot’ reporting, preliminary clinical evaluation in the United Kingdom (UK), and preliminary image evaluation (PIE) in Australia.^1-3 PCE and PIE are a feedback system which allows radiographers to write a short comment, alerting referrers to findings, or lack of, on X-ray examinations in ED.⁴ Importantly, PIE is not a replacement for an official ‘gold standard’ radiology report, but it can assist with clinical decision making for ED clinicians.⁵ While radiographer reporting is now part of extended practice in the UK in response to a significant shortage of radiologists to provide timely radiology reports, PIE is still used widely, particularly in settings where immediate reporting is not available by reporting radiographers or radiologists.⁶

Although there is desire for extended practice into radiographer reporting in NZ and Australia,^{7, 8} and image evaluation is included in undergraduate education,⁹ radiographer reporting in NZ is not currently supported by the Medical Radiation Technologists Board (MRTB).¹⁰ Further, radiologists are an additional barrier to radiographer reporting in Australia and New Zealand, who have outlined they do not support radiographers extending their practice to reporting X-ray examinations.¹¹ This is despite the New Zealand (NZ) public health service having approximately 60 full-time equivalent consultant and registrar radiologists per 1,000,000,¹² which frequently results in delays in radiology reports for ED patients. Furthermore, the distribution of radiologists differs by region, where often rural or provincial regions do not have access to a radiologist on-site.¹¹ The average time for a radiology report to be completed in the Taranaki District, a provincial district, between March 2022 and February 2023 was 4.52 days. Additionally, 24.32% of radiology reports were not completed for more than 7 days.¹³ This delay has led to ED clinicians making treatment decisions without the aid of the radiology report.¹⁴ As many of the ED clinicians are junior doctors, nurse practitioners and clinical nurse specialists, concerns have been raised about the performance of these clinicians to interpret X-ray examinations, which can vary with experience.^{3, 14-16} One study found that ED clinicians sensitivity was as low as 42% with an accuracy of 77%.¹⁷ The most common treatment error in ED is missed fractures, accounting for over 77% of treatment errors.¹⁸ Additionally, a study conducted in Australia demonstrated that 3.1% of fractures are missed at the time of treatment, only to be discovered when the radiology report became available.¹⁹ The same risk may be present within the New Zealand healthcare system as the context is closely aligned to Australia.⁸

Recent coroner's inquests in Australia have outlined that radiographers who see any abnormalities on an X-ray examination should alert referrers to these findings.²⁰ This was supported by the Medical Radiation Practice Board of Australia, and the MRTB, who both stipulate that all radiographers are responsible for recognising significant findings on X-ray examinations and must communicate these findings to the referrer.^{16, 21} To allow this communication to occur in a concise and timely manner, a formal system such as PIE has been suggested.^{5, 14, 16} In the past 10 years, PIE and similar systems have been expanding internationally, with use reported in South Africa, Singapore, Ghana, and significantly in Australia.^{5, 22-24}

There has been extensive research internationally using image evaluation tests demonstrating radiographers' performance in writing PIE comments increases following intensive training.^{4, 19, 22, 23, 25-30} Although these studies demonstrate that radiographers have an increase in performance immediately following intensive training, studies that follow participants at differing time periods after they have received intensive training have mixed results. For example, two studies demonstrated that participants' performance improved at 10–12 weeks post-training,^{19, 27} whereas one study showed a return to baseline performance 6 months following training.³⁰

Additionally, several studies discussed radiographer participants' confidence in participating in PIE and found while training increased their participation in PIE, they were still reluctant to write a PIE when they were unsure of the findings.^31-34 One suggestion to overcome this barrier was to ensure participating radiographers received regular feedback, which also helped improve the accuracy of the comments.³⁴ However, there is no known research to show the effect of regular feedback on participants' performance to perform PIE clinically. In addition, although there has been research conducted in Australia,^{3, 35-38} the UK,^{31, 39, 40} and other countries^{17, 24} to assess radiographers' performance when writing PIE comments, there is no published research assessing the clinical performance of radiographers in New Zealand.

The aim of this study was to determine what effect fortnightly feedback of common errors, with monthly targeted continuing professional development sessions, had on radiographers' performance when detecting and describing acute abnormalities on extremity X-ray examinations in ED.

Methods

Results

Over the 6-month study period, 844 X-ray examinations were evaluated by the participants, greater than the sample size required. The sample size used aligns with multiple other studies enabling direct comparisons.^{31, 35, 36, 40} The mean for sensitivity, specificity, and overall accuracy calculated with a 95% confidence interval were 92.8% (89.9, 95.8), 94.9 (93.1, 96.8), and 94.2 (91.9, 96.5), respectively. The mean comment accuracy was 91.41%. When the month-to-month results were analysed (Table 2), a key finding is the lower sensitivity in May and August, which was caused by these months featuring the highest number of false negatives, almost double the average, with a corresponding drop in comment and overall accuracy. Despite these outliers, the trend demonstrated an increase in sensitivity, specificity, overall accuracy, and comment accuracy (Fig. 1).

The median number of PIEs performed by the participants was 120 (range 24–205), with two significant outliers who performed the fewest number of PIEs, one with a significantly lower sensitivity (Table 3).

Of the 844 examinations, 21 false negatives and 28 false positives were identified. Analysis by anatomical region demonstrates the shoulder, pelvis and knee regions have higher overall accuracy, whereas elbow, wrist and ankle have the lowest overall accuracy (Table 4). The false negatives were assessed (Table 5), and the most common missed abnormalities were in the fingers, hand, and elbow.

There were only two studies that were marked as TP that had additional missed fractures. Both examinations involved complicated fractures, such as a bimalleolar fracture where only a Weber B fracture was noted, and a missed hook of hamate fracture in context of carpometacarpal dislocation.

During the assessment of PIEs, 10 of the 273 (3.7%) X-ray examinations with acute abnormalities had findings identified by participants who were not identified in the radiologist report. A senior radiologist assessed these studies, and in all instances the abnormalities were deemed to be missed by the reporting radiologist. The affected radiology reports were amended, and the referrers were notified ensuring the patient received the appropriate follow-up that was required. Six of these studies were missed abnormalities in the fingers or thumb, one was in an elbow, two were in the hip, and one was in a knee, similar to the findings of the missed abnormalities by the radiographers.

Discussion

This is the first known study about preliminary image evaluation to be conducted in New Zealand. The participants in this study demonstrated a sensitivity, specificity, and overall accuracy at a similar level of, or greater when compared to similar PIE studies conducted internationally (Table 6).^{3, 24, 31, 35, 36, 38, 40} Additionally, the participants demonstrated an increase in their performance over the 6 months of the study period.

The aim of this study was to determine what effect fortnightly feedback of common errors, with monthly targeted continuing professional development sessions, had on radiographers' performance when detecting and describing acute abnormalities on extremity X-ray examinations in ED. To achieve this aim, the comment accuracy was used to form the feedback given to participants with the intention of improving their performance when detecting and describing acute abnormalities, and therefore increase their confidence to participate.³² When the analysis was first conducted on the results from April, a key finding was that despite a high overall accuracy, the comment accuracy was low in comparison. One study conducted in the UK outlined that many PIE comments contained too many words which reduced their comment accuracy.⁴³ The lower comment accuracy in this study implied a similar phenomenon was occurring. It was decided that the participants required further guidance on how to write PIE comments. Feedback was therefore given to the participants to follow the “3Ws of PIE” outlined by Cooper et al.,¹⁴ “What is it, where is it, what is it doing.” While comment accuracy continued to be lower than overall accuracy, the margin was much closer for the remainder of the study. Although it was encouraging that participants' performance when performing PIE improved over 6 months, as PIE was not implemented into the department prior to the start of the study this increase in performance may be due to experiential learning, rather than the ongoing feedback. Further research would be required in a department with PIE previously established to determine whether follow-ups make a difference to participants' performance when detecting and describing acute abnormalities on extremity X-ray examinations. As confidence in writing PIE comments is a major barrier to radiographers participating in a PIE system,⁴⁴ it is suggested that the ongoing feedback could be useful to help remove this barrier for departments planning on implementing PIE in the future.

A further finding in this study is the number of missed abnormalities, by both the participants and the radiologists. These were prominently in the fingers, toes, wrist, and ankle, with avulsion fractures making up the majority of missed fractures. This is a common finding in other studies.^{40, 45} This finding demonstrates the importance of educating radiographers on the appearance of avulsion fractures, and that vigilance is required when searching for acute abnormalities, as small avulsion fractures are easily missed. Of note, 3.7% of abnormalities during this study were detected by the radiographer but missed by the reporting radiologist. Although the majority of the abnormalities were avulsion fractures, this study demonstrated that working together, a PIE with radiology report can help improve the accuracy of acute abnormality detection.

The number of X-ray examinations with missed additional acute abnormalities was minimal in this study, with only two in the context of significant trauma. It is likely these missed abnormalities were due to satisfaction of search, where participants stop searching for abnormalities in cases where obvious abnormality is present.¹⁴ Therefore, it is vital to ensure that when participating in PIE, participants receive education around satisfaction of search.

The abnormality prevalence in this study (34.7%) is higher than the anticipated prevalence based on an internal audit of the preceding year (26%). However, the prevalence found in this study is aligned with, or lower than the prevalence found in other similar studies.^{17, 24, 31, 35, 40} The most likely cause of the higher prevalence is a result of participants not writing PIE comments on studies that did not have an abnormality. This has the potential to skew the sensitivity higher, and the specificity lower.⁴⁶ However, it is also established that the higher prevalence could represent case selection bias, and suggests that participants may only be writing PIE comments on X-ray examinations they feel confident in.⁴⁷ The findings suggest that feedback and continuing professional development sessions can help maintain, or improve PIE performance, and therefore a robust audit and feedback system should be considered as part of a PIE system.

Limitations

The key limitations to this study are the main biases that exist: case selection bias and participant bias.⁴⁶ Case selection bias is where participants would only complete PIEs on X-ray examinations they felt confident in.⁴⁷ Regular communication and training was provided to help build radiographer confidence and minimise this bias. The second bias was participant bias, where only those participants who are interested in PIE volunteered to be a part of the study, and therefore the results are not an indication of the entire population at the study site. As this is a clinical study, this bias was minimised by assessing participants in their daily practice over a period of 6 months, ensuring their performance was assessed on an ongoing basis. Although these were identified as limitations, the biases were minimised as much as possible by ensuring there was an appropriate sample size collected over the entire study period to help minimise the potential for these to impact the results.

Another limitation in this study is the small population size. While the sample size ensured that the results were an accurate demonstration of the PIE performance of radiographers working in the Taranaki district, it cannot be certain that these results would extend to the wider radiographer population in NZ. It is recommended that further study is done to determine the performance of radiographers in NZ as a whole.

Conclusion

This study demonstrates that participants performed PIE to a high standard, which is comparable with radiographers internationally. Additionally, participants improved their performance when detecting and describing acute abnormalities over the course of the study. However, the study was unable to determine if this improvement was due to the ongoing feedback, or experiential learning. This study demonstrates radiographers working in two public hospitals within a single provincial district in NZ have a high level of sensitivity, specificity, and accuracy when performing PIE in ED. This suggests PIE would be useful to assist NZ ED clinicians with their treatment decisions when a radiology report unavailable. In addition, the number of missed acute appendicular abnormalities in ED will likely be reduced. It is recommended that participants in a PIE system receive training and ongoing feedback to provide confidence and support to ensure they can maintain their performance when detecting and describing abnormalities on acute extremity X-ray examinations.

Funding Information

Funds were provided by the New Zealand Institute of Medical Radiation Technology Continuing Education Fund for travel and accommodation funds for the intensive training course facilitator.

Conflict of Interest

The authors declare no conflict of interest.

Ethics Approval

Ethics approval was granted by the University of Auckland Ethics committee (AH25289) and Te Whatu Ora, Taranaki District Chief Medical Advisor.