Cataract surgeons outperform medical students in Eyesi virtual reality cataract surgery: evidence for construct validity

Introduction

Learning cataract surgery is a timely assignment. Besides requiring resource intense training, inexperienced cataract surgeons also have more complications than more experienced ones (Randleman et al. 2007). Capsulorhexis and phacoemulsification are considered to be the most difficult steps for a new cataract surgeon to handle (Dooley & O’Brien 2006). It would be preferable if the initial learning of these steps could take place in a safe and standardized simulated environment instead of in the operation theatre where standardization is difficult if not impossible, and patient safety is compromised. For some time, surgical simulators have been used to train and assess surgical skills (Thijssen & Schijven 2010). For cataract surgery, a few simulators have been presented (Laurell et al. 2004; Mahr & Hodge 2008; Choi et al. 2009), and the Eyesi surgical simulator (VR Magic AG, Mannheim, Germany) has been partly evaluated for validity (Mahr & Hodge 2008; Privett et al. 2010; Selvander & Asman 2010; Le et al. 2011).

In a retrospective study, Belyea et al. (2011) recently found better phacoemulsification usage in real operations associated with training in a virtual reality eye surgery simulator. It has been shown that, for the capsulorhexis procedure, experienced cataract surgeons perform better than residents on the Eyesi surgical simulator (Privett et al. 2010). So far though, no studies have reported the validity and training effects specifically for the hydromaneuvers and phaco divide and conquer modules. The aim of this study was to investigate the construct validity for these modules.

Material and Methods

Seven cataract surgeons and seventeen medical students participated in the study. The group of surgeons were recruited on a voluntarily basis and they were practicing at the Skåne University Hospital or at a local hospital in the region. Five of them had performed 800–10 000 cataract operations, one had performed over 150 cataract operations, and one had done 18 cataract operations. The group of students were attending their 9th semester and were recruited during their 6-week ophthalmology rotation.

The students were given a brief orientation on important anterior chamber structures before the simulator training started. Both the students and the surgeons were instructed in a standardized manner on the function of the simulator. None of the students had previous experience with this or any other intraocular surgical simulator. One of the cataract surgeons had several years before been exposed to an intraocular surgical simulator for demonstration purposes only. The other six surgeons had not trained in an intraocular surgical simulator before.

Surgical simulator

To evaluate the study participants’ cataract surgical skills, the intraocular surgical simulator, Eyesi, was used. This simulator has previously been described (Selvander & Asman 2010), and for this study, the cataract head with a model eye for cataract surgery was used. Probes are inserted into a model eye and a virtual binocular image is created. The image is shown through two oculars giving an image of the anterior segment. Software version 2.5 was used.

Three cataract modules and three manipulating modules were used. The level for each module was chosen based on a pilot study so that tasks would be difficult enough to enable discrimination based on surgical skill and be as close as possible to reality. Cataract modules: the capsulorhexis module (level four of ten) where the trainee has to inject viscoelastics into the anterior chamber, create a flap with a cystotome and pull the flap and perform a capsulorhexis with a forceps (Fig 1A); the hydromaneuvers module (level one of four) where the trainee has to make a hydrodissection of the lens and move the nucleus around to prove that a dissection has been obtained (Fig 1B); the phaco divide and conquer module (level five of six) where the trainee has to divide a nucleus into four quadrants and thereafter remove and emulsify each quadrant with the phaco probe (Fig. 1C). Manipulation modules: the cataract navigation training module (level two of three) where the trainee has to hold an instrument tip steady in mobile spheres scattered at different levels in the anterior chamber (Fig. 2A); the cataract forceps training module (level four of four) where the trainee has to move triangles lying on top of the lens into a fictive basket sitting in the middle of the anterior chamber (Fig. 2B); the cataract cracking and chopping module (level six of eight) where the trainee has to pull each end of a handle simultaneously thereby elongating it into a given length (Fig. 2C). Before using each new module, the trainee was shown an instructional video.

The tasks were performed at a single 60 min session in the above given order three times each except for the phaco divide and conquer module in which two iterations were performed. Several different performance parameters were calculated in the simulator scoring system. A summary parameter (overall score) was obtained for all iterations. For the second iteration, additional parameters were also obtained (Table 1).

Table 1. Comparison of performance parameters between cataract surgeons and medical students’ second iteration (median, range).

	Cataract surgeons	Medical students	p
Module: Capsulorhexis
Overall score (of 100)	54 (0–75)	0 (0–78)	0.035*
Average radius of capsulorhexis value	2.27 (2.09–3.21)	2.43 (1.58–3.02)	0.589
Centering (distance rhexis centre to eye centre)	0.39 (0.04–0.49)	0.56 (0.27–2.19)	0.065
Deviation of rhexis radius from 2.5 mm	0.26 (0.08–0.71)	0.28 (0.01–0.92)	0.949
Injured cornea area value	0 (0–0.67)	0.56 (0–16.3)	0.048*
Maximum radial extension of capsulorhexis value	2.66 (2.57–3.73)	3.59 (2.5–4.62)	0.098
Roundness of capsulorhexis value	0.553 (0.01–0.71)	0 (0–0.95)	0.099
Time (with instruments inserted) duration (second)	118 (73–425)	170 (93–373)	0.153
Module: Hydromaneuvers
Overall score (of 100)	86 (54–90)	81 (44–90)	0.503
Injured cornea area value	0 (0–0)	0 (0–0.57)	–**
Time (with instruments inserted) duration	21 (17–37)	25 (9–104)	0.408
Module: Phaco divide and conquer
Overall score (of 100)	0 (0–95)	0 (0–76)	0.081
Injured cornea area value	0 (0–0.722)	0.204 (0–5.85)	0.086
Time (with instruments inserted) duration	337 (263–480)	399 (207–590)	0.092
Ultrasonic energy value	444 (341–743)	536 (117–883)	0.874
Ultrasonic leakage value	372 (238–608)	414 (105–994)	0.874
Emulsification near capsule no events	3 (0–6)	6 (0–22)	0.051
Posterior capsule torn no events	1 (0–5)	4 (0–10)	0.063
Successful cracking attempts value	2 (1–3)	0 (0–3)	0.014*
Module: Cataract navigation training
Overall score (of 100)	57 (49–81)	55 (0–71)	0.307
Injured cornea area value	0 (0–0)	0 (0–5.61)	0.08
Instrument slipped out of sphere no. events	12 (7–31)	24 (7–53)	0.227
Odometer value	183 (123–315)	246.5 (110–460)	0.215
Time (with instruments inserted) duration	63 (49–91)	93 (41–480)	0.061
Module: Cataract forceps training
Overall score (of 100)	93 (76–100)	76 (4–98)	0.03*
Injured cornea area value	0 (0–0)	0.0185 (0–5.61)	0.021*
Odometer value	81.5 (59.1–92.7)	98.5 (77.5–213)	0.008*
Time (with instruments inserted) duration	36 (20–40)	38 (21–59)	0.024*
Module: Cataract cracking and chopping training
Overall score (of 100)	90 (78–98)	78 (1–98)	0.022*
Injured cornea area value	0 (0–0.185)	0 (0–5.02)	0.323
Instrument slipped out of sphere no. events	3 (1–13)	14 (1–37)	0.015*
Time (with instruments inserted) duration	24 (17–59)	58 (23–253)	0.016*
Injured lens area value	0 (0–0.305)	0.143 (0–4.64)	0.048*

• *Statistical significance.
• **Too many zeros allow for meaningful statistical analysis.

The sessions on the simulator were video recorded as seen on the observer screen (Fig. 3). The video recordings from the second iteration of the capsulorhexis, hydromaneuvers and phaco divide and conquer modules were saved for later evaluation. Because of a technical error, the module hydromaneuvers was not recorded for one of the students. For all the other participants, the appropriate modules were recorded.

Results

Cataract surgeons outperformed medical students with regard to overall score on the modules capsulorhexis (p < 0.001, p = 0.035, p = 0.010 for the tree iterations, respectively), cataract navigation training (p = 0.024, p = 0.307, p = 0.007), cataract forceps training (p = 0.017, p = 0.03, p = 0.028). Less obvious differences in overall score were found for modules cataract cracking and chopping training (p = 0.266, p = 0.022, p = 0.324) and phaco divide and conquer (p = 0.011, p = 0.081 for the two iterations, respectively). No differences in overall score were found on the hydromaneuvers module (p = 0.588, p = 0.503, p = 0.773) (Fig. 4).

Analysing the second iteration in detail, it could be noted that naïve trainees made more damage to the cornea (capsulorhexis, p = 0.048; cataract forceps training, p = 0.021) and lens (cataract cracking and chopping training p = 0.048) during training (Table 1). For the cataract forceps module, naive trainees had higher odometer and time duration value than the surgeons had (p=0.008; p=0.024) and during the phaco procedure, their cracking attempts were less successful (p = 0.014) (Table 1).

The difference between surgeons and students was also evident on the video evaluations of the capsulorhexis, hydromaneuvers and phaco procedures where surgeons received significantly higher and thus better scores from the evaluations of the modified OSATS (p = 0.001) and OSACSS (capsulorhexis, p = 0.003; hydromaneuvers, p = 0.017; phaco divide and conquer, p = 0.001) (Table 2).

Statistical analysis revealed significant correlations between the overall score given by the simulator and the video performance scores (OSACS) for capsulorhexis, hydromaneuvers and phaco (r = 0.669, p < 0.0001; r = 0.525, p = 0.010; r = 0.566, p = 0.004, respectively). Also, the modified OSATS score correlated significantly with the sum of overall score for the three modules above (r = 0.657, p = 0.001). Interrater reliability was high for OSACSS capsulorhexis (r = 0.788), phaco divide and conquer (r = 0.726) and OSATS (r = 0.764) and moderate for OSACSS hydrodissection (r = 0.598).

Discussion

Not surprisingly, differences in performance scores were found between medical students and cataract surgeons in this study. The Eyesi intraocular surgical simulator has previously been validated as a scoring tool for some cataract surgical skills (Privett et al. 2010; Le et al. 2011). We showed that cataract surgeons outperform naïve subjects on the capsulorhexis module. These results are in accordance with the study from Privett et al. where they showed that surgeons had not only better scores but they also had less damage to the cornea, which are findings also supported in our study. Given the differences between cataract surgeons and naïve subjects, a level of proficiency for overall score and injured cornea area can be determined for the capsulorhexis procedure (Table 1).

We could also demonstrate construct validity for the phaco module, and to the best of our knowledge, this is the first report showing construct validity also for this specific module. There were significant differences for successful cracking attempts between surgeons and students. However, the overall score for experienced cataract surgeons was disappointingly low (median 4 and 0 of 100 for the first and second iterations, respectively), and only the first iteration was significantly better than the students’. Two reasons may account for this: (i) inherent shortcomings in the scoring system or (ii) the low number of participants and the heterogeneity in experience among surgeons. Excluding the two junior cataract surgeons would make the surgeon group more homogenous but did not change the overall score median values. Interestingly, comparing the video recordings of the phaco procedures between cataract surgeons and naïve subjects, there were no difficulties in distinguishing surgical skills. These facts indicate that the scoring system in the simulator may be more important of these two explanations pointing at a need for scoring refinement to better distinguish phaco skills. In that sense, the evaluation tools OSACSS and even OSATS are currently more accurate tools to evaluate surgical skill on the phaco module.

For the hydromaneuvers module, the surgeons could not score significantly better than the naïve subjects on the simulator. The material is considerably small and the group of surgeons is heterogeneous. The hydromaneuver was also performed on the lowest and easiest level and perhaps could not be challenging enough to allow separation between the two groups. These are factors that influence the results. However, the video evaluation easily distinguished the surgeons among the participants. Also, the correlation between the video evaluation scores and the simulator overall score was weak for this module. This suggests that the simulator scoring system needs further development also for this module to better distinguish cataract surgical skills.

Belyea et al. (2011) showed that training has an effect on surgical outcome even though their training curriculum was flexible and less controlled allowing for training both in anterior and posterior segment modules. Having found construct validity for three important modules (capsulorhexis, hydromaneuvers and phaco) for cataract surgery calls for the implementation of a structured simulator training programme. For this purpose, it is important to be able to assess the trainee and to measure progress in skills’ training as well as to provide appropriate feedback (Kluger & DeNisi 1996; Mahmood & Darzi 2004). With this study, we can define levels of proficiency for the modules capsulorhexis and phaco based on the metrics from the simulator. Based on the scoring of the video recorded training, we can also set a level of proficiency for the hydromaneuvers procedure as well as the capsulorhexis and phaco procedure and for the three cataract procedures together. It is feasible because, after completion of a trial, the trainee can record the performance, and the video can subsequently be evaluated. While being a time-consuming procedure, it would be preferable to get this process automated via an improved scoring system in the simulator.

A limitation to our study is that it does not include residents. Le et al. (2011) showed a correlation between ophthalmic experience and total score in two manipulation modules (the forceps and antitremor modules). One might assume that similar correlations would exist also for other modules. However, Le et al. grouped their participants (medical students, residents and ophthalmic surgeons) based on level of general competence rather than surgical experience, thus causing overlap in surgical experience between groups. Their study participants also had shorter session times (20 min versus 60 min), and their participants were given only a brief orientation of the simulator. Our medical students were recruited during their ophthalmology rotation. They were thoroughly informed of important structures and important parts of the procedures. All study participants were shown voice-guided instructional videos pointing out important aspects before each new module was commenced. In this way, our naïve group is likely to be more similar to a group of residents without surgical experience.

Surgical simulation allows for training and testing in a safe and uniform manner. In this study, we have shown construct validity on several modules previously not investigated (phaco divide and conquer, hydromaneuvers, cataract cracking and chopping, cataract navigation training) on the Eyesi intraocular surgical simulator and confirmed previously demonstrated construct validity for the capsulorhexis and cataract forceps training modules (Privett et al. 2010; Le et al. 2011). Interestingly, validation of the hydromaneuvers module required the somewhat tedious OSACSS video evaluation tool. Thus, a further development of the scoring system in the simulator for the hydromaneuvers module would be advantageous and make training and evaluation of progress more accessible and immediate.