A Multicenter Validity Study of Four Smartphone Hearing Test Apps in Optimized and Home Environments

Aim

To compare gold standard audiologist-led PTA versus smartphone hearing test results performed both at home and in hospital sound-treated booths.

Study Design

This is a prospective, multicenter validation study conducted between August 2021 and March 2023 at four hospitals providing secondary or tertiary ENT and audiology care. It was designed to validate the feasibility of using four smartphone audiology apps for patient-delivered hearing assessment in comparison to the gold standard of PTA.

Ethical approval was obtained from the Research Ethics Committee (20/WM/0324). Local approval was obtained at each center. All patients provided written informed consent before enrolment. STrengthening the Reporting of OBservational studies in Epidemiology guidelines were utilized.¹⁸

Participants were asked to download a named app onto their personal device. Four apps were selected, two from each of iOS and Android operating systems: Easy Hearing Test (iOS-1), Hearing Test and Ear Age Test (iOS-2), Eartone Hearing Test (Android-1), and Hearing Test (Android-2). Apps were selected on the basis that they were free, available in the United Kingdom (UK), provided a graphical output, and did not require participants to input personal information. The authors did not develop these apps. At the time of this work there were no apps available for both iOS and Android that met the selection criteria on both platforms.

Participants who were already attending outpatient clinic for a clinically-indicated PTA were approached by the study team. After consent was provided, patients were randomized to one app. Randomization was stratified by the operating system of the smartphone that the participant used. Each site was provided with numbered sealed envelopes with two series, marked either “Apple” or “Android”. Within the two series, there were a balanced number of envelopes specifying each of the apps (block randomization of four), with position in the series generated via an online randomizer. The envelopes contained app-specific instructions, consent forms, and inexpensive earbud-style earphones (Prosignal/PSG08468) to use for testing. These had been tested by our group and shown to provide adequate frequency response using KEMAR dummy head.¹⁹

Participants performed the app hearing test once in hospital (in a sound-treated room), and three further times at home (days 1, 3, and 7). They were instructed to perform the home hearing test in a quiet environment avoiding background noise. Participants returned their results by a secure study email address. Following completion, participants were invited to complete a voluntary online Google Forms survey regarding their experience of using the app at home. This consisted of a series of questions where participants could rank their view on a 5-point Likert scale: 1 was rated as “strongly disagree” and 5 as “strongly agree”.

Participants

Eligible participants were individuals over 18 years of age who presented to ENT or audiology clinics for outpatient assessment of their ears or hearing, and who owned a smartphone that ran iOS or Android operating system. Only participants requiring PTA as part of their standard care, as determined by a clinician independent of this study, were eligible for recruitment. Participants were included in analyses if they submitted at least one smartphone app hearing test.

Exclusion criteria included: 1. Active ear infection or discharge; 2. Hearing loss >90 dB at three consecutive frequencies.

The severity of hearing loss was classified according to the British Society of Audiology guidelines: normal <20 dB, mild 21–40 dB, moderate 41–70 dB, severe 71–95 dB, profound >95 dB.

Statistical Analysis

An a priori power calculation was performed to examine correlation across repeats in different environments (repeated measures ANOVA for multiple time points), to compare PTA versus apps (t-test for paired data), and to compare app-assisted hearing testing at the hospital versus at home (t-test) in G-Power (v3.1; Heinrich-Heine-Universität Düsseldorf, Düsseldorf, Germany). This indicated a sample size of n = 34 in each group (minimum n = 136 overall) based on 80% power and alpha of 0.05. Recruited patients who did not submit any data were not counted toward the overall sample size. Participants who submitted at least one smartphone result were included.

The following statistical analyses were performed in MATLAB (v 9.10 [R2021a]; Natick, Massachusetts). Data were examined for normal distribution. Pearson's correlation coefficient (CC) was used when data were normally distributed; Spearman's correlation test was planned if data were not normally distributed. The following comparisons were performed between smartphone app audiometry and PTA: (a) Ear-specific frequency-specific thresholds at 0.5, 1, 2, 4, and 8 kHz; (b) The pure tone average thresholds (mean of 0.5, 1, 2, 4, and 8 kHz). Although not conventional, 8 kHz was included in the pure tone average as it is valuable in ototoxicity monitoring. The following comparisons were performed between smartphone app audiometry performed in optimal (in sound-treated room) and home conditions: (a) Ear-specific frequency-specific thresholds; (b) Pure tone average thresholds.

Test–retest reliability for app pure tone average thresholds (mean of 0.5, 1, 2, 4, and 8 kHz) was examined by intraclass CC (ICC): optimal sound-treated versus final at home conditions, and first versus final at home conditions.

The median, interquartile range, and percentage of app-derived thresholds within ±10 dB of the hospital PTA were calculated at each frequency, and across the pure tone average.

A clinically reasonable correlation was defined as strong (0.6–0.79) or very strong (0.8–1) correlation, and median within ± 10 dB accuracy across all frequencies. Weak correlation was specified as 0–0.39; moderate as 0.4–0.59.

Demographics and Baseline Characteristics

One hundred fifty-six participants across four sites in the UK provided written informed consent (Supplementary Material 1). Seventeen recruited participants were excluded from analyses as they failed to submit any data to the study team. Data from 139 participants was submitted representing 388 smartphone hearing tests and 772 ear app-audiograms in total (two patients submitted unilateral readings). Sixty-five participants (47% of cohort) submitted a complete dataset comprising four app hearing test results.

The mean age of participants was 48 years (range:19–78, SD:15); 69 participants were female (50%). The most common documented reason for undergoing a PTA was chronic otitis media (COM) (Supplementary Material 2). 58% of ears (n = 161) had normal hearing, whilst the remainder had abnormal hearing (mild 26% [n = 71], moderate 13% [n = 36], severe 4% [n = 10]). The mean time to complete the app in the sound-treated room was 6 minutes (range:2–15, SD:2.9).

Smartphone Audiometry Versus PTA: Frequency-Specific Results

The Pearson's CC of PTA versus home app hearing test for the frequency-specific, ear-specific results are presented in Table I and stratified by left and right ears. Across both ears, the frequency-specific scores were strongly correlated at all analyzed frequencies and were better correlated at higher frequencies.

Figure 1 illustrates the frequency-specific distribution of error between PTA and app scores, stratified by the four apps (Figure 1A–D) and overall (Figure 1E). All apps tended to overestimate the severity of hearing loss, particularly at low frequencies. This disagreement was least pronounced for Android-2 (Figure 1D). The median of iOS-1 and iOS-2 were also within ± 10 dBHL of the gold standard at all five frequencies.

The percentage of results within ± 10 dB of PTA are presented in Table II. Overall, the ± 10 dB accuracy was between 54% and 66%. However, this was better for certain apps. For example, iOS-1 and Android-2 had approximately 70%–80% ± 10 dB accuracy, with the highest accuracy at the 2 kHz frequency.

Smartphone Audiometry Versus PTA: Pure Tone Average Results

Table III compares pure tone averages when contrasting app audiometry performed in optimal conditions in hospital versus home. There was a moderate correlation between app audiograms performed in hospital versus PTA (Pearson's CC 0.56). This improved to a rating of “strong” when patients performed the app at home (0.68).

Home hearing thresholds were stratified by app (Table III). Pure tone average thresholds of Android-2 showed a very strong correlation with the PTA (CC 0.83). iOS-1 and Android-1 showed strong correlation (0.77, 0.69, respectively); iOS-2 showed moderate correlation (0.47). The overall ±10 dB percentage accuracy was 63%, being highest for iOS-1 and Android-2 (74% and 75%, respectively).

The violin plot in Figure 2 depicts the pure tone average difference (i.e., the error between PTA and app), stratified by severity of hearing loss: normal, mild, and moderate or severe. Apps tended to overestimate thresholds in listeners without hearing loss.

Smartphone Audiometry in Optimal Noise Versus at Home

Smartphone results performed in optimal noise conditions (i.e., in hospital sound-treated rooms) versus at home were rated as either strong or very strong correlation at all frequencies (Table IV), and across all apps (Table V).

Test–Retest Reliability

Test–retest reliability was high when examining both the optimal noise-to-home condition (intraclass coefficient [ICC] 0.87) and the first to final readings at home (ICC 0.90).

Participant Reception

Forty-six participants (response rate 33%) submitted online feedback surveys; 26 (56.5%) were iOS users and 20 (43.5%) were Android users. The full results are available as Supplementary Material 3. Forty-two participants (91%) agreed/strongly agreed that the smartphone app was easy to use, and 40 (87%) agreed/strongly agreed that they would be happy to use an app to monitor their hearing in the future. Thirty-eight participants (83%) agreed/strongly agreed that they were able to find a suitably quiet place to use the app.

Test Accuracy and Repeatability

The accuracy of smartphone audiometry has previously been shown to be high, with pooled sensitivity and specificity for detecting mild hearing loss of 0.91 and 0.90 respectively, and slightly poorer performance for detecting moderate hearing loss (pooled sensitivity and specificity of 0.94 and 0.87 respectively).⁹ Our pure tone average accuracy (as measured by percentage within ±10 dB of the hospital PTA) was greater than 70% in three out of four apps. Not all apps performed as well as each other, with Android-2 performing the best overall.

All apps tended to overestimate the severity of hearing loss, particularly at lower frequencies. There are several possibilities for this including the impact of ambient noise (although a consistency of this finding when testing in sound-treated room does not support this) and poor calibration/quality of the earbuds. A useful feature for future developers to consider, would be providing inbuilt feedback to the user on the noise of their listening environment. Although perfect accuracy would be ideal, for the purpose of screening, overestimation may be better than underestimation, maximizing sensitivity.

While accuracy was moderate, precision was better, with high test–retest correlation between tests performed at home. This is important for disease monitoring, for example identifying fluctuations in Ménière's that may trigger the need for intervention, monitoring ototoxicity, or research exploring the trajectory of hearing recovery or decline.

We noted a tendency for thresholds on the right ear to be lower (better) than those on the left. The right ear was tested first in three apps (iOS-2, Android-1, Android-2) and so this may reflect fatigue or poorer concentration as testing progressed. Unlike an in-hospital PTA performed by an experienced audiologist, the smartphone apps will not detect poor concentration or inconsistencies in responses.

Effect of Test Environment

Previous work suggested that performance may be poorer in noisier environments, such as in a clinic waiting room¹³; although this finding has not been universal, and the use of noise-canceling headphones may mitigate some ambient noise effects.²⁰ Many apps designed for hearing screening, as opposed to PTA, use a digits-in-noise test (e.g., the World Health Organisation hearWHO app). This has the advantage of using signal-to-noise ratio as the measure of functional hearing, meaning ambient noise pollution is less of an issue. However, it does not provide the sensitivity or frequency-specific data of a PTA. This would be particularly of value where limited high or low-frequency losses may occur (e.g., ototoxicity or Ménière's, respectively).

A meta-analysis found performance outside an audiometric booth to be poorer,⁸ though all published studies not using a sound-treated room have still utilized a quiet school room, clinic room, or office for supervised testing. Surprisingly, we found that results obtained at home were more accurate than those in the sound-treated room. We attribute this to a learning effect, as the first time the participants used the app, they were alone in an audiometric booth. By contrast, home testing was repeated up to three times and always with prior experience. Questionnaire data showed that fewer than 10% of participants disagreed that it was easy to find a quiet place. Our findings therefore support the home environment as suitable, at least in relation to a typical UK home.

Our study involved issuing participants with standard inexpensive earbuds, costing approximately £2.00/pair. A well-fitting earbud may have some sound attenuation properties for noisier environments, with insert earphones found to be best for this.²¹ We chose to standardize earbuds to remove any bias that differing quality or frequency response between smartphones may introduce. App performance may have been better with higher-quality earbuds, as many apps are developed with a stock earbud in mind.

Clinical Application of Smartphone Tests

All apps were able to determine thresholds up to 8 kHz, and accurate high-frequency data adds additional value for ototoxicity monitoring, with daily or regular testing in at-risk individuals potentially allowing ototoxic drugs to be stopped or their dose reduced at an early stage. The ability to regularly test patients and remotely collect data has many possibilities for research, for example, enabling a better understanding of the time course of hearing recovery following idiopathic sudden sensorineural loss or decline in genetic conditions.

The apps' simplicity meant that participants did not struggle and gave positive feedback: 87% of participants agreeing or strongly agreeing that they would be happy to monitor their hearing at home. This fits with a wider trend toward consumer-led health monitoring and wearable health technologies. This can be invaluable to a clinician in observing fluctuating hearing and may also permit patients to take control over their management and seek treatment earlier for relapsing conditions, such as Ménière's or vestibular schwannoma, or to monitor improvement during treatment for sudden sensorineural hearing loss.

Study Limitations

We selected four apps across both common smartphone operating systems. For the purpose of feasibility and patient data security, our choice was limited to apps freely available and not requiring registration using patient identifiable data. We were unable to test the uHear app, which is most widely validated in the literature, as it was not available in UK. Other apps may vary in performance.

Patient age may impact app performance, with poorer results in children and older individuals.^{8, 22} While a strength of our work was the inclusion of participants with a broad age range, we did not test children. It may be that the performance of this group would be poorer.

Smartphones cannot fully replicate a PTA (lack of bone conduction to distinguish conductive loss), and there were no accommodations for masking; the latter being particularly relevant in the monitoring of patients with unilateral hearing loss.

Our results were more accurate at home. Future studies that wish to determine the accuracy of app results at home versus sound-treated rooms, could ensure all participants practice using the app multiple times before data collection is started. Ideally, they would be randomized to perform the test three times in each environment. Participants were limited to performing the app only once in the hospital to reduce their time burden in this voluntary study and due to the availability of sound-treated rooms.

Pure tone average usually does not include 8 kHz as its reliability can be poor compared to lower frequencies. However, 8 kHz appeared just as accurate in our data, and so was included due to its value in ototoxicity monitoring.