A critical readability and quality analysis of internet-based patient information on neck dissections
Abstract
Objective
Patients are increasingly turning to the Internet as a source of healthcare information. Given that neck dissection is a common procedure within the field of Otolaryngology - Head and Neck Surgery, the aim of this study was to evaluate the quality and readability of online patient education materials on neck dissection.
Methods
A Google search was performed using the term “neck dissection.” The first 10 pages of a Google search using the term “neck dissection” were analyzed. The DISCERN instrument was used to assess quality of information. Readability was calculated using the Flesch-Reading Ease, Flesch-Kincaid Grade Level, Gunning-Fog Index, Coleman-Liau Index, and Simple Measure of Gobbledygook Index.
Results
Thirty-one online patient education materials were included. Fifty-five percent (n = 17) of results originated from academic institutions or hospitals. The mean Flesch-Reading Ease score was 61.2 ± 11.9. Fifty-two percent (n = 16) of patient education materials had Flesch-Reading Ease scores above the recommended score of 65. The average reading grade level was 10.5 ± 2.1. The average total DISCERN score was 43.6 ± 10.1. Only 26% of patient education materials (PEMs) had DISCERN scores corresponding to a “good quality” rating. There was a significant positive correlation between DISCERN scores and both Flesch-Reading Ease scores and average reading grade level.
Conclusions
The majority of patient education materials were written above the recommended sixth-grade reading level and the quality of online information pertaining to neck dissections was found to be suboptimal. This research highlights the need for patient education materials regarding neck dissection that are high quality and easily understandable by patients.
Abbreviations
-
- CLI
-
- Coleman-Liau Index
-
- FKGL
-
- Flesch-Kincaid Grade Level
-
- FRE
-
- Flesch-Kincaid Reading Ease
-
- GFI
-
- Gunning-Fog Index
-
- OHNS
-
- Otolaryngology-Head & Neck Surgery
-
- PEMs
-
- Patient education materials
-
- SMOG
-
- Simple Measure of Gobbledygook Index
INTRODUCTION
Cervical lymph node metastases is one of the most important factors affecting prognosis in squamous cell carcinoma of the head and neck.1, 2 As a result, neck dissection is a common procedure that plays a critical role in the surgical management of several head and neck cancers.3, 4 Depending on the type of neoplasm, and extent of metastases, neck dissections can range from a “radical neck dissection,” which removes the sternocleidomastoid, internal jugular vein, and accessory nerve, to a “selective neck dissection,” where varying zones of cervical lymph nodes are selectively removed.5, 6 Potential postoperative complications and sequelae following neck dissection include dysfunction of the shoulder, neck, and upper limb, cervical anesthesia, and pain.5, 7-9 Studies evaluating patient-reported outcomes specific to neck dissections have demonstrated the potential negative impact this procedure can have on patients' quality of life.10 Patient education regarding the risks and benefits of neck dissection is paramount, given the anatomical complexity of the neck and the potential negative consequences of surgery.
Patients are increasingly turning to the Internet for health information.11 Approximately 54% of patients with head and neck cancer use the Internet to search for information about their treatments.12, 13 Thus, internet-based patient education materials (PEMs) are important resources for preoperative patient education. However, the extent to which these materials are helpful to patients can be limited by how easily understandable the material is to the intended audience. Health literacy, defined as an individual's capacity to understand and make use of basic health information, has a significant impact on patients' decision-making and is strongly associated with clinical outcomes.14, 15 An average American adult reads at approximately an eighth-grade reading level.16 Thus, guidelines by the American Medical Association recommend that online medical information be equal to, or less than, a sixth-grade reading level to accommodate individuals from varying educational backgrounds.16 However, the information on the Internet remains largely unregulated and the available online PEMs often vary in readability and quality.17 Moreover, many studies have demonstrated that online PEMs covering various Otolaryngology-Head & Neck Surgery (OHNS) procedures and conditions are lacking in readability and quality.18-22 It is important that the OHNS community is aware of the quality and readability of resources viewed by patients, as patient decisions regarding consent, compliance with treatment and post-operative outcomes can be influenced by how well they understand their diagnosis and treatment plan. Thus, the objective of this study was to evaluate the quality and readability of online PEMs regarding neck dissection.
METHODS
The aim of this study is to evaluate the quality and readability of online PEMs on neck dissection. A cross-sectional analysis of online information was conducted in Toronto, Canada, and all online PEMs were included without any restriction in terms of country of origin.
Data collection
The methodology used in this study is similar to another study conducted by our group (Grose E, BHSc, unpublished data, 2020). Research ethics board approval was not required due to the publicly available nature of this data. A GoogleTM search was performed on April 10, 2020 in Toronto, Canada using the search term “neck dissection” to identify online PEMs. The GoogleTM search engine was chosen as it is the most widely used in North America.23 The first 10 pages of this search were compiled. To prevent the search engine from showing personalized results, the browser was set to incognito mode. Furthermore, prior to the initiation of the search, all search history, cookies, and cached data were erased from the browser and location settings were disabled. The text for all included links was accessed on the same day that the search was conducted. Our exclusion criteria included links that were not written in English, those with access restrictions, scientific webpages, news articles, non-text media, sites that targeted medical professionals, webpages containing less than 100 words, and/or links that did not contain patient information pertaining to neck dissection.
Readability evaluation
All relevant text-based content from the included web pages was downloaded and converted into plain text using Microsoft Word document (Microsoft Corp). Any information deemed not relevant to patient education was removed (ie, formatting, disclaimers). This was done according to standards set by similar bodies of research to avoid skewing readability results.24, 25 The text from each website was evaluated for readability using an online, open access readability calculator (https://readable.com/) which performs the following validated readability instruments: Flesch-Kincaid Reading Ease (FRE), Flesch-Kincaid Grade Level (FKGL), Gunning-Fog Index (GFI), Coleman-Liau Index (CLI), and Simple Measure of Gobbledygook Index (SMOG). Each of these tests uses variables such as sentence length, number of words, and number of syllables to estimate readability of text.26-28 The FRE determines readability using a score of 0 to 100 with higher scores indicating a higher level of readability.29, 30 The optimal score for the FRE is 65.31 The FKGL, GFI, CLI, and SMOG scores report the reading grade level necessary to interpret the text (ie, a score of “6” corresponds to a sixth-grade reading level).29, 32-34 Table 1 describes the formula used to calculate each score. The online calculator (https://readable.com/) has been used in other peer-reviewed publications and uses the validated readability formulas in Table 1.35, 36 All these scores have previously been used to evaluate medical literature.22, 26, 37-40
| Assessment scale | Formula |
|---|---|
| FRE | FRE = 206.835 − (84.6 × average # of syllables per word) − (1.015 × average # of words per sentence) |
| FKGL | FKGL = (11.8 × average # of syllables per word) + (0.39 × average # of words per sentence) − 15.59 |
| GFI | GFI = 0.4 × [(average # of words per sentence) + (average number of words with 3 or more syllables ×100)] |
| CLI | CLI = (0.0588 × average # of letters per 100 words) − (0.296 × average # of sentences per 100 words) − 15.8 |
| SMOG | 3.1291 |
- FRE: Flesch-Kincaid Reading Ease; FKGL: Flesch-Kincaid Grade Level; GFI: Gunning-Fog Index; CLI: Coleman-Liau Index; SMOG: Simple Measure of Gobbledygook Index.
DISCERN grading
DISCERN is a tool designed to assess the reliability and quality of written material on treatment choices, for those with no prior medical knowledge.41 Each of the 16 questions is rated on a scale of 1 to 5. Questions 1 to 8 evaluate the reliability of the website and questions 9 to 15 address the quality of the website's content regarding treatment options. Question 16 is an overall global rating on a scale of 1 to 5. A rating of “1” indicates extensive shortcomings and a rating of “5” indicates minimal shortcomings. A higher cumulative DISCERN score indicates a higher quality of publication with a maximum score of 80 (Table 2). Each question in the DISCERN tool is listed in Table 3. DISCERN scores were completed independently by two reviewers, who were trained in using the DISCERN tool (J.H.K. and E.G.). Any disagreements were resolved by consensus.
| DISCERN score | Quality level |
|---|---|
| < 27 | Very poor quality |
| 27-38 | Poor quality |
| 39-50 | Fair quality |
| 51-62 | Good quality |
| > 62 | Excellent quality |
| Quality criterion | Mean score (standard deviation) |
|---|---|
| Section 1: Reliability | |
| 1. Are the aims clear? | 2.0 (1.4) |
| 2. Does it achieve its aims?a | 4.6 (0.5) |
| 3. Is it relevant? | 3.8 (1.2) |
| 4. Is it clear what sources of information were used to compile the publication (other than the author or producer)? | 1.6 (0.9) |
| 5. Is it clear when the information used or reported in the publication was produced? | 1.9 (1.3) |
| 6. Is it balanced and unbiased? | 2.5 (1.0) |
| 7. Does it provide details of additional sources of support and information? | 2.2 (1.3) |
| 8. Does it refer to areas of uncertainty? | 4.1 (1.2) |
| Total reliability score | 20.3 (6.5) |
| Section 2: Quality | |
| 9. Does it describe how each treatment works? | 4.1 (1.2) |
| 10.Does it describe the benefits of each treatment? | 3.4 (0.9) |
| 11. Does it describe the risks of each treatment? | 3.0 (1.4) |
| 12. Does it describe what would happen if no treatment is used? | 1.2 (0.9) |
| 13. Does it describe how the treatment choices affect overall quality of life? | 3.0 (1.6) |
| 14. Is it clear that there may be more than one possible treatment choice? | 3.4 (1.2) |
| 15. Does it provide support for shared decision-making? | 2.2 (1.5) |
| Total quality score | 23.3 (5.0) |
| 16. Overall Rating of Publication | 2.9 (0.8) |
| Total DISCERN scores | 43.6 (10.1) |
- a Question 2 was not rated if question 1 was rated as a “1.” Question 2 was only rated for 13 websites.
Data analysis
Categorical variables are reported using frequencies and proportions. Continuous variables are presented as means ± standard deviations. Since the first three search results receive more than half the user traffic, separate analyses were performed to compare the readability and quality of these resources to the other included PEMs.42 Unpaired two-tailed t-tests with unequal variances were used to compare the means. Pearson's correlation coefficients between the readability scores and the DISCERN scores were calculated. Inter-rater reliability was analyzed using the weighted kappa (κ) statistic. For all analyses, P < 0.05 was considered significant. Statistical analysis was performed using SPSS software, version 26.0 (IBM Corp).
RESULTS
PEM selection
Overall, 141 websites were retrieved from the Google search and 31 met the inclusion criteria for this study. The majority of the included online PEMs came from academic institutions and hospitals (17/31, 55%), with others coming from professional organizations (6/31, 19%), medical information websites (3/31, 10%), private clinics (3/31,10%), government websites (1/31, 3%), and miscellaneous sources (1/31, 3%). Reasons for exclusion were as follows: scientific journal articles (38/110, 35%), paid advertisements (38/110, 35%), websites targeting medical professionals (17/110, 15%), duplicates (6/110, 5%), websites without information pertaining to neck dissection surgery (3/110, 3%) (ie, websites about exercises to do after neck dissection), websites with access restrictions (3/110, 3%), websites with less than 100 words (2/110, 2%), error links (2/110, 2%), and non-text media (1/110, 1%).
Readability
The average FRE score for all 31 PEMs was 61.2 ± 11.9 with a range of 32.3 to 82.8. Fifty-two percent (16/31) had FRE scores above 65, representing an optimal readability score.31 The average reading grade level from the four different scores was 10.5 ± 2.1. The average grade level produced by the individual readability assessments were as follows: FKGL: 8.7 ± 2.3, GFI:11.6 ± 2.6, CLI: 9.9 ± 1.8, and SMOG:11.7 ± 1.9. The proportion of PEMs with a reading grade level below the eighth-grade was 42% (13/31) according to the FKGL, 7% (2/31) with GFI, 13% (4/31) with CLI, and 0% using SMOG. There was no statistically significant difference between the mean readability scores of the first three search results and the other search results (FRE: 55.0 ± 10.6 vs. 61.8 ± 12.0, P = 0.38, FKGL: 10.0 ± 2.1 vs. 8.6 ± 2.3, P = 0.35, CLI: 11.0 ± 1.6 vs. 9.8 ± 1.8, P = 0.32, GFI: 13.3 ± 2.6 vs. 11.4 ± 2.5, P = 0.34, SMOG: 12.9 ± 1.9 vs. 11.5 ± 1.9, P = 0.34). Table 4 shows the first three search results and their respective readability scores.
| Website name | FRE | FKGL | GFI | CLI | SMOG |
|---|---|---|---|---|---|
| American Head & Neck Society | 47.4 | 11.2 | 14.6 | 12.5 | 13.8 |
| Canadian Cancer Society | 67.1 | 7.6 | 10.3 | 9.3 | 10.7 |
| Thyroid Head and Neck Cancer Foundation | 50.4 | 11.3 | 15.1 | 11.1 | 14.2 |
- FRE: Flesch-Kincaid Reading Ease; FKGL: Flesch-Kincaid Grade Level; GFI: Gunning-Fog Index; CLI: Coleman-Liau Index; SMOG: Simple Measure of Gobbledygook Index.
DISCERN scores
The DISCERN scores ranged from 26 to 60 with 3% (1/31), 29% (9/31), 42% (13/31), 26% (8/31), and 0% of PEMs having scores corresponding to “very poor quality,” “poor quality,” “fair quality,” “good quality,” and “excellent quality,” respectively. The mean score for each question in the DISCERN tool is displayed in Table 3. The weighted κ statistic for total DISCERN scores was 0.75 indicating a substantial agreement between raters. There was no significant difference in the mean total DISCERN scores between the first three search results and the other search results (46.0 ± 3.6 vs. 43.3 ± 10.5, P = 0.38). There was a significant correlation between the DISCERN scores and FRE scores (r = 0.49, P = 0.005) and between the DISCERN scores and average grade level (r = −0.50, P = 0.004).
DISCUSSION
Patients with head and neck cancer typically have complex treatment plans consisting of a combination of surgery, chemotherapy, and/or radiation therapy. A common component of their surgical treatment plan involves a neck dissection. Educational material can provide patients with the information necessary to engage in discussions about their treatment plans with their healthcare providers and better understand the inherent risks and benefits of treatment. Although neck dissection is an essential facet of many head and neck oncologic procedures, it is also associated with significant morbidity and a reduction in health-related quality of life.8 Thus, providing appropriate PEMs can improve communication in informed consent discussions and facilitate a shared decision making process.43 Furthermore, it is well established that robust patient education can improve preoperative anxiety, postoperative satisfaction, health status, and in some cases, lead to cost savings.44, 45 It is estimated that 54% of patients with head and neck cancer use the Internet as a source of health information, thus further necessitating the need for robust PEMs to improve patient satisfaction and outcomes.12, 46-48 The aim of this study was to assess the quality and readability of online neck dissection PEMs, as patients are likely to turn to these supplementary sources for information about this procedure.
While reading grade level is a critical component of health literacy, studies show that Americans tend to read at least three grades lower than their education level, with an average American reading at an eighth-grade level.16, 49 Of the PEMs included in this study, over 50% were written at a grade level above the eighth-grade, irrespective of the readability index used. Furthermore, 48% (15/31) of PEMs had FRE scores below the recommended score of 65.31 Other studies assessing the readability of patient information regarding OHNS procedures and conditions have also indicated that the majority of PEMs exceed the reading level of both the average American adult and the recommended sixth-grade reading level for patient health information.21, 24, 39, 40, 50-52 Within the realm of OHNS, the readability of online information on head and neck cancer using the SMOG was found to be 10.4, which is lower than the mean SMOG score found in the current study of 11.7.50 Evaluating readability with several different scores has been recommended as each score is calculated using different variables.22 A previous study of online information on laryngectomy used five different scores to determine reading grade level and found an average reading grade level of 10.4.39, 50 Similarly, this study also found an average reading grade level of 10.5 suggesting that information available on several head and neck cancer surgeries are above the recommended levels. This is particularly important to consider in the head and neck cancer patient population, as studies have shown that 12% of this patient population has reduced health literacy.53, 54
This study also used the DISCERN score to assess the quality of the PEMs on neck dissection. The mean total DISCERN score was 43.6 out of a maximum of 80 which corresponds to “fair quality.” However, when looking at the distribution of total DISCERN scores, 32% (10/31) of studies had DISCERN scores corresponding to “poor quality” or “very poor quality.” The average global rating (question 16) for the PEMs was 2.9 out of a maximum of 5 indicating the quality and reliability of these PEMs may be suboptimal. This result is comparable to other studies assessing online patient resources for head and neck cancer.50, 55 When considering the individual items assessed in the DISCERN instrument, the PEMs received the lowest score, with a mean score of 1.2 out of 5, on question 12, “does it describe what would happen if no treatment is used?” Omission of this information from PEMs may bias decision-making towards treatment, negatively impacting the informed consent process. The second lowest scoring item, with a mean score of 1.6 out of 5, was question 4, “is it clear what sources of information were used to compile the publication (other than the author or producer)?” This raises concerns about the reliability, potential bias, and accuracy of the information contained within these resources.
While the first three Google search results receive more than half the user traffic, the first three PEMs from this study were of “fair quality”. There were also no significant differences in DISCERN scores or readability of the first three PEMs when compared to the rest of the PEMs. The average reading grade levels of these three sources were 13.0, 9.5, and 12.9 in the order of the search results, which exceed the sixth-grade reading level recommended by the American Medical Association and the eighth-grade reading level of the average American adult.16 Furthermore, only one of the first three search results from the Canadian Cancer Society had a reading grade level below the average American eighth-grade reading level according to only one of the readability scores.16 This demonstrates that the most commonly accessed PEMs for neck dissection remain limited in the quality and readability of the information they provide patients.
This study also explored the relationship between readability and DISCERN scores of PEMs on neck dissection. Previous studies assessing the readability of online PEMs for various OHNS conditions found no statistically significant or a very weak correlation between the readability scores and DISCERN scores.19, 52, 56 Similarly, our group has conducted an analysis of the readability and quality of information on parotidectomy and also found no significant correlation between readability and quality (Grose E, BHSc, unpublished data, 2020). Interestingly, this study found a moderately strong, statistically significant correlation between DISCERN scores and FRE, as well as the average reading grade level. Our findings demonstrate that PEMs on neck dissection that are higher quality and more reliable are also more likely to be readable.
Although this study was novel in its analyses of online PEMs related to neck dissection, this study has some important limitations. Primarily, the readability instruments that are used for the analyses are not sufficient to evaluate all aspects of readability in isolation. For example, while FRE measures certain attributes of words and sentences, SMOG takes into account other attributes, thus capturing different variables. The search strategy in this study is also limited in that it is not possible to perfectly replicate a patient's Google search and to predict the resources that will be chosen for health information. Furthermore, several different terminologies can be used to refer to neck dissections, such as “cervical lymphadenectomy,” which were not accounted for in this study. While scientific journal articles and non-text media such as videos and figures were excluded from the analyses, some patients may turn to these resources for information about neck dissection. Patients may also use physical PEMs, such as pamphlets and brochures provided by their medical professionals. Additionally, patients may read beyond the first 10 pages of their Google search or use different search engines such as Bing or Yahoo. Articles that would be found using these alternative search strategies were not captured in this study. Finally, it should be acknowledged that despite two raters using the DISCERN tool, it is possible for human biases to influence the scores. This must be accounted for when interpreting the results of the DISCERN tool.
CONCLUSIONS
The findings of this study suggest that online PEMs related to neck dissection remain limited. The reading level of current online PEMs often exceeds the level recommended by the American Medical Association. Furthermore, several deficiencies were identified in the quality of these resources. This may lead to patients misunderstanding or misinterpreting crucial information, which can compromise the informed decision-making process. Improving the readability and quality of online PEMs may help patient comprehension, thereby facilitating patient-centered treatment decisions. Clinicians and health care providers should be aware of the adequacy and limitations of online PEMs pertaining to neck dissection and consider supplementing this information with comprehensive in-person counseling. Ensuring these resources are high-quality and easily comprehensible has the potential to improve shared decision making, patient satisfaction, and postoperative outcomes, all of which are important facets of patient-centered healthcare delivery.
ACKNOWLEDGMENTS
Not applicable.
CONFLICTS OF INTEREST
The authors declare that they have no competing interests.
AVAILABILITY OF DATA AND MATERIALS
The generated and analysed during the current study are available from the corresponding author on reasonable request.
AUTHORS CONTRIBUTIONS
EG participated in the design of the study, carried out the study, analyzed the dataset, and drafted the manuscript. JHK carried out the study, acquired the dataset, and drafted the manuscript. JP, ML, JWL, and EM revised the manuscript. All authors read and approved the final manuscript and agree to be accountable for the author's own contributions.
