This paper is in the following e-collection/theme issue:

Information Quality in Digital Media (317) eHealth Literacy / Digital Literacy (352) Quality/Credibility of eHealth Information and Trust Issues (486)

Published on in Vol 4, No 4 (2015): Oct-Dec

Third Molars on the Internet: A Guide for Assessing Information Quality and Readability

Third Molars on the Internet: A Guide for Assessing Information Quality and Readability

Third Molars on the Internet: A Guide for Assessing Information Quality and Readability

Authors of this article:

Kamal Hanna1 Author Orcid Image ;   David Brennan2 Author Orcid Image ;   Paul Sambrook3 Author Orcid Image ;   Jason Armfield4 Author Orcid Image
Article Authors Cited by (15) Tweetations (5) Metrics

    Original Paper

    1PhD Candidate, Australian Research Centre for Population Oral Health (ARCPOH), School of Dentistry, The University of Adelaide, Adelaide, South Australia, Australia

    2Professor, Australian Research Centre for Population Oral Health (ARCPOH), School of Dentistry, The University of Adelaide, Adelaide, South Australia, Australia

    3Consultant OMFS, Department of Oral and Maxillofacial Surgery, Adelaide Dental Hospital, The University of Adelaide, Adelaide, South Australia, Australia

    4Associate Professor, Australian Research Centre for Population Oral Health (ARCPOH), School of Dentistry, The University of Adelaide, Adelaide, South Australia, Australia

    Corresponding Author:

    Kamal Hanna, BDS, PGDip (Laser), MSc (Oral Surgery), MRCS(Ed), Egypt Board OMFS

    PhD Candidate, Australian Research Centre for Population Oral Health (ARCPOH)

    School of Dentistry

    The University of Adelaide

    ARCPOH

    122 Frome Street

    Adelaide, South Australia, 5005

    Australia

    Phone: 61 883135626

    Fax:61 883133070

    Email: kamal.hanna@adelaide.edu.au


    Background: Directing patients suffering from third molars (TMs) problems to high-quality online information is not only medically important, but also could enable better engagement in shared decision making.

    Objectives: This study aimed to develop a scale that measures the scientific information quality (SIQ) for online information concerning wisdom tooth problems and to conduct a quality evaluation for online TMs resources. In addition, the study evaluated whether a specific piece of readability software (Readability Studio Professional 2012) might be reliable in measuring information comprehension, and explored predictors for the SIQ Scale.

    Methods: A cross-sectional sample of websites was retrieved using certain keywords and phrases such as “impacted wisdom tooth problems” using 3 popular search engines. The retrieved websites (n=150) were filtered. The retained 50 websites were evaluated to assess their characteristics, usability, accessibility, trust, readability, SIQ, and their credibility using DISCERN and Health on the Net Code (HoNCode).

    Results: Websites’ mean scale scores varied significantly across website affiliation groups such as governmental, commercial, and treatment provider bodies. The SIQ Scale had a good internal consistency (alpha=.85) and was significantly correlated with DISCERN (r=.82, P<.01) and HoNCode (r=.38, P<.01). Less than 25% of websites had SIQ scores above 75%. The mean readability grade (10.3, SD 1.9) was above the recommended level, and was significantly correlated with the Scientific Information Comprehension Scale (r=.45. P<.01), which provides evidence for convergent validity. Website affiliation and DISCERN were significantly associated with SIQ (P<.01) and explained 76% of the SIQ variance.

    Conclusion: The developed SIQ Scale was found to demonstrate reliability and initial validity. Website affiliation, DISCERN, and HoNCode were significant predictors for the quality of scientific information. The Readability Studio software estimates were associated with scientific information comprehensiveness measures.

    Interact J Med Res 2015;4(4):e19

    doi:10.2196/ijmr.4712

    Keywords

    DISCERN; health information online; Health on the Net Code; readability; Scientific Information Quality Scale; website affiliation; website content analysis; wisdom teeth 


    Wisdom teeth removal is the most commonly performed oral surgical procedure [1]. In addition to patients needing to make a decision regarding whether or not to remove asymptomatic wisdom teeth [2,3], other decisions need to be made regarding anesthetics options, treatment pathways and associated costs, operation timing, and expected recovery [1]. Patients who undergo third molars (TMs) extraction prefer to receive detailed procedural information [4]. Providing those patients with detailed high-quality information is not only medically and legally important in making an informed decision, but also might improve their participation in the process of shared clinical decision making. This might, in turn, improve patient satisfaction and treatment outcomes [5].

    It is not always possible to provide adequate information for patients suffering from TMs problems, because it might be limited by the available consultation time allocated to each patient, given the fact that clinics are often overbooked [6]. The busy nature of oral surgery clinics may hinder surgeons from adequately explaining the provided information, a finding suggested by Ferrús-Torres et al [7]. Lack of sufficient information from professional sources and limitations of information leaflets [8] can result in patients seeking online sources to satisfy their information demands and often before consultation [9]. While the Internet plays an increasing role in dental patient education [10], the quality of online health information varies significantly across websites [11,12]. Therefore, it is argued that clinicians should guide their patients to credible online health resources.

    There can be a potential limitation in the current clinical practice in referring patients to high-quality Internet resources due to clinicians’ lack of time and/or lack of knowledge [13,14]. In addition, the lack of dentists’ ability to discuss the retrieved conflicting Internet-related information with their patients may affect the patient-dentist relationship [10]. To provide patients with guidance in navigating the Internet, clinicians could use the findings from website content analysis studies. However, only a small number of dentally related studies exist and none have covered wisdom tooth problems. The lack of content analysis studies means the absence of an evidence base with which clinicians might be able to guide their patients to credible Internet-based resources. Furthermore, identifying predictors for scientific information quality (SIQ) could make the process of identifying high-quality online resources easier and less time consuming. However, clinicians also need to ensure that the high-quality Internet resources they identify are understandable by their patients.

    Understanding health information is a major domain in health literacy, allowing patients to make appropriate health-related decisions [15]. Patients with higher levels of health literacy have been found to have a better oral health status [16]. To ensure that consumer health information is understandable by the average patient, some health authorities require this information to be at Grade 8 reading level or less (13-14 years of age) [17]. Readability grades are calculated using different readability formulas [18] and are mainly based on word/sentence length and number of syllables per word. These provide a reading grade in relation to the US schooling system, which is set as a reference for readability grading. However, it would be useful to know which of these formulas has the highest association with information comprehensibility. A number of software applications and websites provide a readability-grade estimate for digital documents. Among these software applications, Readability Studio Professional 2012 [19] has been used in some studies [20,21] to calculate readability grades using different formulas. However, readability-grade estimates produced by Readability Studio software need to be assessed for their validity to measure information comprehensibility.

    The aims of this study were to (1) develop and validate a scale that measures SIQ; (2) evaluate the quality and readability of online health information concerning TMs problems; (3) validate the Readability Studio Professional 2012 software for measuring comprehensibility of online information; and (4) explore factors that could predict the SIQ of online health information.


    Website Sampling and Filtering

    To identify high-quality online resources, a cross-sectional sample of websites was selected on October 14, 2013, using advanced search options in Google, Yahoo!, and Bing search engines, with output limited to English language, any location, and specific phrases in the page title. The 3 phrases used were “wisdom tooth removal” OR “wisdom tooth extraction” OR “impacted wisdom tooth problems.” The first 50 results of each search engine output were selected after excluding websites identified as advertisements. A total of 150 websites were initially included. Websites were then filtered by removing duplicates and were reviewed for their relevance as a source for patient information. During this stage, nonfunctional, nonrelevant news articles or blogs were excluded. If a website was found to be relevant, it was categorized as having high, medium, or low relevance based on reporting the predetermined information sections of the SIQ Scale. Only websites of high relevance, according to this classification, were selected for content analysis. Figure 1 shows the flowchart for website sampling and filtering. This review was conducted by KH for consistency and eliminating the need for providing training.

    ‎
    Figure 1. Website content analysis flowchart.
    View this figure

    Characteristics of Website

    A number of website characteristics that might have an association with the quality of provided information were determined. Website affiliation (to which organization the website belongs to) was recorded as governmental, educational institute, treatment provider (hospital/medical or dental practice), nonprofit organization, commercial [22], or other group. There was an open section that was then coded into blogs, hub pages, wiki (like Wikipedia), or news. As content editing could play a role in information quality, websites were categorized into either “open access” or “open content” where the type of content editing was reported for coding. Information delivery format was recorded, as previous research shows the importance of multimedia use to engage patients of low literacy [23]. Information formats were recorded as a multiple response set that included “text within the webpage,” “word/PDF,” “images,” “cartoon animations,” “audio,” “real example,” and “other,” with an open section to enable adding comments, which later were coded into themes. Information communication method was recorded to identify the prevalence of each method. Information communication methods were recorded as a multiple response set that included “fact sheet,” “question and answer (Q&A),” “story,” and “other” with an open section that was coded into other types of information formats.

    Website Evaluation

    To assess different quality aspects of websites under evaluation, several scales were used [24]. Quality aspects included scales assessing usability, accessibility, trustworthiness, readability grade [18], scientific information comprehensiveness, scientific information reporting, scientific information referencing, SIQ, and online health information credibility (Health on the Net [25] and DISCERN [26]). The sum of these scales formed the website total score, which was considered as a collective measure of website quality.

    Usability Scale

    The Usability Scale consisted of items that were partially based on the Minervation Tool (LIDA [27]). These items include registration/subscription to review the information, website navigability, and search ability and were given a score that ranged from 1 to 3 for each item based on the response. For Web 2.0 applications support such as Facebook, Twitter, LinkedIn, and G+, responses were collected as a multiple response set to provide a score that was then trichotomized based on percentile distribution. Usability Scale scores range from 4 to 12.

    Accessibility Scale

    A single-item Binary Scale that was used as a proxy for compliance with Web Content Accessibility Guidelines [28], with scores being 1 for “No” and 3 for “Yes” to increase item weight in the website total score.

    Trust Scale

    The Trust Scale was developed for this study and comprised a 4-item Binary Response Scale measuring trust in a website. Items for this scale were display of the Health on the Net (HoN) seal [25], as it is the most frequently used online consumer health information quality seal, validity of the HoN seal using the HoN toolbar, display of other quality seals, and display of planned review date as an indication for maintaining information currency. Items for this scale were scored 1 for “No” and 3 for “Yes” to increase the scale weight within the website total score. The scale scores range from 4 to 12.

    Mean Readability Grades

    Mean readability grades were computed using Readability Studio Professional 2012 that provides readability grades estimates based on 6 different formulas recommended for the health care industry, which are FORCAST, Fry, Gunning Fog, New Fog, Raygor Estimate, and SMOG. Text from websites was extracted to MS Word (Microsoft, Redmond, WA, USA) where they were prepared for evaluation by the software. In addition, videos were transcribed by the author (KH).

    Scientific Information Comprehension Scale

    The Scientific Information Comprehension (SI Comprehension) Scale was developed specifically for this study. It comprises a 9-item scale that measures the understandability of each section of the scientific information shown in Table 1. Items for this scale were scored on a 5-point Likert-like scale ranging from 1 “difficult to understand” to 5 “easy to understand.” If the item did not exist on the website, it was reported as missing.

    Table 1. Evaluation criteria for scientific information quality concerning wisdom tooth problems.
    Criteria (assessed on a 5-point Likert-like scale: range from 1 for poor to 5 for excellent)Description
    OverviewNumber of third molars (TMs), age of eruption, and etiology of impaction
    PresentationMild pericoronitis to severe infection of facial spaces, swelling, trismus, periodontitis, decay, cyst, or tuners with incidence. No evidence supports the association between TMs and late teen crowding.
    Diagnosis and investigationsDiagnosed by a dentist/oral surgeon, medical and dental history, clinical and radiographic examination, and other radiographs in high-risk TMs
    Treatment optionsRetain functional TMs, symptomatic TMs with untreatable conditions or associated with pathology should be removed, no evidence supports the removal of asymptomatic impacted TMs, shared decision making. Anesthetic options (local anesthetic, intravenous sedation, or general anesthetic). Pathway (minor oral surgery, hospital day case, or hospital inpatient).
    Risk and benefitsIncidence of risks associated with retaining TMs, general surgical risks (pain, bleeding, swelling, etc), anatomical-related risks (numbness of lip or tongue, oroantral fistula), rare risks (tuberosity/mandible fracture)
    Surgical procedureDraping, anesthesia, flap, bone removal, tooth sectioning, tooth removal, socket irrigation, socket inspection, bone filing, suture, and gauze pack
    Postoperative care and recoveryPostoperative instructions, how to control pain, bleeding, swelling, infection, and dry/infected socket. Information about diet and oral hygiene. Expected recovery.
    Costs associated with the treatmentDepend on pathway: direct cost (surgeon, anesthetize, and/or hospital fees), indirect cost (time off work), insurance information
    More information for intravenous sedation and general anesthesia/dental anxiety managementConscious sedation (oral, inhalation and intravenous sedation), general anesthetic
    Scientific Information Reporting Scale

    The Scientific Information Reporting (SI Reporting) Scale is a 9-item binary scale that was developed for this study based on reporting information topics, which can be found on the assessed website. Items for this scale were scored 1 if the information section was covered and 0 if the information section was not covered in the examined website. Full scale scores range from 0 to 9. The SI Reporting Scale was used to identify websites of high relevance as a source of information.

    Scientific Information Referencing Scale

    The Scientific Information Referencing (SI Referencing) Scale is a 9-item Binary Scale that was developed for this study to measure referencing different information sections on the assessed website. Items for this scale were scored 1 if the information section was referenced and 0 if the information section was not referenced, and the full scale scores again range from 0 to 9.

    Scientific Information Quality Scale

    The SIQ Scale is a 9-item Likert-like scale, which was developed to assess various aspects of information that should be provided to patients, based on literature review and authors’ experience in the field (Table 1). Each item was scored on a scale ranging from 1 “poor” to 5 “excellent” against the predetermined criteria for online information concerning TMs problems that was created by this study’s authors using the best available evidence (see Multimedia Appendix 1). Domains for this scale included overview (introduction), presentation, diagnosis, treatment options, risks/benefits, procedural information, postoperative care and recovery, costs, and more information about anxiety control. If the item did not exist on the website, it was reported as missing. The SIQ Scale scores range from 9 to 45.

    Online Consumer Health Information Credibility Tools
    Health on the Net Code Scale

    This was a 14-item scale that was developed by authors [24] based on the criteria for providing the HoN seal [25]. Each item had 3 response options, namely, the website was “not complying” with Health on the Net Code (HoNCode) (scored 1), the website was “partially complying” with HoNCode (scored 2), and the website was “fully complying” with HoNCode (scored 3). The HoNCode consists of the following 8 principles: authorship, complementary information, maintaining privacy, appropriate referencing of information sources, claim policy, transparency, disclose funding source, and clear advertising policy. The HoNCode Scale scores range from 14 to 42.

    DISCERN Scale

    This is a 16-item scale developed by Charnock [26] to assess the credibility of printed consumer health information and was validated for assessment of online consumer health information [29]. Each item was scored 1 for a “definitive no,” 2-4 for “partial yes” (based on reviewer’s judgment), or 5 for a “definitive yes.” The DISCERN items are grouped into 3 main groups: Questions 1-8 are related to reliability of information, Questions 9-15 are related to specific treatment choices, and Question 16 provides an overall quality assessment of the information. The DISCERN Scale scores range from 16 to 80.

    Website Total Score

    The website total score was used as a measure of the total website quality. It was calculated as an unweighted sum of website usability, trust, SIQ, scientific information comprehensiveness, scientific information referencing, scientific information reporting, accessibility, DISCERN, HoNCode Scales, and the reverse-coded mean readability grade. The website total scores range from 57 to 222.

    Reviewer’s Comments

    To allow the evaluator (KH) to provide qualitative feedback on the assessed websites, the researcher commented on areas of biased/unbalanced information. In addition, the researcher commented on factors that might affect information readability and the recommended treatment options. These comments were then coded into themes and subthemes for analysis.

    Data Analysis

    Data were analyzed using IBM SPSS Statistics for Windows version 22.0 (IBM, NY, USA) [30]. Frequencies of websites characteristics were calculated. Means, SDs, and quartile distributions were also calculated for each scale. The internal consistency using Cronbach alpha of each scale was calculated. Pearson r correlation coefficients were calculated between SIQ Scale, DISERN, and HoNCode. In addition, Pearson r correlation between the mean readability grade and the reverse-coded SI Comprehension Scale was measured in an attempt to establish convergent validity. The associations between website affiliation and websites scale scores were tested using one-way analysis of variance (one-way ANOVA) with Tukey post hoc tests.

    To explore predictors for SIQ scores, linear regression was performed after creating dummy variables for website affiliation groups. A block of website affiliation dummies (Model 1) was entered in linear regression, where the “other” group was used as a reference category. In Model 2, DISCERN was added, and in Model 3, DISCERN was removed and replaced by the HoNCode score while statistically controlling for website affiliation. Websites were ranked according to their SIQ score and to their total (unweighted) score. The correlation between the 2 ranking orders was examined using Spearman ranking correlation.

    The website reviewer’s (KH) comments were analyzed using NVivo 10 [31] where comments were coded into themes and subthemes. These themes included biased/unbalanced information (subthemes included areas of biased/unbalanced information), factors affecting information readability (subthemes included repetition, terminologies use, image labeling), and the recommended treatment options (subthemes included obtaining a second opinion, prophylactic removal of all TMs, removal of only symptomatic ones, removal of symptomatic, and seriously think about asymptomatic ones). Cross-tabulation of codes’ frequency by the website affiliation group was obtained for unbalanced/biased information, the recommended treatment and factors affecting information readability, and then weighted according to the percentage of representation of the website affiliation group within the sample.


    Websites Characteristics and Their Usability

    Of the 50 websites available for content analysis, a majority of the reviewed websites (54%, 27/50) were related to a treatment provider after adding 1 website to this group from the “educational institute” group that has a teaching hospital attached to it. A total of 7 of the 50 (14%) websites were related to commercial websites, and governmental and nonprofit organizations websites were equally represented (8%, 4/50). There were 7 “other” group websites (hub pages, blogs, news, and wiki, 14%). A combination of text and image was the most commonly used information format (40%, 20/50). Question and answer was the most predominant information communication method either alone (34%, 17/50) or in combination with fact sheets (22%, 11/50).

    Most websites were open access (74%, 37/50), and the most common form of content editing was posting comments (14%, 7/50). All websites were accessible without either registration or subscription. A majority of websites were judged easy to navigate (62%, 31/50) while slightly above half of the websites (52%, 26/50) had no search facility. Facebook (23% of Web 2.0 applications, 28/121) and Twitter (20% of Web 2.0 applications, 24/121) were the most commonly used Web 2.0 applications.

    Scientific Information Quality

    The developed SIQ Scale had good internal consistency (Cronbach alpha=.85). Furthermore, the SIQ scores were significantly correlated with DISCERN scores (r=.81, P<.01) and HoNCode (r=.38, P<.01). Less than 25% of the evaluated websites had SIQ scores above 75% of the maximum scale score. The overview section was the most reported information section, whereas the cost information section was the least reported.

    Information Credibility Tools

    DISCERN had high internal consistency (Cronbach alpha=.91), whereas that for HoNCode was slightly lower (Cronbach alpha=.80). DISCERN and HoNCode were significantly correlated with each other (r=.71, P<.01) and both scales were significantly correlated with the SIQ Scale (as mentioned earlier).

    Association of Website Affiliation With Website Scores

    One-Way ANOVA showed a significant association between website affiliation and SIQ (F4,45=4.8, P<.01), DISCERN scores (F4,45=4.8, P<.01), and HoNCode score (F4,45=8.8, P<.01). SIQ had an observed power of 90% or over for each of them and had moderate effect size estimates. Website affiliation was also significantly associated with the other scales (Usability, Trust, SI Referencing, and SI Comprehension) except for the mean readability grade where no significant difference was found. Tukey post hoc tests showed that the SIQ mean scores of the “other” website affiliation group was significantly lower than commercial websites (P<.01) and governmental website (P=.01). Table 2 shows the significant association of websites scales mean scores and total score with website affiliation groups.

    Table 2. Quality and readability scores by website affiliation.

    		Website affiliationa
    CommercialTreatment providerGovernmentNonprofit organizationOther
    MeanSDMeanSDMeanSDMeanSDMeanSDP-value significance
    Scientific Information Quality (SIQ)32.6a9.526.0a7.834.8a10.929.0a9.817.0b7.3.01
    DISCERN62.3a9.346.0b10.759.3a17.661.8a12.144.1b13.5<.01
    Health on the Net Code35.4a4.326.8b3.735.3a2.535.8a4.930.7a7.8<.01
    Mean readability grade10.3a0.910.6a1.410.4a3.711.3a2.99.0a2.7.34
    SI Comprehension9.9a,c1.312.8b2.69.8a,b,c2.213.5a,b7.68.3c4.5.01
    SI Referencing10.6a2.49.1b0.59.8a,b0.511.8a3.110.1a,b2.3.01
    Trust5.0a1.44.1b0.44.8a,b1.04.5a,b1.04.0b0.0.03
    Usability10.3a1.48.4b1.110.5a0.69.8a1.310.4a1.4<.01
    SI Reporting16.9a1.616.2a1.417.0a2.016.3a2.213.9b2.1.01
    Accessibility1.0a0.01.1a3.01.5b0.61.0a0.01.0a0.0.02
    Total score153.8a26.1114.3b22.3152.6a29.9144.9a26.9114.0b25.4.02

    aValues in the same row and subtable not sharing the same subscript are significantly different at P<.05 in the two-sided test of equality for column means. Cells with no subscript are not included in the test. Tests assume equal variances.

    Predictors for Scientific Information Quality

    Linear regression models (Table 3) showed that website affiliation alone (Model 1) significantly explained 21% of the adjusted R2 of SIQ scores. Governmental websites had the highest (B) coefficients (B=17.75, P<.01) in comparison to the “other” group that was set as a reference category. After controlling for website affiliation (Model 2), DISCERN scores were found to be significantly associated with the highest SIQ (B=.60, P<.01). Because DISCERN and HoNCode are measuring a close construct, DISCERN was removed from the regression equation and replaced by HoNCode in Model 3. While controlling for website affiliation, HoNCode was found to significantly predict the SIQ (B=.63, P=.02). A regression residual scatter plot showed a random distribution while the P-P plot of the observed and the predicted values of the SIQ scores showed a good model fit (data not presented).

    Table 3. Scientific Information Quality score prediction models.a
    ModelUnstandardized coefficients95% CI for BStandardized coefficientstSignificance (P value)Model summary
    BStandard errorLower boundUpper boundBetaAdjusted R2R2 changeSignificance F change
    1Constant17.003.1610.6423.36
    		5.38<.01.21.28b.01
    Commercial15.574.476.5824.57.583.49<.01
    Treatment provider8.963.531.8516.08.482.54.02
    Governmental17.755.247.2028.30.523.39<.01
    Nonprofit organization12.005.241.4522.55.352.29.03
    2Constant-9.503.16-15.87-3.14
    		-3.01<.01.76.51c<.01
    Commercial4.682.71-0.7710.13.171.73.09
    Treatment provider7.831.963.8711.79.423.99<.01
    Governmental8.683.052.5514.82.252.85.01
    Nonprofit organization1.433.09-4.807.66.040.46.65
    DISCERN0.600.060.480.72.8510.09<.01
    3Constant-2.458.49-19.5614.67
    		-0.29.76.29.09d.02
    Commercial12.594.413.7021.47.472.85.01
    Treatment provider11.433.504.3818.48.613.27<.01
    Governmental14.885.114.5925.17.432.91.01
    Nonprofit organization8.815.14-1.5419.16.261.72.09
    Health on the Net Code Scale0.630.260.111.16.392.45.02

    aThe “other” website affiliation group was used as a reference category.

    bR2change for Model 1: It is change from a null model.

    cR2change for Model 2: It is a change from Model 1.

    dR2change for Model 3: It is a change from Model 1.

    Information Readability Grades and Comprehension

    The mean (SD) readability grade (Figure 2) was 10.3 (1.9). Nonprofit organization websites had the highest mean readability grade, whereas the “other” websites had the lowest mean readability grade (Table 2). There was no significant difference in the mean readability grade among website affiliation groups. One-way ANOVA of readability grade estimates (FORCAST, Fry, Gunning Fog, New Fog Count, Raygor Estimate, and SMOG) grouped by website affiliation showed no significant difference except for FORCAST, which was found to be significantly different (F4,45=3.2, P=.02). Figure 2 shows box plots of 6 different readability grades and the mean readability grade calculated using Readability Studio. After reverse coding of the Scientific Information Comprehension Scale scores, the New FOG readability grade has the highest significant association with it among the used readability formulas (r=.48, p<.01). In addition, the reverse-coded SI Comprehension Scale scores were found to be significantly correlated with the mean readability grade produced by Readability Studio Professional 2012 (r=.45, P<.01).

    ‎
    Figure 2. Box plot of readability grades and mean readability grade.
    View this figure

    Websites’ Ranking

    Websites were ranked according to their SIQ scores. Results showed that the Bupa-UK website [32] had the highest SIQ, followed by that of the NHS-UK [33]. Ranking websites according to their total unweighted scoring showed that Bupa-UK had the highest total score followed by Animated-Teeth [34]. Spearman ranking correlation between both ranking orders were significantly correlated (r=.81, P<.01).

    Analysis of Reviewer’s Comments

    The comment section was analyzed using thematic analysis. Biased or unbalanced information provided was coded. TMs and late teen crowding were the most frequently reported biased information (41% of reported biased/unbalanced information, 13/31). Forcing patients to undergo “sleep dentistry” (16% of reported biased/unbalanced information, 5/31) was an example of unbalanced information, where information providers limited the anesthetic options to general anesthesia or sedation without providing local anesthetic as an option. The treatment provider group was associated with the highest frequency of biased/unbalanced information (74% of reported biased/unbalanced information, 23/31), which was also confirmed by the weighted frequencies of biased/unbalanced information across different website affiliation groups.

    Comprehensibility of information was affected by the use of terminologies without explanation (28% of reported readability issues, 10/35), and/or the use of illustrations that were incorrectly labeled (5% of reported readability issues, 2/35), or inadequately labeled (5% of reported readability issues, 2/35), or sometimes not relevant at all (11% of reported readability issues, 4/35). In addition, poor information presentation and organization (11% of reported readability issues, 5/35) played an important role in the ability of finding information. Furthermore, repetition was found in some of the reviewed websites (11% of reported readability issues, 5/35).

    The most frequently reported treatment option was the removal of symptomatic wisdom teeth and to seriously consider removal of asymptomatic ones (30% of reported treatment options, 7/23), while 4 websites (17% of reported treatment options) recommended the prophylactic removal of all wisdom teeth to “get peace of mind.” A number of websites (28% of reported treatment options, 6/23) recommended the removal of only symptomatic ones. There were instances where patients were advised to get a second opinion (17% of reported treatment options, 4/23) before making a treatment choice related to their wisdom teeth. Coronectomy (removing the crown and retaining the root) as a treatment option for high-risk wisdom teeth was rarely mentioned.


    Preliminary Findings

    In this study, we aimed to provide a guide to assess the quality and readability of online health information with an application on Internet-related information concerning TMs problems using a scale developed for this purpose. The study also identified a shortlist of high-quality resources that might be recommended by clinicians to patients having TMs problems. Because online resources are dynamic, the researchers explored predictors for SIQ that might be used for a quick and easy identification of high-quality online resources.

    To identify high-quality resources, a search was carried out using 3 common search engines (Google, Yahoo!, and Bing), and 3 keywords thought to be used by an average patient. While some authors have claimed that patients do not normally go beyond the first 25 results [35], the number of websites included per search query ranged from 10 to 100 websites. Accordingly, we decided to include the first 50 websites per search engine query. The number of websites remaining for thorough evaluation in this study was considered reasonable according to existing literature where the websites included for final analysis ranged from 21 [36] to 67 [37] with a mean of 38 websites per study. In addition, the observed power for the association between website affiliation and website scores was found to be high.

    Internet information was delivered using mainly question and answer format either alone or together with fact sheets. Preferences of dental patients in relation to information delivery format need further investigation as there is a knowledge gap in the existing literature in this area. In addition, treatment providers should consider using online forums on their websites supported by health professionals to allow for a better engagement with patients [38]. Despite the importance of multimedia use in patient education [23], a combination of text and images was the most commonly used method of presenting information. There were instances where images were not related to the discussed topic, or were inadequately or incorrectly labeled. It is argued that the use of multimedia is associated with high costs due to professionalism, especially if these websites are for small businesses. Efforts should be made by professional and public health organizations to make multimedia available with permission to use at a reduced or no cost. A majority of websites used Facebook and Twitter as social media for sharing of online information. While many people search for information on the Internet for a family member or a friend [9], information sharing is currently powered by using social media.

    Evaluating the quality of scientific information was challenging, especially with the lack of reliable and valid assessment tools. In addition, evaluating the scientific content requires a person who has extensive knowledge in the field. This paper demonstrated that the newly developed SIQ Scale has a high internal consistency and also displayed convergent validity with information credibility tools (DISCERN and HoNCode), which can be used by other researchers. Website affiliation was found to have a significant association with SIQ, usability, accessibility, trust, DISCERN, and HoNCode.

    Linear regression models were used to explore the predictors for SIQ. The importance of this step is to make clinicians spend less time and effort to identify high-quality Internet resources, where no content analysis study is available. Website affiliation was able to significantly predict SIQ. Among different groups of website affiliation, governmental websites were found to be associated with the highest predicted SIQ score compared with the reference category. Credibility indicators—either DISCERN or HoNCode—were able to significantly predict SIQ after statistically controlling for website affiliation. A majority of variance in SIQ scores were explained by website affiliation and DISCERN. This finding is important because it might not only improve clinicians’ ability to identify high-quality online resources but also improve patients’ ability to find these resources by reviewing the governmental websites in light of DISCERN criteria.

    Among the reviewed websites, the recommended treatment options were a reflection of the clinical uncertainty related to asymptomatic wisdom teeth [39]. Despite the lack of evidence supporting prophylactic removal of disease-free asymptomatic impacted wisdom teeth [3], there was a tendency to recommend the removal of asymptomatic wisdom teeth to prevent future problems. Conversely, some websites recommended the removal of only symptomatic third molars. Because of the uncertainty regarding asymptomatic wisdom teeth, some websites advised patients to obtain a second opinion. These findings suggested that clinicians should discuss this uncertainty with their patients before making a shared decision, because patients themselves might be confused due to conflicting information [10]. In addition, some websites were not providing patients with evidence-based information; for example, many websites recommended continuous application of ice packs postoperatively despite the best available information from randomized controlled trial evidence, which showed no significant difference on postoperative edema, pain, and trismus when compared with no intervention [40]. Clinicians have a responsibility to apply the current best evidence in the shared decision-making process to reach a decision that is ethical, and in the best interest of the patient. Although cost is known to provoke anxiety for dental patients [41], it was found to be the least reported information. This suggests that providing cost estimates on websites could be useful in avoiding/reducing potential anxiety related to treatment costs.

    Among the used readability-grade estimates, the New FOG readability grade was the most powerful in predicting scientific information comprehensibility. The significant correlation between the mean readability grade and Scientific Information Comprehension Scale score suggested convergent validity and consequently that the Readability Studio software could be used to assess information comprehensibility. In this study, the estimated mean readability grade was higher than Grade 8 as recommended by some health authorities [17]. Attention should be paid to provide information in a way that is patient centered.

    The strong and significant correlation between websites’ ranking according to their SIQ and their ranking according to total scoring suggested that websites associated with the SIQ were also associated with other quality aspects such as readability, usability, trust, and credibility. Such results suggest that future research might focus on the SIQ Scale, readability-grade estimate, and DISCERN to limit the evaluation process.

    The main limitation of this study lay in 2 main areas: sampling bias and examiner bias that were known to the researchers when conducting data collection and analysis. However, effort was made to minimize their impact by using predetermined assessment criteria and to statistically validate the measurements used. In addition, websites were evaluated by the main author who has appropriate academic qualifications and clinical experience—an approach that has been used in previous research [42]. With regard to sampling bias, the retrieved websites were limited to the keywords that were used and search engines on a certain day.

    The strengths of our study were (1) the contribution to the field of health informatics such as the development and initial validation of the SIQ Scale and the validation of Readability Studio Professional 2012; (2) contribution to current clinical practice by providing a shortlist of high-quality websites (however, clinicians need to consider the dynamic nature of online resources); (3) the development of criteria for patient information concerning wisdom tooth problems (see Multimedia Appendix 1), which might be used as an information sheet covering all areas of wisdom teeth removal and using the best available evidence; (4) use of a statistical approach to analyze website data that has not been used previously in these kind of studies, such as convergent validity, linear regression using dummy variables, and thematic analysis of open comment section using NVivo 10; and (5) the validation of a readability software application that could be used in future research. The SIQ Scale [24] demonstrated some evidence of both reliability and validity in assessing the SIQ; hence, it might be usable in future research related to the assessment of online health information.

    Conclusion

    This study provides clinicians with guidance in assessing Internet resources for patients suffering from wisdom tooth problems. However, clinicians may apply similar techniques when recommending websites to patients who suffer from other dental problems. Consumer health information providers should consider evidence-based information, use of multimedia, and information readability during the process of information production. Readability Studio Professional 2012 was found to be valid as a software application for assessing comprehensibility of online health information. Website affiliation and DISCERN were found to play a major role in the prediction of SIQ. Governmental websites were associated with the highest prediction for SIQ. DISCERN and HoNCode as online information credibility tools were significantly able to predict the SIQ. In instances where no guidance is available, patients could review governmental websites in light of DISCERN criteria to identify high-quality information. The developed SIQ Scale had high internal consistency and established convergent validity, suggesting its use in the future to assess the SIQ of online dental information.

    Acknowledgments

    This paper was part of the principal author’s PhD research project, for which an Adelaide Scholarship International was received. The research received support from the Australian Research Centre for Population Oral Health, The University of Adelaide.

    Conflicts of Interest

    None declared.

    ‎

    Multimedia Appendix 1

    Criteria used for evaluating scientific information quality concerning wisdom teeth problems.

    PDF File (Adobe PDF File), 92KB
    1. Dodson TB, Susarla SM. Impacted wisdom teeth. BMJ Clin Evid 2010;2010 [FREE Full text] [Medline]
    2. McGrath C, Comfort MB, Lo EC, Luo Y. Changes in life quality following third molar surgery—The immediate postoperative period. Br Dent J 2003 Mar 8;194(5):265-268. [CrossRef] [Medline]
    3. Mettes TD, Ghaeminia H, Nienhuijs ME, Perry J, van der Sanden WJ, Plasschaert A. Surgical removal versus retention for the management of asymptomatic impacted wisdom teeth. Cochrane Database Syst Rev 2012;6:CD003879. [CrossRef] [Medline]
    4. Muirhead VE, Marcenes W, Wright D. Do health provider-patient relationships matter? Exploring dentist-patient relationships and oral health-related quality of life in older people. Age Ageing 2014 May;43(3):399-405 [FREE Full text] [CrossRef] [Medline]
    5. van Wijk AJ, Buchanan H, Coulson N, Hoogstraten J. Preparatory information for third molar extraction: Does preference for information and behavioral involvement matter? Patient Educ Couns 2010 Apr;79(1):94-99. [CrossRef] [Medline]
    6. Levinson W, Hudak P, Tricco AC. A systematic review of surgeon-patient communication: Strengths and opportunities for improvement. Patient Educ Couns 2013 Oct;93(1):3-17. [CrossRef] [Medline]
    7. Ferrús-Torres E, Valmaseda-Castellón E, Berini-Aytés L, Gay-Escoda C. Informed consent in oral surgery: The value of written information. J Oral Maxillofac Surg 2011 Jan;69(1):54-58. [CrossRef] [Medline]
    8. Bastable S. Essentials of Patient Education. Sudbury, MA: Jones and Bartlett Publishers; 2006.
    9. Hu J, Luo E, Song E, Xu X, Tan H, Zhao Y, et al. Patients' attitudes towards online dental information and a web-based virtual reality program for clinical dentistry: A pilot investigation in China. Int J Med Inform 2009 Mar;78(3):208-215. [CrossRef] [Medline]
    10. Glick M. The Internet-informed patient: Opportunities for patient-centered care. J Am Dent Assoc 2013 Mar;144(3):239-240. [Medline]
    11. Purcell GP, Wilson P, Delamothe T. The quality of health information on the Internet. BMJ 2002 Mar 9;324(7337):557-558 [FREE Full text] [Medline]
    12. Allam A, Schulz PJ, Nakamoto K. The impact of search engine selection and sorting criteria on vaccination beliefs and attitudes: Two experiments manipulating Google output. J Med Internet Res 2014;16(4):e100 [FREE Full text] [CrossRef] [Medline]
    13. Chestnutt IG, Reynolds K. Perceptions of how the Internet has impacted on dentistry. Br Dent J 2006 Feb 11;200(3):161-165. [CrossRef] [Medline]
    14. Usher WT. Australian health professionals' health website recommendation trends. Health Promot J Austr 2011 Aug;22(2):134-141. [Medline]
    15. Jordan JE, Buchbinder R, Briggs AM, Elsworth GR, Busija L, Batterham R, et al. The health literacy management scale (HeLMS): A measure of an individual's capacity to seek, understand and use health information within the healthcare setting. Patient Educ Couns 2013 May;91(2):228-235. [CrossRef] [Medline]
    16. Brennan D, Spencer J, Roberts-Thomson K. Dental knowledge and oral health among middle-aged adults. Aust N Z J Public Health 2010 Oct;34(5):472-475. [CrossRef] [Medline]
    17. South Australia Health (SA Health). Assessing Readability. Adelaide, South Australia, Australia: SA Health; 2013 Jan.   URL: http:/​/www.​sahealth.sa.gov.au/​wps/​wcm/​connect/​fcb907004e455125ab8eaf8ba24f3db9/​HLT-AssessingReability-T7-PHCS-SQ20130118.​pdf?MOD=AJPERES&CACHEID=fcb907004e455125ab8eaf8ba24f3db9 [accessed 2015-09-20] [WebCite Cache]
    18. Shedlosky-Shoemaker R, Sturm AC, Saleem M, Kelly KM. Tools for assessing readability and quality of health-related web sites. J Genet Couns 2009 Feb;18(1):49-59. [CrossRef] [Medline]
    19. Oleander Software. Readability Studio Professional. Vandalia, OH: Oleander Software; 2012.
    20. Misra P, Agarwal N, Kasabwala K, Hansberry DR, Setzen M, Eloy JA. Readability analysis of healthcare-oriented education resources from the American Academy of Facial Plastic and Reconstructive Surgery. Laryngoscope 2013 Jan;123(1):90-96. [CrossRef] [Medline]
    21. Eloy JA, Li S, Kasabwala K, Agarwal N, Hansberry DR, Baredes S, et al. Readability assessment of patient education materials on major otolaryngology association websites. Otolaryngol Head Neck Surg 2012 Nov;147(5):848-854. [CrossRef] [Medline]
    22. Ni RR, McCreary C. Head and neck cancer information on the internet: Type, accuracy and content. Oral Oncol 2009 Aug;45(8):675-677. [CrossRef] [Medline]
    23. Kandula NR, Malli T, Zei CP, Larsen E, Baker DW. Literacy and retention of information after a multimedia diabetes education program and teach-back. J Health Commun 2011;16:89-102. [CrossRef] [Medline]
    24. Hanna K. Wisdom Tooth Website Evaluation Form. 2013.   URL: https://docs.google.com/forms/d/17B14VEcbuYFXy8eAkLDdsl3AE3GrWMYLalAtrq9HcQY/viewform?usp=send_form [accessed 2015-05-09] [WebCite Cache]
    25. Health on the Net Foundation. HoNCode Membership Application. 2014.   URL: https://www.hon.ch/cgi-bin/HONcode/Inscription/inscription.pl [accessed 2014-07-27] [WebCite Cache]
    26. Charnock D. The DISCERN Handbook. Oxford, UK: Radcliffe Medical Press; 1998.   URL: http://www.discern.org.uk/discern.pdf [accessed 2014-07-27] [WebCite Cache]
    27. Minervation Ltd. The LIDA Instrument. Oxford, UK: Minervation Ltd; 2007.   URL: http://www.minervation.com/wp-content/uploads/2011/04/Minervation-LIDA-instrument-v1-2.pdf [accessed 2015-09-20] [WebCite Cache]
    28. Caldwell B, Cooper M, Reid LG, Vanderheiden G. Web Content Accessibility Guidelines (WCAG) 2.0. 2010 Jun.   URL: http://www.w3.org/WAI/GL/2010/WD-WCAG20-20100617/ [accessed 2015-09-20] [WebCite Cache]
    29. Charnock D, Shepperd S. Learning to DISCERN online: Applying an appraisal tool to health websites in a workshop setting. Health Educ Res 2004 Aug;19(4):440-446 [FREE Full text] [CrossRef] [Medline]
    30. IBM Corporation. IBM SPSS Statistics for Windows Computer Program (Version 22.0). New York, NY: IBM Corporation; 2013.
    31. QSR International Pty Ltd. NVivo Qualitative Data Analysis Software. Burlington, MA: QSR International Pty Ltd; 2012.   URL: http://www.qsrinternational.com/products_nvivo.aspx [accessed 2015-09-20] [WebCite Cache]
    32. Canvin R. Wisdom Teeth Removal. Glasgow, Scotland, UK: Bupa UK; 2013.   URL: http://www.bupa.co.uk/health-information/directory/w/wisdom-teeth-removal [accessed 2014-11-17] [WebCite Cache]
    33. NHS Choices. Wisdom Tooth Removal. London, UK: NHS Choices; 2014.   URL: http://www.nhs.uk/Conditions/Wisdom-tooth-removal/Pages/Introduction.aspx [accessed 2014-11-17] [WebCite Cache]
    34. Animated-Teeth. Is It Really Necessary to Have Your Wisdom Teeth Out?. 2013.   URL: http://www.animated-teeth.com/wisdom_teeth/t6-wisdom-tooth-not-extract.htm [accessed 2014-11-17] [WebCite Cache]
    35. Antonarakis GS, Kiliaridis S. Internet-derived information on cleft lip and palate for families with affected children. Cleft Palate Craniofac J 2009 Jan;46(1):75-80. [CrossRef] [Medline]
    36. Patel U, Cobourne MT. Orthodontic extractions and the Internet: Quality of online information available to the public. Am J Orthod Dentofacial Orthop 2011 Feb;139(2):e103-e109. [CrossRef] [Medline]
    37. Park MW, Jo JH, Park JW. Quality and content of Internet-based information on temporomandibular disorders. J Orofac Pain 2012;26(4):296-306. [Medline]
    38. Kauw D, Repping-Wuts H, Noordzij A, Stikkelbroeck N, Hermus A, Faber M. The contribution of online peer-to-peer communication among patients with adrenal disease to patient-centered care. J Med Internet Res 2015;17(2):e54 [FREE Full text] [CrossRef] [Medline]
    39. Tolstunov L. Third molar uncertainty. J Oral Maxillofac Surg 2013 Feb;71(2):239. [CrossRef] [Medline]
    40. van der Westhuijzen AJ, Becker PJ, Morkel J, Roelse JA. A randomized observer blind comparison of bilateral facial ice pack therapy with no ice therapy following third molar surgery. Int J Oral Maxillofac Surg 2005 May;34(3):281-286. [CrossRef] [Medline]
    41. Armfield JM. The extent and nature of dental fear and phobia in Australia. Aust Dent J 2010 Dec;55(4):368-377. [Medline]
    42. Magunacelaya MB, Glendor U. Surfing for mouth guards: Assessing quality of online information. Dent Traumatol 2011 Oct;27(5):334-343. [CrossRef] [Medline]

    ‎
    ANOVA: analysis of variance
    HoN: Health on the Net
    HoNCode: Health on the Net Code
    SIQ: scientific information quality
    SI Comprehension: Scientific Information Comprehension
    SI Referencing: Scientific Information Referencing
    SI Reporting: Scientific Information Reporting
    TMs: third molars

    Edited by G Eysenbach; submitted 19.05.15; peer-reviewed by S Penteado, A Allam; comments to author 08.07.15; revised version received 20.07.15; accepted 22.07.15; published 06.10.15

    Copyright

    ©Kamal Hanna, David Brennan, Paul Sambrook, Jason Armfield. Originally published in the Interactive Journal of Medical Research (http://www.i-jmr.org/), 06.10.2015.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Interactive Journal of Medical Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.i-jmr.org/, as well as this copyright and license information must be included.