Thoracic Surgery Information on the Internet: A Multilingual Quality Assessment

Introduction

Background

As patients are diagnosed with serious conditions and await complex procedures, it is accepted that they inherently will explore the Internet for answers. Over 80% of patients, health care professionals, and other invested groups utilize the Internet to seek medical information, seeing it as a reliable, trustworthy, and accessible source [1-3]. Industry groups, clinicians, and health care institutions may construct websites with commercial interests in mind [1,4]. In contrast, only a minority of websites are sponsored by government or educational organizations and nonprofit organizations, which may provide objective, unbiased, and hence more accurate information, compared with other sponsors [5,6]. Therefore, the Internet’s accessible source of health information, and frequency of use by the majority, substantiates the need to assess its quality and validity.

Thoracic surgery is a common mode of treatment for many patients with lung cancer. However, patients face a range of extensive and unregulated information regarding conditions and procedures on the Internet, often affecting their expectations and informed decision-making [7,8]. Moreover, language affects the quality of information [9-14], which impacts on multicultural societies and non-English speaking patients who require reliable information.

Health on the Net

Clinicians also require tools both to identify quality information for themselves and also to direct their patients to reliable, high quality Internet resources [11-13]. High quality and reliable health information can be found through the help of several tools [10,11,15]. The Health On the Net (HON) Foundation is one such tool. HON is a not-for-profit multilingual accreditation body that aims to accredit health websites according to its key principles of authority, complementarity, confidentiality, attribution, justifiability, transparency of authorship, sponsorship, and advertising [11]. The HONcode offers directions for users in evaluating and creating a trustworthy and reputable website [16,17]. Of note, website quality has been tested using the HONcode tool across a range of specialties with only a small percentage of websites (7-27%) being routinely accredited [6,18-20].

A comprehensive literature search regarding website information within the sphere of thoracic surgery was undertaken, yielding no studies that evaluate the quality of thoracic surgery-related information on the Internet. In this study, we aimed to evaluate the quality of current Internet information on thoracic surgery websites based on HON principles, and to compare differences between English, French, German, and Spanish language sites. The effect of language relates to what websites appear on different Google search engines (English, French, German, and Spanish), and whether there are any differences in HON-accredited websites. Our secondary goal was to assess and compare information quality based on types of website sponsors.

Methods

Search Engine and Search Terms

Our methodology has been previously described and used [11-13,21]. On this occasion, however, we used the corresponding Google search engine for each respective language search. We performed an Internet search of 20 terms in December 2014 to March 2015 (Table 1) and assessed 12,000 websites. As formal medical terminology has been used for search terms, the same search term used in English was used for the French, German, and Spanish searches on their respective Google search engines. The terms searched were “pectus excavatum,” “pectus carinatum,” “Nuss procedure,” “Ravitch procedure,” “Lorenz bar repair,” “lung cancer,” “nonsmall cell lung cancer,” “small cell lung cancer,” “VATS,” “video-assisted thoracic surgery,” “lung resection,” “lung wedge resection,” “pneumonectomy,” “thoracotomy,” “mediastinoscopy,” “bronchoscopy,” “EBUS,” “endobronchial ultrasound,” and “lung lobectomy.” An expert thoracic surgeon deemed these terms the most common and pertinent medical conditions and procedures for review in this study. These search terms were selected because they are the most objective terms that patients would hear during a consultation. By searching these terms, more meaningful data from websites can be ascertained. Ethics or Institutional Review Board (IRB) approval was not required for this study, since it does not involve patients but only Web-based review of publicly accessible websites.

Internet Searching for Accredited Websites

International and independent, qualified accrediting bodies check HON status at regular times, ensuring that HON certification meets the strict internationally accepted requirements. Moreover, the HON function has been evaluated by many studies, and judged to be a high caliber tool [10-13,22,23].

Access beyond the first page of results by patients is rare [24]. Thus, the first 150 websites yielded by each search were identified and sequentially screened for quality as defined by the HON Foundation. HON principles through the HONcode toolbar function (downloaded from http://www.hon.ch/ for use on any personal computer. HONcode toolbar is easily installed, providing an accessible and visual cue for users) were then applied. According to the HON Foundation website [10], there are 8 criteria evaluated for HONcode certification of a website. These are (1) authoritative (indicate qualifications of authors), (2) complementarity (information should support, not replace, the doctor-patient relationship), (3) privacy (respect privacy and confidentiality of personal data submitted to the site by visitor), (4) attribution (cite the sources of published information, date medical and health pages), (5) justifiability (site must back up claims relating to benefits and performance), (6) transparency (accessible presentation, accurate email contact), (7) financial disclosure (identify funding sources), and (8) advertising policy (clearly distinguish advertising from editorial content). This toolbar automatically activates if a website is accredited by the HON Foundation (HONcode+), as opposed to the toolbar not lighting up, indicating that the website is not HON-accredited (HONcode−). On the basis of the previous studies, approximately 5% of websites could be deemed HONcode+, but have not been accredited yet [10-13].

Analysis of Accredited Websites’ Likelihood of Being Viewed

A secondary analysis of the first 150 websites encountered for each search term was undertaken, as previously described [6,18,25]. First, all returned websites for each search term were divided into tertiles (first 50, middle 50, and last 50). The proportion of accredited sites in each tertile and language was then analyzed and compared by the chi-square test. The purpose of this analysis was to determine whether accredited websites were appearing preferentially—that is, in the pages least likely (last 50) versus most likely (first 50) to be viewed.

Quality Control

For quality control, an English-language search of the control term, “lung cancer,” had nonaccredited sites within the first 150 discovered websites manually evaluated using the HON criteria to determine their HON status to ascertain if they fulfilled the criteria despite not being officially accredited.

Logistic Regression Examining Variables Associated With HON Status

This test was conducted using the three major variables of our study, namely a search term, language, and tertile, of the first 150 websites returned. The reference groups for each variable were excavatum, the first tertile, and English, respectively.

Analysis of Website Sponsors

For all search terms, an analysis was undertaken from English-language websites to determine who the website sponsors were. Only English language websites were examined due to the authors’ lack of proficiency in the other languages. The site sponsors were organized into the following groups: (1) lawyers, (2) nonprofit organizations, (3) government organizations or educational institutions, (4) commercial, (5) thoracic specialists and their professional organizations, (6) Books, articles, and references, (7) other health care professionals, (8) other (social media, forums, personal websites, newspapers, and (9) unrelated.

Sponsorship was determined independently by information on the retrieved Web page regarding its origin; if sponsorship was not obviously apparent, the website was explored until sponsorship could be determined. The concept of sponsorship is not to be confused with the Google terminology of “sponsored links,” which either highlights pages at the start of retrieved search or lists links on the side of the page under a banner. As in a previous analysis, such pages were not included in this study [11].

Statistical Analysis

Comparisons of proportions across types of cancer and language were performed by the chi-square test (or Fisher exact test when counts were <5). All statistical tests were two-sided. Odds ratio and 95% CI were also calculated from the logistic regression analysis. The data analysis for this study was generated by SAS software version 9.1. (SAS Institute Inc).

Results

Internet Search Results for Accredited Websites

The total number of websites for each thoracic surgery-related search term is variable (Table 1). “Lung cancer” had the most websites with approximately 150 million websites followed by “small cell lung cancer” with approximately 112 million websites. “Ravitch procedure” returned the least number, with only 159,890 websites.

The total percentage of HON-accredited sites was notably low across all search terms (median 8%; see Table 1). “Lorenz bar repair,” “EBUS,” “endobronchial ultrasound,” and “VATS” had less than 5% of HON-accredited sites (Table 1).

Regarding linguistic differences (see Table 2 and Figure 1), there was a similar number of HON-accredited thoracic websites across all languages evaluated. English (8%) and German (8%), French (7%) and Spanish (7%) had a similar percentage of HON-accredited sites.

Tertiles were examined to ascertain where HON-accredited websites were more likely to appear. HON accreditation was seen statistically more commonly in the first tertile (0-50 sites) of websites (see Table 3 and Figure 2).

‎View this figure

‎View this figure

Quality Control

For the first 150 “lung cancer” (English) results, we found that 20 sites were accredited by the HON toolbar and 130 were not. We found that 6.9% (9/130) of those nonaccredited sites met HON criteria when assessed manually and 13.2% (79/600) of cancer-related thoracic surgery websites are HON-accredited.

Logistic Regression Examining Variables Associated With HON Status

Odds ratios calculated by search term, language, tertile, and between groups, highlighted significant differences (Table 4). For language, English compared with French, German, or Spanish was just as likely to return an accredited site. The second tertile of websites (51-100) assessed were more likely than the third tertile (101-150) to have accredited sites.

Analysis of Website Sponsors

The sponsor analysis of the 150 websites in English (Table 5) indicated that the most commonly encountered sponsors were “books, articles, and references” (47.1%, 1399/2967) followed by “government or education” (27.7%, 821/2967), “others (social media, forums, personal websites, newspapers” (6.7%, 199/2967), “nonprofit organizations” (5.0%, 149/2967), “commercial” (4.5%, 134/2967), and “thoracic specialists or professional organizations” (3.4%, 100/2967). “Lawyer” (<1%, 9/2967) and “other health care professionals” (<1%, 3/2967) sponsored far less sites. A small percentage (6.3%, 188/2967) of sponsor websites were unrelated to medicine.

Search terms with a larger percentage of “government or education” or “books, articles, and references” were the terms with a larger percentage of HON-accredited websites: “lung cancer,” “nonsmall cell lung cancer,” “small cell lung cancer” with P value <.001; “lung lobectomy,” “lung resection,” and “lung wedge resection” with P value .001; “pneumonectomy,” “bronchoscopy,” and “thoracoscopy” with P value .001.

Discussion

Principal Findings

The aim of this study was to quantify information quality on thoracic surgery-related websites on the Internet. Clinicians may become aware of the lack of quality information regarding thoracic surgery and help to educate patients about the pitfalls of information on the Internet, and direct them to better quality websites.

In summary, the total number of websites for keyword searches varies considerably. The total percentage of HON-accredited websites was markedly low across all search terms. There were minimal linguistic differences in HON-accredited websites, with HON-accredited websites most likely to appear in the first tertile. Nearly half of the websites were books, articles, or references, whereas nearly one-third were governmental or educational.

Comparison With Prior Work

The Internet has developed into an accessible source of health information for everyone. Health websites are guides for patients wanting to better understand their conditions [26]. Web-based health information was sought by 72% of adult Internet users over the last few years [27], a number predicted to grow. Clinicians directing patients to reliable information has many benefits: improving patient-doctor relationships, reinforcing consultation discussions, assisting informed decision-making, providing education before and after events, and helping patients seek appropriate consultation for sensitive topics (eg, urology, gynecology).

There is a stark discrepancy between reliable health information and quality resources that disseminate it. The number of websites providing accurate information for thoracic surgery is not ideal. Only 13% of cancer-related thoracic surgery websites overall were HON-accredited. This is less than in our previous studies, uro-oncology websites [6] in 2009 and surgical oncology websites in 2012 [18], which each returned 18% of oncology-related HON-accredited websites. Similarly, there were 15% of HON-accredited gynecological oncology-related websites [20]. Worse still, only 9% of benign prostate hyperplasia websites were HON-accredited [19]. This reflects our hypothesis that reliable, high-quality health information on the Internet is lacking, specifically for thoracic surgery as well as in a broader context. In the latter study [19], only 7% of nononcology-related websites such as “surgical treatments” were HON-accredited. This figure is comparable with our 10% “Lungsurg” HON-accredited websites. These results are concerning because they imply that patients will encounter unreliable information about their condition, regardless of cancer type. Evidently, this makes website assessment difficult for patients and clinicians alike, potentially leading to distrust of Internet thoracic surgery resources.

It has previously been acknowledged that website quality differs by language [10,11,14]. In our study, whereas English language searches returned more websites overall, both German and English searches returned 8% HON-accredited sites, and French and Spanish searches returned 7% HON-accredited sites. Thoracic surgery information is far more uniform across languages than results from our previous studies [6,18-20], albeit still alarmingly low. It is evident that there is a paucity of high quality, comprehensive information on thoracic surgery available around the world on the Internet, regardless of language. Similarly, HON-accredited websites are more likely to appear in the first tertile overall than in the second or third tertiles. This tertile discrepancy was expected since the Google algorithm generally places the most relevant websites first. Further analysis into the proportion of HON-accredited websites on the first page compared with the first tertile overall may yield interesting results, since it has been known that patients rarely move past the first search page.

Websites also act as a conduit for advertising. Health information is increasingly being controlled by marketing and commercial interests, taking advantage of a significant proportion of the population searching for health information [28]. Consequently, unbiased views are sacrificed for the type of health information offered. However, the majority of sponsors in this study were composed of (1) academic books, articles, and references and (2) government or education. The absence of commercial bodies or marketing in this area implies that thoracic surgery information might not be biased or skewed for marketing purposes, compared with other medical fields previously analyzed [6,18-20]. Notably, the search terms with these sponsors were those with more website results and more HON-accredited websites. This suggests a conscious effort to provide high quality information for these conditions and procedures. Although our study only revealed 1% of websites sponsored by lawyers, a search performed in the United States may show otherwise. This illustrates the unpredictable nature of the Internet.

HONcode is a simple means by which a clinician or patient can objectively correlate a website with high quality information. Compared with other instruments for evaluating website quality, it appears to be a straightforward, valuable tool, and fulfills its goal of identifying reliable health websites [29]. However, HONcode is by no means the only way to rate quality. The DISCERN instrument [30] and LIDA tool [31] are freely available online, designed to help users evaluate the quality of health information on the Internet. The ODPHP’s National Quality Health Website Survey instrument provides a sophisticated method to assess website quality, though is quite time-consuming and subjective [32]. Thus, compared with other, more intensive search tools, HONcode can be used to access reliable information easily by patients and clinicians, who have no prior experience or knowledge. Furthermore, it has been previously shown that website affiliation with HONcode is a significant predictor for scientific information quality [23]. Due to the growing number of websites, the HONcode certification seal is now obtained by voluntary application. However, many high quality websites lack the HONcode seal. In our study, 6% of websites in the control term could have met the criteria and this is consistent with prior research [6,18-20]. Currently, no studies evaluate awareness of HON certification in organizations and patients. Hence, shortcomings of HON may include voluntary application and lack of public awareness. Patients may bypass trustworthy websites, whereas organizations may not actively apply for HON certification. In a wider context, there is a notable lack of congruence of criteria between health information quality assessment tools [33]. Future research may be needed to streamline assessment tools, or streamline health website guidelines so that quality information is standardized. However, this is out of the scope of this paper. More immediately, further research is required to anal awareness of HON. Depending on these results, appropriate steps could then be taken to help clinicians, patients, and organizations to be exposed to HONcode, enabling access to reliable sources of information.

Limitations

It must be said that HONcode is a predictive indictor for high quality websites, which has its drawbacks. Thus, a proportion of websites with objectively high quality information may not fulfill HONcode criteria, and vice versa. As of 2015, HONcode certification is provided as a paid service. This can distort the validity of website information with HONcode criteria.

An inherent limitation of this study involves the search terms used. It cannot be guaranteed that patients would use these terms in their own research of their condition. It is in dispute whether informal search terms would yield websites with better quality information. Conversely, it may result in unrelated website results. However, given that the search terms used in this study are the most formal and objective, informal search terms would likely defer to pages with the formal terms by the Google search algorithm. One solution to this limitation is to encourage clinicians to use the formal medical terms for their patients, thereby empowering patients to research their condition better, ultimately resulting in greater patient education.

As with any Internet study, its dynamic and diverse character produces inherent limitations. In our study, we only performed searches in Melbourne, Australia. It would be interesting to perform multiple searches at various times and locations, analyzing any differences found. “Google” is the most popular search engine (http://searchenginewatch.com), having been used in other studies [10]. However, studies have also shown the impact of social media and health-related videos on YouTube on health care [34]. As these media are not appropriately standardized for health promotion and education, these studies highlight the need for caution among users. Search engines rely on language filters to determine sites returned, but Google enables a multilingual approach. A key advantage of Google may be for clinicians and patients who speak the languages analyzed here, which have a low number of accredited websites. Google translate may provide people with wider access to information online, though quality may vary. The impact of the validity of HON certification once a website has been translated by Google was not investigated in this study.

Conclusions

In conclusion, clinicians must appreciate the lack of validated information of most thoracic surgery websites. Discrepancies are apparent in quality and number of websites between search terms, tertiles, and language. Awareness of this lack of quality can facilitate clinicians in educating patients by using the formal medical term to empower patients to research their condition more comprehensively and thus gain a greater level of understanding. Clinicians must be proactive in identifying and directing patients to trustworthy and accurate information on websites. HONcode is an uncomplicated search tool and can serve as the vanguard to detect appropriate and trustworthy websites.