Web-Based and Telephone Surveys to Assess Public Perception Toward the National Health Insurance in Taiwan: A Comparison of Cost and Results

Introduction

Background

Health care systems often use the reform of payment system to create incentives for health care providers to provide only necessary service within a limited budget which could have influence on population’s accessibility to health care. Numerous studies have examined the impact of global budget system on access to health care. Iglehart et al found that after introducing a global budget, changed allocation of local physicians affected accessibility to medical care [1]. Chu et al also indicated that active suppression of medical expenditures curtailed access to medical care due to bed closures [2]. Many countries use surveys as a monitoring measure aimed at ensuring equity in access to medical care. The Agency for Healthcare Research and Quality (AHRQ) in the United States developed the Consumer Assessment of Healthcare Provider and Systems (CHAPS) with the aim of measuring, reporting on, and improving the quality of health care from the perspective of consumers and patients [3]. Similarly, the Department of Health in the United Kingdom (UK) has been conducting surveys regarding public perceptions of the National Health Service (NHS) and the GP Patient Survey [4,5].

Taiwan implemented a National Health Insurance (NHI) system in 1995 and implemented a global budget payment system in 2002 to contain skyrocketing medical expenditures. The NHI system is a single-payer system which covers more than 99% of residents and contracts with more than 90% of the hospitals and clinics. Like other countries, Taiwan’s National Health Insurance Administration (NHIA) has been conducting a series of surveys on public perception toward the NHI since 2002. The aim of this series of surveys is to understand public perception regarding NHI and health care services and to track how perceptions have changed over time. Questions in this series of surveys comprise core and optional items. Core items are included in every iteration of the annual survey, while optional items are designed specifically for current topics of interest. Every year, telephone survey data are collected using a computer-assisted telephone interviewing (CATI) system, in which samples are selected using a random digit dialing (RDD) system.

A number of problems have been identified in these telephone surveys [6]. Evidence has shown that the gender and race of the interviewer can influence the results [7,8]. The attitude and demeanor of the interviewers may also bias the responses [9,10]. Despite the high response rates, telephone surveys are becoming increasingly expensive, which may be due to the time required to complete surveys using these methods [11]. One alternative to telephone surveys is to use the World Wide Web. Web-based surveys are more likely to be free from interviewer bias due to the nature of being self-administered. They also create a data input file directly. Information obtained from Web-based surveys displays higher test-retest reliability than do telephone surveys [12]. Previous studies have suggested that respondents to Web-based surveys are more likely to describe their personal habits such as drinking [13,14], as well as their concerns regarding traffic safety [15]. This approach also raises concern about the comparability of information obtained using Web-based surveys with that obtained by telephone surveys. Telephone surveys generally obtain a representative national sample; however, Web-based surveys are restricted to respondents who have Internet access.

Study Aims

To the best of our knowledge, no previous study has examined the differences between Web-based surveys and telephone surveys regarding perceptions of NHI or NHI-contracted health care providers and self-reported health status. Our aim was to examine the difference of sociodemographic characteristics, satisfaction of NHI and NHI-contracted health care providers, attitude toward NHI-related issues, behavior in seeking medical advice and self-reported health status between those who completed Web-based surveys and those reached by telephone.

Methods

Development of Questionnaires

To obtain the data necessary to answer the study questions, we formulated a cross-sectional data collection framework using the telephone and internet as communication media. Both survey methods were conducted in August-September 2013. All people aged 20 or older in any region of Taiwan were eligible to take the survey. Screening was conducted by asking respondents whether they were 20 years of age.

The conceptual framework of questionnaires incorporated seven dimensions: satisfaction, health status, use of health service, personal health practice, need for health care, enabling resources, and predisposing characteristics. The draft version of the questionnaire was first reviewed by five experts to enhance content validity. Based on their suggestions, we modified the wording and items on the questionnaires. Then, a pre-test with small sample size was executed to examine the internal reliability of questionnaires. The Cronbach alpha coefficients of these seven dimensions were between .5 and .8. The final questionnaires included 43 items aimed at measuring satisfaction with the NHI and NHI-contracted health care providers, attitudes toward NHI-related issues, use of health service, health status (SF-8 health survey), health behaviors (eg, smoking, drinking, and chewing betel nuts), sociodemographic characteristics, and medical history.

The contents of the Web- and telephone-based questionnaires were identical in terms of the questions asked, the wording, and the order of presentation. The study protocol was approved by the ethics committee at National Taiwan University Hospital (IRB number: 1000401).

Telephone Interview Survey

Individuals aged 20 years and older were eligible for interviews. Stratified random sampling procedure was used to draw respondents for telephone interviews. We contracted a company with experience in conducting telephone interviews; they used the RDD method with CATI to carry out the interview. All interviews were conducted from a central location, and the interviewing process was monitored by supervisors to control quality. All interviewers were well-trained individuals working on a regular basis for the survey company. The household telephone number list was obtained from Chunghwa Telecom, a major provider in Taiwan. To improve the coverage rate and increase the chances of contacting of households that were not registered in the phone book, we randomly replaced the last two digits of the drawn telephone numbers with two different digits to form a new telephone number. If there were missing items, the telephone interviewers would connect participants again to complete the questionnaires. At the end of the survey, the total number of usable responses was 2411 with a sampling error of 2.0%.

Web-Based Survey

The sampling lists used in the Web-based and telephone surveys were obtained from different suppliers to avoid overlap. The panel list used in the Web-based survey included 320,000 individuals who were also matched to the defined target population. The respondents were randomly drawn from panel members aged 20 and older and invited via email. In addition, interested participants could click on a hyperlink to access information related to the study. The participants were required to finish all the items on the questionnaires, and then they could join the lottery. Finally, we obtained 1313 usable responses with sample error of 2.7% in the Web-based survey.

Statistical Analysis

Data were analyzed using SAS version 9.3 for Windows (SAS Institute). Descriptive statistics were used to describe the difference between respondents in the telephone-based and Web-based groups. An independent sample t test and Pearson chi-square test were used to test whether the differences were statistically significant. A P value of less than .05 was considered statistically significant. Multivariate logistic regression and mixed linear regression models were used to test the differences in each dimension of the questionnaire by reporting their odds ratios, coefficients, and standard deviations.

Results

The demographic and clinical characteristics of all 3724 respondents are presented in Table 1. The respondents who filled out the Web-based questionnaire were generally younger (P=.0001), unmarried (P<.001), nonsmokers (P<.001) with a higher education level (P<.001) and higher monthly household income (P<.001), were less likely to suffer from chronic disease (P<.001), and were more likely to be living in the Taipei area (P<.001) when compared to respondents of the telephone survey.

Table 2 presents the satisfaction felt by participants toward NHI and NHI-contracted health care providers, as well as behavior patterns in use of health service. Telephone survey respondents were more likely than Web-based respondents to report a feeling of satisfaction with the NHI (58.4% vs 44.7%; adjusted odds ratio [AOR] 1.97, 95% CI 1.62-2.40) and NHI-contracted health care providers (63.4% vs 48.4%; AOR 2.39, 95% CI 1.96-2.92). No significant differences were observed between telephone and Web-based survey respondents with regard to their behavior in use of health service.

Table 3 illustrates the attitudes of respondents toward NHI-related issues. Results of logistic regression analysis indicate that, after accounting for sociodemographic factors, comorbidity history and self-reported health status, Web-based respondents were more likely than telephone survey respondents to be aware of the details of the NHI policy on the programs specifically designed to assist the disadvantaged (41.1% vs 25.3%; AOR 1.86, 95% CI 1.52-2.27).

Table 4 presents the self-reported health statuses of respondents of the two surveys. Web-based respondents generally had poorer health status with regard to physical functioning (adjusted difference: −1.73; P<.001), role-physical (adjusted difference: −2.25; P<.001), body pain (adjusted difference: −1.94; P<.001), social functioning (adjusted difference: −3.89; P<.001), role-emotional (adjusted difference: −4.06; P<.001) and mental health (adjusted difference: −2.94; P<.001). However, Web-based respondents reported a higher degree of vitality (adjusted difference: 0.94; P=.015) than did the telephone survey respondents.

Discussion

Principal Findings

This study sought to determine whether a Web-based survey could be used as an alternative to RDD telephone surveys. Our results indicate substantial demographic and socioeconomic differences between the individuals who participated in the two surveys. Our findings are consistent with previous studies, in which Web-based survey respondents are generally younger, healthier, and more highly educated than respondents to telephone surveys [16,17]. This was not unexpected, due to the fact that elderly persons still lag behind other age groups with regard to internet usage [18]. The bias did not disappear after weighting according to sociodemographic factors. This raises concerns about the accuracy of Web-based surveys and how representative the information gained is of a population.

Comparison to Prior Work

This is the first study to compare Web-based and RDD telephone surveys with regard to different dimensions. Previous studies have demonstrated that the answers provided by respondents to Web-based surveys often differ from those provided by respondents to telephone surveys [15,19]; however, other studies showed no significant differences. Zuidgeest et al investigated differences between postal surveys and mixed-model surveys (Internet questionnaires with paper follow-up). Their findings failed to indicate any differences in global ratings of health care between respondents to the two surveys [20]. Braunsberger et al compared telephone and Web-based surveys with regard to test-retest reliability in which participants filled out two-wave questionnaires. Their results indicate that the reliability of data from Web panels exceeds that obtained from telephone survey panels [21]. Our results show that despite fundamental differences between those who took the two surveys, behaviors with regard to use of health service and attitudes toward NHI-related issues were similar, except for the awareness of assisting the disadvantaged policy in NHI. This implies that the two data collection modes are able to obtain unbiased data related to these issues. However, if one would like to assess the public perceptions of health care system and population health status, the results from different survey modes could not be compared directly.

Previous studies showed that many of the satisfaction polls done in other countries focused more on assessing consumer and patient satisfaction with the use of health services but rarely assessed the public perception on the health care system. In terms of survey modes, most of the satisfaction surveys collected data via telephone, mail, face-to-face, or mixed methods. Recently, Web-based surveys with advantages of lower cost and more time saving have been broadly used in many fields. In the UK, for example, the GP patient survey used three different methods to collect data: paper, telephone, and Web. Their results indicated that the proportion of patients using Web-based survey was increasing.

Compared to previously mentioned polls done in other countries, this study demonstrated the feasibility of combining satisfaction toward use of health service with awareness and attitudes toward selected policies. Furthermore, we used both telephone and Web-based surveys to collect responses to the same questionnaire and examined whether the responses were comparable. The results indicate that, after adjusting for respondents’ socioeconomic characteristics, use of health service, personal health practice, and health status, the satisfaction toward various issues was significantly different between respondents from telephone surveys and Web-based surveys. Moreover, results of this study also showed that respondents to the telephone survey were more satisfied with NHI and NHI-contracted health care providers than those who took the Web-based survey. Previous studies have indicated that Web-based respondents tend to give more neutral or negative attitudinal evaluations [22]. Evidence also indicates that the techniques, gender, and race of the interviewer can bias the results [7,8]. Thus, it may be reasonable to expect that individuals engaged in a direct conversation with a telephone interviewer may show more positive attitude toward the issues discussed in the survey. Our findings also indicate that Web-based survey respondents are more aware of NHI policies. This is not unexpected considering the higher level of education held by these individuals and their greater access to information via the Internet.

In terms of self-reported health status, Rivara et al investigated the model effect of health-related quality of life among parents and children aged birth to 17 years using Web-based versus telephone surveys [23]. They found that the differences caused by model effect were small but statistically significant in some of the measures. Our findings are consistent with these studies, indicating that the self-reported health status of respondents to telephone surveys was significant better than those responding to Web-based surveys, after controlling for demographic and socioeconomic characteristics, health behaviors, and comorbidity history. When matched by demographic and socioeconomic characteristics and health behaviors, the differences between the two groups remained significant.

Previous studies have indicated that the cost of and time required to complete Web-based surveys (measured per respondent) are far lower than that of other methods [20,21,24]. In this study, the data obtained from the two survey methods was collected by the same commercial research firm. The overall cost of the telephone survey included the cost of acquiring and compiling the survey data followed by preparation of a database for analysis as well as overhead and profit. The overall cost of the Web-based survey included the cost of contacting panel members, directing them to the survey site, creating a platform on which to conduct the survey as well as an allocation for overhead and profit. The average cost for each usable telephone survey sample was US$3.98, whereas the average cost of obtaining each usable Web-based survey sample was US$0.71, 17.8% of the cost of the telephone survey samples.

Nonetheless, several issues should be considered when interpreting these results. Any survey of perception and behavior is subject to various forms of self-selection and self-reported bias. As discussed earlier, the RDD telephone and Web-based surveys used in this study were not immune to these limitations. We attempted to minimize the effects of these issues by ensuring informed consent and standardized data collection procedures. Despite the fact that we employed an RDD contact method for the telephone survey, the contacted individuals were still able to choose whether to participate in the interview. Similarly, the Web-based survey, which required respondents to fill out the survey online, is prone to the issue of self-selection. Further research should address this issue by using a random assignment process to determine if the same population who agrees to be interviewed would choose a different format for the interview process.

Conclusions

Web-based surveys offer cost savings and efficiency in obtaining data, especially on awareness and attitudes toward NHI-related issues. But when assessing the public’s satisfaction with the health care system and population health status, survey results could not be compared directly because different methods of survey could yield different outcomes.