This paper is in the following e-collection/theme issue:

Information Quality in Digital Media (347) Consumer & Patient Education and Shared-Decision Making (699)

Published on in Vol 6, No 2 (2017): Jul-Dec

Quality of Patient Information Websites About Congenital Heart Defects: Mixed-Methods Study of Perspectives Among Individuals With Experience of a Prenatal Diagnosis

Quality of Patient Information Websites About Congenital Heart Defects: Mixed-Methods Study of Perspectives Among Individuals With Experience of a Prenatal Diagnosis

Quality of Patient Information Websites About Congenital Heart Defects: Mixed-Methods Study of Perspectives Among Individuals With Experience of a Prenatal Diagnosis

Authors of this article:

Tommy Carlsson1 Author Orcid Image ;   Ulla Melander Marttala2 Author Orcid Image ;   Barbro Wadensten3 Author Orcid Image ;   Gunnar Bergman4 Author Orcid Image ;   Ove Axelsson1, 5 Author Orcid Image ;   Elisabet Mattsson1, 6 Author Orcid Image
Article Authors Cited by (4) Tweetations (5) Metrics

    Original Paper

    1Department of Women's and Children's Health, Uppsala University, Uppsala, Sweden

    2Department of Scandinavian Languages, Uppsala University, Uppsala, Sweden

    3Department of Public Health and Caring Sciences, Uppsala University, Uppsala, Sweden

    4Department of Women’s and Children’s Health, Karolinska Institutet, Stockholm, Sweden

    5Centre for Clinical Research Sörmland, Uppsala University, Eskilstuna, Sweden

    6Department of Health Care Sciences, Ersta Sköndal Bräcke University College, Stockholm, Sweden

    Corresponding Author:

    Tommy Carlsson, RM, CCRN, MMSc

    Department of Women's and Children's Health

    Uppsala University

    Husargatan 3, D11:1

    Box 564

    Uppsala, SE-75237

    Sweden

    Phone: 46 736143454

    Email: tommy.carlsson@kbh.uu.se


    Background: When a heart defect is prenatally diagnosed in the fetus, expectant parents experience a great need for information about various topics. After the diagnosis, the Web is used for supplemental information, and the scarcity of research calls attention to the need to explore patient information websites from the perspectives of the intended consumers.

    Objective: The overarching aim of this study was to explore the quality of Swedish patient information websites about congenital heart defects, from the perspectives of individuals with experience of a prenatal diagnosis of congenital heart defect in the fetus.

    Methods: This was a mixed-methods study of websites identified through systematic searches in the two most used Web-based search engines. Of the total 80 screened hits, 10 hits led to patient information websites about congenital heart defects. A quality assessment tool inspired by a previous study was used to evaluate each website’s appearance, details, relevance, suitability, information about treatment choices, and overall quality. Answers were given on a 5-point Likert scale, ranging from 1, representing the lowest score, to 5, representing the highest score. Each website was assessed individually by persons with experience of continued (n=4) and terminated (n=5) pregnancy following a prenatal diagnosis. Assessments were analyzed with Kendall’s coefficient of concordance W, Mann-Whitney U test, Friedman’s test, and a Wilcoxon-Nemenyi-McDonald-Thompson test. In addition, each assessor submitted written responses to open-ended questions in the quality assessment tool, and two joint focus group discussions were conducted with each group of assessors. The qualitative data were analyzed with inductive manifest content analysis.

    Results: Assessments represented a low score (median=2.0) for treatment choices and moderate scores (median=3.0) for appearance, details, relevance, suitability, and overall quality. No website had a median of the highest achievable score for any of the questions in the quality assessment tool. Medians of the lowest achievable score were found in questions about treatment choices (n=4 websites), details (n=2 websites), suitability (n=1 website), and overall quality (n=1 website). Websites had significantly different scores for appearance (P=.01), details (P<.001), relevance (P<.001), suitability (P<.001), treatment choices (P=.04), and overall quality (P<.001). The content analysis of the qualitative data generated six categories: (1) advertisements, (2) comprehensiveness, (3) design, (4) illustrations and pictures, (5) language, and (6) trustworthiness. Various issues with the included websites were highlighted, including the use of inappropriate advertisements, biased information, poor illustrations, complex language, and poor trustworthiness.

    Conclusions: From the perspectives of the intended consumers, patient information websites about congenital heart defects are, to a large extent, inadequate tools for supplemental information following a prenatal diagnosis. Health professionals should initiate discussions with patients about their intentions to use the Web, inform them about the varied quality in the Web-based landscape, and offer recommendations for appropriate Web-based sources.

    Interact J Med Res 2017;6(2):e15

    doi:10.2196/ijmr.7844

    Keywords

    congenital heart defects; consumer health information; information literacy; Internet; popular works; pregnancy; prenatal diagnosis 


    Background

    Many countries around the world include obstetric ultrasound examinations as part of routine maternity care. One of the investigations possible with the ultrasound examination involves assessment of the anatomy of fetuses, with the purpose to detect or rule out fetal anomalies [1]. The most common types of fetal anomalies are congenital heart defects, defined as malformations of the heart and great vessels [2]. The incidence of congenital heart defects is approximately 8 in 1000 live births [2], with an incidence of 2.5 to 3.0 for the most severe, which require specialist cardiologic care [3]. Between the years 2000 and 2005, roughly 1 in 5 cases of all congenital heart defects were detected before birth [4]. However, the rates of prenatal diagnoses have increased in recent years [5-7], and studies report that over 40% of the most severe defects are prenatally diagnosed today [2,4,7].

    Expectant parents view the obstetric ultrasound examination as an important step toward parenthood [8-10] but are often unprepared for adverse news [11-13] and experience an emotional crisis when faced with a prenatal diagnosis [11,14]. Depending on legislation and availability of induced abortion, the diagnosis may require a decision to continue or terminate the pregnancy. Expectant parents faced with the diagnosis experience this decision as difficult [11,15]. The decision involves various informational [11,16,17] considerations such as the process of induced abortion and what to expect of the postnatal situation if the pregnancy is continued [17]. Consequently, these persons experience a great need for information and highly value that which is offered during consultations with specialist health professionals [17,18]. However, the psychological distress experienced in connection to the diagnosis negatively influences comprehension and retention of the information offered [12,17], and research suggests that these individuals use the Web for supplemental information [16,17,19].

    Patients view the Web as an important source of health-related information [20] and commonly use it for such purposes [20,21]. During pregnancy, women frequently use it to search for information about a variety of topics such as fetal development, pregnancy complications, and antenatal care [22]. As a large source of potentially highly accessible information [23,24], it holds promise to promote patient equity and empowerment [23,25-27]. However, studies indicate that consumers experience various difficulties when using the Web for health-related information [28,29]. Further complicating the situation, there is a lack of overarching systematic activities that aim to assess and control the quality of the available Web-based information [23], resulting in a risk of contact with information of poor quality. Combined, these issues call attention to the disorganized state of the Web [24]. The informational difficulties [11,15-17] and ethical dilemmas [30] related to decision making regarding whether to continue or terminate the pregnancy highlight the need to explore the relevance, level of details, suitability, and information about treatment choices in patient information websites. Moreover, research indicates that users of Web-based information place high importance on website appearance [31-33], raising questions concerning whether the design of these websites correspond to the preferences of the intended consumers.

    As a step toward ensuring high-quality patient information materials, research suggests a need for developers to involve the intended consumers in the stages of production [34]. However, studies investigating the quality of Web-based information often use researchers or health professionals as assessors, raising questions about its applicability in clinical settings. In light of the identified differences in perspectives between researchers, health professionals, and patients [35,36], more research that employs new methods to investigate the perspectives of the actual consumers of the information is needed. No study has yet been published that explores consumers’ perspectives on websites about congenital heart defects.

    Objectives

    The overall aim of this study was to explore the quality of patient information websites about congenital heart defects, from the perspectives of individuals with personal experience of a prenatal diagnosis of congenital heart defect in the fetus. Specifically, we set out to address the following research questions:

    • How do individuals with personal experience of a prenatal diagnosis of congenital heart defect in the fetus assess patient information websites about congenital heart defects with regard to appearance, details, relevance, suitability, information about treatment choices, and overall quality?
    • Are there any differences in the aforementioned assessments between (1) assessors with continued and terminated pregnancy, and (2) websites about congenital heart defects?
    • What perspectives do individuals with experience of a prenatal diagnosis bring up when accessing these websites?

    Study Context

    In Sweden, all expectant parents are offered a routine ultrasound examination at approximately 18 weeks of gestation. One of the purposes of the examination is to assess the fetal anatomy, to detect or rule out possible fetal anomalies. Swedish law permits termination of pregnancy up to 18 completed weeks of gestation. At later gestations, approval must be granted by the National Board of Health and Welfare. Few pregnancies are terminated after 22 completed weeks.

    Study Design

    Through mixed-methods design with quantitative and qualitative approaches, we aspired to utilize the strengths and to offset the weaknesses of each approach [37]. Mixed-methods designs are presented as promising strategies to optimize assessments of Web-based health-related information [38].

    Patient Information Websites

    The two most popular search engines on the Web, Google and Bing [39], were used to search for Swedish patient information websites about congenital heart defects. Swedish search terms for heart defect (hjärtfel) and fetus heart defect (foster hjärtfel) were used in searches during May 2015. In total, this resulted in 291,990 hits. We performed the searches in the incognito mode of our Web browser, to minimize influence from previous search patterns. The first 20 hits of each search were screened for inclusion by the first author. In total, 80 hits were screened; nine hits were deemed irrelevant, as they did not include any information about congenital heart defects. Relevant hits that included information about congenital heart defects leading to newspaper websites (n=19), communities or blogs (n=9), websites specifically produced for health professionals (n=5), scientific papers (n=3), and password-protected websites (n=1) were excluded. After correcting for duplicate hits (n=24), 10 Swedish patient information websites about congenital heart defects were included (Figure 1).

    The first author decided which search terms to use. After the assessments, we asked the assessors, with experience of a prenatal diagnosis, which search terms they preferred to use when searching for patient information websites about congenital heart defects in connection to the diagnosis. The most commonly reported words in these search terms were heart defect (n=29 times mentioned) and fetus (n=11 times mentioned) (Multimedia Appendix 1).

    The included websites were affiliated with charities and private organizations (n=3); government, hospitals, and clinics (n=3); independent information websites (n=2); and pharmaceutical companies (n=2). Of the included websites, 9 appeared in the first 10 hits of the searches, and 8 were identified in more than one of the searches (Multimedia Appendices 2 and 3).

    ‎
    Figure 1. Search procedure to identify Swedish patient information websites about congenital heart defects. VC=virtual community or blog; HP=website specifically produced for health professionals; NP=newspaper website; PP=password-protected website; SC=scientific paper.
    View this figure

    Assessors

    Assessors were purposefully recruited [40] to strive for variation with regard to country of birth, educational level, and age. They were recruited from 2 units for fetal cardiology in Sweden. Through her clinical network, the last author was responsible for the recruitment of assessors with continued pregnancy. In total, 8 potential assessors were asked to participate; 2 declined because of fear of rekindled painful memories. When contacted for further information, 2 females declined because (1) she felt like it was enough with her partner participating in the study (n=1 potential assessor) and (2) lack of time (n=1 potential assessor). Thus, 2 females and 2 males with continued pregnancy, assessed the websites. The first author was responsible for the recruitment of persons with terminated pregnancy. Potential assessors were identified when they took part in a previous interview study that used consecutive clinical recruitment [17]. In total, 8 potential assessors were asked to participate; 1 male declined because of lack of interest. When contacted for further information, 1 female and 1 male declined because of lack of time. Thus, 3 females and 2 males with terminated pregnancy assessed the websites.

    Assessors with continued pregnancy were parents of children with congenital heart defects who were 2 (n=2 assessors) and 3 (n=2 assessors) years old. Assessors’ ages ranged between 23 and 43 years. Regarding country of birth, 3 assessors with continued pregnancy were born outside Sweden, and all with terminated pregnancy were born in Sweden. Of the assessors, 2 had senior high school and 7 had university/college as highest educational level (Multimedia Appendix 4). Assessors with terminated pregnancy had undergone the procedure 20 (n=1 assessor), 24 (n=2 assessors), 27 (n=1 assessor), and 38 (n=1 assessor) weeks before assessments. Of the assessors, 4 with continued (n=2 assessors) and terminated (n=2 assessors) pregnancy were couples, and so experienced the same prenatal diagnosis. Assessors with continued pregnancy had experience of ventricular septal defect (n=2 assessors) or a combination of transposition of the great arteries, ventricular septal defect, and pulmonary stenosis (n=2 assessors). Assessors with terminated pregnancy had experience of atrioventricular septal defect with associated trisomy 21 (n=2 assessors), aortic stenosis (n=2 assessors), or a combination of Ebstein’s anomaly and multiple structural malformations (n=1 assessor).

    Data Collection

    Quantitative Data Collection

    During 2015, individual written assessments were performed by the assessors for each of the included website. A quality assessment tool was developed (Table 1), inspired by a previous study within fetal cardiology [41]. The tool included 6 questions regarding each website’s appearance, details, relevance, suitability, information about treatment choices, and overall quality. Answers were given on a 5-point Likert scale, ranging from 1, representing lowest score, to 5, representing the highest score. A score of 3 represented quality criterion partly fulfilled, neither unsuitable nor suitable, and moderate overall quality. With 9 assessments for each of the 10 websites, we received a total of 90 assessments for each question in the quality assessment tool. In total, 8 assessors completed the assessments during a 4-hour workshop and 1 assessor at his home. Assessors were instructed to assess each included website separately and to access the sections that they found relevant to the subject of congenital heart defects when faced with a prenatal diagnosis (irrespective of which heart defect they had experienced). Because we aimed to collect assessments grounded in the perspectives of the intended consumers of the included websites, we did not provide further specific instructions to the assessors.

    Qualitative Data Collection

    Each question in the quality assessment tool was supplemented by a written open-ended question asking whether the assessors had any opinions related to the specific question. Additionally, assessors were asked to leave written comments regarding their perspectives on the positive and negative aspects of each website. Two focus group discussions were held regarding perspectives on the included websites, one with assessors who continued (n=4 assessors) and one with assessors who terminated (n=5 assessors) the pregnancy. The discussions were performed with the aid of a computer connected to a projector and were digitally recorded with audiovisual screen recordings. The first author moderated the discussions, and the second author attended the sessions as an observer. The websites were accessed one at a time, and assessors were encouraged to freely discuss aspects perceived as relevant to them. The discussions lasted 55 min for the assessors with continued pregnancy and 46 min for the assessors with terminated pregnancy. The first author transcribed the recordings verbatim.

    Table 1. Quality assessment tool.
    QuestionScale
    Is the appearance appropriate for the target audience?1 (no); 2; 3 (partly); 4; 5 (yes)
    Is the level of detail appropriate for the target audience?1 (no); 2; 3 (partly); 4; 5 (yes)
    Is the content of the website relevant?1 (no); 2; 3 (partly); 4; 5 (yes)
    How suitable is the website as a source of information following a detection of congenital heart defect in the fetus?1 (very unsuitable); 2; 3 (neither unsuitable nor suitable); 4; 5 (very suitable)
    Is it clear that more than one treatment choice exists?1 (no); 2; 3 (partly); 4; 5 (yes)
    How is the overall quality of the website as a source of information following a detection of congenital heart defect in the fetus?1 (low); 2; 3 (moderate); 4; 5 (high)

    Data Analysis

    Quantitative Data Analysis

    Quantitative data were analyzed with R version 3.2.2. (R Foundation for Statistical Computing, Austria). Kendall’s coefficient of concordance W was used to determine interrater reliability, with W ≥.21 representing fair, W ≥.41 moderate, W ≥.61 substantial, and W ≥.81 almost perfect concordance [42]. The median scores of all 10 included websites were calculated for each assessor, and the Mann-Whitney U test was used to compare these with the groups of assessors who continued and terminated the pregnancy. Friedman’s test was used to compare the scores of the websites, and the Wilcoxon- Nemenyi-McDonald-Thompson test was used as a posthoc test to investigate possible differences between the scores of specific websites. P values of <.05 were considered statistically significant.

    Qualitative Data Analysis

    Qualitative data were analyzed with Nvivo for Mac version 11.4.0. (QRS International Pty Ltd., Australia). The written responses to the open-ended questions and the transcripts of the focus group discussions were analyzed with inductive qualitative manifest content analysis, inspired by the outline presented by Graneheim and Lundman [43]. The materials were read repeatedly to gain an overall understanding. Meaning units were identified, defined as words, sentences or paragraphs containing aspects related to each other through a common content and context. Meaning units were condensed, that is, shortened without losing content and context. Condensed meaning units were each assigned descriptive codes, functioning as labels of the content. Codes were structured into categories, defined as collections of codes that shared a similar commonality with internally homogenous and externally heterogeneous manifest content, that is, the visible content described with as little interpretation as possible [43].


    Quantitative Results

    Inspecting all of the assessments (n=90), the highest proportion of the lowest score (1) was found for treatment choices (n=35, 39% of total assessment scores). By contrast, the highest proportion of the highest score (5) was found for relevance (n=16, 18% of total assessment scores); see Figure 2.

    For all assessors, the interrater reliability ranged between 0.18 and 0.40 for the 6 questions in the quality assessment tool. Interrater reliability ranged between 0.36 and 0.66 for assessors with continued pregnancy and between 0.29 and 0.42 for assessors with terminated pregnancy. With the exception of treatment choices (median=2.0), all questions had a median total score of 3.0, representing moderate quality (Table 2). There were no significant differences between median scores of the assessments from assessors with continued and terminated pregnancy. Figure 3 presents the distributions of the assessment scores among the assessors with continued and terminated pregnancy.

    Figure 4 presents the distributions of the assessment scores for each of the included websites. No website had a median score of 5.0, representing high quality, for any of the questions in the quality assessment tool. Median scores of 1.0, representing low quality, were found in questions about treatment choices (n=4 websites), details (n=2 websites), suitability (n=1 website), and overall quality (n=1 website). Website 10 had median scores of 1.0 for 4 of the 6 questions.

    Table 2. Interrater reliability assessment scores for the questions in the quality assessment tool, with distributions of the assessment from assessors with continued pregnancy (n=40 assessments) and terminated pregnancy (n=50 assessments).
    QuestionsContinued pregnancyTerminated pregnancyTotal

    		Interrater reliabilityaMedianInterquartile rangeInterrater reliabilityaMedianInterquartile rangeInterrater reliabilityaMedianInterquartile range
    Appearance0.363.020.343.020.233.02
    Details0.603.030.423.020.383.03
    Relevance0.543.010.323.010.313.01
    Suitability0.663.020.423.010.403.02
    Treatment choices0.433.020.322.020.182.02
    Overall quality0.573.020.292.510.313.01

    aKendall’s coefficient of concordance W.

    ‎
    Figure 2. Distribution (percentage) of all assessment scores (n=90) for each question in the quality assessment tool. 1=low quality, 5=high quality.
    View this figure
    ‎
    Figure 3. Distributions and medians (horizontal lines in dotplots) of the assessment scores from each of the assessors with continued (n=4) and terminated (n=5) pregnancy. 1=low quality; 5=high quality; C=continued pregnancy; T=terminated pregnancy.
    View this figure
    ‎
    Figure 4. Distributions and medians (horizontal lines in dotplots) of the assessment scores for each of the websites included (n=10), with significant differences between websites indicated with brackets above plots. 1=low quality; 5=high quality; W=website; *P<.05.
    View this figure
    Appearance

    Friedman’s test revealed significant differences between websites (χ29=20.9, P=.01). Post hoc revealed significantly lower score for website 10 than website 3 (R=43.5, P=.008).

    Details

    Friedman’s test revealed significant differences between websites (χ29=33.7, P<.001). Post hoc revealed a significantly lower score for website 6 than websites 1 (R=38.0, P=.05) and 2 (R=39.0, P=.04), for website 9 than websites 1 (R=45.5, P<.001) and 2 (R=46.5, P=.002), and for website 10 than websites 1 (R=45.0, P=.005) and 2 (R=46, P=.003).

    Relevance

    Friedman’s test revealed significant differences between websites (χ29=28.9, P<.001). Post hoc revealed a significantly lower score for website 10 than websites 1 (R=44.5, P=.004), 2 (R=47.5, P=.002), 3 (R=39.5, P=.02), and 5 (R=37.5, P=.045).

    Suitability

    Friedman’s test revealed significant differences between websites (χ29=35.7, P<.001). Post hoc revealed significantly lower score for website 10 than websites 1 (R=54.5, P<.001), 3 (R=51.5, P<.001), and 5 (R=39.5, P<.04).

    Treatment Choices

    Friedman’s test revealed significant differences between websites (χ29=17.7, P=.04). Post hoc revealed no significant differences.

    Overall Quality

    Friedman’s test revealed significant differences between websites (χ29=28.1, P<.001). Post hoc revealed significantly lower score for website 10 than websites 1 (R=39.5, P=.03), 2 (R=47.0, P<.002), and 3 (R=42.0, P<.01).

    Qualitative Results

    Qualitative analysis generated six categories: (1) advertisements, (2) comprehensiveness, (3) design, (4) illustrations and pictures, (5) language, and (6) trustworthiness.

    Advertisements

    Advertisements made it difficult to recognize relevant information and were considered inappropriate in the context of prenatal diagnosis. When advertisements were used, the assessors regarded websites as less serious.

    Children’s party [in an advertisement]! Not really the first thing that comes to mind…It was more like that disturbed me.
    [Male 3, continued pregnancy]
    Comprehensiveness

    Overall, medical information about the normal cardiovascular system and specific heart defects were regarded as sufficient and of good quality. Statistics, as well as information about postnatal care and quality of life, were appreciated. However, assessors pointed out a lack of information about the postnatal situation for families and inclusion of irrelevant topics. Assessors in both groups appreciated stories of previous cases, considered these to provide realistic insights into postnatal situation. However, those with terminated pregnancy pointed out a lack of stories about previous cases that terminated the pregnancy.

    It doesn’t say anything about how you would perform the surgery and nothing about other types of knowledge necessary about related subjects, how it is for the family...How will the child feel, how will it affect the parents.
    [Female 5, continued pregnancy]

    Information was considered belittling regarding the consequences of living with a severe heart defect, portraying an overly optimistic view of treatment and prognosis of severe defects. Balanced information about the alternatives to continue or terminate the pregnancy was considered lacking, and assessors with terminated pregnancy pointed out that there was insufficient information about induced abortions. The assessors observed that only 2 websites mentioned the possibility of terminating the pregnancy, but these were considered too simplistic and to lack relevant information about the actual process associated with induced abortion.

    Doesn’t include anything about pregnancy termination. They kind of direct the reader toward “it’s just to keep going...”
    [Male 2, terminated pregnancy]
    Design

    Clean interfaces were appreciated, with easy navigation between different subjects covered in the information. However, disorganized structures resulted in many unnecessary clicks required to continue reading, which hindered information uptake.

    I didn’t even see the information on that website because wherever you clicked you got to a new window and were transferred to another [website]. So for me, I did not even get as far as you did because I got so annoyed with the website.
    [Female 1, terminated pregnancy]
    Illustrations and Pictures

    Clear and easy-to-read illustrations of the anatomy of the normal cardiovascular system and specific heart defects were appreciated and considered to promote information comprehension, and website quality was regarded as lower when websites lacked these tools. Assessors described other illustrations as irrelevant, unsuitable, and complex. Those with terminated pregnancy mentioned that use of pictures portraying children were inappropriate for the context of prenatal diagnosis. In contrast, those with continued pregnancy appreciated pictures of hospitalized children, as this was considered to promote realistic expectations of the postnatal situation.

    One thing that I just don’t appreciate is when you go into these [websites], and you see a picture of [a child]...Because that becomes, how would our child have looked like [...] You don’t want that picture in the back of your head.
    [Male 2, terminated pregnancy]
    Language

    Assessors appreciated websites that used understandable, easy-to-read language with comprehensible explanations and the option to read more by clicking on complex words. However, they described that websites used complex language with unnecessary medical terminology, which for them made it seem like the information was produced for health professionals rather than for laypersons.

    Maybe a little too complicated in the text.
    [Male 3, continued pregnancy]
    Trustworthiness

    Websites were considered trustworthy when information was updated, included references, and had specialist health professionals as authors of medical content. When one or more of these criteria were lacking, assessors raised suspicions about the information’s accuracy. Websites from pharmaceutical companies were considered to be less trustworthy because of the possibility that they were profit driven. One website contained a disclaimer that the content had not been professionally reviewed, which made assessors frustrated as they considered themselves incapable of judging the accuracy. Another website had not been updated in more than a decade, which was considered a serious shortcoming.

    One thing I reacted upon when I sat and read, when you accessed certain websites, you saw last time updated in 2002, and then it didn’t feel very current.
    [Male 2, terminated pregnancy]

    Principal Findings

    This study aimed to explore the quality of patient information websites about congenital heart defects and to provide new insights into these sources from the perspectives of the intended consumers. To the extent of our knowledge, this study is the first to use individuals with experience of a prenatal diagnosis as assessors to explore the quality of information websites about congenital heart defects. Overall, the investigated quality criteria were quantitatively assessed as either partially or not fulfilled. Various issues were described during discussions, indicating a need for further improvement of Web-based information about congenital heart defects. The suitability was assessed as neither unsuitable nor suitable, and the assessors raised issues regarding untrustworthy information belittling regarding the consequences of living with a severe heart defect. All of these aspects are known risks with Web-based information [23].

    Parents of children with congenital heart defects want information about appropriate websites in connection to a prenatal diagnosis [17,44], but a previous study suggests that few cardiologists provide these recommendations [45]. According to the assessments in this study, some websites are particularly poor choices as sources for supplemental Web-based information. For example, one website affiliated with a clinic, identified in three searches and placed high in the search rank, had significantly lower assessments across five of the six quality criteria. Combined, the quantitative and qualitative findings indicate a need for health professionals to acknowledge the existing quality deficits and varied quality of available websites and bring this subject up for discussion during consultations with individuals who plan to use it for supplemental information. These efforts are even more important in light of the reports that consumers of health information use suboptimal search strategies [28], are unsuccessful at finding satisfactory Web-based information about health-related topics [46], and rarely discuss Web-based information with health professionals [22]. However, we acknowledge that health professionals may lack the time needed to feel adequately updated about available Web-based sources [24,47] and experience challenges when consulting patients who read Web-based information [47,48]. To adequately meet these issues, there seems to be a need for systematic efforts from overarching institutions in charge of publishing such recommendations.

    Our findings indicate a lack of Web-based information about pregnancy termination and a bias of information toward pregnancy continuation in websites about congenital heart defects. It is crucial that individuals faced with a prenatal diagnosis reach informed decisions on whether to continue or terminate the pregnancy. Research suggests that persons faced with a prenatal diagnosis need information about the option to terminate the pregnancy and use the Web for supplemental information [17]. However, few English websites about congenital heart defects contain this type of information, which the findings of this study also confirm for Swedish settings [41]. Preparatory information is a key aspect to quality abortion care [49], and previous studies report that patients have unanswered questions and feel unprepared for the abortion process [50-52]. It is of great concern that patients experience information about pregnancy termination as insufficient and high-blown, both from health professionals and from Web-based sources [17,50]. Moreover, previous studies indicate that websites about induced abortions contain inaccurate, misleading [53,54], unsuitable, and low-quality information [55], calling attention to the need for patients to cautiously use and interpret Web-based information about these topics. In light of our findings, health professionals must consider these aspects when consulting individuals who decide on pregnancy termination and must make efforts to ensure that patients come into contact with sufficient and relevant high-quality Web-based information. The findings illustrate the importance of coordinated informational approaches between units for fetal cardiology and fetal medicine.

    The complex language and unnecessary medical terminology described by the assessors raises concerns regarding the readability of these sources. Readability levels above that recommended to the general, nonmedical audience have been observed among websites about noninvasive prenatal testing [56] and other pregnancy-related topics [57,58], echoing our findings. Illustrations are possible pedagogic tools to overcome readability issues and are especially important when offering information to individuals with low health literacy [59], that is, the degree to which individuals have the capacity to obtain, process, and understand health information [60]. Assessors appreciated clear and easy-to-read illustrations, described as beneficial to promote comprehension. However, they also pointed toward the use of complex and irrelevant illustrations, further illustrating the problems related to poor readability. In light of these findings, we recommend the use of supplemental high-quality illustrations when offering information about congenital heart defects, but efforts need to be made to ensure that they are understandable to the intended consumers and have a clear purpose. Moreover, website developers need to take steps to improve the readability of their Web-based information about congenital heart defects.

    Strengths and Limitations

    The strength of this study is that we used mixed-methods design [38] to quantitatively investigate a set of variables while still remaining open to other perspectives with an inductive qualitative approach [61]. Assessors with personal experience assessed the websites, with experience of both continued and terminated pregnancy following a prenatal diagnosis. Typically, studies investigating the quality of information websites about reproductive health issues use a limited number of researchers or health professionals as assessors, frequently between 1 and 4 assessors [41,53,56,62-64]. In light of the identified differences in perspectives between researchers, health professionals, and patients [35,36], we argue that by using 9 laypersons with personal experience as assessors, our study provides insight that may more closely correspond to the views and preferences of the intended consumers of the patient information.

    We used both female and male assessors with experience of either continuation or termination of pregnancy, which implies that the assessments include views and preferences of laypersons with various backgrounds and experiences. However, considering the variations in severity, prognosis, and treatment of the possible heart defects that can be prenatally diagnosed today [65,66], we acknowledge that the assessment of our assessors may differ from laypersons with experiences of other prenatal diagnoses. The interrater reliability ranged between 0.18 and 0.66, with higher coefficients of concordance for assessors who shared similar pregnancy outcome. This indicates at least fair to moderate concordance between the assessors used in this study [42]. Moreover, we did not use any validated instruments to assess the websites, as these include an extensive number of items [38]. We urge a need for development of less extensive alternatives, suitable for the purpose of quality assessment that uses laypersons as assessors.

    We did not collect any information concerning the literacy levels of the assessors. It is possible that their levels were above or below the population norms among the intended consumers. Consequently, we cannot make any claims regarding their literacy levels and their representativeness. The results need to be interpreted with this in mind.

    We wanted the focus group discussions to reflect the topics of importance for the assessors and not be colored by our preconceptions as researchers. Thus, no discussion guide was used to guide the discussions, and the assessors were encouraged to freely discuss aspects perceived as relevant to them. We argue that this approach was an appropriate way to stay close to the perspectives of the assessors.

    Swedish websites were included, an important aspect to consider in the context of prenatal diagnosis, which differs in routines and legislation between countries. However, previous assessments of English websites with a health professional or researcher as an assessor indicate that the identified problems in this study transcend various settings and are similarly perceived by professionals and laypersons [41]. It is important to bear in mind that consumers rarely search beyond the first 10 hits in searches performed in search engines [28]. Consequently, we argue that strengths of this study are that we screened the first 20 hits and performed the searches in the most commonly used search engines [39]. Moreover, 24 of the 80 screened hits were duplicates, indicating that we achieved saturation in the inclusion of available websites. The first author chose the search terms. Consequently, it is possible that these do not fully correspond to those used by the intended consumers. However, after the data collection, we asked the assessors which search terms they preferred to use when searching for patient information websites about congenital heart defects in connection to the diagnosis. The answers confirmed the used search terms (Multimedia Appendix 1), indicating adequate representativeness in relation to those search terms that are used among the intended consumers. Moreover, the first author screened the hits and decided which hits to include as patient information websites. It is possible that laypersons would include other hits.

    Suggestions for Future Research

    We encourage researchers to conduct studies that investigate how to improve the quality of Web-based information about congenital heart defects. Additionally, more research is needed to explore how individuals with low educational levels, individuals with low health literacy, and immigrants experience these websites.

    Recommendations for Website Developers

    The results indicate that website developers must ensure that sufficient information about treatment choices are included, with balanced information concerning pregnancy termination and possible consequences of living with a severe heart defect. Efforts should be taken to include statistics, as well as information about postnatal care and quality of life. If previous cases of persons faced with prenatal diagnosis are presented, make sure to include examples of cases with continued and terminated pregnancies. Clean and easily navigated designs, without advertisements, are important aspects when developing and maintaining websites. Clear and easy-to-read illustrations of the anatomy of the normal cardiovascular system and specific heart defects are desired among the intended consumers and should be included in Web-based patient information. Details should always be provided concerning references, authorship, and date of production. Website developers should strive toward continuous updates of the website and clearly state when it has been updated.

    Conclusions

    From the perspectives of the intended consumers, patient information websites about congenital heart defects do not fulfill quality criteria concerning information about treatment choices and only partly fulfill quality criteria concerning appearance, details, relevance, suitability, and overall quality. There are differences between existing websites, indicating variations in the Web-based landscape. The findings indicate that these websites include inappropriate advertisements, biased information, poor illustrations, complex language, and have issues with trustworthiness. Improvement is needed to meet the needs among the intended consumers for unbiased, complete, and trustworthy high-quality patient information websites. Combined, the findings suggest that existing patient information websites about congenital heart defects are, to a large extent, inadequate tools for supplemental information following a prenatal diagnosis.

    When counseling expectant parents faced with a prenatal diagnosis of congenital heart defect in the fetus, we encourage health professionals to initiate discussions about their intentions to use the Web. Professionals should inform these persons about the varied quality in the Web-based landscape and offer recommendations for appropriate websites that include unbiased high-quality supplemental information. We encourage website developers to avoid (1) using inappropriate advertisements and pictures, (2) publishing information that the intended consumers may consider belittling regarding the impact of living with a severe heart defect, and (3) using complex illustrations without a clear purpose. Furthermore, we encourage them to (1) include information about pregnancy termination, (2) write information in a readable and understandable language, without the use of unnecessary medical terminology, (3) include stories from persons with lived experience of continuation and termination of pregnancy, (4) continuously update their websites with accurate information written by specialist health professionals, and (4) include a list of references.

    Acknowledgments

    We would like to thank the assessors for their contributions. The study received financial support from the regional research foundation Uppsala and Örebro. Tommy Carlsson was also financed by Uppsala County Council (“ALF-medel”). The funders had no role in the study design, data collection, analysis, and interpretation in the writing of the report and in the decision to submit the paper for publication. The Regional Ethics Committee in Uppsala, Sweden, approved this study (Reference number 2014/504). The assessors received SEK 1500 as compensation for lost working hours and travel expenses.

    Authors' Contributions

    TC, UMM, and EM conceived and designed the study. TC and UMM collected the data. TC analyzed the data and wrote the first draft of the manuscript. UMM, BW, GB, OA, and EM read the draft and provided comments for purposes of revision. All authors approved the final version of the manuscript.

    Conflicts of Interest

    None declared.

    ‎

    Multimedia Appendix 1

    Number of words reported more than one time in search terms preferred by the assessors.

    PDF File (Adobe PDF File), 31KB
    ‎

    Multimedia Appendix 2

    Search rank order and outcome of the screened hits.

    JPG File, 1MB
    ‎

    Multimedia Appendix 3

    Affiliation, number of searches where identified, and search ranks for each included website.

    PDF File (Adobe PDF File), 34KB
    ‎

    Multimedia Appendix 4

    Characteristics of the assessors with continued (n=4) and terminated (n=5) pregnancy.

    PDF File (Adobe PDF File), 48KB
    1. Campbell S. A short history of sonography in obstetrics and gynaecology. Facts Views Vis Obgyn 2013;5(3):213-229 [FREE Full text] [Medline]
    2. Dolk H, Loane M, Garne E. The prevalence of congenital anomalies in Europe. Adv Exp Med Biol 2010;686:349-364. [CrossRef] [Medline]
    3. Hoffman JIE, Kaplan S. The incidence of congenital heart disease. J Am Coll Cardiol 2002 Jun 19;39(12):1890-1900 [FREE Full text] [Medline]
    4. Dolk H, Loane M, Garne E, European Surveillance of Congenital Anomalies (EUROCAT) Working Group. Congenital heart defects in Europe: prevalence and perinatal mortality, 2000 to 2005. Circulation 2011 Mar 01;123(8):841-849 [FREE Full text] [CrossRef] [Medline]
    5. Bergman G, Borgström E, Lundell B, Sonesson SE. [Improved prenatal diagnosis of congenital heart defects. A follow-up study of prenatal ultrasound screening]. Lakartidningen 2008;105(12-13):899-903. [Medline]
    6. National Institute for Cardiovascular Outcomes Research. Nicor. 2014. National congenital heart disease audit report 2010-2013   URL: https:/​/nicor4.​nicor.org.uk/​chd/​an_paeds.nsf/​9791867eff401e0d8025716f004bb8f2/​5983f27e0b3ff3b080257d5d005cec4a/​$FILE/​NCHDA%20Report%202014%20v5.​pdf [WebCite Cache]
    7. SWEDCON. UCR. 2015. Nationellt register för medfödda hjärtsjukdomar   URL: http://www.ucr.uu.se/swedcon/arsrapporter/arsrapport-swedcon-2015/download [WebCite Cache]
    8. Draper J. 'It's the first scientific evidence': men's experience of pregnancy confirmation. J Adv Nurs 2002 Sep;39(6):563-570. [Medline]
    9. Ekelin M, Crang-Svalenius E, Dykes AK. A qualitative study of mothers' and fathers' experiences of routine ultrasound examination in Sweden. Midwifery 2004 Dec;20(4):335-344. [CrossRef] [Medline]
    10. Dykes K, Stjernqvist K. The importance of ultrasound to first-time mothers' thoughts about their unborn child. J Reprod Infant Psychol 2001;19(2):95-104. [CrossRef]
    11. Sandelowski M, Barroso J. The travesty of choosing after positive prenatal diagnosis. J Obstet Gynecol Neonatal Nurs 2005;34(3):307-318. [CrossRef] [Medline]
    12. Larsson AK, Svalenius EC, Lundqvist A, Dykes AK. Parents' experiences of an abnormal ultrasound examination - vacillating between emotional confusion and sense of reality. Reprod Health 2010 Jun 14;7:10 [FREE Full text] [CrossRef] [Medline]
    13. Åhman A, Lindgren P, Sarkadi A. Facts first, then reaction--expectant fathers' experiences of an ultrasound screening identifying soft markers. Midwifery 2012 Oct;28(5):e667-e675. [CrossRef] [Medline]
    14. McKechnie AC, Pridham K, Tluczek A. Preparing heart and mind for becoming a parent following a diagnosis of fetal anomaly. Qual Health Res 2015 Sep;25(9):1182-1198. [CrossRef] [Medline]
    15. Asplin N, Wessel H, Marions L, Öhman SG. Pregnant women's perspectives on decision-making when a fetal malformation is detected by ultrasound examination. Sex Reprod Healthc 2013 Jun;4(2):79-84. [CrossRef] [Medline]
    16. Carlsson T, Bergman G, Melander Marttala U, Wadensten B, Mattsson E. Information following a diagnosis of congenital heart defect: experiences among parents to prenatally diagnosed children. PLoS One 2015 Feb 18;10(2):e0117995 [FREE Full text] [CrossRef] [Medline]
    17. Carlsson T, Bergman G, Wadensten B, Mattsson E. Experiences of informational needs and received information following a prenatal diagnosis of congenital heart defect. Prenat Diagn 2016 Jun;36(6):515-522. [CrossRef] [Medline]
    18. Menahem S, Grimwade J. Counselling strategies in the prenatal diagnosis of major heart abnormality. Heart Lung Circ 2004 Sep;13(3):261-265. [CrossRef] [Medline]
    19. Bratt EL, Järvholm S, Ekman-Joelsson BM, Mattson LÅ, Mellander M. Parent's experiences of counselling and their need for support following a prenatal diagnosis of congenital heart disease--a qualitative study in a Swedish context. BMC Pregnancy Childbirth 2015 Aug 15;15:171 [FREE Full text] [CrossRef] [Medline]
    20. Kummervold PE, Chronaki CE, Lausen B, Prokosch HU, Rasmussen J, Santana S, et al. eHealth trends in Europe 2005-2007: a population-based survey. J Med Internet Res 2008 Nov 17;10(4):e42 [FREE Full text] [CrossRef] [Medline]
    21. Fox S, Duggan M. Pew Research Center. 2013. Health online   URL: http://www.pewinternet.org/files/old-media//Files/Reports/PIP_HealthOnline.pdf [accessed 2017-08-17] [WebCite Cache]
    22. Sayakhot P, Carolan-Olah M. Internet use by pregnant women seeking pregnancy-related information: a systematic review. BMC Pregnancy Childbirth 2016 Mar 28;16:65 [FREE Full text] [CrossRef] [Medline]
    23. Cline RJ, Haynes KM. Consumer health information seeking on the internet: the state of the art. Health Educ Res 2001 Dec;16(6):671-692 [FREE Full text] [Medline]
    24. Eysenbach G, Jadad AR. Evidence-based patient choice and consumer health informatics in the internet age. J Med Internet Res 2001;3(2):E19 [FREE Full text] [CrossRef] [Medline]
    25. Broom A. Virtually he@lthy: the impact of internet use on disease experience and the doctor-patient relationship. Qual Health Res 2005 Mar;15(3):325-345. [CrossRef] [Medline]
    26. Groen WG, Kuijpers W, Oldenburg HS, Wouters MW, Aaronson NK, van Harten HW. Empowerment of cancer survivors through information technology: an integrative review. J Med Internet Res 2015 Nov 27;17(11):e270 [FREE Full text] [CrossRef] [Medline]
    27. Kreps GL, Neuhauser L. New directions in eHealth communication: opportunities and challenges. Patient Educ Couns 2010 Mar;78(3):329-336. [CrossRef] [Medline]
    28. Eysenbach G, Köhler C. How do consumers search for and appraise health information on the world wide web? qualitative study using focus groups, usability tests, and in-depth interviews. BMJ 2002 Mar 9;324(7337):573-577 [FREE Full text] [Medline]
    29. Peterson G, Aslani P, Williams KA. How do consumers search for and appraise information on medicines on the internet? a qualitative study using focus groups. J Med Internet Res 2003 Dec 19;5(4):e33 [FREE Full text] [CrossRef] [Medline]
    30. Howe D. Ethics of prenatal ultrasound. Best Pract Res Clin Obstet Gynaecol 2014 Apr;28(3):443-451. [CrossRef] [Medline]
    31. Franck LS, Noble G. Here's an idea: ask the users! Young people's views on navigation, design and content of a health information website. J Child Health Care 2007 Dec;11(4):287-297. [CrossRef] [Medline]
    32. Tractinsky N, Katz AS, Ikar D. What is beautiful is usable. Interact Comput 2000 Dec;13(2):127-145. [CrossRef]
    33. Cyr D. Modeling web site design across cultures: relationships to trust, satisfaction, and e-loyalty. JMIS 2008;24(4):47-72. [CrossRef]
    34. Coulter A, Entwistle V, Gilbert D. Sharing decisions with patients: is the information good enough? BMJ 1999 Jan 30;318(7179):318-322 [FREE Full text] [Medline]
    35. Tallon D, Chard J, Dieppe P. Relation between agendas of the research community and the research consumer. Lancet 2000 Jun 10;355(9220):2037-2040. [CrossRef] [Medline]
    36. Hordern AJ, Street AF. Communicating about patient sexuality and intimacy after cancer: mismatched expectations and unmet needs. Med J Aust 2007 Mar 5;186(5):224-227. [Medline]
    37. Watkins D, Gioia D. Mixed methods research. Oxford, England: Oxford University Press; 2015.
    38. Beaunoyer E, Arsenault M, Lomanowska AM, Guitton MJ. Understanding online health information: evaluation, tools, and strategies. Patient Educ Couns 2017 Feb;100(2):183-189. [CrossRef] [Medline]
    39. Ebizmba. 2017. Top 15 most popular search engines   URL: http://www.ebizmba.com/articles/search-engines [accessed 2017-04-08] [WebCite Cache]
    40. Patton M. Qualitative research and evaluation methods. Thousand Oaks, CA: Sage Publications; 2002.
    41. Carlsson T, Bergman G, Karlsson AM, Mattsson E. Content and quality of information websites about congenital heart defects following a prenatal diagnosis. Interact J Med Res 2015 Jan 21;4(1):e4 [FREE Full text] [CrossRef] [Medline]
    42. Gisev N, Bell JS, Chen TF. Interrater agreement and interrater reliability: Key concepts, approaches, and applications. Res Social Adm Pharm 2013;9(3):330-338. [CrossRef]
    43. Graneheim UH, Lundman B. Qualitative content analysis in nursing research: concepts, procedures and measures to achieve trustworthiness. Nurse Educ Today 2004 Feb;24(2):105-112. [CrossRef] [Medline]
    44. Arya B, Glickstein JS, Levasseur SM, Williams IA. Parents of children with congenital heart disease prefer more information than cardiologists provide. Congenit Heart Dis 2013;8(1):78-85 [FREE Full text] [CrossRef] [Medline]
    45. Hilton-Kamm D, Sklansky M, Chang RK. How not to tell parents about their child's new diagnosis of congenital heart disease: an internet survey of 841 parents. Pediatr Cardiol 2014 Feb;35(2):239-252. [CrossRef] [Medline]
    46. Zeng QT, Kogan S, Plovnick RM, Crowell J, Lacroix EM, Greenes RA. Positive attitudes and failed queries: an exploration of the conundrums of consumer health information retrieval. Int J Med Inform 2004 Feb;73(1):45-55. [CrossRef] [Medline]
    47. Ahmad F, Hudak PL, Bercovitz K, Hollenberg E, Levinson W. Are physicians ready for patients with internet-based health information? J Med Internet Res 2006 Sep 29;8(3):e22 [FREE Full text] [CrossRef] [Medline]
    48. Sommerhalder K, Abraham A, Zufferey MC, Barth J, Abel T. Internet information and medical consultations: experiences from patients' and physicians' perspectives. Patient Educ Couns 2009 Nov;77(2):266-271. [CrossRef] [Medline]
    49. Dennis A, Blanchard K, Bessenaar T. Identifying indicators for quality abortion care: a systematic literature review. J Fam Plann Reprod Health Care 2017 Jan;43(1):7-15. [CrossRef] [Medline]
    50. Mukkavaara I, Öhrling K, Lindberg I. Women's experiences after an induced second trimester abortion. Midwifery 2012 Oct;28(5):e720-e725. [CrossRef] [Medline]
    51. Andersson IM, Christensson K, Gemzell-Danielsson K. Experiences, feelings and thoughts of women undergoing second trimester medical termination of pregnancy. PLoS One 2014 Dec 29;9(12):e115957 [FREE Full text] [CrossRef] [Medline]
    52. Carlsson T, Bergman G, Karlsson AM, Wadensten B, Mattsson E. Experiences of termination of pregnancy for a fetal anomaly: a qualitative study of virtual community messages. Midwifery 2016 Oct;41:54-60. [CrossRef] [Medline]
    53. Bryant AG, Narasimhan S, Bryant-Comstock K, Levi EE. Crisis pregnancy center websites: information, misinformation and disinformation. Contraception 2014 Dec;90(6):601-605. [CrossRef] [Medline]
    54. Rowlands S. Misinformation on abortion. Eur J Contracept Reprod Health Care 2011 Aug;16(4):233-240. [CrossRef] [Medline]
    55. Carlsson T, Axelsson O. Patient information websites about medically induced second-trimester abortions: a descriptive study of quality, suitability, and issues. J Med Internet Res 2017 Jan 10;19(1):e8 [FREE Full text] [CrossRef] [Medline]
    56. Mercer MB, Agatisa PK, Farrell RM. What patients are reading about noninvasive prenatal testing: an evaluation of internet content and implications for patient-centered care. Prenat Diagn 2014 Oct;34(10):986-993. [CrossRef] [Medline]
    57. Sacks S, Abenhaim HA. How evidence-based is the information on the internet about nausea and vomiting of pregnancy? J Obstet Gynaecol Can 2013 Aug;35(8):697-703. [Medline]
    58. Bantan N, Abenhaim HA. Vaginal births after caesarean: what does Google think about it? Women Birth 2015 Mar;28(1):21-24. [CrossRef] [Medline]
    59. Houts PS, Doak CC, Doak LG, Loscalzo MJ. The role of pictures in improving health communication: a review of research on attention, comprehension, recall, and adherence. Patient Educ Couns 2006 May;61(2):173-190. [CrossRef] [Medline]
    60. Institute of Medicine (US) Committee on Health Literacy, Nielsen-Bohlman L, Panzer AM, Kindig DA. Health Literacy: A Prescription to End Confusion. Washington, DC: National Academies Press; 2004.
    61. Elo S, Kyngäs H. The qualitative content analysis process. J Adv Nurs 2008 Apr;62(1):107-115. [CrossRef] [Medline]
    62. Rezniczek GA, Küppers L, Heuer H, Hefler LA, Buerkle B, Tempfer CB. Quality of websites of obstetrics and gynecology departments: a cross-sectional study. BMC Pregnancy Childbirth 2015 Apr 26;15:103 [FREE Full text] [CrossRef] [Medline]
    63. Fioretti BT, Reiter M, Betrán AP, Torloni MR. Googling caesarean section: a survey on the quality of the information available on the internet. BJOG 2015 Apr;122(5):731-739 [FREE Full text] [CrossRef] [Medline]
    64. Rahnavardi M, Arabi MS, Ardalan G, Zamani N, Jahanbin M, Sohani F, et al. Accuracy and coverage of reproductive health information on the internet accessed in English and Persian from Iran. J Fam Plann Reprod Health Care 2008 Jul;34(3):153-157 [FREE Full text] [CrossRef] [Medline]
    65. Allan LD, Huggon IC. Counselling following a diagnosis of congenital heart disease. Prenat Diagn 2004 Dec 30;24(13):1136-1142. [CrossRef] [Medline]
    66. Warnes CA. The adult with congenital heart disease: born to be bad? J Am Coll Cardiol 2005 Jul 05;46(1):1-8 [FREE Full text] [CrossRef] [Medline]

    Edited by TR Soron; submitted 08.04.17; peer-reviewed by G Mohan; comments to author 12.06.17; accepted 24.06.17; published 12.09.17

    Copyright

    ©Tommy Carlsson, Ulla Melander Marttala, Barbro Wadensten, Gunnar Bergman, Ove Axelsson, Elisabet Mattsson. Originally published in the Interactive Journal of Medical Research (http://www.i-jmr.org/), 12.09.2017.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Interactive Journal of Medical Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.i-jmr.org/, as well as this copyright and license information must be included.