Standardization of Questions in Rare Disease Registries: The PRISM Library Project

Introduction

The value and utility of patient registries are largely contingent upon the use of both high-quality research questions and structured and relevant answer sets accompanying them. The interoperability of registries or registry data—including the feeding of registry data into more rigorous clinical studies and regulatory submissions for new agents—depends upon the use of data standards; yet, there is currently no clear specification for developing registry questions nor are there banks of existing questions to support registry developers. The diverse nature of registries, sponsors, and disease-specific data requirements complicate efforts at standardization in registry applications. This paper introduces the [Rare Disease] PRISM (Patient Registry Item Specifications and Metadata for Rare Disease) project, a library of standardized questions covering a broad spectrum of rare diseases that can be used to support the development of new registries in all disease areas. If widely used, PRISM will enable the re-use of questions across registries, hence reducing the variation in registry data collection and facilitating a bottom-up standardization of patient registries. Although it was developed to support the rare disease community’s urgent need for shared resources, the PRISM library of patient-directed registry questions can be a valuable resource to registries for all diseases.

Background

Rare Diseases Registries

Often patient registries are a helpful first step in estimating the impact and understanding the etiology of rare diseases—both requisites for the development of new diagnostics and therapeutics. Because of the small numbers of patients affected by rare diseases, these registries present unique challenges related to registry design, enrollment of patients, and data collection [1-5], as well as for analysis and interpretation of registry data [5,6]. In recent years, the U.S. Food and Drug Administration (FDA) and umbrella patient advocacy support organizations (eg, the Genetic Alliance) have publicly encouraged rare disease patient advocacy groups (PAGs) to develop registries as part of a comprehensive strategic research plan. As such, a proliferation of patient registries is underway, and there are currently no dedicated or centralized efforts for developing standards that will reduce the time required to develop new patient registries and facilitate opportunities for shared data.

Patient Registry Variation and Standards

There is tremendous variability in the type of data and specific questions that patient registries collect, due in part to the lack of registry-specific data standards and also to the heterogeneity of registries’ purposes and sponsors. Patient registries can be designed for many purposes, including public health surveillance, epidemiologic and longitudinal research, patient education, research recruitment, and population monitoring for the safety of post-marketed drugs and devices. Patient registries can include data reported by patients, researchers, or clinicians. (A characterization of registry types is summarized in Richesson and Vehik [2].) Sponsors and developers of patient registries are varied as well and include governments, academic scientists, and clinical investigators. Often patient registries (and the supporting questions, typically targeted to patients and caregivers rather than physicians) are developed ad hoc by the PAGs themselves, and there is currently no clear specification for standards or banks of existing questions for them to access. Further, the content of these registries (ie, the questions and associated answer sets) may change over time as more becomes known about the disease and its clinical variations, or when new therapeutics and devices become available. There is a tremendous need—especially for the thousands of rare diseases that do not have patient registries—for resources that help registry sponsors and developers to identify well-constructed and meaningful questions.

There is also a clear role for data standards to promote shared efficiencies in registry development and enable opportunities for data sharing. These needs are particularly pronounced for rare diseases, which have sparse resources and significantly fewer—and highly distributed—domain experts and affected patients. An important standards challenge is the fact that there is no central control of patient registries—there is no single funding or regulatory agency that can oversee all the different registry types and implementations. Because registries are developed by many sponsors to address distinct functions, a top-down standards effort would require countless stakeholders and is not feasible. Additionally, there is no central authority to monitor or enforce standards compliance once developed. Given the tremendous need for standards and the scope of data collected across disease-specific registries, and given that there is no incentive or regulatory means to develop standards or enforce compliance, alternatives to complement traditional Standards Development Organizations (SDOs) are needed. Non-traditional strategies for developing and promoting standards can be effective and embraced across various rare diseases if these various research communities perceive them as accessible, useful, helpful, and easy to adopt.

The PRISM project (funded by an American Recovery and Reinvestment Act (ARRA) grant administered through the National Library of Medicine (NIH), NLM Grant Number 1RC1LM010455-01, and supported by the Office of Rare Diseases Research) was developed to provide a useful resource to promote the efficient development of patient registries and standardized quality data collection by supporting the sharing and re-use of existing registry questions and data standards. The fundamental idea behind PRISM is that if registry developers could access questions used by other rare disease registries, they could consider and likely re-use these questions, thereby reducing the variation in questions/data collection across various patient registries, and leading to a bottom-up development of standards. In addition, the utility and scalability of PRISM is based on the notion that the PRISM content is accessible and open to any registry sponsor, regardless of prior standards knowledge or experience. The authors are engaged with various standards and informatics organizations and deliberately designed PRISM to facilitate linkage with other standards and research resources as appropriate. The PRISM project, therefore, provides not only a library of questions but a vehicle for the registry developers and rare disease organizations to interact, learn, and develop consensus requirements, which can in turn be directed to various standards development organizations (eg, CDISC [7], HL7 [8], LOINC [9]), research initiatives (eg, caDSR [10] and CSHARE, PROMIS [11,12], PhenX [13]), and national interoperability initiatives emerging from the U.S. DHHS, Office of the National Coordinator [14,15].

As a demonstration project, PRISM explored foundational issues related to the types of questions included in the bank (relative to other standards and question repositories) and the inclusion of metadata that will facilitate their search and retrieval. We describe our methodological approach to developing the PRISM library in the Methods section and present the resulting library structure, features, and composition afterward in the Results section.

Methods

The first questions identified for PRISM included a convenience sample of questions from well-established (>5 years) natural history studies in various diseases (metabolic [16], vascular [17], developmental disorders [18]) and several existing registries. The questions were obtained largely from the NIH-funded Rare Diseases Clinical Research Network as the authors worked in the data center for the network and had familiarity with the various studies and investigators. These were examined by many experts (both terminology experts at the College of American Pathologists (CAP) and research and informatics experts at the University of South Florida (USF)) for face validity, including commonality, clarity, and organization. Questions were re-worded slightly, as needed, to make the full semantics of the question clear and to make the question generalizable to multiple diseases as possible. For example, a registry data entry item “Genetic test?” would be entered into PRISM as “Have you had a genetic test to confirm your diagnosis?”, representing the intended semantics of the question in a generalizable way, rather than “Have you had a genetic test to confirm your Rett Syndrome diagnosis?”, which was the intent of the question from the source document. In addition, some registry questions were taken from a variety of established rare disease patient registries that authors were acquainted with. To ensure that questions were “stable” and field-tested, authors used well-established content that had been considered final by a PAG after a multi-disciplinary review process and pilot testing. Generic content relative to many or all patient registries was also incorporated, including standard elements from the Rare Diseases Clinical Research Network (RDCRN) Contact Registry (supporting over 200 rare diseases) [19], the OMB and NIH demographics [20], selected questions from the NINDS Common Data Elements [21], and all of the data elements recommended for the Global Registry for Rare Diseases [22], an initiative led by the U.S. Office of Rare Diseases Research. A typology was developed to organize the questions in PRISM and to support internal curation of the library. Questions were indexed by one or more keywords that describe the general content category (eg, demographic, medications, medical history, special histories, etc.). The specific representations of the metadata keywords include common forms (eg, Demographics, Medical History) and form headings (eg, Medication History, Special Education Services) that group related data (by type or by source) for registries and observational studies.

We describe our selected strategy and design features for PRISM in the next section in terms of content, search and retrieval requirements, indexing model and metadata, and strategy for growing the content of the PRISM library. The authors thoroughly explored other standards throughout the design of PRISM and present the relationships and definitions between PRISM and other efforts related to standardized questions and patient reported data. Finally, in the discussion section, we describe immediate future directions for PRISM, including requirements for sharing the library, interface design, and future plans for maintenance and governance of PRISM.

Results

Content

At present, PRISM contains over 2,200 questions. A sample of 224 questions and selected metadata is presented in Appendix 1. Many questions (especially the most general, such as “List current medications you are taking” and “List any other major diseases you have had”) are relevant to virtually all rare diseases, and others are relevant to a great many rare diseases (eg, “Do you require an assistive device for walking?”, or “Was your child born full-term?”). However, the majority of PRISM questions—such as “Does your child hoard food?”, “How many times per week does child pick at own skin until it bleeds?”, “Does your child need TV to fall asleep?”, and “Approximately how many bone fractures have you had in your lifetime?”—are relevant only to particular diseases, and often the valid answer sets associated with each question are disease-specific. For example, valid answers to the question “What age was your child’s first bone fracture?” would include prenatal ages in a patient registry targeted to Osteogenesis Imperfecta, thereby requiring different units and ranges of expected values. The obvious challenge that thousands of such disease-specific questions (relevant to the thousands of rare diseases seeking to develop registries) will present for traditional standards development organizations is what drove the design decision for the scope of PRISM. The scope of PRISM, therefore, deliberately includes a range of disease-specific content, along with narrative definitions and metadata for indexing and source preservation. The metadata was selected to facilitate linkages between rare disease communities and existing SDOs and controlled terminologies, as described in the following sections.

Related Efforts

To prevent overlapping with other standards efforts related to the collection of clinical data, a deliberate search for relevant standardization efforts was undertaken by the authors. This search of existing standards and informatics and library resources revealed several related and potentially relevant efforts, and informed the design of PRISM to leverage related efforts. As described in the introduction, the focus of PRISM is on patient-reported questions that are not a part of standardized, validated patient assessment instruments. Given that focus, several organizations are engaged in various attempts to inventory and codify standardized assessment instruments and items, specifically Clinical LOINC [23], PROMIS, the caDSR and NINDS Common Data Elements [24], and to some extent SNOMED CT. Because PRISM was developed with the intent to leverage and coordinate existing standards as much as possible, the PRISM scope was clearly defined at the onset to prevent any overlap. However, some areas of potential overlap with other initiatives exist. Table 1 and Table 2 describe the potential relationship of PRISM to other initiatives that are also developing repositories of questions or data elements.

Some data elements, like height and weight are in clinical data element repositories such as Clinical LOINC and research data element registries like caDSR. To enable “one-stop shopping” for registry developers, these items are also in PRISM, with the idea that future PRISM interfaces can identify linkages to these other standards where and when appropriate. These linkages can inform PRISM users that they indeed are using items from another designated standard and can also inform PRISM curators to ensure that they do not create or support future variations in that item.

In the interest of rapidly assembling content relevant to patient registries for any and all rare diseases, the initial PRISM strategy has been to accept virtually all questions, with the idea that either patient communities or curators might later filter, rate, or rank them for PRISM users. Despite limiting the inclusion of questions to those from well-established registries, some questions used in rare disease registries were poorly constructed or not as clear as they could be. Because PRISM is motivated to address the registry question needs for a spectrum of registry designs and diseases, it does contain registry data elements (in actual use) that might not be ideally constructed or might actually conflict with another value set. Regardless, the liberal acceptance policy of PRISM increases the breadth and volume of questions and ultimately increases the value of PRISM as a central resource for questions (which should be supplemented by advice on selection and use). We hope that others use PRISM as a resource for standards development or build applications that can facilitate the ranking or endorsement of certain PRISM questions over others in specific diseases or contexts.

Search and Retrieval Requirements

As mentioned in the previous section, the growth of the library brought with it challenges for curation and use. Internally, a process was developed to add new content without duplicating questions and ensure that related questions or variants could be indexed for effective retrieval and comparison. We operated under the assumption that PRISM needed to be useful to provide value. We explored user roles and searching techniques to determine the best method for indexing question and answer sets (QAS). Our indexing scheme (including the use of controlled terminology) is described later. A key strategy for identifying the search and retrieval requirements was employing use cases.

The PRISM team developed narrative use cases (for question searching, re-use, and new registry development) that have informed our indexing approaches. For our indexing strategy, PRISM also developed use cases of retrieval requirements as a guide of needed functionality and detail, and leveraged existing metadata and controlled vocabularies. The use cases describe the development of new registries using PRISM as a resource for the questions. Specifically, the use cases describe the development of a prospective vasculitis and pregnancy registry—including the re-use of PRISM questions and the submission of new questions to PRISM—and the subsequent development of another registry using these same pregnancy-related questions. Finally, the use cases articulate a demonstration and initial proof of the interoperability of data collected from different registries and data sources. These functional use cases ultimately will also support the evaluation of PRISM. The use cases were developed by the authors and not widely vetted in the rare disease community. They can be found on PRISM website [25].

Indexing Model and Metadata

A critical and largely unaddressed problem for registries (and clinical research data collection in general) is the need for patient registry questions and answers to be indexed in such a way that they can be retrieved for re-use, for example, to support rapid development of another related rare disease registry. In essence, indexing is the practice of applying metadata (structured and descriptive information) to items in a database for efficient and accurate retrieval [26]. An important measure of indexing is specificity, which refers to the detail or precision of the indexing process and its depth [26(p161)]. Achieving the appropriate level of specificity in encoding the question and answer sets was a continuous challenge. Our approach was guided by the ideal situation where the pairs would be encoded to convey the semantic meaning of the clinical concepts and at the same time allow efficient collocation. The specificity and overall indexing approach informed the detailed use cases discussed earlier. The indexing scheme also addresses semantic ambiguities inherent in research questions and answer sets, and includes metadata for the following constructs:

1. Context (type of study, disease or treatment of interest, etc.)

2. Format of questions and location of semantics

3. Who is asking the question (patient, relative, doctor)?

4. Audience or person being asked the question (patient, family member or caregiver)

5. Relevant data standards for specific answer sets

Metadata and Controlled Vocabularies

Metadata are used to describe information resource-type features of questions, such as terms and attributes, and controlled vocabularies represent the actual content. Both are important for retrieval and, ultimately, interoperability. PRISM uses Dublin Core (DC) as a metadata framework for indexing PRISM QAS, and within DC metadata uses controlled terminology to reflect the semantics for each question. Our approach is described in detail in [27]. Terminology control, when implemented correctly and consistently, can dramatically improve the quality of search results in most contexts. SNOMED CT is an ONC recommended standard for many aspects of the electronic medical record, and previous research has indicated that SNOMED CT is also well suited for clinical concepts in research [28].

The use of Dublin Core metadata to annotate various QAS in PRISM offers a way to employ the most appropriate controlled vocabulary(s) for the content, while preserving retrievability. In addition to selected Dublin Core metadata elements and controlled terminologies, other decisions were made to ensure that these elements and vocabularies were used appropriately and consistently. Specifically, assuring each QAS is usable, reproducible, and understandable on its own merit. For example, a form that addresses gynecological issues may include a QAS addressing menstrual symptoms, such as “Do you experience cramps?” From the context of the form, this can refer only to menstrual cramps and may be coded with or without the “menstrual semantics”. However, when a later user searches the registry for library questions about abdominal, leg or other body site cramps, this question may be inappropriately selected. The QAS metadata (including the embedded SNOMED CT codes and the narrative definition of the question that PRISM includes) can be used to easily disambiguate this term.

Under the leadership of terminology experts from the College of American Pathologists, guidelines for using SNOMED CT were developed related to post-coordination, selection of hierarchies, and level of specificity. Guidelines for the use of SCT within the PRISM Library data model were developed collaboratively by the PRISM team and included the following decisions:

1. Assigning codes for entire question and answer set groups vs. discrete codes for questions and answers

2. An approach to take the (semantically) closest available SNOMED CT concept rather than creating a new one

3. Consequently, we considered but rejected the idea to create a SNOMED extension (“Ref Set”) mechanism

4. Versioning and change protocols were developed between USF and CAP partners

Procedures for Adding New Items / Growing PRISM content

As described in the previous section, PRISM has developed a useful, consistent, and standards-compliant solution for the encoding of questions in PRISM. Ultimately, the metadata model and indexing strategy will be tested as the content of PRISM grows. Implementation plans should ensure that as the size of the PRISM library grows, duplicate questions are not inadvertently added. (Anecdotally, this is an issue with other question and metadata repositories, owing to the fact that a complete search of existing content must be undertaken before new content is added, and this search is both time-consuming and generally not incentivized.) For PRISM to remain a useful resource to rare disease registry developers, and for PRISM to support the goal to reduce question variation across rare disease registries, the library should not contain obvious duplicates and the library indexing model should be sufficient for users to search and retrieve relevant registry questions from PRISM before they re-create their own variation. As new content is introduced to the PRISM outreach team at USF, they first determine if the content is in scope of PRISM (ie, is not purview of other standards initiatives shown in Table 1 and Table 2). Then, as questions are identified for possible addition into the library, library professionals perform multiple and semantically enhanced queries to compare to QAS already in the database. Through collaborations with rare disease stakeholders who are part of the PRISM scientific review committee, they are then edited for structure and clarity, and definitions are provided. After the questions have been edited, semantic and administrative metadata are applied (including controlled terminology) before integrating them into the library’s database.

Discussion

Limited Scope

One of the biggest challenges of the PRISM project has been to keep the scope reasonable and practical. Since the targeted audience for PRISM is researchers and registry developers (including PAGs with non-research backgrounds), we took a minimalist approach to coding with the goal of easy retrieval. It is our expectation that easy retrieval will drive increased usage, which will cultivate de facto standards, and those standards will ultimately support interoperability (see Figure 1).

We recognize that interoperability between registry questions and other data (eg, EHR data) would require more sophisticated coding with SNOMED CT and other data standards. Given the short duration of this project and the desire for maximum retrievability, we determined that this level of coding would be out of the scope for PRISM at this time.

It is not clear at this point how large a corpus of sharable items that there is among disparate and different rare diseases. Likewise, it is still unclear whether items for rare diseases are likely to be similar to items for more common diseases, and if so, whether there would be value in finding a way to include and reuse those items via PRISM as well. The future use and evaluation of PRISM content by multiple disease representatives will yield information on the reusability of the questions in PRISM within and across rare and common diseases, as well as provide practical examples of cross-disease standards and determination of standards gaps. To understand the reusability and generalizability of PRISM questions, additional work needs to explore the validity and reproducibility of the categorization and indexing of questions.

‎View this figure

The Future

PRISM fills a void for the rare diseases research and registry development communities. The PRISM library resource can support standardized data collection in patient registries by reducing unnecessary variation. PRISM is free and available to search through the project website [25]. Authors are also in the process of making the PRISM library metadata model and content accessible through the National Center for Biomedical Ontology [29] Bioportal. While the library content is available, we encourage innovators and developers to build tools that integrate PRISM content into registry development efforts. It is possible that the PRISM library resource could be leveraged in large research data collection tools such as the NIH-funded REDCap project.

Authors are hopeful for future funding that will allow PRISM content to grow to meet the needs of the thousands of rare diseases registry applications and to allow computer mediated methods for adding and presenting content. The notion of using a distributed community of registry developers to curate this resource by commenting on and ranking items—as with the demonstration of caDSR content that is described in [30]—is very appealing. Such projects would require extensive marketing and publicity of PRISM to a comprehensive group of rare disease registry stakeholders and researchers in order to bolster the extensive use of PRISM that would be required to effectively demonstrate a bottom-up community driven development of standards. An increased usage and future growth of PRISM will in turn require formal governance for PRISM and official policies related to content and a submissions and update processes.

The PRISM developers, with the cooperation and support of the National Organization for Rare Disorders (NORD), are working to make the PRISM resource available and useful to registry developers representing all rare diseases and all countries. Currently, several rare disease patient advocacy organizations are participating in focus groups and expert interviews to inform the development of best interfaces and retrieval strategies to ensure that PRISM is a useful and accessible community-driven resource. In addition, we are developing international collaborations to explore the translation of items to support global rare disease research. Our overarching goal is that—given the sheer number of rare diseases, the variety of registry designs, and the number of languages that might need to be addressed—the PRISM leadership seizes and implements standards opportunities without burdening resource-strapped rare disease communities.

Summary

The lack of a clear set of standards and specifications for data collection using patient registries represents a significant data standards gap in an explosively growing application area—important to both drug development and patient-directed health communities. Standardization of patient registries can enable the interoperability of health and research data, as registries should be able to receive data from health care system or transmit data into various clinical research or pharmacovigilance applications. PRISM can be used to facilitate interoperability of existing and newly developed registries and to ensure that moving forward, registries use standard sets of questions. Without the use of such a resource, the proliferation of patient registries and variation of data collection questions will be inevitable. This central resource, the PRISM library, will support a bottom-up and incremental standards promulgation. By using a standard set of metadata elements and SNOMED CT to facilitate the retrieval and re-use of existing questions and standards, PRISM will reduce variation in the rare diseases registry community and assist registry implementers to produce high quality registries much more efficiently than ever before. Once variation in patient registries is reduced (ie, “standards” emerge), then issues related to harmonizing, mapping, and relating to the different standards communities for health care (eg, HL7) and research (eg, CDISC) can be addressed in an efficient manner. In this approach, the standardization of patient registry questions can serve to improve efficiencies, collaboration, and resource sharing across the entire drug development process.

Information regarding the development of PRISM and access can be found on the PRISM website [24] .