Challenges in Systematic Reviews: Synthesis of Topics Related to the Delivery, Organization, and Financing of Health Care

Quantitative systematic reviews have traditionally focused on estimating the effectiveness of specific therapeutic interventions, assessing the accuracy of diagnostic tests, and, to a lesser extent, quantifying epidemiologic relationships (1-3). Such meta-analyses lend themselves to clear formulation of problems with research questions such as the following: Does a particular intervention increase survival in patients with a particular diagnosis? What are the sensitivity and specificity of a given diagnostic test? What is the probability that patients with a particular risk factor are likely to develop a given disease? The literature searches, development of inclusion and exclusion criteria, and selection of quantitative methods for combining evidence follow readily from such research questions. By contrast, systematic reviews of some important policy issues, such as those related to the delivery, organization, and financing of health care, present 3 key challenges to established methods of systematic reviews (4). First, such reviews may require considerable effort at the problem formulation stage to narrow the scope of the question to one that can feasibly be addressed and to define appropriate inclusion and exclusion criteria that facilitate literature searches. They may address research questions such as the following: What is the relative effect of private for-profit versus private not-for-profit delivery of care on patient outcomes? Which strategies are the most effective for promoting preventive care? Second, they may not lend themselves to straightforward literature searches, as some of the literature addressing these complex questions falls outside the traditional medical literature and may be difficult to retrieve (5). For example, the relevant literature may consist principally of institutional reports, white papers, and other grey literature, which are often not well cataloged in electronic databases. Third, the included articles may be highly heterogeneous with respect to study design (including the key reported outcomes) and quality. This can lead to difficulties appraising and interpreting the evidence and can severely limit opportunities to synthesize the data quantitatively, as well as qualitatively. There is increasing interest in using high-quality systematic reviews of topics related to the delivery, organization, and financing of health care. For example, economic evaluations of technologies are increasingly being used to inform coverage decisions by organizations such as the National Institute for Clinical Excellence (NICE) in the United Kingdom and the Australian Government’s Pharmaceutical Benefits Advisory Committee (PBAC) (6). Several research groups are developing expertise in performing high-quality systematic reviews on these topics. Notably, within the Cochrane Collaboration, the Cochrane Effective Practice and Organisation of Care Group has performed numerous reviews on related topics (7, 8). The Agency for Healthcare Research and Quality (AHRQ) has increasingly asked its Evidence-based Practice Centers to synthesize evidence on complex, policy-related topics that do not lend themselves to traditional methods of evidence synthesis. In this review, we discuss innovative methods used by Evidence-based Practice Centers and other researchers to synthesize evidence related to the study of the organization, delivery, and financing of health care. We reviewed 94 Evidence-based Practice Center reviews and other published systematic reviews to identify reviews that specified their method for problem formulation, discussed the challenges of using nontraditional or medical literatures, and synthesized broad-based literature relevant to health care delivery, organization, or financing. In this supplement article, we review approaches used in these systematic reviews for problem formulation, data collection, and analysis and interpretation. Methodologic Innovations Described in Published Reviews We found 13 published reviews of broad topics related to the delivery, organization, and financing of health care that specified their methods for problem formulation, data collection, and data analysis and interpretation. The topics meeting these criteria broadly included patient safety (9-11), quality improvement (12, 13), bioterrorism preparedness (14, 15), health promotion (16), health care financing (17-19), telemedicine (20), and health literacy (21). We present the results of our review in the Appendix Table. In the sections that follow, we summarize innovative approaches to solving the unusual methodologic challenges characteristic of these types of evidence synthesis topics. We have organized these sections according to 3 of the stages of evidence synthesis from the Cooper model of research synthesis: problem formulation, data collection, and data analysis and interpretation (22). Challenge: Formulating Research Questions and Inclusion Criteria The problem formulation stage has 3 research objectives: development of the key research questions, construction of the definitions of the key concepts that distinguish relevant from irrelevant studies, and the establishment of inclusion and exclusion criteria for the review (22). In general, given the complexity and breadth of the topics addressed in the included reviews, the authors often described the use of conceptual frameworks for problem formulation. Specifically, they often adapted theoretical frameworks from related fields to develop conceptual models that they then used to specify key research questions and direct their literature searches. Systematic Methods for Defining Concepts and Inclusion Criteria The evidence related to health care delivery, organization, and financing often describes complex systems processes. Without clearly stated definitions of the processes of interest, it can be difficult to identify the research questions, develop appropriate literature searches, and formulate inclusion and exclusion criteria. Thus, for some projects, the essential task of the problem formulation stage is the defining of relevant concepts. For example, in the review Measures of Patient Safety Based on Hospital Administrative Data: The Patient Safety Indicators, the authors evaluated patient safety indicators developed from administrative data (9). They began by developing a set of standardized definitions of 9 key concepts relevant to their topic, including quality of health care, quality indicators, patient safety, patient safety indicators, medical error, and adverse events (9). They used these definitions to develop a list of potential indicators of medical errors, patient safety, or potentially preventable complications (9). The authors then invited 21 professional clinical organizations and 16 surgical subspecialty organizations to submit nominations of experts to evaluate potential patient safety indicators. An expanded expert advisory group of 76 clinicians (divided into 11 panels) reviewed and evaluated indicators according to a method adapted from the RAND/UCLA Appropriateness Method (23). This consisted of an initial independent assessment of each indicator by clinician panelists using an initial questionnaire, a conference call among all panelists, followed by a final independent assessment by clinician panelists using the same questionnaire (9). The panel process served to refine definitions of some indicators, add new measures, and dismiss indicators with major concerns from further consideration (9). Finally, a subset of 34 of the most promising indicators was developed and additional literature was sought for studies describing these indicators. This unique use of a series of expert advisory panels produced inclusion criteria with considerable face validity given the breath and depth of the clinical specialists involved in the process. Use of a Decision Analytic Framework To Define Key Research Questions The authors of the review Bioterrorism Preparedness and Response: Use of Information Technologies and Decision Support Systems used an approach based on decision theory for defining the key research questions of their study (14). Their review evaluated whether existing information technologies and decision support systems would be useful for clinicians and public health officials preparing for and responding to bioterrorism (14). The authors were faced with a broad topic that involved highly heterogeneous technologies, deployed in different settings, for different users, and for different aspects of bioterrorism preparedness planning and responses. To specify the key research questions of their study, the authors used a decision analytic framework to identify the key decisions of clinicians and public health officials preparing for and responding to bioterrorism (Figures 1 and 2) (24). For example, they identified 4 key decisions of clinicians during a bioterrorism response: diagnostic decisions, management decisions (for example, how to select antibiotics for a given patient), prevention decisions (for example, which patients to place in isolation), and decisions about when to report suspicious cases to public health officials. The authors used a graphical method for representing decision problems, called an influence diagram, to help identify information needed to make such decisions (Figure 2) (24). The use of a decision analytic framework and influence diagram schematic also enabled the authors to identify and evaluate the relationships between the uncertain events that affect the decisions and to identify what is observable to the decision makers (24). They used this analytic framework to inform the key questions for their literature searches and syntheses of the literature. For example, this approach identified data about sensitivity, specificity, and post-test probability of disease for diagnostic tests as critical data for clinicians’ diagnostic decisions. Figure 1. Approach to problem formulation, data collection, and ana