Efficacy, effectiveness, and comparative effectiveness in liver disease†
Potential conflict of interest: Dr. El-Serag received grants from Bayer and Schering-Plough. He is also a consultant for Vertex. Dr. Talwalkar received grants from Salix and Merck.
From Efficacy to Effectiveness
The term “translational research” is commonly used to describe efforts made toward bridging the gap between discoveries made at “the bench” to the patient's “bedside” by moving basic discoveries into a candidate health application such as the production of a new treatment or diagnostic test, which is typically assessed in clinical trials. However, the benefits of therapies and diagnostic tests observed in those studies are often reduced once they are implemented in clinical practice. Therefore, “efficacy” reflects the degree to which an intervention produces the expected result under carefully controlled conditions chosen to maximize the likelihood of observing an effect if it exists. By contrast, “effectiveness” addresses the extent to which an intervention is beneficial when deployed in medical practice settings and broader populations.1 It takes into account the benefits and harms of an intervention, and therefore can often be more relevant to health care decisions of providers and patients as well as policy evaluation. Efficacy is largely determined by the biological effects of a therapy, whereas effectiveness takes into account external factors such as individual patient characteristics, health system features, and societal influences. Given the possible chasm between efficacy and effectiveness, another frontier of translational research looms, which addresses the gap between clinical trials and the “real world” impact of individual-level or population-level interventions on health outcomes.2, 3
The concept of effectiveness encompasses the impact of the healthcare system and human behavior. Figure 1 illustrates that in addition to pharmacological and physiological efficacy, other components must be available for an intervention to be effective. These include (1) availability and accessibility of the intervention to patients who can obtain benefit in appropriate health care settings, (2) identification of patients who are appropriate for the intervention, (3) recommendation of the intervention by providers, (4) acceptance of the intervention by patients, and (5) adherence to treatment at the recommended dosing for therapeutic coverage to fully achieve the benefits of therapy. Given the multiplicity of components involved with determining effectiveness, one could assess the interactions between these factors to come up with an estimate of how one particular intervention would perform in everyday practice. Models of effectiveness can be operationalized to evaluate therapeutic (e.g., treatment of viral hepatitis) or diagnostic (e.g., surveillance for hepatocellular carcinoma) measures in large populations. For example, in Table 1, under a scenario which assumes 50% efficacy for current hepatitis C therapy (therapy A) to achieve a sustained viral response in patients with genotype 1 infection, the overall effectiveness could be as low as 17% given relatively small decrements in the rates of access to care, accurate diagnostic testing, proper treatment recommendations by the providers, and ultimately acceptance/adherence to treatment by the patient. If a new therapy B is developed with increased efficacy (70%), the overall effectiveness of therapy B would be only slightly higher than therapy A (24%) if all other parameters remained unchanged. However, if the less efficacious therapy A becomes more widely available, its overall effectiveness might become greater than that of the more efficacious therapy B, based on greater confidence by providers and better acceptance and adherence by patients. These scenarios, however, are somewhat simplistic in the assumption the probabilities of access, recommendation, and adherence are independent of each other. It is likely, for example, that lack of access (e.g., among the underinsured or uninsured) would adversely influence the rates of diagnosis, recommendation, and adherence, thus leading to even lower effectiveness rate than shown in the example.
The possible steps that bridge efficacy to effectiveness.
| Efficacy | Access | Diagnosis | Recommendation | Acceptance | Adherence | Community Effectiveness | |
|---|---|---|---|---|---|---|---|
| Therapy A | 50% | 80% | 85% | 85% | 85% | 70% | 17% |
| Therapy B | 70% | 80% | 85% | 85% | 85% | 70% | 24% |
| Therapy A with modifications | 50% | 90% | 90% | 90% | 90% | 80% | 26% |
- All estimates are hypothetical except for community effectiveness, which is calculated by multiplying all the preceding probabilities.
As this example illustrates, the observed efficacy of treatments within clinical trials may not be easily replicated in the community. Therefore, it is also imperative to conduct studies for evaluating the effectiveness of interventions (Fig. 1).
Initial observations of effectiveness stem from the documentation of wide variations in the use of diagnostic and therapeutic modalities by geographic area and other demographic factors. Variations in the utilization of health services can be a consequence of overutilization or underutilization of recommended care as well as disparities in care associated with sex and race (Fig. 2).4 Even in populations with more equitable access to care (e.g., Medicare and veteran populations), a number of studies have shown that health services utilization patterns and outcomes are unfavorable to black patients as compared with whites.5 Providers' knowledge and attitudes toward therapeutic or diagnostic procedures can also be a major explanation of inappropriate utilization or disparity. For example, it has also been shown that physicians provide less information and do not encourage as much participation in black patients compared with white patients. Finally, the dynamics in the interaction between patients and healthcare providers should also be considered.6 Variations may be appropriate, however, if they could be explained by disease-related factors (e.g., presence of known contraindications) or patient preferences (e.g., patients refusing a certain therapy).
Underutilization of health services is a major reason for reduced effectiveness of these services. The figure shows a schematic presentation of the appropriate and inappropriate underutilization of services, and the possible reasons for underutilization.
“Effectiveness” studies of therapeutic and diagnostic interventions within liver disorders remain scarce, except for a few studies in the effectiveness of hepatitis C virus antiviral treatment.7-12 A Focused Study Group held in the 2006 Annual Meeting of the American Association for the Study of Liver Diseases highlighted the chasm between efficacy and effectiveness of several practices, including hepatitis C antiviral therapy, screening for hepatocellular carcinoma, and treatment of hepatocellular carcinoma.13 Where present, the evidence indicated marked underutilization of these interventions. Underutilization seems to follow some disturbing patterns in relation to ethnicity, poverty, and sex.6 Perhaps even more striking was the dearth of studies to examine most of the important links mediating efficacy to effectiveness shown in Fig. 2.
Abbreviations
CER, comparative effectiveness research; IOM, Institute of Medicine; NIH, National Institutes of Health.
The New Buzzword: Comparative Effectiveness
-
Consistent with the definition of effectiveness, CER is conducted in settings that are similar to those in which the intervention will be used in practice. An emphasis is placed on external validity, or the ability to generalize results to real-world decision making.
-
CER measures outcomes, both benefits and harms, that are important to patients. This is familiar to clinicians because they routinely address risks and benefits of an intervention in practice. Assessment of patient-reported outcomes is important for CER studies in which patient ratings of effectiveness or adverse events may differ from clinical measures.
-
Methods used for CER range from nonexperimental studies (observational settings) to experiments (randomized and nonrandomized controlled trials) to synthesis of existing studies (systematic reviews and meta-analysis, technology assessments, and decision analysis).
-
CER not only informs a specific clinical decision from the patient perspective but also directs a health policy decision from the population perspective. Clinical questions refer to the health care of individual patients, including preventive, screening, diagnostic, therapeutic, monitoring, or rehabilitative interventions. Policy questions refer to the health and health care of populations through knowledge synthesis and transfer strategies, public health programs, or initiatives involving the organization, delivery, or payment for health services.
-
CER focuses on the individual rather than the average patient by analyzing results at the population and subgroup levels. Utilization of subgroup results and clinical prediction rules assists providers and patients in individualizing management decisions. Applying new knowledge in genomics, systems biology, and other biomedical sciences in subgroups of patients with demographic, ethnic, physiologic, and genetic characteristics introduces new possibilities of individualized and predictive medicine.
-
CER compares at least two alternative interventions, each with the potential to be “best practice.” For many clinical decisions, “optimal usual care” reflecting current standards is an appropriate potential comparator, which may include the alternative of “watchful waiting.”
The process by which these 100 priority topics were selected and prioritized was exhaustive and iterative. It is worth mentioning that a part of this process was to solicit input from a Web-based questionnaire that was distributed to individuals and organizations. This survey generated 1268 responses in 32 organ system categories ranging from oral health to skin disorders. Of the 32 categories, the only one that received 0 (zero) nomination was “liver and biliary tract disorders.” Table 2 lists topic areas with some hepatology relevance. The only disease that directly concerned hepatology was hepatitis C, which was listed under infectious disease and appeared in the last quartile of the 100 topics. Other topics that are tangentially related to hepatology were treatment of obesity for related outcomes (which should include nonalcoholic fatty liver disease) and treatment of liver metastasis. Although many hepatologists may argue that there are other disorders with a demonstrably higher need for further research, it is therefore imperative that we become more engaged in future efforts at directing CER initiatives toward diseases and conditions within our discipline.
| Research Area | Topic | Priority Q uartile |
|---|---|---|
| Infectious disease | Compare the effectiveness of alternative clinical management strategies for hepatitis C, including alternative duration of therapy for patients based on viral genomic profile and patient risk factors (e.g., behavior-related risk factors). | 4 |
| Nutrition | Compare the effectiveness of treatment strategies for obesity (e.g., bariatric surgery, behavioral interventions, pharmacologic treatment) on the resolution of obesity-related outcomes such as diabetes, hypertension, and musculoskeletal disorders. | 3 |
| Oncology | Compare the effectiveness of surgical resection, observation, or ablative techniques on disease-free and overall survival, tumor recurrence, quality of life, and toxicity in patients with liver metastases. | 4 |
CER Methods: Strengths, Limitations, and Requirements
It is important that quality standards for scientific validity for CER remain as rigorous as traditional biomedical scientific research. The conduct of CER, however, is likely to face additional obstacles when compared to more conventional areas of clinical or epidemiological research. Although it is a desirable option with high internal validity, the utilization of prospective, randomized trials requiring large sample sizes may take so long or be so expensive as to render the study unfeasible or unethical. This is especially important in CER, where study subjects tend to be more heterogeneous than those in typical efficacy trials. Innovative study designs such as cluster randomized trials in which the subjects are assigned to intervention or control in groups (clusters) defined by a common feature, such as the same physician or health plan, can be used. CER may also include cost-effectiveness or economic analyses which take into account resource utilization and expected benefits of competing alternate strategies explicitly, demonstrating sometimes that a more expensive approach offers better value than other lower-cost approaches. Weaknesses of those analyses include lack of standardization or transparency in methodology; limitations related to outcome modeling, which is often necessary when relevant data are not available from effectiveness trials or high-quality observation studies; and suspicion that the analyses tend to discourage the use of expensive forms of care and lead to denial of needed care. However, these models can identify gaps in knowledge that can stimulate primary research to more concisely determine the effectiveness of interventions and diagnostic testing methods. Of note, cost was not a major endpoint or outcome stressed in both the IOM report and NIH Challenge Grant offerings in 2009, given societal and governmental concerns about avoiding “rationing” from an economic perspective.
Effectiveness research using observational data such as health care claims databases and patient registries are susceptible to unidentified bias and confounders that may weaken the level of evidence obtained from these studies. Thus, studies must have appropriate comparison group to determine the effectiveness of a new intervention. Studies must be adequately powered so that the conclusion can be drawn with confidence, both statistically and clinically. Retrospective studies, while aided by the progressive availability of electronic medical records for large numbers of patients, are often limited by biases or confounders that limit the reliability of data. As a result, it is critical that research strategies incorporate plans to match specific study methodologies to the question being asked and to the population and/or database that is available. Close attention must be paid to accounting for potential biases and adjusting for confounding factors. Recommendations have been made in the IOM report to use electronic health registries and databases for particular types of CER where these study designs may be well-suited to answer specific questions about diagnostic tests and treatments.
Finally, it is imperative that the investigators have rigorous training in epidemiological research, health services research, and statistical methods to ensure methodological robustness and study validity. Additional resources are necessary such as new research infrastructure to answer clinical and policy questions as well as develop and test innovative methodologic frameworks. Future initiatives from NIH and the Agency for Healthcare Research and Quality are expected to involve requests for proposals to develop CER-mentored training programs to expand the pool of qualified investigators in this field.
Effectiveness Research: Where to Go from Here?
The recent emphasis on CER should not be regarded as a mandate that all patient-oriented research must focus on comparative effectiveness or even effectiveness. Naturally, for early phase studies of an intervention, it is critical to evaluate their safety and efficacy under a defined set of circumstances. Once an intervention has been shown to be efficacious, CER to address effectiveness in settings different from the efficacy studies may inform physicians, patients, and policy makers.
Ultimately, in order for CER to impact health care delivery or outcomes, the results must be communicated effectively to patients and providers and integrated into the health care delivery system. Although the traditional model of biomedical research has devoted considerable attention and resources to developing new therapies, enhancing the potential benefits of what we already have and improving nonmedical or health system factors has not been studied in depth. CER recognizes that both types of research are crucial in our quest to improve the health of patients. Even for skeptics of effectiveness research, it should be evident that information on variables other than efficacy cannot be easily obtained from clinical trials alone. Thus, data from high-quality observational or quasi-experimental studies become critical in the assessment of the overall effectiveness of therapeutic or diagnostic modalities in large patient populations.
The recently passed Healthcare Reform Law of 2010 places further emphasis on CER. For example, the law authorizes the establishment of a nonprofit corporation known as the Patient-Centered Outcome Research Institute (PCORI), which is expected to oversee the conduct of CER and dissemination of the research findings.16 The extent to which CER will be supported and used while the reform is implemented in the coming years remains to be seen. It is certain, however, that interest will remain high on promoting research to determine the effectiveness of new (and existing) therapies in the context of usual medical practice settings within the United States in the foreseeable future.
