home

Section V: Research Design

Key Points

  • Pragmatic trials often involve randomizing groups (cluster randomization) vs. individuals to avoid contamination across individuals, or spill-over, of the intervention.
  • Some participants may demand that everyone gets the intervention, for example in quality improvement studies.  In this situation, a stepped wedge design can be very useful.
  • Taking a patient- or stake-holder centered approach to trial design is foundational to pragmatic research.  This approach is also key for accelerating the eventual uptake of the research findings into practice.  
  • PRECIS-2 identifies three domains related to study design (i.e., primary outcome, follow-up and primary analysis) 

 

Learning Objectives:

To describe and apply statistical considerations to pragmatic trial research design

&
To review the PRECIS-2 domains of follow-up, primary outcome and primary analysis

Statistical Considerations - Trial Designs

Cluster Randomization

  • Cluster randomization trials randomize groups of individuals to receive different interventions.  Groups can be clinics, hospital, worksites and entire communities.  This trial design has become increasing popular in public health and clinical trial research.
  • In the context of pragmatic trials, the benefits of increased efficiency and decreased risk of experimental contamination often outweigh the resulting loss in statistical precision from the effects of variance inflation. 
  • Special Considerations: Identifying the unit of inference based upon the level of cluster randomization selected. There is potential for imbalance between groups when randomizing a relatively small number of clusters.  Assessing the value of the intra-cluster correlation (ICC) may be difficult when planning the trial size.

 

Donner A, Klar N.  Pitfalls of and Controversies in Cluster Randomization Trials. Am J Public Health. 2004;94:416–422

Stepped Wedge

  • Stepped wedge randomized trial designs involve sequential roll-out of an intervention to participants (individuals or clusters) over a number of time periods. 
  • By the end of the study, all participants will have received the intervention, although the order in which participants receive the intervention is determined at random. 
  • The design is particularly relevant where it is predicted that the intervention will do more good than harm (thereby, making a parallel design, in which certain participants do not receive the intervention unethical or unattractive to participants) and/or where, for logistical, practical or financial reasons, it is difficult to deliver the intervention simultaneously to all participants. 
  • Special Consideration:  Stepped wedge trials are vulnerable to time varying confounding.  They work best when the desired effect on individuals is achieved within a short period of time. 

 

Brown CA, Lilford RJ. The stepped wedge trial design: a systematic review. BMC Medical Research Methodology 2006, 6:54  doi:10.1186/1471-2288-6-54

Effectiveness-Implementation Hybrid

This design strategy involves blending design components of clinical effectiveness and implementation research and is described by 3 hybrid types: 

  • 1. Testing Effectiveness
  • 2. Dual Testing
  • 3. Testing Strategy
Testing clinical effectiveness, while gathering information on implementation
Dual testing of clinical effectiveness and implementation interventions/strategies
Testing of an implementation strategy, while gathering information on clinical effectiveness

Special Consideration:

Traditional clinical and implementation research have not shared many design features – for example, unit of analysis, typical unit of randomization, outcome measures, and targets of the intervention being tested. Hybrid designs are new and the field is still evolving on how best to blend these design components.  However, the information they provide could speed the translation of research findings into routine practice.

 

Curran, GM, Mark Bauer, Brian Mittman, Jeffrey M. Pyne, Cheryl Stetler. Effectiveness-implementation Hybrid Designs: Combining Elements of Clinical Effectiveness and Implementation Research to Enhance Public Health Impact. Med Care. 2012 Mar; 50(3): 217–226. 

Statistical Considerations - Analysis

  • Intention-to-Treat (ITT)

  • Contextual Factors

ITT analysis includes every subject who is randomized according to their treatment assignment. It ignores noncompliance, protocol deviations, withdrawal, and anything that happens after randomization.  In this regard, it reflects usual care practices.

Special Consideration:  Addressing patient drop-out (data missingness) is particularly critical when using ITT in longitudinal studies.  Thus, a natural tension exists between ensuring protocol adherence and minimizing follow-up burden.

Characteristics of the setting can affect implementation and effectiveness of interventions.  Analyses should explore potential moderators (effect modifiers) that are present at baseline using multilevel modeling with time x treatment x moderator interactions.

Special Consideration: Ensuring that data on possible contextual factors are collected.

Pragmatic Trials Vs. Pre-Post Observational Studies

Different statistical considerations described above are necessary to account for selection biases using observational study designs.

 

Pre-Post Evaluations

Are one form of pragmatic research trial design that does not involve randomization

Discovery/Development

Pragmatic Trials

Involve randomization of the treatment or the intervention.  

AHRQ's Users Guide for Developing a Protocol for Observational Comparitive Effectiveness Research
The user's guide identifies best practices for designing observational CER studies and standardizes the review of study protocols with checklists in each chapter.
PCORI's Standards for Causal Inference Methods in Analyses 
Describes the development a set of minimum standards for causal inference methods for observational and experimental studies in patient-centered outcomes research and comparative effectiveness research.
Duke's Pragmatic Trials Living Textbook

Designed by the NIH Collaboratory to provide a complete suite of information on how to understand, design, conduct, analyze & disseminate pragmatic clinical trials (PCTs).

5 Recommendations to Strengthen Research

Necessary Pragmatic Research Trial Infrastructure

An important challenge is the need to develop infrastructure to support pragmatic clinical trials, which compare interventions in usual practice settings and subjects.

The NIH Clinical and Translational Science Awards Consortium reported on five recommendations related to strengthening the research infrastructure for pragmatic clinical trials 

Concannon et al, 2013

1. Develop A Learning Network

FRAMEWORK

1

  • Share research opportunities across diverse funding agencies
  • Share key lessons learned

2. Support Community and Stakeholder Engagement

  • Establish a standing infrastructure for routinely engaging communities, practices, and stakeholders in trial development, implementation, and dissemination activities
Mullins CD, Abdulhalim AM, Lavellee DC. Continuous Patient Engagement in Comparative Effectiveness Research JAMA  2012; 307 (15): 1587-1588.

3. Address Regulatory Challenges

  • Modify IRB process to support joint approvals
  • Develop strategies to streamline multi-institutional contracting

4. Provide Information Technology Solutions

  • Implement secure, standards-based, interoperable information systems across sites and institutions
  • Develop a comprehensive dictionary of data elements across data platforms

5. Expand Research Methods

FRAMEWORK

5

  • Focus method development on study design and analytical approaches that help measure and interpret treatment, site, and patient (subject) heterogeneity

PRECIS-2 Domains in Study Design 

Hover to Reveal Description
Click to Reveal

PRECIS-2 Domain: Primary Outcome

To what extent is the trial's primary outcome relevant to participants?

PRECIS-2 Domain: Follow-up

How different is the intensity of measurement and follow-up of participants in the trial and the likely follow-up in usual care?

PRECIS-2 Domain: Primary Analysis

To what extent are all data included in the analysis of the primary outcome? 

Dissemination

The targeted distribution of information and intervention materials to a specific public health or clinical practice audience. The intent is to spread (“scale up”) and sustain knowledge and the associated evidence-based interventions.

Implementation

The use of strategies to adopt and integrate evidence-based health interventions and change practice patterns within specific settings....

Dissemination and implementation research.

Intends to bridge the gap between public health, clinical research, and everyday practice by building a knowledge base about how health information, interventions, and new clinical practices and policies are transmitted and translated for public health and health care service use in specific settings.

Applying PRECIS-2 Domains: Research Design

More Pragmatic . . .

More Explanatory . . .

Primary Outcome

  • Of obvious importance to participants

 

 

 

  • Uses a surrogate, physiological outcome in which the direct relevance to participants is not clearly evident
  • Requires central adjudication or assessment expertise not available in usual care
  • Measured at a time point not consistent with usual care

Follow-Up

  • Participant burden is no more than usual care

 

 

  • More frequent or longer visits compared to usual follow-up
  • More extensive data collection than what is routinely collected
  • Special follow-up triggered by an outcome or intervening event

Primary Analysis

  • Intention-to-treat using all available data (all participants count)

 

  • Excludes ineligible post-randomization participants
  • Includes only completers or those following the treatment protocol

Other Resources

Research Design: An Example of Pragmatic Trials applied

P. Michael Ho, MD, PhD

Mike Ho presents an example of pragmatic trial study design: clopidogrel treatment

Statistical Considerations                                                                                              

L. Miriam Dickinson, PhD

Key lessons learned in encountering statistical considerations in the field, presented by Miriam Dickinson

Case Study for Patient-Centered Pragmatic Clinical Trials (PCT's)                                

C Daniel Mullins, PhD 

Daniel Mullins discusses the importance of patient-centerdness in PCT research design at the 2014 CRISP Pragmatic Trials workshop

Key Takeaways

Click To Reveal Flipcard Answer

What can be said about the PRECIS-2 domains as they apply to research design?

  • Primary outcome measures are relevant to the participants (and to the users of the research findings).
  • Measurement and follow-up intensity reflect usual care (or what could be accomplished under typical practice conditions)
  • The primary analysis includes everyone who was randomized (an intention-to-treat analysis), regardless of their adherence to the intervention.

Patient-Centerdness is foundational to pragmatic trials research, why?

Among other benefits, this approach (along with stakeholder consideration) will actually help to accelerate the eventual uptake of the research findings into practice!

List the 5 recommendations from the NIH to strengthen research infrastructure

  1. Develop a learning network
  2. Support community and stakeholder engagement
  3. Address regulatory challenges
  4. Provide information technology solutions
  5. Expand research methods

References

Click Each Reference to View the Abstract and Download Full Texts.