The Real World Evidence Equation

Although real world evidence (RWE) has become increasingly popular for its insight into the use and performance of marketed products (i.e., in the real world rather than in controlled clinical trials), many pharmaceutical companies struggle with the basic decision of which study design should be used to address a given research question. The lack of certainty is understandable; the “toolbox” of study types available to researchers is actually quite extensive, and there are pros and cons to each. In this paper, we present general guidelines to help companies choose a study design that will ensure their Real-World Data (RWD) collection will be fit for their specific purpose.

Many of the studies used to develop RWE do not demand the same level of administrative rigor as randomized controlled trials (RCTs) and, hence, they are often thought to be easier to plan and conduct. Yet, there are so many possible stakeholders to satisfy, so many study designs from which to choose and so many different local regulations to accommodate, the design of real-world research actually presents a higher level of complexity. Manufacturers of pharmaceutical products, medical devices or diagnostics and, increasingly, also consumer products, who seek RWE must contend with:

• Varied stakeholder needs. To be successful, RWE studies must meet the particular stakeholder’s requirements for evidence—both in terms of the applicability and validity of the results. The external stakeholders interested in RWE include regulators, Health Technology Assessment (HTA) bodies/payers, healthcare professionals, patients/ caregivers, patient advocacy groups, the scientific community and the media. Internal manufacturer stakeholders include Clinical Development, Medical Affairs, Safety/Pharmacovigilance, Regulatory Affairs, Market Access/HEOR, Marketing and Communications and multi-functional brand teams. Each group, internally and externally, of course, will have different needs and a different focus. Moreover, a study may have multiple stakeholders across multiple countries. Thus, study planners must realize that it is unrealistic for a single study to fulfill the needs of all stakeholders; studies should be designed to address specific objectives which must be clarified at the outset.
• A lack of global harmonization in requirements. Regulatory, ethical and legal requirements for RWE are driven by the study design, the type of data to be collected and the patient population, etc., which is why each country has its own views on realworld studies and its own approval and notification requirements. Existing guidance, exhaustive though it may seem given that documents often consist of hundreds of pages, can’t cover everything, and researchers must proceed carefully to ensure that designs will meet with regulatory, ethical, legal and other required approval by the relevant stakeholders.
• Different perspectives on evidence validity. As RWD is collected in real-life situations, the focus is on generating evidence applicable to the respective population as a whole—in other words to achieve “external validity.” With a naturalistic study approach, data may be considered “messy” by RCT standards. Data can be missing (patients may not complete certain measurements, for example) or present challenges to statistical analysis (there may be inherent differences in study and control populations, for instance); thus, the “internal validity” of comparisons between study groups is often limited in RWE studies.
• Complexity of designs and approaches. In the RWE area, clinical, pharmacoepidemiological, epidemiological, as well as social and market research approaches, not only meet, they also overlap. There is a large toolbox of data sources, study designs, and analytical approaches adding to the challenge of study planning. It starts with the basic question of finding the best data source and extends to the final question of the level of quality required. Quality standards should be governed by the design and purpose of the study rather than by the function of the study manager or the capabilities of the contracted vendor.

In general, RWE picks up where RCTs leave off in the life cycle of a pharmaceutical product. Across the life of a product, RWE can be used for three main purposes:

• To manage product risk. As regulators are yielding to pressure to speed products to market, they are increasingly requiring manufacturers to implement post-approval programs that monitor and manage risk in the broad patient population.
• To support access/reimbursement. In the past, pharmaceutical products needed to clear three hurdles en route to market through studies proving quality, safety and efficacy. In many countries, HTA agencies now impose a fourth hurdle: cost effectiveness. Similarly, payers are looking closely at the value of the drugs or devices that they cover and more and more often are making their access and pricing agreements conditional upon receiving risk/benefit analyses and budget impact/cost data.
• To inform and educate stakeholders. The audience for product and disease information extends beyond regulators, payers and providers to include patients, caregivers and advocacy groups. When the message platforms of educational and promotional materials developed for all of these audiences are based on a product’s superior benefit/risk profile in actual, real-life practice, they can be more credible. Valid insight on the epidemiology, burden and economic implications of a disease will provide important background information.

As seen in Figure 1, there is a hierarchy, but also some overlap, in the ways that a particular body of RWE can be used. On one end of the spectrum, regulatory purposes usually require the highest administrative rigor, such as detailed documentation of all study processes, and also the highest evidence levels. This typically involves clinical research and pharmacoepidemiology, and the key study types are interventional RCTs and non-interventional studies (NIS). Studies undertaken for communication purposes vary in their requirements, but their administrative needs are often low – the focus is on quality of data and results rather than on ensuring every step is documented. On the other side of the spectrum, qualitative and quantitative social and market research, opinion surveys or preference studies typically require neither high evidence validity nor administrative rigor because they don’t collect individual-subject health information.

Requirements of HTA agencies are increasing in scope, and studies to address their needs are being set up more like clinical trials than simple payer research. Observational studies investigating burden of illness, treatment patterns or population-based epidemiology are often performed to substantiate HTAs or value dossier submissions.

RWE resulting from highly rigorous clinical or pharmacoepidemiological research has always been used for market access discussions and repurposed for use in communication and education of other audiences. Vice versa, data published in scientific, peer-reviewed journals for communication purposes increasingly serve to support product claims with regulators and HTAs, e.g. to provide epidemiology, burden of illness results or deliver product profile/ preference information (market/social research).

Although the manufacturer’s internal stakeholders for regulatory information differ from typical internal communication stakeholders, the audiences for, and uses of, RWE can be quite fluid if the study is designed appropriately.

What design and type of study is best for a given research objective? Despite the hierarchy displayed in Figure 1, there is, unfortunately, no simple answer to that quite reasonable question. There are so many variables driven by the interplay of the indication, the product profile and the type of outcomes required that only general guidance can be provided.

To start, one must look at any RWE research issue from two different angles: First from the top (or strategic/scientific level) down and then from the bottom (available data sources and operational constraints) up. (See Figure 2).

The “strategic/scientific considerations” originate from the purpose of the study—who will use the information and in what way. Depending on the study purposes, specific outcome information will need to be collected, ranging from clinical to patient-reported to economic outcomes. Outcome categories and instruments, in turn, influence the data collection needs and, ultimately the study design.

The “operational considerations” revolve around the availability of data sources and the general feasibility (what data are available, what can be collected easily and who needs to be involved) and degrees of quality (what level of administrative quality and data validity is expected by the study stakeholders). In the RWE area, study feasibility depends heavily on central and local regulations, ethical and legal requirements and the related study requirements for quality.

From an operational perspective, there is also a hierarchy, or sequence, of approaches, starting with readily available data from existing sources and ending with complex, longterm interventional or observational prospective study approaches.

If secondary data such as claims or electronic medical record (EMR) databases are available to fulfill the study purpose and cover the respective outcomes needs, a retrospective database study will often provide a cost-effective and timely solution (Figure 3). To some extent, this is also true for a primary-research, retrospective chart-review design. However, retrospective approaches—database or chart-based—have limitations in the feasibility of follow-up or reconciliation, e.g., for adverse event reports for safety purposes, and the availability of patient-reported outcome (PRO) information is usually minimal.

For some RWE research purposes, health information on individual subjects is not required. The research questions can be answered via aggregated data—such as percentages—or through opinion-based research, which is the domain of social and market research. Obviously, the evidence validity of these approaches is relatively low, but may be sufficient for some purposes and research questions—and studies can be performed relatively quickly, e.g., using physician or subject panels.

Selecting the optimal RWE study design from the outset saves researchers both time and money. In fact, it can often be that getting it “not quite right” is as costly as “getting it wrong,” so pharmaceutical companies are well-advised to give careful thought to the purpose of the study, the intended stakeholders, the outcomes sought and the operational factors that will influence what is feasible and what level of data quality can be obtained. While there are no hard and fast answers that can help in selecting the best study design, we have provided some general guidance for working through the cascade of decisions that must be made. The issue is so complex and the stakes are so high that we recommend that researchers seek the counsel of experts in designing their RWE studies. Properly designed, RWE research can demonstrate superior Benefit/Risk profiles at tolerable cost over a product’s life cycle.