Content Validity, Concept Elicitation, and Disease Conceptual Modeling under FDA PFDD Guidance
In Part 1 of this series, we defined Clinical Outcome Assessments (COAs) and explored why demonstrating a tangible treatment benefit in daily life is central to modern drug evaluation. Yet recognizing the value of the patient experience is only the beginning. The core scientific challenge lies in converting qualitative, lived experiences into standardized, reproducible measurement tools suitable for regulatory review.
Under the FDA’s Patient-Focused Drug Development (PFDD) framework, establishing content validity is the mandatory first milestone. Before calculating statistical correlations or factor loadings, researchers must qualitatively prove that an instrument measures what matters most to patients, using language they understand and can answer accurately. Without qualitative content validity, even the most sophisticated statistical modeling cannot rescue a flawed instrument.
Here is the 7-step roadmap used to construct, adapt, and refine robust COAs.
The 7-Step Development Framework
- Qualitative Literature Review: Map existing literature, patient narratives, and disease burden.
- Concept Elicitation (CE) Interviews: Conduct open-ended, semi-structured interviews with patients and caregivers.
- Conceptual Disease Modeling: Structure themes into a causal hierarchy and demonstrate concept saturation.
- Review of Existing Instruments: Benchmark available scales against the conceptual model and the target Context of Use (COU).
- Instrument Adaptation or Construction: Draft items, define recall periods, and format response options.
- Cognitive Debriefing: Verify patient comprehension and readability using think-aloud methods.
- Quantitative Psychometric Evaluation: Transition to statistical testing of reliability, validity, and responsiveness.
Step 1: Qualitative Literature Review
Before speaking directly with patients, researchers conduct a systematic review of clinical trials, qualitative studies, and outcomes research literature. The objective is to compile an inventory of disease-specific symptoms, functional limitations, and quality-of-life impacts.
In modern research programs, teams increasingly supplement peer-reviewed literature with social listening—analyzing de-identified discussions across verified patient advocacy forums to capture daily challenges that rarely surface in routine clinical visits.
Step 2: Qualitative Concept Elicitation (CE) Interviews
Primary qualitative research centers on one-on-one, semi-structured interviews with patients, caregivers, and expert clinicians. To prevent investigator bias, interview guides follow a disciplined sequence:
- Spontaneous Elicitation: The interviewer begins with broad, open-ended questions (e.g., “How does your condition affect your morning routine?”). This allows participants to identify and prioritize their symptoms using their own vocabulary.
- Targeted Probing: If vital clinical areas are not mentioned spontaneously, the interviewer introduces neutral, non-leading follow-ups (e.g., “You mentioned difficulty walking; can you describe what that feels like on stairs?”).
Interviewers strictly avoid leading phrasing, double-barreled questions, or clinical jargon. This non-leading discipline is essential because it prevents the interviewer’s preconceptions from contaminating the patient’s conceptual vocabulary.
Step 3: Disease Conceptual Modeling and Concept Saturation
Interview transcripts undergo thematic coding to construct a Disease Conceptual Model. This framework organizes the patient experience into a logical, three-tiered causal hierarchy:
- Biological Symptoms: Core physical manifestations directly caused by pathology (e.g., muscle weakness, tremor, or shortness of breath).
- Proximal Functional Impacts: Direct functional limitations resulting from those symptoms (e.g., difficulty climbing stairs, inability to button a shirt, or trouble holding a pen).
- Distal HRQoL Impacts: Downstream psychological, social, and economic consequences (e.g., anxiety, loss of workplace productivity, or social isolation).
This hierarchy ensures that the final instrument captures not only biological symptoms, but also their direct functional and life-level consequences. During this step, researchers must also document Concept Saturation:
Concept saturation occurs when conducting additional interviews yields no new relevant concepts, themes, or insights.
Regulators require formal proof of saturation across interview cohorts to confirm that the sample size was sufficient and that the resulting scale captures the full spectrum of the patient experience.
Step 4: Critical Review of Existing Instruments
Armed with a validated conceptual model, researchers evaluate whether existing, qualified COAs cover the identified concepts within the target Context of Use.
In clinical practice, legacy instruments often fall short: they may rely on outdated diagnostic classifications, contain questions irrelevant to modern lifestyles, or lack cultural adaptability for global trials. If an established instrument has strong psychometric properties but lacks a critical symptom domain, adapting that tool is often faster and more cost-effective than building an entirely new scale from scratch.
Step 5: Instrument Adaptation or Construction
When drafting new items (or modifying existing ones), researchers translate concepts of interest into specific survey questions:
- Item Phrasing: Questions incorporate natural patient phrasing identified directly in interview transcripts rather than formal medical terminology.
- Recall Period Selection: The look-back window is chosen based on symptom dynamics. Highly fluctuating symptoms (e.g., acute pain, nausea) require short recall windows (such as “the past 24 hours” or “right now”) to minimize memory distortion. More stable physical functions (e.g., walking or dressing) typically use a “past 7 days” recall period.
- Response Options: Response categories (e.g., 4- or 5-point Likert scales, numeric rating scales) must be balanced, mutually exclusive, and easy for patients to distinguish.
A notable real-world example is the SMAIS (Spinal Muscular Atrophy Independence Scale): an initial draft of 30 items derived from literature was refined to 29 daily functional tasks after qualitative patient input confirmed which activities best represented meaningful independence.
Step 6: Cognitive Debriefing (Testing Comprehension)
Drafting questions is not enough; investigators must verify that patients interpret the items exactly as intended. In Cognitive Debriefing, an independent group of patients reviews the draft instrument using the Think-Aloud method.
Participants read instructions, questions, and response scales aloud while explaining their interpretation and thought process in real time. This identifies confusing phrasing, ambiguous instructions, or inappropriate recall windows. Cognitive debriefing typically involves two iterative rounds of testing, allowing for revisions and re-testing before finalizing the scale.
Step 7: Transition to Quantitative Psychometrics
Once qualitative content validity is established through Steps 1 to 6, the instrument moves into formal quantitative validation. Quantitative modeling can evaluate consistency and refine score precision, but it cannot compensate for missing, irrelevant, or poorly defined concepts.
In Part 3 of this series, we will examine the quantitative engine of COA validation: Classical Test Theory (CTT), Item Response Theory (IRT), and the science of setting Meaningful Change Thresholds.
Looking for the complete roadmap? Read the Series Overview & Table of Contents
Note: This series draws on and adapts core concepts from Genentech’s Coursera course “Data Sciences in Pharma: Patient Centered Outcomes Research,” together with the FDA’s Patient-Focused Drug Development (PFDD) guidance.