QPPV.COM

Risk Minimisation Effectiveness Evaluation

A practical guide to evaluating whether risk minimisation measures achieve their intended objectives and improve safe use of medicinal products.

Audio Lesson 11 min

Risk Minimisation Effectiveness Evaluation

Introduction

The implementation of a risk minimisation measure does not guarantee that risk has been reduced.

Educational materials may be distributed but never read. Monitoring recommendations may be included in prescribing information but not followed. Pregnancy prevention programmes may exist but fail to prevent fetal exposure.

For this reason, modern pharmacovigilance systems increasingly focus on effectiveness evaluation.

Risk minimisation effectiveness evaluation seeks to determine whether a risk minimisation measure achieves its intended objective and contributes to safer use of a medicinal product.

The question is no longer:

Was the measure implemented?

The question is:

Did the measure work?

Why Effectiveness Evaluation Matters

Risk minimisation measures often require substantial effort and resources.

Examples include:

Regulators expect evidence that these activities provide value and contribute to public health protection.

Without effectiveness evaluation, it may be impossible to determine whether a programme should:

Regulatory Expectations

Within modern RMPs, effectiveness evaluation has become an important component of risk management.

Regulators increasingly expect organisations to:

The level of evaluation should be proportionate to the importance of the safety concern and the complexity of the intervention.

Key regulatory references include: - EMA Good Pharmacovigilance Practices (GVP) Module V – Risk Management Systems (and subsequent updates) - GVP Module VIII – Post-Authorisation Safety Studies (for PASS methodologies) - ENCePP Guide on Methodological Standards in Pharmacoepidemiology - CIOMS IX Practical Approaches to Risk Minimisation - National guidance relevant to specific measures (e.g., pregnancy prevention programmes)

Inspectors will frequently seek evidence that the evaluation approach is consistent with these guidance documents and that regulatory commitments (in RMP and product labelling) have been met.

Fundamental Principle

Evaluation should begin with a clearly defined objective.

For example:

Risk: Teratogenicity

Objective: Prevent exposure during pregnancy

Measure: Pregnancy Prevention Programme

Evaluation: Has exposure during pregnancy been reduced?

Without a clearly defined objective, meaningful evaluation becomes difficult.

What Does Success Look Like?

A risk minimisation programme should have predefined goals.

Examples may include:

Success criteria should be defined before evaluation begins and be expressed in measurable terms (numerator, denominator, timeframe, acceptable thresholds).

Evaluation Framework

A common framework is:

Safety Concern ↓ Risk Minimisation Measure ↓ Behaviour Change ↓ Clinical Outcome

Evaluation may occur at one or more stages of this pathway.

Process Indicators

Process indicators evaluate implementation.

Typical questions include:

Examples include:

Process indicators are useful but have limitations: they document activity, not necessarily impact.

Outcome Indicators

Outcome indicators evaluate the effect of the intervention.

Examples include:

Outcome indicators generally provide stronger evidence of effectiveness. They are, however, more difficult to measure and may require primary data collection or linkage to healthcare databases.

Knowledge Surveys

Knowledge surveys are widely used to evaluate educational materials.

Examples of assessed topics include:

Surveys help determine whether key safety messages have been understood. Survey design must consider representativeness, sampling, response bias and validation of instruments.

Behavioural Assessments

Behavioural assessments evaluate whether healthcare professionals or patients modify their actions.

Examples include:

These assessments often provide more meaningful information than knowledge surveys alone and can be derived from claims, electronic health records (EHR), registries or primary data collection.

Drug Utilisation Studies

Drug Utilisation Studies (DUS) are frequently used to evaluate risk minimisation measures.

Examples include assessment of:

DUS studies provide insight into real-world implementation and are often designed as retrospective cohort studies or interrupted time series.

PASS and Effectiveness Evaluation

Post-Authorisation Safety Studies may support effectiveness evaluation.

Examples include:

PASS methodologies are particularly useful when evaluation requires large populations or long-term follow-up. Where relevant, the study should follow GVP Module VIII requirements and ENCePP methodological standards.

Pregnancy Prevention Programmes

Pregnancy Prevention Programmes often require extensive effectiveness evaluation.

Possible measures include:

Because fetal exposure may have serious consequences, regulators frequently expect robust evaluation with predefined, quantitative success criteria and a clear plan for action if goals are not met.

Educational Material Evaluation

Educational programmes may be evaluated using:

A useful principle is:

Distributed ≠ Read

Read ≠ Understood

Understood ≠ Behaviour Changed

Evaluation should extend beyond simple distribution metrics whenever possible.

Measuring Rare Outcomes

Some risks occur infrequently.

Examples include:

In such situations, direct outcome measurement may be difficult.

Alternative approaches may include:

The chosen approach should be scientifically justified and documented in a study protocol or evaluation plan.

Defining Success Criteria

Success criteria should be established prospectively.

Examples may include:

Predefined criteria improve interpretation of findings and are a common focus during inspections.

Lifecycle Management

Evaluation findings should influence programme management.

Possible outcomes include:

Evaluation should support decision-making rather than exist solely for regulatory reporting. The decision pathway (who decides, on what evidence, and within what timeframe) should be defined in governance documentation.

Challenges in Effectiveness Evaluation

Several practical challenges exist.

Attribution

Changes may result from multiple factors. Robust designs and sensitivity analyses can help with causal interpretation.

Rare Events

Outcomes may be difficult to measure directly. Consider proxy and exposure-based indicators.

Data Availability

Relevant information may be unavailable or incomplete. Data agreements and feasibility assessments are essential.

Survey Bias

Participants may not represent the wider population. Use sampling frames and weighting where possible.

Resource Requirements

Robust evaluations may require substantial investment. The scope should be proportionate to the residual risk and regulatory commitments.

These limitations should be recognised when interpreting findings and must be documented.

Common Regulatory Deficiencies

Recurring deficiencies include:

No Defined Objective

Programme purpose is unclear.

Reliance on Distribution Metrics Alone

No assessment of understanding or behaviour.

Weak Success Criteria

Evaluation lacks meaningful endpoints.

Failure to Use Results

Findings do not influence programme management.

Inadequate Documentation

Evaluation methods and conclusions are poorly documented.

These issues frequently generate regulatory questions during inspections and regulatory reviews.

Inspection and Audit Considerations

Inspectors may review:

The emphasis is often on demonstrating a systematic, pre-specified approach, transparent decision-making, and adequate governance rather than achieving a specific outcome alone.

Role of the QPPV

The QPPV should understand:

Inspectors frequently assess QPPV awareness of major risk management activities and their effectiveness. The QPPV is expected to sign off on periodic safety reports and be able to explain evaluation outcomes and resultant actions.

Characteristics of Effective Evaluation Programmes

Effective programmes generally demonstrate:

The objective is to understand whether risk minimisation measures improve patient safety.

Structured Evaluation Checklist (Inspection-Ready)

The following checklist is presented as a practical tool to make evaluations inspection-ready and operational. Use it to assess preparedness before study start and to document readiness for inspection.

  1. Linkage and Justification
  2. [ ] Clear linkage of the evaluation to specific RMP safety concerns and regulatory commitments.
  3. [ ] Scientific rationale for chosen indicators (process/outcome) documented.

  4. Inspection relevance: Inspectors expect explicit linkage between the RMP, risk minimisation measure, and the evaluation.

  5. Objectives and Success Criteria

  6. [ ] Primary and secondary objectives clearly stated.
  7. [ ] Quantitative, time-bound success criteria defined (numerator, denominator, timeframe).
  8. [ ] Pre-specification of clinically meaningful effect sizes and statistical thresholds.

  9. Inspection relevance: Predefined success criteria reduce post-hoc interpretation bias.

  10. Study Protocol or Evaluation Plan

  11. [ ] Protocol/SAP finalized prior to data access and analysis.
  12. [ ] Version control and approval signatures (sponsor, PI, QPPV).
  13. [ ] Feasibility assessment completed and documented.

  14. Inspection relevance: Inspectors will request protocol versions and approval records.

  15. Data Sources and Quality

  16. [ ] Data sources identified and described (EHR, claims, registries, surveys, distribution logs).
  17. [ ] Data sharing agreements and data protection arrangements in place.
  18. [ ] Data quality assessments and validation plans (e.g., coding validation, completeness checks).

  19. Inspection relevance: Contracts, DTA, and data provenance are commonly inspected.

  20. Study Design and Methods

  21. [ ] Design selected with justification (e.g., ITS, cohort, cross-sectional survey, registry).
  22. [ ] Population definitions, inclusion/exclusion criteria, exposure and outcome algorithms specified.
  23. [ ] Sample size/power calculations documented where relevant.

  24. Inspection relevance: Inspectors assess methodological appropriateness and justification.

  25. Bias Control and Confounding

  26. [ ] Confounding control strategies (e.g., ITS adjustments, propensity scores) specified.
  27. [ ] Plans for sensitivity analyses and negative/positive controls included.

  28. Inspection relevance: Demonstrating awareness and mitigation of bias strengthens findings.

  29. Analysis Plan

  30. [ ] Detailed statistical analysis plan (SAP) including handling of missing data and multiplicity.
  31. [ ] Predefined primary and secondary analyses and threshold for success.
  32. [ ] Statistical software and reproducibility requirements (analytic code retention).

  33. Inspection relevance: SAPs, code and outputs are routinely requested.

  34. Governance and Oversight

  35. [ ] Roles and responsibilities defined (sponsor, study lead, statistician, QPPV).
  36. [ ] Steering committee or governance body established where necessary.
  37. [ ] Audit and monitoring plan included.
  38. [ ] Process for protocol amendments and reporting to authorities defined.

  39. Inspection relevance: Minutes and logs demonstrating governance are inspected.

  40. Ethics and Regulatory Approvals

  41. [ ] Ethics/IRB approvals obtained for primary data collection where required.
  42. [ ] Notifications to competent authorities (if PASS/PASS-like) completed.

  43. Inspection relevance: Inspectors review approvals and informed consent processes when applicable.

  44. Documentation and Archiving

    • [ ] Master file with protocol, SAP, investigator list, contracts, dataset specifications, CRFs, code, output files.
    • [ ] Documented version control and secure archiving.
    • [ ] Retention timelines aligned with regulatory requirements.
    • Inspection relevance: Immediate availability of the master file is expected during inspections.

  45. Reporting and Decision-Making

    • [ ] Reporting plan including interim and final reports, timelines, and distribution lists.
    • [ ] Decision criteria and action plan if success criteria not met.
    • [ ] Process to update the RMP and regulatory submissions defined.
    • Inspection relevance: Inspectors look for evidence that findings led to action or formal rationale for no change.

  46. Communication and Stakeholder Management

    • [ ] Stakeholder map and communication plan (HCPs, patients, regulators) in place.
    • [ ] Materials to support findings (slide decks, FAQ) prepared.
    • Inspection relevance: Transparency with regulators and stakeholders is essential.

Use this checklist as a working file within the study master file and attach evidence items to each checked box for inspection readiness.

Sample Study Protocol — Template (Inspection-Ready and Operational)

This sample protocol is an operational template for a post-implementation effectiveness evaluation. It is intentionally structured to meet regulatory expectations and inspection scrutiny. Replace bracketed text with product- and measure-specific detail.

Title: Evaluation of the Effectiveness of [Risk Minimisation Measure] for [Product Name] in Reducing [Specific Safety Concern]

Protocol ID: [Sponsor-YYYY-MM-DD-XX] Version: [v1.0] Date: [YYYY-MM-DD] Sponsor: [Company Name] Study type: [Retrospective cohort / Interrupted time series / Prospective registry / Cross-sectional survey / Mixed-methods] Linked RMP section: [RMP section and specific measure identifier]

  1. Background and Rationale
  2. Brief summary of the safety concern, approved risk minimisation measure(s), regulatory commitments, and reason for evaluation.

  3. Link to RMP, risk communication materials and previous evaluations.

  4. Objectives

  5. Primary objective:
    • To assess whether [risk minimisation measure] reduced [primary outcome] within [timeframe] compared with baseline.

  6. Secondary objectives:

    • To assess [awareness/knowledge/process indicators].
    • To evaluate subgroups ([region], [specialist vs primary care], [age groups]).
    • To assess unintended consequences (e.g., reduced prescribing in indicated populations).

  7. Predefined Success Criteria (Primary and Secondary)

  8. Primary success criterion (example):
    • A relative reduction of ≥[X]% in [primary outcome rate] from baseline to [post-intervention period], with a two-sided 95% confidence interval excluding a reduction of <[Y]%.
    • OR an absolute target (e.g., prescribing outside authorised conditions <5% within 12 months).
  9. Secondary success criteria (examples):
    • ≥[90]% of prescribers correctly answer key knowledge items in a post-intervention survey.
    • ≥[80]% compliance with monitoring requirements within [timeframe].

  10. Rationale for thresholds and timeframe, tied to clinical relevance and regulatory commitments.

  11. Study Endpoints (Definitions)

  12. Primary endpoint:
    • Numerator, denominator, measurement method and coding algorithms (ICD, ATC, procedure codes).
  13. Secondary endpoints:
    • Process indicators (distribution rates, training completion)
    • Knowledge/behavioural endpoints (survey scores, de-prescribing events)

  14. All case definitions provided in appendices (coding lists, validation algorithms).

  15. Data Sources

  16. Primary data sources:
    • [EHR database name], [claims database], [national registry], [product distribution logs], [survey platform].
  17. Data provenance and quality:
    • Description of data owners, access agreements, data refresh frequency, completeness, and prior validation studies.

  18. Data protection:

    • GDPR/comparable law compliance, pseudonymisation/ anonymisation methods, data transfer agreements.

  19. Study Design and Population

  20. Design: [e.g., Interrupted Time Series (ITS) with monthly aggregated rates for 24 months pre- and 12 months post-intervention].
  21. Population:
    • Inclusion criteria (e.g., all patients with at least one prescription of [product] during the observation period).
    • Exclusion criteria (e.g., clinical trial participants, short observation windows).
  22. Exposure definition:
    • How exposure is defined (prescription fill, dispensing, recorded administration).

  23. Observation periods:

    • Clear pre-intervention and post-intervention time windows and justification for length (seasonality considerations).

  24. Sample Size and Power (where applicable)

  25. Power calculations for the primary endpoint, assumptions and minimum detectable effect size.
  26. For ITS, justification of number of time points and expected variability.

  27. For surveys, sample size to achieve target precision (e.g., 95% CI width) and representativeness strategy.

  28. Data Collection and Management

  29. Data extraction specifications and datasets to be generated (raw extract, analysis-ready dataset).
  30. Data cleaning rules, reconciling discrepancies, handling duplicates.
  31. Documentation: data dictionary, derivation algorithms, CRFs for primary collection.

  32. Quality control procedures and validation steps (manual review of a sample of records, coder training).

  33. Statistical Analysis Plan (SAP) — summary

  34. Primary analysis:
    • Modelling approach (e.g., segmented regression ITS; Poisson/negative binomial regression for rates; logistic regression for binary outcomes).
    • Model covariates (seasonality, calendar time, co-interventions, secular trends).
    • Estimands (absolute change, relative change) and presentation (point estimates, 95% CIs).
  35. Secondary analyses:
    • Subgroup analyses, stratified analyses by region or prescriber type.
  36. Sensitivity analyses:
    • Alternative definitions, lag periods, excluding transition windows, use of negative/positive controls.
  37. Handling of missing data:
    • Imputation methods or complete-case rationale.
  38. Multiplicity:
    • Approach to multiple comparisons (hierarchical testing, emphasis on primary endpoint).
  39. Software and reproducibility:
    • Statistical software and version; code retention policy with version control.

  40. Interim analyses:

    • Pre-specified if applicable, with stopping rules or decision thresholds.

  41. Bias Assessment and Confounding Control

    • Identification of potential biases (confounding by indication, measurement error, co-interventions).
    • Control strategies (ITS to control time trends, use of comparison groups, propensity score methods).

  42. Ethics and Regulatory Considerations

    • Ethics approvals (IRB/EC) required for primary data collection or patient contact; status and timelines.
    • Regulatory notifications (e.g., PASS notification to competent authorities) where applicable.
    • Data protection and consent approach described; anonymisation/pseudonymisation methods.

  43. Governance and Oversight

    • Sponsor responsibilities and delegated parties.
    • Study steering committee composition, roles and frequency of meetings.
    • Statistical review: independent statistician review planned.
    • QPPV responsibilities: overview of QPPV oversight, sign-off on final report and regulatory submissions.
    • Audit and monitoring: internal monitoring plan and external audit rights.
    • Amendment control: SOPs for protocol amendment, regulatory notification and documentation of rationale.
    • Escalation plan: predefined triggers that require immediate review and regulatory notification (e.g., failure to meet critical safety thresholds).

  44. Quality Assurance

    • Data verification steps, source data verification where applicable.
    • Validation of coding algorithms and outcome measures.
    • Traceability from raw data to final tables and outputs, including retention of intermediate datasets.

  45. Timeline and Milestones

    • Feasibility completion: [date]
    • Protocol finalization and approvals: [date]
    • Data extraction: [date]
    • Analysis start and completion: [date]
    • Interim report (if applicable): [date]
    • Final report and submission to authorities: [date]
    • Review meeting and RMP update (if indicated): [date]

  46. Reporting and Dissemination

    • Format of final study report (ICH E3-style), contents and signatories.
    • Reporting to regulators and commitments to update the RMP or product information.
    • Publication/dissemination policy and embargoes.
    • Communication plan for stakeholders including HCPs and patients if changes are required.

  47. Limitations

    • Anticipated limitations and their likely impact on interpretation.
    • Pre-planned approaches to mitigate or document these limitations.

  48. Documentation and Inspection Readiness (Appendix)

    • List of documents to be maintained in the study master file and audit dossier:
    • Protocol and signed approvals
    • SAP and signed approvals
    • Feasibility report
    • Data transfer agreements and contracts
    • Ethics approvals and communications
    • Data dictionaries and derivation algorithms
    • Raw output files, analytic code, and code execution logs
    • Minutes of governance meetings, steering committee decisions
    • QC checks and validation reports
    • Final study report and regulatory correspondence
    • Statement of data retention and location for inspection purposes.

  49. Appendices

    • Coding algorithms (ICD, ATC lists)
    • Data extraction specifications
    • Survey questionnaires and validation information
    • Sample size calculations detail
    • Example analytic code snippets and variable lists

Practical Implementation Notes

Governance Discussion (Operational and Regulatory Context)

A robust governance framework is critical for credible and inspectable effectiveness evaluations.

Key governance elements include:

Inspection relevance: inspectors will evaluate whether governance structures are appropriate for the complexity of the evaluation and whether decisions taken were recorded and followed SOPs and regulatory requirements.

Inspection-Ready Checklist: Documents to Present

When preparing for inspection, ensure the following are indexed and readily available:

This package demonstrates transparency and traceability and is frequently requested during inspections.

Key Takeaways

Risk minimisation effectiveness evaluation requires a systematic, pre-specified and well-governed approach. The evaluation must be linked to the RMP, incorporate predefined success criteria, use appropriate methods and data, and be documented in a protocol/SAP that is inspection-ready.

Regulators and inspectors place emphasis on prospective planning, robust governance, transparent documentation, and clear decision pathways that translate evaluation findings into action for patient safety.

References

  1. EMA Good Pharmacovigilance Practices (GVP) Module V – Risk Management Systems.
  2. CIOMS IX Practical Approaches to Risk Minimisation.
  3. EMA Risk Management Plan Template.
  4. Commission Implementing Regulation (EU) No 520/2012.
  5. EMA Guidance on Risk Minimisation Measures.
  6. ENCePP Guide on Methodological Standards in Pharmacoepidemiology.
  7. ICH E2E Pharmacovigilance Planning.
  8. EMA GVP Module VIII – Post-authorisation Safety Studies (PASS).
  9. EU PAS Register — best practice for registration of pharmacoepidemiological studies.

Related Articles

Additional Risk Minimisation Measures

A practical guide to Additional Risk Minimisation Measures, including educational materials, pregnancy prevention programmes, controlled access systems and effectiveness evaluation.

Additional Pharmacovigilance Activities

A practical guide to Additional Pharmacovigilance Activities, including PASS studies, registries, enhanced monitoring and regulatory expectations.

PASS and Risk Management Plans

A practical guide to the role of PASS studies in Risk Management Plans, including study selection, regulatory expectations, governance and lifecycle management.

Last reviewed: 2026-06-11