Disproportionality Analysis in Pharmacovigilance

Introduction

Modern pharmacovigilance systems generate very large volumes of safety data. National spontaneous reporting databases, EudraVigilance, the FDA Adverse Event Reporting System (FAERS) and other safety databases may contain millions of Individual Case Safety Reports (ICSRs). Manual review of every possible product-event combination is therefore impractical.

Disproportionality analysis was developed to assist with signal detection in large safety databases. The objective is to identify product-event combinations that are reported more frequently than would be expected based on overall reporting patterns within the database.

These methods are widely used by regulatory authorities, Marketing Authorisation Holders and signal management teams. They are particularly useful as screening tools because they can rapidly identify observations that may warrant further evaluation.

However, disproportionality analyses do not establish causality. A statistical association identified through disproportionality analysis should be regarded as a hypothesis requiring further investigation rather than evidence that a medicinal product causes a particular event.

Purpose of Disproportionality Analysis

Disproportionality analysis seeks to identify reporting patterns that differ from what would be expected if no association existed between a medicinal product and an event.

If a particular adverse event is reported more frequently for a product than would be expected relative to other products in the database, the observation may warrant further review.

The output of disproportionality analysis is therefore a potential signal rather than a regulatory conclusion. These methods are intended to support signal detection activities and should be interpreted within the broader context of signal validation and signal assessment.

Disproportionality Analysis Within Signal Management

Disproportionality analysis forms part of signal detection rather than signal assessment.

A simplified relationship is:

Safety Database
        ↓
Disproportionality Analysis
        ↓
Potential Statistical Signal
        ↓
Signal Validation
        ↓
Signal Assessment

The statistical output generated by disproportionality methods is only one source of information. Clinical review, literature findings, epidemiological evidence and biological plausibility remain important components of signal management.

Core Measures: ROR, PRR, IC and EBGM — Comparison Table

The table below summarises the principal measures used for disproportionality screening: formulae, common interpretation thresholds, typical use-cases and practical notes about implementation and inspection relevance.

Notes: - The table gives canonical forms; specific implementations (e.g., inclusion/exclusion rules, stratification, shrinkage priors) vary between organisations and databases. Regulatory expectations (see GVP Module IX and EVDAS guidance) require transparent documentation of chosen methods and thresholds. - "Observed" = a; "Expected" = (product_total * event_total)/N. Shrinkage methods adjust Observed/Expected to reduce influence of small counts.

Worked numerical example (step-by-step)

This worked example demonstrates calculation of ROR, PRR, IC and a simplified empirical Bayes shrinkage estimate (illustrative only; MGPS/EBGM uses more sophisticated mixture priors).

Assume the following 2x2 counts extracted from a safety database:

• a = reports for product of interest with the event = 20
• b = reports for product of interest with other events = 980 (product total = 1,000)
• c = reports for same event with all other products = 200
• d = reports for other products with other events = 79,800 (others total = 80,000)
• N = a + b + c + d = 81,000
• Event total = a + c = 220

1) Reporting Odds Ratio (ROR)

ROR = (a/b) / (c/d)

• a/b = 20/980 = 0.020408
• c/d = 200/79,800 = 0.0025063
• ROR = 0.020408 / 0.0025063 ≈ 8.14

Approximate 95% CI for log(ROR): - SE(log ROR) = sqrt(1/a + 1/b + 1/c + 1/d) = sqrt(1/20 + 1/980 + 1/200 + 1/79,800) ≈ 0.2368 - log(ROR) = ln(8.14) ≈ 2.099 - 95% CI log = 2.099 ± 1.96 × 0.2368 → lower log = 1.635, upper log = 2.563 - 95% CI ROR ≈ [exp(1.635), exp(2.563)] = [5.13, 12.99]

Interpretation: ROR 8.14 (95% CI 5.13–12.99) strongly suggestive of disproportional reporting.

2) Proportional Reporting Ratio (PRR)

PRR = [a/(a+b)] / [c/(c+d)]

• a/(a+b) = 20/1,000 = 0.02
• c/(c+d) = 200/80,000 = 0.0025
• PRR = 0.02 / 0.0025 = 8.0

Historical rule-of-thumb: PRR ≥ 2, chi-square ≥ 4 and a ≥ 3 triggers a flag. Here PRR = 8 and a = 20, indicating a strong disproportionality by PRR.

3) Information Component (IC) — basic calculation

IC = log2(Observed / Expected)

• Observed = a = 20
• Expected = (product_total * event_total) / N = (1,000 * 220) / 81,000 ≈ 2.716
• Observed/Expected = 20 / 2.716 ≈ 7.366
• IC = log2(7.366) ≈ 2.88

Interpretation: A positive IC indicates more reports than expected. Regulatory practice (UMC/WHO) flags when IC025 (the lower credibility limit) > 0. For this magnitude (Observed much greater than Expected), IC025 would typically be > 0.

4) Simplified empirical Bayes shrinkage illustration (conjugate Gamma-Poisson)

This is a didactic simplification. MGPS/EBGM as used in practice uses a mixture of gamma priors and returns EBGM and EB05 (lower bound). A simple conjugate prior Gamma(α, β) with α = 1, β = 1 gives a posterior mean RR estimate:

Posterior mean RR = (α + Observed) / (β + Expected)

• Using α = β = 1:
• Posterior mean RR = (1 + 20) / (1 + 2.716) = 21 / 3.716 ≈ 5.65

This "shrinks" the naive Observed/Expected (≈7.37) towards the prior expectation (1.0). In MGPS/EBGM, EBGM would be produced alongside EB05 (lower 90% bound); if EB05 > 1 the combination is generally considered disproportionate.

Key point from the example: - All four measures indicate strong disproportionality here, but magnitudes differ because of the shrinkage approach (EBGM) and different scalings (ROR vs PRR vs IC). - In practice, one should present both the raw counts and the chosen disproportionality metrics plus confidence/credibility bounds, and then proceed to clinical review.

Practical implementation details

This section lists practical points to ensure robust operationalisation of disproportionality analysis.

• Data pre-processing:
• De-duplicate ICSRs (clear SOP for deduplication).
• Map events to consistent terminology (MedDRA coding level — PT preferred; ability to examine HLGT/HLT for clustered signals).
• Apply consistent product coding (active substance, brand, route, indication).

• Decide inclusion/exclusion: spontaneous reports only vs all reports, literature cases, solicited reports, clinical trial reports.

• Choice of time-window:

• Rolling windows (e.g., last 3, 6, 12 months) are common; define windows in SOPs.

• Consider both cumulative and interval analyses to detect emergent trends and avoid dilution.

• Stratification and subgrouping:

• Age, sex, region, reporting source (healthcare professional vs consumer), seriousness, route of administration and indication can be stratified to reduce confounding and masking.

• Pre-specify stratification schemes where feasible (e.g., paediatric vs adult, country-level).

• Shrinkage and small counts:

• Use shrinkage (IC, EBGM) to reduce false positives when counts are small.

• Document prior choices and their rationale; for MGPS/EBGM implementations, provide prior mixture detail.

• Masking and confounding:

• Be aware of masking (strongly reported product-event combinations can obscure signals for other products).

• Implement methods to detect masking (e.g., sequential removal of high-count products) where resources allow.

• Multiple testing and prioritisation:

• There are hundreds of thousands of product-event pairs; implement triage rules (thresholds, minimum counts, clinical filters).

• Combine quantitative flags with qualitative criteria (seriousness, novelty, plausibility).

• Software and reproducibility:

• Use validated software (internal or commercial). Record software version, libraries, and parameter settings.

• Store code in version control; keep dataset snapshots that generated each run.

• Outputs and reporting:

• Produce ranked lists, with counts, metrics (ROR, PRR, IC, EBGM), CIs/credibility bounds, and time series plots.

• Include links to underlying ICSRs and prepared medical summaries to facilitate rapid case review.

• Integration with signal management:

• Define workflows for screening → triage → validation → assessment.
• Capture decisions, rationale, and escalation steps in the signal tracking system.

Regulatory context

Disproportionality analyses are explicitly referenced by regulators as part of signal management frameworks:

• EMA: GVP Module IX (Signal Management) outlines expectations for detection activities, the need to document methods, thresholds, validation and follow-up.
• EudraVigilance: EVDAS provides outputs used by Member States and MAHs; its documentation stresses that EVDAS signals require follow-up and are not definitive regulatory signals.
• UMC/WHO: IC and IC025 are standard outputs in global signal detection and are incorporated into WHO methodologies.
• FDA: FAERS analyses and empirical Bayesian approaches (EBGM) are referenced in FDA publications; FAERS outputs form part of FDA surveillance but require clinical follow-up.

Regulators expect transparency on methodology, governance and how statistical outputs feed into the overall signal management process. During inspections, authorities will review whether an organisation’s approach aligns with regulatory requirements and good practice.

Inspection relevance and what inspectors commonly review

Inspectors commonly examine the following aspects of disproportionality programmes:

• Method selection and justification:
• Rationale for chosen metrics (ROR, PRR, IC, EBGM) and thresholds.

• Evidence that methods are suitable for the organisation’s product portfolio and databases.

• Documentation and SOPs:

• SOPs describing data sources, pre-processing, de-duplication, coding standards (MedDRA version), and analytic flows.

• Detailed specifications on thresholds, time-windows, stratification, and prioritisation rules.

• Validation and testing:

• Validation of analytic code and tools (unit tests, reconciliations with reference outputs).

• Performance characteristics (sensitivity, specificity) if evaluated locally.

• Records and traceability:

• Archived datasets and reports that produced signals.

• Audit trail of decisions (who reviewed, what was concluded, and why actions were taken or not taken).

• Governance and escalation:

• Defined roles (signal detection analyst, medical reviewer, QPPV/designee, safety committee).

• Clear escalation paths and timelines for urgent signals.

• Reproducibility:

• Ability to reproduce results (same input dataset producing same output).

• Version control for code and data snapshots.

• Integration with regulatory reporting:

• Evidence that identified signals are considered when preparing PSURs/PBRERs, RMPs and expedited reports where relevant.

Inspectors often focus less on the choice of a particular statistical metric and more on whether the programme is robustly implemented, documented, validated and governed.

Governance and oversight

Effective governance ensures that disproportionality analyses translate into robust decision-making.

Key governance elements:

• Policy and SOP framework:
• Company policy on statistical signal detection that references GVP and local regulatory expectations.

• SOPs covering data management, analytic procedures, thresholds, triage, validation and documentation.

• Roles & responsibilities:

• Data analysts/statisticians implement and run analyses.
• Signal detection leads prioritise and triage outputs.
• Clinical safety physicians perform validation and assessment.

• QPPV or pharmacovigilance head maintains overall accountability.

• Decision-making forum:

• Regular safety signal review committee meetings with documented minutes and action items.

• Pre-defined criteria for escalation (e.g., serious events, high disproportionality, public health impact).

• Metrics and quality assurance:

• KPIs for timeliness of review, proportion of signals closed at validation, number of regulatory actions stemming from detection.

• Periodic audits of the signal detection process and external benchmarking.

• Training and competence:

• Ongoing training for analysts and medical reviewers on methods, interpretation and regulatory expectations.

• Change control:

• Formal change control for algorithm modifications, software upgrades and threshold changes including impact assessments and back-testing.

Inspection-readiness checklist (concise)

Prepare the following items to demonstrate inspection readiness for disproportionality analysis:

• Governance & policies:
• Current policy on signal detection referencing GVP Module IX.

• Organisational chart showing roles and responsibilities (analysts, reviewers, QPPV).

• SOPs and work instructions:

• SOPs for data extraction, deduplication, MedDRA mapping/version control, analytic methods and triage.

• Work instructions for routine runs (time-windows, stratifications).

• Methodology documentation:

• Formal description of the statistical methods (ROR, PRR, IC, EBGM) and precise formulas used.

• Thresholds, rationale and any supplementary filters (e.g., seriousness, minimum counts).

• Validation and testing:

• Validation report for analysis software (test cases, reconciliation with expected outputs).

• Code repository access and version history for scripts used (with change control records).

• Run artefacts:

• Most recent dataset snapshot and run output (ranked lists with counts and metrics).

• Archived older runs to demonstrate trend analysis and reproducibility.

• Signal records:

• Example signal files demonstrating workflow from statistical flag → triage → validation → outcome.

• Minutes from signal review meetings and evidence of actions or closure rationale.

• Technical configuration:

• Software name/version and list of statistical libraries.

• Environment documentation (production/validation separation).

• Training & competence:

• Training records for staff performing and interpreting disproportionality analyses.

• Risk assessments:

• Impact assessment for changes to detection algorithms or thresholds.

• Evidence of periodic review of detection performance (false positives/negatives).

• Integration with regulatory reporting:

• Evidence that signal outcomes have been considered for PBRERs/RMP updates and expedited reports where applicable.

Provide these materials in a way that permits rapid demonstration of reproducibility (e.g., a short SOP-driven run that reproduces a selected result).

Common operational pitfalls and mitigation

• Pitfall: Lack of deduplication or inconsistent coding → false signals.

• Mitigation: Robust deduplication SOP, MedDRA mapping governance, review of high-ranked combinations.

• Pitfall: Undocumented thresholds and unapproved algorithm changes.

• Mitigation: Change control, versioned policies, impact assessment and back-testing.

• Pitfall: Over-reliance on a single metric.

• Mitigation: Use multiple complementary measures and integrate qualitative clinical review.

• Pitfall: Poor linkage between statistical flags and signal management workflow.

• Mitigation: Clear triage SOPs, automated routing to reviewers, and tracking of actions.

Clinical review and escalation

Statistical flags must enter a documented clinical review pathway. That review should:

• Access underlying cases and narratives.
• Evaluate temporality, dechallenge/rechallenge, concomitant drugs and confounders.
• Search literature and other data sources (EHRs, registries) where feasible.
• Decide on next steps: monitor, request more data, perform epidemiological study, regulatory communication.
• Document the rationale and timeline for closure or escalation.

Escalation criteria should be explicit (for example: serious outcome, high ROR/EB05, plausible mechanism, vulnerable population affected).

Key takeaways

• ROR, PRR, IC and EBGM are complementary tools for screening large safety databases; each has strengths and limitations.
• Shrinkage methods (IC, EBGM) provide stability for sparse data; simple ratios (ROR, PRR) are intuitive and widely used.
• Robust implementation requires pre-specified methods, reliable data processing, validation, governance and documented triage pathways.
• Regulatory expectations prioritise transparency, reproducibility and appropriate clinical follow-up; inspectors focus on documentation, validation and governance rather than on a single chosen statistical method.
• Maintain inspection readiness by keeping policies, SOPs, validation evidence, run outputs and signal records organised and accessible.

References

1. EMA Good Pharmacovigilance Practices (GVP) Module IX – Signal Management.
2. EudraVigilance Data Analysis System (EVDAS) Guidance.
3. CIOMS VIII Practical Aspects of Signal Detection in Pharmacovigilance.
4. Bate A, Evans SJW. Quantitative Signal Detection Using Spontaneous ADR Reporting.
5. Hauben M, Aronson JK. Defining Signal and Its Subtypes in Pharmacovigilance.
6. van Puijenbroek EP et al. A Comparison of Measures of Disproportionality for Signal Detection.
7. Uppsala Monitoring Centre. Bayesian Signal Detection Methodology.
8. ICH E2E Pharmacovigilance Planning.