Signal Management KPIs

Introduction

Signal management systems generate large volumes of operational data. Organisations may track hundreds of individual activities including signal detection reviews, validation decisions, assessment timelines, governance actions and escalation activities. While these operational measures are useful for process management, senior stakeholders generally require a smaller set of indicators that provide visibility of overall system performance.

Key Performance Indicators (KPIs) serve this purpose.

KPIs are selected measures used to assess whether a signal management system is operating effectively, meeting defined objectives and supporting pharmacovigilance compliance. They provide a mechanism through which governance committees, senior management and QPPVs can monitor system health without reviewing every individual activity.

Well-designed KPIs support oversight and decision-making. Poorly designed KPIs create reporting burden while providing little practical value.

KPI Versus Metric

The terms KPI and metric are often used interchangeably, but they are not identical.

A metric is a measurement.

Examples include:

• Number of validated signals
• Number of open assessments
• Number of committee meetings
• Number of actions assigned

A KPI is a management indicator derived from one or more metrics.

Examples include:

• Percentage of assessments completed within target timelines
• Percentage of overdue actions
• Percentage of validated signals assessed within required timeframes

KPIs are intended to support decisions.

Metrics provide information.

KPIs provide management insight.

Why KPIs Matter

Signal management governance depends upon visibility.

Without KPIs, it may be difficult to determine:

• Whether reviews are occurring on time
• Whether assessment backlogs are increasing
• Whether governance actions are being completed
• Whether escalation processes are functioning
• Whether resources are adequate

KPIs provide an early warning system for process deterioration.

They also help organisations demonstrate ongoing oversight of pharmacovigilance activities.

Characteristics of Effective KPIs

Effective KPIs generally possess several characteristics.

They should be:

• Relevant
• Objective
• Consistent
• Actionable
• Understandable
• Trendable

Most importantly, a KPI should influence behaviour or decision-making.

If no action would ever be taken based on a KPI result, the value of the KPI should be questioned.

Operational KPIs

Operational KPIs evaluate routine signal management activities.

Examples include:

Signal Validation Timeliness

Percentage of signal validations completed within target timelines.

Signal Assessment Timeliness

Percentage of assessments completed within defined review periods.

Assessment Backlog

Number of overdue assessments relative to total open assessments.

Signal Closure Timeliness

Percentage of signals closed within planned timelines.

These KPIs help identify workflow bottlenecks and resource constraints.

Compliance KPIs

Compliance KPIs evaluate adherence to procedures and regulatory expectations.

Examples include:

Documentation Completeness

Percentage of reviewed records meeting documentation requirements.

Governance Compliance

Percentage of signals reviewed through required governance processes.

Escalation Compliance

Percentage of signals escalated according to procedural requirements.

Review Schedule Compliance

Percentage of scheduled signal reviews completed on time.

Compliance KPIs are commonly reviewed during audits and inspections.

Governance KPIs

Governance KPIs assess oversight effectiveness.

Examples include:

Committee Action Closure Rate

Percentage of committee actions completed by due dates.

Overdue Governance Actions

Number of governance actions remaining open beyond target timelines.

Committee Attendance

Attendance rates for required governance participants.

Escalation Timeliness

Percentage of escalations performed within defined timelines.

Governance KPIs provide visibility regarding organisational control of signal management activities.

Risk-Based KPIs

Risk-based KPIs focus on significant safety concerns.

Examples include:

Open High-Priority Signals

Number of validated signals classified as high priority.

Emerging Safety Issues

Number of open Emerging Safety Issues.

Benefit-Risk Signals

Signals with potential impact on benefit-risk evaluation.

Critical Regulatory Commitments

Open signal-related regulatory commitments.

These KPIs often receive particular attention from QPPVs and senior governance bodies.

QPPV Dashboard KPIs

QPPVs typically require concise, risk-focused reporting.

A practical QPPV dashboard may include:

Signal Inventory

• Open signals
• Validated signals
• Open assessments
• Closed assessments

Timeliness

• Assessment completion rate
• Validation completion rate
• Overdue assessments

Emerging Safety Issues

• New ESIs
• Open ESIs
• Closed ESIs

Governance

• Open committee actions
• Overdue actions
• Escalated concerns

Risk Overview

• High-priority signals
• Signals affecting benefit-risk balance
• Significant ongoing assessments

The objective is visibility rather than operational detail.

Committee Dashboard KPIs

Signal management committees often require additional operational information.

Examples include:

• Assessment backlog
• Validation outcomes
• Escalation activity
• Open actions
• Trend analysis
• Resource utilisation

Committee dashboards should support discussion and decision-making rather than simply present data.

Trending KPIs

Single KPI values are often less informative than trends.

Examples include:

• Increasing overdue assessments
• Rising validation timelines
• Growing action backlogs
• Increasing numbers of high-priority signals

Trend analysis helps organisations identify problems before they become significant compliance concerns.

For this reason, KPI reviews should generally include historical comparisons.

KPI Thresholds

KPIs are most useful when supported by predefined thresholds.

Many organisations utilise traffic-light systems.

Example:

Assessment Timeliness

• Green: ≥95%
• Amber: 85%–94%
• Red: <85%

Action Closure Rate

• Green: ≥95%
• Amber: 90%–94%
• Red: <90%

Thresholds should reflect organisational expectations and risk tolerance.

Explicit KPI Definitions, Formulas and Implementation Details

To be inspection-ready, each KPI must be unambiguously defined, traceable to data sources, reproducible by independent calculation and supported by documented rationale for thresholds. The sections below contain explicit KPI definitions, calculation formulas, inclusion/exclusion rules, recommended calculation frequency and practical implementation notes.

Note: Time periods and target timelines are organisationally defined (e.g., 30/60/90 days) and must be referenced to applicable SOPs or service level agreements (SLAs).

1. Assessment Completion Rate (Signal Assessment Timeliness)

• Definition: Proportion of completed signal assessments that were finalised within the defined target assessment period.
• Formula:
• Assessment Completion Rate (%) = (Number of signal assessments completed within target period during reporting period / Number of signal assessments completed during reporting period) × 100
• Numerator: Count of signal assessments with completion date ≤ validation/assessment start date + target assessment timeframe.
• Denominator: Count of all signal assessments with documented completion date within the reporting period.
• Inclusion/Exclusion:
• Include: All company-initiated and joint assessments where the company is responsible for the assessment deliverable.
• Exclude: Assessments on hold with documented reason (regulatory pause, pending external data) if SOP permits exclusion; require documented criteria.
• Frequency: Monthly; trend reported quarterly to governance.
• Data sources: Signal management system (e.g., ArisG, Argus Signal, PV database), assessment records, timestamps from document control system.
• Implementation notes: Ensure consistent definition of "assessment start date" (e.g., date of validation decision) and audit trails to support timestamps.

2. Validation Timeliness

• Definition: Proportion of candidate signals validated (accepted or rejected) within the target validation timeframe from detection to validation decision.
• Formula:
• Validation Timeliness (%) = (Number of validations completed within defined timeframe during period / Total number of validations completed during period) × 100
• Numerator: Validations where validation decision date − detection/referral date ≤ validation timeframe.
• Denominator: Total validations with decision date in period.
• Inclusion/Exclusion: Exclude duplicates and administrative rejects per SOP.
• Frequency: Weekly operational; monthly governance summary.
• Data sources: Signal detection logs, validation decision records, detection algorithms output.

3. Assessment Backlog Ratio

• Definition: Proportion of open assessments that are overdue relative to total open assessments.
• Formula:
• Assessment Backlog Ratio (%) = (Number of open assessments past target completion date / Total number of open assessments) × 100
• Numerator: Open assessments with current date > planned completion date and no documented exception.
• Denominator: Total open assessments as of report date.
• Frequency: Weekly.
• Data sources: Signal tracking system, action trackers, committee minutes for documented exceptions.
• Implementation notes: Maintain reasons for overdue status for trend and root-cause analysis.

4. Committee Action Closure Rate

• Definition: Proportion of governance committee actions closed by their agreed due date.
• Formula:
• Committee Action Closure Rate (%) = (Number of committee actions completed by due date / Total number of committee actions with due dates in period) × 100
• Numerator: Actions with completion date ≤ due date.
• Denominator: All actions assigned with definitive due dates within reporting period.
• Frequency: Monthly after committee meeting.
• Data sources: Committee minutes, action tracker, CAPA system.
• Inspection evidence: Signed committee minutes, action assignments with owners, evidence of closure (deliverables, dates).

5. Overdue Governance Actions

• Definition: Absolute count of governance actions that have exceeded their due date and remain open.
• Calculation:
• Overdue Governance Actions = Count(actions where status = 'open' AND current date > due date)
• Frequency: Weekly.
• Data sources: Action tracker, governance system.
• Implementation notes: For inspection readiness, include justification for each overdue action, mitigation steps and updated target dates.

6. Documentation Completeness Rate

• Definition: Percentage of assessments or validation records that meet predefined documentation completeness criteria.
• Formula:
• Documentation Completeness Rate (%) = (Number of records meeting all checklist items / Number of records sampled) × 100
• Numerator: Records that pass a completeness checklist (e.g., dates, authorship, rationale, references).
• Denominator: Size of sample or total records reviewed (sampling strategy must be documented).
• Frequency: Quarterly or per inspection-sampling cadence.
• Data sources: Assessment dossiers, document management system, completeness checklist.
• Implementation notes: Retain completed checklists and version control evidence.

7. Escalation Compliance Rate

• Definition: Proportion of signals that were escalated according to predefined procedural criteria within the required timeframes.
• Formula:
• Escalation Compliance (%) = (Number of required escalations completed within timeframe / Number of instances requiring escalation) × 100
• Frequency: Monthly.
• Data sources: Signal database, escalation logs, email trails or governance ticketing system.

8. Open High-Priority Signals

• Definition: Count of validated signals currently classified as high priority (organisation’s priority classification).
• Calculation:
• Open High-Priority Signals = Count(signals where priority = 'High' AND status != 'Closed')
• Frequency: Weekly.
• Data sources: Signal priority registry, safety database.
• Implementation notes: Document criteria for 'High' classification and any re-classification events.

9. Emerging Safety Issues (ESI) Status

• Definition: Inventory and class-level status counts for ESIs (new, ongoing, closed).
• Calculations:
• New ESIs (period) = Count(ESIs with validation date in reporting period)
• Open ESIs = Count(ESIs with status = 'Open')
• Closed ESIs (period) = Count(ESIs closed during reporting period)
• Frequency: Monthly.
• Data sources: ESI registry, signal database, committee minutes.

10. Root Cause and CAPA Rate

• Definition: Percentage of KPI breaches that have an identified root cause and an initiated CAPA within specified governance timelines.
• Formula:
• Root Cause & CAPA Rate (%) = (Number of KPI breaches with documented root cause and CAPA initiated / Total KPI breaches in period) × 100
• Frequency: Monthly for operational issues, quarterly for trend review.
• Data sources: CAPA system, root-cause analysis documentation, KPI review logs.

Data Sources and Data Lineage

Accurate KPIs depend on clear data lineage and trusted sources. Primary data sources commonly used:

• Signal management system (primary): signal records, detection outputs, validation and assessment timestamps, priority fields.
• Safety database (ICSR system): case counts, aggregated case-level data, timelines where relevant.
• Document management system: assessment reports, sign-off evidence, version history.
• Committee governance system: meeting agendas, minutes, attendance records, action trackers.
• CAPA and quality management systems: investigations, corrective/preventive actions and status.
• Master data (product master, country master, roles): to ensure consistent inclusion/exclusion rules.

Practical implementation details:

• Define a single source of truth for each KPI data element (e.g., "assessment completion date = date recorded in SignalSystem.Assessment.CompletionDate").
• Maintain a data dictionary that maps KPI fields to system tables/fields and to SOP definitions.
• Implement automated extracts where feasible (SQL, API), and retain extraction scripts with version control.
• Perform routine reconciliations between data sources (e.g., signal system vs. document management system) and log reconciliation outcomes.
• Document data refresh frequency and latency (e.g., daily extract at 02:00 UTC).

Sample Queries and Pseudocode

Below are illustrative pseudocode snippets to guide implementation. Adapt to specific systems and SQL dialects. Replace field and table names with local schema equivalents.

• Assessment Completion Rate (pseudocode): SELECT SUM(CASE WHEN DATEDIFF(day, AssessmentStartDate, CompletionDate) <= @TargetDays THEN 1 ELSE 0 END) AS OnTimeCount, COUNT() AS TotalCompleted, (SUM(CASE WHEN DATEDIFF(day, AssessmentStartDate, CompletionDate) <= @TargetDays THEN 1 ELSE 0 END) * 100.0) / COUNT() AS CompletionRatePct FROM SignalAssessments WHERE CompletionDate BETWEEN @PeriodStart AND @PeriodEnd AND AssessmentType IN ('CompanyOwned','Co-sponsor') -- per SOP

• Overdue Governance Actions: SELECT COUNT(*) AS OverdueActions FROM CommitteeActions WHERE Status = 'Open' AND DueDate < GETDATE()

Ensure all queries are stored in a controlled repository and that results are reproducible by audit.

Sample Dashboard Visuals (Design Templates)

Dashboards should present concise KPI cards, trend charts and drill-down capability. Below are recommended visual components and ASCII mockups to illustrate layout. For inspections, include snapshot exports (PDF) and underlying data tables.

1. Executive KPI Row (Top-level, one-line summary)
2. KPI card: Assessment Completion Rate — 92% (green)
3. KPI card: Validation Timeliness — 88% (amber)
4. KPI card: Open High-Priority Signals — 4 (red)
5. KPI card: Open ESIs — 2 (amber)
6. KPI card: Committee Action Closure Rate — 96% (green)

ASCII card example: [Assessment Completion Rate] 92% ▲ +2% vs prior month

1. Trend Panel
2. Line chart: Assessment Completion Rate (last 12 months)
3. Line chart: Validation Timeliness (last 12 months)
4. Stacked bar: Open vs Overdue Assessments (monthly)

ASCII mini-trend: Month: J F M A M J Rate%: 90 91 89 92 93 92 (sparkline)

1. Backlog and Workload Panel
2. Heatmap by product: number of open assessments by priority
3. Bar chart: Ageing of open assessments (0–30, 31–60, 61–90, >90 days)

ASCII ageing table: Priority | 0–30 | 31–60 | 61–90 | >90 High | 2 | 1 | 1 | 0 Medium | 10 | 3 | 1 | 0 Low | 5 | 2 | 0 | 0

1. Governance Panel
2. Table: Open actions with owner, due date, days overdue, status

3. Bar: Committee attendance % per member over last 6 meetings

4. Drill-Down Capability

5. From KPI card to a detailed report listing constituent records (IDs, dates, owner, reason for delay)
6. Export functionality for inspection packs (PDF or CSV)

Design notes: - Include timestamp of last data refresh and data source name on every dashboard. - Provide links to SOPs, definitions, and data dictionary. - Ensure calendar-based filtering (month, quarter, year) and product/country filters. - Export-ready snapshots: include a "print-friendly" layout for each dashboard view to support inspection requests.

Inspection Evidence and Reproducibility

Inspectors will evaluate not only KPI values but the systems and processes that produce them. Prepare the following artefacts for inspection:

• KPI definitions document (single master document) that includes formula, numerator/denominator, inclusion/exclusion rules, target timeframe, business rationale and threshold values.
• Data dictionary mapping KPI fields to system table/field names and queries.
• Stored query scripts or API call documentation and execution logs.
• Timestamped dashboard exports (PDF) for requested periods and the underlying raw data CSVs.
• SOP references demonstrating governance for KPI generation, review and remediation.
• Evidence of periodic KPI review meetings (minutes), decisions taken, CAPAs initiated and closure evidence.
• Reconciliation reports between signal system and document management system showing data consistency checks.
• Change log for KPI definitions and dashboard configuration (who changed what and when).
• Training records for users responsible for KPI generation and interpretation.

Governance Checklist (Inspection-Ready)

Use the checklist below to ensure KPI programme maturity and inspection readiness. For each item, retain supporting evidence.

1. KPI Ownership and Accountability
2. [ ] Each KPI has an assigned owner (name, role).

3. Evidence: KPI definitions document with owner listed.

4. Formal KPI Definitions

5. [ ] KPI formula, numerator, denominator, inclusion/exclusion rules documented.
6. [ ] References to applicable SOPs and timelines included.

7. Evidence: Central KPI definitions file.

8. Single Source of Truth

9. [ ] Data source for each KPI field specified and authoritative.

10. Evidence: Data dictionary and system mapping.

11. Reproducible Calculations

12. [ ] Stored queries/API scripts exist and are version controlled.

13. Evidence: Query repository, script execution logs.

14. Data Quality Controls

15. [ ] Routine reconciliations and exception reports performed and documented.

16. Evidence: Reconciliation logs and issue registers.

17. Thresholds and Rationale

18. [ ] Thresholds defined, risk-based and agreed by governance.

19. Evidence: KPI threshold policy and governance approval record.

20. Review Cadence and Escalation

21. [ ] KPI review frequency defined (operational vs governance) and adhered to.
22. [ ] Escalation pathways defined for KPI breaches.

23. Evidence: Review calendar, meeting minutes, escalation records.

24. Audit Trail and Change Control

25. [ ] Changes to KPI definitions or dashboard configuration are under change control.

26. Evidence: Change control logs, version history.

27. Documentation Completeness & Sampled Evidence

28. [ ] Samples of assessment dossiers and completeness checklists retained.

29. Evidence: Sample packages, completed checklists.

30. Committee Integration

    • [ ] KPIs are presented to appropriate committees and QPPV with actions documented.
    • Evidence: Committee packs, minutes, action log entries.

31. CAPA and Root Cause Linkage

    • [ ] KPI breaches linked to root-cause analyses and CAPAs where required.
    • Evidence: CAPA records and closure evidence.

32. Training and Competency

    • [ ] Staff responsible for KPI production and interpretation are trained.
    • Evidence: Training records and competency assessments.

33. Archive and Retention

    • [ ] Historical KPI data and dashboard snapshots archived per records retention policy.
    • Evidence: Archive logs and retrieval tests.

34. Regulatory Alignment

    • [ ] KPI programme mapped to applicable GVP modules and inspection expectations (GVP Module IX, Module I, Module III).
    • Evidence: Mapping document with regulatory citations.

Completing and maintaining this checklist with documentary evidence strengthens inspection readiness and demonstrates a controlled approach to KPI governance.

KPI Programme Governance and Change Management

KPI governance should be integrated into the pharmacovigilance quality system. Key governance activities:

• Periodic review of KPI portfolio for ongoing relevance and risk alignment.
• Formal approval of KPI thresholds by governance committees and QPPV.
• Annual review of KPI definitions and data sources or when significant process/system changes occur.
• Controlled deprecation and introduction of KPIs with retrospective impact analysis.
• Inclusion of KPI performance in management review and PSMF content where relevant.

Change management considerations:

• Implement changes under quality change control with impact assessment on historical comparability.
• Communicate changes to all dashboard consumers and provide transitional mapping for trend continuity.
• Re-run historical calculations when definition changes materially affect trend interpretation and retain both historical and new metric versions for transparency.

KPI Review During Inspections (Expanded)

Inspectors typically focus on:

• The set of KPIs that the organisation uses and why those were chosen.
• Documentation showing exact KPI definitions and how calculations were derived.
• Evidence the QPPV receives and reviews relevant KPI information.
• Samples of KPI-driven actions: examples where KPI trends triggered investigations or CAPAs.
• Availability of raw data to reproduce KPI values.

To address this, prepare:

• A concise inspection pack including KPI definitions, last 12 months of KPI trends, representative drill-down reports for red/amber KPI events, CAPA narratives and meeting minutes demonstrating management response.
• Demonstrable reproducibility: allow inspectors to run stored queries or provide the exact scripts and CSV outputs.
• Cross-reference to GVP modules and supporting evidence in the PSMF.

Characteristics of Mature KPI Programmes

Mature KPI programmes generally demonstrate:

• Clear objectives
• Risk-based indicators
• Consistent reporting
• Trend analysis
• Defined thresholds
• Governance review
• Action-oriented oversight

The purpose of a KPI programme is not measurement for its own sake.

Its purpose is to support effective management of signal-related risks and pharmacovigilance performance.

Key Takeaways

KPIs provide management-level visibility of signal management performance and governance effectiveness.

KPIs differ from operational metrics because they are designed to support decision-making and oversight.

Useful KPI programmes typically include operational, compliance, governance and risk-based indicators.

QPPVs and governance committees commonly utilise KPI dashboards to monitor significant safety concerns and system performance.

Inspectors frequently assess not only which KPIs are reported but how KPI information is used to drive actions and continuous improvement. A controlled, documented and reproducible KPI programme with clear ownership, data lineage and evidence of governance is essential for inspection readiness.

References

1. EMA Good Pharmacovigilance Practices (GVP) Module IX – Signal Management.
2. EMA Good Pharmacovigilance Practices (GVP) Module I – Pharmacovigilance Systems and Their Quality Systems.
3. EMA Good Pharmacovigilance Practices (GVP) Module III – Pharmacovigilance Inspections.
4. Commission Implementing Regulation (EU) No 520/2012.
5. CIOMS VIII Practical Aspects of Signal Detection in Pharmacovigilance.
6. ICH E2E Pharmacovigilance Planning.
7. ICH Q10 Pharmaceutical Quality System.