Signal Management KPIs

Introduction

Signal management systems generate large volumes of operational data. Organisations may track hundreds of individual activities including signal detection reviews, validation decisions, assessment timelines, governance actions and escalation activities. While these operational measures are useful for process management, senior stakeholders generally require a smaller set of indicators that provide visibility of overall system performance.

Key Performance Indicators (KPIs) serve this purpose.

KPIs are selected measures used to assess whether a signal management system is operating effectively, meeting defined objectives and supporting pharmacovigilance compliance. They provide a mechanism through which governance committees, senior management and QPPVs can monitor system health without reviewing every individual activity.

Well-designed KPIs support oversight and decision-making. Poorly designed KPIs create reporting burden while providing little practical value.

KPI Versus Metric

The terms KPI and metric are often used interchangeably, but they are not identical.

A metric is a measurement.

Examples include:

• Number of validated signals
• Number of open assessments
• Number of committee meetings
• Number of actions assigned

A KPI is a management indicator derived from one or more metrics.

Examples include:

• Percentage of assessments completed within target timelines
• Percentage of overdue actions
• Percentage of validated signals assessed within required timeframes

KPIs are intended to support decisions.

Metrics provide information.

KPIs provide management insight.

Why KPIs Matter

Signal management governance depends upon visibility.

Without KPIs, it may be difficult to determine:

• Whether reviews are occurring on time
• Whether assessment backlogs are increasing
• Whether governance actions are being completed
• Whether escalation processes are functioning
• Whether resources are adequate

KPIs provide an early warning system for process deterioration.

They also help organisations demonstrate ongoing oversight of pharmacovigilance activities.

Characteristics of Effective KPIs

Effective KPIs generally possess several characteristics.

They should be:

• Relevant
• Objective
• Consistent
• Actionable
• Understandable
• Trendable

Most importantly, a KPI should influence behaviour or decision-making.

If no action would ever be taken based on a KPI result, the value of the KPI should be questioned.

Operational KPIs

Operational KPIs evaluate routine signal management activities.

Examples include:

Signal Validation Timeliness

Percentage of signal validations completed within target timelines.

Signal Assessment Timeliness

Percentage of assessments completed within defined review periods.

Assessment Backlog

Number of overdue assessments relative to total open assessments.

Signal Closure Timeliness

Percentage of signals closed within planned timelines.

These KPIs help identify workflow bottlenecks and resource constraints.

Compliance KPIs

Compliance KPIs evaluate adherence to procedures and regulatory expectations.

Examples include:

Documentation Completeness

Percentage of reviewed records meeting documentation requirements.

Governance Compliance

Percentage of signals reviewed through required governance processes.

Escalation Compliance

Percentage of signals escalated according to procedural requirements.

Review Schedule Compliance

Percentage of scheduled signal reviews completed on time.

Compliance KPIs are commonly reviewed during audits and inspections.

Governance KPIs

Governance KPIs assess oversight effectiveness.

Examples include:

Committee Action Closure Rate

Percentage of committee actions completed by due dates.

Overdue Governance Actions

Number of governance actions remaining open beyond target timelines.

Committee Attendance

Attendance rates for required governance participants.

Escalation Timeliness

Percentage of escalations performed within defined timelines.

Governance KPIs provide visibility regarding organisational control of signal management activities.

Risk-Based KPIs

Risk-based KPIs focus on significant safety concerns.

Examples include:

Open High-Priority Signals

Number of validated signals classified as high priority.

Emerging Safety Issues

Number of open Emerging Safety Issues.

Benefit-Risk Signals

Signals with potential impact on benefit-risk evaluation.

Critical Regulatory Commitments

Open signal-related regulatory commitments.

These KPIs often receive particular attention from QPPVs and senior governance bodies.

QPPV Dashboard KPIs

QPPVs typically require concise, risk-focused reporting.

A practical QPPV dashboard may include:

Signal Inventory

• Open signals
• Validated signals
• Open assessments
• Closed assessments

Timeliness

• Assessment completion rate
• Validation completion rate
• Overdue assessments

Emerging Safety Issues

• New ESIs
• Open ESIs
• Closed ESIs

Governance

• Open committee actions
• Overdue actions
• Escalated concerns

Risk Overview

• High-priority signals
• Signals affecting benefit-risk balance
• Significant ongoing assessments

The objective is visibility rather than operational detail.

Committee Dashboard KPIs

Signal management committees often require additional operational information.

Examples include:

• Assessment backlog
• Validation outcomes
• Escalation activity
• Open actions
• Trend analysis
• Resource utilisation

Committee dashboards should support discussion and decision-making rather than simply present data.

Trending KPIs

Single KPI values are often less informative than trends.

Examples include:

• Increasing overdue assessments
• Rising validation timelines
• Growing action backlogs
• Increasing numbers of high-priority signals

Trend analysis helps organisations identify problems before they become significant compliance concerns.

For this reason, KPI reviews should generally include historical comparisons.

KPI Thresholds

KPIs are most useful when supported by predefined thresholds.

Many organisations utilise traffic-light systems.

Example:

Assessment Timeliness

• Green: ≥95%
• Amber: 85%–94%
• Red: <85%

Action Closure Rate

• Green: ≥95%
• Amber: 90%–94%
• Red: <90%

Thresholds should reflect organisational expectations and risk tolerance.

Common KPI Mistakes

Several recurring problems occur in KPI design.

Measuring Activity Rather Than Performance

Example:

• Number of committee meetings

This measures activity but not effectiveness.

Excessive KPI Volume

Too many KPIs dilute management attention.

Lack of Defined Actions

KPIs should trigger investigation or action when thresholds are breached.

Lack of Trend Analysis

Isolated values may conceal important developments.

Absence of Risk Focus

KPIs should support management of significant safety concerns rather than merely operational reporting.

KPI Review During Inspections

Inspectors frequently explore KPI programmes when evaluating signal management systems.

Typical questions include:

• Which KPIs are reviewed?
• Who reviews them?
• How often are they reviewed?
• What actions result from adverse trends?
• Can examples be provided?

Inspectors are generally interested in how KPIs are used rather than how many KPIs exist.

A small number of meaningful KPIs is often more valuable than a large volume of rarely reviewed measures.

KPI Governance

KPIs should themselves be subject to governance.

Organisations should define:

• KPI ownership
• Review frequency
• Escalation thresholds
• Reporting recipients
• Corrective action expectations

Governance ensures that KPIs remain relevant and continue to support decision-making.

Characteristics of Mature KPI Programmes

Mature KPI programmes generally demonstrate:

• Clear objectives
• Risk-based indicators
• Consistent reporting
• Trend analysis
• Defined thresholds
• Governance review
• Action-oriented oversight

The purpose of a KPI programme is not measurement for its own sake.

Its purpose is to support effective management of signal-related risks and pharmacovigilance performance.

Key Takeaways

KPIs provide management-level visibility of signal management performance and governance effectiveness.

KPIs differ from operational metrics because they are designed to support decision-making and oversight.

Useful KPI programmes typically include operational, compliance, governance and risk-based indicators.

QPPVs and governance committees commonly utilise KPI dashboards to monitor significant safety concerns and system performance.

Inspectors frequently assess not only which KPIs are reported but how KPI information is used to drive actions and continuous improvement.

References

1. EMA Good Pharmacovigilance Practices (GVP) Module IX – Signal Management.
2. EMA Good Pharmacovigilance Practices (GVP) Module I – Pharmacovigilance Systems and Their Quality Systems.
3. EMA Good Pharmacovigilance Practices (GVP) Module III – Pharmacovigilance Inspections.
4. Commission Implementing Regulation (EU) No 520/2012.
5. CIOMS VIII Practical Aspects of Signal Detection in Pharmacovigilance.
6. ICH E2E Pharmacovigilance Planning.
7. ICH Q10 Pharmaceutical Quality System.