Signal Management Metrics

Introduction

Signal management systems generate substantial volumes of information. Organisations perform signal detection activities, validate observations, conduct assessments, implement actions and escalate important safety concerns. Without objective measures of performance, it can be difficult to determine whether these activities are operating effectively.

Metrics provide a structured approach to monitoring signal management processes. They allow organisations to evaluate workload, timeliness, compliance, governance effectiveness and operational performance.

Metrics are not a substitute for scientific judgement. A signal management system cannot be evaluated solely through numbers. However, metrics can identify trends, highlight emerging issues and support informed governance decisions.

Well-designed metrics help organisations understand whether processes are functioning as intended and whether significant risks may be developing within the signal management system itself.

Why Metrics Matter

Signal management involves multiple activities occurring over extended periods.

These activities include:

• Signal detection
• Validation
• Prioritisation
• Assessment
• Escalation
• Action implementation
• Closure

Metrics provide visibility across these activities and support:

• Governance oversight
• Resource planning
• Process improvement
• Inspection readiness
• Compliance monitoring

Without metrics, organisations may struggle to identify delays, backlogs or systemic weaknesses.

Objectives of Signal Management Metrics

Metrics should support decision-making.

The purpose of a metric is not simply to generate data.

Effective metrics should help answer questions such as:

• Are signals being reviewed on time?
• Are assessments progressing appropriately?
• Are actions being completed?
• Are governance processes functioning effectively?
• Are resources sufficient?

Metrics that do not support meaningful decisions often create reporting burden without improving oversight.

Categories of Signal Management Metrics

Most metrics fall into several broad categories.

These include:

• Volume metrics
• Timeliness metrics
• Quality metrics
• Governance metrics
• Outcome metrics
• Compliance metrics

A balanced metric programme typically includes measures from multiple categories.

Over-reliance on a single category may provide an incomplete picture of system performance.

Volume Metrics

Volume metrics measure workload and activity levels.

Examples include:

• Number of detected observations
• Number of validated signals
• Number of ongoing assessments
• Number of closed signals
• Number of escalated concerns

Volume metrics help organisations understand workload trends and resource requirements.

However, volume alone provides little information regarding quality or effectiveness.

A large number of signals does not necessarily indicate a problem, nor does a small number indicate success.

Timeliness Metrics

Timeliness is one of the most commonly monitored aspects of signal management.

Examples include:

• Detection review timelines
• Validation completion times
• Assessment completion times
• Escalation timelines
• Action completion timelines

Timeliness metrics help identify process delays and operational bottlenecks.

Because delayed evaluation of important safety concerns may create regulatory and public health risks, timeliness often receives significant management attention.

Validation Metrics

Validation metrics focus on the review of detected observations.

Examples include:

• Number of observations reviewed
• Percentage validated
• Percentage closed during validation
• Average validation duration

These metrics help organisations understand whether validation activities are functioning efficiently.

Unexpected changes in validation outcomes may indicate changes in reporting patterns, detection methods or review practices.

Assessment Metrics

Assessment metrics evaluate the progression of validated signals through detailed review activities.

Examples include:

• Open assessments
• Closed assessments
• Assessment duration
• Overdue assessments
• Assessment backlog

Assessment metrics are particularly useful for identifying resource constraints and workflow challenges.

A growing backlog may indicate increasing workload or insufficient capacity.

Escalation Metrics

Escalation metrics help organisations understand how significant concerns move through governance structures.

Examples include:

• Signals escalated to committees
• Emerging Safety Issues identified
• Escalations to senior management
• Escalations to the QPPV

These metrics provide visibility into the significance of reviewed safety concerns and the effectiveness of governance processes.

Governance Metrics

Governance metrics evaluate oversight activities.

Examples include:

• Committee meeting frequency
• Attendance rates
• Action completion rates
• Governance review timelines

Governance metrics help demonstrate that oversight activities are functioning appropriately.

Inspectors frequently review governance effectiveness indirectly through these types of measures.

Action Tracking Metrics

Signal assessments frequently generate actions.

Examples include:

• Additional analyses
• Literature reviews
• Regulatory submissions
• Product information updates

Useful metrics may include:

• Number of open actions
• Overdue actions
• Action completion rates
• Average completion times

Failure to monitor actions may result in assessments that are completed scientifically but not implemented operationally.

Compliance Metrics

Compliance metrics evaluate adherence to internal procedures and regulatory expectations.

Examples include:

• Reviews completed within target timelines
• Assessments approved appropriately
• Documentation completeness
• Escalation compliance

Compliance metrics are often used during audits and inspections to assess process control.

However, compliance should not be confused with effectiveness.

A process may meet timelines while still failing to identify important safety concerns.

Outcome Metrics

Outcome metrics evaluate the consequences of signal management activities.

Examples include:

• Product information updates
• Risk minimisation measures implemented
• Additional studies initiated
• Regulatory actions arising from signals

Outcome metrics may help organisations understand the broader impact of signal management activities.

Interpretation requires caution because outcomes are influenced by many factors beyond process performance.

Metrics for QPPV Oversight

The QPPV typically requires a high-level view of signal management performance.

Examples of metrics commonly reviewed by QPPVs include:

• Significant validated signals
• Emerging Safety Issues
• Assessment backlogs
• Overdue assessments
• Escalated concerns
• Open critical actions

These metrics support oversight without requiring review of every operational activity.

The objective is to provide visibility regarding significant risks and system performance.

Metrics for Committees

Governance committees often utilise metrics to support decision-making.

Typical committee dashboards may include:

• Open signal inventory
• High-priority signals
• Assessment status
• Action tracking
• Escalation summaries

Committee metrics should support discussion and decision-making rather than serve solely as informational reports.

Trending and Analysis

Individual metrics are often less informative than trends.

Examples include:

• Increasing assessment backlogs
• Rising numbers of escalations
• Changes in validation rates
• Growing action delays

Trend analysis may identify emerging process weaknesses before they become significant compliance concerns.

For this reason, many organisations review metrics over multiple reporting periods rather than focusing exclusively on current values.

Common Metric Pitfalls

Several problems occur frequently.

Measuring Activity Instead of Performance

Large numbers of completed reviews may not indicate effective signal management.

Excessive Metrics

Too many metrics may obscure important information.

Lack of Context

Metrics without explanation may be misleading.

Focus on Timelines Alone

Timeliness is important but does not measure scientific quality.

Failure to Trend Data

Single data points rarely provide meaningful insight.

Lack of Action

Metrics should inform decisions. Metrics that never influence actions provide limited value.

Metrics During Inspections

Inspectors may review signal management metrics to understand system performance.

Common areas of interest include:

• Assessment backlogs
• Timeliness trends
• Escalation activities
• Governance effectiveness
• Action completion rates

Inspectors often focus on how metrics are used rather than merely whether they exist.

The ability to demonstrate that metrics support oversight and continuous improvement is generally more important than the number of metrics reported.

Characteristics of Effective Metric Programmes

Effective programmes generally demonstrate:

• Clear objectives
• Meaningful measures
• Reliable data
• Trend analysis
• Governance visibility
• Action-oriented reporting

Metrics should support understanding rather than create unnecessary complexity.

The most useful metrics are often those that directly support decisions and oversight activities.

Key Takeaways

Signal management metrics provide visibility into process performance, governance effectiveness, compliance and operational workload.

Useful metrics may address volume, timeliness, quality, governance, actions and outcomes.

The QPPV and governance committees commonly rely on metrics to support oversight of significant safety concerns and process performance.

Trend analysis is generally more informative than isolated measurements.

Metrics should support decision-making, continuous improvement and regulatory compliance rather than exist solely for reporting purposes.

References

1. EMA Good Pharmacovigilance Practices (GVP) Module IX – Signal Management.
2. EMA Good Pharmacovigilance Practices (GVP) Module I – Pharmacovigilance Systems and Their Quality Systems.
3. EMA Good Pharmacovigilance Practices (GVP) Module III – Pharmacovigilance Inspections.
4. Commission Implementing Regulation (EU) No 520/2012.
5. CIOMS VIII Practical Aspects of Signal Detection in Pharmacovigilance.
6. ICH E2E Pharmacovigilance Planning.
7. ICH Q10 Pharmaceutical Quality System.
8. ISPE Quality Metrics Guidance.