PSMF Metrics and KPIs

Introduction

Many organisations maintain a PSMF.

Far fewer actively measure whether it remains healthy.

This distinction is important.

A PSMF can exist and still:

• Contain outdated information
• Miss critical vendors
• Omit products
• Misrepresent governance structures
• Create inspection risk

The challenge is that document quality often deteriorates gradually.

Without objective indicators, organisations may not recognise problems until an audit or inspection occurs.

Metrics help solve this problem.

They transform PSMF governance from a subjective activity into a measurable one.

Why Metrics Matter

Historically, PSMF maintenance was often viewed as a documentation exercise.

Modern pharmacovigilance systems are too complex for that approach.

A mature organisation should be able to answer questions such as:

• Is the PSMF current?
• Are annexes accurate?
• Are inventories complete?
• Are updates occurring on time?
• Are governance reviews effective?

Metrics provide evidence-based answers.

The Purpose of PSMF Metrics

The objective of metrics is not reporting.

The objective is governance.

Effective metrics help organisations:

• Detect deterioration early
• Monitor compliance
• Prioritise resources
• Support inspections
• Support QPPV oversight

Metrics should therefore focus on risk rather than activity.

Characteristics of Useful Metrics

Strong metrics are:

• Objective
• Repeatable
• Actionable
• Risk-based
• Easy to understand

Weak metrics often measure activity rather than control.

For example:

Weak Metric

Number of PSMF pages.

Strong Metric

Percentage of annexes reviewed on schedule.

The second metric provides meaningful governance information.

PSMF Health Framework

A practical approach is to group metrics into several categories.

Completeness

Is information present?

Accuracy

Is information correct?

Timeliness

Is information current?

Governance

Are controls functioning?

Inspection Readiness

Could the organisation defend the document during an inspection?

Together these categories provide a balanced view of PSMF health.

Completeness Metrics

Completeness metrics assess whether required information exists.

Examples include:

Annex Completion Rate

Measures whether required annexes exist.

Example:

Vendor Inventory Coverage

Measures whether all PV vendors appear within the inventory.

Product Inventory Coverage

Measures whether all products are included.

Missing information often represents one of the most common inspection findings.

Accuracy Metrics

Accuracy is frequently more important than completeness.

An inaccurate inventory may be more problematic than an incomplete one.

Examples include:

Inventory Accuracy Rate

Assesses whether sampled entries are correct.

Vendor Accuracy Rate

Measures agreement between vendor inventories and contractual records.

Organisational Accuracy Rate

Assesses whether organisational charts reflect actual reporting structures.

Regular verification activities support these metrics.

Timeliness Metrics

Timeliness measures whether information remains current.

Examples include:

Overdue Updates

Number of updates exceeding defined timelines.

Average Update Time

Time between a business change and PSMF update.

Review Compliance

Percentage of reviews completed on schedule.

A document can be accurate today but inaccurate tomorrow.

Timeliness metrics help prevent gradual deterioration.

Product Inventory Metrics

Product inventories often represent one of the largest maintenance burdens.

Useful measures include:

Products Pending Update

Products awaiting inclusion or revision.

Inventory Review Compliance

Percentage of scheduled reviews completed.

Product Reconciliation Findings

Differences identified during verification activities.

These metrics are particularly important following acquisitions and divestments.

Vendor Oversight Metrics

Vendor inventories often change frequently.

Examples include:

Vendor Inventory Accuracy

Percentage of vendors accurately recorded.

Missing Vendor Rate

Vendors identified outside formal inventories.

Oversight Review Completion

Completion of planned vendor oversight reviews.

For additional information see:

[[vendor-oversight]]

Governance Metrics

Governance metrics assess whether oversight mechanisms function effectively.

Examples include:

Governance Review Completion

Scheduled reviews completed on time.

Escalation Compliance

Issues escalated according to procedures.

Action Item Closure

Governance actions completed within agreed timelines.

These indicators help determine whether oversight is active rather than theoretical.

QPPV Oversight Metrics

The PSMF should support effective QPPV oversight.

Examples include:

Significant Changes Reported

Percentage of significant changes communicated to the QPPV.

Governance Review Participation

Attendance and participation within governance meetings.

Risk Review Completion

Scheduled risk reviews completed on time.

These metrics provide visibility regarding oversight effectiveness.

For additional information see:

[[psmf-qppv-oversight]]

Change Control Metrics

Many PSMF deficiencies originate from poor change control.

Useful indicators include:

Change Assessment Completion

Percentage of significant changes assessed for PSMF impact.

Change-to-Update Time

Time between approved change and documented update.

Outstanding Change Actions

Open activities awaiting completion.

Strong performance here often correlates with strong inspection outcomes.

Audit Metrics

Audits help verify whether the PSMF reflects reality.

Examples include:

Audit Coverage

Percentage of planned audits completed.

PSMF Findings

Audit observations related to the PSMF.

Repeat Findings

Issues recurring after previous CAPAs.

Repeat findings frequently indicate ineffective governance.

CAPA Metrics

CAPA performance often predicts future inspection outcomes.

Useful indicators include:

Open CAPAs

Current active CAPAs.

Overdue CAPAs

Actions exceeding target dates.

CAPA Effectiveness Rate

Percentage of CAPAs verified as effective.

Repeat Deficiencies

Issues recurring after closure.

These metrics help assess whether improvements are sustainable.

Inspection Readiness Metrics

Many organisations claim to be inspection-ready.

Metrics provide evidence.

Examples include:

Annex Currency

Percentage of annexes updated within required timelines.

Inventory Reconciliation Success

Percentage of reconciliations completed without discrepancies.

Critical Deficiency Count

Number of unresolved high-risk issues.

Inspection Readiness Score

Composite indicator combining multiple measures.

Such indicators provide management visibility regarding inspection preparedness.

Dashboard Design

An effective dashboard should remain simple.

A practical PSMF dashboard may include:

The objective is visibility rather than volume.

Common Metric Mistakes

Several mistakes occur repeatedly.

Measuring Activity Rather Than Risk

Example:

• Number of updates performed

Instead of:

• Number of overdue updates

Excessive Complexity

Large dashboards often obscure important information.

Lack of Thresholds

Metrics without escalation criteria provide limited value.

Lack of Trend Analysis

Single values rarely tell the full story.

Trend analysis often reveals deterioration earlier.

What Great Organisations Measure

High-performing organisations focus on:

• Accuracy
• Timeliness
• Governance effectiveness
• Inspection readiness

They use metrics to drive action rather than simply generate reports.

Importantly, they view the PSMF as a living governance system rather than a static document.

Key Takeaways

• Metrics help detect deterioration before inspections identify deficiencies.
• Effective metrics focus on risk rather than activity.
• Completeness, accuracy and timeliness are foundational measurement areas.
• Vendor inventories and product inventories require ongoing monitoring.
• Governance metrics help determine whether oversight is functioning effectively.
• CAPA and audit metrics provide insight into continuous improvement.
• Mature organisations use dashboards to support QPPV oversight and inspection readiness.

References

1. EMA Good Pharmacovigilance Practices (GVP) Module I – Pharmacovigilance Systems and Their Quality Systems.
2. EMA Good Pharmacovigilance Practices (GVP) Module II – Pharmacovigilance System Master File.
3. EMA Good Pharmacovigilance Practices (GVP) Module III – Pharmacovigilance Inspections.
4. Regulation (EC) No 726/2004.
5. Directive 2001/83/EC.
6. Commission Implementing Regulation (EU) No 520/2012.
7. EMA Questions and Answers on Pharmacovigilance System Master Files.