QPPV.COM

Vendor KPIs and Metrics in Pharmacovigilance

A practical guide to vendor performance measurement, KPIs, dashboards, governance metrics and inspection-ready oversight frameworks.

Audio Lesson 9 min

Vendor KPIs and Metrics in Pharmacovigilance

Introduction

Vendor oversight depends upon visibility.

Without objective information, organisations cannot determine whether outsourced activities are:

This is why metrics are fundamental to modern vendor governance.

Well-designed KPIs help organisations identify risks before they become inspection findings.

Poorly designed KPIs often create the illusion of control while hiding meaningful problems.

For this reason, mature organisations place significant emphasis on performance measurement.

Why Metrics Matter

Most vendor relationships generate large amounts of information.

Examples include:

The challenge is determining which information actually matters.

Metrics help transform raw information into decision-support tools.

A useful principle is:

Metrics should support action.

If a metric cannot influence a decision, its value may be limited.

The Purpose of Vendor KPIs

The primary objectives are:

The objective is not reporting for its own sake.

The objective is maintaining control of outsourced activities.

Characteristics of Useful KPIs

Strong KPIs are:

Weak KPIs often measure activity rather than performance.

Example:

Weak KPI

Number of cases processed.

Strong KPI

Percentage of cases processed within required timelines.

The second metric provides meaningful oversight information.

KPI Categories

A practical framework is to group metrics into categories.

Compliance

Are regulatory obligations being met?

Quality

Are activities performed correctly?

Operational Performance

Are services delivered effectively?

Governance

Are oversight activities functioning?

Improvement

Are problems being resolved?

Together these categories provide a balanced view.

Compliance KPIs

Compliance metrics are often among the most important indicators.

Examples include:

Timeliness Compliance

Measures whether activities occur within required timelines.

Examples:

Submission Compliance

Measures whether required submissions occur correctly.

Reconciliation Compliance

Measures completion of reconciliation activities.

These indicators often receive significant inspection attention.

Quality KPIs

Quality metrics assess whether activities are performed accurately.

Examples include:

Quality Review Findings

Number of findings identified during review activities.

Error Rate

Percentage of records containing errors.

Rework Rate

Percentage of work requiring correction.

Quality Trend Analysis

Monitoring changes over time.

Quality metrics frequently reveal emerging problems before compliance failures occur.

Operational KPIs

Operational metrics assess delivery effectiveness.

Examples include:

Throughput

Volume of work completed.

Capacity Utilisation

Resource usage.

Turnaround Time

Time required to complete activities.

Backlog Levels

Outstanding work.

Operational indicators help identify resource pressures.

Governance KPIs

Governance metrics assess oversight effectiveness.

Examples include:

Governance Meeting Attendance

Participation in scheduled reviews.

Action Item Closure

Completion of governance actions.

Escalation Compliance

Timely escalation of significant issues.

Risk Review Completion

Completion of planned reviews.

Governance metrics are often overlooked despite their importance.

Audit KPIs

Audit metrics provide insight into control effectiveness.

Examples include:

Audit Findings

Number of findings identified.

Repeat Findings

Previously identified issues that recur.

Audit Coverage

Percentage of planned audits completed.

Finding Closure

Timely resolution of findings.

These indicators often predict future inspection outcomes.

For additional information see:

[[vendor-audits]]

CAPA KPIs

CAPA metrics assess whether problems are being resolved effectively.

Examples include:

Open CAPAs

Current active CAPAs.

Overdue CAPAs

Actions exceeding target dates.

CAPA Effectiveness

Percentage of CAPAs verified as effective.

Repeat Deficiencies

Recurring issues after closure.

Strong CAPA performance often correlates with mature governance.

Risk-Based KPI Models

Not all vendors require identical metrics.

A low-risk vendor may require:

A critical vendor may require:

Risk classification should influence monitoring intensity.

For additional information see:

[[vendor-risk-assessment]]

KPI Thresholds

Metrics become more useful when thresholds are defined.

Example:

KPI Green Amber Red
Timeliness ≥98% 95–97% <95%
Quality ≥97% 94–96% <94%
CAPA Closure ≥90% 80–89% <80%

Thresholds support escalation decisions.

Single data points rarely tell the full story.

Trend analysis often provides greater value.

Examples include:

Stable Performance

Consistent results over time.

Gradual Deterioration

Small declines over several months.

Sudden Changes

Rapid shifts requiring investigation.

Many significant compliance issues become visible through trends before major failures occur.

Vendor Dashboards

A useful dashboard should provide visibility rather than complexity.

Typical dashboard areas include:

Area Example KPI
Compliance Timeliness
Quality Error Rate
Operations Backlog
Governance Action Closure
Audits Open Findings
CAPAs Overdue CAPAs

The objective is to support decision making.

Metrics and the QPPV

The QPPV cannot personally monitor every operational detail.

Metrics help provide visibility.

Particularly important areas may include:

Metrics support effective oversight by highlighting areas requiring attention.

For additional discussion see:

[[vendor-oversight-for-qppvs]]

Metrics and Inspections

Inspectors frequently review:

A common question is:

How does the organisation know when vendor performance is deteriorating?

Metrics often form a significant part of the answer.

Regulatory expectations for measurement and oversight are described in EMA GVP Module I (pharmacovigilance systems and their quality systems), Module III (inspections) and in ICH Q9 (risk management). Inspectors expect documented, reproducible measurement systems with evidence trails, escalation criteria and governance actions.

Common KPI Mistakes

Several weaknesses occur repeatedly.

Measuring Activity

Tracking workload rather than effectiveness.

Too Many Metrics

Large dashboards become difficult to interpret.

No Thresholds

Performance expectations are unclear.

No Trend Analysis

Deterioration remains unnoticed.

No Action

Metrics are collected but not used.

These weaknesses significantly reduce value.

Characteristics of Mature Measurement Programmes

High-performing organisations generally demonstrate:

Risk-Based Monitoring

Metrics align with risk.

Defined Thresholds

Escalation criteria exist.

Trend Analysis

Performance is assessed over time.

Governance Integration

Metrics influence decisions.

Continuous Improvement

Metrics support organisational learning.

These characteristics strengthen vendor oversight significantly.

Key Takeaways

Detailed KPI Definition Table

The table below provides a practical, inspection-oriented KPI definition set that can be adopted and adapted. Each KPI entry includes a precise definition, numerator/denominator, calculation method, authoritative data sources, the responsible owner(s), threshold bands, reporting frequency and the evidence an inspector would expect to see.

KPI Description Numerator Denominator Calculation Data sources Owner Thresholds (G/A/R) Reporting frequency Required evidence Inspection relevance
Initial Case Intake Timeliness Percentage of new safety reports entered into the PV database within target intake time (SLA, e.g., 3 business days) Number of new cases entered within intake SLA Total number of new cases received during period Numerator ÷ Denominator × 100 Vendor case intake logs, PV safety database timestamped entries, email receipts, eCRF or portal receipts Vendor PV Lead; Sponsor Vendor Manager ≥98% / 95–97% / <95% Weekly/monthly (monthly for governance) Extract of PV database with timestamps, intake logs, SLA, sample cases showing timestamps, intake SOP Demonstrates timeliness of data capture and evidence of date/time stamps; inspectors verify chain of custody and timeliness
ICSR Expedited Reporting Compliance (15/7/90) % of expedited reports (e.g., CIOMS/MedDRA serious) reported to health authorities within regulatory timeframe Number of expedited reports submitted within required regulatory timeline Total number of expedited reports due in period Numerator ÷ Denominator × 100 PV database, regulatory submission logs, submission confirmations, vendor eCTD/E2B submission records Vendor PV Lead; Regulatory Affairs / QPPV ≥99% / 97–98% / <97% Monthly Submission confirmations, submission timestamps, proof of distribution, SOMS/Regulatory tracker, SOPs for expedited reporting Core regulatory KPI — inspectors check proofs of submission, timelines and escalation when missed
Case Quality Error Rate (post-QC) % of cases with at least one quality finding after QC/quality review Number of cases with ≥1 QC finding Total number of cases reviewed by QC in period Numerator ÷ Denominator × 100 QC review logs, QC checklists, PV database flags, audit trails Vendor Quality Manager; Sponsor QA ≤3% / 3–6% / >6% Monthly QC checklists, examples of corrected cases, root cause analyses, training records Indicates effectiveness of vendor processes and need for remediation; inspectors review samples and trend
ICSR Completeness Score % of required data elements completed (per minimum dataset) across sampled cases Sum of completed required data elements across sample Maximum possible required data elements across sample (Numerator ÷ Denominator) × 100 PV database extracts, case ICSRs, minimum dataset checklist Vendor PV Lead ≥95% / 90–94% / <90% Monthly/quarterly Minimum dataset checklist, sample ICSR bundles, database extracts, mapping documents Demonstrates data quality and compliance with reporting standards; inspectors review sample completeness
Follow-up Reporting Timeliness % of follow-up reports transmitted within target (e.g., within 30/90 days depending on dataset) Number of follow-up reports transmitted within SLA Total number of follow-up reports due Numerator ÷ Denominator × 100 PV database follow-up status, correspondence logs, submission confirmations Vendor PV Lead; Regulatory Affairs ≥95% / 90–94% / <90% Monthly Follow-up schedule, sample follow-up reports, submission confirmations Inspectors check follow-up handling and documentation of medical information
Backlog (>30 days) % of open cases older than 30 days since intake Number of open cases >30 days Total open cases Numerator ÷ Denominator × 100 PV database case status reports, case aging reports Vendor Operations Manager; Sponsor Vendor Manager ≤3% / 3–7% / >7% Weekly/monthly Case aging reports, root-cause analysis for aged cases, capacity plans Aging backlogs are red flags in inspections — provide plan to clear and evidence of resource allocation
Reconciliation Completion Rate % of reconciliations completed between vendor and sponsor (e.g., ADR case lists, payment/transaction reconciliations) Number of reconciliations completed by due date Number of reconciliations scheduled Numerator ÷ Denominator × 100 Reconciliation trackers, signed reconciliation logs, emails Vendor Finance/Operations; Sponsor PV Lead 100% / 95–99% / <95% Monthly/quarterly Signed reconciliation records, evidence of remediation for discrepancies Inspectors expect reconciliations where required and documented resolution of discrepancies
Audit Findings per Audit Average number of findings per audit (or % audits with ≥1 critical/major) Total number of findings (by severity) identified in period Number of audits performed Numerator ÷ Denominator Audit reports, audit trails, management responses Sponsor QA / Vendor QA Low/no criticals; ≤2 major/critical per audit acceptable / >2 major or any critical triggers escalation Quarterly Audit reports, management response, corrective action plans, evidence of verification Audit findings and closure demonstrate control environment; repeat findings are a key inspector focus
CAPA Overdue % % of CAPAs overdue beyond agreed target completion date Number of CAPAs overdue Total open CAPAs Numerator ÷ Denominator × 100 CAPA tracker, CAPA logs, verification records Sponsor QA; Vendor QA ≤5% / 5–15% / >15% Monthly CAPA tracker export, evidence of extension approvals, verification records Inspectors review CAPA backlog, evidence of tracking, and effectiveness verification
CAPA Effectiveness Rate % of CAPAs closed and subsequently verified effective at pre-defined verification interval Number of CAPAs verified effective Number of CAPAs closed and due for effectiveness check Numerator ÷ Denominator × 100 CAPA tracker, verification reports, trend analyses Sponsor QA; Vendor QA ≥90% / 80–89% / <80% Quarterly Verification reports, metrics pre/post CAPA, trend tables Demonstrates that corrective actions produce sustained improvement — inspector interest in effectiveness
Escalation Timeliness % of issues meeting escalation criteria that are escalated within defined timeline Number of escalations completed within SLA Total number of issues meeting escalation criteria Numerator ÷ Denominator × 100 Escalation logs, meeting minutes, email evidence Sponsor PV Lead; Vendor Senior Manager ≥95% / 90–94% / <90% Monthly Escalation log, emails, evidence of decision and follow-up actions Inspectors verify that significant issues are escalated per SOPs and that appropriate actions were taken
Governance Action Closure % of governance actions (e.g., from oversight meetings) closed by agreed due date Number of governance actions closed on time Total governance actions assigned Numerator ÷ Denominator × 100 Governance minutes, action trackers Sponsor Governance Lead; QPPV oversight ≥95% / 90–94% / <90% Monthly/quarterly Meeting minutes, action tracker export, evidence for closed actions Governance processes and timely closure are central to inspections — minutes and evidence should be available
E2B / Transmission Success Rate % of electronic transmissions (E2B messages) successfully transmitted and accepted by authorities Number of successful transmissions accepted by recipient Total number of intended transmissions Numerator ÷ Denominator × 100 Transmission logs, delivery receipts, RA confirmations Vendor IT/Regulatory 100% / 98–99% / <98% Monthly Transmission logs, delivery receipts, retry and error logs, change control if tool updated Inspectors look for reliable electronic exchange and evidence of handling failed transmissions
Deviation Closure Timeliness % of deviations closed by due date Number of deviations closed on time Total deviations opened in period Numerator ÷ Denominator ×100 Deviation logs, CAPA/CAPA linkage Vendor QA; Sponsor QA ≥95% / 90–94% / <90% Monthly Deviation register exports, investigation reports, closure evidence Demonstrates quality culture; inspectors review deviation investigations and closures
Training Compliance (PV SOPs) % of vendor staff trained/recertified on applicable PV SOPs within interval Number of staff with current training records Number of staff requiring training Numerator ÷ Denominator × 100 LMS reports, attendance logs, sign-off sheets Vendor HR/Quality; Sponsor Training Lead 100% / 95–99% / <95% Quarterly Training records, training matrix, certificates Training evidence is commonly requested in inspections as indicator of competence

Notes on thresholds: thresholds should be set against regulatory requirements, contractual SLAs and risk appetite. The sample thresholds above are commonly used starting points; organisations must calibrate bands based on risk classification and historical performance.

Practical implementation details for KPI delivery

Inspection-Readiness Implementation Checklist

The following checklist translates KPI definitions into actionable steps that prepare a vendor oversight programme for inspection. Each item includes the actions required, suggested owner(s), frequency and the specific evidence inspectors will expect to review.

  1. KPI Definitions Document (KPI Dictionary)
  2. Action: Create and maintain a controlled KPI definitions document listing each KPI, precise definitions, numerator/denominator, calculation script, data source, owner, thresholds and reporting cadence.
  3. Owner: Sponsor PV Lead / QA; Reviewed/approved by QPPV.
  4. Frequency: Review annually or after major process changes.

  5. Evidence: Signed KPI dictionary, version history, change log.

  6. Data Source and Lineage Mapping

  7. Action: Map all source systems (PV database, vendor CRM, audit system, CAPA tracker, LMS) to KPI inputs; document ETL processes and responsibilities.
  8. Owner: Vendor IT / Sponsor IT / Data Owner.
  9. Frequency: Initially and after system changes.

  10. Evidence: Data lineage diagrams, ETL runbooks, validation records.

  11. Data Extraction and Validation Procedures

  12. Action: Implement and document extraction procedures and reconciliation checks between source and dashboard extracts.
  13. Owner: Vendor Operations / Sponsor Data Lead.
  14. Frequency: Each reporting period; validation after any change.

  15. Evidence: Reconciliation logs, checksum files, date-stamped extracts.

  16. KPI Calculation Scripts and Version Control

  17. Action: Store calculation scripts (SQL, PowerBI DAX, R/Python) in a controlled repository with access control and change history.
  18. Owner: Data Analyst / Vendor IT.
  19. Frequency: As needed; validate after change.

  20. Evidence: Script repository, change control records, validation reports.

  21. Dashboard and Report Validation

  22. Action: Validate dashboards against raw extracts; document test cases and pass/fail criteria.
  23. Owner: Sponsor QA; Vendor Operations.
  24. Frequency: Quarterly and before inspection.

  25. Evidence: Validation test cases, test results, sign-off.

  26. Evidence Retention and Pack Preparation

  27. Action: For each KPI reporting period, compile an evidence pack: raw extracts, reconciliation logs, sample cases, meeting minutes, CAPA evidence and any corrective actions.
  28. Owner: Sponsor Vendor Manager; Vendor PV Lead.
  29. Frequency: Monthly reports; maintain historical packs for inspection (recommended minimum 2 years typical, longer if required).

  30. Evidence: Pack index, stored in controlled repository with access log.

  31. SOPs and Work Instructions

  32. Action: Maintain SOPs covering KPI governance, data management, escalation, metric definitions, and inspection response procedures.
  33. Owner: Sponsor QA; Vendor QA.
  34. Frequency: Review annually or after process change.

  35. Evidence: SOPs with revision history, approved copies.

  36. Escalation Matrix and Governance Meeting Cadence

  37. Action: Define escalation criteria tied to KPI thresholds, specify roles to be notified and required timeline for action, and set governance meeting schedules (weekly operational, monthly management, quarterly executive).
  38. Owner: Sponsor PV Lead; QPPV.
  39. Frequency: As events occur; meetings per schedule.

  40. Evidence: Escalation matrix, action logs, meeting agendas and minutes, escalation emails.

  41. CAPA and Remediation Tracking

  42. Action: Ensure CAPAs are linked to KPI failures, tracked with due dates and effectiveness checks. CAPA ownership must be clear.
  43. Owner: Sponsor QA; Vendor QA.
  44. Frequency: Ongoing; monthly review.

  45. Evidence: CAPA tracker exports, root cause analyses, verification reports.

  46. Audit and Oversight Programme Alignment

    • Action: Align KPI programme with audit schedule; use audits to validate KPI integrity.
    • Owner: Sponsor QA; Audit function.
    • Frequency: As per audit cycle.
    • Evidence: Audit reports, responses, evidence of KPI audit trails.

  47. Training and Competency Records

    • Action: Ensure staff using KPI systems are trained; maintain training matrix and certificates.
    • Owner: Vendor HR/Training; Sponsor Training Lead.
    • Frequency: Annually or as per local policy.
    • Evidence: LMS reports, training records.

  48. Mock Inspections and Evidence Walkthroughs

    • Action: Run mock inspections focused on KPI evidence packs, data lineage and sample case audits to stress-test readiness.
    • Owner: Sponsor QA / PV Lead.
    • Frequency: Annually; before regulatory inspections.
    • Evidence: Mock inspection reports, action plans and closure evidence.

  49. Data Integrity and ALCOA+ Checks

    • Action: Implement regular checks for data integrity (timeliness, completeness, uniqueness) and document corrective actions for exceptions.
    • Owner: Vendor IT / QA.
    • Frequency: Quarterly.
    • Evidence: Data integrity checks, exception logs, remediation records.

  50. Contractual and Quality Agreement Review

    • Action: Ensure SLAs, Quality Agreements and Data Processing Agreements explicitly capture KPI requirements, thresholds, reporting cadence and audit access.
    • Owner: Legal; Vendor Management; Sponsor QA.
    • Frequency: At contract setup and annually.
    • Evidence: Signed Quality Agreement, SLA, appendices naming KPIs.

  51. Produce an Inspection Evidence Pack (ready-to-issue)

    • Action: Pre-assemble a standard evidence pack to provide on request: KPI dictionary, latest dashboard exports, three months of monthly packs, reconciliation outputs, sample case bundles and audit reports.
    • Owner: Sponsor PV Lead; QA.
    • Frequency: Maintain continuously; update monthly.
    • Evidence: Pack index, secure storage path, access logs.

  52. Contact Directory and Responsibility Matrix

    • Action: Maintain an up-to-date contact list of vendor and sponsor leads for escalation during inspections.
    • Owner: Vendor Manager; Sponsor PV Lead.
    • Frequency: Quarterly.
    • Evidence: Contact list within evidence pack and governance minutes.

  53. Regulatory Change and Impact Assessment Log

    • Action: Document regulatory changes that impact KPIs (e.g., changes to reporting timelines, new electronic submission rules) and perform impact assessments.
    • Owner: Regulatory Affairs; Sponsor PV Lead.
    • Frequency: As required.
    • Evidence: Change log, impact assessments, approved implementation plans.

What inspectors typically request (prepare these in advance) - KPI dictionary and definition approvals signed by sponsor/QPPV/QA. - Raw data extracts used to produce KPI reports (with timestamps and user logs). - Dashboard export and calculation scripts. - Monthly KPI reports with evidence packs for the reporting period (reconciliations, sample cases). - SOPs covering KPI governance, data extraction, and escalation. - Recent governance meeting minutes and action trackers linked to KPI results. - Audit reports referencing KPI processes and CAPA evidence. - Quality agreements and SLA clauses specifying KPIs and reporting cadence. - Records of mock inspections and corrective actions taken.

Governance, Roles and Regulatory Context

To make KPI programmes effective and defensible to inspectors, governance and role clarity are essential. Below is a summary of responsibilities and regulatory expectations.

Roles and responsibilities - QPPV: Overall responsibility for pharmacovigilance system performance; must be able to rely on KPI outputs to demonstrate oversight. Approves critical KPI definitions and receives escalations for significant trends or threshold breaches. - Sponsor PV Lead: Day-to-day owner of KPI framework, responsible for KPI dictionary maintenance, reporting and first-line analysis. - Sponsor QA: Ensures KPI programme is validated, auditable and aligned with company quality system; leads audits of vendor KPI implementation. - Vendor PV Lead: Operational owner for delivering KPI inputs, providing evidence, and executing remediation actions under the SLA. - Vendor QA/Operations: Ensure data integrity, application of SOPs, and prompt response to findings arising from KPI trends. - Regulatory Affairs: Supports interpretation of reporting timelines and evidence for submission-related KPIs. - Data/IT owner: Maintains extracts, dashboards and validates data pipeline and access controls.

Governance structure and escalation - Define tiered governance: operational (weekly), management (monthly) and executive (quarterly). - Map KPI thresholds to required actions and persons to be notified (e.g., automatic email to Sponsor PV Lead at amber, escalation to QPPV and Sponsor Executive at red). - Ensure governance meeting minutes clearly document decisions, assigned actions with due dates, and linkage back to KPI evidence.

Regulatory context and inspection relevance - EMA GVP Module I requires systems and quality systems to provide assurance that the PV system works consistently. KPI programmes provide demonstrable evidence of system performance. - GVP Module III (Inspections) expects inspectors to see performance monitoring, escalation and corrective actions. KPI evidence packs are commonly requested and scrutinised. - ICH Q9 (Quality Risk Management) supports a risk-based selection and prioritisation of KPIs; critical vendors and activities require more intensive monitoring. - Data integrity expectations (ALCOA+) apply equally to KPI data as to case data; inspectors will challenge missing audit trails, inconsistent extracts and manual manipulations without justification.

Governance aspects inspectors will evaluate - Clarity of KPI definitions and authority lines. - Evidence that KPI calculations are repeatable and reproducible. - Integrity of data sources and reconciliation between source systems and reports. - Clear escalation and remediation when thresholds are breached. - Evidence of review and oversight by sponsor/QPPV and QA. - Documentation of decisions arising from KPI trends (minutes, email trails, CAPA initiation).

Practical tips for maintaining inspection-ready KPI programmes

References

  1. EMA Good Pharmacovigilance Practices (GVP) Module I – Pharmacovigilance Systems and Their Quality Systems.
  2. EMA Good Pharmacovigilance Practices (GVP) Module III – Pharmacovigilance Inspections.
  3. EMA Good Pharmacovigilance Practices (GVP) Module II – Pharmacovigilance System Master File.
  4. Regulation (EC) No 726/2004.
  5. Directive 2001/83/EC.
  6. Commission Implementing Regulation (EU) No 520/2012.
  7. ICH Q9 Quality Risk Management.
  8. ICH E2E Pharmacovigilance Planning.

Related Articles

Critical Vendor Management in Pharmacovigilance

A comprehensive guide to critical vendor identification, oversight, governance, risk management and QPPV visibility.

Vendor Oversight for QPPVs

An advanced guide to vendor oversight from the QPPV perspective, including governance, risk management, escalation, inspections and organisational accountability.

Vendor Oversight in Pharmacovigilance

A comprehensive guide to vendor oversight, outsourcing governance, accountability, risk management and inspection readiness within modern pharmacovigilance systems.

Last reviewed: 2026-06-11