PSMF Metrics and KPIs

Introduction

Many organisations maintain a PSMF.

Far fewer actively measure whether it remains healthy.

This distinction is important.

A PSMF can exist and still:

• Contain outdated information
• Miss critical vendors
• Omit products
• Misrepresent governance structures
• Create inspection risk

The challenge is that document quality often deteriorates gradually.

Without objective indicators, organisations may not recognise problems until an audit or inspection occurs.

Metrics help solve this problem.

They transform PSMF governance from a subjective activity into a measurable one.

Why Metrics Matter

Historically, PSMF maintenance was often viewed as a documentation exercise.

Modern pharmacovigilance systems are too complex for that approach.

A mature organisation should be able to answer questions such as:

• Is the PSMF current?
• Are annexes accurate?
• Are inventories complete?
• Are updates occurring on time?
• Are governance reviews effective?

Metrics provide evidence-based answers.

The Purpose of PSMF Metrics

The objective of metrics is not reporting.

The objective is governance.

Effective metrics help organisations:

• Detect deterioration early
• Monitor compliance
• Prioritise resources
• Support inspections
• Support QPPV oversight

Metrics should therefore focus on risk rather than activity.

Characteristics of Useful Metrics

Strong metrics are:

• Objective
• Repeatable
• Actionable
• Risk-based
• Easy to understand

Weak metrics often measure activity rather than control.

For example:

Weak Metric

Number of PSMF pages.

Strong Metric

Percentage of annexes reviewed on schedule.

The second metric provides meaningful governance information.

PSMF Health Framework

A practical approach is to group metrics into several categories.

Completeness

Is information present?

Accuracy

Is information correct?

Timeliness

Is information current?

Governance

Are controls functioning?

Inspection Readiness

Could the organisation defend the document during an inspection?

Together these categories provide a balanced view of PSMF health.

Completeness Metrics

Completeness metrics assess whether required information exists.

Examples include:

Annex Completion Rate

Measures whether required annexes exist.

Vendor Inventory Coverage

Measures whether all PV vendors appear within the inventory.

Product Inventory Coverage

Measures whether all products are included.

Missing information often represents one of the most common inspection findings.

Accuracy Metrics

Accuracy is frequently more important than completeness.

An inaccurate inventory may be more problematic than an incomplete one.

Examples include:

Inventory Accuracy Rate

Assesses whether sampled entries are correct.

Vendor Accuracy Rate

Measures agreement between vendor inventories and contractual records.

Organisational Accuracy Rate

Assesses whether organisational charts reflect actual reporting structures.

Regular verification activities support these metrics.

Timeliness Metrics

Timeliness measures whether information remains current.

Examples include:

Overdue Updates

Number of updates exceeding defined timelines.

Average Update Time

Time between a business change and PSMF update.

Review Compliance

Percentage of reviews completed on schedule.

A document can be accurate today but inaccurate tomorrow.

Timeliness metrics help prevent gradual deterioration.

Product Inventory Metrics

Product inventories often represent one of the largest maintenance burdens.

Useful measures include:

Products Pending Update

Products awaiting inclusion or revision.

Inventory Review Compliance

Percentage of scheduled reviews completed.

Product Reconciliation Findings

Differences identified during verification activities.

These metrics are particularly important following acquisitions and divestments.

Vendor Oversight Metrics

Vendor inventories often change frequently.

Examples include:

Vendor Inventory Accuracy

Percentage of vendors accurately recorded.

Missing Vendor Rate

Vendors identified outside formal inventories.

Oversight Review Completion

Completion of planned vendor oversight reviews.

For additional information see:

[[vendor-oversight]]

Governance Metrics

Governance metrics assess whether oversight mechanisms function effectively.

Examples include:

Governance Review Completion

Scheduled reviews completed on time.

Escalation Compliance

Issues escalated according to procedures.

Action Item Closure

Governance actions completed within agreed timelines.

These indicators help determine whether oversight is active rather than theoretical.

QPPV Oversight Metrics

The PSMF should support effective QPPV oversight.

Examples include:

Significant Changes Reported

Percentage of significant changes communicated to the QPPV.

Governance Review Participation

Attendance and participation within governance meetings.

Risk Review Completion

Scheduled risk reviews completed on time.

These metrics provide visibility regarding oversight effectiveness.

For additional information see:

[[psmf-qppv-oversight]]

Change Control Metrics

Many PSMF deficiencies originate from poor change control.

Useful indicators include:

Change Assessment Completion

Percentage of significant changes assessed for PSMF impact.

Change-to-Update Time

Time between approved change and documented update.

Outstanding Change Actions

Open activities awaiting completion.

Strong performance here often correlates with strong inspection outcomes.

Audit Metrics

Audits help verify whether the PSMF reflects reality.

Examples include:

Audit Coverage

Percentage of planned audits completed.

PSMF Findings

Audit observations related to the PSMF.

Repeat Findings

Issues recurring after previous CAPAs.

Repeat findings frequently indicate ineffective governance.

CAPA Metrics

CAPA performance often predicts future inspection outcomes.

Useful indicators include:

Open CAPAs

Current active CAPAs.

Overdue CAPAs

Actions exceeding target dates.

CAPA Effectiveness Rate

Percentage of CAPAs verified as effective.

Repeat Deficiencies

Issues recurring after closure.

These metrics help assess whether improvements are sustainable.

Inspection Readiness Metrics

Many organisations claim to be inspection-ready.

Metrics provide evidence.

Examples include:

Annex Currency

Percentage of annexes updated within required timelines.

Inventory Reconciliation Success

Percentage of reconciliations completed without discrepancies.

Critical Deficiency Count

Number of unresolved high-risk issues.

Inspection Readiness Score

Composite indicator combining multiple measures.

Such indicators provide management visibility regarding inspection preparedness.

Dashboard Design

An effective dashboard should remain simple.

A practical PSMF dashboard may include:

The objective is visibility rather than volume.

KPI Definition Table — actionable and inspection-ready

The table below provides a standard set of KPIs with an actionable definition for each. For inspection purposes, each KPI definition must be documented in a KPI SOP or annex to the PSMF, with the data source, calculation logic and owner clearly recorded. Thresholds shown are example RAG (Green/Amber/Red) bands; adapt to organisational risk tolerance and regulatory expectations.

Notes on thresholds and sampling: - Sample sizes for accuracy metrics should be pre-defined in SOPs using risk-based sampling (e.g., stratified random sampling across product lines, geographies and vendors). - Composite Inspection Readiness Score must have a documented methodology (weights, normalization) and be reproducible from source data.

Practical implementation details

1. KPI governance and documentation
2. Each KPI must be defined in a KPI definition document (SOP or PSMF annex) containing: purpose, scope, calculation steps, acceptable sources, sample size method, owner, evidence retention requirements, and change control provisions.

3. Maintain a KPI register listing current KPI versions, last review date and approval history. Treat KPI definitions as controlled documents subject to change control.

4. Data sources and master data governance

5. Establish master sources of truth (e.g., regulatory product master, contract repository, vendor master, PSMF document control system). Document the primary and secondary sources for each KPI.
6. Validate electronic feeds or exports used for KPI calculation. Retain raw extracts used in KPI computation to support inspection queries.

7. Implement periodic reconciliation between systems (e.g., PV safety database, regulatory master, procurement) with documented reconciliation procedures.

8. Measurement and sampling

9. For accuracy metrics adopt risk-based sampling: define strata (high-risk products, critical vendors, recent changes), sample sizes, and acceptance criteria in the KPI SOP.

10. Use statistical techniques where appropriate (e.g., confidence intervals) to justify sample sizes and thresholds during inspections.

11. Automation and tooling

12. Where possible automate KPI extraction to reduce manual error. Use validated reporting tools or business intelligence systems (with change logs and versioning).

13. For automated KPIs maintain validation evidence showing report logic and data lineage. Include screenshots, SQL queries or report definitions in inspection evidence packs.

14. Evidence and audit trail

15. For each KPI cycle retain: data extracts, calculation worksheets, dashboard exports, meeting minutes where results were reviewed, and related CAPA/task records.

16. Evidence must be retained in accordance with the organisation’s document retention policy and be readily retrievable for inspections.

17. Escalation and thresholds

18. Define escalation pathways linked to threshold bands (e.g., amber triggers local remediation plan; red triggers executive escalation and immediate CAPA initiation).

19. Document expected remedial timelines and responsible parties for each threshold breach.

20. Reporting cadence and audience

21. Present KPIs at appropriate governance levels: operational KPIs to PSMF owners and product teams; aggregated KPIs to the QPPV and Head of PV; readiness KPIs to executive management.

22. Include trend charts (3–12 months) to demonstrate trajectory; inspectors often look for trends rather than single point values.

23. Integration with CAPA and change control

24. Link KPI failures to the CAPA system. For persistent issues, document root cause analysis and effectiveness checks with date-stamped evidence.
25. Include KPI impacts in change assessments: changes to organisational structure, vendors, or systems must prompt KPI review and re-baselining if required.

Regulatory context and inspection relevance

• EMA GVP Module II (PSMF) and Module I (quality systems) require that pharmacovigilance systems be described and maintained; regulators expect the PSMF to be accurate, complete and up to date. Well-defined KPIs provide objective evidence that this requirement is being met.
• GVP Module III and inspection guidance emphasise the need for systems and controls; inspectors will look for documented controls, evidence of monitoring, remediation of deficiencies and QPPV oversight. KPI records, SOPs and evidence packs are routinely requested during inspections and remote assessments.
• Under Directive 2001/83/EC and Regulation (EC) No 726/2004 the QPPV has responsibilities for ensuring the PV system functions. KPI outputs should therefore be visible to and discussed with the QPPV; KPI governance must document QPPV access and review expectations.
• Inspectors frequently focus on:
• Currency of annexes, especially MAH/portfolio details and QPPV contact information.
• Reconciliations between PV system data and PSMF.
• Evidence that governance reviews occur and that action items are closed.
• CAPA effectiveness where KPI failures have been identified.
• Practical inspection readiness: maintain a compact evidence binder (electronic) that links each KPI to its source data, calculation, recent trend, associated meeting minutes and any CAPA actions. This accelerates responses during opening meetings and document requests.

Governance discussion

• Roles and responsibilities:
• PSMF Owner: day-to-day KPI steward, maintains KPI evidence and executes updates.
• QPPV: oversight recipient and decision-maker for high-severity findings or thresholds breaches that affect patient safety or regulatory obligations.
• Head of PV / Governance Lead: ensures KPIs are included in PV governance fora and that thresholds prompt adequate escalation.
• Quality: owns CAPA effectiveness verification and KPI SOP controls.

• IT/Data Stewards: maintain report validation and data lineage for automated KPIs.

• KPI lifecycle management:

• Review KPI set and thresholds annually (or sooner after organisational change). Document rationale for thresholds and any re-baselining.
• Maintain version control and a change history for KPI definitions to demonstrate traceability during inspections.

• Embed KPI review into periodic management reviews and QPPV reporting; include action plans, resource requests and trend analysis.

• Independence and objectivity:

• Where possible separate KPI calculation and governance review from operational teams subject to those KPIs. Quality oversight of KPI methodology reduces conflict of interest.
• Use independent audit to verify KPI methodology and evidence periodically.

Practical examples of inspection evidence package

For each KPI inspected, prepare: - KPI definition document (SOP or annex) showing purpose, formula, data sources, sampling method and owner. - Raw data extract used to compute the KPI for the period under review. - Calculation worksheet or validated report showing the computation steps. - Trend graph for the prior 6–12 months. - Minutes from governance meeting where KPI was reviewed and action items agreed. - CAPA or task records if KPI fell into amber/red and the subsequent remediation and effectiveness check.

Common metric mistakes (summary)

• Measuring activity rather than control.
• Excessive complexity without documented rationale.
• Lack of thresholds and escalation criteria.
• Poor sampling methodology for accuracy metrics.
• Missing evidence or broken data lineage for automated KPIs.

What Great Organisations Measure

High-performing organisations focus on: - Accuracy - Timeliness - Governance effectiveness - Inspection readiness

They use metrics to drive action rather than simply generate reports.

Importantly, they view the PSMF as a living governance system rather than a static document. They document KPI definitions, retain evidence, validate data sources and demonstrate linkage between KPI failures and effective CAPA.

Key Takeaways

• Metrics help detect deterioration before inspections identify deficiencies.
• Effective metrics focus on risk rather than activity.
• Completeness, accuracy and timeliness are foundational measurement areas.
• Vendor inventories and product inventories require ongoing monitoring.
• Governance metrics help determine whether oversight is functioning effectively.
• CAPA and audit metrics provide insight into continuous improvement.
• Mature organisations use dashboards and documented KPI definitions to support QPPV oversight and inspection readiness.

New: Sample KPI SOP (Template)

This section provides a complete sample KPI SOP that can be adopted as a controlled document or PSMF annex. It is intentionally prescriptive to support inspection readiness: the SOP contains defined sections inspectors commonly review, including data source provenance, calculation logic, sampling methodology, evidence retention and escalation.

Title: SOP-PV-PSMF-KPI-001 — PSMF KPI Management and Evidence Generation

Version: 1.0 Effective date: [YYYY-MM-DD] Owner: PSMF Owner Approver: Head of PV, Quality, QPPV (as applicable)

1. Purpose

2. Define responsibilities, procedures and evidence requirements for the definition, calculation, reporting and retention of KPIs used to monitor the Pharmacovigilance System Master File (PSMF).

3. Scope

4. Applies to all KPIs recorded in the PSMF KPI register including completeness, accuracy, timeliness, governance and inspection readiness KPIs. Includes manual and automated KPI processes, evidence retention and controls for supporting calculations.

5. Definitions

6. KPI: Key Performance Indicator.
7. Evidence pack: set of documents required to reproduce a KPI value (raw extract, calculation worksheet, report logic, timestamped meeting minutes).
8. RAG: Red/Amber/Green thresholds.
9. Data owner: owner of the master data source.

10. PSMF Owner: responsible for the PSMF and KPI steward.

11. Roles and responsibilities

12. PSMF Owner: maintain KPI register, execute KPI calculations, assemble evidence packs, present KPI results to governance.
13. Data Owners (Contracts, Regulatory, Vendor Master, IT): provide and attest to source data.
14. Quality: review SOP compliance, verify CAPA effectiveness for KPI failures, perform independent checks.
15. QPPV: review inspection readiness KPIs and confirm acceptability.

16. IT/Data Stewards: validate automated reports, maintain report version control, and provide audit trails.

17. KPI lifecycle and governance

18. New KPI proposals submitted to KPI Governance Forum using a standard template (purpose, calculation, data source, owner, frequency, thresholds).
19. KPI definitions are version controlled and approved by Head of PV and Quality.
20. Review frequency: KPI definitions reviewed annually or after a major change (e.g., acquisition, system change).

21. Retirement: KPIs may be retired with documented rationale and archived evidence.

22. KPI definition template (required fields)

23. KPI name, unique identifier
24. Purpose and regulatory rationale (link to GVP clause if applicable)
25. Calculation: explicit formula with numerator/denominator and allowed exceptions
26. Data sources: system name, table/field, owner, extraction method, time zone
27. Frequency and reporting cadence
28. Sample strategy (if applicable): strata, sample size calculation, selection method, random seed
29. Thresholds: R/A/G with escalation actions linked to each band
30. Evidence requirements: list of files, retention period, file names conventions
31. Responsible roles for calculation and approval

32. Change control and version history

33. Data extraction and validation

34. Manual extracts: include export timestamp, user ID, export file name, and MD5 checksum.
35. Automated reports: retain report definition, SQL/ETL script, validation test cases and change log.

36. Validation: each KPI extract requires a signed data attestation from the data owner confirming single source of truth and extract accuracy.

37. Sampling methodology (accuracy KPIs)

38. Use risk-based stratified random sampling. Define strata (e.g., high-risk products, high-value vendors, recently changed records).
39. Minimum sample size: calculated using binomial confidence interval approach or risk-based pragmatic approach; document calculation in the worksheet.

40. Sampling reproducibility: preserve random seed and selection date to enable repopulation.

41. Calculation and worksheet requirements

42. Maintain calculation worksheets (spreadsheet or BI export) that show raw extract rows, selection method, inclusion/exclusion logic, intermediate counts and final KPI value.

43. Worksheets must be unambiguous and time-stamped; changes must be tracked using spreadsheet change history or document control.

44. Evidence retention and naming conventions

45. Evidence must be retained for a period consistent with PV document retention policy (e.g., 5 years or regulatory-specified).
46. File naming convention (example): KPI__.xlsx

47. Store evidence in controlled document repository with access controls and audit trail.

48. Escalation and actions

49. Define escalation triggers for each threshold and required documentation for each escalation (escalation memo, CAPA initiation, timelines for remediation).

50. For red band breaches, the QPPV and Head of PV must be notified within 24 hours with a completed impact assessment and immediate mitigation plan.

51. Inspection readiness and evidence pack assembly

52. On inspection request, provide the following per KPI for the requested period:
53. KPI definition (approved version)
54. Raw extract(s) with export metadata
55. Calculation worksheet (step-by-step)
56. Report definition or SQL query (for automated KPIs)
57. Trend chart and latest governance minutes
58. Any CAPA or remediation evidence

59. Maintain a pre-assembled inspection evidence binder updated monthly for the top 10 inspection-critical KPIs.

60. Change control for KPI definitions

61. All KPI definition changes occur via the organisation’s change control process and require a documented rationale, impact assessment, and approval.

62. Independent verification and audit

63. Quality performs annual verification of a subset of KPIs, including re-execution of calculations using source extracts and review of evidence packs.

64. Audit will review compliance with this SOP during scheduled audits or when inspection findings indicate concerns.

65. Appendices (mandatory)

66. Appendix A: KPI definition template (fillable)
67. Appendix B: Sampling calculation examples
68. Appendix C: Evidence pack checklist
69. Appendix D: Example calculation worksheet template
70. Appendix E: Escalation flowchart with contact details

New: Worked numerical example (raw data extract and calculation worksheet)

The following worked example demonstrates the end-to-end evidence generation for a common accuracy KPI: Vendor Inventory Accuracy. It includes a raw data extract, documented sample selection, a calculation worksheet with intermediate steps, and the final KPI value. This example also shows how to include the evidence in an inspection-ready pack.

KPI: Vendor Inventory Accuracy (KPI_ID: KPI-VEND-ACC-Q1) Purpose: Validate that vendors contracted to perform PV activities are accurately recorded in the PSMF vendor inventory. Formula: (Number of sampled vendors with accurate PSMF entries / Number sampled) × 100% Data sources: - Contract Repository (source of truth for contracted vendors) — System: ContractsDB - PSMF Vendor Inventory — System: PSMF_DocStore Period: 1 April 2026 to 30 April 2026 Sample strategy: Risk-based stratified sample with oversampling of 'critical' vendors. Seeded random sampling for reproducibility.

1) Raw data extract (combined view)

This combined extract is an export performed on 2026-05-01 at 09:12 UTC. File name: KPI_VEND_ACC_202604_ContractsDB_PSMFDS_ks.xlsx Export metadata: - Exported by: JD (Vendor Oversight Lead) - Export timestamp: 2026-05-01T09:12:03Z - MD5 checksum: a1b2c3d4e5f67890123456789abcdef0

Sample of the raw extract (first 20 rows shown):

(Full extract contains 120 contracted PV vendors; the table above is the sample portion.)

Interpretation of raw fields: - In_PSMF: indicates whether the vendor is listed in the PSMF vendor index. - VendorName_PSMF: the vendor name as recorded in the PSMF (may differ in minor spelling).

2) Sample selection (reproducible, risk-based)

Rules in SOP: - Strata: High criticality, Medium criticality, Low criticality. - Sampling fractions: High = 50% of high-criticality vendors (rounded up), Medium = 15% of medium, Low = 10% of low. - Random seed: 2026-05-01 (documented in worksheet) Counts from the extract: - High criticality vendors = 20 - Medium criticality vendors = 50 - Low criticality vendors = 50

Sample sizes: - High: ceil(20 × 0.50) = 10 - Medium: round(50 × 0.15) = 8 - Low: round(50 × 0.10) = 5 Total sample size = 10 + 8 + 5 = 23 vendors

Documented random selection: random generator with seed 2026-05-01 generated the following VendorIDs: V001, V004, V006, V009, V012, V016, V018, V021, V025, V030* (high; if vendor IDs beyond 20 appear, adjust per full list). For transparency, all selected VendorIDs and selection script are retained in Appendix B of the worksheet.

(For this example, we will use a reduced reproducible sample of 15 to keep the worked example compact; the SOP-compliant sample was 23. The worksheet documents the deviation, rationale, and approval by Quality to use a focused 15-sample check for a rapid cycle verification. All deviations are signed and stored.)

Final sample used (15 vendors), selected with seed 2026-05-01: V001, V002, V003, V004, V008, V009, V010, V011, V012, V013, V014, V016, V018, V019, V020

The sample selection worksheet contains: - full script (Python/R/Excel), seed, date/time, and list of indices selected - approval memo for sample size deviation signed by Quality (if applicable)

3) Accuracy verification rules

For each sampled vendor, assess the following PSMF entry fields against the Contract Repository: - Vendor presence in PSMF (In_PSMF = Y) - Vendor name match (allow minor spelling differences; define acceptance as exact match after normalized whitespace and case-insensitive comparison; other deviations require documented justification) - Contract number recorded in PSMF (where applicable) - PV service recorded in PSMF matches contracted PV service

A vendor is scored as "Accurate" only if all assessed fields meet acceptance criteria; otherwise scored "Not Accurate". All assessed fields and evidence references are recorded in the worksheet.

4) Calculation worksheet (step-by-step)

Calculation worksheet file: KPI_VEND_ACC_202604_Worksheet_ks.xlsx Worksheet tabs: - Tab 1: RawExtract (copy of combined extract rows for sampled VendorIDs) - Tab 2: VerificationChecklist (one row per sampled vendor with field-by-field verification and evidence links) - Tab 3: IntermediateCounts - Tab 4: FinalCalculation - Tab 5: EvidenceIndex (links to contracts pdf, PSMF screenshots, extraction metadata)

Tab: VerificationChecklist (excerpt)

Notes: - For vendors with In_PSMF = N, the NameMatch and other checks are not applicable; they are counted as Not Accurate unless an immediate remediation explanation exists and is accepted by Quality. - EvidenceLink files are stored in a secure evidence repository with access control and retention metadata.

Tab: IntermediateCounts

Tab: FinalCalculation

• Numerator = Number of sampled vendors with accurate PSMF entries = 11
• Denominator = Number sampled = 15
• Vendor Inventory Accuracy = 11 / 15 × 100% = 73.33%

RAG assessment (per KPI definition table): - G ≥ 95%; A 90–94.9%; R < 90% - Result: 73.33% → Red

5) Evidence pack assembly for inspection

Files included in the KPI evidence pack (folder: /InspectionEvidence/KPI-VEND-ACC-202604): - 1_KPI_Definition_KPI-VEND-ACC-Q1_v1.0.pdf - 2_RawExtract_ContractsDB_PSMF_20260501.csv (with export metadata and MD5) - 3_Worksheet_KPI_VEND_ACC_202604_Worksheet_ks.xlsx (calculation steps, selection script) - 4_Script_RandomSelection_seed20260501.py (or .R) and output list - 5_ReportDefinition_VendorInventoryAccuracy.sql (for automated or reproducible calculation) - 6_PSMF_screenshots_VendorEntries.zip (screen grabs for each sampled vendor) - 7_Contract_pdfs.zip (contracts for sampled vendors) - 8_Governance_Minutes_20260505_KPIReview.pdf (minutes where results were reviewed) - 9_CAPA_Initiation_Form_20260506.pdf (CAPA opened for remediation) - 10_QA_Approval_SampleDeviation_20260502.pdf (if sample deviation occurred) - 11_ExportChecksumManifest.txt

Each file uses the naming convention defined in the SOP and includes a timestamp, preparer initials and version.

6) Remediation and CAPA trace

Because the KPI result was Red (73.33%), the following actions were initiated and recorded in the evidence pack: - On 2026-05-05 governance meeting: immediate remediation plan approved (documented in Gov Minutes). - CAPA opened (CAPA-ID: CAPA-PSMF-2026-004) to update PSMF with missing vendors V003, V009, V014 and to reconcile vendor naming variations; CAPA owner assigned with target completion 2026-06-15. - Interim mitigation: manual PSMF addendum created and QPPV notified on 2026-05-06.

All CAPA actions, closure evidence and effectiveness verification are linked to KPI evidence pack and will be presented to Quality for closure verification.

New: Composite Inspection Readiness Score — worked example

A composite Inspection Readiness Score can be useful to provide a single management view. The composite must be fully documented (weights and normalization) in the KPI SOP and evidence pack.

Example KPI set and weights: - Annex Currency (K1) — weight 30% - Vendor Inventory Accuracy (K2) — weight 20% - Product Inventory Coverage (K3) — weight 20% - Overdue Updates (K4) — weight 15% (inverse: lower is better) - CAPA Effectiveness Rate (K5) — weight 15%

Normalization rules: - Each KPI scaled to 0–100% on desired polarity (for K4 Overdue Updates, score = max(0, 100 − (OverdueCount × penalty))); define penalty logic in SOP). - Weighted sum = Σ (normalized KPI_i × weight_i)

Observed KPI values for the current month (example): - K1 Annex Currency = 92% (within amber band) - K2 Vendor Inventory Accuracy = 73.33% (as calculated above) - K3 Product Inventory Coverage = 99.2% - K4 Overdue Updates = 2 items (threshold: G=0 ; A ≤ 3 ; R > 3). Using a simple penalty of 20 points per overdue item up to a cap of 100: score_K4 = max(0, 100 − (2 × 20)) = 60 - K5 CAPA Effectiveness Rate = 88%

Normalize (already 0–100):

Compute weighted score: - K1 contribution = 92 × 0.30 = 27.6 - K2 contribution = 73.33 × 0.20 = 14.666 - K3 contribution = 99.2 × 0.20 = 19.84 - K4 contribution = 60 × 0.15 = 9.0 - K5 contribution = 88 × 0.15 = 13.2

Inspection Readiness Score = 27.6 + 14.666 + 19.84 + 9.0 + 13.2 = 84.306 ≈ 84.3%

RAG per composite thresholds (as defined in KPI table): - G ≥ 90%; A 75–89%; R < 75% - Result: 84.3% → Amber

Evidence to support composite: - Link each KPI evidence pack (as described for Vendor KPI) to the composite calculation worksheet showing calculations, weights, and links to underlying raw extracts and governance minutes. - Preserve versioned composite calculation (filename: InspectionReadinessComposite_202604_Worksheet_v1.0.xlsx) and approval sign-off by Head of PV.

Inspection relevance: - Inspectors will request: KPI definitions, raw extracts for each KPI, calculation worksheets, trend history, and governance minutes. The composite score alone is insufficient; the inspector will verify source data and calculation reproducibility. Providing the underlying evidence pack for each KPI expedites inspection review.

Evidence retention, audit trails and demonstration of reproducibility

To be inspection-ready, follow these minimum practices for each KPI cycle: - Store raw extracts with metadata (who exported, when, system, checksum). - Store calculation worksheets showing intermediate steps, not just final dashboard numbers. - Store automated report definitions (SQL, ETL, BI report) with version history and validation notes. - Keep governance meeting minutes where KPI results were reviewed and actions assigned. - Record signatures/approvals (electronic or scanned) for KPI definition changes or sample deviations. - Ensure Quality independently re-performs at least one KPI calculation per year and records verification results.

During an inspection, readiness is demonstrated by: - Providing the KPI SOP and KPI definition(s). - Presenting the raw extracts with export metadata. - Walking through the calculation worksheet and reproducing the KPI in real time (or within an agreed timeframe) using preserved scripts or SQL. - Showing governance minutes and CAPA records following any amber/red findings. - Demonstrating traceability from dashboard number → worksheet → extract → source system.

Practical tips for building the evidence pack (operational checklist)

• Pre-build an evidence binder for top N KPIs (e.g., top 10 inspection-critical) and update monthly.
• Use standardized file names and a single repository path per KPI per period.
• Include a one-page KPI summary (definition, current value, trend arrow, last reviewed date, owner) as the cover page for each evidence pack.
• Use hashed checksums and a manifest file for each period to protect integrity; record checksums and storage location in the KPI register.
• Ensure access permissions allow inspectors to view files in read-only mode or provide certified copies if required by regulatory authority.

Closing governance notes

• The sample KPI SOP and worked example above illustrate the level of detail regulators expect: documented methods, reproducible calculations, and complete evidence.
• Governance must not view these artefacts as optional. During an inspection the absence of raw extracts or calculation worksheets will be interpreted as a lack of control and will likely become a finding.
• Assigning clear roles (data owner, PSMF Owner, Quality, QPPV) and documenting attestation steps reduces inspection risk and demonstrates proactive oversight.

References

1. EMA Good Pharmacovigilance Practices (GVP) Module I – Pharmacovigilance Systems and Their Quality Systems.
2. EMA Good Pharmacovigilance Practices (GVP) Module II – Pharmacovigilance System Master File.
3. EMA Good Pharmacovigilance Practices (GVP) Module III – Pharmacovigilance Inspections.
4. Regulation (EC) No 726/2004.
5. Directive 2001/83/EC.
6. Commission Implementing Regulation (EU) No 520/2012.
7. EMA Questions and Answers on Pharmacovigilance System Master Files.