Proton Pump Inhibitors and Hypomagnesaemia: A Signal Evaluation
- Proton Pump Inhibitors and Hypomagnesaemia: A Signal Evaluation
- Signal Evaluation
- Pharmacological Plausibility
- The Role of Alternative Causes
- A Particularly Useful Clinical Clue: Renal Magnesium Conservation
- Severe Clinical Consequences
- What Strengthens the Causal Association?
- What Would We Expect If PPIs Were Not Causal?
- What Remains Uncertain?
- Signal Prioritisation
- What Would a Pharmacovigilance Team Review?
- Historical Signal Evolution
- A Two-Level Conclusion
- What This Example Teaches About Signal Evaluation
- 1. Dechallenge can be powerful
- 2. Rechallenge can be exceptionally informative
- 3. Do not deliberately create rechallenge
- 4. Long latency does not exclude causality
- 5. Laboratory physiology matters
- 6. Class effects should be tested
- 7. Epidemiology and individual cases can tell different stories
- 8. Mechanism and causality should be separated
- 9. Severity matters
- 10. The strongest conclusion is often layered
- Key Takeaways
- References
Introduction
Some pharmacovigilance signals are difficult because the epidemiological evidence is inconsistent.
Others are difficult because the event is rare, the mechanism is incompletely understood, and the strongest evidence comes from individual patients rather than large comparative studies.
Proton pump inhibitor (PPI)-associated hypomagnesaemia is a useful example of the second type.
The association illustrates several important principles of signal evaluation:
- rare adverse reactions may first become apparent through individual case reports;
- a strong dechallenge and rechallenge pattern can provide important causal evidence;
- recurrence after exposure to different drugs within the same class can support a class effect;
- a long latency does not exclude causality;
- mechanistic plausibility can evolve after a signal has been identified;
- epidemiological studies may show substantial heterogeneity;
- the absence of a completely established molecular mechanism does not necessarily mean that the adverse reaction is unproven;
- a signal can become sufficiently convincing to influence product information and clinical practice even when some aspects of its mechanism and incidence remain uncertain.
This article evaluates the PPI–hypomagnesaemia association as a pharmacovigilance case study.
The objective is not to reproduce a regulatory assessment for any individual PPI. It is to demonstrate how a safety professional can move from case reports to a broader causal assessment and distinguish what is strongly supported from what remains uncertain.
The Signal
The signal can be expressed simply:
Long-term exposure to proton pump inhibitors may cause hypomagnesaemia, sometimes severe enough to produce hypocalcaemia, hypokalaemia, neurological manifestations or cardiac arrhythmias.
The signal is unusual because the initial clinical pattern can be difficult to recognise.
Patients may have been taking a PPI for years.
They may have no obvious symptoms until magnesium depletion becomes severe.
The clinical presentation may then be attributed to another condition, particularly when several electrolyte abnormalities occur simultaneously.
Reported manifestations include:
- muscle weakness;
- tremor;
- paraesthesia;
- tetany;
- seizures;
- hypocalcaemia;
- hypokalaemia;
- QT prolongation;
- ventricular arrhythmias;
- and other consequences of severe magnesium depletion.
Several published case series describe severe biochemical abnormalities and clinically important complications in long-term PPI users. 0
The important pharmacovigilance question is therefore not merely whether low magnesium occurs in PPI users.
The question is:
Does PPI exposure cause clinically important hypomagnesaemia in at least some patients, and how strong is the evidence for that causal association?
Why the Signal Is Interesting
At first sight, hypomagnesaemia has many possible causes.
These include:
- gastrointestinal losses;
- renal magnesium wasting;
- diuretic therapy;
- alcohol use;
- malnutrition;
- endocrine disorders;
- inherited disorders of magnesium handling;
- gastrointestinal disease;
- and other medicines.
That creates a substantial background rate of potential confounding.
A pharmacovigilance assessment therefore needs evidence that distinguishes PPI-associated hypomagnesaemia from ordinary hypomagnesaemia occurring in patients who happen to take PPIs.
This is where the individual cases become particularly informative.
Signal Evaluation
1. The Initial Case Pattern
The earliest reports were notable not simply because patients had low magnesium.
They were notable because the abnormalities behaved as though they were related to PPI exposure.
A particularly informative early report described two long-term PPI users with severe magnesium depletion. Renal magnesium handling demonstrated avid renal conservation, supporting impaired gastrointestinal absorption rather than excessive renal loss. Magnesium abnormalities resolved after withdrawal of the PPI. 1
This is a valuable causal clue.
If the kidneys are retaining magnesium appropriately while serum magnesium remains profoundly low, renal wasting becomes a less satisfactory explanation.
If the abnormality subsequently resolves after withdrawal of the PPI, the medication becomes a more credible explanation.
This is an example of how clinical physiology can strengthen a pharmacovigilance case.
2. Dechallenge
Dechallenge means observing what happens after the suspected medicine is withdrawn.
In PPI-associated hypomagnesaemia, dechallenge has repeatedly been informative.
A clinical case series involving eight patients with severe hypomagnesaemia reported prompt resolution after stopping PPI therapy in carefully monitored patients. Magnesium replacement alone was relatively ineffective while PPI therapy continued. 2
This pattern is important.
If supplementation is unable to correct the abnormality adequately while exposure continues, but magnesium normalises after withdrawal, the temporal relationship becomes considerably more persuasive.
It is not perfect proof.
A disease process could theoretically improve coincidentally after treatment changes.
But repeated, physiologically coherent dechallenge across independent cases is stronger evidence than a single spontaneous report.
3. Rechallenge
Rechallenge occurs when the suspected adverse reaction returns after the medicine is reintroduced.
This is one of the most informative observations available in individual-case causality assessment.
In PPI-associated hypomagnesaemia, rechallenge has been reported repeatedly.
A systematic review of published cases found that discontinuation of PPIs resulted in rapid recovery, with a median recovery time of approximately four days in the cases where this was documented. Rechallenge was followed by recurrence within approximately four days. 3
Other individual reports describe recurrence after re-exposure, including recurrence after a different PPI. 4
This creates a particularly informative sequence:
Exposure → hypomagnesaemia → withdrawal → recovery → re-exposure → recurrence
That pattern is difficult to dismiss as random coincidence when it occurs repeatedly.
Why Rechallenge Is Powerful
Suppose a patient develops hypomagnesaemia after several years of PPI therapy.
That observation alone is weak.
There are many alternative explanations.
Now suppose the PPI is stopped and magnesium normalises.
The causal argument becomes stronger.
Now suppose the PPI is restarted because symptoms recur and hypomagnesaemia returns rapidly.
The causal argument becomes substantially stronger again.
This is why pharmacovigilance should not treat all case reports as equivalent.
A case containing a well-documented positive rechallenge has a different evidentiary value from a case in which a patient simply happens to have hypomagnesaemia while taking a PPI.
An Important Ethical Point
Rechallenge is informative but should not be deliberately performed merely to strengthen a pharmacovigilance case when recurrence could expose the patient to serious harm.
Many of the strongest rechallenge observations in the PPI literature arose because the clinical need for acid suppression led to re-exposure.
This distinction matters.
Pharmacovigilance should use clinically occurring evidence.
It should not create unnecessary risk merely to generate evidence.
4. Is This a Class Effect?
A major question is whether the reaction is restricted to one PPI or represents a broader class effect.
Published cases involve multiple PPIs, including:
- omeprazole;
- esomeprazole;
- pantoprazole;
- rabeprazole;
- and other agents.
The systematic review of published cases concluded that PPI-induced hypomagnesaemia appeared to be a drug-class effect. 5
A case series similarly reported severe hypomagnesaemia in patients exposed to several different PPIs. 6
This pattern is useful because it reduces the likelihood that the association is caused by an idiosyncratic impurity, formulation-specific problem or isolated manufacturing issue.
However, class effect should be interpreted carefully.
Evidence that several drugs in a class can cause an adverse reaction does not prove that:
- every patient has the same susceptibility;
- every PPI carries identical risk;
- potency is irrelevant;
- duration of exposure is irrelevant;
- or the mechanism is completely understood.
It supports a class-level pharmacovigilance hypothesis.
5. Latency: Why Did It Take Years?
One of the more interesting characteristics of PPI-associated hypomagnesaemia is latency.
The systematic review of published cases reported a median duration of PPI exposure of approximately 5.5 years before hypomagnesaemia was identified, with a very broad range from approximately two weeks to 13 years. 7
This creates an important teaching point.
A long latency can initially appear inconsistent with causality.
But if the mechanism involves gradual depletion of total body magnesium stores, long exposure may actually be biologically coherent.
Two patients in an early report had evidence of severe magnesium depletion and renal conservation, consistent with a failure of intestinal magnesium absorption. 8
Therefore:
Long latency should trigger mechanistic investigation rather than automatic rejection of causality.
The temporal relationship must be interpreted according to the proposed biological process.
6. What Does the Epidemiology Show?
The epidemiological evidence is less straightforward than the individual-case evidence.
A 2015 meta-analysis of nine observational studies involving approximately 110,000 patients reported an increased risk of hypomagnesaemia among PPI users, with a pooled relative risk of 1.43 (95% CI 1.08–1.88). A sensitivity analysis restricted to higher-quality studies produced a pooled RR of 1.63 (95% CI 1.14–2.23). 9
These results support an association.
But observational studies have important limitations.
Patients prescribed PPIs often differ systematically from people who are not prescribed PPIs.
They may:
- be older;
- have more comorbidities;
- use diuretics;
- have renal disease;
- have gastrointestinal disease;
- be hospitalised more frequently;
- use other medicines associated with electrolyte disturbances.
These factors can produce confounding.
A later systematic review and meta-analysis illustrates the uncertainty.
The 2019 review included 15 studies and approximately 129,000 participants. The pooled analysis suggested increased risk, but heterogeneity was very high, and subgroup analyses did not consistently demonstrate an association. The authors concluded that the association remained unclear because of substantial heterogeneity. 10
This creates an important contrast.
Individual cases
Strong dechallenge and rechallenge evidence.
Observational epidemiology
Evidence of association, but heterogeneous.
That is not contradictory.
The two evidence streams answer different questions.
7. Why the Case Reports May Be More Informative Than the Crude Epidemiology
For common adverse events, spontaneous cases may be relatively weak evidence because the background rate is high.
Hypomagnesaemia is different.
The most informative cases can contain:
- severe biochemical abnormalities;
- a plausible alternative-diagnosis assessment;
- renal magnesium conservation;
- failure of supplementation while the PPI continues;
- recovery after withdrawal;
- recurrence after rechallenge;
- recurrence with another PPI.
A carefully documented sequence like this contains multiple causal clues.
A large observational database may contain hundreds of cases but lack detailed information on:
- magnesium intake;
- renal magnesium excretion;
- supplementation;
- precise timing of recovery;
- rechallenge;
- alternative causes.
Therefore, larger sample size does not automatically mean stronger causal evidence.
Evidence quality depends on the question being asked.
Pharmacological Plausibility
What Is Established?
PPIs suppress gastric acid secretion by inhibiting the gastric H+/K+-ATPase.
Magnesium absorption occurs through both passive and active intestinal processes.
Evidence from clinical and experimental studies suggests that PPI exposure can interfere with intestinal magnesium absorption.
The exact molecular pathway, however, has not always been fully established.
This distinction is important.
What Is Plausible?
One proposed mechanism involves altered intestinal magnesium transport, including effects on the transient receptor potential melastatin channels TRPM6 and TRPM7.
Experimental and clinical literature has provided evidence consistent with altered magnesium transport during PPI exposure. 11
Another proposed explanation involves changes in intestinal magnesium solubility or absorption associated with altered luminal conditions.
These mechanisms are biologically plausible.
But the precise contribution of each pathway remains uncertain.
What Is Not Fully Established?
It is not necessary to claim that one molecular pathway has been conclusively demonstrated in every patient with PPI-associated hypomagnesaemia.
The pharmacovigilance conclusion can be stronger than the mechanistic conclusion.
This is a general lesson:
A drug reaction can be causally well supported even when the complete molecular mechanism remains incompletely characterised.
Causality is assessed from the totality of evidence.
Mechanism is one component of that evidence.
The Role of Alternative Causes
A good case evaluation must actively investigate other explanations.
Potential alternative causes include:
Diuretics
Loop and thiazide diuretics can affect magnesium balance and are frequent concomitant medicines in patients taking PPIs.
Renal disease
Renal handling of electrolytes can be abnormal in complex medical populations.
Gastrointestinal losses
Diarrhoea and malabsorption can independently reduce magnesium.
Poor nutritional intake
Reduced intake can contribute to magnesium depletion.
Alcohol
Chronic alcohol use can produce magnesium deficiency through multiple mechanisms.
Other medicines
Several medicines can influence magnesium balance.
The more convincingly these alternatives are excluded, the stronger the individual case becomes.
A Particularly Useful Clinical Clue: Renal Magnesium Conservation
One of the strongest mechanistic clues in the early reports was the renal response.
When serum magnesium falls, the kidney normally attempts to conserve magnesium.
If urinary magnesium excretion is appropriately low, renal wasting is less likely to explain the deficiency.
In reported PPI cases, marked renal conservation supported an intestinal rather than renal mechanism. 12
This is an excellent example of how clinical laboratory data can convert a vague adverse-event report into a mechanistically informative case.
The difference is substantial.
Compare:
"Patient taking PPI developed low magnesium."
with:
"Patient taking PPI developed profound hypomagnesaemia; urinary magnesium excretion demonstrated appropriate renal conservation; supplementation was insufficient while treatment continued; magnesium normalised after withdrawal and recurred after re-exposure."
The second case provides much stronger causal information.
Severe Clinical Consequences
Hypomagnesaemia is not merely a laboratory abnormality.
Severe magnesium depletion can produce secondary abnormalities, including:
- hypocalcaemia;
- hypokalaemia;
- neuromuscular manifestations;
- QT prolongation;
- and potentially serious cardiac arrhythmias.
Case series have documented severe biochemical abnormalities and cardiac or neurological consequences in patients with PPI-associated hypomagnesaemia. 13
This matters during signal prioritisation.
A rare event can still represent an important safety concern when:
- the outcome is potentially serious;
- the exposure is widespread;
- the event may be preventable;
- the diagnosis can be missed;
- and withdrawal of the suspected medicine can reverse the abnormality.
What Strengthens the Causal Association?
The PPI–hypomagnesaemia signal contains several classical causal clues.
| Evidence element | Observation | Effect on causal assessment |
|---|---|---|
| Temporality | Hypomagnesaemia develops during PPI exposure | Supports |
| Dechallenge | Magnesium improves after withdrawal | Strongly supports |
| Rechallenge | Hypomagnesaemia recurs after re-exposure | Strongly supports |
| Class consistency | Multiple PPIs implicated | Supports class effect |
| Biological plausibility | Altered intestinal magnesium absorption is plausible | Supports |
| Alternative causes | Some cases provide evidence against renal wasting | Supports |
| Epidemiology | Observational association reported | Supports |
| Epidemiological heterogeneity | Results vary substantially between studies | Limits precision |
| Long latency | Often months or years | Requires mechanistic interpretation |
| Clinical severity | Severe symptomatic cases reported | Increases clinical importance |
No single row proves causality.
The strength comes from the convergence of independent evidence.
What Would We Expect If PPIs Were Not Causal?
This counterfactual question is useful in every signal evaluation.
If PPIs were not causally related to hypomagnesaemia, we would expect:
- no consistent improvement after withdrawal;
- no reproducible recurrence after rechallenge;
- no recurrence across different PPIs;
- alternative explanations to account for the cases;
- and no coherent physiological pattern.
The published literature does not fit that expectation particularly well.
Repeated dechallenge and rechallenge observations are difficult to explain solely through background hypomagnesaemia.
That is why the causal evidence from individual cases is stronger than the heterogeneous epidemiological association might initially suggest.
What Remains Uncertain?
A rigorous conclusion must also identify the unresolved questions.
Incidence
The true incidence of clinically significant PPI-associated hypomagnesaemia is uncertain.
Susceptibility
It is unclear why some long-term users develop profound hypomagnesaemia while many others do not.
Duration threshold
The literature does not establish a single exposure duration at which risk begins.
Dose-response
A clear dose-response relationship has not been consistently demonstrated.
Molecular mechanism
The precise molecular mechanisms remain incompletely resolved.
Population risk
The degree to which risk varies with age, renal function, nutritional status, concomitant medicines and genetic susceptibility remains uncertain.
These uncertainties do not erase the signal.
They define the remaining questions.
Signal Prioritisation
This signal would reasonably receive significant pharmacovigilance attention because several important characteristics converge:
- potentially severe clinical consequences;
- plausible biological mechanism;
- repeated individual cases;
- positive dechallenge;
- positive rechallenge;
- apparent class effect;
- potential for delayed recognition;
- widespread exposure to PPIs;
- and reversibility after withdrawal.
The signal therefore illustrates an important distinction:
A signal can be highly credible even when its incidence and molecular mechanism remain incompletely quantified.
What Would a Pharmacovigilance Team Review?
A practical assessment would include several evidence streams.
Individual Case Safety Reports
Review:
- duration of PPI exposure;
- serum magnesium concentration;
- associated calcium and potassium abnormalities;
- symptoms;
- renal magnesium excretion where available;
- concomitant medicines;
- gastrointestinal losses;
- renal disease;
- nutritional status;
- dechallenge;
- rechallenge;
- recurrence with another PPI.
Cases with documented positive rechallenge should receive particular attention.
Literature
Characterise:
- case reports;
- case series;
- systematic reviews;
- epidemiological studies;
- mechanistic studies;
- exposure duration;
- class consistency;
- severity;
- clinical outcomes.
Do not treat all literature as equivalent.
A case report with a positive rechallenge answers a different question from a cross-sectional database study.
Aggregate Safety Data
Review whether cases show:
- increasing reporting;
- concentration among long-term users;
- class-wide distribution;
- severe electrolyte abnormalities;
- recurrent dechallenge/rechallenge patterns;
- associated cardiac or neurological events.
Product Information
The assessment should consider whether the existing safety information appropriately reflects:
- the adverse reaction;
- relevant risk factors;
- the possibility of severe manifestations;
- and appropriate clinical management or monitoring considerations where applicable.
The objective is not to add warnings simply because a signal exists.
The objective is to determine whether the existing information remains adequate in light of the evidence.
Historical Signal Evolution
One of the most valuable ways to study pharmacovigilance is to examine how the interpretation of a signal changes over time.
The PPI–hypomagnesaemia story illustrates this particularly well.
Stage 1: Individual observation
A clinician observes unexplained severe hypomagnesaemia in a long-term PPI user.
Stage 2: Repeated cases
Additional cases demonstrate similar abnormalities.
Stage 3: Dechallenge
Withdrawal repeatedly results in recovery.
Stage 4: Rechallenge
Re-exposure results in recurrence.
Stage 5: Class pattern
Multiple PPIs produce similar observations.
Stage 6: Mechanistic investigation
Studies investigate intestinal magnesium absorption and transport.
Stage 7: Epidemiological evaluation
Population studies investigate whether PPI users have a higher prevalence or incidence of hypomagnesaemia.
Stage 8: Regulatory and clinical recognition
The adverse reaction becomes incorporated into pharmacovigilance and clinical risk management.
This is the evolution of a signal from observation to increasingly coherent causal evidence.
A Two-Level Conclusion
Evidence conclusion
The evidence supports a causal association between PPI exposure and hypomagnesaemia in susceptible patients.
The strongest evidence comes from repeated, physiologically coherent individual cases showing recovery after withdrawal and recurrence after re-exposure, including recurrence with different PPIs. 14
Observational studies also support an association, although their results are heterogeneous and are subject to confounding and differences in outcome definitions. 15
Mechanistic conclusion
An effect on intestinal magnesium absorption is biologically plausible and supported by clinical and experimental observations, but the precise molecular mechanism and determinants of individual susceptibility remain incompletely established. 16
The overall pharmacovigilance conclusion is therefore stronger than the mechanistic conclusion:
PPI exposure can cause clinically important hypomagnesaemia, particularly during prolonged treatment, although the incidence, susceptibility factors and precise molecular mechanism remain incompletely characterised.
That is a sufficiently strong causal conclusion without pretending that every mechanistic question has been solved.
What This Example Teaches About Signal Evaluation
The PPI–hypomagnesaemia signal demonstrates several general principles.
1. Dechallenge can be powerful
Repeated improvement after withdrawal can materially strengthen causal assessment.
2. Rechallenge can be exceptionally informative
A reproducible recurrence after re-exposure is one of the strongest individual-case clues available.
3. Do not deliberately create rechallenge
Existing clinical rechallenges can provide powerful evidence without deliberately exposing patients to unnecessary risk.
4. Long latency does not exclude causality
A cumulative depletion mechanism may produce an adverse event after years of exposure.
5. Laboratory physiology matters
Urinary magnesium measurements helped distinguish gastrointestinal from renal causes in important cases.
6. Class effects should be tested
Similar reactions with several members of a drug class can strengthen the causal interpretation.
7. Epidemiology and individual cases can tell different stories
Heterogeneous population studies do not necessarily invalidate strong individual-case evidence.
8. Mechanism and causality should be separated
The adverse reaction can be well supported even when the exact molecular pathway remains uncertain.
9. Severity matters
A rare laboratory abnormality becomes more important when it can produce arrhythmias, seizures or other serious consequences.
10. The strongest conclusion is often layered
A good pharmacovigilance assessment should separately state:
- what is established;
- what is strongly supported;
- what is plausible;
- and what remains uncertain.
Key Takeaways
PPI-associated hypomagnesaemia is a particularly instructive historical pharmacovigilance signal because its strongest evidence does not come from a single large epidemiological study.
The important observations are:
-
Severe hypomagnesaemia has been repeatedly reported during PPI exposure.
-
Some cases demonstrate appropriate renal magnesium conservation, supporting impaired intestinal absorption rather than renal wasting. 17
-
Withdrawal of PPI therapy can result in rapid recovery.
-
Rechallenge can produce rapid recurrence, providing strong individual-case evidence. 18
-
Similar reactions have occurred with multiple PPIs, supporting a possible class effect. 19
-
The latency can be prolonged, sometimes extending over several years. 20
-
Observational epidemiological studies generally support an association, but substantial heterogeneity limits precise estimation of risk. 21
-
An effect on intestinal magnesium absorption is biologically plausible, although the precise molecular mechanism and susceptibility factors remain incompletely established. 22
-
Severe hypomagnesaemia can produce clinically important secondary abnormalities, including hypocalcaemia, hypokalaemia and potentially serious cardiac or neurological manifestations. 23
-
The case demonstrates why pharmacovigilance signal evaluation should integrate individual cases, dechallenge, rechallenge, physiology, class effects, epidemiology and mechanism rather than relying on a single evidence source.
References
-
European Medicines Agency. Guideline on good pharmacovigilance practices (GVP) Module IX – Signal management. EMA. 24
-
Hess MW, Hoenderop JGJ, Bindels RJM, Drenth JPH. Systematic review: hypomagnesaemia induced by proton pump inhibition. Aliment Pharmacol Ther. 2012;36(5):405-413. doi:10.1111/j.1365-2036.2012.05201.x. 25
-
Cundy T, Mackay J. Severe hypomagnesaemia in long-term users of proton-pump inhibitors. Clin Endocrinol (Oxf). 2008;69(2):338-341. 26
-
Hoorn EJ, van der Hoek J, de Man RA, Kuipers EJ, Bolwerk C, Zietse R. A case series of proton pump inhibitor-induced hypomagnesemia. Am J Kidney Dis. 2010;56(1):112-116. 27
-
Mackay JD, Bladon PT. Hypomagnesaemia due to proton-pump inhibitor therapy: a clinical case series. QJM. 2010;103(6):387-395. 28
-
Park CH, Kim EH, Roh YH, Kim HY, Lee SK. The association between the use of proton pump inhibitors and the risk of hypomagnesemia: a systematic review and meta-analysis. PLoS One. 2014;9(11):e112558. 29
-
Cheungpasitporn W, Thongprayoon C, Kittanamongkolchai W. Proton pump inhibitors linked to hypomagnesemia: a systematic review and meta-analysis of observational studies. Ren Fail. 2015;37(7):1237-1241. 30
-
Liao S, Gan L, Mei Z. Does the use of proton pump inhibitors increase the risk of hypomagnesemia: an updated systematic review and meta-analysis. Medicine (Baltimore). 2019;98(13):e15011. 31
-
Toh JW, Ong E, Wilson R. Hypomagnesaemia associated with long-term use of proton pump inhibitors. Gastroenterol Rep (Oxf). 2015;3(3):243-249. 32
-
Epstein M, McGrath S, Law F. Proton-pump inhibitors and hypomagnesemic hypoparathyroidism. N Engl J Med. 2006;355:1834-1836.
-
Kuipers MT, Thang HD, Arntzenius AB. Hypomagnesaemia due to use of proton pump inhibitors—a review. Neth J Med. 2009;67:169-172.