Malaria vaccine efficacy: the difficulty of detecting and diagnosing malaria
© O'Meara et al; licensee BioMed Central Ltd. 2007
Received: 11 February 2007
Accepted: 26 March 2007
Published: 26 March 2007
New sources of funding have revitalized efforts to control malaria. An effective vaccine would be a tremendous asset in the fight against this devastating disease and increasing financial and scientific resources are being invested to develop one. A few candidates have been tested in Phase I and II clinical trials, and several others are poised to begin trials soon. Some studies have been promising, and others disappointing.
It is difficult to compare the results of these clinical trials; even independent trials of the same vaccine give highly discrepant results. One major obstacle in evaluating malaria vaccines is the difficulty of diagnosing clinical malaria. This analysis evaluates the impact of diagnostic error, particularly that introduced by microscopy, on the outcome of efficacy trials of malaria vaccines and make recommendations for improving future trials.
Vaccines which protect against infectious diseases have saved many lives in the past 200 years, have allowed the eradication of smallpox and near-eradication of polio, and are widely regarded as extremely cost-effective interventions. Vaccines against many common diseases, such as measles and pertussis, have become part of routine paediatric care and public health programs worldwide.
Malaria is estimated to be the deadliest paediatric disease: hundreds of thousands of children under five succumb to malaria each year. An effective vaccine would be a major advance in public health and is actively being sought. Several candidates have been tested in clinical trials and many more are in various stages of development. Many aspects of vaccine trial design are important in ensuring reliable outcomes, but among the most critical are the choice of primary endpoint and thereby the definition of vaccine efficacy. In practice, clinical endpoints depend on three equally important components of malaria diagnosis – detection, quantification and case definition. The factors which contribute to accuracy in each component, specifically with respect to microscopic diagnosis, and their impact on the calculation of vaccine efficacy for various types of Phase II and III clinical trial endpoints are examined.
Measuring vaccine efficacy
Comparison of endpoints of vaccine trials. The mode of action of the vaccine will determine the type of trial and endpoint that is used to evaluate their efficacy.
Target of vaccine*
Type of follow up
Sporozoite challenge (Phase IIa)
Delayed patency (if vaccinees become infected)
Number of primary merozoites**
Field trial in endemic setting (Phase IIb/III)
Time to first infection or first episode
ACD (for time to infection) ACD or PCD (for first episodes)
Incidence of disease episodes
ACD or PCD
For multiple episodes 
Field trial in endemic setting
Incidence of disease episodes (first episode or multiple episodes) (Phase IIb/III)
ACD or PCD
[15, 16, 37]
Density of infection (Phase IIb)
Vaccine efficacy is calculated as a ratio, which leads to the common misconception that uncertainty (i.e. false positives or false negatives) in the numerator and uncertainty in the denominator will cancel or balance. This is not the case, particularly when sensitivity and specificity may differ in the vaccinated and the control groups. The 95% confidence intervals with which VE is typically reported give the range of values within which one can be 95% certain that the true VE lies, based on the observations, but they reflect only one type of uncertainty. They describe the role of random chance in the observations and depend on the sample size and number of observed disease events. They do not reflect measurement error or other types of uncertainty that may be introduced during the trial. The effect of the uncertainty of the measurement technique on the reliability of VE calculations has not been fully evaluated and must be properly incorporated.
When making measurements with categorical conclusions, such as the presence or absence of parasites, measurement uncertainty can decrease the sensitivity and specificity of the endpoint. Low sensitivity results in a misclassification of true positives as negatives and may arise as a result of measurement error or a case definition that is too stringent. If the ex ante sensitivity is low, it can be counterbalanced by increasing the sample size. However, if the ex post sensitivity (as evaluated after the intervention) is lower in the vaccinated group than in the control group, a systematic bias is introduced and the VE will be overestimated.
Low sensitivity can have a similar effect on Phase IIb trials which measure time to first infection in naturally exposed populations; high rates of false negatives and variable sensitivity can obscure a difference between the vaccinated and unvaccinated groups. False negatives have less serious consequences for trial interpretation when the time to first clinical episodes is measured by passive case detection. If no parasites are detected at first presentation and patients are not treated for malaria, then presumably symptoms will persist until parasites are detected and the case is counted.
The specificity of malaria microscopy is also less than perfect, and rates of false positives can reach 24% . The impact of false positives on malaria chemoprophylaxis trials has been evaluated. The analysis is applicable for clinical trials of pre-erythrocytic vaccines where smears are prepared regularly to detect infection; even a 1% decrease in specificity can reduce the observed protective efficacy by 30%.
Quantifying malaria by microscopy is a difficult task, but the importance of this well-known fact is often overlooked. Accurate density determination is critical for at least two endpoints used in clinical trials of malaria vaccines. Blood-stage vaccines can be evaluated by the reduction in density of the blood-stage infection in the vaccinated versus the control group (Phase IIb), and clinical episodes of malaria for any type of vaccine are often defined by a specific density threshold (Phase IIb, III; see next section). Uncertainty in density determination may reduce both the sensitivity and specificity of these endpoints, and the magnitude of the uncertainty is inversely proportional to the density. When density is altered by the intervention (vaccination), the result is a different magnitude of error in the vaccinated versus the control group.
Factors other than measurement error can contribute to uncertainty in parasite density measurements. For example, parasite density is reported as parasites per microliter of blood, but because parasites are counted using white blood cells (WBCs) as an index, a conversion factor -generally a uniform approximation of 8,000 WBCs/μl- is used to convert the number of parasites per WBC into parasites per microliter. However, WBC counts may vary with age, infection status and other factors. Counts can be > 13,000 WBC/μl in asymptomatically infected infants and as low as 5,400 WBC/μl in asymptomatically infected adults. WBC counts in malaria-infected, febrile adults are significantly lower than in uninfected, febrile adults. Thus, adopting a single, uniform approximation as a conversion factor can obscure the true parasite density and potentially lead to even larger errors than the pure counting errors described above- e.g. over-estimating parasite densities by 30% relative to true WBC counts. Using complete blood counts to determine the per-patient WBC count or counting by volume, as has been done in some studies (i.e. [9, 15–17]), can reduce the problem of patient-to-patient variation in WBC counts.
Temporal fluctuations in parasite density in peripheral blood are common and may have greater consequences for parasite density measurement than uncertainty associated with microscopic quantification. Peaks and troughs of density may arise from sequestration, or immune response and the rise of new antigenic variants. Thus it is seldom clear how to interpret a single measurement of parasite density in an individual. For example, infections quantified 24–48 hours apart in each of 11 patients prior to drug treatment showed an average increase of 5,500 parasites per microliter, with a range of -17,000 to 46,000. A longitudinal study of adults in highly endemic areas reported frequent fluctuations of 100-fold in parasite density within as few as 6 hours. Similar results were seen in children monitored daily. More studies are necessary to determine whether a single density determination is an appropriate measurement endpoint for clinical trials.
It is worth noting that body temperature can also fluctuate on the timescale of hours. Choosing a temperature threshold to define a febrile malaria episode raises similar issues to choosing a single density threshold.
In malaria-endemic areas with moderate to high transmission, the presence of parasites alone or parasites with fever are not adequate indicators of a clinical malaria episode. It is not possible to definitively diagnose clinical malaria; therefore, criteria must be chosen for which a patient will be considered to be experiencing a clinical malaria episode. The uncertainty in definition will lead to misclassification of cases, both false positives and false negatives, resulting in decreased sensitivity and specificity.
The fraction of parasitaemic, symptomatic individuals whose malaise is caused by parasites is referred to as the attributable fraction. The attributable fraction can be calculated by comparing the proportion of febrile individuals over a range of parasite densities. Most commonly, a logistic regression model is fit to the data to describe the probability that a fever can be attributed to malaria at any density. The overall attributable fraction is calculated by averaging this probability over all cases. The logistic regression model can be used to calculate the sensitivity and specificity of a particular cut-off density by estimating the number of true cases that will not be counted because parasite density falls below the threshold (false negatives) and the number of non-malaria febrile cases that will be counted as positive because the density exceeds the threshold (false positives). Cases of malaria are then defined by a particular cut-off density which gives the required sensitivity and specificity. As parasite density decreases, the probability that symptoms are due to the observed parasitemia decreases. Therefore, the specificity of the case definition decreases with decreasing threshold density, and the observed vaccine efficacy should also decrease.
The logistic regression model for determining attributable fraction has been used to determine the sensitivity and specificity of case definitions in numerous epidemiological settings. The cut-off density which gives a case definition with sufficiently high sensitivity and specificity depends on many interrelated variables. The appropriate case definition changes as a function of transmission intensity, and, therefore, season . Age, which together with transmission intensity determines previous exposure and immunity, has been correlated to changes in attributable fraction [13, 22, 23, 24], and sensitivity and specificity of a particular case definition. Vaccination is designed to alter the immune status, therefore, the case definition will necessarily have different sensitivities and specificities in the vaccinated compared to the unvaccinated group. All of these factors complicate comparisons between trials that use density cut-off case definitions.
How to reduce uncertainty?
Understanding the sources and impact of uncertainty can lead to more robust endpoint definitions in malaria vaccine trials. The sensitivity and specificity of microscopy are highly reader-dependent. However, particularly in the case of sensitivity, there are intrinsic limitations to microscopy that cannot be completely resolved by improving reader performance. The shape of the density-error relationship amongst the study microscopists should be determined before the study begins. "Over-reading" paradigms, for example having two microscopists read every slide and a third microscopist re-read slides with highly discrepant results, increase the sensitivity and specificity of microscopy and allow on-going, 'real-time' evaluation of the density-error relationship.
Density-ratio endpoints seem to be less sensitive to uncertainty than threshold-dependent endpoints, particularly when the majority of cases fall close to the threshold value. To reduce the error of a threshold-dependent endpoint, a cut-off value based on the parasite:leukocyte ratio rather than an absolute density may be used. Evidence indicates that the parasite:leukocyte ratio is bimodal [13, 25], giving a natural choice for a cut-off value. Deviations from this cutoff value due to experimental error would result in fewer misclassified cases. This observation is consistent with data from other studies which showed that WBC counts were lower in symptomatic, infected individuals compared to those with similar symptoms but no parasites [14, 26].
Errors in density measurement have little impact on vaccine efficacy calculated by attributable fraction.
Mode of action of vaccine
30% reduction in density 25% reduction in febrile episodes
50% reduction in density 50% reduction in febrile episodes
70% reduction in density 25% reduction in febrile episodes
To obtain accurate results, maximize the ability to compare and distinguish between vaccine candidates, and avoid scuttling promising candidates, it is essential to validate the choice of endpoint and the sensitivity and specificity of the endpoint. Measurement error has a significant impact on the quality and reliability of the outcome and should be considered when developing clinical trial protocols. The type of endpoint chosen determines the extent to which measurement error may affect the calculation of VE, ranging from completely obscuring the true efficacy to differences of a few percentage points. Calculating VE using attributable fraction may reduce the error introduced by counting parasites and improve the ability to compare between vaccine trials.
The authors would like to thank Allan Saul for helpful discussions and insightful critique of this work.
- Smith PG, Rodrigues LC, Fine PEM: Assessment of the Protective Efficacy of Vaccines against Common Diseases Using Case-Control and Cohort Studies. Int J Epidemiol. 1984, 13 (1): 87-93. 10.1093/ije/13.1.87.View ArticlePubMed
- Blackwelder WC, Storsaeter J, Olin P, Hallander HO: Acellular Pertussis Vaccines - Efficacy and Evaluation of Clinical Case Definitions. Am J Dis Child. 1991, 145 (11): 1285-1289.View ArticlePubMed
- Lachenbruch PA: Sensitivity, specificity, and vaccine efficacy. Control Clin Trials. 1998, 19 (6): 569-574. 10.1016/S0197-2456(98)00042-7.View ArticlePubMed
- Prudhomme O'Meara W, Barcus M, Wongsrichanalai C, Muth S, Maguire J, Jordan R, Prescott WR, McKenzie FE: Reader technique as a source of variability in determining malaria parasite density by microscopy. Malaria Journal. 2006, 5: 118-10.1186/1475-2875-5-118.View Article
- Bejon P, Andrews L, Andersen RF, Dunachie S, Webster D, Walther M, Gilbert SC, Peto T, Hill AVS: Calculation of liver-to-blood inocula, parasite growth rates, and preerythrocytic vaccine efficacy, from serial quantitative polymerase chain reaction studies of volunteers challenged with malaria sporozoites. J Infect Dis. 2005, 191 (4): 619-626. 10.1086/427243.View ArticlePubMed
- McKenzie FE, Sirichaisinthop J, Miller RS, Gasser RA, Wongsrichanalai C: Dependence of malaria detection and species diagnosis by microscopy on parasite density. Am J Trop Med Hyg. 2003, 69 (4): 372-376.PubMed CentralPubMed
- Ohrt C, Purnomo, Sutamihardja MA, Tang D, Kain KC: Impact of Microscopy error on estimates of protective efficacy in malaria-prevention trials. J Infect Dis. 2002, 186 (4): 540-546. 10.1086/341938.View ArticlePubMed
- O'Meara WP, McKenzie FE, Magill AJ, Forney JR, Permpanich B, Lucas C, Gasser RA, Wongsrichanalai C: Sources of variability in determining malaria parasite density by microscopy. Am J Trop Med Hyg. 2005, 73 (3): 593-598.PubMed CentralPubMed
- Genton B, Betuela I, Felger I, Al-Yaman F, Anders RF, Saul A, Rare L, Baisor M, Lorry K, Brown GV, Pye D, Irving DO, Smith TA, Beck HP, Alpers MP: A recombinant blood-stage malaria vaccine reduces Plasmodium falciparum density and exerts selective pressure on parasite populations in a phase 1-2b trial in Papua New Guinea. J Infect Dis. 2002, 185 (6): 820-827. 10.1086/339342.View ArticlePubMed
- Warhurst DC, Williams JE: Laboratory diagnosis of malaria. J Clin Pathol. 1996, 49: 533-538.PubMed CentralView ArticlePubMed
- Rogier C, Commenges D, Trape JF: Evidence for an age-dependent pyrogenic threshold of Plasmodium falciparum parasitemia in highly endemic populations. Am J Trop Med Hyg. 1996, 54 (6): 613-619.PubMed
- McKenzie FE, Prudhomme WA, Magill AJ, Forney JR, Permpanich B, Lucas C, Gasser RA, Wongsrichanalai C: White blood cell counts and malaria. J Infect Dis. 2005, 192 (2): 323-330. 10.1086/431152.PubMed CentralView ArticlePubMed
- Genton B, Alyaman F, Alpers MP: Assessment of Malaria Morbidity in Vaccine Efficacy Trials. Lancet. 1995, 345 (8952): 804-804. 10.1016/S0140-6736(95)90688-6.View ArticlePubMed
- Delley V, Bouvier P, Breslow N, Doumbo O, Sagara I, Diakite M, Mauris A, Dolo A, Rougemont A: What does a single determination of malaria parasite density mean? A longitudinal survey in Mali. Trop Med Int Health. 2000, 5 (6): 404-412. 10.1046/j.1365-3156.2000.00566.x.View ArticlePubMed
- Farnert A, Snounou G, Rooth I, Bjorkman A: Daily dynamics of Plasmodium falciparum subpopulations in asymptomatic children in a holoendemic area. Am J Trop Med Hyg. 1997, 56 (5): 538-547.PubMed
- Smith T, Schellenberg JA, Hayes R: Attributable Fraction Estimates and Case Definitions for Malaria in Endemic Areas. Stat Med. 1994, 13 (22): 2345-2358.View ArticlePubMed
- Mwangi TW, Ross A, Snow RW, Marsh K: Case definitions of clinical malaria under different transmission conditions in Kilifi District, Kenya. J Infect Dis. 2005, 191 (11): 1932-1939. 10.1086/430006.PubMed CentralView ArticlePubMed
- Dicko A, Mantel C, Kouriba B, Sagara I, Thera MA, Doumbia S, Diallo M, Poudiougou B, Diakite M, Doumbo OK: Season, fever prevalence and pyrogenic threshold for malaria disease definition in an endemic area of Mali. Trop Med Int Health. 2005, 10 (6): 550-556. 10.1111/j.1365-3156.2005.01418.x.View ArticlePubMed
- Vounatsou P, Smith T, Kitua AY, Alonso PL, Tanner M: Apparent tolerance of Plasmodium falciparum in infants in a highly endemic area. Parasitology. 2000, 120: 1-9. 10.1017/S0031182099005211.View ArticlePubMed
- Henry MC, Assi SB, Rogier C, Dossou-Yovo J, Chandre F, Guillet P, Carnevale P: Protective efficacy of lambda-cyhalothrin treated nets in Anopheles gambiae pyrethroid resistance areas of Cote d'Ivoire. Am J Trop Med Hyg. 2005, 73 (5): 859-864.PubMed
- Alonso PL, Sacarlal J, Aponte JJ, Leach A, Macete E, Milman J, Mandomando I, Spiessens B, Guinovart C, Espasa M, Bassat Q, Aide P, Ofori-Anyinam O, Navia MM, Corachan S, Ceuppens M, Dubois MC, Demoitie MA, Dubovsky F, Menendez C, Tornieporth N, Ballou WR, Thompson R, Cohen J: Efficacy of the RTS,S/AS02A vaccine against Plasmodium falciparum infection and disease in young African children: randomised controlled trial. Lancet. 2004, 364 (9443): 1411-1420. 10.1016/S0140-6736(04)17223-1.View ArticlePubMed
- Maguire JD, Lederman ER, Barcus MJ, Prudhomme O'Meara WA, Jordon RG, Duong S, Muth S, Sismadi P, Bangs MJ, Prescott WR, Baird JK, Wongsrichanalai C: Production and Validation of Durable, High Quality Standardized Malaria Microscopy Slides for Teaching, Testing and Quality Assurance during an Era of Declining Diagnostic Proficiency. Malaria Journal. 2006, 5: 92-10.1186/1475-2875-5-92.PubMed CentralView ArticlePubMed
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.