Skip to main content

Keep the quality high: the benefits of lot testing for the quality control of malaria rapid diagnostic tests

Abstract

Background

The production and use of malaria rapid diagnostic tests (RDTs) has risen dramatically over the past 20 years. In view of weak or non-existing in vitro diagnostics (IVD) regulations and post-marketing surveillance (PMS) systems in malaria endemic countries, the World Health Organization, later joined by the Foundation for Innovative New Diagnostics, established an independent, centralized performance evaluation and Lot Testing (LT) programme to safeguard against poor quality of RDTs being distributed through the public health sector of malaria endemic countries. RDT performances and manufacturer quality management systems have evolved over the past decade raising questions about the future need for a centralized LT programme.

Results

Between 2007 and 2017, 6056 lots have been evaluated, representing approximately 1.6 Billion RDTs. A total of 69 lots (1.1%) failed the quality control. Of these failures, 26 were detected at receipt of the RDT lot in the LT laboratory, representing an estimated 7.9 million poor quality RDTs, and LT requesters were advised that RDTs were not of sufficient quality for use in patient management. Forty-three were detected after long-term storage in the laboratory, of which 24 (56%) were found to be due to a major issue with insufficient buffer volume in single use buffer vials, others predominantly showing loss of sensitivity. The annual cost of running the programme, based on expenses recorded in years 2014–2016, an estimated volume of 700 lots per year and including replenishment of quality control samples, was estimated at US$ 178,500 ($US 255 per lot tested).

Conclusions

Despite the clear benefits of the centralized LT programme and its low cost compared with the potential costs of each country establishing its own PMS system for RDTs, funding concerns have made its future beyond 2020 uncertain. In order to manage the risks of misdiagnosis due to low quality RDTs, and to ensure the continued safety and reliability of malaria case management, there is a need to ensure that an effective and implementable approach to RDT quality control continues to be available to programmes in endemic countries.

Background

As RDTs became increasingly used in the early 2000s, critical quality limitations became apparent [1,2,3] which could not be identified and resolved by the weak regulatory oversight in endemic countries. To respond to this need for an independent quality control (QC) system, the World Health Organization (WHO), together with the Research Institute for Tropical Medicine, Philippines, and the Institut Pasteur du Cambodge, Cambodia, initiated a lot-testing (LT) programme in 2003, which then expanded in collaboration with the Foundation for Innovative New Diagnostics (FIND) to a Global RDT Evaluation Programme, comprising the Product Testing and LT programmes [4, 5]. The former informed WHO procurement criteria until RDT prequalification became a requirement in December 2017. LT assesses lot quality before deployment at the request of procurers, manufacturers, or National Malaria Control Programmes (NMCPs), using a testing algorithm with standardized QC samples derived from patient blood [6] (Fig. 1).

Fig. 1
figure1

Overview of the Lot Testing process. a Schematic overview of the overall lot testing process, based on samples at 200 parasites per microlitre of blood (p/µL) and with antigen concentrations within a standardized range corresponding to this parasitaemia, *indicates a condition that applies to combination tests only. b Schematic overview of the testing procedure and pass/fail criteria. RDTs must detect all repeats of all samples at 200 p/µL in order to pass. False positives and anomalies, such as red background, flow failure, etc., are reported as comments and not used for pass/fail decisions. Insufficient buffer is reported as a “fail” result, #confirmatory testing in a second laboratory, if necessary, is performed according to the same two-step procedure than for the initial testing and using a different set of samples. c Main components and activities required for the LT programme are indicated. LT = Lot Testing, Pf = P. falciparum, EQA = External quality assessment, QMS = Quality management system

Within a few years, LT has become a component of the procurement processes of all major public sector procurers, including Médecins Sans Frontières (MSF), the United States President’s Malaria Initiative (PMI), and the United Nations International Children’s Emergency Fund (UNICEF), and it became mandatory for procurement under grants from the Global Fund (GF). When the funding of the UNITAID programme ended in 2017, the WHO leveraged funding to support one laboratory, but stopped replenishing QC sample stocks and the future of the programme is now unclear [7]. Since mid-2017, the GF ceased to require WHO-coordinated LT, requesting instead that countries “arrange for the monitoring of the quality of diagnostic products procured with grant funds in line with relevant WHO guidelines on post-marketing surveillance (PMS) of in vitro diagnostics (IVDs)” [8]. Precise guidance on testing procedures and QC standards is however not available, except some example procedures for LT of HIV RDTs [9]. Other procurers continue to utilize the service.

Impact of the lot-testing programme

From 2007 to 2017, 6056 lots were evaluated, representing an estimated 1.6 billion RDTs [10] (unpublished LT programme data). Overall, 69 (1.1%) failures were detected, of which 26 at RDT receipt and 43 after long-term storage (Table 1). The 26 initial failures correspond to an estimated 7.9 million RDTs, which could have reached end-users and potentially led to incorrect clinical decisions. Most failures (20/26, 76.9%) were due to non-detection of Plasmodium vivax samples at 200 parasites/µL, and failure rates did not change much over time, demonstrating the importance of continued testing.

Table 1 Summary of Lot Testing failures (2007–2017)

More than half of the long-term failures (24/43, 56%) were due to partly evaporated buffer in single-use buffer vials [10,11,12]. Throughout 2014 to June 2017, this problem affected 24 lots of 5 products from 3 WHO-prequalified manufacturers. The detection of this issue triggered a WHO Notice of Concern discouraging procurement of these products until problem resolution [13]. Without LT, such issues may go undetected, as proactive post-marketing surveillance (PMS) for diagnostics is weak and health workers are usually not trained to detect and report quality issues, often being unaware of existing complaint systems [14, 15]. The programme’s formal reporting of RDT anomalies, such as incomplete clearing and red background [16], also formed the basis for successful product replacement in some cases. The ‘non-routine’ LT assessed RDTs withdrawn from the field in various countries and allowed differentiating between product defects, operator errors, and parasite factors. Ruling out product defects led to the subsequent confirmation of high Plasmodium falciparum hrp2/hrp3 gene deletion rates in Eritrea as the cause of false negative RDT results [17], impacting on the use of HRP2-based RDTs in this region.

There is a high probability that the LT programme has also impacted on RDT quality through the pressure placed on manufacturers to meet LT quality standards for all lots sold in the public sector, and/or through the potential publicity associated with failure. A lot failure typically leads to lot replacement and is associated with financial and reputational costs for manufacturers. In a survey conducted in 2014, 31% (11/35) of manufacturers indicated that the LT Programme had triggered improvements of their lot-release procedures or QC panel characteristics [18].

The future of LT

These findings demonstrate important contributions of the LT programme detecting defects even in WHO prequalified products. Other existing processes verifying RDT quality cannot fully substitute for such an independent programme. While manufacturers have their own lot-release procedures, assessed by the WHO prequalification of diagnostics (PQ) process [19], there is no established mechanism to ensure continued adherence to good manufacturing practice and PMS after a product is prequalified. Furthermore, it is the responsibility of the NMCPs and National Regulatory Authorities (NRAs) to perform PMS, but the processes are poorly defined or essentially non-existent in most low resource countries [14, 15].

India and Nigeria have established their own LT systems through collaborations with the WHO/GMP and FIND, respectively, using samples prepared locally as per the publicly available WHO-FIND procedures [20]. Other countries may adopt such a system, however the running costs, the complexity of preparing and characterizing QC samples and ensuring laboratory certification represent a considerable investment in light of the infrequent testing need of a single country. QC samples for RDTs need to have known antigen concentrations but validated antigen quantification methods are not routinely available in endemic countries. From the manufacturer’s or RDT procurer’s point of view, it may be challenging to deal with different laboratories and turnover times for individual countries. This does not obviate the importance of more country ownership and building efficient capacity at country or regional level to manage product quality—including post-distribution quality monitoring—across not just malaria diagnostics but those for other diseases. However, this capacity has to be matched against overall programme costs, and the ability to verify quality of the testing process and site.

The WHO-FIND system used instead an international network of reference laboratories with centralized sample characterization (Fig. 1) and is the only example of a centralized, internationally-recognized LT programme for diagnostics. The main challenges faced, e.g. developing and confirming samples of known reactivity, adapting sample collection sites and periods to changing malaria epidemiology, or facing low sample outputs within the required antigen concentration ranges, were easier to absorb with the centralized mechanism than it would have been for a country-specific programme with limited funding.

Financially, the centralized LT programme is probably the most cost-effective investment. Based on expenses incurred for the 2014–2016 period, the mean cost per lot tested has been estimated to US$ 255 (Table 2). With approximately 700 lots tested per year, this translates to an annual cost of around $US 178,500 in a market of over 400 million tests and an estimated budget of 120 million USD annually [21, 22]. This seems a small price to pay to prevent incorrect diagnoses with poor quality RDTs, and to maintain confidence in RDT results.

Table 2 Estimation of costs for Lot Testing (2014–2016)

Building a framework for RDT quality control

The LT programme is only one part of an intervention framework that ensures RDT quality. Ideally, the NRAs and NMCPs should implement and support proactive and reactive PMS activities with adequate capacity and funding, for which major donors can play a crucial role.

A number of useful resources are publicly available (Table 3), such as the WHO guidelines for PMS of IVDs [9], a troubleshooting guide for frequently observed anomalies and errors [23], and a protocol to help setting up a process for detecting, investigating, and acting upon quality issues [24]. Training manuals, videos, job aids and quizzes have been developed by the WHO, some with extensive field testing in Zambia [25,26,27]. Reference materials also exist, including panels of culture-derived P. falciparum parasites [28], international standards [29], or HRP2, pLDH and aldolase recombinant proteins [30, 31], to be used as such or to develop secondary reference materials. These well characterized, easy to handle and temperature-stable materials may be used in complement to LT with patient-derived samples, e.g. for cross-checking RDTs withdrawn from the field, however standard procedures for their use need to be developed and widely accepted. Panels of pre-diluted recombinant proteins have been evaluated with national reference laboratories and found easy to use for RDT QC (FIND, unpublished work).

Table 3 List of available resource to support the PMS and correct use of malaria RDTs

Conclusions

The WHO strategy is to make RDTs available as close as possible to patients. The tests are often used at community level and results are not confirmed a posteriori, as for tuberculosis or HIV. Total confidence in test results and quality of tests is, therefore, essential. Although tools exist for development of country-level LT, it is unrealistic to expect well-functioning processes to be in place in a near or medium-term in most endemic countries. Regional level capacity may be within reach, but first requires a concerted effort to fund inter-country mechanisms to support it. While the WHO has discussed options for continuation of the programme with key stakeholders, there remains a lack of engagement, heightening the risk of a drop in quality. Continuation of the centralized LT programme would represent a low-cost investment, while in the meantime countries and donors should invest more efforts to develop national or regional capacity. Any form of LT would significantly enhance the overall safeguarding framework, including national regulations and WHO prequalification, to help sustaining the high quality of RDTs.

Availability of data and materials

Not applicable.

Abbreviations

EQA:

External quality assessment

FIND:

Foundation for innovative new diagnostics

GF:

The Global Fund to fight AIDS, Tuberculosis and Malaria

GMP:

Global malaria programme

HIV:

Human immunodeficiency virus

HRP2:

Histidine rich protein 2

IPC:

Institut Pasteur du Cambodge

IVD:

In-vitro diagnostics

LT:

Lot testing

RDT:

Malaria rapid diagnostic test

MSF:

Médecins Sans Frontières

NMCP:

National malaria control programme

NRA:

National regulatory authority

pLDH:

Plasmodium lactate deshydrogenase

PMI:

United States President’s Malaria Initiative

PMS:

Post-marketing surveillance

PQ:

Prequalification of diagnostics

PT:

Product testing

RITM:

Research institute for tropical medicine

QC:

Quality control

QMS:

Quality management system

SOP:

Standard operating procedures

UNICEF:

United Nations International Children’s Emergency Fund

WHO:

World Health Organization

References

  1. 1.

    Bell D, Wongsrichanalai C, Barnwell JW. Ensuring quality and access for malaria diagnosis: how can it be achieved? Nat Rev Microbiol. 2006;4:682–95.

    CAS  Article  Google Scholar 

  2. 2.

    Mouatcho JC, Goldring JP. Malaria rapid diagnostic tests: challenges and prospects. J Med Microbiol. 2013;62:1491–505.

    CAS  Article  Google Scholar 

  3. 3.

    Wongsrichanalai C, Barcus MJ, Muth S, Sutamihardja A, Wernsdorfer WH. A review of malaria diagnostic tools: microscopy and rapid diagnostic test (RDT). Am J Trop Med Hyg. 2007;77:119–27.

    Article  Google Scholar 

  4. 4.

    WHO. WHO-FIND malaria RDT evaluation programme. Geneva, World Health Organization, 2017. https://www.who.int/malaria/areas/diagnosis/rapid-diagnostic-tests/rdt-evaluation-programme/en/. Accessed 17 Feb 2020.

  5. 5.

    Cunningham J, Jones S, Gatton ML, Barnwell JW, Cheng Q, Chiodini PL, et al. A review of the WHO malaria rapid diagnostic test product testing programme (2008–2018): performance, procurement and policy. Malar J. 2019;18:387.

    Article  Google Scholar 

  6. 6.

    Lot testing. WHO: Geneva. https://www.who.int/malaria/areas/diagnosis/rapid-diagnostic-tests/lot-testing/en/. Accessed 17 Feb 2020.

  7. 7.

    WHO. Changes to the WHO-FIND Malaria RDT Lot Testing Programme. Geneva: World Health Organization. 2017. https://www.who.int/malaria/news/2017/rdt-lot-testing/en/. Accessed 18 Feb 2020.

  8. 8.

    The global fund. global fund quality assurance policy for diagnostics products (Issued on 14 December 2010, most recently amended on 4 May 2017). Geneva: World Health Organization. 2017. https://www.theglobalfund.org/media/5885/psm_qadiagnostics_policy_en.pdf?u=636607536980000000. Accessed 17 Feb 2020.

  9. 9.

    WHO. Post-market surveillance of in vitro diagnostics. Geneva: World Health Organization. 2015. https://apps.who.int/iris/bitstream/handle/10665/255576/9789241509213-eng.pdf;jsessionid=E93138DC88FDD4ABD2D215B6406002AB?sequence=1. Accessed 19 Feb 2020.

  10. 10.

    WHO. Results of the FIND-WHO Lot Testing Programme for Malaria Rapid Diagnostic Tests (RDTs). Geneva: World Health Organization. 2017. https://www.finddx.org/wp-content/uploads/2017/08/Malaria-lot-testing-results-2007-endJune-2017_30AUG17.pdf. Accessed 18 Feb 2020.

  11. 11.

    UNITAID. Malaria Diagnostics Technology and Market Landscape, 3rd Edn. Geneva: World Health Organization. 2016. https://www.ghdonline.org/uploads/Unitaid-Malaria-Dx-Tech-Mkt-Landscape-3rd-Ed-April-2016.pdf. Accessed 18 Feb 2020.

  12. 12.

    Harvey SA, Incardona S, Martin N, Lussiana C, Streat E, Dolan S, et al. Quality issues with malaria rapid diagnostic test accessories and buffer packaging: findings from a 5-country private sector project in Africa. Malar J. 2017;16:160.

    Article  Google Scholar 

  13. 13.

    WHO. WHO Information Notice for Users, 2016/01, version 8. Geneva: World Health Organization. 2016. https://www.who.int/diagnostics_laboratory/procurement/160429_single_buffer_ampulla_information_notice_for_users_v8.pdf?ua=1. Accessed 19 Feb 2020.

  14. 14.

    McNerney R, Sollis K, Peeling RW. Improving access to new diagnostics through harmonised regulation: priorities for action. Afr J Lab Med. 2014;3:123.

    Article  Google Scholar 

  15. 15.

    Rugera SP, McNerney R, Poon AK, Akimana G, Mariki RF, Kajumbula H, et al. Regulation of medical diagnostics and medical devices in the East African community partner states. BMC Health Serv Res. 2014;14:524.

    Article  Google Scholar 

  16. 16.

    FIND. Guide for the interpretation of observations noted during lot testing of malaria RDTs. Geneva: Foundation for Innovative New Diagnostics. 2013. https://www.finddx.org/wp-content/uploads/2016/02/malaria_rdt_guide_for_observations_30jul13.pdf. Accessed 19 Feb 2020.

  17. 17.

    Berhane A, Anderson K, Mihreteab S, Gresty K, Rogier E, Mohamed S, et al. Major threat to malaria control programs by Plasmodium falciparum lacking histidine-rich protein 2 Eritrea. Emerg Infect Dis. 2018;24:462–70.

    CAS  Article  Google Scholar 

  18. 18.

    Incardona S, Serra-Casas E, Champouillon N, Nsanzabana C, Cunningham J, Gonzalez IJ. Global survey of malaria rapid diagnostic test (RDT) sales, procurement and lot verification practices: assessing the use of the WHO-FIND Malaria RDT evaluation programme (2011–2014). Malar J. 2017;16:196.

    Article  Google Scholar 

  19. 19.

    WHO. Prequalification of in vitro diagnostics. Geneva: World Health Organization. https://www.who.int/diagnostics_laboratory/evaluations/en/. Accessed 19 Feb 2020.

  20. 20.

    WHO, FIND. methods manual for laboratory quality control testing of malaria rapid diagnostic tests. Version 8. Geneva: World Health Organization and Foundation for Innovative New Diagnostics. 2016. https://www.who.int/malaria/publications/rdt-lab-quality-manual/en/. Accessed 19 Feb 2020.

  21. 21.

    WHO. World Malaria Report 2019. Geneva: World Health Organization. 2019. https://www.who.int/malaria/publications/world-malaria-report-2019/en/. Accessed 04 June 2020.

  22. 22.

    UNITAID. The state of the malaria RDT market 2018, Geneva, Switzerland, 2018. https://unitaid.org/assets/The-state-of-the-malaria-RDT-market-2018.pdf. Accessed 04 June 2020.

  23. 23.

    FIND, Johns Hopkins School of Public Health, Malaria Consortium, Population Services International and WHO. Troubleshooting guide for supervisors overseeing users of malaria RDTs. Geneva, Switzerland, 2015. https://www.finddx.org/wp-content/uploads/2016/10/RDT-supervisors-guide-2016.pdf. Accessed 19 Feb 2020.

  24. 24.

    FIND. Protocol on responding to problems with malaria RDTs. Geneva, Foundation for Innovative New Diagnostics, 2016. https://www.finddx.org/wp-content/uploads/2016/10/Malaria-RDT-protocol-24JUN16-FINAL.pdf. Accessed 19 Feb 2020.

  25. 25.

    WHO. Training materials for use of malaria and G6PD RDTs. Geneva: World Health Organization. https://www.who.int/malaria/areas/diagnosis/rapid-diagnostic-tests/job-aids/en/. Accessed 19 Feb 2020.

  26. 26.

    Harvey SA, Jennings L, Chinyama M, Masaninga F, Mulholland K, Bell DR. Improving community health worker use of malaria rapid diagnostic tests in Zambia: package instructions, job aid and job aid-plus-training. Malar J. 2008;7:160.

    Article  Google Scholar 

  27. 27.

    FIND. Guides, manuals and implementation tools for malaria rapid diagnostic tests. Geneva: Foundation for Innovative New Diagnostics. https://www.finddx.org/reports-and-landscapes/guides-manuals-implementation-tools-for-malaria-rdts/. Accessed 02 March 2020.

  28. 28.

    Zeptometrix Corporation. Malaria P. falciparum culture panels. New York. https://www.zeptometrix.com/categories/assay-developers/microorganisms. Accessed 19 Feb 2020.

  29. 29.

    National Institute for Biological Standards and Controls (NIBSC). Plasmodium falciparum antigens (1st International Standard). South Mimms, UK. https://www.nibsc.org/products/brm_product_catalogue/detail_page.aspx?catid=16/376. Accessed 19 Feb 2020.

  30. 30.

    Microcoat Biotechnologie GmbH. Malaria reference materials: P. falciparum HRP-2, recombinant. Bernried am Starnberger See, Germany. https://www.microcoat.de/Products/malaria-reference-materials-p-falciparum-hrp-2-recombinant/. Accessed 27 Feb 2020.

  31. 31.

    SPAN Diagnostics SARL. Recombinant antigens for Diagnostics. Malaria: Plasmodium Aldolase (P. falciparum); pLDH (P. falciparum); pLDH (P. vivax); pHRP(II) (P. falciparum). https://spandiag.com/recombinant-antigens-for-diagnostics/. Accessed 27 Feb 2020.

Download references

Acknowledgements

The authors wish to acknowledge all staff from the IPC and RITM laboratories who have contributed to the RDT Lot Testing activities, and all staff from WHO and FIND who have contributed to their coordination. Editorial assistance for later drafts of the manuscript was provided by Rachel Wright, PhD, funded by FIND, according to Good Publication Practice guidelines.

Funding

Our work on malaria and specifically RDT quality systems was enabled by grants from the Bill and Melinda Gates Foundation, the Global Fund to fight AIDS, Tuberculosis and Malaria, the United States President’s Malaria Initiative (via the United States Agency for International Development) and UNITAID, as well as funding from the Department of Foreign Affairs and Trade of the Australian government. Publication of this article was funded by FIND.

Author information

Affiliations

Authors

Contributions

SD, SI and XCD conceived the synopsis of the publication. AC drafted the first draft of the manuscript and SD, SI and XCD developed the final draft. JC, FA, TF, JL, CAL, DM, SN, JBS and BW contributed to the lot testing activities, coordination and data. All co-authors contributed to the final draft and reviewed. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Sandra Incardona.

Ethics declarations

Ethical approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Incardona, S., Bell, D., Campillo, A. et al. Keep the quality high: the benefits of lot testing for the quality control of malaria rapid diagnostic tests. Malar J 19, 247 (2020). https://doi.org/10.1186/s12936-020-03324-3

Download citation

Keywords

  • Malaria
  • Diagnostics
  • Rapid diagnostic test
  • Lot testing
  • Quality control
  • Post-market surveillance