Evaluating the quality of routinely reported data on malaria commodity stocks in Guinea, 2014–2016

Background Ensuring malaria commodity availability at health facilities is a cornerstone of malaria control. Since 2013, the Guinea National Malaria Control Programme has been routinely collecting data on stock levels of key malaria commodities through a monthly routine malaria information system (RMIS). In parallel, biannual end-user verification (EUV) surveys have also assessed malaria commodity availability at a subset of health facilities, potentially representing a duplication of efforts. Methods Data on 12 malaria commodity stock levels verified during four EUV surveys conducted between 2014 and 2016 was compared to data for the corresponding months submitted by the same health facilities through the RMIS. The sensitivity and specificity of the RMIS in detecting stock-outs was calculated, as was the percent difference between average stock levels reported through the two systems. Results Of the 171 health facilities visited during the four EUV surveys, 129 (75%) had data available in the RMIS. Of 351 commodity stock-outs observed during the EUV in the sampled reporting health facilities, 256 (73%) were also signaled through the corresponding RMIS reports. When the presence of malaria commodity stocks was confirmed during the EUV surveys, the RMIS also reported available stock 87% (677/775) of the time. For all commodities, the median percent difference in average stock levels between the EUV and RMIS was 4% (interquartile range − 7 to 27%). Conclusion The concordance between stock levels reported through the RMIS and those verified during the EUV visits provides certain evidence that RMIS data can inform quantification and procurement decisions. However, lower than acceptable rates of reporting and incomplete detection of stock-outs from facilities that do report suggest that further systems strengthening is needed to improve RMIS reporting completeness and data quality.

ensure timely diagnosis and correct treatment of confirmed malaria cases. Ensuring effective treatment requires a steady and reliable supply of malaria tests and medicines at health facilities.
Key malaria commodities include malaria rapid diagnostic tests (RDTs), medicines used for treatment of uncomplicated malaria by artemisinin-based combination therapy (ACT), quinine tablets, injectable forms of quinine and artemisinin for treatment of severe malaria, sulfadoxine-pyrimethamine (SP) for intermittent preventive treatment in pregnancy, and LLINs for routine distribution to pregnant women and infants. These commodities are financed and centrally procured by the government of Guinea and donors, primarily the Global Fund to Fight AIDS, Tuberculosis and Malaria (GF) and the U.S. President's Malaria Initiative (PMI), and stored at the central medical stores. From there, they are distributed to regional depots, from which Guinea's roughly 400 public hospitals and health centres are supplied when they directly request commodities from the regional depot based on needs, using a "pull system". Health centres in turn supply the community health workers and health posts in their catchment area. Malaria commodities are provided free of charge to patients of all ages.
Monitoring stocks of malaria commodities throughout the system is a priority activity for the NMCP and its partners. Stock availability is an important indicator of health system readiness and sudden interruptions in the supply chain may affect the quality of service and health programmes [5]. Moreover, frequent stock-outs of essential anti-malarial drugs due to improperly managed supply chains can affect healthcare provider prescription behavior [6], increase patient healthcare costs [7], and impede universal access to quality malaria case management [5,8,9]. However, while there has been work on developing tools for malaria supply chain management in endemic countries [10], few formal studies of optimal supply chain policies have been performed [11][12][13].
During initial expansion of malaria control efforts in Guinea, the country's logistic management and information system (LMIS) at the time was originally targeted to manage and improve management of commodities at the level of the central medical stores and regional depots. Because the LMIS was not designed to collect data on commodity stocks at the health-facility level, the Guinea NMCP decided to include data on malaria commodity stocks as part of its monthly routine malaria information system (RMIS), parallel to and independent of the larger health management information system. A parallel malaria-only reporting system was deemed necessary in the absence of a functional health management information system at the time. The RMIS is supported by PMI and GF and was rolled out in 2013, reaching nationwide coverage in 2014. District-level reporting improved from 66% completeness in late 2014, shortly after expansion, to consistent 100% completeness since early 2016. Data on stock levels at the beginning of the month, stock levels at the end of the month, number of days of stock-out, and consumption rates, including use, expiration, and loss, are reported through the RMIS by hospitals and health centres for 12 malaria commodities. Monthly data are recorded in paper forms at health facilities, and then transferred to the district health authorities, where they are inputted into an electronic database and sent to the national level. The Guinea NMCP routinely analyses the reported data to identify areas with under-and overstock of commodities and monitors stock levels and consumption rates to forecast commodity needs. The malaria monthly bulletin issued by NMCP has included district-level commodity indicators since its inception in late 2014 and is a key tool in allowing NMCP and its partners to track malaria commodity levels.
When PMI was launched in Guinea in 2012 and the RMIS was not yet operational, PMI implemented periodic end-use verification (EUV) surveys to monitor malaria commodity availability at regional warehouses and in peripheral health facilities [14]. Originally, the EUV surveys were only implemented in the 19 health districts supported by PMI, but starting July 2014, implementation expanded nationwide. During these surveys, teams from the national level visited a cross-sectional sample of hospitals and health centres to record stocks of malaria commodities, examine stock cards, document storage conditions, and review outpatient registries. The surveys were implemented by the PMI-funded Systems for Improved Access to Pharmaceuticals and Services project, but included participation from NMCP and district health authorities. They followed a standard methodology to track PMI-procured commodities from source to end-user.
Since 2016, the improved timeliness and completeness of the RMIS have raised the possibility that it might already be collecting sufficiently reliable commodity data to provide the same kind of accountability that the EUV surveys were originally designed to provide. If that were the case, then the resources currently devoted to the EUV surveys could be redirected to other monitoring and evaluation activities. To explore this possibility, NMCP and its partners undertook a systematic assessment comparing the RMIS commodity reporting to the results of the EUV.

Methods
A retrospective analysis was conducted comparing data on commodity stock-outs and stock levels reported through the RMIS to data collected in health facilities during four EUV surveys. Using the EUV data as the gold standard, the RMIS data were evaluated on reporting completeness, sensitivity and specificity of stock-out detection, and the relative difference in reported stock levels. The RMIS data were obtained from the NMCP national malaria database, and the EUV data were provided by the PMI-funded Systems for Improved Access to Pharmaceuticals and Services project in Guinea.

Data analysis
For each health facility visited during the EUV surveys, the RMIS database was searched by matching on health facility name and district. Variations in spelling and health facility name required manual matching of health facilities in the two databases. Data on day-of-visit commodity stock levels verified during the surveys were abstracted from the EUV database. Data on beginning-of-month and end-of-month stock levels and the number of days with stock-out were abstracted from the RMIS for the month corresponding to the EUV visit for each health facility in the EUV. Data on twelve malaria commodities were analysed: RDTs, the four age/weight formulations of amodiaquine-artesunate (AS-AQ), oral quinine tablets, injectable quinine, injectable artesunate, intramuscular artemether, SP, and LLINs. Data on stocks of AL, which was not in wide use in Guinea in the period covered by the assessment, were considered separately. The July 2014 and December 2014 EUV surveys did not capture data on AL, so indicators for AL for these surveys were not calculated. Moreover, in the period covered by the four surveys, the RMIS data recording and reporting forms did not stratify AL stocks by its four age/weight formulations, so data on AL stocks collected during the EUV surveys, which did stratify by the different formulations, were considered together; AL was considered in stock if any formulation of AL was documented, and AL stock levels were considered to be the sum across all AL formulations.
Reporting completeness of the RMIS was assessed by calculating the proportion of the health facilities included in each EUV survey that had submitted their monthly malaria report through the RMIS for the month of the survey.
The sensitivity and specificity of the RMIS to detect stock-outs of malaria commodities were also assessed. Sensitivity was calculated as the proportion of health facilities with a stock-out during the EUV survey that also reported a stock-out in the RMIS report for the corresponding month. The specificity was defined as the proportion of health facilities without a stock-out during the EUV survey that did not report a stock-out in their corresponding RMIS report. Health facilities that had not submitted RMIS data during the month of the survey were excluded from the analysis of specificity and sensitivity. Stock-outs in the EUV were defined as zero stock level recorded during the day of the survey visit. In the RMIS data, a health facility was considered to have reported a stock-out if beginning-of-month stock levels were reported as zero, end-of-month stock levels were reported as zero, or the number of days of stock-out during the month were greater than zero. The sensitivity and specificity were calculated separately for each commodity and each survey.
The stock levels reported through the RMIS and counted during the EUV visits were compared separately for each survey and then jointly across all surveys. Because day-of-visit data were not available from the RMIS, the average monthly stock levels were analysed, defined as the mean of the beginning-of-month Table 1 Proportion of health facilities visited during four end-user verification (EUV) surveys that also submitted reports through the routine malaria information system, Guinea  and end-of-month stock levels. For each commodity, the total mean monthly stock level per health facility was calculated for the RMIS dataset, and the average percent difference between the RMIS and EUV datasets was calculated. Additionally, for each individual health facility, the percent difference between the EUV and RMIS data was calculated for each commodity. The distribution of health-facility-level percent difference was visualized using boxplots, stratifying by commodity. The proportion of health facilities with percent difference < 25%, 25-50%, and > 50% was calculated. Health facilities that had not submitted RMIS data during the month of the survey were excluded from the analysis of stock levels. Abstracted data were collected in an Excel (Microsoft, Redmond, USA) spreadsheet and data analysis was performed using R version 3.3.2 (R Foundation for Statistical Computing, Vienna, Austria).

Completeness of health facility reporting
Out of the 171 health facilities visited during the four EUVs, 75% (129/171) had submitted corresponding monthly reports through the RMIS (Table 1). Completeness ranged from 67% (30/45) in the October 2015 survey to 87% (27/31) in the July 2014 survey.

Stock-out detection
There were 351 instances of stock-outs for the 12 malaria commodities assessed recorded during the four EUV surveys in health facilities that had submitted RMIS reports. Of these commodity stock-outs, 256 (73%) were also signaled through the corresponding RMIS reports ( Table 2). The sensitivity of the RMIS in detecting stock-outs of all commodities ranged from 68% (119/176) in the August 2016 survey to 84% (53/63) in the July 2014 survey. Across all surveys, sensitivity of the RMIS was highest in detecting injectable artemether stock-outs (sensitivity 98%, 51/52) and least sensitive for SP (sensitivity 31%, 5/16). The RMIS was also highly specific in capturing stock availability. For facilities visited during the EUV for which there was a corresponding RMIS report, there were 775 instances across all commodities and all four surveys where the EUV reported available stock. In the RMIS database, 677 (87%) of these instances were reported having stock available during the corresponding months. The specificity for all commodities ranged from 84% (173/205) in October 2015 to 91% (130/143) in December 2014.
The RMIS was most specific for detecting stock availability of the ASAQ formulations and AL, with specificity exceeding 88% for each. Specificity was lowest for injectable artemether, 43% (6/14).

Stock levels
Average commodity stock levels reported through the RMIS were close to the average stock levels verified during the EUV surveys ( Figs. 1 and 2). The median percent difference in average stock levels between the EUV and RMIS across all commodities in the four surveys was 4%, with the interquartile range spanning − 7% to 27% (Fig. 3).
The concordance between average stock levels across the two data sources belied substantial variation at the individual health facility level (Fig. 3). However, the median percent difference was close to zero for all commodities, and, for all commodities with the exception of injectable artemether, more than half of all absolute percent differences were less than 50% ( Table 3). The distribution of percent difference was stable across the four surveys (Fig. 4).

Discussion
The RMIS data on overall malaria commodity stock levels generally agreed with the stock levels counted during the field visits. However, the detection of stock-outs through the RMIS is still incomplete. About a quarter of health facilities do not report through RMIS, and in the three quarters of health facilities that had submitted malaria reports for the months of the EUV visits, the RMIS was sufficiently sensitive to detect only three quarters of all stock-outs observed in the EUV. The RMIS reported stock-outs not captured by the EUV in 13% of cases. The concordance between the two data sources should be interpreted in the context of the limitations of the methodology of the comparison. Because the EUV surveys only record the actual stock levels on the day of the health facility visit, the data from the monthly RMIS reports and the EUV data could not be compared directly. As a result, the definition of stock-outs differed between the two data sources, and the sensitivity and specificity reported here need to be interpreted in that context. In particular, the sensitivity could have been overestimated and the specificity underestimated since the RMIS can capture stock-outs that occur at any point during the month. In addition, the comparison of stock levels was not a one-to-one comparison, which could account for the observation that although mean stock A further limitation is that there were differences in how AL stock levels were captured in the two systems. Finally, non-reporting health facilities might be more likely to have stock-outs or poor data recording practices, and thus the concordance between the EUV and RMIS stock levels reported here might be an over-estimate. Ultimately, the complexities in validating the RMIS data against the EUV data reflect fundamental differences in the reporting systems' objectives and designs. The RMIS was designed to provide complete, timely data on key health facility malaria indicators for the entire country throughout the year. In contrast, EUV surveys were designed to provide periodic audits of health facilities for the purposes of accountability. The imperfect data completeness, sensitivity and specificity of the RMIS are balanced out by a much larger amount of data collected during the RMIS. For 2016, while the EUV survey visited a total of 64 health facilities, yielding 768 data-points on stock availability, the RMIS collected a total of 5280 monthly health facility reports across the entire year, for a total of 63,360 data points. The spatial and temporal scope of the RMIS allows continuous detection of stockouts throughout the country. Similarly, the representativeness of the consumption and stock level data in the RMIS allows for accurate quantification and forecasting of malaria commodity needs. Finally, the fact that the RMIS is managed and implemented by the NMCP and district health authorities means that it is sustainable in the long term and permits the kind of routine analysis embodied by the production and dissemination of the monthly malaria bulletins. Because it involves teams visiting health facilities, the EUV methodology allows for the incidental collection of data that are not normally available through the RMIS. Verification of commodity stock by survey teams allows for independent assessment of the availability of donorsourced malaria commodities. In addition, survey teams assess other aspects of management of malaria commodities, such as storage conditions and the correct use of stock cards. EUV surveys in Guinea also include a retrospective review of outpatient registries, providing insight into malaria case management practices. Finally, EUVs provide a chance for capacity building for participating NMCP and district health authority staff, while also providing an incidental opportunity for supportive supervision of health facility staff.
The results presented here suggest that the data reported through the Guinea RMIS could be used to monitor commodity stocks in health facilities. In this context, the EUVs do not provide substantial additional actionable information on malaria commodity stock levels and the stock verification component of the EUVs might represent an unnecessary duplication of efforts, on the condition that the RMIS can routinely be queried to provide data on commodity stock levels and stock-outs. Instead, the added utility of the EUV surveys is concentrated in the secondary data incidentally collected during the surveys; for example, storage conditions and case management practices and their role in supervision and capacity building. In recent years, the Guinea NMCP has developed and rolled out a comprehensive supervision tool to be used during malaria-related supervisory visits to health facilities. The additional components of the EUV, such as those related to storage conditions and commodity management practices, are either already included in the supervision tool or could be easily integrated into the existing tool. Resources currently devoted to the implementation of the EUV survey could instead be repositioned to support routine supervisory visits, including increasing their frequency, expanding nationaland district-level participation, and enhancing their focus on data quality. As malaria control efforts in Guinea mature, the focus of NMCP and its partners has been shifting from scale-up of interventions to monitoring and tracking their impact. In parallel, the periodic collection of data through household and health facility surveys has yielded to continuous, routine information systems.

Conclusions
Discontinuation of surveys, originally intended to be temporary measures in the absence of well-functioning routine systems, will need to be informed by systematic evaluations like the one reported here that can assess whether the sensitivity and specificity of routine information systems are adequate and can provide evidence that the routine information systems are sufficiently robust and accurate. Ultimately, the assessment of the commodity data in the RMIS reported here was made possible because of the unique opportunity to have an independent dataset on commodity stocks in the form of the EUV surveys. The ability to rely on the RMIS to provide high-quality commodity data will benefit from future assessments that cross-check independently collected data, such as from data quality audits, with the routinely collected RMIS data. These sorts of assessments are particularly timely as Guinea pursues a modernization of its ensemble HMIS, based on the increasingly popular open-source District Health Information System 2 (DHIS2) reporting platform. This new system has an integrated health-facility-level LMIS, whose framework is modeled on the commodity component of the RMIS. Eventually, the RMIS itself is planned to be fully integrated into the new HMIS, cementing it as a foundation of routine disease surveillance in Guinea.