Analysis on external competency assessment for malaria microscopists in China

Abstract

Background

In order to meet the requirement of malaria elimination (ME), three courses of the External Competency Assessment of Malaria Microscopists (ECAMM) were conducted during 2017–2018 in China by facilitators designated by the World Health Organization (WHO-ECAMM). A training course with a model copied from the WHO-ECAMM course was also held a week ahead of ECAMM in March 2018. Thirty-six participants completed these courses and obtained different results.

Methods

The slide structures, agendas, score calculations, and the levels of certifications of the four courses strictly adhered to the WHO guidelines. All the data were collected in Excel 2016 and analysed in Graphpad Prism5 or SPSS 23. Significant differences were evaluated in Graphpad Prism5 by two-tailed paired t tests between the pre-assessment and final-assessment for each of the four courses, as well as one-way ANOVAs with Kruskal–Wallis tests and Dunn’s post hoc tests among the final assessments of the four courses. Correlations between participants’ competency results and their ages, years working on malaria, and numbers of malaria cases reported in their provinces were evaluated by bivariate correlations (two-tailed) and linear regression (excluding cases pairwise) in SPSS 23. The Pearson correlation coefficients (r values), P values (two tailed), adjusted R square (Adjusted R²), standardized coefficients (β) and Sig. P values were recorded. The percentages of participants who gave the right answer to each slide (PPS) in the final assessments of the three WHO-ECAMM courses were calculated. Correlation analysis between PPS and parasitaemia (100–2000 parasites/μL) of Plasmodium falciparum slides used in species identification and parasite counting, were also evaluated via bivariate correlations (two-tailed) tests.

Results

Among the 36 participants, 16 participants were certificated as Level 1 (two from NRL), 10 were certified as Level 2 (one from NRL). Within the same course, participants had improved their average scores from pre-assessments to final assessments. The numbers of malaria cases reported in participants’ provinces were strongly correlated to their species identification (SI) scores; r = 0.45, P = 0.040, n = 21; r = 0.57, P = 0.001, n = 32; r = 0.56, P = 0.007). The parasitaemia of P. falciparum within 100–2000 parasites/μL was correlated significantly (r = 0.44, P = 0.008, n = 36) with the PPS of all counting slides but not with slides for identification (r = − 0.018, P = 0.93, n = 30).

Conclusions

The analysis and comparison of participants’ competency results not only verified that the model of the WHO-ECAMM course had strong power in improving and assessing microscopists’ competencies but also reflected the correlation between decreased numbers of indigenous malaria cases and microscopists’ competencies in certain areas in China.

Background

China is planning to declare malaria elimination (ME) in 2020 [1]. To support the declaration and sustain the achievements in ME, a sensitive surveillance system based on precise and timely diagnosis of malaria cases in current and future China is required [2,3,4,5,6,7]. Hence, certificated microscopists with high levels of competency in malaria diagnosis were essential. In the past dozen years, the model for the External Competency Assessment of Malaria Microscopists (ECAMM) courses—developed by the World Health Organization (WHO)—has been conducted in more than 100 selected countries in the West Pacific Region, Southeast Asia Region, Africa Region and Eastern Mediterranean Region. This model had been verified to be an effective model to evaluate competencies of microscopists. In order to meet the requirement of ME, three WHO-ECAMM courses were held during 2017–2018, which were attended by 36 Chinese microscopists from either the National Reference Laboratory for Malaria Diagnosis (NRL) or 31 provincial laboratories. To obtain preliminary knowledge prior to the WHO-ECAMM course, 12 microscopists from the course in March 2018 received a training course with a model copied from the WHO-ECAMM course. The 36 participants had different backgrounds and obtained different results from the WHO-ECAMM courses. In this study, the competency results from these four courses were reported and analysed to determin ways to best sustain and enhance these competencies in China.

Methods

Administration of WHO-ECAMM and the training courses

WHO-ECAMM courses were held at the National Institute of Parasitic Diseases (NIPD) at the Chinese Center for Disease Control and Prevention (China CDC) in Shanghai by the facilitator, Ken Lilley (from WHO), in September 2017, March 2018 and November 2018. The slide structure and agenda strictly followed WHO guidelines [8, 9]. The training course was conducted a week ahead of the WHO-ECAMM course that was held in March 2018. The slide structure (Table 1), agenda, score calculations and level of certification of the training course were copied from WHO-ECAMM [8, 9], in which all participants were required to read 74 slides in 4 days, of which 56 slides were used for competency assessment. The slides for the four courses were all from the slide banks of the Research Institute for Tropical Medicine (RITM). Participants’ scores or accuracy (%) for parasite detection (PD), species identification (SI), and parasite counting (PC) were calculated by the following three formulas:

$${\text{PD}} = {\text{Pp}} \times 100/\left( {Pp + FNp} \right) ; {\text{ SI}} = {\text{Np}} \times 100/\left( {Np + FPp} \right);{\text{ and PC}} = {\text{CCs}} \times 100/{\text{T}} .$$

Table 1 Slides structure in training course

Pp denotes the number of slides with malaria parasites (positives slides) noted by the participant. FNp denotes the number of false negatives noted by the participant. Np denotes the number of negatives noted by the participant. FPp denotes the number of false positives noted by the participant. CCs denotes the number of slides with PC within 25% of the true count noted by the participant. T denotes the total number used for assessment. A participant was certificated as having a high level (1 or 2 levels) of competency only when a participant’s score of PD and SI were both equal to or higher than 80% and their PC score was equal to or higher than 40% (Table 2).

Table 2 Competency levels and criteria

Background of participants from the three different WHO-ECAMM courses

For the 12 participants in the WHO-ECAMM course in March 2018, seven of whom came from non-malaria-epidemic provinces, one participant had attended the WHO-ECAMM course in the past and all received the training a week ahead of the main course. Twenty-four participants in the other two WHO-ECAMM courses were from NRL and malaria-epidemic provinces. Nine of the 12 participants attending the course in November 2018 had taken part in the WHO-ECAMM course at least once in the past three to 6 years. However, most of the participants (11/12) from the course in September 2017 had neither of the above experiences.

Data collection

All the data were collected in Excel 2016. Detailed basic information of the 36 participants (e.g., ages and years working on malaria) from the three WHO-ECAMM courses and their competency results—including PD, SI, PC and certification levels (CLs)—were collected and consulted from mission reports of the WHO-ECAMM facilitator. The malaria situations (reported cases) in 2014–2017 in the 31 provinces from which the 32 participants worked were gathered (two participants from same province) [10,11,12,13]. Also, the answer of each participant given to each slide in the final assessment of the three WHO-ECAMM courses was recorded.

Data analysis

All the data were analysed in Graphpad Prism5 or SPSS 23. To compare the competency results (mean values with 95% confidence interval, CI) between the pre-assessments and final-assessments from each of the four courses, significant differences (P < 0.05, significant; P ≥ 0.05, not significant) were evaluated by two-tailed paired t tests in Graphpad Prism5. Additionally, the final-assessments of the four courses were evaluated by one-way ANOVAs with Kruskal–Wallis tests and Dunn’s post hoc tests in Graphpad Prism5. The Pearson correlation coefficients (r values) and P values (two tailed) among the 36 participants’ competency results of PD, SI and PC scores were evaluated by bivariate correlations (two-tailed) in SPSS 23. To verify the factors influencing participants’ competency results, the correlations between participants’ competency results and their ages, years working on malaria, and numbers of malaria cases reported in their provinces were also evaluated. Firstly, the 36 participants’ ages, years working on malaria, and the numbers of malaria cases reported in the 31 provinces in the different courses were counted and compared. The correlation analysis was carried out within each of the four combinations that were assembled by data in the final assessments of the four courses by bivariate correlations and linear regression in SPSS 23. The correlation coefficients (r values), P values (two tailed), adjusted R square (Adjusted R²), standardized coefficients (β) and Sig. P values were recorded. Combination 1 included data in the WHO-ECAMM course in September 2017 and in the training course in which participants attended prior to the main course. Combination 2 included data of Combination 1 and those in the WHO-ECAMM course in November 2018. Data for Combinations 3 and 4 were all obtained from the WHO-ECAMM courses. Combination 3 included data from September 2017 and November 2018 in which participants had not received similar training over the past 3 years. Combination 4 included data from the three WHO-ECAMM courses.

The percentages of participants who gave the right answer to each slide (PPS) in the final assessment of the three WHO-ECAMM courses were calculated and recorded.

Pearson correlation coefficients (r values) and P values (two tailed) between the PPS and parasitaemia (100–2000 parasites/μL) of the Plasmodium falciparum slides used in species identification and parasite counting were evaluated by bivariate correlations (two-tailed) test separately.

Results

Competency results in the training course

The competency results in the pre-assessment and final-assessment were shown in Table 3. One participant from NRL was certified at Level 1, four participants from provincial laboratories were certificated as Level 2, and the other participants were certified as either Level 3 or Level 4. All participants improved their scores. The average scores improved as follows: PD increased from 92% (CI 84–100%) to 93% (CI 88–97%); PC increased from 28% (CI 16–41%) to 42% (CI 31–54%); and SI increased significantly from 58% (CI 40–76%) to 77% (CI 69–86%; P = 0.007, n = 12, paired t test).

Table 3 Competency results of 12 participants in training course

Competency results of the three WHO-ECAMM courses

The competency results of all participants in the three WHO-ECAMM courses were shown in Table 4. In total, 16 participants were certificated as Level 1 (two from NRL), 10 participants were certified as Level 2 (one from NRL), and the others were certified as Level 3 or Level 4. Separately, in the course from September 2017, two participants were certificated as Level 1, four participants were certified as Level 2, three participants were certified as Level 3, and three participants were certified as Level 4. In the course from March 2018, six participants were certificated as Level 1, five participants were certified as Level 2, and one participant was certified as Level 4. In the course from November 2018, eight participants were certificated as Level 1, one participant was certified as Level 2, and three participants were certified as Level 3.

Table 4 Competency results of 36 participants in WHO-ECAMM courses

Correlation analysis showed that the 36 participants’ competencies of SI were significantly correlated with PC (r = 0.48, P = 0.003; n = 36). All average scores of the 36 participants in the final assessments had improved compared with those in pre-assessments, except for the average PD score in September 2017, which was 98% (CI 95–101%) and 97% (CI 95–99%; P = 0.63, n = 12), respectively. Also, participants from September 2017 had improved their SI and PC scores significantly, which increased from 71% (CI 59–84%) to 85% (CI 80–90%; P = 0.014, n = 12), and from 25% (CI 13–37%) to 42% (CI 31–54%; P = 0.036; n = 12), respectively. The participants from March 2018 improved their SI score significantly from 87% (CI 81–93%) to 94% (CI 91–97%; P = 0.021, n = 12). Participants from November 2018 improved both their PD and SI scores significantly from 92% (CI 87–97%) to 98% (CI 96–101%; P = 0.044, n = 12), and from 80% (CI 69–91%) to 94% (CI 91–97%) (P = 0.0065), respectively.

Among the three courses, participants’ average PD scores in the final assessments were 97% (CI 95–99%), 100% (CI 99–101%) and 98% (CI 96–101%). Following a one-way ANOVA with a Kruskal–Wallis test, there was no significant difference among these scores (P = 0.059; n = 12, 12, 12). For the average scores of SI in the three final assessments, participants from the three courses obtained SIs of 85%, 94% and 94%. Participants’ average SI scores in the two courses from 2018 were significantly higher than that in September 2017 (P = 0.0034, Kruskal–Wallis test; n = 12, 12, 12), but there was no significant difference between the courses during 2018. For the average score of PC in the final assessments, participants from the three courses obtained PC scores of 42% (CI 31–54%), 56% (CI 46–66%) and 50% (CI 43–57%). There were no significant differences among these scores (Kruskal–Wallis test, n = 12, 12, 12).

Among the four courses, the competency results in the final assessments of the training course were the lowest. However, analysis showed that 12 participants in this course improved their average scores of PD, SI or PC (P = 0.0084, P = 0.0003, P = 0.0043; Kruskal–Wallis test; n = 12, 12, 12, 12) from pre-assessment in the training course to the final-assessment in the WHO-ECAMM course in March 2018.

Influence of parasitaemia on participants’ competency results of P. falciparum identification and counting

The correlation analysis showed that there was no correlation between the PPS and parasitaemia of P. falciparum slides (r = − 0.018, P = 0.93, n = 30) that were used in species identification. However, these two datasets correlated significantly (r = 0.44, P = 0.008, n = 36) when the slides were used for parasite counting and their parasitaemia were within 100–2000 parasites/μL.

Analysis of participants’ backgrounds and basic information

The backgrounds and basic information of the 36 participants from 31 provincial laboratories and NRL in the different courses were analysed and compared (Table 5, 6). The average age of participants in the three WHO-ECAMM courses was 38 years old (CI 36–40 years old, n = 36). Among them, the youngest and oldest participants were 28 and 54 years old. There was no significant difference among the ages of participants from the three different WHO-ECAMM courses (P = 0.61). All of the participants had experience working on malaria. They had worked on malaria for at least 1 year and for as long as 32 years. On average, they had worked in malaria diagnosis for 12 years (CI 9–14 years, n = 36). The numbers of malaria cases in the 31 provinces ranged from 1 to 469 (Mean number = 101 cases, CI 59–144 cases, n = 31). Participants from provinces where more cases were reported tended to work on malaria longer (r = 0.37, P = 0.036). There were no variations in their ages in terms of the differences in the numbers of malaria cases among their different provinces (r = 0.21, P = 0.25).

Table 5 Basic information of all participants in 3 WHO-ECAMM courses

Table 6 Basic information of all participants in 4 combinations

Participants from the WHO-ECAMM course of March 2018 had worked in malaria diagnosis significantly less time (mean = 6 years, CI 3–8 years, n = 12) than those from the other two courses (mean = 16 years, CI 10–21 years, n = 12; mean = 14 years, CI 9–19 years, n = 12; P = 0.0040, Kruskal–Wallis test). The average number of malaria cases (Mean = 199 cases, CI 111–289 cases, n = 11) in provinces from participants in the course of November 2018 was significantly greater than those in provinces from participants in the other two courses (mean = 68 cases, CI 19–116 cases, n = 11; mean = 54 cases, CI − 20 to 116 cases, n = 10; P = 0.005, Kruskal–Wallis test).

Evaluation of correlation between competency results and factors

The correlation analysis results of data in the four combinations were shown in Table 7. Nearly none of the combinations showed a correlation of participants’ competency results with their ages and years that participants worked on malaria.

Table 7 Correlation values of 3 factors and competency results among 4 combinations

For the numbers of malaria cases in provinces, the correlation analysis between them and participants’ SI scores presented a significant positive correlation in Combinations 1, 2 and 3 (r = 0.45, P = 0.040, n = 21; r = 0.57, P = 0.001, n = 32; r = 0.56, P = 0.007, n = 22). The regression results were also well supported these correlations in Combination 2 and 3 (Adjusted R² = 0.28, 0.20; β = 0.50, 0.57; Sig. P = 0.006, 0.011; n = 32, 22). However, the correlation disappeared or degraded in Combination 4 (r = 0.30, P = 0.092, Adjusted R² = 0.13; β = 0.45; Sig. P = 0.021; n = 32). In all correlation analysis, no factors were shown to correlate with participants’ PD scores (|r| ≤ 0.38, P ≥ 0.11).

Discussion and conclusion

In 2017, no indigenous cases of malaria were reported in China, which gave a boost to all Chinese people [10, 14]. This represented a crucial milestone for China in terms of reaching the ME goal. However, it should be noted that imported cases were still reported in all 31 provinces in China, including seven non-epidemic provinces [5, 6, 10,11,12,13]. Additionally, the malaria patients were not restricted to known areas or designated by natural environments any longer. If the cases could not be found in time, especially in some areas where malaria cases were seldom or had not been reported for a long time, severe cases (even potentially fatal cases) [5, 6, 10,11,12,13] and/or retransmission could occur [15,16,17]. Hence, continued efforts were needed to strengthen the malaria monitoring system, which depends on a strong and widely distributed laboratory network that can ensure accurate and timely diagnoses of malaria cases.

From September 2017 to the end of November 2018, three WHO-ECAMM courses were held to determine microscopists’ competency levels and to strengthen the laboratory network for malaria diagnosis in China. Thirty-six participants were certificated in these courses and their certifications were valid until 2020. Thirty-two of them covered all provinces in China. Most of them were young and experienced. Sixteen participants (including two from NRL) were certificated as Level 1 and 10 participants (including one from NRL) were certified as Level 2. All of them were either trainers, facilitators, or inspectors in the laboratory network of malaria diagnosis for achieving the ME goal in China. However, because malaria activities were organized in provincial units under a national level in China, at least one microscopist with a high level (1 or 2 level) in each province was essential for the laboratory network. Thus, 26 microscopists with high levels is not sufficient for such a large area of China. Hence, more training courses and WHO-ECAMM courses were required to strengthen and certify microscopists’ competencies.

Analysis on the competency results of the participants showed that there were differences among the three WHO-ECAMM courses, especially in terms of SI and PD scores. PD and SI were the key requirements in the accurate diagnosis of malaria. These two competencies not only influenced each other, but also influenced PC competency greatly. As such, it was necessary to find the factors that might influence these two competencies. Thus participants’ basic information and backgrounds were investigated. The competency results in the training course were also applied in this investigation because the results reflected their original competencies in their routine work. The results showed that (data in Combination 1, 2 and 3) if there were no training the numbers of malaria cases reported in participants’ provinces significantly correlated to participants’ SI scores. In contrast, competencies of even experienced microscopists in provinces where malaria gradually became a rare disease concomitantly decreased because they did not have enough malaria slides to review in their routine work over a long period of time [5, 6, 10,11,12,13, 18]. Additionally, microscopists in provinces where malaria cases were less reported usually would not remain at their posts for a long time, and new staff in these provinces would not be able to improve their competencies because they would have less of a chance of reading slides [4, 18,19,20,21]. These outcomes were also roughly in accordance with the trend of participants’ competency results in different courses and their backgrounds and/or basic information.

Surprisingly, although their previous competency results in the training course were worse, participants’ results in the WHO-ECAMM course of March 2018 were distinctly better than those in 2017 and were not inferior to those in November 2018. This significant change was ascribed to the training course held ahead of the WHO-ECAMM course. The training course had a model that was copied from the WHO-ECAMM and it improved these participants’ competencies greatly and led to the disappearance or degradation of the influence of participants’ previous working experiences (Combination 4). This power was consistent with the significant improvement of participants’ assessment results from pre-assessment to final assessment in each course.

Because of the strong power of the model of the WHO-ECAMM course in improving and assessing microscopists’ competencies, this model provides a new perspective for simultaneously training and assessing microscopists in China, especially in areas where malaria cases have been occasional or seldom reported in recent years. In these areas, microscopists usually had worked for a period but their competency could not sustain or improve due to the nature of their routine work. Even slides might be inadequate for conducting training courses in some provinces. As such, this kind of model (WHO-ECAMM) might not be applicable in this case.

Besides participants’ experiences, the slides themselves also influenced their competency results. Extensive misidentification of P. falciparum was not regarded as a difficult point for all microscopists but only for those who had less of a chance to examine the slides. However, parasite counting was difficult for all microscopists, especially when counting slides with low parasitaemia (one less or more parasite counted might influence the results greatly). Additionally, according to the analysis, participants’ PC competencies were significantly influenced by their SI competencies. Hence, microscopists’ capabilities both on SI should be strengthened before making improvements in PC competency. Additionally, counting skills that strictly followed the rules recommended by WHO were also practical for accurate counting. All participants in the course of March 2018 and nine participants in the course of November 2018 had experienced formal counting practice repeatedly. Consequently, they mastered their counting skills more skillfully than those in the course of September 2017, in which no participants had this experience. However, the low score or PPS in counting also indicated that most of the participants in China seldom counted parasites in their routine work.