Simulated Malaria Online Tool: an instrument for evaluating healthcare providers' practices and contributing to the evidence base for certifying malaria elimination and preventing its re-establishment

Abstract

Background

Healthcare providers (HCPs) practice and correct management of suspected malaria (CMSM) are central components of malaria elimination and prevention of re-establishment (POR) in countries in the elimination phase. However, knowledge of malaria surveillance systems and HCPs practices often wanes in countries aiming to eliminate malaria due to the low numbers of cases. The study aimed to implement a valid Simulated Malaria Online Tool (SMOT) for assessment HCP performance in CMSM and POR in a malaria-free area.

Methods

HCPs were evaluated using SMOT tool based on four criteria including presenting a suspected malaria case for detection of HCPs’ failures in recognition (a), diagnosis (b), appropriate treatment (c), and urgent reporting (d); and compared with simulated patients (SP). Multiple logistic regression analysis was carried out to estimate adjusted odds ratios (ORs) for the risk of HCPs failures.

Results

The overall failure proportion was 237 (83%), and the majority of failures were in recognition (a). There was no significant difference between the SMOT and SP based on all failure criteria (P > 0.05). The private clinic (93%) and the public specialized clinic (70%) had the highest and lowest failure proportions. After passing the recognition stage (a), the overall failure proportions decreased to 47.8% and 25.0% for total HCPs and infectious disease specialists, respectively. In the final analysis, private sector (AOR = 4.36: 1.25–15.2), not-specialist providers (AOR = 2.84: 1.29–6.25) and HCPs with ≥ 5 years’ experience (AOR = 2.03: 1.01–6.25) increased the risk of failure.

Conclusion

Findings confirmed the implementation of SMOT tool in settings where malaria transmission is low or interrupted. The tool is able to identify sub-groups of providers needing strengthening, and contributes to the prevention of malaria re-establishment.

Background

Effective case management of suspected malaria (CMSM) is a central component of malaria elimination and prevention of reestablishment (POR) in an area where malaria transmission is low or has been interrupted [1]. CMSM includes identification, early case detection and/or diagnosis, and prompt treatment with appropriate anti-malarial drugs, in the first 24 h after diagnosis, to prevent severe disease and fatal outcomes as well as onward transmission [2, 3]. The World Health Organization (WHO) is leading the fight against malaria and aims to eliminate the disease in at least 35 countries by 2030 and prevent its re-establishment in countries that have already eliminated it. It also expects to see a 90% reduction in malaria incidence and a 90% reduction in malaria-related mortality by that year [4]. In the elimination phase and in the prevention of malaria reestablishment (POR), case management should also include urgent reporting, which may trigger a response to manage any transmission risk related to the case [5].

Despite efforts by countries in the elimination phase and/or low transmission areas, waning knowledge around CMSM remains a significant drawback in numerous settings [6]. Especially, Health care providers (HCPs) may experience a decrease in awareness due to the low incidence of malaria cases in low-transmission areas or areas that have been interrupted for many years. Therefore, the nonappearance of reported malaria cases may not inevitably mean that malaria has been eliminated [2, 7]. The awareness of health systems to malaria, recognized as vigilance, is a principal element of malaria surveillance following elimination, and it needs to be in place before malaria elimination certification can be granted [5, 8].

Furthermore, in settings where malaria transmission is low or has been interrupted, it is difficult to evaluate the performance of HCPs and malaria surveillance systems due to the absence of actual malaria cases in the field [9]. Therefore, the present study developed and validated a Simulated Malaria Online Tool (SMOT), to evaluate HCPs' practice in relation to the CMSM for these settings [1]. This study was carried out to evaluate HCPs’ practice in CMSM in East Azerbaijan Province of Iran, where local malaria transmission has been interrupted for many years.

Methods

Study design and setting

The study was conducted to investigate the feasibility of applying SMOT to evaluate first-line HCPs’ practice in relation to CMCM in the context of prevention of reestablishment of malaria and documentation for malaria elimination certification. Previously, development and validation of the SMOT tool against live encounters with simulated suspected malaria patients has been demonstrated in a malaria-free area [1]. The tool had been designed online for several reasons, including the possibility of including the main elements of a real patient visit, the possibility of remote evaluation by health managers and malaria surveillance systems, and the inability to modify the answers (to simulate a real world encounters with suspected malaria) given to the first questions as the patient scenario progressed.

In the current study, the application of SMOT was performed in the context of the malaria surveillance programme, East Azerbaijan Province in 2021, where no locally transmitted malaria cases have been reported since 2005 [7].

The study samples were first-line HCPs who worked in the private and public sectors including infectious disease specialists, emergency medicine medical doctors (worked in the emergency wards and/or medical settings), internal medicine and other specialists, family physicians, other medical doctors (MDs), community health workers (Behvarz and Moragheb in Persian), nurses and nurses assistants, and midwives.

The sample size was set at 285 HCPs based on the expected failure proportion (p = 40%) in appropriate CMSM based on previous study in this area [2], type I error (α = 0.05), d = 0.15 p, and 10% non-responders compensation.

For sampling, the roster of all HCPs in East Azerbaijan province (N = 4953) was acquired through Tabriz University of Medical Sciences. HCPs were selected using stratified random sampling. First, HCPs were divided into two categories of MD and non-MD by the public and private sectors, and then into emergency and non-emergency subgroups according to the likelihood of the occurrence of febrile suspected malaria cases for each type of health facility/HCP. This likelihood was determined through the previous evidence [7], consultation with experts, and records of reported new malaria cases in the last 20 years.

The questionnaire link was sent to them through an online method (email, social network, or SMS). After 48 h, the study contacted those who did not respond, leading to a minimal non-response rate and achieving the desired sample size as they could conveniently respond online at any time. In the case of non-response, a provider of the same type and stratum was randomly chosen from the entire population of that stratum. The study response rate was 88.5%. The responses, along with their specifics including baseline and demographic characteristics, type of healthcare provider, type of health facility, specialist, private and/or public sector, were accessed and reviewed on an online platform.

Definition of failure in the correct management of suspected malaria

If one or more of the following criteria were fulfilled, an HCP encountering a suspected malaria case would be considered a failure [10]:

• (a) Does not elicit a travel history in case the patient lives outside an area where malaria transmission occurs, and does not perform or request a diagnostic test for malaria (the exceptional case, where a provider has a patient tested without a travel history is accepted as a pass);

• (b) Does not recommend that patients with suspected malaria undergo diagnostic testing within 24 h of their encounter;

• (c) Does not ensure appropriate treatment within 12 h after a test result positive for malaria;

• (d) Does not report the case to the appropriate public health authority within 12 h after a test result positive for malaria.

Measurement (implementation of SMOT)

HCPs’ practice in CMSM was assessed using SMOT, which has been validated by Azizi et al. [1] with the same categories of HCPs as in the current study and in the same area. The sensitivity and agreement of SMOT in comparison with the simulated patient methodology considered the gold standard were 98.7% and 96.6%, respectively.

SMOT was made available on an online platform with link (https://survey.porsline.ir/s/sSfqXFQ), and sent to HCPs through email and/or virtual networks. Although respondents were aware of their assessment, they did not realize it was related to malaria [1]. In an area where malaria transmission is no longer present, SMOT simulates a febrile suspected malaria case: the case should be questioned about a history of travel to a malaria risk area. The design allows for every question to be answered based on the previous responses. If the respondent asks about their travel history as part of their response to the previous question(s), the travel history details will be triggered.

The tool was planned to assess four performances in CMSM: (a) recognition of a suspected malaria case, (b) malaria testing and/or prescription of a correct malaria diagnostic test, (c) prescription of appropriate anti-malarial drugs, and (d) notification and/or submission of an urgent report of a confirmed malaria case. Even when a respondent failed in the prior criteria, especially after criteria (a) and (b), downstream measures can still be assessed, the tool presents a confirmed positive test result for providers to assess criteria (c) treatment and (d) reporting.

Finally, SMOT considered HCPs into two groups of failure and pass. Providers were classified in the failure group if they failed in at least one of the four criteria (a), (b), (c) and (d). Flowchart and details of SMOT performance and structure is presented in Appendix 1. Details of the tool's performance and psychometric properties have been reported previously [1].

Statistical analysis

The SPSS software (version 21.0, Chicago, IL, USA) was used for data analysis. The Chi-square (χ2) test was used to examine categorical variables. Independent t-test was conducted for quantitative variables when normal distribution and homogeneity of variance were met [11, 12]. Multiple logistic regression analysis was carried out to estimate the crude and adjusted odds ratios (ORs) with 95% confidence intervals (CIs) for the risk of failure. For modelling, all independent variables were initially evaluated through simple logistic regression. Subsequently, variables with a p-value below 0.2 underwent further analysis via multiple logistic regression employing the Enter method. The Hosmer and Lemeshow statistic was utilized to assess data congruence and the model's goodness of fit [12, 13].

Results

A total of 285 HCPs were evaluated with the SMOT tool. The response rate was 88.5%. The mean and standard deviation of the age was 39.16 ± 10.35. Most of the respondents (60%) were general physicians (practitioners). Approximately, 80% of the respondents worked in the public sector, and 46 and 24 percent were in public community-based health facilities and emergency wards, respectively.

Of the HCPs evaluated, 237 (83.15%) had a failure outcome. The average age of providers with pass outcome was higher than that of HCPs with failure outcome, but with no significant difference. A significant relationship (P = 0.045) was found between work experience (< 5) and risk of failure (Table 1).

Table 1 Baseline and demographic characteristics of HCPs in the Correct management of suspected malaria (CMSM)

The highest failure proportion in the CMSM was observed among the general medical practitioners (87.2%). The failure proportion among specialists was significantly lower than among non-specialists (p = 0.023). Regarding health facilities, the highest failure proportions (95.2% and 93.0%) were among MDs who working in private offices and private clinics, respectively. However, the lowest failure proportion was reported in public specialized clinics with 70.5%. Overall, the failure proportion in the private and public sectors was 92.7% and 80.8%, respectively. There was a significant relationship between the health facility, private or public sector, and HCP type (specialist or generalist) with the risk of failure in CMSM (P < 0.05) (Table 2).

Table 2 HCPs practice in CMSM by type of provider and health facilities

Table 3 shows the feasibility and comparison of failure proportions (main (a–d) failures and sub-failures criteria) by SMOT and validation study in an area without local malaria transmission. The failure proportions were compared with the simulated patient (validity study) [1]; There were no significant differences between SMOT and the simulated patient for all failure criteria (P > 0.05 for all criteria). Among the 4 main criteria (a–d), criterion (a) related to not eliciting travel history or suspicion of malaria was the most common at 193 (67.7%), while the failure to prescribe a correct diagnostic test was the lowest at 70 (13.3%). Regarding sub-failure criteria, the most common failure was the lack of knowledge or not using a rapid diagnostic test, accounting for 245 cases (85.9%).

Table 3 Comparison of failure proportions by SMOT and simulated malaria patients; in an area with local malaria-free

The study found almost 48 (17.0%) of HCPs passed all 4 (a-d) criteria in relation to CMSM and only 6 (2.1%) of HCPs were failed in 4 all criteria (a–d). Failure in 3 of 4 main criteria with 90 (31.5%) had the highest failures frequency (Table 4).

Table 4 Proportion of healthcare providers failure criteria in CMSM

Table 5 demonstrates the crude and adjusted ORs and 95% CIs for the risk of HCPs' failures in the CMSM in the presence of covariates. The final analysis showed that the type of HCP (non-specialist) (OR = 2.84; 95% CI 1.29–6.25) and the type of service provided sector (private) (OR = 4.36; 95% CI 1.25—15.20), and less than 5 years' work experience (OR = 2.03; 95% CI 1.01–4.19) increased significantly the likelihood of HCPs' failures in the CMSM.

Table 5 Results of multiple logistic regression analysis to estimate crude and adjusted Odds Ratios (ORs) and 95% confidence Intervals (CIs) for HCP failures in CMSM

Table 6 shows the failure rates following the completion of criterion (a): identification of malaria and/or inquiry about travel history by all healthcare providers (HCPs) and Infectious Disease Specialists. In total, 32.3% and 51.6% of all HCPs and infectious disease specialists met criterion (a). Among them, 47.8% and 25.0% of all HCPs and infectious disease specialists, respectively, failed at least one of the 3 criteria (b, c, and d). Following the completion of criterion (a), no failures were observed in criterion (b)—ordering appropriate diagnostic tests—for infectious disease specialists. Furthermore, over 87% of patients were referred to the healthcare system by infectious disease specialists after meeting criterion (a).

Table 6 Healthcare providers' failures criteria after passing criterion (a); recognition of suspected malaria case; by the overall and infectious disease specialists HCPs

Discussion

This study is the second phase (implementation in the field) of developing and validating the Simulated Malaria Online Tool (SMOT), where malaria transmission has been interrupted. The findings from the initial validation phase have already been published [1]. This study examined the feasibility of SMOT in the field and evaluated various Health care Providers' (HCPs) practices and malaria surveillance programmes in the CMSM, an area where local malaria transmission has been halted since 2005. In the elimination phase and beyond, HCPs’ readiness and practice in appropriate malaria case management are crucial for malaria elimination and preventing re-establishment. The study results show that SMOT can assess HCPs' practices in malaria case management, including identifying febrile patients suspected of malaria, making a diagnosis, providing appropriate treatment, and promptly reporting cases. The study found that SMOT could identify HCPs' practices in malaria case management, such as recognizing a febrile patient suspected of malaria, making a diagnosis, providing appropriate treatment with first-line anti-malarial drugs, and prompt reporting in a ‘‘malaria-free’’ context or those soon to receive that certification.

In the validation study of SMOT by Azizi et al. [1], it was compared with a simulated patient, considered as the gold standard, to simulate a real encounter of healthcare providers with a febrile patient suspected of malaria. The failure proportions (criteria a to d) of SMOT in the present study were found to be similar to those in the study by Azizi et al. when compared with the simulated patient as the gold standard. No significant differences were observed in the failure proportions between SMOT and the simulated patient. These findings indicate the feasibility of using SMOT to assess healthcare provider practices related to CMSM in countries with low or no malaria transmission, aiming to prevent the reintroduction of malaria.

The failure rate was 83% overall. Most failures were related to criterion (a), which involved not asking about travel history and thus missing the possibility of malaria. A smaller proportion of failures was also seen for criterion (b), which involved misdiagnosing malaria.

Furthermore, the study findings emphasized specific categories of healthcare providers and facilities where intervention is urgently needed to enhance case management. The evidence from this study can also aid in achieving and expediting malaria elimination certification from the WHO. Despite the crucial need for preparedness and attentiveness among providers to prevent the resurgence of malaria in countries undergoing elimination, there has been a lack of emphasis on assessing provider performance. This concern has also been underscored in meta-analysis studies [2, 3, 9]

Nevertheless, in this study, when the performance of healthcare providers (HCPs) was assessed after meeting criterion a (malaria recognition), their performance notably improved, particularly among infectious diseases specialists. The overall failure rate was 47.8% for all providers post meeting criterion a, showing a 25% decrease for infectious diseases specialists. Among infectious diseases specialists, following the fulfillment of criterion a, the failure rate in diagnosis criterion (b) dropped to zero, and improvements were also observed in other criteria.

Besides, findings revealed that following the identification of malaria, 76% of all healthcare providers and 87.5% of infectious disease specialists reported consulting with specialized levels of the healthcare system. This discovery is crucial as it ensures that when malaria is suspected, subsequent actions such as diagnosis, treatment, and prompt reporting are typically carried out accurately.

Since 2005, there has been no local transmission in the study area. The sensitivity, knowledge, and effective management of malaria cases by HCPs appear to have diminished. Another factor contributing to the high failure rates could be that the study was carried out during the COVID-19 pandemic, when HCPs were mainly focused on this particular disease, presenting significant challenges for malaria control programmes [14].

Therefore, one reason for the high failure rates of HCPs is the lack of consideration for the health system's performance beyond just HCPs. It is anticipated that many failures occurring post-malaria diagnosis are linked to the health system's effectiveness, and some providers may opt to refer patients to the health system initially to ensure proper management and care for malaria cases [14, 15].

The final analysis, based on multiple logistic regression and adjusting for potential confounders, revealed that non-specialist healthcare providers, private health service delivery sector, and HCPs with less than 5 years of work experience increased the risk of failure in the CMSM. Among specialists, the failure rate was the lowest, while personal and private clinics had the highest failure rate. These results underscored the effective performance of public (governmental) health facilities in the national malaria control program. The lower failure rate in public specialized clinics can be attributed to the presence of specialist doctors, as their failure rate was the lowest compared to other healthcare providers.

Consistent with the current study findings, Selemani et al. in Tanzania[16] found that HCPs with work experience of more than 3 years were more successful in case management. Similar to the present study, another study also showed that the type of HCP is one of the most significant variables in the correct management of malaria cases. Meta-analysis studies have also shown that the experience and work history of HCP, the type of HCP, the type of health facility providing malaria services, and the expectations and compliance of patients are considered to be the most important effective factors in the correct management of malaria [17].

Strengthen and limitation

Evaluation of healthcare providers' actual performance requires real cases in the field. However, in areas where malaria transmission has been interrupted, real cases are scarce. To address this issue, the study utilized SMOT to simulate real-life scenarios. Although the non-response rate was low in this study (11%), field experiences suggest that non-response rates are typically high among busy, experienced HCPs with extensive work histories [18]. This may slightly elevate the likelihood of failure compared to real-life situations.

Conclusion

Findings confirmed the feasibility and implementation of SMOT in settings where local malaria transmission has been interrupted or malaria transmission is low. Healthcare providers' adherence to appropriate malaria case management, especially in detecting suspected malaria cases, needs improvement. Meanwhile, implementing and evaluating healthcare providers' practices using SMOT is crucial in preventing malaria re-establishment.

Appendix 1

See Fig. 1

Fig. 1

Flowchart of SMOT tool for detecting failures in CMSM (S1–12: Stages)