Quality of malaria case management at outpatient health facilities in Angola

Background Angola's malaria case-management policy recommends treatment with artemether-lumefantrine (AL). In 2006, AL implementation began in Huambo Province, which involved training health workers (HWs), supervision, delivering AL to health facilities, and improving malaria testing with microscopy and rapid diagnostic tests (RDTs). Implementation was complicated by a policy that was sometimes ambiguous. Methods Fourteen months after implementation began, a cross-sectional survey was conducted in 33 outpatient facilities in Huambo Province to assess their readiness to manage malaria and the quality of malaria case-management for patients of all ages. Consultations were observed, patients were interviewed and re-examined, and HWs were interviewed. Results Ninety-three HWs and 177 consultations were evaluated, although many sampled consultations were missed. All facilities had AL in-stock and at least one HW trained to use AL and RDTs. However, anti-malarial stock-outs in the previous three months were common, clinical supervision was infrequent, and HWs had important knowledge gaps. Except for fever history, clinical assessments were often incomplete. Although testing was recommended for all patients with suspected malaria, only 30.7% of such patients were tested. Correct testing was significantly associated with caseloads < 25 patients/day (odds ratio: 18.4; p < 0.0001) and elevated patient temperature (odds ratio: 2.5 per 1°C increase; p = 0.007). Testing was more common among AL-trained HWs, but the association was borderline significant (p = 0.072). When the malaria test was negative, HWs often diagnosed patients with malaria (57.8%) and prescribed anti-malarials (60.0%). Sixty-six percent of malaria-related diagnoses were correct, 20.1% were minor errors, and 13.9% were major (potentially life-threatening) errors. Only 49.0% of malaria treatments were correct, 5.4% were minor errors, and 45.6% were major errors. HWs almost always dosed AL correctly and gave accurate dosing instructions to patients; however, other aspects of counseling needed improvement. Conclusion By late-2007, substantial progress had been made to implement the malaria case-management policy in a setting with weak infrastructure. However, policy ambiguities, under-use of malaria testing, and distrust of negative test results led to many incorrect malaria diagnoses and treatments. In 2009, Angola published a policy that clarified many issues. As problems identified in this survey are not unique to Angola, better strategies for improving HW performance are urgently needed.


Background
Malaria is a major cause of morbidity and mortality in sub-Saharan Africa, and prompt treatment with an effective anti-malarial is a pillar of malaria control. For decades, as most health facilities in Africa did not perform diagnostic testing, uncomplicated malaria was typically managed with a syndromic approach in which all patients with febrile illness were treated with chloroquine. Since 2000, in the face of increasing resistance to chloroquine and other commonly-used anti-malarials, 45 African countries changed their case-management policies to recommend artemisinin-based combination therapy (ACT) as the first-line treatment for uncomplicated malaria [1,2]. To reduce unnecessary ACT use, malaria diagnostic testing with microscopy or rapid diagnostic tests (RDTs) is often recommended.
Despite substantial investments by donors, African governments, and other partners to support these new diagnostic and treatment policies, remarkably little is known about how well the policies have been implemented. Published studies of health worker adherence to ACT policies have only been done in three low-income countries (Kenya, Uganda, and Zambia) [3][4][5][6][7][8][9]; and use of testing was evaluated in only two (Kenya and Zambia) [5][6][7]. Results from these studies are concerning. The proportion of patients needing ACT who received it was often low (range: 26-64%), as was the proportion of patients needing testing who were tested (range: 27-56%). Notably, one study from an upper-middle income country (South Africa, where malaria transmission is very limited and health facilities in malaria endemic areas are staffed with personnel from the malaria control programme to perform testing) [1] found that all patients with a febrile illness were tested and all test-positive patients were treated with anti-malarials.
Although malaria transmission, malaria control programs, and levels of socioeconomic development vary widely across Africa, evaluations of real-world implementation of ACT in specific settings can reveal programmatic strengths, weaknesses, and lessons that might benefit other parts of the continent. This report presents such an evaluation in Huambo Province, Angola, where the scaleup of ACT and diagnostic testing was supported by the U.S. President's Malaria Initiative [10]. First, a narrative is provided that describes how the ACT policy was implemented. Second, a survey is presented that assessed health facility and health worker readiness to manage malaria cases and the quality of malaria case management.

ACT implementation
Angola's ACT policy was adopted in 2004, although as early as 2003 ACT had been supplied and short trainings conducted in some health facilities in Huambo. In Febru-ary 2006, the National Malaria Control Programme (NMCP) began disseminating the policy with a five-day training course for 20 trainers from across Angola, including one from Huambo. While many trainings were held over the next several years, it is notable that the ACT policy document was only finalized in 2008 and published in 2009 [11]. Furthermore, in 2007 (when the survey described in this report was being planned), the draft policy document was somewhat ambiguous on malaria diagnosis [12]. The document defined a malaria case as anyone with fever and a positive test for Plasmodium. It then had a "clinical case" section that listed signs and symptoms of uncomplicated malaria without further explanation. The document stated that malaria cannot be ruled out in patients with a negative test and no other cause of fever and that the World Health Organization (WHO) recommends that malaria can be diagnosed among children < 5 years old (under-5s) based on clinical findings without laboratory testing. While these statements provided general guidance on diagnosing malaria, they left some questions. For example: 1) should under-5s with fever be tested?; 2) how should non-fever signs and symptoms be used to decide which patients should be tested?; and 3) in patients with fever and a negative test, which non-malaria causes of fever should be ruled out before treating with an anti-malarial? (Note that the recently published policy clarifies many of these issues-see details below [11].) In Huambo Province, the recommended ACT was artemether-lumefantrine (AL). In 2007, in response to the ambiguous policy, local malaria control staff developed training materials that were based on the draft NMCP policy document, as well as WHO guidelines when the NMCP policy was unclear. However, local staff did not want to be seen as making their own policy. Therefore, although the training materials provided some additional precision on what health workers should do during consultations, they were still not a comprehensive case-management guideline. From August-September 2006, the training materials were used in 3-day courses for health workers from 16 health facilities. From January-November 2007, 570 staff throughout the province were trained. Courses typically lasted three days, involved clinicians, pharmacists and laboratory technicians, and included instruction on RDT use. Participants were expected to share the information with others in their facilities (i.e., informal, cascade training), although there was a multiyear plan to train additional health workers with the formal three-day course. A multi-year plan was needed because the public sector employs unusually large numbers of health workers (after Angola's civil war, many nurses from the opposition faction were integrated into the health system). Scale-up activities were conducted with substantial external assistance, including staff from the NMCP, WHO, The MENTOR Initiative (a non-governmental organization [13]), the U.S. Agency for International Development, and the U.S. Centers for Disease Control and Prevention.
In September 2007 (a time when the availability of malaria testing was still quite limited in Angola), the NMCP Director modified the ACT policy to be in line with WHO guidelines [14]: under-5s with suspected malaria in areas with moderate or high transmission did not need testing and could be presumptively treated for malaria. The change was announced during a meeting, but not incorporated into a published policy until 2009 [10]. Interestingly, one year after the September 2007 announcement, partners sometimes still had different understandings of what presumptive treatment meant (treat all under-5s with suspected malaria versus only treat if no other cause of fever was found). The recently published policy specifies that under-5s with a febrile illness from areas with moderate or high malaria transmission should be treated for malaria without testing [11]. Other issues clarified by the published policy include the definition of a confirmed malaria case, how patients should be managed if malaria testing is not available, and the possibility that patients with cerebral malaria can have a negative test.
AL and RDTs were usually first supplied with the trainings. Subsequently, the pharmaceutical management plan called for monthly deliveries to health facilities. In practice, the time period varied. For several months before the survey described in this report, commodities were not supplied because of transportation difficulties from the central warehouse in Luanda, Angola's capital. However, commodities were delivered shortly before the survey.
Regarding other ACT implementation activities before the survey, AL-related supervision and microscopy training were just beginning. Regarding user fees, the policy calls for free ACT and RDT testing at public facilities.

Study setting
Huambo Province, in the highlands of central Angola, has meso-endemic, stable malaria transmission [15]. Transmission peaks from November-April, with Plasmodium falciparum causing > 90% of infections. The population (2.3 million) is impoverished, and agriculture is a primary source of economic activity [16]. About half (45%) live in rural areas; and there is one urban center, the city of Huambo. During Angola's long civil war , Huambo was particularly hard hit. Much of the health infrastructure was left severely damaged or destroyed, and many land mines remain.

Study design and inclusion criteria
We conducted a cross-sectional cluster survey from October-November, 2007. A cluster was defined as all patient consultations performed in an eligible health facility during regular working hours (Monday-Friday, 8 am-3 pm) during the survey period. Facilities were eligible if they were a public or private facility in Huambo Province that provided outpatient care and where the ACT policy had been implemented (i.e., at least one health worker trained on the policy and ACT delivered anytime in the past). Fifty-seven eligible facilities were identified (eight hospitals, 48 health centers, and one health post); all were government-run. Health workers were eligible if they performed outpatient consultations with sick patients at an eligible facility. Inclusion criteria for outpatient consultations were any initial consultation for patients seeking care for any illness at eligible facilities during regular working hours. Initial consultation meant the patient's first visit to the facility for the current illness.

Sample size and sampling
To minimize costs, a sample size of 30 health facilities was chosen, which is the smallest number of clusters that many experts advise for cluster surveys [17]. Assuming 10% of facilities would be permanently closed or inaccessible, the total sample size was 33 facilities. The list was ordered by facility type (hospitals, health centers, and health posts) and municipality (i.e., district), and 33 facilities were selected with systematic sampling. Systematic sampling was used to choose dates for each facility visit.
Survey teams attempted to include all initial consultations using the "follow-the-patient" approach (see below). However, in high-volume clinics, not all consultations could be included. Surveyors reviewed patient registers and averaged patients in the preceding 5 weekdays to estimate daily caseload. Estimated caseload determined the sampling fraction of patients: if < 21 patients were expected, the sampling fraction was 100% (include all initial consultations); if 22-42 patients were expected, the sampling fraction was 50% (every other initial consultation), etc. If not all consultations could be included, patients were selected with systematic sampling (e.g., select every third patient, with the first chosen randomly).

Data collection
Two teams collected data, each consisting of three surveyors (all medical professionals), a laboratory technician, and a driver. Survey organizers closely supervised data collection. For eight days, surveyors practiced survey procedures in a classroom setting and in health facilities not in the sample. Concordance testing assessed observation skills (with role-playing and videos of consultations) and re-examination skills (with hospitalized patients), and training continued until concordance (i.e., percent agree-ment between surveyors and a gold standard determined by survey organizers) was at least 90%; concordance was typically about 96%.
Data were collected with standardized forms written in Portuguese. Questions for patients were translated into Umbundu, the local language and back-translated to verify accuracy. Interviews were conducted in the language that subjects were most comfortable speaking.
Dates of survey visits were not announced in advance to health facility staff. During visits, surveyors arrived before 8 am. Teams met with facility directors and health workers performing consultations to introduce themselves, describe the visit's purpose, and request verbal consent from health workers.
As ill patients arrived at facilities, drivers identified patients coming for an initial consultation and gave these patients a card with a sequential identification number. The cards were used to sample patients and confirm the total number of initial consultations that day. Surveyors met selected patients and requested verbal consent. Surveyors silently observed consultations with a checklist to record health worker practices. After the consultation, surveyors asked health workers for the patient's diagnoses and prescribed treatments. Surveyors then followed patients through all other parts of the facility visit (e.g., laboratory, pharmacy) and silently recorded malariarelated treatment instructions and counseling messages. When patients were ready to leave the facility, surveyors conducted an interview and focused re-examination in a room out of view of the health facility staff to avoid influencing routine practices. Dispensed medications were recorded. If surveyors identified a patient who had not been adequately treated for malaria or any other serious illness, they provided treatment free of charge.
Non-pregnant patients ≥ 5 years old with suspected malaria (defined below) were tested. Survey team laboratory technicians drew several drops of blood by finger stick with a single-use sterile lancet to make a blood smear and perform a RDT (Paracheck ® , Orchid Diagnostics, Mumbai, India). Blood smears were air dried in the field; after the survey, they were stained with Giemsa and read by an expert microscopist. The RDT was read in the field after 15-25 minutes to ensure that parasitemic patients were treated with an anti-malarial. The "old" (pre-September 2007) NMCP guidelines recommended testing for all patients with suspected malaria, however the survey did not test under-5s and pregnant women with suspected malaria. The reasons were to minimize unnecessary patient discomfort and the time patients spent with surveyors. Due to a misunderstanding, it was thought the policy stated that these two patient groups should be treated with an anti-malarial regardless of the test result. Therefore, as testing did not seem to affect the treatment, these patients were not tested.
At the end of the visit, surveyors interviewed health workers to obtain information on training, supervision, and knowledge of malaria case management. Surveyors also assessed the facility's equipment and drug stocks.

Definitions
Defining the standard for assessing the quality of malaria testing, diagnosis, and treatment was challenging because: 1) policy and training materials sometimes lacked precision; 2) the survey protocol did not include testing for all patients who should have been tested; and 3) the NMCP policy changed just before the survey. Therefore, using NMCP guidelines, an analysis algorithm was developed with enough precision to analyze case-management quality in our sample of consultations ( Figure 1). The algorithm was based on the "old" pre-September 2007 policy (test all patients with suspected malaria) as implemented in Huambo Province because the "new" policy (announced in September 2007, which recommended presumptive treatment for under-5s) was just starting in Huambo--only three health workers had been trained in the new policy, and they only saw four patients in the analysis. The algorithm incorporated case-management principles actually conveyed to health workers during training and provided disease classifications for the somewhat ambiguous instances in which patients that should have been tested were not and vice-versa.
The analysis algorithm was used to make a "gold standard" determination of the patient's diagnosis and treatment. Case-management quality was assessed by comparing the observed health worker's diagnosis and treatment with the gold standard. The analysis algorithm did not use survey laboratory results; instead, it used information that should have been available to health workers (e.g., results of microscopy and RDTs ordered by health workers). This approach is best for evaluating adherence to a guideline, as it prevents classifying health worker practices as erroneous if laboratory results from the health facility did not match survey laboratory results. Survey laboratory results were used in separate analyses. The analysis algorithm used clinical signs and symptoms from patient interviews and re-examinations performed by surveyors.
As per the Huambo training materials, suspected malaria was defined as either fever (by history or measured axillary temperature > 37.5°C) or at least three of the following: headache, joint pain, chills, sweating, anaemia, cough (for children only), anorexia, fatigue, vomiting, or diarrhoea. Uncomplicated malaria was defined as malaria (determined by the analysis algorithm) without signs of severe illness (Figure 2), and complicated malaria was defined as malaria with at least one sign of severe illness. Definitions of non-malaria causes of fever used by the algorithm are shown in Figure 2.
Definitions of treatment quality were based on surveyormeasured patient weights and the NMCP's anti-malarial dosing guidelines [12] (Figure 3). Malaria treatment quality was categorized as: 1) recommended (health worker's prescribed treatment exactly matched the analysis algorithm, including drug type, dosage, and treatment duration), 2) adequate (treatments were not recommended, but still considered life-saving), or 3) inadequate (neither recommended nor adequate). These three categories correspond to correct treatment, minor errors, and major errors, respectively [18].

Analysis
Data were double-entered with SPSS Data Entry version 1.0 (SPSS Inc., Chicago, Illinois) and validated against paper questionnaires. Analyses were conducted with SAS version 9.1 (SAS Institute, Inc., Cary, North Carolina). Weights (for patient-level indicators only) were the product of two components: (1/patient selection probability) × a non-response adjustment (number of eligible patients selected for enrollment [i.e., patients enrolled and included in the analysis + selected patients missed by surveyors + refusals + withdrawals + patients lost to followup]/patients included in the analysis).
Descriptive analyses of patient and health worker-level indicators were performed with the SURVEYFREQ procedure, which uses the Taylor expansion method to account for cluster sampling and unequal analysis weights. Health Algorithm used to analyse the quality of malaria diagnosis and treatment, as it was applied in outpatient health facilities, Hua-mbo Province, Angola Figure 1 Algorithm used to analyse the quality of malaria diagnosis and treatment, as it was applied in outpatient health facilities, Huambo Province, Angola. a Defined as either fever (history of fever or axillary temperature > 37.5°C), or at least 3 of the following: headache, joint pain, chills, sweating, anaemia (palmor pallor), cough (applies to children only), anorexia, fatigue, vomiting, or diarrhoea. b Error (health worker's decision did not follow policy documents and training materials). c Result not available on the day of the consultation, patient asked to return the next day (this only occurred for 2 patients). d This part of the algorithm was not explicitly included in policy documents or training materials; however, the decision could be logically inferred from policy documents or training materials. e Defined as cerebral dysfunction, cerebral malaria, disseminated intravascular coagulopathy, haemoglobinuria, hepatic dysfunction, hyperthermia, pulmonary oedema, renal insufficiency, severe anaemia, or shock. For details, see Figure 2. f Do not treat for malaria now; wait until result is ready and treat only if test is positive. g Defined as dysentery, hepatitis, influenza-like illness, measles, otitis, pneumonia, or urinary tract infection. For details, see Figure 2. worker indicators were not weighted, as selection probabilities were equal. Patient-level indicators were weighted, unless otherwise noted. As the sample size of health facilities (n = 33) was a relatively large proportion of all facilities in the sampling frame (N = 57), confidence intervals (CIs) for health facility-level indicators were adjusted with the finite population correction factor--i.e., for a proportion, p, the standard error = {square-root [p(1 -p)/n]} × {square-root [(N -n)/(N -1)]}. If p = 0% or 100% (and thus the estimated standard error would be zero), CIs were estimated using the exact binomial method.
Logistic regression modeling was performed using the SURVEYLOGISTIC procedure, which also uses the Taylor expansion method to address clustering and weighting. All models were weighted. Variables with a univariate pvalue < 0.15 were included in a multivariate model (the full model). Reduced models were tested by removing variables with a multivariate p-value > 0.10. Variables with a multivariate p-value > 0.10 were not removed if odds ratios of other variables changed by > 20%. To assess confounding by excluded variables (univariate p-value ≥ 0.15), we added them to the reduced model one at a time and retained them if odds ratios of other variables changed by > 20%. Hypothesis testing and CI estimation were done with an alpha of 0.05.

Enrollment
Survey teams visited all 33 sampled health facilities as scheduled. In these facilities, 100 health workers performed outpatient consultations; 93 (93.0%) were interviewed, and seven left before interviews could be administered. No health worker refused to participate. Of Signs of severe illness in the guidelines and how they were operationalized in the survey 1. "Cerebral dysfunction and cerebral malaria" in the guidelines was defined as: history of convulsions, .
Other causes of febrile illness and how they were operationalized in the survey 1. "Dysentery" in the guidelines was defined as: diarrhea and bloody stools. 2. "Hepatitis" was defined as: jaundice.
Note. The following signs of severe illness in the national policy were excluded in the survey analysis because testing could not be performed in the field (most health facilities also lacked the tests): hyperparasitemia (>100,000 parasites/mm 3 ), hypoglycemia, hyponatremia, and metabolic acidosis.
the 93 workers interviewed, 64 treated at least one included patient; and of the seven workers missed, six treated at least one included patient. Thus, patient-level indicators of case-management quality reflected the practices of 70 health workers (64 interviewed, six missed).
Altogether, 778 patients came for initial consultations; 389 of whom were selected for inclusion. Of these 389 patients, 177 were included, 170 were missed (i.e., no surveyor was available to begin the "follow the patient" procedure), 38 refused, three withdrew after enrollment, and one patient left the facility before completing all survey steps. Although many selected patients were not surveyed (i.e., 170 missed + 38 refusals + three withdrawals + one lost, or 212 patients), most (170/212, or 80.2%) were missed patients. These missed patients were unlikely to have been much different from included patients, as they were essentially missing at random. If missed patients are excluded, the "participation rate" (i.e., the proportion of patients approached for consent who were included) was 80.8% (177 included/[177 + 38 refusals + three withdrawals + one lost]).  ask about fever and measure temperatures for all patients; solely relying on chief complaints could lead to many missed cases of febrile illness.

Health facility characteristics
As defined in Huambo, 136 (77.8%) patients had suspected malaria (fever or three non-fever symptoms), half (48.0%) of whom had signs of a non-malaria illness-most commonly, respiratory infections (  Figure 1, footnote 7).

Patient characteristics: quality of clinical assessment and use of diagnostic testing
Clinical assessments were evaluated by estimating the proportion of patients for whom health workers had determined if a given sign or symptom was present. "Determined" meant the worker was exposed to the information by any means (e.g., spontaneously offered by the patient, provided by the patient in response to a question, or obviously evident Multivariable modeling identified two factors significantly associated with testing (Table 4). First, the odds of testing among patients (needing testing) seen by health workers with caseloads < 25 patients/day were 18-fold greater than for those seen by workers with higher caseloads. Based on unadjusted results, the proportion of patients tested by health workers with lower and higher With the analysis algorithm as the standard, 66.1% of health workers' malaria-related diagnoses were correct, 20.1% were minor errors, and 13.9% were major (potentially life-threatening) errors (Table 5). With survey microscopy as the standard: a) for the one microscopy-positive case of uncomplicated malaria, the health worker's diagnosis was uncomplicated malaria (correct); b) for the one microscopy-positive case of complicated malaria, the worker's diagnosis was uncomplicated malaria (major error); and c) for the 70 microscopy-negative cases, workers correctly diagnosed no malaria in 34 cases (48.6%, unweighted) and incorrectly over-diagnosed malaria in 36 cases (51.4%, unweighted).

Laboratory assessment
Twenty-two staff from a convenience sample of 20 facilities were assessed as they performed RDTs. All the correct steps (one drop of blood and six drops of buffer solution) were followed in only half (11/22) of observations. The most common error was using too little buffer solution (two to five drops). The amount of blood was usually correct. Most staff knew that one should wait 15 minutes or a little longer, although one-third might have thought that waiting times were < 15 minutes or were unsure. Additionally, in some facilities, test results were reported in batches instead of reporting individual results as they were ready. This practice caused unnecessarily long waiting times (≥ 1 hour) for some patients (RDT results should be available in < 30 minutes).

Patient characteristics: quality of treatment and counseling
Among all 177 patients, 61.4% of prescribed malariarelated treatments were correct, 22.3% were minor errors, and 16.3% were major (potentially life-threatening) errors ( Table 6). The most common errors were prescribing no anti-malarials for patients with uncomplicated malaria and prescribing AL for patients without malaria. Errors such as under-dosing AL and using ineffective antimalarials were uncommon.
Among the 59 patients with malaria, the quality of prescribed treatments was lower: only 49.0% were correct, 5.4% were minor errors, and 45.6% were major errors ( Table 6). In an analysis of treatment quality from the patient's perspective (i.e., patient left the facility with the anti-malarial in hand and demonstrated knowledge on administering it at home), only 27.1% of patients received recommended care, 5.4% received adequate (but not recommended) care, and 67.5% received inadequate care.
An analysis of 62 patients prescribed AL (whether or not it was indicated) revealed that health workers almost always dosed AL correctly, but the quality of counseling was mixed (Table 7). Nearly all patients received the correct dose (95.1%) and complete dosing instructions (88.2%). However, only 60.9% of patients could repeat all the dosing instructions. Furthermore, few patients were given the first dose during the consultation (10.7%) or advised to take the medicine with food (31.3%).

Graphical pathway analysis
A graphical pathway analysis was performed to link results of individual steps of the case-management process and identify strengths and weaknesses of health worker practices. For simplicity, percentages are unweighted. Among the 40 patients without febrile ill- ness/suspected malaria (and thus no malaria), we found that a large majority of patients (35/40, or 87.5%) were not tested for malaria, nearly all (34/35, or 97.1%) of these untested patients were not diagnosed with malaria; and of the 34 patients without a malaria diagnosis, none were treated with an anti-malarial (Figure 4, steps along the bottom of the figure). In other words, most patients were managed correctly at all points in the case-management process.
Among the 78 patients with febrile illness/suspected malaria but no gold standard malaria diagnosis, many (36/78, or 46.2%) were not tested, even though they should have been ( Figure 5). Among the 41 patients with a negative test result, over half (24/41, or 58.5%) were diagnosed with malaria. Nearly all (33/36, or 91.7%) patients diagnosed with malaria were prescribed an effective anti-malarial; and of the 42 patients without a malaria diagnosis, none were treated with anti-malarials.
Among the 59 patients with febrile illness/suspected malaria and a gold standard malaria diagnosis, most (42/ 58, or 72.4%) were not tested, even though they should have been ( Figure 6). Of 26 patients who did not receive anti-malarials (major error), none had been tested. All 17 patients with a positive malaria test were diagnosed with malaria. A large majority of patients (31/35, or 88.6%) diagnosed with malaria were prescribed effective antimalarials; and of the 24 patients without a malaria diagnosis, none were treated with anti-malarials.
In summary, many patients who should have been tested were not, which led to many incorrect diagnoses.  Major error (health worker incorrectly "under-diagnosed" complicated malaria as uncomplicated malaria, or under-diagnosed uncomplicated malaria as no malaria) 25 13.9 (8.0-19.7) CI = confidence interval; NC = not calculated. a The "gold standard" malaria diagnosis (against which health worker diagnoses were compared) was defined by applying the analysis algorithm ( Figure 1) to patient clinical signs and symptoms (assessed by surveyors, but information that should have been available to observed health workers) and laboratory data available to observed health workers (i.e., not the survey team's laboratory results). b Health workers' diagnoses of the 24 patients were gastrointestinal illnesses (n = 10; e.g., gastritis, intestinal parasites, and dysentery), respiratory illnesses (n = 5; e.g., bronchitis), skin problems (n = 2; e.g., scabies and skin boils), and other (n = 7; e.g., dental caries, trauma, chicken pox, and malnutrition).
nosed with uncomplicated malaria, a large majority (83.6%) received AL (usually correctly dosed); and among 102 patients without a diagnosis of malaria, nearly all (97.3%) received no anti-malarial treatment.

Design effects and intraclass correlation coefficients
Design effects and intraclass correlation coefficients ( ) were examined for 24 key patient-level indicators of casemanagement quality related to patient assessment, diagnosis, treatment, and counseling. Weighted design effects ranged from about 1.0 (no correlation) to 3.4 (moderate correlation); the median was 1.7. Weighted values of ranged from just under zero to about 1.0; the median was 0.3. Design effects for the two indicators of correct malaria treatment were close to one (i.e., 1.1 and 1.6), which reflects relatively little correlation.
As heterogeneous analysis weights tend to increase design effects and decrease precision [19], an unweighted analysis was performed to quantify the effect of heterogeneous weights (final weights ranged from 1.7-40.6). As expected, unweighted design effects and were usually lower than values from the weighted analysis. An examination of the ratio of weighted design effects to unweighted design effects revealed a median ratio of 1.48. In other words, heterogeneous weights typically increased design effects by 48%, which widened CIs by 22% (i.e., square-root [1.48] = 1.22) (details available on-line) (see Additional file 1).

Discussion
This survey evaluated about 14 months of activities to scale-up Angola's ACT policy in a province with weak infrastructure that was the focus of a well-funded malaria

Quality of malaria treatment for the 59 malaria cases (quality in terms of antimalarials obtained by patients and patient recall of treatment instructions)
Patient left the health facility with the recommended anti-malarial and knowledge of how to administer the drug at home AL = artemether-lumefantrine; CI = confidence interval. a The "gold standard" malaria diagnosis (against which health worker treatments were compared) was defined by applying the analysis algorithm ( Figure 1) to patient clinical signs and symptoms (assessed by surveyors, but information that should have been available to observed health workers) and laboratory data available to observed health workers (i.e., not the survey team's laboratory results). b Quality in terms of anti-malarials prescribed by health workers. No error means that patients received recommended treatment in exact accordance with guidelines (malaria cases treated with the recommended anti-malarial with the recommended dosage, and non-malaria cases received no anti-malarial treatment). Minor error means that malaria cases received non-recommended, but still life-saving, anti-malarial treatment (either an overdose of a recommended anti-malarial, or an adequate dose of a non-recommended anti-malarial); and non-malaria cases received unnecessary anti-malarial treatment that was unlikely to cause serious harm. Major error means that malaria cases did not receive life-saving treatment (no anti-malarial, an ineffective anti-malarial, or an under-dosed anti-malarial). c The 2 minor errors were: 1 case of uncomplicated malaria treated with correctly dosed quinine, and 1 case of uncomplicated malaria in a child < 5 kg treated with AL (with a dosage appropriate for a child 5-14 kg). d The 30 major errors were: 26 cases of uncomplicated malaria not treated with anti-malarials, 2 cases of uncomplicated malaria treated with ineffective anti-malarials (1 treated with amodiaquine, 1 treated with sulfadoxine-pyrimethamine), 1 case of uncomplicated malaria treated with underdosed AL, and 1 case of complicated malaria treated with underdosed AL.
control initiative. As the survey occurred in the middle of a multi-year implementation plan and as training and supervision have continued since the survey, the current situation is probably better than what is reflected in this report. However, the programmatic strengths and weaknesses we identified were notable, and the results reveal problems that might exist today. Moreover, as the setting is typical of many parts of the developing world, the broader lessons might be relevant to other low-income countries.
The first main finding was related to the fact that when a policy is ambiguous, overly complex, communicated imprecisely, or changed frequently, confusion can ripple throughout the health system. In Angola, in 2007, parts of the NMCP policy lacked clarity. In Huambo, local staff responded by adding details in training materials to fill gaps. However, despite these good intentions, the result was that nearly everyone involved in scale-up activities had a different interpretation of the guidelines. Additionally, the complex definition of suspected malaria likely contributed to the finding that no health worker knew the full definition. These observations underscore the impor-tance of developing policies and training materials that are clearly written and as simple as possible. Donors and other development partners should insist that these fundamental programmatic building blocks be in place before large-scale implementation activities are undertaken. Furthermore, the statement that patients might have malaria even with a negative test, which appears in policy statements from Angola and other countries [5,20], should be replaced with more specific guidance. At the international level, experts can assist by helping to answer the question: precisely who should be treated when a test is negative? Without specific guidance, health workers are justified in ignoring negative test results; and if instances exist when test results can be ignored, some workers might conclude that testing has limited value and can be skipped.
The second finding was that many case-management elements were in place, although some additional strengthening was needed. All facilities had AL in-stock, the ability to perform malaria testing, and AL-trained health workers. Although scale-up was still ongoing, aspects of health facility preparedness requiring improvement during the survey were: oral and injectable quinine were frequently not in-stock, few facilities had all AL blister packs continuously in the previous three months, one-quarter of patients were seen by health workers without AL training, and supervision on AL use was uncommon and often did not involve observation and feedback. Some of these deficiencies have been observed elsewhere [9]. Additionally, the health worker knowledge assessment revealed two major gaps: no one knew the full definition of whom to test (although most could identify which patients needed testing in case scenarios), and most did not trust negative test results. These findings illustrate the importance of basic program management, in terms of stocking essential medicines, scheduling trained health workers to perform consultations, and providing clinical supervision as training occurs. Although it might not seem directly linked to lowering malaria mortality, strengthening managerial capacity should be a priority of today's well-funded disease control initiatives.
The third main finding was related to case-management quality. As many indicators are needed for a comprehensive description, we constructed a causal diagram to display the key practices and show the causal chain of observed problems (Figure 7). Clearly, this diagram is simplistic, as it omits many fundamental environmental factors of potential importance, such as low salaries, poor motivation, weak infrastructure, and perceived patient expectations. As illustrated at the top of Figure 7, the casemanagement process begins with patient assessment. Except for fever history, assessments were often incomplete. Other studies have found similar deficiencies [20,21].
Regarding malaria testing, the main problem was substantial under-use. We explored testing practices in-depth because testing and trusting the result was the key to making a correct diagnosis and correct diagnosis was the key to prescribing appropriate treatments. Interestingly, the factors significantly associated with testing were caseload and patient temperature (root causes on the left of Figure  7). Similar analyses from Tanzania [22] and Zambia [21] in the pre-ACT era found a variety of factors positively associated with testing: patient age ≥ 15 years, fever, fever history, and longer travel time to clinic. Testing was less likely if patients had a skin problem or single-etiology illness (e.g., respiratory infection alone). These results illustrate the importance of non-intervention environmental factors [23] and could be used to guide future quality improvement activities (e.g., changing schedules to reduce high caseloads, and teaching health workers to test all patients with fever history, not just those with elevated temperatures). AL training had a borderline association with testing; but even ignoring statistical significance, the Graphical pathway analysis of the case-management process for 40 patients without suspected malaria Figure 4 Graphical pathway analysis of the case-management process for 40 patients without suspected malaria. AL = artemether-lumefantrine; HW = health worker. absolute proportion of patients tested by AL-trained workers was low (38.1%). Supervision on AL was not associated with testing, perhaps because it focused on pharmaceutical management rather than clinical practice. It was notable that health worker knowledge was not associated with correct testing, which might explain the modest effect of training. The lack of association between health worker knowledge and practice has been observed in other settings [24][25][26], and this phenomenon should make program managers consider other quality improvement strategies besides training--such as incentives, job aids, and quality management principles to name a few [23].
The patterns of diagnosis and treatment quality were similar to those of testing. Only half to two-thirds of patients were correctly diagnosed and treated. The two most common diagnostic errors were missing malaria cases and over-diagnosing malaria among patients without it. Overdiagnosis can be partly explained by workers' apparent distrust of negative test results, which echoes results of the knowledge assessment. Erroneous diagnoses translated into inappropriate treatments because, in general, workers correctly treated the diagnoses they made. Additionally, health workers rarely administered the first dose of AL during the consultation (perhaps because no food was available), and many aspects of counseling needed improvement--problems found in other settings [4,9]. An analysis from the patient's perspective, which was perhaps the most relevant quality indicator (i.e., patients needing treatment left the facility with the correct medicine and knowledge on administering it at home), found that only about one-third of patients received adequate (i.e., lifesaving) care.
These results point to what might be the most important lesson of our evaluation and other similar studies [3][4][5][6][7][8][9]: commonly-used scale-up strategies such as providing commodities, cascade training, job aids, and a little supervision of uncertain quality, seem to have only a moderate effect. Moreover, implementation of these strategies is time-consuming. For malaria control initiatives to achieve better results faster, new approaches are needed.
There were several important positive findings related to treatment. For example, it was reassuring that nearly all patients that health workers diagnosed with malaria were prescribed AL at the correct dose and provided with accurate dosing instructions. Also, there was little use of ineffective and non-recommended anti-malarials--a problem observed elsewhere [3,6,27].

Methodological lessons
This survey provided several methodological lessons. First, it was challenging to develop the definition of the gold standard against which case-management quality Graphical pathway analysis of the case-management process for 78 patients with suspected malaria but no gold standard malaria diagnosis Figure 5 Graphical pathway analysis of the case-management process for 78 patients with suspected malaria but no gold standard malaria diagnosis. AL = artemether-lumefantrine; HW = health worker. The second lesson concerned the "follow the patient" sampling method. This method allows surveyors to observe all parts of the patient's clinic visit and might reduce loss after enrollment. However, clearly, many patients were missed because of an insufficient number of surveyors and because health workers did not always see patients in the order they arrived at the facility (e.g., if patient 1 was selected for inclusion but not immediately seen by a health worker, a surveyor had to wait for patient 1's consultation, during which time other sampled patients might be seen by health workers). Also, the method led to widely varying analysis weights, which reduced precision. Another approach, which is probably better, is to have surveyors at fixed stations (e.g., observers spend the entire visit observing consultations). The disadvantage of the fixed-station method is that a small survey team might miss consultations in facilities with several consultation rooms. The solution is to either randomly select consultation rooms (and record selection probabilities, so results can be weighted) or determine in advance the number of consultation rooms in each sampled facility and create a small pool of extra surveyors to work at the larger facilities.
Finally, this survey demonstrated the positive bias that can occur when quality is evaluated only by measuring how well prescriptions match health worker diagnoses. With this approach, quality seemed excellent. Most (83.6%) patients whom workers diagnosed with malaria received AL, and nearly all (97.3%) patients whom workers did not diagnose with malaria were not prescribed anti-malarials. In contrast, when quality was assessed with a gold standard based on expert re-examination, results were much lower.
Graphical pathway analysis of the case-management process for 59 patients with a gold standard malaria diagnosis Figure 6 Graphical pathway analysis of the case-management process for 59 patients with a gold standard malaria diagnosis. AL = artemether-lumefantrine; HW = health worker. a None of these 26 patients had been tested for malaria, although all were seen at health facilities in which testing was available.

Limitations
The survey has several important limitations. First, it only included facilities in which ACT implementation activities had occurred. Therefore, results cannot be generalized to facilities where many patients might seek care--especially government-run health posts and private-sector facilities. Second, the response rate was low because surveyors missed eligible patients, although it seems unlikely that missing these patients introduced a large bias. Third, directly observing consultations could have caused health workers to be more careful than usual [28,29], thus overestimating case-management quality somewhat. Fourth, some patient groups with suspected malaria were not tested. This issue, along with the survey period occurring before the peak malaria season (the rains began late), led to a patient sample size that was too small to evaluate the performance characteristics of microscopy and RDTs adequately. Moreover, by missing the peak malaria season, the parasite prevalence might not be a reliable estimate for characterizing the level of malaria transmission intensity in the province. Lastly, there was some ambiguity in the gold standard diagnosis and treatment for cases in which health workers did not test patients who should have been tested or vice-versa, although the survey's general conclusions were unlikely to have been affected.

Conclusion
Huambo Province is a challenging place to scale-up a new case-management policy that recommends a new drug, emphasizes diagnostic testing, and introduces a new test.
As of late-2007, the scale-up process was well underway, and the survey had important positive findings. However, key gaps were found. In particular, the ambiguous policy, under-use of malaria testing, and distrust of negative test results led to many incorrect malaria diagnoses and treatments. In 2009, Angola published a policy that clarified many issues. As problems identified in the survey are not unique to Angola, better strategies for improving health worker adherence to guidelines are urgently needed. Surveys that assess case-management quality with observations of consultations can provide crucial information for developing local recommendations and guiding scale-up elsewhere.

Abbreviations
ACT: artemisinin-based combination therapy; AL: artemether-lumefantrine (i.e., Coartem®); CI: confidence Causal diagram of the case-management process in outpatient health facilities, Huambo Province, Angola Figure 7 Causal diagram of the case-management process in outpatient health facilities, Huambo Province, Angola.