Implementation of the LLIN campaign

Activities which formed the core basis of the universal LLIN campaign in Ghana included pre-registration of persons and sleeping places, door-to-door distribution of LLINs by volunteers including hang-up activities, and post-distribution ‘keep-up’ behaviour change communication activities.

Trained volunteers registered the number of people and sleeping spaces in households in the districts to benefit from the Hang-Up Campaign. This was used as the basis for the allocation of the LLINs and related items. In the first instance, the number of LLINs was calculated as number of household members divided by two; if this was greater than or less than the number of sleeping spaces then the number of LLINs was adjusted to equal the number of sleeping spaces.

Trained volunteers were informed of the final LLIN allocations to households in their community and collected the LLINs from the nearest pre-positioning site. The volunteers then delivered the allocated number of LLINs to each household. Where they were permitted to enter, they helped the occupants to hang the nets above each sleeping space in the household. Where they were not permitted to enter, they were instructed to provide rope and nails. In both cases the volunteers also delivered behaviour change communication (BCC) messages about use.

The volunteers who conducted the household registration and hang-up were recruited from the communities in which they worked. They relied on local knowledge of community members and especially household heads to identify registered households for LLIN distribution. Each hang-up team (made up of three volunteers) had a member who had participated in the household registration in the area (s) where the team carried out the hang-up.

Study setting

This paper focuses on evaluation of the LLIN Campaign in Brong Ahafo, Central and Western regions, which were selected for pragmatic reasons, based on timing of data collection relative to campaign implementation and funding support. The selected regions cover all three of Ghana’s ecological zones: Central and Western regions are in southern Ghana and include parts of the coastal and forest ecological zones; Brong Ahafo is in central Ghana and includes parts of the forest and savannah ecological zones. There are nevertheless socio-economic and cultural differences between these regions and those in the north or east of the country. Likewise, since the selection of Brong Ahafo, Central and Western regions for this evaluation was not random it is not possible to collate the regional results to produce a statistically representative national average. This would also be hard to interpret as universal campaign implementation was phased across the country, starting in Eastern region in December 2010 and ending in Greater Accra Region in October 2012.

Overview of evaluation design

An uncontrolled before-after design was chosen for this evaluation, with attribution of effects of the LLIN Campaign through collection of data on source of nets owned by households and a thorough process evaluation [7]. A mixed methods approach was taken, involving quantitative pre- and post-campaign household surveys, post-campaign in-depth interviews and focus group discussions with key stakeholders, and a costing analysis.

Because the same implementation strategy for the campaign was used in all ten regions of Ghana, it was not possible to use a randomized controlled trial or even a controlled before-and-after design to allow for the individual components to be evaluated separately. Therefore, the incremental cost effectiveness of the ‘hang-up’ component was estimated using reported variation in the implementation of hang-up activities and LLIN use. To facilitate this comparison, detailed questions on exposure to each element of the campaign process were included in the post-campaign household survey questionnaire.

LLIN distribution in Central and Western regions took place in November-December 2011; distribution in Brong Ahafo was in May-June 2012. The post-campaign household survey was conducted in September-October 2012, approximately 11 months after LLIN distribution in Central and Western regions and five months in Brong Ahafo region.

Collection of costs data

Financial and economic costs were collected from the societal perspective, meaning that direct and indirect costs to both LLIN providers and recipients were incorporated. An ingredients approach was used to identify all resources required to deliver LLINs through a mass universal campaign with hang-up activities [8].

Financial costs were obtained retrospectively from the financial reports and accounts of the implementation partners. Research and evaluation costs were not included. Costs were measured in Ghanaian Cedis (GHC) or United States Dollars (USD), depending on the currency of the original expenditure. Costs in GHC were converted to USD according to the average exchange rate for the year of the expenditure (1 USD equivalent to 1.42 GHC in 2010, 1.52 GHC in 2011, and 1.81 GHC in 2012). All costs were adjusted for inflation and are presented as 2012 USD using the consumer price indices available from the International Monetary Fund [9].

Economic costs recognize that the cost of using resources means that these resources are unavailable for productive use elsewhere, and include costs such as donated goods or volunteered time spent on the intervention. Information on the time that volunteers spent on campaign activities, what they would have been doing if not working on the campaign and estimates of their usual income was collected during focus group discussions. The time spent by GHS personnel and other partners on training and supervision for which they did not receive direct salary support was also valued and included as an economic cost.

Capital goods with an expected lifespan of more than one year were annualized using a discount rate of 3% according to the guidelines of the World Health Organization [10]. The lifespan of cars and motorbikes used in the campaign was estimated to be 5 years, based on information from UNICEF. An average useful lifespan of three years was assumed for the LLINs [3]. Other one-off costs of the campaign were also treated as capital costs, including sensitization, household registration and LLIN distribution and hang-up; essentially these are investments, which are expected to last as long as the useful LLIN lifespan. These costs were annualized across the average LLIN lifespan of three years using a discount rate of 3%. Consistent with the running costs of the UNICEF Ghana office, overheads were included at a fixed value of 7% of all financial provider-level expenditure. Economic costs are presented as the average annual economic cost over the effective lifespan of the LLIN.

Questions on whether the household had made some contribution to the volunteer for hang-up and the value of the contribution were included in the household survey questionnaire, along with questions to investigate the amount of time any member of the household had to wait at home for the LLIN hang up visit.

Measurement of campaign effectiveness

The number of LLINs distributed in each region was obtained from GHS district reports. The effect of the campaign on LLIN ownership and use was measured using household surveys. Pre-campaign baseline data on LLIN ownership and use is provided at regional level by the Multiple Indicator Cluster Survey (MICS) conducted in September-December 2011 [11]. Post-campaign data on LLIN ownership and use was collected in Brong Ahafo, Central and Western regions in September-October 2012 using the same survey design and standardized questionnaire as MICS to allow direct comparison. Additional questions on households’ exposure to each phase of the LLIN Campaign were also included in the post-campaign questionnaire, including whether campaign LLINs were hung by the volunteer or not. Briefly, the post-campaign household survey followed a two-stage cluster sample design. At least 543 households were required in each of the three evaluation regions to give an estimate of the proportion of children under five who slept under an LLIN the night before the survey to within 8% precision (further details to be reported elsewhere; Awini et al., personal communication).

Data processing and analysis was conducted in Stata 12.0, using ‘svy’ commands to ensure confidence intervals were appropriately adjusted according to the design of the surveys. Determinants of post-campaign LLIN use were explored using logistic regression; explanatory variables with a p-value of <0.1 in univariable analyses were included in the final multivariable model.

Cost effectiveness analysis

For each of these outcomes, an incremental cost effectiveness ratio (ICER) was calculated i.e. the additional effect (person using an LLIN or death averted) for additional cost of the LLIN Campaign compared to no campaign.

Sensitivity analyses

The cost analyses involve a number of important assumptions. To investigate the effect of these assumptions on the results, each was varied in turn in one-way sensitivity analyses. Estimates of the useful lifespan of an LLIN vary by setting, therefore a lower value of two years and upper value of five years were explored [3]. These changes in LLIN lifespan were applied to all annualized campaign costs. The financial cost of the LLINs decreased considerably over the period of the LLIN Campaign. Cost estimates of USD 3.25 and USD 4.80 per LLIN used in the sensitivity analysis were median 2012 prices for a rectangular net (as distributed in the LLIN Campaign) or a conical net, respectively [12]. To explore the importance of assumptions regarding the predicted mortality impact of the LLIN Campaign (such as the influence of transmission intensity), the upper and lower uncertainty limits for the pooled estimate of 5.5 child deaths averted per 1,000 children protected by an LLIN were used (95% CI: 3.39, 7.67) [1]. The discount rate was also varied with lower and upper values of 0% and 10% [13].

Estimating the incremental cost-effectiveness of hang-up activities

The study design did not allow direct comparison of a universal LLIN campaign with and without hang-up activities. However, the effect of the hang-up was derived by comparing LLIN use in households in which volunteers had hung-up one or more LLINs with use in households where the volunteer did not hang the net (s). Costs were estimated of a universal campaign without hang-up visits to households. In this alternative scenario, LLIN distribution was conducted from a fixed-point without any household follow-up visits for hang-up: it was assumed that the same volunteers involved in household registration attended a fixed distribution point for three days to deliver nets to recipients; the number of volunteers needed for registration and fixed-point distribution is approximately one-third that for door-to-door distribution. No time was included for house-to-house visits by volunteers. Supervision of distribution was reduced from ten days to three days. All other costs were kept constant.

The effectiveness on LLIN use of this campaign strategy was assumed to be lower, based on data from the post-campaign household survey data which found that LLIN use amongst individuals and children under five living in households where all or some of the nets were hung by a volunteer was significantly higher than LLIN use amongst those living in households where nets weren’t hung by the volunteer.

Thus, the ICER of a campaign with hang-up activities compared to one without was calculated. The sensitivity of this predicted ICER to the assumption about the additional effect of hang up was explored by varying the odds ratio of LLIN use by children under five living in households where nets were or were not hung by a campaign volunteer.

Ethical considerations

Approval for the evaluation was granted by Ghana Health Service. Ethics approval was granted by the Observational/Interventions Research Ethics Committee of the London School of Hygiene & Tropical Medicine. Appropriate authorities were informed in each region and district involved in the evaluation. Individual informed written consent to participate in the study was obtained from heads of households and key informants.

Financial and economic costs of the LLIN campaign

The total financial cost of the campaign from the provider perspective across all three regions was USD 23.85 million (USD 6.78 million in Brong Ahafo region, USD 7.44 million in Central region, and USD 9.63 million in Western region); the average annual economic cost from the societal perspective was USD 10.64 million (USD 3.21 million, USD 3.34 million and USD 4.10 million in each region, respectively) (Additional file 1). LLINs, transport and storage comprise around 80% of the total financial cost (from both the provider and societal perspectives); this is reduced to around 65% of the annual economic cost once the costs associated with volunteer and household time have been included (combined figures provided in Table 1; regional breakdown presented in Figure 1).

	Total financial cost		Average annual economic cost
	USD (2012 prices)	%	USD (2012 prices)	%
LLINs, transport & storage	5.38	79.1	1.90	65.4
LLINs	4.96	72.9	1.75	60.4
Non-LLIN materials	0.36	5.3	0.13	4.3
Transport & storage	0.06	0.9	0.02	0.7
Household registration phase	0.21	3.1	0.17	5.7
Regional planning & training of trainers	0.03	0.5	0.02	0.5
Training of volunteers	0.03	0.5	0.02	0.8
Registration exercise	0.09	1.3	0.10	3.3
Supervision	0.06	0.8	0.03	1.1
LLIN distribution (including hang-up)	0.32	4.8	0.38	12.9
Logistics training	0.05	0.7	0.02	0.7
Training of volunteers	0.10	1.5	0.06	2.1
Hang-up exercise	-	-	0.20	6.9
Supervision	0.17	2.6	0.09	3.3
Information, education & communication	0.17	2.5	0.10	3.4
Social mobilization	0.11	1.6	0.04	1.4
Keep-up planning & supervision	0.01	0.2	0.01	0.5
IEC materials & activities	0.05	0.7	0.04	1.6
Overheads	0.43	6.3	0.15	5.3
Total provider cost	6.51		2.69
Household-level costs	0.29	4.3	0.21	7.3
Contribution to volunteers	0.29	4.3	0.10	3.5
Waiting time	-	-	0.11	3.7
Total societal cost	6.80		2.90

Costs are presented in USD 2012 prices and summarized across all three regions (Brong Ahafo, Central & Western).

Effectiveness of the LLIN campaign

Significant improvements were achieved in LLIN ownership and use in all three regions that can be attributed to the LLIN Campaign. Household ownership of LLINs and use by all individuals, children under five years and pregnant women doubled between pre- and post-campaign household surveys. For example across the three regions, the proportion of households owning at least one LLIN increased from 42.5% pre-campaign to 85.3% post-campaign; the proportion of all individuals and children under five sleeping under an LLIN increased from 21.3% to 54.4% and 33.9% to 63.0%, respectively. Post-campaign LLIN ownership and use was higher in Brong Ahafo than Central and Western regions, although the relative improvements were similar across all three regions (Table 2).

The odds of sleeping under an LLIN were also significantly higher for children living in households with at least one LLIN observed hanging on the day of the visit, those living in households with at least one LLIN per two people and those living in the poorest households (Table 3).

Cost per additional person and per additional child under five using an LLIN

The annual economic cost per additional person using an LLIN following the LLIN Campaign was USD 3.37 in Brong Ahafo, rising to USD 5.59 and USD 7.45 in Central and Western regions, respectively (Table 2). The higher costs in Central and Western regions compared to Brong Ahafo are due to a combination of the higher unit costs per LLIN delivered and the lower number of additional individuals using an LLIN as a result of the campaign.

A similar pattern was seen for the cost per additional child under five sleeping under an LLIN due to the LLIN Campaign: the economic cost in Western region (USD 47.86) was almost double that in the Brong Ahafo region (USD 27.68), with the cost in Central region somewhere in between (USD 41.21) (Table 2). This incremental approach assumes that if there had not been a mass universal campaign then LLIN use would have remained at pre-campaign levels and that there would be no costs associated with maintaining this level of LLIN use.

Cost per additional death averted

At the LLIN use levels measured by the post-campaign survey, an estimated 1,608 child deaths were averted across the three regions in 2012 (637, 445, and 471 in Brong Ahafo, Central and Western regions, respectively). It is important to note that these deaths are additional to those that would have been averted with pre-campaign LLIN use i.e. are those directly attributable to the LLIN Campaign rather than the absolute number of deaths that could be averted by post-campaign LLIN use levels. The economic cost per child death averted was USD 6,619 (USD 5,032 in Brong Ahafo, USD 7,493 in Central and USD 8,702 in Western regions) (Table 2). Although the relative improvement in LLIN use by under-fives after the LLIN Campaign was similar across all three regions, a greater number of deaths are predicted to be averted in Brong Ahafo due to the higher overall proportions. In addition, because the overall costs for LLIN delivery were lower in Brong Ahafo, the financial and economic cost per death averted (whatever the perspective) was considerably lower in Brong Ahafo than Central and Western regions.

Sensitivity analysis

The ICER was sensitive to useful lifespan of the net; a reduction in LLIN lifespan from three years to two years increases the cost per LLIN delivered and cost per death averted by around 50%. Conversely, if the lifespan of LLINs could be increased to five years then the cost per LLIN delivered and per death averted drop by around 40% (combined figures presented in Figure 2; regional breakdown presented in Additional file 2).

Using the median 2012 bulk purchase cost of a rectangular or conical net as the lower and upper unit costs showed the ICER was sensitive to the LLIN purchase price, and the regional differences reflect the variation in the price paid when procuring the nets. The ICER was very sensitive to the estimate of protective efficacy of the LLINs: if the lower uncertainty limit was used (3.4 deaths averted per 1,000 children protected by an LLIN), cost per death averted increased by around 60%; if the upper uncertainty limit was used (7.7 deaths averted per 1,000 children protected by an LLIN), cost per death averted decreased by around 30% (Figure 2; Additional file 2). The ICER was least sensitive to the choice of discount rate.

Estimating the incremental cost-effectiveness of hang-up activities

In the comparison scenario of a universal campaign without hang-up visits to households, the number of volunteer-days and supervisor-days were reduced by replacing house-to-house distribution with fixed-point distribution and removing follow-up visits to households for hang-up. This reduced the financial provider-side costs by USD 0.10 (2%) per LLIN delivered. Although the direct financial savings were relatively modest, this alternative campaign design has the potential to substantially reduce the annual economic cost from the societal perspective by around 10%. This difference is because the unpaid time of volunteers is valued in the economic analysis so a reduction in the number of volunteer-days leads to a reduction in the overall full economic costs of the campaign.

However, the household survey data show that use was significantly higher in households where all or some of the campaign LLINs were hung by the volunteer (Table 3). Therefore, in the campaign delivery scenario in which no hang-up visits were conducted, the use by children under five that could have been achieved was adjusted accordingly. By reducing the proportion of children under five sleeping under an LLIN post-campaign in this scenario, the number of all-cause child deaths that could have been averted was lower and therefore the financial cost per child death averted increased by approximately 40%; the annual economic cost per child death averted also increased by approximately 30%.

The sensitivity of the predicted ICER (of a universal LLIN campaign with hang-up activities compared to one without) to the assumption about the additional effect of hang-up was explored. As the odds of a child under five using an LLIN in a household where LLINs had been hung by a volunteer increased, the cost per death averted decreased. A universal campaign with hang-up activities dominates a universal campaign without hang-up activities at an odds ratio of approximately 1.1 or greater (Figure 3). Thus, a minimum additional effect of hang-up activities of a 10% increase in under-five LLIN use is needed to achieve a cost-saving of a universal campaign with hang-up compared to a universal campaign without hang-up in this setting.