Process evaluation of the Senegal-Community Nutrition Project: an adequacy assessment of a large scale urban project

Overview of the Community Nutrition Project in Senegal

In Senegal in the beginning of the 90s, the prevalence of underweight among children under-five increased in urban areas (WHO 2005) and marked differences in prevalence were to be expected between neighbourhoods owing to their contrasting socio-economic levels. In response to this situation, the Government of Senegal initiated the nation-wide CNP funded by the World Bank (World Bank 1995) and implemented by a private agency AGETIP (Agence d'Exécution des Travaux d'Intérêt Public; Marek et al. 1999).

The CNP featured both a nutrition component and a water component. The objectives of the nutrition component were: (i) to halt further deterioration in the nutritional status of children <3 years of age in targeted poor urban neighbourhoods; (ii) through nutrition education, to initiate changes in attitudes and feeding practices of mothers. Services were provided in specific buildings called Community Nutrition Centres (CNC), by legal entities specifically created for the purpose which consisted of four people (generally with a bachelor's degree), usually previously unemployed and living in the targeted neighbourhood (Diallo et al. 1997). Physicians specially employed for the project supervised the delivery of services.

To reach the most vulnerable groups among the poor, the first level of targeting was geographical, focusing on districts comprising a high rate of poor households (Sadio & Diop 1994). Poor households were defined as those whose average per capita monthly expenditure level was below the cost of a food basket equivalent to 2400 kcal/day. In the selected districts, every 6- to 36-month-old child was theoretically eligible. However, as the priority outcome was to bring malnourished children back to normal growth, a second level of targeting was the children's nutritional status in order to select eligible children for participating in the CNP nutrition component. According to CNP terms of reference, malnutrition was defined as underweight (low weight-for-age) identified with a colour-based growth chart which used the threshold of approximately 80% of the median of the National Centre for Health Statistics (NCHS) reference (WHO 1983). A child aged 6–36 months was eligible if s/he was: malnourished; the sibling of a malnourished child; or well nourished but had not gained weight during the last 2 months, when previously participating in CNP.

The project design has been documented in detail elsewhere (World Bank 1995; République du Sénégal, AGETIP, Projet de Nutrition Communautaire 1997). Basically, the CNP provided targeted children with monthly growth monitoring and promotion and a weekly food supplementation (a flour mix made of local ingredients), provided that their mothers attended weekly nutrition and health education sessions, for a period of 6 months. CNP also provided targeted children with: referral to health services for unvaccinated children, for severely malnourished children, for children who failed to gain weight over 2 months and for sick children; and home visits to follow-up participating children who were referred or who did not come to the services.

After a 6-month pilot phase in three cities, the CNP started a 5-year programme in almost all urban areas in the country. CNP activities were phased in gradually and after 4 years, a total of 121 000 children were enrolled in the CNP in 25 towns (République du Sénégal, AGETIP, Projet de Nutrition Communautaire 1996–1999).

Expected processes of the Community Nutrition Project in Senegal

Process objectives were developed and agreed upon with the Government, AGETIP, and key stakeholders during an objectives-oriented project planning (Ziel-Orientierte Project Planung: ZOPP) participatory workshop held in July 1994 in Dakar, Senegal. The objectives of the project were then clarified, project activities and intended results identified and indicators to monitor results defined (World Bank 1995).

The CNP planned to offer a total number of seven weightings (monthly), 24 rations of supplement (weekly) and 24 educational sessions (weekly) per beneficiary during each cohort of 6 month duration. Expected processes comprised the recruitment of at least 90% of underweight children in each cohort. Moreover, the CNP assumed that 20% of participating children would still present criteria to be eligible at the end of a 6-month period and offered them the opportunity of being recruited again. CNP's objectives were to weigh 90% of participating children monthly, to deliver 90% of the planned food supplements and to reach an attendance of 80% of the mothers at education sessions. CNP objectives also included target values for the main process outcome that is 80% of nutritional recovery at the end of a cohort in children underweight at recruitment (World Bank 1995; République du Sénégal, AGETIP, Projet de Nutrition Communautaire 1997).

Subjects of the process evaluation study

The present study focuses on two of the risk groups targeted by the nutrition component of CNP: children aged 6–36 months and their mothers. Other targeted groups such as pregnant or lactating women, or children not targeted but allowed to follow some of the program were not included in this study.

In Diourbel, an inland city of about 77 000 inhabitants in 1988 (République du Sénégal, Ministère de l’économie, des finances et du plan, Direction de la Prévision et de la statistique 1992), the district of ‘Keur Cheikh Ibra’ had the highest rate of poor households (15%) and was the first to be targeted by the CNP. It featured about 1500 children of 6–36 months of age and five CNCs were implemented. The process evaluation covered the first 2 years, i.e. four cohorts, including all participating children from these five CNCs. Monitoring record cards of all the participating children who underwent at least one body weight measurement, i.e. the one used for recruitment, were collected in the CNCs and computerized.

Data collected for the process evaluation

Data from the individual monitoring record cards provided: individual characteristics of the child and mother; attendance yes or no (y/n) of the child at each of the monthly weighing; attendance (y/n) by the mother at each of the weekly educational sessions; whether (y/n) each of the weekly supplement rations was actually given to the mother for the child; and the child's growth in weight. Children's body weight measurements had been taken by CNC teams to the nearest 100 g using a hanging baby scale (Salter). The resulting colour as reported by the CNC workers on the growth chart was entered in our files and used to code the diagnosis of underweight by CNP workers. The reason for recruitment reported by the CNC workers on individual cards was entered as such in our files. Moreover, the child's age and body weight computerized from the individual cards were also used to calculate the exact anthropometric weight-for-age index and to classify the child as underweight or not with respect to the threshold of 80% of the sex-specific NCHS reference median (WHO 1983).

Process evaluation components

Two kinds of factors (recruitment of participants and services provided) were examined in the present study through fidelity, extent and reach (Table 1). Fidelity to the design of underweight diagnosis by CNP workers was not tested in the present sample, but a study carried out during the programme in five cities including Diourbel showed that the quality of age determination, weighting and charting by using the colour-based growth chart was good (Diallo & Zeitlin 1998). However, we tested the consequence of the design issue, which is using an average chart for both sexes, on the validity of the diagnosis of underweight and, consequently, on the reach at the individual level. The fidelity of recruitment was assessed through complying with various inclusion criteria. Therefore, the reach at the CNP level was assessed here by the rate of underweight at recruitment. The participants’ attendance at the services reflected the amount of services delivered (extent).

Data management and statistical analysis

Computerization of data from individual record cards was validated by double entry. The data entry and the computation of weight-for-age index were performed with Epi-Info 6.04d (Dean et al. 1994). Data management and statistical analyses were performed using the SAS system (SAS Institute Inc., Cary, NC, USA), release 8.0.

The validity of the diagnosis of underweight by CNC workers was assessed by comparison with the diagnosis based on calculation from weight, age and sex, considered as the reference value, through a sensitivity/specificity analysis (Selvin 1996). Sensitivity was the probability to be classified as malnourished by the CNC workers when the child's computed weight-for-age index was <80% (ability to correctly identify underweight children). Specificity was the probability of being classified as non-malnourished by the CNC workers when the child's computed weight-for-age index was ≥80% (ability to correctly classify non-underweight children).

Prevalence ratios (PR; Rothman & Greenland 1998) were used to compare rates of underweight at recruitment between groups. Adjusted PRs were estimated using a suitable generalized linear model, i.e. modified Poisson regression (Traissac et al. 1999; Zou 2004).

In order to compare with the CNP's process objectives, performance in providing the services was assessed as the ratio of the number of sessions actually attended by the total number of participants to the number of sessions proposed, calculated on the basis of seven weightings, 24 rations of supplement and 24 educational sessions planned.

The sample of participating children resulting from the collection of individual cards in the CNCs was exhaustive with respect to the population studied. As the calculated values for the means, rates and measures of association were those of the population under study, there was no need to take into account sampling variability and consequently no confidence intervals and/or P-value for associations were computed (Korn & Graubard 1999).

Ethical considerations

Investigators involved in this study accessed information on CNP participants by entering data from individual forms stored in the CNCs with full agreement of CNP leaders. The name of the subject was not entered in the data file ensuring confidentiality protection for individually identifiable information. Data were analysed independently of CNP staff/employees and the investigators were given the responsibility to submit this work for publication as project coordinators signed authorization granted for any analysing and writing. During the course of the intervention, the results were used to provide recommendations to CNP leaders and workers.

Diagnosis of underweight

Results were obtained in children for whom the duration of growth monitoring was at least 4 1/2 months, i.e. in 3864 children. At recruitment, sensitivity was higher than specificity in the total sample (Table 2). Few underweight children included were misclassified (sensitivity of 90%); conversely, 39% of non-underweight children were misclassified (specificity of 61%). On the other hand, sensitivity was lower than specificity at the end of follow-up. A high proportion of underweight children (30%, i.e. sensitivity of 70%) were misclassified at the end and only 3% of the non-underweight children were misclassified (specificity of 97%) when they left the CNP cohort. These results varied among CNCs and, to a lesser extent, also among cohorts. However, important variations appeared in the quality of the diagnosis of underweight as a function of gender (Table 2), notably for specificity at recruitment and for sensitivity at the end of the cohort.

From diagnosis made by CNC workers, rate of underweight at recruitment was only 72.9% when the CNP objective was 90%, suggesting a low fidelity to targeting criteria and leading to a lower reach than expected. Using the calculated weight-for-age index for underweight diagnosis showed a decreased reach, i.e. 66.4% of underweight at recruitment. All the subsequent results presented are based on the computerized weight-for-age index based on sex-specific references.

Recruitment

The four cohorts included a total of 4084 children. The fourth cohort included far fewer children (less than half) than the previous ones (Table 3). Among the participating children, the youngest were recruited more often than the oldest and the sex ratio was 0.99.

Reach of underweight at recruitment (Table 3) varied by CNC, and increased during the first three cohorts, then declined to the lowest rate in the fourth cohort. Youngest (6–17 months) participating children were more often underweight at recruitment than 18–36 months old ones, whatever the CNC or the cohort (detailed data not shown). A marked difference in the rate of underweight at recruitment was observed between boys and girls, leading to a sex ratio of 1.44 among underweight participants. When adjusted for all five variables in Table 3, no PR value changed markedly. Not even one CNC or cohort showed the expected reach of 90% underweight children recruited.

Among the children who were not underweight at recruitment, only 2.2% and 3.8% were siblings of an underweight child or had failed to gain weight, respectively, as classified by the CNC workers. In the second, third and fourth cohorts, 30.4% (26.3–37.7% per cohort and 13.0–52.8% per CNC) of the children had already been included in a preceding cohort, which was in excess of the 20% rate planned. Among the previous participating children, 0.4% were classified as having not gained weight during the three final weightings of the previous cohort and only 61.6% (53.3–64.4% per cohort and 46.1–80.5% per CNC) were underweight at recruitment. All these data confirm the low fidelity to recruitment criteria.

Attendance at services

The total of actual weighing sessions represented 93% of the weighing sessions offered for all the participating children, with no distinction between underweight and non-underweight children. Nearly 92% of the children, underweight or not, attended at least six of the seven weightings planned (Table 4), with a proportion per CNC ranging from 87% to 97%, and per cohort from 88% (third cohort) to 96% (second cohort) (detailed data not shown). With regard to the CNP objective of 90% of children weighed monthly, extent of growth monitoring was good whatever the underweight status, thus allowing a satisfactory reach in the targeted group of underweight children. Consequently, the mean duration was of 5.8 ± 0.9 months for a planned duration of 6 months.

The extent of food supplement for children was very low as effective rations given represented 45% of the total rations offered, with a higher coverage of underweight (48%) than non-underweight children (39%), when the CNP objective was to deliver 90% of the rations. Moreover, the fact that non-underweight children received some food supplement (leakage) reflects a problem of fidelity to criteria for service delivery. The percentage of children who received ≥14 rations per CNC ranged from 25% to 44%. In the second cohort no children received ≥14 rations whereas the proportion was 27.7%, 56.8% and 73.6% in the first, third and fourth cohort, respectively (detailed data not shown). This reflects a clear improvement in extent over time during the second year of the project.

Actual attendance at the education sessions by mothers represented 62% of the total number of sessions planned, again with a higher reach of mothers of underweight children (64%vs. 56%). Therefore, extent was insufficient as expected attendance was 80%. Extent by CNC ranged from 15.6% to 63.6% of mothers who attended ≥18 sessions (detailed data not shown). There was a clear increase in attendance at education sessions from the first to the fourth cohort (20%, 30%, 41% and 58% of mothers attended ≥18 sessions, respectively).

The only differences by gender were a higher mean number of food supplement received (10.1 vs. 7.5) and a higher number of education sessions attended by their mother (14.0 vs. 12.2) in girls when compared with boys.

Outcome expected in the target group: nutritional recovery in participating children

Nutritional recovery was assessed among the participating children who were underweight at recruitment. Recovery was defined as the ratio of children who were no longer underweight at the end of follow-up to the children who were underweight at recruitment. This was calculated again when the duration of growth monitoring was at least 4 1/2 months (n = 2570 children underweight at recruitment). Nutritional recovery was 61% (between 57% and 67% per cohort, between 56% and 68% per CNC, and there was no difference between boys and girls).

Underweight diagnosis and recruitment

The explanation of the incorrect selection of targeted children may be found in a combination of several biases. First, the main explanation of a ‘chart’ bias may be found in the use of a single reference for boys and girls together. The CNCs’ growth chart had a threshold determined from a ‘mean’ value of the two NCHS references of body weight for the age of boys and of girls (WHO 1983), thus giving an intermediate value between the correct one for boys and the correct one for girls (FAO 1992; Latham 1997). This threshold was too low for boys and too high for girls. Therefore, it is likely that an underweight boy with a weight just below the correct reference value for boys would not be classified as underweight, and a normal weight girl with a weight value just above the proper reference for girls would be classified as underweight by CNP workers. Many community nutrition programmes do use a single chart for both sexes. The bias introduced by this practice is important enough to underline that it is a significant weakness of those programmes for at least two reasons. First, it leads to an assessment of underweight that differs from the diagnosis based on sex-specific references largely used to evaluate underweight status in population or impact of nutritional intervention. Second, it leads to under coverage of targeted underweight girls in contexts where female children could be already challenged with other disadvantages. However, there was no disadvantage towards girls concerning delivery of services from CNP.

Another source of bias could be that the objective of CNC workers was to include as many underweight children as possible at the beginning and to retrieve as few as possible at the end. We could therefore suggest the hypothesis of a ‘yield’ bias at the group (cohort) level. Third, this kind of error could also be due to a ‘humanitarian’ bias at the individual (child) level to avoid excluding borderline normal weight children, as previously reported (Soeters 1986). One reason for this systematic bias could be that with the chart method, the measurer knows immediately whether or not the child is malnourished and tends, at recruitment, to classify children who are slightly above the threshold point as being under it. Finally, implementing the eligibility criteria can lead to perceptions of unfairness (Marchione 2005). The individual nutritional targeting among the population who came to the CNCs was not always well accepted by mothers and the CNC workers may have felt obliged to include non-target children (‘social’ bias). Better accounting for such problems met by programme workers is critical for the improvement of program targeting (Lee et al. 2005).

For comparison purpose, sensitivity/specificity of the assessment of very low weight-for-age by using the growth chart in Africa when compared with weight-for-age<-3 z-scores was of 62.0%/98.7% under conditions which exclude some bias inducing constraints e.g. those from targeting or meeting programme objectives (Hamer et al. 2004).

Inclusion of normal weight children could be considered potentially useful in an intervention in terms of prevention of malnutrition. However, a too low proportion of underweight participants has to be taken seriously in the case where all malnourished children in the overall population would not be recruited: CNP services are delivered at the level of participating children and their mothers while improvement in nutritional status is expected as an impact at the level of the overall population.

Process outputs and outcome: attendance at the services and recovery from underweight

One can assume that high attendance rate at the monthly weightings should be considerably helped by the fact that the other two services were delivered weekly. Moreover, the weekly food supplement for the child could be one of the reasons for the mother's regular attendance at the weekly education sessions. However, attendance at education sessions was better than attendance at food supplement delivery, which was only half the CNP objective. This may be explained by a low fidelity of implementation: the supplement was not delivered to project staff on time or in sufficient quantities, especially at the beginning of the programme. Over time, these practical points improved and the two weekly services showed higher attendance. Moreover, in the three cohorts that followed the cohorts reported here, the attendance at the education sessions reached 71–86% of the total number of sessions planned and in five new CNCs opened in Diourbel, the coverage of the education sessions in their first three cohorts increased from 65% to 81% (République du Sénégal, AGETIP, Projet de Nutrition Communautaire 1996–1999).

The higher rate of initially underweight children in the groups of mothers who presented the best weekly attendance suggests that CNC workers were efficient in raising mothers’ awareness of the nutritional status of their children and of the importance of attending the sessions. However, out of the 100 g/child/day of food supplement given to each child by the CNP, a child on average ate only 25 g (Treche 1998), the food supplement being diverted to the family. It was not possible to confirm the output hypothesis, namely, that the supplement will be consumed in addition to the normal daily diet and thereby increase total daily caloric intake. Generally, mothers were satisfied with the CNC's services, but they expected more activities, mainly diversification of the food supplement, integration of health care, literacy programs and income-generating activities (Ndiaye 1999). Indeed, whatever the quality of the services provided, their use by the mothers is often determined by conflicts with income generating activities. Even in the most efficient CNC or cohort, the expected recovery rate of 80% was not reached during the first 2 years of the CNP in Diourbel.

Potential contribution to the Community Nutrition Project's impact

The expected impact of CNP is at least a stabilization, or at the best, a decrease of the prevalence of malnutrition of children in the population of the target neighbourhood. Among process outcomes, recovery from underweight in children under programme can have a direct effect immediately on impact, whereas changes in feeding practices of the mothers could have a less direct or immediate effect provided they can have benefit on child growth. A wrong underweight diagnosis, as well as applying incorrect criteria for recruitment, harmed the quality of individual targeting and therefore the CNP reach of underweight children in the overall population, thus compromising the potential extent of the direct impact. Irregular supplies of supplement could certainly have reduced, first, the direct impact through supplement consumption, if any, and, second, the attendance of mothers to education sessions; this problem needed to be addressed through appropriate action. High attendance of mothers to growth monitoring and promotion probably improved mothers’ awareness of the child's growth, but cannot be considered to have a positive impact alone as its efficiency remains to be shown (Save the Children UK 2003; Hossain et al. 2005). Good attendance to growth promotion and temperate attendance to education sessions cannot by themselves ensure effective performance in mothers training. However, we showed, in another city in Senegal, that changes in knowledge, attitudes and practices of mothers were generally favourable during the first 2-year period (Méjean et al. 2004). Therefore, a benefit for impact could be expected. It has to be noted, further, that behaviour changes resulting from education could strongly depend on the context, e.g. household resources.

Finally, beyond process results, an explanation for lack of recovery could rely on the design of choosing underweight as nutritional criterion. Indeed, many underweight children are stunted but not necessarily wasted. The fact that wasting or stunting are not differentiated by weight-for-age index could prevent the right action to be carried out toward the right problem.

The CNP leaders and workers took heed of process evaluation results thus improving implementation successfully as demonstrated also by process results from following cohorts carried out after the four ones studied here. Such accountability is likely to be one of the ways that could contribute to enhance potential of impact.

Generalizable interest

This study concerns one of the more recent large scale community nutrition programs. Evaluation was integrated into the CNP at the design phase (World Bank 1995) as now clearly recommended (Habicht et al. 1999). Independent and transparent presentation of the process evaluation of such interventions is needed to show what works in different contexts. From our study, it appeared that: a wrong design of underweight diagnosis could largely compromise reach of targeting and, moreover, could imply a risk of disadvantage towards girls; even with clearly defined design and objectives, incorrect criteria for enrolment can be frequently applied; in this context of program follow-up, growth monitoring could be effectively implemented in routine settings and on a large scale; dependence on the supplement supplier led to low fidelity of implementation. As a more in-depth analysis of data from CNP monitoring system, moreover done by an independent source from CNP, our study allowed assessing to what extent could the performances of an intervention be misinterpreted or exaggerated, as also already reported (Save the Children UK 2003). Training and supportive supervision are at heart of a well functioning program. Contrary to inadequate training and supervision of nutrition workers reported for two other large scale nutrition projects in Africa (Save the Children UK 2003), CNP ensured training of all CNC workers by local consultants or training institutions, and only the best supervisors among those who entered the training were selected. Each supervisor looks after five teams each managing a CNC. This positive point is favourable to a possible generalization of the implementation as described here.