Improving survey data on pregnancy-related deaths in low-and middle-income countries: a validation study in Senegal

Introduction

The maternal mortality ratio (MMR) is a key indicator of the Millennium Development Goals (MDG5), and a proposed indicator of the post-2015 Sustainable Development Goals (SDGs). It is the number of maternal deaths per 100 000 live births, where maternal deaths refer to ‘the death of a woman while pregnant or within 42 days of termination of pregnancy, irrespective of the duration and site of the pregnancy, from any cause related to or aggravated by the pregnancy or its management, but not from accidental or incidental causes’ 1.

In many low- and middle-income countries (LMICs), vital registration systems are too incomplete to measure MMRs 2, 3. Instead, data on causes of death among women of reproductive ages (e.g. 15–49 years) often come from censuses and siblings’ survival histories (SSH). SSH are collected during surveys like the Demographic and Health Surveys (DHS) using a standardised questionnaire (‘DHS questionnaire’ thereafter): respondents list their maternal siblings (their ‘sibship’); then they report the age of surviving siblings, as well as the age at death and years since death of deceased siblings. If a respondent's sister died at reproductive ages, interviewers ask whether she died while pregnant, during delivery or within a certain time after a delivery. Some DHS ask the respondent whether their sister died within 42 days of delivery, whereas others use a different interval (e.g. 2 months). If the answer is yes to any of the three questions above, the death is classified as ‘pregnancy-related’ (PR). PR deaths are proxies for maternal deaths even though they include deaths from accidents and incidental causes. They may also include a small number of deaths that occurred outside the risk period defined by ICD-10 if the survey elicited deaths within 2 months of delivery (instead of 42 days).

SSH can yield independent estimates of maternal mortality rates: reported PR deaths constitute the events included in the numerator, whereas the person-years included in the denominator are calculated using information on dates of birth and death of a respondent's sister(s). As PR deaths are rare events however, this ‘sisterhood method’ 4 seldom allows detecting significant MMR declines in LMICs (for an exception, see 5).

SSH also play a key role in the statistical models used by the United Nations Maternal Mortality Estimation Inter-Agency Group (UN MMEIG) and the Institute for Health Metrics and Evaluation (IHME) to track global progress towards MDG5 6, 7. These models build on three components: (i) the proportion of maternal deaths among deaths of females of reproductive ages (PM thereafter), (ii) the death rate from all causes at reproductive ages (the ‘mortality envelope’, noted μ below) and (iii) the general fertility rate (GFR), that is the number of births per person-years lived by women of reproductive ages. Each of these components is estimated independently and may be adjusted using cross-country regressions. They are then combined as:

(1)

For LMICs with limited vital registration, both UN MMEIG and IHME use GFR estimates described in the World Population Prospects 8. To estimate the mortality envelope, UN MMEIG and IHME combine censuses, surveys and model life tables 9, 10. But to obtain estimates of PMs, they predominantly rely on SSH: IHME used SSH in 38 African countries with limited vital registration 6, whereas SSH represented 81% of the observations pertaining to African countries in the UN MMEIG data set 11.

The validity of MMR estimates produced by UN MMEIG and IHME thus hinges critically on the accuracy of SSH estimates of the PM. But such data have not been thoroughly validated. Evaluations of SSH data quality have focused on investigating whether SSH reproduce known fertility and mortality trends 12-14. Several studies have also compared MMR estimates obtained from SSH using the sisterhood method with MMR estimates from census 15, 13 or longitudinal data (e.g. 16, 17. Discrepancies between MMR estimates could, however, be due to either (i) biases in SSH estimates of the PM, or (ii) other errors that affect the reporting of person-years 18.

Only one study in Matlab (Bangladesh) directly measured bias in SSH estimates of the PM 19, 20. It linked SSH data to prospective mortality records collected for the same sibship by a health and demographic surveillance system (HDSS). HDSS monitors births, deaths and other demographic events prospectively in small populations during regular household visits 21, 22. In Matlab, SSH slightly underestimated the PM compared with HDSS data, because abortion-related PR deaths were misclassified as non-PR. This study has prompted the adoption of adjustment factors in statistical models of the MMR. For example, the UN MMEIG systematically inflates SSH estimates of the PM by 10% prior to including them into its MMR model 11.

We recently conducted another validation study of SSH data in a rural population of Senegal 18. In this study, we have two objectives. First, we assess whether SSH collected using the DHS questionnaire also underestimate the PM in this sub-Saharan setting. Second, we test the hypothesis that a modified SSH questionnaire, the siblings’ survival calendar (SSC), yields more accurate survey data on causes of death among women of reproductive age than the DHS questionnaire.

Data and methods

Sampling of respondents

We first selected a stratified sample of (i) sibships in which one sibling (male or female) died between 15 and 59 years old since the beginning of the HDSS (n = 592), and (ii) other sibships, in which no adult deaths were recorded by the HDSS (n = 500). We randomly allocated the sampled sibships to the DHS or SSC groups (Figure 1), and we selected at random up to two surviving members of each sibship. Respondents included both men and women aged 15–59 years old. We recruited eight interviewers who had previously conducted one of the Senegal DHS. Interviewers were unaware of the composition of respondents’ sibships, including whether they were affected by PR or non-PR deaths. In total, 609 respondents from the DHS group were interviewed vs. 580 from the SSC group.

Analytical framework

We considered a SSH survey conducted in a population with n deaths among women of reproductive age in the past 15 years, according to a reference data set (i.e. HDSS). Among these, m were PR deaths. The ‘true’ PM is m/n. We assumed that there is at most one death per sibship. All n deceased women have at least one surviving sibling who is interviewed during the survey. We defined a variable S, with S = 1 if the respondent was member of a sibship with a PR death according to the HDSS, and 0 otherwise. We defined a variable R, with R = 1 if the respondent reported that one of her sisters died at reproductive ages (irrespective of cause) and 0 otherwise; and a variable C, with C = 1 if s/he reported a cause of death (PR vs. non-PR) that was concordant with the cause recorded by HDSS and 0 otherwise. Selective omissions occur when P(R¦S = 1) ≠ P(R¦S = 0). P(C¦R = 1, S = 1) is the sensitivity of SSH, that is the probability of correctly classifying the cause of a reported death among respondents in sibships with a PR death; and P(C¦R = 1, S = 0) is the specificity of SSH, that is the probability of correctly classifying the cause of a reported death among respondents in sibships with a non-PR death. The SSH estimate of PM is as follows:

(2)

Results

Validation results

The DHS questionnaire was affected by selective omissions of deaths (Table 3). Among DHS respondents whose sister died of PR causes according to HDSS, 27/28 reported a death at reproductive ages (96.4%, Table 3). This was significantly lower (39/55, 70.9%) among DHS respondents whose sister died of non-PR causes (P = 0.007). Among SSC respondents, we could not detect selective omissions of deaths (P = 0.469).

Table 3. Selective omissions of deaths among women of reproductive age during siblings’ survival histories, by type of questionnaire and sibship composition

	Sibships with non pregnancy-related deaths		Sibships with one pregnancy-related death		P-value	Unadjusted OR (95% CI)	Adjusted OR (95% CI)
	Reported/expected	% (95% CI)	Reported/expected	% (95% CI)			Model 1	Model 2
DHS questionnaire	39/55	70.9 (56.6, 82.0)	27/28	96.4 (78.9, 99.5)	0.007	11.1 (1.41, 87.0)	11.6 (1.59, 84.4)	10.7 (1.28, 88.9)
SSC questionnaire	50/59	84.8 (70.6, 92.8)	20/22	90.9 (71.4, 97.6)	0.469	1.8 (0.36, 9.02)	2.2 (0.45, 10.6)	1.4 (0.27, 7.39)

• ‘Reported’ refers to the number of respondents who reported a death at reproductive ages (15–49) among their sisters over the past 15 years; ‘Expected’ refers to the number of respondents who were members of sibships in which there was at least one PR/non-PR death in the past 15 years; P-values test the null hypothesis of no difference in proportion of respondents reporting a death at reproductive ages over the past 15 years between sibships with a PR death and sibships with only non-PR deaths. This P-value is based on a χ² test. Standard errors are adjusted for the clustering of respondents within sibships. OR = odds ratio; ORs are obtained from logistic regressions in which the dependent variable is a dummy variable taking value 1 if the survey respondent reported a death at reproductive ages (irrespective of cause), and 0 if no such death was reported. The key dependent variable is a dummy variable taking value 1 if the respondent was member of a sibship in which one death was PR, and 0 if the respondent was member of a sibship where the death was non-PR. OR > 1 thus indicates that respondents were more likely to report a death at reproductive ages (irrespective of cause) if their sister had died of PR causes. Model 1 includes additional control for covariates associated with assignment to the DHS vs. SSC questionnaire in bivariate analyses at the P = 0.2 level (see Table 2). Model 2 includes controls for all variables listed in Table 2. Among the 10 respondents in the DHS group for which HDSS data did not permit assessing the sister's cause of death, two reported a PR death. Among the seven respondents from the SSC group in a similar case, one reported a PR death.

The DHS and SSC questionnaires had similar sensitivity (92.6% vs. 90.0%, P = 0.744), but the SSC had significantly higher specificity than the DHS questionnaire (Table 4). In sibships with non-PR deaths according to HDSS, 48/50 SSC respondents correctly classified the reported death as non-PR (95.4%) vs. 31/39 DHS respondents (79.5%, P = 0.014).

Table 4. Sensitivity and specificity of reports of deaths among women of reproductive age collected during SSH survey, by type of questionnaire

	DHS questionnaire		SSC questionnaire		P-value	Unadjusted OR (95% CI)	Adjusted OR (95% CI)
	Correct/reported	% (95% CI)	Correct/reported	% (95% CI)			Model 1	Model 2
Sensitivity	25/27	92.6 (74.6, 98.2)	18/20	90.0 (69.5, 97.3)	0.744	1.39 (0.19, 10.2)	0.97 (0.14, 6.63)	–*
Specificity	31/39	79.5 (64.1, 89.4)	48/50	96.0 (85.5, 99.0)	0.014	6.19 (1.22, 31.4)	5.92 (1.12, 31.3)	6.60 (1.22, 35.7)

• Sensitivity refers to the proportion of deaths classified as pregnancy-related using the HDSS data, which were also reported as pregnancy-related during the SSH survey. Specificity refers to the proportion of deaths classified as not pregnancy-related using the HDSS data, which were also reported as not pregnancy-related during the SSH survey. P-values measure differences in sensitivity between the SSC and DHS questionnaires. They were based on a χ² test of association. Standard errors were adjusted for the clustering of respondents within sibships. OR = odds ratio; ORs are obtained from logistic regressions in which the dependent variable is a dummy variable taking value 1 if the survey reported a cause of death (PR vs. non-PR) that was concordant with the cause of death ascertained by the HDSS. The key dependent variable is a dummy variable taking value 1 if the respondent was assigned to an interview with the SSC questionnaire, and 0 if s/he was assigned to the DHS group. In that case, an OR > 1 thus indicates that respondents assigned to the SSC group were more likely to report a concordant cause of death than respondents assigned to the DHS group. Model 1 includes additional control for covariates associated with assignment to the DHS vs. SSC questionnaire in bivariate analyses at the P = 0.2 level (see Table 2). Model 2 includes controls for all variables listed in Table 2. *There were too few cases to estimate this model.

The DHS questionnaire consistently overestimated the PM (Figure 2). For example, in a population where the true PM is 10%, the estimated PM obtained from the DHS questionnaire was approximately 30%. The SSC yielded PM estimates that were, at most, four percentage points higher than the true PM.

Discussion

In LMICs with limited vital registration, maternal mortality is often estimated from survey data. It is thus crucial to validate and improve SSH, that is the instrument used to record causes of death among women of reproductive ages during surveys. We compared SSH collected using two different questionnaires to high-quality prospective mortality data from a HDSS in Senegal. The DHS questionnaire was prone to selective omissions of deaths, with PR deaths more frequently reported. Among reported deaths, the DHS questionnaire had high sensitivity, but low specificity: respondents misclassified non-PR deaths as PR. These reporting errors lead to large overestimates of the PM.

These findings contrast with another validation study in Matlab (Bangladesh), where the DHS questionnaire underestimated the PM 20. In Matlab, there were no selective omissions of deaths, but SSH respondents misclassified abortion-related deaths as non-PR, either because they were not aware of their sister's pregnancy or because of social desirability bias associated with abortions. The DHS questionnaire likely had higher sensitivity in Niakhar, because abortion-related deaths are less common in rural Senegal: most PR deaths occur during or after delivery 24, 27.

The DHS questionnaire was less accurate in recording non-PR deaths in Niakhar. This may be because SSH respondents under-report non-PR deaths due to stigmatised causes of death. In Niakhar, epilepsy – an often-stigmatised condition in sub-Saharan countries 28 – was the most frequent cause of death attributed by physicians among omitted non-PR deaths (5/16, 31.3%). SSH respondents may also inaccurately assess the timing of their sister's death, that is whether it took place within 42 days of delivery.

This study also constitutes a unique attempt to improve the accuracy of SSH data on PR mortality. Compared with the DHS questionnaire, a new SSH questionnaire, the siblings’ survival calendar (SSC), was not affected by selective omission of deaths and had significantly higher specificity. Probing and recall cues incorporated in the SSC likely helped respondents report of a number of non-PR deaths, which were omitted during DHS interviews. The event history calendar likely helped respondents determine the timing of their sisters’ deaths. The SSC yielded virtually unbiased estimates of the PM.

There are several limitations. First, the HDSS does not constitute a gold standard on PR mortality 29. However, our findings remained after accounting for the possibility of misclassifications in the HDSS data set (Appendix S2). Second, our analyses are limited by small sample size, particularly our tests of the difference in sensitivity between questionnaires. We also could not test whether the effects of the SSC varied by sibship or respondent characteristics. Third, our study included men and individuals aged 50–59 years old as respondents. These two groups are not always included in SSH data collection during DHS. There are, however, calls to more systematically include men and older respondents in SSH data collection during DHS and other surveys 30, 31. Fourth, we did not consider the impact of reporting errors on confidence intervals calculated for MMR estimates. The width of confidence intervals of MMR estimates depends on a number of parameters including sample size, cluster survey structure and uncertainty associated with model specification 6, 11. A full assessment of the impact of reporting errors on confidence intervals thus requires more complex analytical frameworks and simulations. Fifth, a respondent may sometimes report that her sister is alive, when in fact she died before the survey. Such vital status errors may affect PM estimates, but they are very rare: in a previous study in Senegal, we found that the DHS questionnaire had 100% specificity and 99.6% sensitivity in recording the vital status of adult siblings 18. Sixth, we tested a version of the DHS questionnaire, which elicited PR deaths having occurred within 42 days of the delivery. Some DHS, however, use 2 months instead to elicit post-partum PR deaths. According to the Niakhar HDSS, there were no deaths that took place between 42 and 60 days after a delivery. The results of this validation are thus likely not affected by different choices of a risk period for PR deaths. Finally, the generalisability of our findings may be limited because (i) HDSS only provided reference data for a subset of the Niakhar population, (ii) inhabitants of the Niakhar area may be more aware of causes of deaths than respondents in other settings due to long-term demographic surveillance, and (iii) the Niakhar HDSS population has significantly lower schooling than the rest of Senegal 23.

Our results nonetheless have important implications. First, they suggest that the statistical models currently used to track progress towards MDG5 should be re-examined. Some of these models adjust SSH data for forgetting of adult deaths, but do not attempt to correct misreporting of causes of deaths in SSH data 6, 31. Other models assume that SSH universally underestimate PR mortality and apply a single correction factor to all SSH data sets 11. All things being equal, these analytical strategies may lead to overestimates of the level of maternal mortality in Senegal and similar settings. Second, our results suggest that the accuracy of SSH can be improved by adopting a new questionnaire, the SSC. Further research is needed to validate data collected by the SSC in additional epidemiological settings (e.g. populations with high HIV prevalence), and to address possible barriers to the integration of the SSC into large-scale surveys. Such barriers include additional training and interviewing time. Refining training protocols and/or transferring the SSC to mobile data collection can address these barriers. Ultimately, the SSC may permit better tracking progress towards the reduction in maternal mortality.