Journal of Social Research & Policy, Vol. 5, Issue 1, July 2014

The Impact of Non-Response Weighting: Empirical Evidence from Modelling Residential Mobility HUSAM SADIG1 University of Essex, UK

AHMED BANANY University of the West of England, UK

Abstract This paper examines whether non-response weighting affects estimates resulting from the analysis of the desire for residential mobility. The assumption here is that residential mobility itself can cause survey non-response. If that is the case, estimates related to residential mobility that are only constructed based on the set of responding units are likely to be biased. In turn, non-response weighting may reduce the potential bias, resulting in different estimates. Data are from the British Household Panel Survey. Analysis of the desire for residential mobility is conducted. Weighted and un-weighted estimates are produced and compared. The results suggest that implementing the weights in the analysis yields different estimates, which in turn indicates bias reduction. Keywords: Survey non-response; Weighting; Bias; Precision; Residential mobility.

Introduction The typical size of the population in social science studies makes it impossible to collect data from every unit. A representative random sample may therefore be the best means to make inference and draw conclusions about the population of interest. However, when sample members are selected randomly, they cannot be replaced. Thus, non-response may be a dilemma in some surveys. This is because if there is a systematic difference between respondents and nonrespondents, in terms of what the survey is measuring, non-response will distort the distribution of the sample, as it will be biased towards the characteristics of respondents. Therefore, nonresponse weighting is an important aspect of the survey because it modifies the distribution of the responding sample and makes it similar to that of the selected sample. Non-response bias pervades estimates more if the cause of non-response is linked to the statistics under investigation. One of the areas in social science phenomena that may be directly linked to non-response is residential mobility (RM). If many non-respondents did not respond because they have moved house, estimates resulting from analyses linked to RM, which will be solely based on respondents, may be biased. Thus, a common controversy in social science is whether non-response weighting can eliminate the potential bias resulting from non-response in the context of investigating RM. This paper investigates the impact of weighting on estimates relating to the desire for residential mobility. The analysis focuses on the effect of weighting rather than the construction of the weights. While weighting may have an impact on estimates from a large range of analyses, our investigation is limited to estimates resulting from the analysis of the 1

Postal Address: Institute for Social and Economic Research, University of Essex, Wivenhoe Park, Colchester, Essex, CO4 3SQ UK. E-mail Address: [email protected]

92 | JSRP

Husam Sadig, Ahmed Banany

desire for residential mobility. The paper provides a brief background on residential mobility and non-response before illustrating the link between the two phenomena. Our discussion assumes a longitudinal type of survey.

Residential mobility Residential mobility (RM) is defined as the phenomenon with which individuals change their residence. Many people move house as they change their jobs, partners and personal networks. On the one hand, RM can associate with a few negative phenomena, such as high rates of crime and early school leaving (Tonnessen, Telle & Syse, 2013; Haelermans & De Witte, 2013). On the other hand, RM may have a number of positive impacts on individuals. For example, in a highly mobile population, an individual’s social networks are more open and people can make friendships more quickly (see for example Oishi, Lun & Sherman, 2007). Early research on residential mobility focused on residential dissatisfaction as the main factor influencing decision-making regarding residential mobility (Speare’s, 1974). However, research in recent years has shown that residential dissatisfaction may not be the only main factor affecting house moving; characteristics of a neighborhood and household structure may also have a major effect (Deane, 1990; McHugh, Gober & Reid, 1990; Rabe & Taylor, 2009). In many surveys, respondents who intend to move are asked the main reason for their decision to move. The most frequently reported answers to this relate to: home ownership, housing characteristics, life events and changes in employment status. However, often, respondents can only report one main reason for their intention to move. Thus, other possible factors that may be equally important cannot be identified. In this paper, the analysis is concerned with investigating the determinants of a future desire for moving house, regardless of the main reason reported by the respondent.

Non-response and weighting Non-response is a failure to collect measurements from sampled members (Groves et al., 2004). The partial failure to collect measurements (i.e. answers are not received for some items in the questionnaire) is referred to as item non-response. Meanwhile, a complete failure to obtain measurements (i.e. failure to conduct the interview with a sampled member) is called unit nonresponse (Lynn, 2008; Groves & Couper, 1998). In this paper, our discussion is limited to unit non-response, and the term non-response is used as shorthand for unit non-response. Non-response may occur for a number of reasons, the major are: failure to contact a sampled member (non-contact); failure to obtain cooperation from a contacted sampled member (refusal); and inability to conduct the interview with a contacted sampled member who is willing to cooperate with the survey (Lepkowski & Couper, 2002; Groves et al., 2004). However, nonresponse due to inability represents a small proportion of non-response in surveys. Most of nonresponse in surveys nowadays comes from non-contact and refusals (Brick, 2013). Non-response affects survey-based estimates in at least two ways: it results in a smaller sample size which in turn produces less precise estimates; but more importantly, if nonrespondents have different values than respondents on variables that are components of the statistics being estimated, estimates that are solely based on the responding sample may be biased (Lynn, 1996; Sadig, 2011). In longitudinal surveys where data is collected from the same sampled members on multiple occasions (waves), there are two types of non-response: wave non-response refers to the situation where a sample member is absent from the survey for one or more waves, but he or she returns to the survey and continue providing data; and attrition is the permanent drop out from the survey. In order to deal with non-response, it is important to distinguish between three missing data mechanisms (the process that generates the missingness). There are three main types of missing data mechanisms in the literature that can be distinguished (given by Rubin, 1976; Alison, 2000,

The Impact of Non-Response Weighting: Empirical Evidence from Modelling Residential Mobility

93 | JSRP

Little & Rubin, 2002). If Y represents a substantive survey variable for which some values are missing for some of the sample members; X represents a set of auxiliary variables that is fully observed for all sample members; Z is a variable that external to the survey and uncorrelated with X and Y; and R is an indicator indicates whether values of Y are observed or not, then: Missing completely at random (MCAR) is a situation where the missingness is caused by the outside phenomenon Z (i.e. R and Z are correlated). In this case, estimates derived from Y may not be biased and post-survey adjustments, such as weighting, may not change this. Missing at random (MAR) refers to the situation where the missingness is partially due to Z and partially due to X, so that there is an indirect relationship between Y and R (i.e. the relationship between Y and R is conditional on X). If this is the case, the missingness may cause bias to estimates derived from Y, but fortunately, by using X, a number of post-survey adjustments, including weighting, may be used to reduce this bias. Not missing at random (NMAR) in this case there is a direct relationship between Y and R (and may be also between X and R; and Z and R). In other words, the missingness is caused by the survey variables. In this situation, estimates derived from Y will be biased. Unfortunately, this will limit the choice of the post-survey adjustments that can be used to deal with the problem, as some methods, such as weighting, cannot help reducing the bias in this case. In practice, however, survey researchers will not know which of the three missing data mechanisms applies to the data. Because, such knowledge requires full measurements on the selected sample in terms of the survey key variables, which in turn makes addressing the problem of missing data unnecessary in the first place. Thus, survey researchers have to assume one of the three missing data patterns. MAR is the most assumed missing data mechanism, as it allows the implementation of a wider range of post-survey adjustments. Weighting is one of these adjustments, and to which our discussion will be limited in this paper. Non-response weighting assigns more values to the responding units that are similar to the non-responding units to compensate for the missing sampled members. As a result, the distribution of the responding sample will resemble that of the selected sample, and hence estimates may not be biased towards characteristics of the responding sample.

Link between residential mobility and non-response Not all non-response causes bias to estimates. Sometimes it does and sometimes it does not. In principles, estimates are biased when the reason for non-response is linked to the statistics in question. For example, consider non-response in a survey that aims at estimating the proportion of gay men in a town. Because of the sensitivity of the survey topic, it would be reasonable to assume that some of the non-response is because of the survey topic. Consequently, using only the data provided by the responding sample may lead to under estimating the statistics in question (proportion of gay men). Similarly, residential mobility may cause non-response. This may occur if the sampled member move house after the sampling stage and prior to data collection, or, in the case of longitudinal surveys, a sample member may move house between the data collection points without informing the survey organisation. In both cases, if the survey organisation fails to locate the moving sampled member, this will result in non-contact (i.e. non-response). As a result, estimates resulting from modeling the desire for residential mobility, which are only based on the responding units, may be biased against those who moved house (nonrespondents). The bias will afflict estimates if those who moved houses have different values than respondents on the variables included in the estimation. In which case, weighting may reduce bias in estimates if it is used in the estimation. This is because weighting will increase the influence of the responding units who are similar to the non-respondents who moved house (probably those who have desire to move house in the future) to balance the sample by compensating for the missing units (Lynn, 2005; Sadig, 2011). Because of the direct relationship between residential mobility and non-response, this paper aims to investigate the factors affecting the desire for residential mobility, and assesses if nonresponse weighting affects estimates resulting from this investigation.

94 | JSRP

Husam Sadig, Ahmed Banany

Data and methods The data are from the 18 waves of the British Household Panel Survey (BHPS). The BHPS is a longitudinal survey conducted through the period of 1991 to 2008 before Understanding Society took over in 2009 as the UK and the world’s largest survey. BHPS interviewed the same sampled members every year for 18 years and collected data on 9 main areas: labour markets, income, savings and wealth, household and family organization, housing, consumption, health, social and political values, education and training. Also, the BHPS provides data users with a set of longitudinal non-response weights2 at every wave which aims at adjusting for non-response at all data collection points up to the latest. Additionally, the BHPS provides a set of weights at the first data collection point (wave 1). This set of weights corrects for the unbalanced probabilities of selection and corrects for the initial non-response simultaneously. In this paper, the longitudinal set of weights at wave 18 was multiplied by wave 1 set of weights. The resulting set of weights is thus assumed to correct for the unequal probabilities of selection and for non-response in all of the 18 waves. This set of weights was used in the analysis here. In the BHPS, respondents are asked every year whether they prefer to move house. If respondents report a preference for moving house, in this paper, this was taken as an indication of a desire for residential mobility. Accordingly, the dependent variable in the empirical analysis was a categorical variable, indicating whether the respondent has an intention to move from their current house. {

(1)

Where The dependent variable in the empirical analysis. This setting permits the application of a binary outcome panel data model. Thus, random effects (RE) logistic regression was used to model the desire for residential mobility. To assess the impact of non-response weighting on estimates, the same model was estimated twice, with and without weighting. By comparing estimates resulting from the two models, conclusion can be drawn about the effect of weighting on modeling the desire for residential mobility. Since the method is held constant, differences between estimates resulting from the two models will be due to the implementation of the weights in the estimation. The independent variables are selected from three categories: individual characteristics, housing characteristics and life-course events. Namely, these variables are sex, age, employment status, financial situation, household size, income, marital status, number of children aged 16 or under, ownership of a car, type of household, number of rooms in the house, type of house, liking present neighbourhood and whether respondent has money in savings. The analysis was restricted to those aged 16 or over, and was implemented in STATA.

Results Descriptive results Table 1 presents the percentages of those with desire to move in the 18-year period of the BHPS by main respondents characteristics. The calculation was done separately for every year before aggregating the 18 years by taking the average. The table presents weighted and unweighted percentages. Also, the table presents 95% Confidence Intervals (CI) for the unweighted proportions. CI’s are used here to assess whether the weighted proportions are within

2

In the BHPS the weights are calculated using a ‘weighting classes’ method. In every wave, sample members are categorized into a number of classes using the variables: age, sex, race, employment status, income, education, region and tenure, which are measured in the previous wave. The response rate is then calculated for each class conditional on responding in the previous wave. The weight for a respondent in a given class is calculated as the inverse of the class response rate. Every wave, the resultant weights are multiplied by the longitudinal weights from the previous wave accounting for the loss of respondents between each two adjacent waves Taylor (2010).

The Impact of Non-Response Weighting: Empirical Evidence from Modelling Residential Mobility

95 | JSRP

the calculated CI’s of the relevant unadjusted proportions. If any weighted proportion falls out of the CI of its corresponding un-weighted proportion, this may be taken as an indication of a significant difference between the two proportions, and hence clear impact of weighting. In general (for both weighted and un-weighted percentages), the table depicts association between the desire to move and the majority of the characteristics. For example, employed respondents show less desire to move than unemployed respondents (30.44%<69.56% and 28.20<71.80%). This explains the stability in residential status when having a job (employed individuals are more settled; meanwhile unemployed people have more preference for moving with the motive of looking for a job). Also, those with small households have more desire to move house than those who have large households (97.46>2.54% and 98.13%>1.87). This is probably because having a small household is associated with ease and low-cost mobility, while a large household may find moving more difficult and costly. Focusing on the comparison between the weighted and un-weighted percentages, there are eight weighted percentages that fall out of the boundaries of the CI of their equivalent un-weighted proportions. These are related to the following categories: have no dependent children, have dependent children, unemployed, employed, home owner, renter, small household and large household. These results indicate that, in the analysis of the desire for residential mobility, weighting does have an impact on estimates. Implementing the weights results in different proportions of those who have desire to move house in some categories suggesting significantly different proportions in comparison with the un-weighted analysis. Thus, as BHPS weights are assumed to reduce non-response bias, it can be said that ignoring the weights in this analysis will result in biased percentages. Table 1: Sample members with desire to move: percentages by main variables Variable Un-weighted % 95% CI Weighted % Gender Men 54.74% 53.38% – 56.10% 53.86% Women 45.26% 43.60% – 46.62% 46.14% Parenthood status Have no dependent children 65.62% 64.32% – 66.92% 67.17%* Have dependent children 34.38% 33.08% – 35.68% 32.83%* Work status Unemployed 69.56% 68.30% – 70.82% 71.80%* Employed 30.44% 29.18% – 31.70% 28.20%* Financial situation Living comfortably 61.81% 60.48% – 63.14% 61.96% Doing alright 28.38% 27.15% – 29.61% 29.01% Finding it difficult 9.81% 09.00% – 10.62% 09.03% Cohabitation Not living with a partner 74.34% 73.15% – 75.53% 75.05% Living with a partner 25.65% 24.46% – 26.84% 24.95% Housing tenure Home owner 23.28% 22.12% – 24.44% 21.31%* Renter 76.72% 75.56% – 77.88% 78.69%* Household size Small household 97.46% 97.03% – 97.89% 98.13%* Large household 2.54% 02.11% – 02.97% 01.87%*  CI is a 95% confidence interval for the un-weighted percentages. * indicates a significant difference between the weighted and un-weighted proportions. Multivariate analysis Table 2 presents the results from the random effects model of the desire for residential mobility. The table presents weighted and un-weighted coefficients. Additionally, 95% CI are

96 | JSRP

Husam Sadig, Ahmed Banany

presented for un-weighted coefficients. Similar to our descriptive statistics, CI is also used here to assess if weighting results in significantly different coefficients. With regard to the factors affecting the desire for residential mobility, for both un-weighted and weighted models, the analysis shows that most factors are significant. For example, renters have more desire for residential mobility than homeowners ( ̂ = 1.778, p < 0.01; ̂ = 1.518, p < 0.05). Also, a single-person household favours moving more than a multi-person household ( ̂ = 1.444, p < 0.01; ̂ = 1.932, p < 0.01). These results are inline with the literature on residential mobility indicating that it is higher amongst renters and small households than amongst homeowners and large households. Concentrating on the differences between the un-weighted and weighted models, our results distinguish between differences relating to precision and bias. a) Precision: The weighted model results in two differences relating to two variables (renter and living with a partner). Both variables are less significant in the weighted model than in the ̂ un-weighted model ( = 1.778, p < 0.01; weighted: ̂ = 1.518, p < 0.05) ̂ and ( = 0.957, p < 0.05; weighted: ̂ = 0.811, p < 0.10). b) Bias: This is based on the assessment of the CIs. Looking at the coefficients in the weighted model, it can be seen that there are two coefficients that are different in their magnitude from their equivalent coefficients in the un-weighted model. These are: Likes neighbourhood and single-person household. Both coefficients fall out of the CI of their equivalent coefficients in the un-weighted model. This is clearly showing the effect of the weights in dealing with non-response bias. Table 2: Random effects logistic regression models of the desire for Residential Mobility Un-weighted CI Weighted Year 1997 to 2002 0.957 0.674 1.240 0.931 Year 2003 to 2008 0.775** 0.459 1.090 0.804** Female 0.768* 0.588 0.947 0.762* Age 0.837** 0.693 0.980 0.711** Likes neighborhood 0.836*** 0.647 1.024 0.588***b Renter 1.778*** 0.947 2.608 1.518**a Unemployed 1.302* 1.059 1.544 1.399* Out of the labor force 0.785 0.532 1.038 0.649 Financially okay 0.816* 0.415 1.217 0.619* Financially Struggling 0.854* 0.601 1.107 0.739* Having savings 1.136 0.903 1.369 1.109 Annual income/1000 0.854** 0.601 1.107 0.751** Member of large household 0.733** 0.433 1.023 0.601** Living with partner 0.957** 0.674 1.240 0.811*a Number of dependent children 0.716** 0.324 1.108 0.675** Having a car 1.286 1.090 1.482 1.296 Living in flat 1.234** 0.821 1.646 1.391** Living in business premises 1.288*** 0.996 1.580 1.317*** Living in bedsitter 0.789 0.693 0.885 0.770 Single-person household 1.444*** 0.981 1.906 1.932***b Living in large accommodation 0.748*** 0.569 0.926 0.608*** rho 0.36 0.43 N 5132 5132 Note: Entries are the odds ratios. Data are from BHPS 1991-2008. rho represents the percentage of variance that is due to differences across respondents, and the values in the table

The Impact of Non-Response Weighting: Empirical Evidence from Modelling Residential Mobility

97 | JSRP

indicate enough variability between respondents to favor a random effects model. a indicates differences in significance levels between equivalent coefficients in the two models. b indicates significant differences between equivalent coefficients in the two models. The reference categories of the categorical independent variables are: year 1991 to 1996, male, do not like neighbourhood, Home owner, employed, having good financial situation, do not have savings, member of small household, not living with partner, not having a car, living in a house, nonsingle person household and living in small accommodation respectively. * p < 0.10, ** p < 0.05, *** p < 0.01. These results confirm our descriptive results. They indicate that implementing non-response weights in the analysis of the desire for residential mobility may have a significant impact on estimates. The sizes of the weights increase the variability in the data. In turn, this affects some estimates and increases their standard errors. As a result, these estimates became less significant. Moreover, the weights affect the calculations of some estimates by increasing the influence of the responding units that are similar to the non-responding units resulting in different sizes of these estimates (bias reduction).

Conclusion This paper is concerned with the investigation of the impact of non-response weighting on estimates in the context of analyzing the desire for residential mobility. Our analysis suggests the following: Some estimates resulting from the analysis of the desire for residential mobility, which is solely based on data from the responding sample, are biased. Using non-response weight adjustments in the analysis have a significant impact on these estimates. This impact may be in one of two forms: a) A change in the standard error of an estimate resulting in a different significance level of the estimate. b) Bias reduction, which results in a different magnitude of an estimate. Assuming that the weights construction reflects the non-response process in the sample properly, the weights can be said to deal with non-response error effectively. To the extent this assumption is satisfied, the weighting will tend to reduce the bias in estimates. These findings have important implications, especially in the interpretations and conclusions drawn from these types of analyses. Ignoring the weights and analyzing the data as if the response rate is 100%, may produce misleading results. The investigation in this paper shows that a few of the un-weighted estimates, mistakenly, appear more precise or with incorrect magnitude, in similar analyses, more estimates may be affected. In longitudinal surveys, it is typically the responsibility of the survey organisations to design the weights and include them in data files to be used by analysts. However, it is rarely the case that secondary data users incorporate weights in their analysis. The common attitude amongst data users is that weighting is a sophisticated technique that is only implemented by experienced statisticians, and it does not matter at the stage of producing basic survey reports (Lynn, 1996). In our opinion, the effect of non-response is under estimated by such attitude. The principles underlying the effect of non-response are clear, convincing and can be proven practically. The set of findings in this paper is one demonstration of this. Thus, survey organisations should emphasise the importance of using the weights in their quality reports, prepare and properly document the weights construction process. In turn, data analysts should consider implementing the weights in their analysis to improve the quality of their results. It is desirable and not difficult to apply weighting in many data analysis software, and with most of the statistics techniques.

98 | JSRP

Husam Sadig, Ahmed Banany

References 1.

Allison, P. D. (2000). Missing data. Thousand Oaks, CA: Sage.

2.

Brick, J. (2013). Nonresponse and Weighting Adjustments: A critical review. Journal of Official Statistics, 29(3), pp. 329-353. http://dx.doi.org/10.2478/jos-2013-0026

3.

Deane, G. (1990). Mobility and adjustments: paths to the resolution of residential stress, Demography, 27(1), pp. 65–79. http://dx.doi.org/10.2307/2061553

4.

Groves, R., & Couper, M. (1998). Nonresponse in Household Interview Surveys. New York: Wiley.

5.

Groves, R., Fowler, F., Couper, M., Lepkowski, J., Singer, E., & Tourangeau, R. (2004). Survey Methodology. New York: Wiley.

6.

Haelermans, C., & De Witte, K. (2013). Does residential mobility improve educational outcomes? Evidence from the Netherlands. Top Institute for EvidenceBased Education Research Working Papers, No. 13/14. Maastricht: Maastricht University.

7.

Lepkowski, J., & Couper, M. (2002). Nonresponse in the Second Wave of Longitudinal Household Surveys, in R. Groves, D. Dillman, J. Eltinge, & R. Little (Eds.) Survey Nonresponse, (pp. 259-272). New York: John Wiley.

8.

Little, R. J. A., & Rubin, D. B. (2002). Statistical Analysis with Missing Data. New York: Wiley.

9.

Lynn, P. (1996). Weighting for Non-response. R. Banks et al. (Eds.), Survey and statistical computing 1996, (pp. 205-214). ASC.

10. Lynn, P. (2005). Weighting. K. Kempf-Leonard, Encyclopedia of Social Measurement, (pp.967-973). Academic press. 11. Lynn, P. (2008). The Problem of Nonresponse. E.D. De Leeuw, J.J. Hox., & D.A. Dillman, International Handbook of Survey Methodology. (pp. 35-55). New York: Lawrence Erlbaum. 12. McHugh, K. E., Gober, P., & Reid, N. (1990). Determinants of Short- and LongTerm Mobility Expectations for Home Owners and Renters. Demography, 27(1), pp. 81-95. http://dx.doi.org/10.2307/2061554 13. Oishi, S., Lun, J., & Sherman, G. (2007). Residential Mobility, Self-concept, and Positive Effect in Social Interaction. Journal of Personality and Social Psychology. 93(1), pp. 131-141. http://dx.doi.org/10.1037/0022-3514.93.1.131 14. Rabe, B., & Taylor, M. (2009). Residential Mobility, Neighbourhood quality and lifecourse events, Institute for Social and Economic Research Working Papers, No. 200928. Colchester: University of Essex. 15. Rubin, D. B. (1976). Inference and Missing Data (with discussion). Biometrika, 63(3), pp. 581-592. http://dx.doi.org/10.1093/biomet/63.3.581 16. Sadig, H. (2011). Non-response Weight Adjustments in Longitudinal Surveys. In Survey Research Methods and Applications, (pp.207-210). Pisa: Edizioni plus-Pisa University Press.

The Impact of Non-Response Weighting: Empirical Evidence from Modelling Residential Mobility

99 | JSRP

17. Speare, A. (1974). Residential Satisfaction as an Intervening Variable in Residential Mobility. Demography, 11(2), pp. 173-188. http://dx.doi.org/10.2307/2060556 18. Taylor, M. (Ed.). (2010). British Household Panel Survey User Manual Volume A. Colchester: Institute for Social and Economic Research, University of Essex. 19. Tonnessen, M., Telle, K., & Syse, A. (2013). Childhood Residential Mobility and Adult outcomes. Statistics Norway Discussion Papers, No. 750. Oslo: Statistics Norway.