Robustness of items within and across surveys

Sociologists often draw conclusion based on one single survey. Estimates from survey items can however be affected by different errors such as nonobservation bias, interviewer effects, question designs or measurement effects more generally. The sources of these errors heavily depend on the survey design and the budget allocated to it (mode, contact procedure, refusal conversion etc.). In addition, weights may correlate little with substantive variables and using weights in the analysis model may not be sufficient to correct for nonobservation bias. In this paper we compare the mean values of three often analysed items (political interest, satisfaction with democracy and health) and try to find the possible sources of error explaining found differences. We analyse discrepancies of means within (using or not using post-stratification weights) and between six different surveys run at the same time: The Swiss part of the European Social Survey in 2010, the Swiss part of the International Social Survey Programme (ISSP) in 2011, the Swiss Household Panel in 2011, the Swiss electoral study (CATI and web) in 2011 and the Swiss Labor Force Survey in 2010. Results show that while there are small differences within surveys, large differences may occur between surveys. The differences in means can probably be explained by selection bias that coverage and nonresponse weight adjustments fail to correct for, or measurement bias due to question wording, different answer categories, or different modes. It should be kept in mind that it is very difficult, in a non-experimental set-up, to identify and disentangle the different sources of error. The paper raise however awareness about drawing conclusions based on a single survey and the survey errors that should be taken into account.