Making uncommon experiences visible in the survey life cycle

PDF
Publication

FORS Guide Nº 24

DOI
https://doi.org/10.24449/FG-2024-00024
How to cite

Nora Dasoki, Guy Elcheroth, Sandrine Morel, Erika Antal & Robin Tillmann (2024). Making Uncommon experiences visible in the survey life cycle. FORS Guide, 24, Version 1.0. 1-18. https://doi.org/10.24449/FG-2024-00024

Abstract

This FORS Guide discusses how methodological practices and dissemination policies are likely to produce cumulative filters throughout the survey life cycle, acting as barriers to the representation of uncommon social experiences. A reflexive approach is proposed to researchers to observe how and why some of these filters make it less likely that minority populations become study samples, that non-normative events are reported by participants or noticed by researchers, and that “atypical” data are not stored, shared, and reused. This guide has been developed by members of the 'Data diversity and public good research’ group within the FORS-SSP scientific research program.

Recommendations

  • Be curious – do your best to know your data. Sincere curiosity is the necessary foundation for any constructive action; nothing can be done about bias in survey as long as we are blind to it. If you have the chance to collect you own survey data, re-reading section 2 of this guide might help you ask relevant questions about sources of bias in your survey design. If you use data produced by others, read the methodological documentation, use this guide to read between the lines, and try to exchange with someone who was there when methodological decisions were taken, to understand how priorities were set.
  • Be humble – don’t over-claim the representativeness of your findings. Once you have followed recommendation 1 wholeheartedly, you are likely already inoculated against too broad claims regarding representativeness. Try always to replace them with more precise statements, which will truly inform your readers on which parts of your reference population, and which kinds of experiences, your survey allows measures with the greatest highest precision, and where are its likely blind spots. Keep these statements in mind yourself when analyzing your data and interpreting your findings.
  • Be transparent – help others to know their data. If you contribute to the production of data that will likely be used by other researchers, you can do much to make it easier for them to follow recommendations 1 and 2. If you document the survey design process and fieldwork in a clear and descriptive manner, it will be easier for them to understand your work and make good usage of your data, than if you opt for much conceptual jargon or write as if you need to praise and market your data.
  • Be generous – share what you can. When you have to decide which data and meta-data to share with other researchers, don’t be blocked by the question whether they are likely to meet standard expectations. From a systemic view, standard expectations can actually be a serious source of systematic bias in the data that circulate and inform the scientific consensus. The more you share, the more likely you will hence contribute to remove systematic bias, especially if your data are “different”. If you have ethical reservations, try to overcome an all-or-nothing approach, seek competent ethical advice and let yourself help with those obstacles that can be removed without infringing on the protection of research participants, or other stakeholders.
  • Be fair – don’t assume uncommon experiences away. The fact that certain social groups have not participated in a survey, that survey participants did not report certain types of events, or that your analyses do not reveal certain relations, is not yet proof that these groups, events or relations do not exist. Re-reading section 3 of this paper might inspire you not to interpret or write down your findings as if the question of their existence is irrelevant, but to give fair consideration to both substantive and methodological interpretations of empirical absences.
  • Be original – worry first about not reproducing existing biases. Survey researchers are intuitively inclined to worry first about new biases they might introduce when deviating from established survey practices (which are more likely to be visible, and finger-pointed by peers). From a systemic perspective however, reproducing the same biases as everyone else is much more damaging than creating new biases: The former contributes to entrenched systematic bias, while the latter is more likely to increase variability in aggregate research outcomes. Thinking about this aspect might encourage you to weigh risks differently next time you consider departing from beaten tracks.
  • Be purposeful – choose one uncommon experience and make it visible. After having read this paper, you might feel a vague sense of discouragement: If there are so many ways in which the experiences of minorities are masked in surveys, sensitive events or non-normative behaviors, the challenge might be too huge to tackle. But you might try to turn things upside-down: If so many blind spots are our baseline, you can already make a difference by tackling one of them earnestly. Unless you are leading one of the few very high-resourced surveys, focusing on one specific group or experience, giving yourself the methodological means to let it appear in open daylight, and then pursuing this goal throughout the different stages of the research cycle, is certainly your best chance to take the field a step further without dispersing limited resources – including your own cognitive focus – over many tiny battlefields.

    • Copyright

    © the author 2024. This work is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0)

    Publication year

    2024

    Quantitative data anonymisation: practical guidance for anonymising sensitive social science data