A Comparison of Online Panels with GSS and ANES Data

Abstract

In the past five years, researchers have increasingly used low cost data collection methods to conduct surveys. Survey data collection software platforms such as Qualtrics and Survey Monkey allow researchers to easily and cheaply create questionnaires for distribution. Similarly, low cost methods are available to recruit survey participants. Some of these include online panels, Amazon’s Mechanical Turk (MTurk), and Google Consumer Surveys. With these tools, researchers are much less dependent on professional survey researchers to conduct surveys. More importantly however, survey researchers are now using non-probability online panels as a substitute for probability samples. In 2010 and 2013, AAPOR released task force reports that analyzed the challenges encountered when using online panels and nonprobability samples for high quality survey research. In 2014, a book on the use of online panels in survey research included chapters written by respected survey researchers (Callegaro et al, 2014). In the past five years, at least 20 peer-reviewed methods articles were published on the use of Mechanical Turk for social and behavioral science research. Despite the cautions raised about the appropriate use of online panels, they are being used more often, e.g., the YouGov panel and CBS News. In this presentation, we will discuss the results of a number of experiments we conducted that compared distributions in questions asked recently in the ANES and GSS to similar questions asked with MTurk samples and a Qualtrics online panel. In addition, we will show how simple multivariate models are similar and different using data from both the probability and non-probability samples. This presentation will contribute to the continuing research into the appropriate uses of online panels for survey research.