Can Text Message Surveys Produce Accurate Results?
A question people reasonably ask about any new research method – including text message surveys – is whether the new mode produces accurate data. There are many different types of benchmarks against which to compare survey estimates. But for those focused on election research, the natural comparison to make is between estimates of candidate support and corresponding election results.
Before the 2018 midterm elections, we rigorously assessed the accuracy of short, live-interviewer text message surveys. Working in partnership with Civis Analytics, we fielded one arm of a multi-modal pre-election survey across 77 congressional districts between September 26 and November 2, 2018. Specifically, we conducted live-interviewer text message surveys where the entirety of the interview occurred within the text message conversation. Civis subsequently provided us weighted estimates based only on the text-message surveys, which are reported here (this research was first presented at the 2019 meeting of the American Association for Public Opinion Research).
To evaluate the performance of these electorates, we limited the set of districts to those where there were at least 100 interviews conducted via text message, and where there was also at least one survey conducted by the New York Times Upshot section and Siena College that year, who polled 96 Congressional districts between September 4 and November 4. The Times / Siena poll provided raw data and detailed para-data, like stratum-specific response rates, in real time, allowing us to compare text message surveys not just to elections but to a highly-rated phone-only polling operation. The overlap between the Times Upshot / Siena polled districts and those where we collected at least 100 interviews left a set of 20 districts for comparison.
First, we examined response rates across the two surveys. We saw that the short, live-interviewer short message surveys produced a higher response rate overall, and among voters with cell-phones, than did the Times Upshot/ Siena’s live-caller phone polls. This was also true for every demographic group examined except the oldest voters, who responded at higher rates to the phone polls.
Second, we compared the accuracy of the estimates between our text message surveys and the Times Upshot / Siena polling. We focus here on the estimated Democratic vote share, as margin estimates effectively double the error estimates. The graph below reports the estimates of Democratic party vote share for each.
To dig into these comparisons more systematically, we looked at three measures:
Bias: the difference between actual Democratic vote share and estimated Democratic vote share for each district averaged over all districts, so that negative values are over estimates of Democrats support and smaller values are less biased on average.
Absolute error: the absolute difference between actual Democratic vote share and estimated Democratic vote share for each district, averaged over all districts, so that smaller values reflect more accurate estimates.
Weighted error: We regress actual Democratic Vote Share on Estimated Democratic Vote Share, weighting by sample size, reporting standard error of the regression, so that smaller values reflect more accurate estimates.
The table below reports these metrics for three sets of surveys: the text message surveys fielded by Survey 160, the Times Upshot / Siena phone polls, and the poll averages produced by FiveThirtyEight.
| Average N | Average Error | Absolute Error | Weighted Error | |
|---|---|---|---|---|
| Survey160 Text Message Polls |
315 | -0.009 | 0.022 | 0.021 |
| Times Upshot / Siena Phone Polls |
493 | -0.001 | 0.019 | 0.020 |
| FiveThirtyEight poll average estimates | NA | 0.003 | 0.013 | 0.017 |
Phone polls had about 50% larger sample sizes than the text message surveys on average. And yet, on average these phone polls were only 0.3 percentage points more accurate (that is, smaller absolute error). Compared to averages of multiple surveys, texting was only 0.9 percentage points less accurate despite the much smaller sample sizes.
Additionally, when we presented this research at the 2019 annual meeting of the American Association for Public Opinion Research, we shared a panel with members of the Siena College team, who reported that they completed approximately one survey per interviewer hour. Between the higher response rate and the ability to field surveys to multiple respondents concurrently with the asynchronous communication afforded by text messaging, Survey 160 completed ten surveys per interviewer hour. That represents a substantial cost savings over telephone surveys.
This research substantiated that texting can produce accurate survey data, at least as compared to high quality phone surveys and even polling averages and can allow researchers to collect these data at a fraction of the cost that would be required to do so by phone. This finding was a basis for our product development focus. Subsequent research has shown that the accuracy of surveys can be improved by addition of other modes to reach those without cell phones, but prioritizing texting while backfilling with more expensive survey methods offers a superior way to maintain high quality while minimizing costs.
