Text to Web or Live Interviewer Text?

What’s the best way to do a text message survey? TL,DR: It depends on the length.  

There is more than one way to use text messages to conduct survey research. The more common way to field surveys is to send links to a web survey platform where respondents self-administer an interview. Alternatively, Survey 160’s technology allows researchers to conduct what we call a “live interview text message survey,” in which the entirety of the interview is conducted back and forth by text message, with the interviewer coding respondents’ answers using the Survey 160 platform.

But which of these two text message survey tools is best, by what criteria, in which situations? 

To answer this question, at Survey 160 we have conducted a series of experiments in which we randomly assigned respondents to one of these two modes, as well as to a third “choice” condition where respondents could select how to complete the survey. We used these experiments to look at several different outcomes. 

First, which method has better response rates? To answer this question we have conducted a series of randomized controlled trials in which respondents are assigned, after initial contact, to complete the survey back and forth over the text message conversation, are provided a link to a web survey, or are given a choice.  Study 1 was a survey of President Joe Biden’s approval and other political topics, conducted with a national sample of registerd voters in February 2021. Study 2 was a survey on vaccine behavior and other health-related attitudes conducted in Colorado and Oregon in July 2021. In these early experiments, we observed higher response rates for the live interviewer approach, but among people who began the survey on text-to-web, the dropoff rates were much lower.

Study 1 Completion Rates

Study 2 Completion Rates

If Live text interviews have higher cooperation but also higher drop-off than text-to-web surveys, then there is necessarily a “break-even” point in terms of survey length above which text-to-web surveys have higher completion rates and lower costs. To identify where that break-even point is, in terms of survey length, we conducted a new study in April 2022 in a survey of Pennsylvanians asking about political attitudes. In this third experiment, in addition to randomly assigning survey mode, we also randomly varied the length of the survey instrument across respondents by assigning people to between 0 and 6 additional blocks of questions (“modules”) after a few core question items on demographics and primary election vote choice. What we found was that as the survey progressed, the response rates of the live SMS and the text-to-web surveys equalized. Looking at the response rate by number of randomly-assigned additional modules, we see a clear decrease in response rate with additional modules. At the same time, the difference in completion rate by text to web does not consistently or statistically significantly vary by the number of additional survey modules respondents are assigned.

Study 3: Completion by point in survey

Study 3: Completion by assigned length

The implications for response rates – and by extension, cost – are clear. For shorter surveys, live-interviewer text surveys are preferable, while for longer surveys, you should go with text-to-web. In general, for surveys shorter than 15 questions, we recommend live interviewer text surveys, while for surveys longer than 25 questions we generally recommend text-to-web surveys. 

In between 15 to 25 questions, there are some other considerations that merit discussion. For instance, we have generally, but not uniformly, found that weighting live-interviewer SMS data produces lower design effects than text-to-web, as shown in the graph below. We have also examined differences in weighted survey accuracy compared to external benchmarks like election results or other polling averages, but have not found significant differences in the experiments we have conducted.

Study 1 Design Effect

Study 3 Design Effect

Finally, it’s worth considering what modes respondents themselves prefer. Across these three studies, we found that when presented with a choice, majorities of respondents consistently chose live interviewer surveys.

Of course these are the results from just three studies, and using the best practices developed by Survey 160, so experiences using other platforms and research protocols may differ. But they provide context for our recommendations to our clients: use live interviewer text messages for short surveys, but for longer surveys you may get higher response rates and lower costs using text-to-web.