Survey length specificity

One common question we get when designing a text message survey is whether to specify how long it will take respondents to complete. Intuitively, if it’s short, it is helpful to tell that to respondents, but if it is long, it probably is not. The question is at what point does a short survey become a long survey?

In 2020, focusing on live interviewer surveys, we examined exactly this question. We built into our 2020 client work a series of A/B tests into survey intro messages to respondents. Some people received a “specific” treatment, where the number of questions in the survey was listed explicitly. Others were randomly assigned to receive a “vague” message, describing the survey as short, brief, or a similar description. 

We then conducted a meta-analysis of the 2.6m attempted interviews over 46 separate projects to evaluate how the effect of (randomly-assigned) specificity varied with (not randomly assigned) length. We found (as the graph shows) that the effect of specificity stopped being beneficial around 12 questions.

While this analysis focuses on specifying the number of questions in a live interviewer text survey, we have also experimented with telling people how long (in minutes) it will take to complete a web survey in a text-to-web survey. Typically, when we include this information it is in a second text message, after respondents have agreed to take the survey, as contextualizing information along with the URL to the web survey platform. While we have not conducted enough of these experiments yet to nail down the exact break-even point, we have generally found it to be useful to tell respondents the time it will take when that time is ten minutes or less. In other cases, we have found this information to be counterproductive. For example, in this experiment conducted in partnership with Gradient Metrics for the Cornell Midterm Survey project we tested the inclusion of the following statement: “The survey takes most people under 20 minutes.” We found that this statement significantly decreased the proportion of respondents (among those who agreed to participate in an initial text message) who started the web survey (p<0.1), and we terminated the experiment early. 

Overall, it can be helpful to tell your survey respondents that the survey is brief, and to be specific in doing so … as long as it is in fact short. When the median web survey response time is 10 minutes or shorter, we recommend including a statement along the lines of the following “This survey takes most people 10 minutes or less to complete”. For longer survey lengths, we recommend additional A/B testing to refine this best practice.