Timing your Texting

Alex Dawahare, Ali Raphael, Lily Collins, Pedro Antenucci, and Kevin Collins, Survey 160

Last year, the Journal of Survey Statistics and Methodology published a paper by NORC researchers on, among other topics, the timing of text message delivery. They found no difference between a 10am or 5pm text message delivery time.

Operationally it is useful to know if the time of delivery of a text message has any impact on the survey. At Survey 160, we field text message surveys somewhat differently from others, asking for specific survey participation before sending a link, or in other cases conducting the entirety of the survey over text message. So we wanted to see if we could replicate that null effect. Our full methodological description is below, but in an internal survey we randomly assigned people to be contacted at 12 pm and 4 pm eastern time. On average, we found no substantive or statistically significant difference. There’s more variation when breaking out by time zone, but these differences are not statistically significant. 

We wanted to go one step further and look across a broader range of studies and larger range of times. We conducted a pooled meta-analysis of about 900,000 interview attempts over ten surveys, looking at both cooperation rates (who says they’ll take a survey) and completion rates (who finishes the survey). In this case, we looked at not text-to-web surveys but live-interviewer SMS surveys, where the entire interview happens back and forth via text message, with interviewers coding responses using Survey 160’s texting software. And we looked at not just conversations that are started in the morning and afternoon, but also those in the evening (we can continue these live interviews over multiple days). 

The results provide an interesting window into respondent behavior, and have informed our decision to add a new software feature to further optimize survey completion rates. 

As before,  we do not see substantively or statistically significant differences in cooperation rate by time of day. However, we do see a small but meaningful drop off in completion rate for the evening (morning and afternoon are not significantly different). We excluded the last day of the survey from this analysis to limit one possible explanation, that people started the survey but didn’t have an opportunity to complete it.

New Software Feature

To better account for the respondent's tendency to be less likely to continue the survey when first contacted at the end of the day, we’ve introduced a new feature: “response only mode.” When activated, interviewers stop sending new initial messages, but continue to field responses. Survey Managers at Survey 160 can activate and deactivate the tool, and our software development roadmap includes a plan to dynamically automate this functionality based on the parameters of the survey.

Methodological Notes

The time-of-day randomized controlled trial was fielded as part of a survey of US adults fielded from February 5 through February 11, 2025. Respondents were sampled from, or matched to, the TargetSmart commercial file. There were two attempts made to respondents, except for those who refused on the first attempt, though only the first attempt was randomized and only responses to the first attempt were included in the analysis here. Additionally, the handful of attempted interviews of respondents in Alaska and Hawaii were excluded from this analysis.

The pooled re-analysis included 10 live interviewer sms surveys, each between 4 and 10 questions in length, representing 901,934 attempted conversations, conducted between December 2023 and February 2024. Regression model controls for day of week and day of survey (first day, second day, etc). The final day of each survey is excluded from the analysis, because anyone who is initially contacted late in the final day may not have an opportunity to complete the survey. Models include campaign-level fixed effects and clustered standard errors.

Initial contact is randomly assigned by our software within sampling strata, though if sampling strata are upweighted more of those up-weighted strata may be started earlier in the course of the survey.

Times are broken into three day parts: morning (before 12pm), afternoon (12 pm to 5 pm), and evening (5 pm onwards). These are all local times, but due to our survey protocol at the times of these surveys, that meant that Eastern time zones were texted (locally) later in the day, on average, than were Pacific time zones (for example).