How Texting Around the Holidays Impacts Survey Performance
A common question we get at Survey 160 when trying to decide the fielding window for a text message survey is whether texting around major U.S. holidays is likely to impact performance. In short, the data suggest that texting should be paused on most major U.S. holidays, but not Election Day. If you are thinking about surveying over Christmas and New Year’s Day, we advise against it
When measuring the performance for a survey, we focus on two key metrics: the engagement rate and the completion rate. For clarity, the engagement rate is the percentage of people that respond to the intro text message asking them to participate in the survey (regardless of the substance of the response) and the completion rate is the percentage of respondents that complete the survey.
To assess the effect of holiday timing on these outcomes, we pooled metadata from 108 of our fielded surveys from 2021-2024 that overlapped one or more holidays, and then compared within-survey performance on holidays vs. before or after holidays (the fact that these comparisons are being made within surveys means you should avoid comparing, for instance, the rates on July 4th to those on Juneteenth, as there may be very different surveys in the field over those windows). This analysis suggests that for the vast majority of major U.S. holidays the engagement rate for a survey declines by 1-percentage-point on the day of the holiday relative to the average before the holiday. As an illustration, prior to the holiday, the average engagement rate for the surveys that fielded through Juneteenth was 12.5%. On the day of the holiday it declined to 10.6%. The charts below show a snapshot of the results for a few of the holidays that were included in the analysis.
The completion rate, one of the most important metrics of performance for a survey, followed a somewhat similar pattern with an average decrease of .5-percent-points on the holiday relative to the average before it. Further illustrating this point, prior to the holiday, the completion rate for the surveys that fielded through Juneteenth was 2.6%. On the day of the holiday, the figure was 2.1%, an overall relative decrease of 19.24%.
One interesting exception to this rule is Election Day, where the data consistently shows noticeable increases in the engagement and completion rates for the surveys. On average, for the surveys that field through Election Day, we see a .2-percentage-point increase in the engagement rate on the day of the event compared to the baseline before it.
Similarly, we see a .5-percentage-point increase in the completion rate on Election Day compared to the baseline before it.
Overall, we recommend pausing surveys on most major U.S. holidays. The notable exception to this is on Election Day where we see an increased engagement and completion rate.
