Using texting with other modes to optimize for cost and accuracy

Most of our work at Survey 160 involves fielding the text-message arm of multi-mode surveys. As a result, we are frequently engaged in research on how to optimize the fielding of multi-mode surveys. One common approach is to use a single sampling frame, like a voter file, and contacting some voters with live phone calls and some with text messages. But who should you contact with which mode? And in what order?  

To better advise our clients, in the run up to the 2023 elections, we conducted a survey fielding experiment just before the 2023 Kentucky gubernatorial election. We randomly split a sample. Group A received two rounds of texts to cell phones and IVRs to landlines, while Group B received two rounds of live calls. (When respondents had both a landline and cell phone on the file, we randomly selected which to use.) After these two rounds, we switched mode for a third round of contact, using the opposite modes to backfill strata with unmet quotas. 

We presented this research at the 2024 meeting of the American Association for Public Opinion Research (AAPOR). Below, we highlight some takeaways from this research, including the relative costs, accuracy, and sample composition of different approaches. (TL;DR: Prioritize texting, but consider other modes to reach voters without cell phones, and/or to backfill strata with unmet quotas)

Takeaway #1: Text-to-web (T2W) and interactive voice response (IVR) were both dramatically cheaper than live calls to cells and landlines, respectively. 

There were two reasons for this. First, the response rates for T2W in this survey was actually higher than live calls, which may be an outlier. 

Additionally, the base costs per contact are much lower for T2W and IVR. Even though our texts are all sent by human operators, a given interviewer can complete many more surveys per interviewer hour by text. As a result, there are much lower costs for T2W and IVR compared to their live caller counterparts. Live calls to cells are particularly costly because of the regulatory obligation to hand dial rather than use predictive dialer technology.

Takeaway #2: There’s meaningful differences between cells and landlines, regardless of mode. 

Across several different dimensions, we saw similarity across mode, but differences across phone type (cell or landline). For starters, the gender match to a selected respondent is higher for cells than landlines. This is not entirely surprising, since there may be multiple residents of the same household sharing a single landline. 

We also see meaningful differences in resulting sample composition, with landline respondents much more likely to lack a college degree, but much less likely to be Gen Z or Millennial.

Takeaway #3: In addition to costing less, texting was dramatically more accurate than live calls when comparing weighted samples.
However, perhaps because of the aforementioned differences in landline and cell phone responses, adding IVR to text-to-web improved the accuracy of the estimates. And additionally backfilling strata with unmet quotas using live calls improved the accuracy even more. 

Takeaway #4: Counterfactual sample allocation rules using voter file variables could improve accuracy even more, though this needs further testing.

One thing this experiment allows us to do is to estimate, counterfactually, what would have happened if we texted one group of demographically-defined voters and called the rest, or vice versa. For example, if you text young people and call old people, what happens? We call these “counterfactual sample  allocation rules”

And indeed, we find that such counterfactual  sample allocation rules would have improved accuracy on margin even more in this sample, including (a) text + IVR to respondents under 50, with live calls to respondents over 50, (b) text + IVR to white respondents, with live calls to non-white respondents, and best of all, (c) text + IVR to respondents who did not vote in 2020 or only voted in the general election, while calling people who voted in the primary election. This final counterfactual sample allocation would have produced essentially zero error on the Democrat-Republican margin. 

However, the nature of such testing is likely to produce some outliers if we test enough alternative rules. As such, we recommend further experimentation to do confirmatory testing.