Introducing Dynamic Response Rate Adjusted Stratified Sampling
All survey researchers know that response rates are not identical across different groups of people. The most common solution to this issue is to weight or otherwise calibrate data after the survey is completed, but researchers have also shown there are benefits for accounting for the expected response rate in the sampling procedures. While there are some stable patterns of non-response within and across mode, these patterns can also change in response to the political environment. Additionally, not all researchers have enough data to develop a well-calibrated predictive model of survey response in advance of fielding a survey.
To overcome these limitations, and to help researchers account for differential response rates across sampling strata in real time, we at Survey 160 have developed a tool we call Dynamic Response Rate Adjusted Stratified Sampling, or DRASS for short. The details of this algorithm are described in this paper, but the intuition is straightforward: observe the response rates of each sampling stratum in real time, and as texting agents draw new samples from the database, up-weight low-responding groups and down-weight high-responding groups. The expectation is that this should produce samples that are closer to target distributions, lower design effects after weighting, and more accurate data, though (because more respondents in low-responding groups are contacted) at somewhat higher costs.
To test how well DRASS achieves these goals, we fielded an experimental survey of five elections across three states in November of 2022. We randomly assigned respondents to either the DRASS condition, or left this algorithm off. We constructed 2 sets of weights, one that simply weighted back to the sample distributions of the stratifying variables, and a set of “advanced” weights that also included weighting factors for educational attainment, based off of Current Population Study November supplement measures of registered voters, and the interaction of Age cohort and Gender.
As expected, there was a marginally higher cost per completed interview for the DRASS treatment group. Also as expected, the design effect was lower (looking at either set of weights) for the DRASS treatment group. After accounting for the effect of the design effect on the effective sample size, though, the cost per effective complete was actually lower with DRASS activated.
We also examined the effect on accuracy. With the “Advanced” weights, we see that the DRASS condition not only beats the control condition, but had a lower absolute error on the Democrat minus Republican Margin (or put another way, greater accuracy) than the median poll in the FiveThirtyEight database for those contests.
We generally recommend using DRASS to produce samples that are closer to the target distributions of sampling strata. If your organization does have the data and resources to put together a survey response predictive model and use that model in constructing your sampling, DRASS can be even more efficient, as that sampling ensures that there are enough respondents in low-responding groups in order to neither exhaust the sample nor require the purchase of unnecessarily large samples in the first place.
