Rigor Matters: Representative Stimulus Assignment

Representative sampling is critical for a survey total respondent pool. If a survey sample does not align with the modeled population, generalization is compromised.

Representativeness at the stimulus level as well. Bias can occur when the in-survey population assigned a stimulus (e.g., brands, ads, categories, etc.) does not align with its target population. Accommodating balanced stimulus assignment often involves the careful application of purpose-built toolkits.

Survey Use Case Example

Consider a simple case, an ice cream survey with a deep dive into one of the flavors that respondents have recently consumed (Vanilla [70%], Chocolate [50%], and Strawberry [30%]). Imagine that in this world, every strawberry consumer also ate chocolate ice-creams but never any vanilla one. The crossed-use table might look like this:

Now, imagine in this world, researchers run an ice cream survey with a "deep dive" for one flavor. They assign each respondent a single flavor which this person consumed, and ask detailed questions about it. In this case, the assignment is based on lowest counts: if a respondent has consumed several favors, they are assigned to the qualifying response with the least answers so far. During field, when the overall N was 100, the assignment table might look as follows.

Most strawberry-qualifying respondents are assigned to it, because it is the lowest incident flavor. Those who like both vanilla and chocolate are predominantly shunted to chocolate. As a result, vanilla deep divers are less likely to be chocolate aficionados, and chocolate deep divers are less likely to be fans of strawberry. Research findings elicited from this flavor assignment will be imperfectly representative.

Addressing Complex Challenges

This is the simplest of examples. Real stimulus sets may have complex correlational relationships. These effects can profoundly skew stimulus populations.

Addressing them well calls for a multitude of tools that sometimes require concurrent access to and intense calculation of previously gathered survey respondents. This needs to happen quickly so as not to inconvenience respondents. The stimulus assignment control utilities that IntelliSurvey supports seamlessly include:

These techniques (and others) can be in combination.  Our platform supports multiple tracked assignment schematics, which can be either independent or interdependent.

Of course, any deterministic process can elicit skewed in-sample stimulus assignment. To that end, we enable the injection of randomness via respondent-level parameterized lotteries.

Rigor Matters

If you wrestle with these challenges in your practice, please get in touch. We love to geek out with fellow researchers!