Balancing the Scales: Using Survey Quotas for Representative Research

Survey quotas are vital for gathering representative, accurate research results. Without them, your data collection will lead to demographic over- or under-representation, with disproportionate results. 

For example, let’s say you survey the general population about a given topic. Without quotas in place, you’re subject to whoever passes the screener and fills out the survey. You run the survey, and low and behold, 75% of the responses are from women when you really wanted an equal representation of men and women. To correct this, quotas are used to ensure balanced sampling.

What we’ll cover in this post:

• What are survey quotas?
• Why quotas matter
• Types of quotas
• How to use quotas
• Best practices

What Are Survey Quotas?

Quotas are predetermined targets for the number of respondents from different, mutually exclusive subgroups. They require a specific number or percentage to be met, and once that number has been reached, the quota is considered fulfilled. From that point, any new respondents who meet the criteria will be excused from the survey.

One way to think about quotas is as color-coded buckets of varying sizes. Respondents are also color-coded and can only be placed in the bucket that matches their color. Once a bucket is full, it can no longer fit any more people, even if their color matches. When that happens, any additional respondents are turned away because the bucket has no more room.

Why Quotas Matter

Quotas play a significant role in survey operations. They help ensure sample representativeness, mitigate sampling bias, and facilitate comparisons across different segments of the population. In doing so, quotas help enhance the validity of research findings.

Quotas also help achieve statistical reliability. By setting appropriate quotas, you can confirm that the sample size for each subgroup is sufficient for meaningful analysis and interpretation of the data. 

Types of Quotas

Multiple types of quotas are available to help you reach your goals. Many quotas are based on demographics, firmographics, geographics, or behavioral data, though they can also be based on answers to specific survey questions. 

Overall Response Quota

This quota limits the total number of responses for the entire survey. 

Example: Your survey target is n=200. Anyone who passes the screener can participate. Once 200 responses have been completed, no further respondents will be allowed to participate. 

Simple Quota

A simple quota has a single qualification that needs to be met.

Example: Your survey target is still n=200; however, you want 50% of the respondents to be women. In this instance, you would set a quota for 100 women. Once 100 women have completed the survey, the quota has been met, and no further women will be accepted to participate.

Compound Quota

Compound quotas require a respondent to meet multiple criteria to fulfill the quota. 

Example: Your survey target is still n=200, and you want 50% to be women (criteria #1) who also have children (criteria #2). In this case, you would still set a quota for n=100, but the difference here is that BOTH criteria must be met in order to fill the quota. 

How to Use Quotas Effectively

Plan Your Quotas

Quotas can be extremely useful in ensuring a representative sample of key groups. When planning your quotas, ask yourself the following questions:

• What is your target population?
• Which demographics or characteristics are most relevant?
• How do population proportions impact your quota parameters?

Determine Quota Sizes

To determine quota sizes, start with your target population. Identify which characteristics are important to your survey (e.g. age, gender, geographic location, or other attributes) and set your quotas accordingly. You can leverage existing information about your target population, such as census data, market research reports, and previous survey results, to inform your quota composition.

It is also important to consider sample size requirements. You’ll need to calculate the minimum sample size needed for each quota group to reach statistically significant results. You can use online calculators or statistical formulas to determine sample size based on the desired confidence level and margin of error. 

Be sure to balance the need for representative samples with practical constraints such as time and budget. This will help you determine how many respondents you can realistically survey within each quota group. 

Set Up Quotas In Your Survey Tool

Quota setup will vary from platform to platform. Generally speaking, the more complex your quota configuration, the more advanced your technology will need to be. 

With IntelliSurvey, you can leverage our team for programming or do it yourself. IntelliBuilder, our drag-and-drop survey builder, is great for non-technical users who want to add simple quotas. For tech-savvy users who want to code quotas themselves, our plain-English SPL (survey programming language) allows you to code both simple and more complex quotas.

Here's what it looks like to add a gender quota in IntelliBuilder. 

Using the SPL is equally easy, as shown in this example of a pizza survey. 

Monitor and Manage Quotas In Field

Once you’ve set your quotas, it’s important to monitor them during fielding. By monitoring quotas, you can determine which ones are being filled quickly and which may require additional effort to reach. Sometimes, quotas may need to be tweaked to match the desired sample composition and maintain representativeness. 

Tip: Don’t wait until after fielding to clean your data! As poor-quality respondents are removed, you may no longer be able to fill your quotas. At IntelliSurvey, we use CheatSweep™, our proprietary data cleansing system, as well as live review during fielding to ensure bad data and bad actors don’t make it through to your final results.

To seamlessly monitor quotas and make any necessary adjustments on the fly, simply access the Quotas applet and apply changes as needed. 

Best Practices for Using Quotas

Conduct pre-survey research to understand the target population. Collaborate with stakeholders to ensure you have accounted for various data needs and requirements.

Define quotas clearly: Clearly define the quotas based on relevant demographic or other criteria. This could include age, gender, income level, geographic location, etc. Always ensure that quotas align with the research objectives and target population. 

Set realistic quota targets: Based on the demographics of your target population, set reasonable and achievable quota targets. 

Monitor quotas closely: Use real-time reporting tools to monitor quota rates, track progress, and identify imbalances or underrepresented groups. Continuously review and adjust as needed based on data trends and feedback. 

Leverage real-time data cleansing to ensure poor-quality respondents are discarded and are not included in your quotas. 

Use quota sampling techniques: Employ quota sampling techniques such as sequential sampling or stratified sampling to ensure quotas are filled evenly across different demographic groups. This helps avoid oversampling or undersampling particular subgroups. 

Plan for contingencies: Have contingency plans in place in case quotas are not filling as expected or if certain groups are difficult to reach. This could include blending additional sample sources, increasing incentives, or extending the time the survey is in field. 

Mastering the Success of Your Survey Quotas

Implementing survey quotas helps ensure the representativeness of your data, allowing you to collect balanced responses across various subgroups while preventing demographic overrepresentation or underrepresentation. 

With over two decades of experience in field management, IntelliSurvey stands ready to assist you with broad and niche audiences alike. To learn more about how we can elevate the fielding of your next study, please contact us.