Skip to content

How to Minimize Survey Bias


Looking for new insights to inform your company's decision-making? Survey-based research is a powerful tool when used correctly. Business owners know that making data-based decisions is essential to success, but how do you ensure that you're gathering high-quality data that will help your company reach its goals?

Inaccurate data causes costly blunders in product development, marketing campaigns, and company strategy, eroding customer trust and damaging employee satisfaction. 

So what's a business leader to do? The key is to be aware of the potential for bias in survey-based research and to take steps to minimize it. A dedicated partner like IntelliSurvey expertly programs your surveys so they're on target and reduce survey bias, giving you the best possible data every time. With well-designed surveys that aim to reduce the effects of bias, you'll achieve maximum benefit from your survey investment and gather mountains of reliable and actionable data.

Knowledge is power when it comes to avoiding the pitfalls of bias in surveys. In this article, we'll cover the most damaging types of bias in surveys and how to prevent them from skewing your survey results so you can use your hard-earned data to confidently make decisions for your organization.

Selection Bias and Sample Size Issues

Selection bias, also known as sampling bias, results from unrepresentative samples (a sample is the specific group of people from whom you wish to gather data). We've outlined several leading causes of selection bias below.

Convenience Sampling

This common form of selection bias occurs when researchers pick respondents because they are easy to access, rather than because they reflect the appropriate audience for the topic. It is tempting to reach out to people nearby or who will be keen to answer. However, that very proximity or motivation often correlates with opinions about the topic and can skew the results deeply. 

Let’s take the example of customer satisfaction. If you only survey people who follow your brand on social media, your results will be unreliable. Followers are much more likely to be advocates than the average customer. This survey satisfaction score will be much higher than the real one, because of sample selection bias. To find the real satisfaction score, screen your survey sample starting from the general population and ensure representation with quotas. This will balance the right mix of advocates, neutral and detractors in your study.

The Self-selection Fallacy

When respondents are allowed to self-select into taking a survey, the results aren't trustworthy. Why? Because respondents who self-select are more likely to have a bias (in either direction) toward the survey topic. On top of that, when respondents receive an incentive to complete the survey, some people will be tempted to lie to qualify and get the reward for the project while they do not really belong in your target audience. The answer from these cheaters will blur the results.

To avoid self-selection, use non-leading language until you have confirmed the respondent belongs in your target audience. Don’t disclose the topic of the study in your invitations. In your screener, use neutral questions that don’t let the respondent guess the topic. For example, don’t ask “Have you purchased shoes in the past 12 months? Yes/No” but “Which of the following have you purchased in the past 12 months? Shoes/ Clothes/ A car/ White goods (e.g. fridge, washing machine, etc.)/ Groceries/ Cinema tickets/ Travel tickets/ None”. To make it even better, randomize the order in which the respondents show the list of answers in the second question.

Non-response Bias

When people who do not respond to the survey differ from those who do (especially in ways relevant to the research question), inaccurate data will abound. For example, younger men drop out significantly more than other demographics when the surveys get longer. Even if you start from a representative sample source, you will naturally see a percentage of males that is significantly too low among the people completing a 40-minute survey. Ensuring this percentage is in line can be done with quotas on completes. Carefully defining quotas is key to minimizing non-response and data accuracy.

Insufficient Sample Size

Another cause of inaccurate survey results is when the sample size is too small to extrapolate with confidence to the full target population. The smaller the sample, the more likely the results are far from the metrics you want to assess in the full population due to selection error. The larger the sample, the more robust the results. 

Picking the right sample size is always a trade-off between minimizing the margin of error of the results and practical constraints like feasibility, timing, and costs.

How To Do It Right?

Selecting a representative sample is the gold standard to get robust insights. To be representative, 2 criteria must be met. First, the sample composition needs to be accurate: the sample should have the right mix of respondents, mirroring the target population. Second, the sample size should be large enough.

For example, suppose you are conducting a survey about the online behaviors of American adults. You will need to compose a sample of adults from all 50 states, with the right distribution of ages, genders, ethnicities, and income levels. The sample size for each question and each sub-group analyzed should be at least n=384 to keep the error margin below 5%. 

At IntelliSurvey, we understand representativity. Our expert programmers and field team are here to advise you on fielding methodology that zeros in on the optimal bundle of sample sources for your specific needs. We source from a broad range of panels including consumer, B2B, HCP, and patient surveys in over 50 countries. You can rest assured that your sample is carefully targeted to minimize bias.

What Is Response Bias?

Have you ever encountered a survey question you didn't want to answer with the whole truth? Maybe you feared being judged or didn't want to rock the boat. Now, imagine the same situation but on a larger scale and potentially millions of dollars riding on the answers to those questions. That's the essence of survey response bias.

People want to sound favorable or be seen by others in a certain way. Social pressure or a desire to please the surveyor all come into play. And sometimes, they simply don’t know or remember what their real answer to a question is. Even trust-worthy respondents can give wrong answers when placed in the wrong situation.

There are several ways to overcome response bias to ensure accurate survey results. Let's take a look at some of the most common response bias offenders and how to limit them.

Observation Bias

Observation bias occurs when respondents skew their answers to align with what they think the survey hopes to prove. For example, people might show interest in product development concepts because they’re the subject of a survey, but they wouldn’t have any interest in the product on the shelf. 

Results are also often skewed as a result of unintentional bias in survey questions. For example, a question like  "Don't you think taxes are too high?" leads respondents to think they probably are; while "Do you think taxes are fair?" suggests they are not. Neutral wording using a Likert scale brings better results: “What do you think of the level of taxes?” with answers on the scale from 0=” Significantly too low” to 10=” Significantly too high” and a mid-point at 5= ”Fair”.

Best practice: Make sure your questions are clear, concise, and free of loaded language. It doesn't hurt to remind respondents that they should answer honestly, without worrying about what the researcher might want to hear. Introduction screens like “There is no right or wrong answer, we want to understand what you really think” can be effective.

Avoid cues in the survey that point to the study goal. If the respondent cannot guess what you are trying to prove, they cannot skew answers to please you.

Social Desirability Bias

This is when respondents submit answers they think are the most socially acceptable. For example, someone might claim to recycle when they don't or say they never spread gossip when they do it all the time. Unchecked, this bias also results in inaccurate information about important metrics like age, income, or education level.

Best practice: Conduct surveys anonymously and design questions neutrally to minimize the impact of social desirability bias.

Acquiescence Bias

Some people socially tend to agree more than they disagree. This is true in general, and in particular of survey respondents. While acquiescence bias is present to some degree in every culture, it’s stronger in Asia and reaches its apex in the Middle East. This can cause meaningless differences across markets in multi-country projects.

Best practice: For countries with strong acquiescence bias, use question formats that do not ask them to agree, like rankings, contra-scales, or MaxDiff. 

You can replace agreement batteries by using contra-scales with statements on both sides and a scale in the middle. Instead of asking “Price is more important than quality” on a scale from Strongly Disagree to Strongly Agree, place on the left “Price is the most important” and on the right “Quality is the most important” (this is even more effective if you randomize the sides of the statements).

Contra-scales, ranking questions, and MaxDiffs (which also establish a rank order) all remove acquiescence bias. They also allow for international comparison across cultures.

Reduce Survey Bias, Every Time

At IntelliSurvey, we’ve developed best practice methodologies through the thousands of surveys we run every year, ongoing research reviews, as well as our own research on research exercises. We recommend validated designs tailored to each study. Our teams will set up, monitor, and manage your project to minimize the risk of biases threatening your data and provide a solid foundation for decision-making. 

Contact us today to discuss your upcoming research studies.

Subscribe to our Blog