Survey Data Integrity: Moving Beyond Door Screening and Speeding

In the realm of survey research, ensuring data integrity is paramount. Door screening and speeding have long been employed to identify potential survey fraud. However, their effectiveness is limited, capturing only a fraction of the problematic data. In this blog post, we delve deeper into the shortcomings of relying solely on door screening and speed and explore advanced techniques IntelliSurvey employs to enhance survey data integrity.

Traditional Fraud Detection Limitations in Survey Research

Speeding Challenges

While speeding is a common practice to filter out respondents completing surveys too quickly, it can have unintended consequences. Speeding measures often exclude fast readers and individuals who genuinely provide thoughtful responses in a shorter time frame. This exclusion may inadvertently bias the data and limit the representation of certain demographics, such as younger individuals or males.

Another limitation of traditional speed flags is their failure to account for the different paths respondents take within the survey. The traditional rule of removing respondents below a third of the median survey time can create a bias. For example, if some respondents skip a 10-question loop while others go through it multiple times, the traditional rule may disproportionately remove respondents who did not go through the loop. To address this bias, IntelliSurvey recommends focusing on the time spent per question rather than the total survey time. This approach ensures a more balanced cleaning process across different paths and minimizes potential data bias.

Door Screening Challenges

Door screening covers a number of digital fingerprinting techniques based on IP, device, and cookie data to identify fraud.

One of the challenges lies in the complex nature of fraud patterns. Fraudsters are constantly evolving their tactics to bypass door screening measures, making it difficult to capture all instances of fraudulent activity.

Additionally, door screening may inadvertently flag legitimate respondents as potential fraud cases due to factors such as shared IP addresses or device usage. This can result in the exclusion of genuine participants, leading to a loss of valuable data and potentially affecting the representativeness of the sample.

Limitations

Speeding and door screening are still very useful cleaning methods when appropriately applied. These methods are particularly valuable in removing live a first layer of fraud. However, combining them only captures about half of the fraudsters, calling for more advanced techniques.

Advanced Fraud Detection

Manual Checks: Seeking Patterns and Contradictory Answers

One of the key aspects of fraud detection is seeking patterns in respondent behavior. Fraudsters try to guess the topic and qualify, and they select more options than a real respondent to increase their chances. When designing questionnaires, including questions that catch seeking patterns is a great tool to identify fraud.

Other manual traps can be designed, such as contradictory answers. However, this task can be delicate to avoid disturbing the flow of the survey for good respondents who resent too obvious quality checks.

Quantitative Patterns: In-Survey Bad Behaviour Metrics

Proprietary survey platforms allow for collecting thousands of in-survey behavior data points. Does the respondent scroll through all answers? Click in straight lines? Move their mouse in a human way on the screen? Type faster than a computer? Copy-paste a lot?

All of these indicators can be leveraged to assess whether the respondent is human and how much attention they are paying to the survey.

Open-End Patterns: Assessing Response Quality

Individual respondent answers to open-ends are also a good indicator of data quality. Do they demonstrate an understanding of the question? Show emotions when they should? Are they written with a realistic tone? Have some typos? Are they in the expected language?

Bad quality can also be flagged by identifying too-similar answers or answers that exhibit signs of being generated from a common source.

In Conclusion

Relying solely on door screening and speeding to identify survey fraud is insufficient to ensure data integrity. By recognizing the limitations of these traditional methods, researchers can explore advanced techniques to enhance survey data integrity. IntelliSurvey's commitment to robust fraud detection, incorporating a blend of automated algorithms and human expertise, empowers researchers to mitigate the risks associated with fraudulent responses. 

Don't settle for surface-level measures that compromise the reliability of your research findings—partner with IntelliSurvey to safeguard against fraudulent responses and unlock the true potential of your survey data.