Comparing Panel Provider Quality & Unveiling Survey Fraud

Have you ever wondered how survey fraud rates vary across sample provider types? While there is no one-size-fits-all answer, our recent experiment showed that the impact can be huge. In fact, some mainstream sample sources exhibit over double the amount of fraud than higher-quality alternatives.

Our Survey Panel Provider Quality Experiment

IntelliSurvey recently conducted an 8-minute survey to explore the quantitative research respondent experience in the United States. 

For observation purposes, we allowed all respondents to complete the survey, in contrast to our usual live cleaning practices. A total of 2,000 surveys were collected from four major sources in the industry:

Panel A: a large, established mainstream panel company using managed services

Panel B1 (Direct): a large, tech-first panel company using managed services

Panel B2 (Aggregator): the same panel provider as B1 using their panels through an aggregator

Aggregator C: an aggregator sample, excluding B1

Once collected, we identified fraudulent respondents through a combination of IP and device finger-printing, automated in-survey behavior checks with our proprietary cleaning algorithm CheatSweep™, as well as manual consistency checks and open-end reviews.

The Results

On average, we found 31% fraud across all panel providers. However, not all sources were equal - that rate varied massively between vendors.

As shown in the chart below, aggregator quality was notably inferior compared to going direct to panels. When purchasing direct from a vendor, the data quality was significantly better, thus minimizing the time needed for data cleansing. Interestingly, the biggest brand (Panel A) did not offer the best quality. Panel B1 outperformed it.

It's worth noting that we did not explore any third-tier vendors in this study. However, in our rejected vendor pilots, we routinely encountered fraud rates exceeding a staggering 80%!

How IntelliSurvey Can Help

Do you have a clear understanding of the fraud levels associated with your panel sources? At IntelliSurvey, we have the expertise and tools to help you navigate this complex landscape. Make sure the data in your survey is accurate by getting in touch with us today.