The Sample Bubble in the Form of a Conference Recap
Blog | March 27, 2024
By Danielle Chinitz
As I arrived to SampleCon 2024 at the beautiful Lake Oconee in Georgia, I was excited to meet clients from overseas, new potential partners, and prospects. Day one was filled with lots of networking and face-to-face time with longtime friends from the industry. Between all the networking a few core themes emerged from the content.
Here’s a quick recap of my highlights for you from the only sample-focused conference in the industry.
The Respondent Experience is (still) in Focus.
I knew I wanted to attend one of the sessions aptly titled “The Good, the Bad, and the Ugly of Data Insights.” The panel mainly consisted of first timers at the conference, with a nice mix of buyers and sellers of sample. The content of the discussion was focused predominantly on how we treat respondents and how respondents have continued to be commoditized. Unfortunately, it felt to me like this was anything but a new conversation. In my experience, this dates back a decade, if not more. The people with the power to change things for the better are still just ruminating on the current state.
Is it time to raise CPI?
In this session, a case was made to raise CPIs for quant work. One proposal suggested survey incentives should align with qualitative incentive models. The thought was that the increased CPI will positively impact fraud and fuel the market research industry with the corresponding larger budgets. This can produce more experimentation, specifically surrounding AI.
Wait a minute, though. Paying respondents more will reduce fraud? Also, I thought CPIs were going up to cover incentives. How then does the money become available for AI experimentation?
Maybe the conference name Sample”Con” means something else entirely.
Data Quality
Day three was all about global data quality initiatives. This panel seated industry veterans across Research firms, ResTech, and panel companies. Given the discussion topic and multiple collaboration sessions following the speakers, I looked forward to this day most.
Let me share some claims made during this session. You may want to brace yourself:
- Normal dupe rates within panel companies was purported to be 30%.
- “Internet traffic” -> low dupe, high fraud
- Belief that definition of fraud is “subjective”
- We need to find a balance with false positives. “Being OK with a little fraud means more completes.”
- Fragmentation causes loopholes – one system should control all fraud.
- Privacy laws increase fraud; GDPR removed the use of super cookies*
I’ll refrain from commenting on every point above, but this list is a perfect example of the sample bubble not being connected to researchers’ realities. They’re saying the quiet part out loud here.
Let’s look at a few examples of where OpinionRoute’s view differs from the conference conversation.
We know that dupe rates should never be 30%, unless you create faulty Field Plans.
30% duplicate rates are absolutely not the norm. We have been tracking this data for a decade across multiple clients, countries, and vendors – averages are far below 10%. However, you can see an acceleration of these levels if you blend the overlapping vendors. In other words, don’t use a vendor who is also using that same vendor. Common sense to researchers.
Researchers do not need to lower quality standards to appease sample firms.
Being “OK with a bit of fraud if it helps conversion rate” as a concept shows why today’s researcher cannot rely on the sample vendors alone for data quality. The ‘conversion is king’ thought process, which has become increasingly popular over the years, overshadows the fact that quality is the leading factor in the trust foundation of our industry. We cannot and should never compromise on that to the supply chain economic model.
Privacy Compliance Builds Trust with Real Consumers.
Lastly, one should never point the finger at privacy laws as a risk to our ability to deliver quality insights. Any reputable fraud-prevention technology in 2024 is not cookie-based anyway. Real people clear their browsing history regularly because they do not want to be tracked or targeted constantly. There is no correlation between cleaning your history or removing cookies and the intent to commit survey fraud.
Link Security is great, but not possible for most MRX teams.
During a breakout session, I decided to join the survey link security discussion. Again, there was a diverse set of companies sitting around the table, most with many years in the industry. The goal of this team was to draft what standardization surrounding survey link security for online quant work will look like. Our output included link hashing, link encryption, and server-to-server communication. There was consensus around the table that this was a good idea. But, if the people in the room represented 10% of the industry, how do we drive this mainstream? (by the way, we do this for Market Researchers already).
Maybe I’m too indoctrinated into OpinionRoute’s relentless focus on the MRX firm, but even as a sample veteran, the hot takes at this show stopped me in my tracks. I’ve never felt more secure about our purpose than I did leaving Lake Oconee.
We openly welcome collaboration for lunch-and-learn sessions. If you feel like your team can benefit from a session, please reach out so we can build together.