Can we rely on survey data?

 

Public opinion surveys ahead of the 2020 election were the most inaccurate in 40 years. 

 

In trying to ascertain why, the American Association for Public Opinion Research noted that 2020 was not like 2016, when the failure to include enough low education voters led to an undercounting of Trump supporters.

 

Instead, the issue with 2020 was that some voters (Trump supporters) declined to participate in polls altogether. (Zitner, 2021)

 

Net, net poor sample selection was responsible for both errors.

 

Unfortunately the problem with sample selection has increased ever since people started using cell phones and the once highly reliable method of sampling via phone became obsolete.  Research companies have responded by moving some polling online.  But this presents its own set of problems.

 

In testing results from phone versus online research (the mode effect), Pew Research noted that phone respondents were more likely to give extreme responses.  They also found that online survey respondents tend to favor the first answer choice while people on the phone tend to favor the last choice that they hear.  

 

Overall, discrepancies were greater for more attitudinal questions, e.g. assessing the quality ones social life.  (McGinty, 2021)

 

Even if you do get the correct sample, there are still a multitude of ways customer surveys can go wrong.  These include not asking enough questions, asking too many questions, or asking the wrong question.  Like - how likely are you to recommend us? (Dholakia, 2021)

 

Then there's the question of compensation.  That $1 you sent me with the survey is insulting not motivating.  And telling me that I will be entered in a contest makes me want to ask: "Do you get paid in lottery tickets?" 

 

Which brings us back to the original question.  Is survey data usable?  And how? 

 

Do you participate in surveys?  Which ones and why?  If you opt out of surveys, do you shun all surveys or just some?  Have you participated in online research?   Do you think you behaved differently than you would have on the telephone?  Or in person (focus groups)?  

 

Have you been paid to participate in surveys?  Would it affect your participation rate if you are paid?  Finally, how carefully are you looking at the data that you are using?  Have you been conscious of sample size and margin of error?

 

 

Zitner, A. (2021, May 13)  Trump-Biden Was Worst Presidential Polling Miss in 40 Years, Panel Says.  wsj.com.  Retrieved September 7, 2021, from  https://www.wsj.com/articles/trump-biden-was-worst-presidential-polling-miss-in-40-years-panel-says-11620909178

 

McGinty, J. (2021, April 30) What Happens When Opinion Polls Move Online.  wsj.com.  Retrieved September 7, 2021, from  https://www.wsj.com/articles/what-happens-when-opinion-polls-move-online-11619775001

 

Dholakia, U. (2021, May 23)  The Five Biggest Mistakes Companies Make With Customer Surveys.  wsj.com.  Retrieved September 7, 2021, from  https://www.wsj.com/articles/the-five-biggest-mistakes-companies-make-with-customer-surveys-11621632499

Why are researchers asking the wrong question?

 

Words matter.  Just changing one word in a research question can vastly impact the results. 

A definitive study from Loftus & Palmer in 1974 demonstrated that changing the descriptor of an accident from "bumped" to "smashed" changes the testimony from eyewitnesses.  (McLeod, 2014)

 

You can check out a recreation of the experiment from Brain Games here...

https://www.youtube.com/watch?v=qQ-96BLaKYQ

 

I recently had the opportunity to interact with a customer service person at an organization I do business with.  The interaction which was above and beyond the typical contact went well, so I was prepared to provide feedback to the company.

 

When I opened the email the company sent, I expected to see the question: "How satisfied were you with the customer service that you received?"  I would have responded with a 10 out of 10. 

 

But the email they sent me asked - "Based on your recent call with Company X customer service, how likely are you to recommend Company X to a friend or colleague?"  My answer to that question is 0 out of 10.  

 

Why?  Because you don't pay me to market your services and I don't work for free.

 

I am not sure why companies don't get that. 

Because I didn't want my answer to reflect poorly upon the person that I spoke with, I chose not to answer at all.

This is not an isolated incidence.  I am not sure when companies decided that if they only have one question to ask this would be the one.  

But I am sure it's not providing them with any meaningful feedback.

 

McLeod, S. (2014)  Loftus & Palmer. simplypsychology.org.  Retrieved July 6, 2021, from

https://www.simplypsychology.org/loftus-palmer.html

Would you date someone who doesn't recycle?

 

According to a survey from decluttr, 47% of young Americans would not be in a relationship with someone who doesn't recycle.

 

The research was fielded in 2020, and the sample was comprised of 1,332 Millennials and Gen Z's. (2020)

 

The folks who fielded this research were mostly interested in recycling of  E-waste.  And report that 60% of young people don't know what e-waste means.

 

I'm confused.  They can't be recycling e-waste if they don't know what it is.  So does that mean they don't date?

 

Often companies with an agenda do some superficial research to support their point-of-view.  Do you think that is what happened here?  What do you think their agenda is?

 

Do you think that it makes sense to include both Millennials and Gen Z's in the study?  Have you noticed differences between these generations?

 

Meanwhile a Harris poll of 1,000 American adults done on behalf of ISRI (Institute of Scrap Recycling Industries) showed that only 33% of adults 18-34 always recycle, while 48% of adults 35+ always recycle. (Leblanc, 2019)

 

If that's true then the data about dating makes even less sense.

 

So, how do you decide if research is worth using?  Have you seen issues with conflicting research before?  How do you choose which to believe?

 

 

(2020) How are Americas biggest technology users approaching E-waste?  decluttr.com.  Retrieved February 1, 2021, from  https://www.decluttr.com/us/store/e-waste

 

Leblanc, R. (2019, March 12)  Recycling beliefs vary between generations.  thebalancesmb.com.  Retrieved February 1, 2021, from  https://www.thebalancesmb.com/who-recycles-more-young-or-old-2877918