Public opinion surveys ahead of the 2020 election were the most inaccurate in 40 years.
In trying to ascertain why, the American Association for Public Opinion Research noted that 2020 was not like 2016, when the failure to include enough low education voters led to an undercounting of Trump supporters.
Instead, the issue with 2020 was that some voters (Trump supporters) declined to participate in polls altogether. (Zitner, 2021)
Net, net poor sample selection was responsible for both errors.
Unfortunately the problem with sample selection has increased ever since people started using cell phones and the once highly reliable method of sampling via phone became obsolete. Research companies have responded by moving some polling online. But this presents its own set of problems.
In testing results from phone versus online research (the mode effect), Pew Research noted that phone respondents were more likely to give extreme responses. They also found that online survey respondents tend to favor the first answer choice while people on the phone tend to favor the last choice that they hear.
Overall, discrepancies were greater for more attitudinal questions, e.g. assessing the quality ones social life. (McGinty, 2021)
Even if you do get the correct sample, there are still a multitude of ways customer surveys can go wrong. These include not asking enough questions, asking too many questions, or asking the wrong question. Like - how likely are you to recommend us? (Dholakia, 2021)
Then there's the question of compensation. That $1 you sent me with the survey is insulting not motivating. And telling me that I will be entered in a contest makes me want to ask: "Do you get paid in lottery tickets?"
Which brings us back to the original question. Is survey data usable? And how?
Do you participate in surveys? Which ones and why? If you opt out of surveys, do you shun all surveys or just some? Have you participated in online research? Do you think you behaved differently than you would have on the telephone? Or in person (focus groups)?
Have you been paid to participate in surveys? Would it affect your participation rate if you are paid? Finally, how carefully are you looking at the data that you are using? Have you been conscious of sample size and margin of error?
Zitner, A. (2021, May 13) Trump-Biden Was Worst Presidential Polling Miss in 40 Years, Panel Says. wsj.com. Retrieved September 7, 2021, from https://www.wsj.com/articles/trump-biden-was-worst-presidential-polling-miss-in-40-years-panel-says-11620909178
McGinty, J. (2021, April 30) What Happens When Opinion Polls Move Online. wsj.com. Retrieved September 7, 2021, from https://www.wsj.com/articles/what-happens-when-opinion-polls-move-online-11619775001
Dholakia, U. (2021, May 23) The Five Biggest Mistakes Companies Make With Customer Surveys. wsj.com. Retrieved September 7, 2021, from https://www.wsj.com/articles/the-five-biggest-mistakes-companies-make-with-customer-surveys-11621632499
5 comments:
Survey data can be extremely useful when done correctly. It is crucial for market researchers to present the right questions to the right audience. In my opinion, one of the most important things market researchers can do is screen their potential respondents before enrolling them in a survey. This will allow them to "weed out" the weak applicants and recruit the most ideal respondents.
I actually really enjoy participating in surveys and focus groups! I have strong opinions and like to express them. I only participate when the subject matter is important to me and I feel as though my opinion would be of value to the research company. I have participated in online, telephone and in-person surveys on various topics from skincare products to lifestyle habits. If a subject matter is not important to me, I likely wouldn't bother taking the survey.
Personally, I feel as though it is significantly easier to give an unfair or extreme response when participating in online research. I also find it very easy to get distracted or quickly click through questions. I feel as though I give the most genuine feedback when I am participating on the phone or in-person because I am more focused on the subject matter and feel a stronger responsibility to provide the highest quality responses.
I have also participated in paid surveys and focus groups. Getting paid has a positive impact on my participation rate, as long as the price is right. If I am only being offered a couple of dollars for a 20+ minute survey, then I likely wouldn't participate or would feel a sense of resentment, which would not produce quality results. When I am getting paid, it feels like I am doing a job where I have to produce my best work.
I have never really been conscious of sample size or margin of error, but this is definitely something I will consider during future surveys!
Surveys can be tricky. Especially the ones online. I have never participated in one but I believe online surveys are not 100% accurate. If I was ever in the position to have to do one I don't know how focused I would be to give the correct answers especially for a topic that I am not interested in. In-person surveys are more accurate in my opinion. If a person asked me the questions face to face I would more likely pay more attention to my answers.
Moving on, paid surveys are a great motivator for people to spend time to do complete the questions but it all depends on the amount of money.
I did some research and I found this:
"A survey’s response rate has always been viewed as an important indicator of survey quality. However, recent studies have shown that surveys with lower response rates (near 20%) had more accurate measurements compared to surveys with higher response rates (near 60 or 70%)."
I believe that the surveys with lower response rates are more efficient because this means that the researcher addressed the right audience. This 20% actually cared to respond to the survey and provide truthful answers whereas the 60-70% probably answered the survey hastily.
https://surveyanyplace.com/blog/average-survey-response-rate/
In a world of instant gratification, surveying tends to go against the common-day objective of greed. Buzzfeed, an entertainment news source, has an array of surveys on their website in which the results are visible to the viewer immediately after they respond. Although, the information is not finalized as the survey is continuous, the individual is instantly gratified knowing if their preference for olives is one of popularity or not.
But, government issued surveys don’t tend to work that way and it appears that most people yearn for an incentive (often immediate) for their participation. I wonder if a reformed way of data collection could be devised resulting in more accurate data and individuals who are willing to participate and not remain in a state of habituation? How can we express the urgency of survey participation in the modern world?
Target audiences sometimes are hard to represent a whole group; many methods are used for survey data collection and statistical analysis. However, there are some cons to the survey date; firstly, the data is collected from a sample of consumers. Secondly, just like professor PJ mentioned in the blog, even if the sample is correct, other ways may cause the surveys wrong such as asking wrong questions. Finally, in my opinion, survey data might be helpful sometimes but rely heavily on it doesn't seem like a wise decision.
Surveys aren't reliable. Whether it's displaying data of a certain demographic, to possibilities of what can a company do better over time, I dont think surveys en masse are any good. There's too many variables that one would have to look at to have an effective survey. In small amounts, let's say a small business of 10-15 people, a survey is effective because its everyone answering most likely with honest intent about a direct problem. Any time a big corporation would ask for a survey saying "What can we do better" is almost pointless. What if directing their attention to my problem causes another issue for others? What if this issue causes a snowball effect to even greater problems? It just seems unreasonable to me.
Post a Comment