Can we trust survey data?

 

Back in the day, we could learn many things from telephone surveys.

 

But then computers and cell phones were born and people started dropping their land lines.  Not surprisingly those that still have land lines are different than those who do not.  Think less educated, less wealthy and older.

 

So clearly land line only surveys are not going to be projectable.

 

Instead research companies have begun to combine cell phone and computer data with the information received from land lines.  Weighting the responses has proved to be tricky.

 

Perhaps more troubling are the results from a recent study by Pew Research to determine the validity of combined responses.  Pew found that answers from phones versus computers varied by anywhere from 0 to 18 percentage points.  That's a big difference.  And companies are likely to make decisions based on any skew greater than 10%.  Possibly wrong decisions.

 

Interestingly the disparities had to do with intensity - so they aren't contradictory.  That's a relief.

 

Apparently phone respondents were more likely to give extreme responses.  They are also more likely to favor the last choice they hear, while online respondents are more likely to favor the first choice.

 

Discrepancies were especially large when respondents were asked to access their quality of family and social life.  Telephone respondents were more likely to be satisfied than online respondents.  They were also more likely to say gay, lesbian, Black and Hispanics face a lot of discrimination.

 

Meanwhile, online respondents were more likely to rate political figures negatively.

 

Researchers noted that it could be possible that telephone respondents were more inclined to give socially responsible answers. (McGinty, 2021)

 

So where does that leave us?

 

Can we use research gathered exclusively online?  Or should people look for combined data?  Should we verify research before we use it?  How would we do that?  Is the the best option for people to do their own research?

 

Have you participated in a phone survey or live research?  Did you have a desire to please the interviewer?  Have you noticed any pattern to your answers when you participate in online research?

 

 

McGinty, J. (2021, April 30)  What Happens When Opinion Polls Move Online?  wsj.com.  Retrieved January 24, 2022, from  https://www.wsj.com/articles/what-happens-when-opinion-polls-move-online-11619775001

 

Can you use research that failed the reproducibility test?

In 2015, the University of Virginia led a new Reproducibility Project that repeated 100 classic psychological studies and they were only able to successfully replicate one-third of them.

But the article goes on to allow for the possibility that one of the factors causing reproducibility failures could be the passage of time.  Specifically, in 1988, a study was done which concluded that our facial expressions can influence our mood - so the more we smile the happier we'll be. 

https://www.ncbi.nlm.nih.gov/pubmed/3379579

The stimuli for the experiment was a Far Side comic by Gary Larson.  I'll bet you have never heard of it or him.  Humor has changed quite a bit since the 80's so I wouldn't be at all surprised if the experiment could be replicated now, but only with a contemporary comic.

The failed University of Amsterdam in the Netherlands replication study also had a problem of its own - subject bias.  As with most psychology studies it used psychology students for the sample.  And since this was a classic research study, they may have already been familiar with it.

https://psychcentral.com/blog/archives/2010/08/26/psychology-secrets-most-psychology-studies-are-college-student-biased/

So what do you think?  Does this invalidate the results of the initial study?  Which issue concerns you more?  Would you use the results?  If so, how?  Have you seen other evidence supporting the basic thesis that facial expressions can influence mood?

(2016, September 26) MacDonald, F. Two More Classic Psychology Studies Just Failed The Reproducibility Test.  sciencealert.com  Retrieved September 11, 2017, from

https://www.sciencealert.com/two-more-classic-psychology-studies-just-failed-the-reproducibility-test

You keep telling yourself that you bought that fancy new phone so you can hear better.

But we all know that you bought it to boost your self-esteem.  That's because all purchase decisions are emotional. 

There has been a significant amount of research done over the past few decades about why people buy.  Back in the early 90's, even before the scientific proof began to emerge, advertising agencies recognized the patterns and shifted their focus from product features to consumer insights.  And strategic planners were born.

The latest research strongly confirms that belief.  The Advertising Research Foundation has concluded that "likeability" is the most predictive measure of whether an advertisement will increase sales.  Specifically,  emotional response outweighs content when it comes to purchase intent by a 3-1 margin on television and a 2-1 margin in print.  (Murray, 2013)

Why?  Emotions compel us to do something.  It's the way we are wired.  Enjoy your new iPhone.

Murray, P. (2013, February 26)  How Emotions Influence What We Buy.  psychologytoday.com.  Retrieved December 30, 2014, from  http://www.psychologytoday.com/blog/inside-the-consumer-mind/201302/how-emotions-influence-what-we-buy

Can internet research provide the insights that marketers seek?

12/3/09

Thanks to new technologies, a new form of internet research has emerged. It involves aggregating all available web chatter to gather information. That’s right. Your search activities, YouTube posts and tweets are being carefully monitored by companies looking to glean insights about consumer trends.

Based on this type of info Harrah’s determined that what traveler’s tend to chat about is the iconic view of the Las Vegas Strip from its Paris Las Vegas hotel. As a result of this data, they changed the photo on their home page. They also noticed that customers cared about room size, so they started including data about square footage in their marketing messages. The result? Online bookings increased by a double-digit percentage. That’s impressive given the current state of the economy. (Steel, 2009)

But, how do you feel about big brother looking over your shoulder? And what about the large percentage of the population that doesn’t engage online? How will their needs be addressed in an all digital future?


Steel, E. (2009, November 23). Marketers Find Web Chat Can Be Inspiring. Wall Street Journal, pB8.