|Date:||October 12, 2004 / year-entry #364|
|Summary:||Philip Su's discussion of early versions of Microsoft Money triggered a topic that had been sitting in the back of my mind for a while: That people lie on surveys and focus groups, often unwittingly. I can think of three types of lies offhand. (I'm not counting malicious lying; that is, intentional lying for the...|
Philip Su's discussion of early versions of Microsoft Money triggered a topic that had been sitting in the back of my mind for a while: That people lie on surveys and focus groups, often unwittingly. I can think of three types of lies offhand. (I'm not counting malicious lying; that is, intentional lying for the purpose of undermining the results of the survey or focus group.)
First, people lie about the reasons why they do things.
This is one of those unconscious lies. People claim that they want a computer to do their personal finances, to organize their recipes, to mail-merge their Christmas card labels.
They are lying.
Those are the things people wish they would use their computer for. That's before the reality hits them of how much work it is to track every expenditure, transcribe every recipe, type in every address. In reality, they end up using the computer to play video games, surf the web, and email jokes to each other.
Just because people say they would do something doesn't mean they will. That leads to the second class of focus group lying: The polite lie, also known as "say what the sponsor wants to hear".
The following story is true, but the names have been changed.
A company conducted focus groups for their Product X, which had as its main competitor Product Q. They asked people who were using Product Q, "Why do you use Product Q instead of Product X?" The respondents gave their reasons: "Because Product Q has feature F," "Because Product Q performs G faster," "Because Product Q lets me do activity H." They added, "If Product X did all that and was cheaper, we'd switch to it."
Armed with this valuable insight, the company expended time, effort, and money in adding feature F to Product X, making Product X do G faster, and adding the ability to do activity H. They lowered the price and sat back and waited for the customers to beat a path to their door.
But the customers didn't come.
Because the customers were lying. In reality, they had no intention of switching from Product Q to Product X at all. They grew up with Product Q, they were used to the way Product Q worked, they simply liked Product Q. Product Q had what in the hot bubble-days was called "mindshare", but what in older days was called "brand loyalty" or just "inertia".
When asked to justify why they preferred Product Q, the people in the focus group couldn't say, "I don't know; I just like it." That would be perceived as an "unhelpful" answer, and besides it would be subconsciously admitting that they were being manipulated by Product Q's marketing! Instead, they made up reasons to justify their preference to themselves and consequently to the sponsor of the focus group.
Result: Company wastes tremendous effort on the wrong thing.
(Closely related to this is the phenomenon of saying—and even believing—"I'd pay ten bucks for that!" Yet when the opportunity arises to buy it for $10, you decline. I do this myself.)
The third example of lying that occurred to me is the one where you don't even realize that you are contradicting yourself. My favorite example of this was a poll on the subject of congestion charging on highways in the United States. The idea behind congestion charging is to create a toll road and vary the cost of driving on the road depending on how heavy traffic is. Respondents were asked two questions:
Surprisingly, most people answered "No" to the first question and "Yes" to the second. But if you stop and think about it, if people avoid driving during peak traffic hours, then congestion would be reduced because there are fewer cars on the road. An answer of "Yes" to the second question logically implies an answer of "Yes" to the first question.
(One may be able to explain this by arguing that, "Well, sure congestion charging would be effective for influencing my driving behavior, but I don't see how it would affect enough other people to make it worthwhile. I'm special." Sort of how most people rate themselves as above-average drivers.)
What I believe happened was that people reacted by saying to themselves, "I am opposed to congestion charging," and concluding, "Therefore, I must do what I can to prevent it from happening." Proclaiming on surveys that it would never work is one way of accomplishing this.
When I shared my brilliant theories with some of my colleagues, one of them, a program manager on the Office team, added his own observation (which I have edited slightly):
I'm sure there are other ways people subconsciously lie on surveys and focus groups, but those are the ones that came to mind.
[Insignificant typos fixed, October 13.]
<-- Back to Old New Thing Archive Index