| Date: | February 10, 2010 / year-entry #47 |
| Tags: | non-computer |
| Orig Link: | https://blogs.msdn.microsoft.com/oldnewthing/20100210-00/?p=14983 |
| Comments: | 54 |
| Summary: | It seems that the United States has gone satisfaction survey mad. You get your oil changed, they ask you to fill out a satisfaction survey. You make a doctor's appointment, they ask you to fill out a satisfaction survey. You call the company technical support phone line, they ask you to fill out a satisfaction... |
It seems that the United States has gone satisfaction survey mad. You get your oil changed, they ask you to fill out a satisfaction survey. You make a doctor's appointment, they ask you to fill out a satisfaction survey. You call the company technical support phone line, they ask you to fill out a satisfaction survey. These surveys typically ask you to rate how well various aspects of the interaction went, be it how easy it was to make the appointment, how knowledgeable the person who helped you was, how long you had to wait, whether the music in the waiting room was soothing, and what really bothers me is that all of these satisfaction surveys are striving to score "exceeds expectations" on every question. Um, if you are exceeding all expectations, then people will adjust their expectations so that what was previously "exceeding expectations" is now just "meeting expectations". In other words, the only way to consistently exceed expectations is to intentionally set low expectations. I think these people look to Lake Wobegon as their ideal town, because "all the children are above average". Sometimes I try to explain this on the satisfaction survey. One such survey asked, "How can we improve our performance from 'somewhat satisfied' to 'extremely satisfied'?" I replied,
Another time, I simply responded,
Bonus chatter: One time, I was told by the person assisting me that I would receive a satisfaction survey in the mail and that "the only thing they look at is whether you rated something exceeded expectations or not. All the other ratings are meaningless." Of course, all management is doing is devaluing the meaning of "exceeds expectations." Now, when somebody actually exceeds expectations, you have no way of saying "This really did exceed my expectations; it's not just one of those perfunctory exceeds expectations that everybody is required to give." I wonder if they will just go one higher and create a category called significantly exceeded expectations. Of course, the natural next step is to require all customer service representatives to meet this new level of service. Merely exceeding expectations won't be good enough any more. |
Comments (54)
Comments are closed. |
Yes, the local branches of the big home improvement store chains have surveys all of the time. They ask if your expectations were exceeded, but they don’t bother to ask what your expectations ARE.
I occasionally note in my comments that my expectations are high, and so if I rate the place as "meets expectations" then I’m perfectly satisfied. Other people might have low expectations!
But I’m sure no one pays any attention to my comments.
Don’t they need to know what the expectations are before the "exceeds" part means anything to them? There has to be something to peg that on.
You are aware that filling these things out is optional, right? Unless somebody was really great or really poor, I send these things off to a landfill somewhere.
In your most recent TechNet Magazine column, you mention that Vista made the time portion of the file timestamp friendlier. If you see the person responsible for this, do me a favor and punch them in the face. I understand why this would be done for non-technical users, but as a power-user / developer / tester this seemingly small change drives me insane.
Back in the day MS Facilities (late 90s. maybe) sent around a survey about how I liked my desk. It ran to several pages of questions. I clicked thru to just look at them, and after I’d looked at the 5th page of questions, decided that merely looking at it was a sign of insanity, and stopped. I also hoped that anyone who took the time and answered the thing would be immediately fired.
Spinal Tap’s surveys go up to eleven.
The social sciences are jealous of real science and want to be able to measure the unmeasurable.
"The social sciences are jealous of real science and want to be able to measure the unmeasurable."
Meanwhile, the real sciences are jealous of the social sciences, because the latter can get funding time and again without ever having to publish a meaningful measurement.
I’ll say that your book exceeds expectations, and overall I am somewhat satisfied with Amazon’s shipping thereof.
This post merely met expectations
I recently had a support ticket with a vendor and the support guy helped me solve the problem just fine. Then instead of closing the ticket, I got an e-mail from him asking me to rate him from 1 to 10 ("1 being lowest", he said to describe the scale) before he closed it. The lack of anonymity was awkward, as well as the generic scale that I’m sure I don’t interpret like the majority.
Well Mr. Vendor Support Guy, although you answered quickly and your answer was helpful, that was exactly my expectation so I’d like to rate you a 5, which I assumes means I’m satisfied. A 7 at most because hey, what if you had done something truly exceptional like flown out and fixed the problem yourself? That’d be worth a 10, so if I rate you a 10 this time then I’m kind of setting my expectations low. Except you also got a minor detail wrong, so I’d like to take off a point for not being thorough and testing your solution after typing it into the e-mail. So a 4. Or 6 at most.
But it was too awkward because I knew what he wanted, so I gave him a 9. And I bet he thought I was a jerk for giving him a 9.
This reminds me of those surveys where you’re asked to rate something on a scale like:
Poor, Satisfactory, Good, Excellent,
a scale on which everyone knows that rating somebody or something as "Satisfactory" is the politest way of expressing dissatisfaction.
It’s like they said in Chuck:
"If everything is awesome, then nothing is awesome, awesome becomes bland!"
I get one in the mail every time I take my car to the dealer "If you can’t check "exceeded Expectations on every box, call me and I’ll take care of it personally. Signed, [Manager]" It always confuses me too, becaue my expectations for Car repair places are remarkably low, such that it’s not hard to meet my expecations (that is to say, not utterly fail) yet I’m hardly satisfied. I should call up and say "Honestly, you met my expectations, which was to say service was slow, and communication was poor, just like I’ve come to expect from Car dealer service centers. And since it took only 4 hours instead of the 6 I budgeted, i suppose you exceeded them there! Yet, I’m not exactly satisfied, so how do I check this box?"
I am dissatisfied by John’s bringing back the random Vista-bashing in completely unrelated blog entries. I rank John’s post 2 out of 10. (Would be 1, but I don’t want to get him fired!)
tsblke,
Its worse with BMW… everytime I take my car into the dealer, whether its a major repair or just an oil change, a couple of days later I get a call from the BMW mothership asking how my experience was.
I interpret it more as (to paraphrase George HW Bush) "Message: we care" than any serious attempt at quality measurement. As other people have said, even though its allegedly a five point scale its interpreted as a binary result.
A few times I wanted to give response to the quality of service of one or another company — especially what annoys me. So I attempted to fill the survey only to find that they are carefully designed not to allow you to show what’s wrong.
So my theory is: there are these "survey companies" that get money for making surveys. Now the director of the "Survey" company (you can imagine Dogbert) goes to the pointy haired boss of the company that can potentially pay for it. PHB will gladly pay Dogbert for survey, but he of course doesn’t want that survey to show his incompetence. No worry, Dogbert demonstrates him that it’s impossible thanks to carefully designed questions: the survey can only show that he’s awesome. Nothing else is possible. Result: PHB has the survey to show to his upper levels. "Survey" company gets money. Users just get annoyed.
All such surveys are therefore absolutely useless. Competent managers can find out what sucks in their company only some other way.
Social scientists have written some of the most thorough, insightful and useful texts on measurement theory and practice. Unfortunately they are rarely well read and the typwe of scales being referred to here (mainly Likert) are used without effectively knowledge of their parameters and social implication – Raymonds point. Excellent point. Well made.
They should be using the outstandingly flawless WES (Wendy Experience Scale): http://wendyhome.com/2009/03/24/wes/
Back in the old days when we were really trying to figure out our customers, our surveys were read to
Collect written comments.
The only results circulated were generic problems (e.g. ‘filled order wrong’, ‘did not return call’) and the written comments. None of the multiple choice were even tallied.
Managers were required (and others encouraged) to read the incident lists and written comments.
The company believed that if the results didn’t give enough information make improvement, then management didn’t understand what was happening well enough to manage!
Back in the old days when we were really trying to figure out our customers, our surveys were read to
Collect written comments.
The only results circulated were generic problems (e.g. ‘filled order wrong’, ‘did not return call’) and the written comments. None of the multiple choice were even tallied.
Managers were required (and others encouraged) to read the incident lists and written comments.
The company believed that if the results didn’t give enough information make improvement, then management didn’t understand what was happening well enough to manage!
I can second the "only exceeds expectations matters" bit. I know some people who work for a major pharmacy chain in Ontario and their customer surveys work that way too. Not only is this asinine survey made up of one "right" answer and 5 wrong ones, but they actually use the scores to determine employee raises.
When I worked in support they actually read the surveys. The numbers were rolled up in to an average score whose value was actually realistic. For example, our goal was an average csat score of 3.8 / 5 in each of the various categories.
That goal exceeded expectations.
I’m somewhat satisfied by Raymond’s blog.
The BMW guy is paid based on what you put on the survey. Sometimes you’ll see people essentially bribe you for a 10.
The next major step that will be taken in customer satisfaction and other areas of life (and in some ways is already being taken) is that people will not be permitted to exceed expectations. I mean, imagine the harm that does to the poor guy who merely meets expectations.
Just ask Harrison Bergeron.
I decided a long time ago to always give low scores on any "satisfaction survey" (this goes double for any online version of a survey). I figure that every company and group has room for improvement and such improvement will only happen if they perceive a need for it.
If everybody is "somewhat satisfied" there’s no reason to try and improve the service.
"the only thing they look at is whether you rated something ‘exceeded expectations’ or not."
Is that so the MBAs know where to cut the budget, cleanliness exceeded expectations – sack the cleaner.
It is, of course, trivial to have all employees be above average.
First, remove all possibility of independent thought – employees should be able to follow a flowchart to do their jobs. This makes everybody’s performance very close even.
Second pick a person (at random) and sabotage their efforts somehow. This makes one person’s performance much poorer than the rest.
"Average (mean) performance" is then very close to the common level of almost all employees, and much higher than the one employee that was thrown to the wolves.
Third, fire that person.
Result: everybody’s performance is above average.
I find ironic the relevance to the review process that exists at Microsoft, or at least did a decade ago, last I worked there. In particular, on a scale from 1 to 5 (or was it 0 to 5?), 3.5 was considered "average", and you only got a 3.5 if your manager felt that you’d exceeded expectations.
Do a solid job, meet your schedule, write code with a minimum of bugs, and that’ll earn you just a 3.0. Get a few 3.0 scores in a row, and now you’re the "team underachiever".
So much for the true meaning of "meet expectations".
Microsoft surveys often ask if an experience has resulted in increased satisfaction with Microsoft. If one is already satisfied with Microsoft, having something go as expected would not increase satisfaction. Answering "no" implies that your good experience was in fact bad.
I have a high opinion of Microsoft based on years of experience. An everyday experience like searching the KB and finding a hotfix is unlikely to improve my satisfaction. However, I must either lie and say it did, or tell the truth and give the impression that my experience was bad. Catch 22.
Declaring "Exceeds Expectations" as the baseline is the same kind of thinking that gave us "give 110%". In fact, I’m surprised that number hasn’t been subject to inflation.
I went to my Toyota dealer on Monday for the service. Normally I expect there will be a follow up survey either by phone or email. But there is none this time, I think that they are so busy this time for the big recall. The survey is meaningless now. So here you go next time when the survey appears again, the result will be positively positive.
My Honda dealer stopped the calls after my car left warranty.
The biggest problem with the surveys is that you can’t express dissatisfaction at having to call in the first place. I would far rather have the issue not arise or be able to fix it myself. Having to call means it is a big problem and I am dissatisfied no matter what.
One company I worked over fifteen years ago used to include survey cards with the software product (ah the days before downloads). On a whim once we tried to track down what happened to them. It turned out they were stuffed in boxes in a warehouse and never looked at.
Consumer goods companies still pull tricks like this today. Usually there is some warranty card that goes to some random city in Colorado. It asks you all sorts of irrelevant stuff (household income, education, how likely you are to subscribe to magazines etc).
@Roger
People here probably know, but the "warranty card" that comes with all manner of consumer crap is nothing of the sort. It’s little more than a gimmick to get you to respond to their questionnaire which they then turn around and sell to some third party.
Warranties do not need to be "activated" by sending in the included card (at least in the United States). You are issued the warranty upon purchasing the product.
Aren’t they like most tests? The score themselves don’t really mean anything, but comparing scores does have meaning. Same as SAT scores. What does a 1420 mean? Nothing unless you know what everyone else is scoring. In the same light, a 8.3 means nothing until you put it in context. So you aren’t looking at the exact score as much as how you rate compared to others (or how one attribute rates compared to others).
“I am easily satisfied with the very best.”
Sir Winston Churchill
Sometimes it’s not the companies themselves want the survey for direction of improvement.
For example, in order to retain your company’s Microsoft Certified Partner status, you have to ask at least 10 of your customers to fill in surveys each year for the CSAT requirement.
These surveys aggravate me because they’re so general as to be meaningless. "Cleanliness: 1 2 … 10". Well, maybe the plates and silverware were sparkling clean, but the floor was grimy. How do I rate that? *Specific* feedback would be useful to fix the problem, but a rating from 1 to 10 does nothing…
…unless the only reason they’re really asking is that they ship the cards back to HQ, which decides which restaurant gets the bonus for the most 10s.
Maurits: that’s also why nearly everybody has below average number of hands but above average number of fingers.
This post reminds me of MSFT performance review-speak
I really like the "I’m satisfied with somewhat satisfied" line. I’d throw in an occasional "Your service rocks, but the survey was a great downer".
It’s just an oil change. I don’t wake up in the morning quibbling in excited anticipation of meeting a mechanic.
Personal experience suggests the employees may have hidden motives for some of the surveys.
There is a Safeway near my parents that is clearly experimental. At one point they installed "Safeway Select TV" above the checkout lines. They blared loud commercials at you and had flashy graphics.
Like magic, all of the lines suddenly acquired several stacks of surveys for shoppers to fill out. If someone asked or complained about the TV the employees immediately handed them a pen. They couldn’t stand it.
The TVs vanished within a week, but I noticed what the management learned was that you need to avoid annoying the employees or get a torrent of negative surveys. They replaced the TVs with speakers and made sure none of them were located near the checkout aisles.
The average person has one breast and one testicle.
"So top grade’s O for ‘Outstanding,’" she [Hermione] was saying, "and then there’s A-"
"No, E," George corrected her, "E for ‘Exceeds Expectations.’ And I’ve always thought Fred and I should’ve got E in everything, because we exceeded expectations just by turning up for the exams."
(Pedant mode: /engage/)
What you, sir, were doing, was /quivering./
/I/ am quibbling.
I was aiming for the sound of the word mroe than assuming it actually exists.
Can I play the "not a native speaker card" now?
This has gone on in military fitness reports for two thirds of a century at least. The Caine Mutiny, set in the middle of WW II, mentions the damaging implications of "above average" on a fitness report–it meant something like "utterly incompetent." As far as I know this still goes on. I know a recently retired officer whose career was essentially crippled by an "above average" on judgment about 1990.
By the way, pretty much the first thing I had to do after downloading Visual Studio Express was to fill out a survey. In fairness to MSDN, I’ll add this did not try to set average as the midpoint of the scale.
Not satisfied by being satisfied?
Expect to expect to exceeds expectations?
This is speak is meaningless corporate jargon.
I normally just always tick the lowest possible score for everything.
And if they come back to me asking "why such low scores" I explain that there wasn’t a box to indicate how satisfied (or dissatisfied) on a scale of 1 to 10 was I with being asked to fill in another bloody survey!
Many organizations now use these surveys for compensation or bonuses (or for performance of contracts in outsources call centers, etc).
If the line worker is minimum wage, and reasonably friendly or helpful even if not necessarily perfect at their job, this is something to encourage so I always score them highly on the questions they have control over, and fire my ire at the questions that represent management decisions or poor investment in systems.
I stay at a lot of Holiday Inns. They send out surveys. The management says anything less than the top mark is a ding on their record. Some I’ve talked to are actually paranoid about it.
One place I was stuck at over four month period had rotten TV reception over their internal cable. The further you were away from the control room the worse the reception.
We were understanding for a while, but it got old. Especially when the best picture was on a rolling ad for Dish Network.
We used the survey to get them to get the replacement they had been talking about for months.
Ah.. It must be MSPoll time :)
I’m just waiting to receive a survey for feedback on a survey…
Holds true in economics as well. Raise minimum wage…
What drives me crazy is that "exceeds expectations" is really meant to be interpreted as a *good* thing, when it might also be pretty bad. I mean… if I expect a service to be bad, it can exceed my expections by being even worse…
By the same principle, this is why in general do not tip wait-staff, unless their service was above ordinary. This expectation that tips are mandatory does my head in (and yes, I know they may be underpaid, but that’s not my problem to fix).
@Tex: coming from a country where rounding up to the next euro is often good enough, I’ve struggled with this a lot in the US. On one side, it’s the local custom, so I feel oblieged to follow.
OTOH, It looks like a conspiracy to not tell how much it would cost to dine here. It’s even worse with cabs drivers, who have no hesitation to show they feel entitled to a big "tip". For a country that is oh so open about "making money", not telling me clearly what a service costs feels strangely out of place.
However, I decided for me that not tipping is hurting the wrong people. Yes – it’s in their power to unite and stand up against this practice, but not tipping them is about as effective as an embargo.
Tex,
It may not be your problem to fix, but the way to protest is to refuse to eat at restaurants that have tipping.
Protesting by taking money out of the pockets of the wait staff is punishing people who didn’t set the policy, and are just trying to make a living. It also makes you look…well, "cheap" is the most polite word I can think of.