LETTERS

Customer Inter@ction Solutions, Jul 2005

Ms. Schelmetic,

I wanted to drop a note thanking you for the plug in your "Last Call" ["Clean Up In The Self-Service Aisle," May 2005] editorial for the need for more and better testing of self-service systems.

Testing for ease of use and caller satisfaction is the business my company was created to perform, and after about four years of providing service, there are a number of stories to share. One significant point is that bad test methodology can be worse than no testing at all.

Examples?

The million-dollar systems tested only by employees under pressure to launch who knew the jargon and how the system was supposed to work; and which were shut off within weeks of launch due to caller complaints. (NDAs in place so I can't name names, but this is such a frequent occurrence that we always counsel against "Friends and Family" tests. Show off if you wish, but call it a demo and not a true performance test.)

The speech technology ASR company that was blamed and whose reputation nose-dived when the customer insisted on installing an automated system despite insufficient processing power in the back-office to handle the call volumes.

The sometimes horribly poor survey and testing methodologies used and often manipulated by the workers to disguise the truth about service levels. When you call Comcast and are asked to participate in a post-call survey, the agent gets a screen pop and therefore knows the caller has agreed to be surveyed. Transfer to the survey is in the agent's control. Do you think callers finishing a negative call get transferred? General Motors has every new car buyer take a post-sale satisfaction survey, and sales people are compensated for 100 percent positive results. But it is the sales person who "assists" the car buyer in completing the survey.

The call center manager measured and rewarded for getting the automated system to handle a higher percentage of calls. His solution was to make it MORE difficult to zero out to an agent, allowing him to meet his goal.

Writing objective questions, participation bias, sample bias, etc., are huge issues too. "Hello, were you satisfied with our service or would you rather be poked with a sharp stick?"

Sample size is also critical. If a problem affects five percent of callers; 100 sample calls will reliably hit the problem just once. That is seldom enough to even recognize a problem exists. "We improved our satisfaction scores five percent over last month!" Great. But your follow-up survey sample of 100 customers was biased because only those with positive experiences were asked to participate, and besides, a sample of 100 has a 10 percent margin of error.

My point is that some poorly thoughtout systems might be due as much to bad testing as no testing. "Do it yourself" survey writing might result in some insights, but is insufficient to create a quality self-service caller experience.

So as we advise the industry, it isn't enough to say more testing is needed. We have to emphasize GOOD testing as well.

Again, thanks for the editorial. If we keep preaching, perhaps it will sink in someday.

Rick Rappe, Vice President, Business Development, VocaLabs

www.vocalabs.com

Editor's note: Rick, thanks for the heads-up about Comcast. As 100 percent of my dealings with Comcast are, shall we say, somewhat contentious, from now on I'll specifically request to be put into the post-call survey system. That'll be more fun than a new sled after a blizzard!

- Tracey Schelmetic