Are your customer surveys accurate? Put another way: Are you giving your company faulty informa­tion based on inadequate research methods?

Malpractice is a hard word — it directly implies professional incompetence through negligence, ignorance or intent. Doctors and other professionals carry insurance for malpractice in the event that a patient or client perceives a lack of professional competence. For contact center professionals, however, there is no such fall back for professional incompetence, whether intentional or not.

You should never put yourself in a position where your competence can be called into question. That's why so many call center managers are skating on thin ice when it comes to their customer satisfaction measurements: There are demonstrable failings with many of the typical practices used by call center managers. By definition, an ineffective measurement program generates errors from negligence, ignorance and/or intentional wrongdoing. You have a fiduciary responsibility to your company — and recommendations made based on erroneous customer data do, indeed, meet the definition of malpractice.

Measurement programs must meet certain scientific criteria to be statistically valid with an acceptable confidence level and level of precision or tolerated error. Without these considerations, you are guilty of Survey Malpractice. To defend your program with “it has always been this way” or “we were told to do a survey” is not sufficient. Research laws adhered to in academia apply to the business world. A deficient survey yields inaccurate data and results in invalid conclusions no matter who conducts it.

Before assuming that Survey Mal­practice does not apply to your program, consider the following telltale signs of errors and biases:

• Measuring too many things. Your survey evaluation of a five-minute contact center service experience takes the customer 15 minutes to complete and has 40 questions on it. While everyone in your organization has a need for customer intelligence, you should not be fielding only one survey to get all of the answers.
  Should the contact center be measuring satisfaction with the in-home repair service, the accounting and invoicing process, the latest marketing campaign, or the distribution network? Not all on one survey.
• Not measuring enough things. An overall satisfaction question and a question about agent courtesy does not a survey make. Without a robust set of measurement constructs, answers to questions will not be found. Asking three or four questions will not facilitate a change in a management process, won’t enable effective agent coaching and is unlikely to be a valid measure to include in an incentive or performance plan.
• Measuring questions with an unreliable scale. In school, everyone agreed on what test scores meant: a 95 was an A, an 85 was a B and a 75 was a C. Everything in between has its own marks associated with it, as well. Yet, when it comes to our service measurement, we tend to give customers limited responses. What do the categories excellent, good, fair and poor really mean? Should you translate them into numbers for a grade from the customer? Giving limited options does not permit robust analysis, and statistical analysis is generally applied incorrectly. Using a categorical scale or a scale that is too small (like many typical 5-point survey questions) is not adequate for the evaluation of service delivery.
• Measuring the wrong things or the right things the wrong way. Surveys should not be designed to tell you what you want to hear but rather what you need to hear. Constructs that are measured should have a purpose in the overall measurement plan. Each item should have a definitive plan for use within the evaluation process. The right things to measure will focus on several overall company measures that affect your center (or your center’s value statement to the organization), the agents and issue/problem resolution.
• Asking for an evaluation after memory has degraded. When we think about time, 24 to 48 hours does not seem like a long time. But when you are measuring satisfaction with your service, it is the difference between an accurate evaluation and a flawed one. Do you remember exactly how you felt after you called your telephone company about an issue? Could you accurately rate that particular experience 48 hours later, after other calls to the same company, or others, have been made? Yet you are asking your customers to do just that with a delayed measurement, and you are opening the door to inaccurate reporting and compromised decision-making for your organization. This is also an unfair evaluation of your agents.
• Accuracy and credibility of service providers and product vendors. As with any technology or service, the user assumes responsibility for applying the correct tool, or applying the tool correctly. You can purchase software to schedule agents but if you do not apply the functionality correctly, you will be responsible for the error.
  There are plenty of homegrown or vendor-supplied tools to field a survey, but again, if you do not apply the functionality correctly, you will be responsible for the error. You can locate a service provider that is interested in selling you something, but it is usually something that fits into their cookie cutter approach and that will not be suited to your specific requirements.
• Wiggle room via correction factors. If you are using correction factors to account for issues in the data or to placate the agents or the management team, some aspect of the survey design is flawed. A common adjustment is to collect 11 survey evaluations per agent and delete everyone’s lowest score. With a valid measurement that includes numeric scores as well as explanations for scores and a rigorous quality control process, adjustments in the final scores are not necessary. Making excuses for the results or allowing holes to be poked diminishes/undermines the effectiveness of the program and highlights an opening for Survey Malpractice claims.

How To Do It Right

In addition to adding important insight for management, customer score explanations (survey comments) enable a more accurate level of individual accountability. In research, the collection of comments is referred to as qualitative data. This critical element is often omitted because it may be difficult to collect, to process and to analyze. Without this feature, a measurement program is immediately susceptible to errors that are not corrected.

The most common error when customers respond to surveys is that they reverse the scale (scale flipping), which means they want to be giving you positive scores, but are instead giving you negative ones by mistake. Therefore, comments can be reviewed to ensure correct interpretation of the scale and of the question. Proper assignment of the survey is evident as well. Survey scoring or assignment errors account for an average of a 5 to 7 percent variation in scores pre- and post-quality control. This is the amount of negligent exposure from errors in the data. Controlling for a 5 percent score variation lessens the need for correction factors, ensures more reliability in the results upon which to base management decisions, and allows your QA team to coach with confidence.

The bottom line for quality control is to avoid a need to defend this performance metric as a human resource tool. By instituting stringent quality control, all surveys pass the “is it fair?” test, and the right agents are held accountable for the right data. Your center’s quality assurance and monitoring program requires continuous calibration, rules and guidelines. In other words, a stringent process is needed to be successful. Your survey program is no different.

Research is a science as is the application of the results. Garbage in equals garbage out. Dirty data is the equivalent of garbage for your measurement program. Consider­ing the implications to personnel management and the overall management of the center and its ultimate value as the relationship manager for the company, it is better to do NO surveys than to do poor surveys. Critically evaluate your program or proposed program on the following criteria:

1. Use of the correct measurement methodology. The customer chooses the communication channel to your organization and your survey of that experience should match the channel. Avoid survey channel slamming by employing immediate post-call surveys to evaluate telephonic customer service delivery and email/Web surveys to evaluate the electronic channel. Mixed methodologies are likely to create Survey Malpractice exposure. To avoid Survey Malpractice, ensure that you are measuring the rights things through the right channel.

2. Appropriate length. Protect your response rate and survey validity by fielding a survey that is just right, not too short and not too long. Monitor the point at which respondents drop off to determine if the survey is too long. Consider running more than one survey if the scope of the measurement need grows. Measuring becomes an addiction for those who can benefit from your continuous contact with customers. Do not attempt to measure everything in one survey. Run additional surveys for requests outside your scope to prevent excessive additions to your Voice of the Caller survey.

3. Actionable results. Infor­ma­tion must be more than just interesting to know and what management wants to hear. Customer intelligence drives the organization and should be highly sought after. If your team and company executives are not clamoring for the information, you have room for improvement. Managing the customer experience is continuous and must be proactive rather than reactive. The contact center is an effective intelligence partner for the company, beyond its own measurement needs. The value the measurement program in the contact center brings to the organization can erase the misconception of being a “cost” center, as long as it is done correctly.

4. Accurate results. Too much time, effort and resources are wasted chasing the wrong things from questionable survey results. More time, effort and resources need to go to the design of the survey and its implementation. Again, NO survey is better than a bad survey. Action on results is expected, so the direction for action must be accurate.

5. Measurement should complement existing programs. Surveying is part of a holistic approach. Call monitoring is your view of quality and surveying is the callers’ view of quality. It should not be one or the other, but how to use them each to complement quality assessment with a holistic view of the caller experience.

6. Retain the customer relationship. Measurement is intended to understand and then to reengineer the experience to be successful for the company and for the customer. When service interactions fail, quantification is not enough. It is irresponsible of the management team to not have immediate triggers to serve as a safety net for the relationship. Build your survey tool to include a request for contact if the customer needs additional attention. This way, you will be able to save a customer relationship that might otherwise have been lost. Delayed measurement programs do not include the benefit of an immediate trigger.

7. Request Resolution and Problem Resolution must be quantified. The various call resolution calculations such as first-contact resolution (FCR), are most important from the customers’ perspective. (See FCR Unraveled: Getting to the Heart of First-Contact Resolution.) Just because your team or the agent views the call as “closed” does not mean the customer has the same impression. Reporting and acting on any other FCR metric is negligence. A series of questions around the resolution topic must be included in your survey.

We all know that cheap can sometimes be very expensive. Cost-only decisions are rarely apples-to-apples comparisons. Find­ing the least expensive survey option does allow you to check the measurement requirement off of your list of things to do. However, with a low quality and value plan, the Survey Malpractice implications could easily be more than 100 times your annual measurement budget. Biased and inaccurate results affect the contact center’s position within the organization as the momentum of effectiveness breeding efficiency never occurs. The right dials are not tweaked, and improvement is not made and certainly cannot be quantified.

Unfair measurement used for performance evaluation causes low agent morale and higher turnover. A lack of quality control enables scores that could be invalid and/or assigned to the wrong agent to be included in the scoring model. And with all of these situations in play, unhappy managers equal unhappy agents, which in turn equal unhappy customers.

Ignorance about measurement contributes to Survey Malpractice. Design a customer intelligence plan and secure a greater ROI from the right measurement. Use the customer-driven quality monitoring to be less of a report card summary and more of a diagnostic tool to achieve best-in-class performance. Agents respond to fair and equitable feedback and productive coaching and this translates to happier customers. Educate all parts of the company about your measurement program and leverage the information to help them to achieve their goals. If the data are sound and analyzed properly, the results will be sound and, therefore, defendable. This leads to a greater ROI, but more importantly to greater value — to your organization, to your employees, and most importantly, to your customers.

Jodie is the author of the upcoming book, “Survey Pain Relief: Transforming Customer Insights into Action,” which will be released at ACCE 2008.