Can the integrity of a customer survey be impacted by self-administration?

Oh yes indeed!  Let’s take some lessons from a case study. An organization in the travel industry had outstanding reviews from all its clients on its self-administered customer surveys. Customer surveys were passed out by the tour director on the second to last day of the tour and were collected by the tour director the following morning.

Customer reviews were really good. In fact, the company was able to boast they received a 99% recommendation rate from their guests. All in all, things looked rather positive for the company. However, they had noticed a trend toward gradual decreases in returning clients and, even more puzzlingly, it seemed that few people were actually being referred to the travel company. Somehow things didn’t add up. Why weren’t people referring their beloved travel company?

One of the new company directors made a controversial decision to outsource the survey to a third party. Much to the company’s surprise, the information gathered by the third party didn’t even seem to be from the same company. Only seventy percent of customers said they might consider recommending the travel company to others. Hardly a glowing endorsement. Furthermore (and despite the substantial cash outlay for the tour packages offered by the company) several clients abandoned ship and left early—they weren’t even around to answer the customer survey!  Finally, it surfaced that tour directors became scapegoats when clients were dissatisfied with company offerings…even if the points of dissatisfaction were beyond the tour director’s control. Knowing this, it wouldn’t be terribly surprising if several dozen surveys from less-than-happy clients went missing or clients were too scared to reveal their true opinions while they were still on the tour!

The above represents a case study of poor survey administration practices that likely led to skewed response rates as well as affirmation and socially desirable response biases. Ultimately, it helped create a completely inaccurate picture of the organizational state. Unfortunately, this sort of occurrence is not terribly uncommon. Despite best intentions, customer survey data is often collected and used in the wrong way. When this happens, customer surveys can actually block the path to improvement.