A weekend in the Bush?
The banner photo above shows view of Australian landscape from our holiday lodge. The Retreat Bushland Lodge
Employee Surveys Gone WrongEmployee surveys gone wrong Over 25 years of conducting employee surveys and communication audits I’ve seen many problems. Given that the clients I’ve worked with have been enlightened enough to want employee feedback by way of surveys, it’s surprising that so many surveys provide misleading findings. Here are some examples from Kidding yourself Executives in an international organization knew the executive team was “on the nose” from pre-survey focus groups I’d conducted. They did not want to have to report to employees just how unpopular they were. So they refused to allow me to include items in the survey on senior executives. I protested in vain that in The survey results showed excellent scores on a couple of dozen factors. But compared to these, overall employee satisfaction was much lower. It stands to reason that if employees are satisfied with a couple of dozen aspects of their employment relationship, (for example communication, their immediate manager, job satisfaction, training, career prospects, service quality and work environment) then they’ll be satisfied overall. But not so in this case. HR managers from all around the world could not work out what was wrong. How could overall satisfaction be lower than scores for every survey category? The reason of course was the missing factor – the executive team - almost certainly a key driver. The lesson here is that if you do a correlational analysis to identify the key drivers of satisfaction or engagement, you can’t leave out a category that might prove to be significant. The result is a quite misleading analysis. Coordinated lying Employee surveys have become a joke in many companies where bonuses are at least partially influenced by employee survey results. Sometimes it is only the manager’s bonus, but managers have ways to encourage staff to rate them well when there is money at stake. In a subsidiary of a global energy company the overall satisfaction score (the factor used for bonus calculations) is artificially high every year as employees know their bonus is determined by this score. The scores on other factors are much lower. (The reverse situation to that in the case above.) If you tie survey results to financial incentives expect employees to tell you lies. Your survey will be inaccurate and a waste of money. Blatant cheating Recently the CEO of the same energy company announced in the staff magazine that overall employee satisfaction was up to 93% positive. I could not believe the improvement. Over many years it had improved from 48% positive into the 80s, but it had been relatively stable for some years. HR told me “it’s amazing what you can achieve when you work at it”. But I discovered later that at least some employees knew HR had changed the index by omitting the low scoring items making up the overall satisfaction factor. This is a clever but unethical way make HR look good. And the HR guy disappeared not long after for reasons that were not announced. If you cheat and your employees find out then your survey becomes a joke. You do too. Impossible targets The HR chief of a high-tech subsidiary of a large telco phoned in a panic. He and the top team had realised they were in danger of missing out on their significant annual bonus. They had included an item in their bonus calculation stating that employee satisfaction had to increase by 10% each year in order for bonuses to be paid. They’d overlooked that fact that over the years as employee satisfaction scores improved (with efforts in the right areas) it got harder to increase the improvement by 10%. It was easy when satisfaction was low and well-nigh impossible when scores were way above the national average. Don’t set impossible targets around your survey results. Not only might they not increase as you’d like, there may be external factors over which you have no control that can affect the results. Benchmarking gives perspective After I’d completed a round of focus groups on communication in a utility the CEO bragged he was about to spend a fortune on a recognition program. I couldn’t believe it as my focus groups showed recognition was relatively good - the best I’d seen in years. I asked why. The CEO gave me latest employee survey results showing recognition items averaged 52% positive. This does not sound good. But when I benchmarked the results against national norms I found that 52% was fully 10% above the national norm. (We Aussies are not very good at recognition.) When all of their survey results were compared to national norms, recognition was the seventh worst score. They had overlooked six more worthy areas to work on. Moreover, there was no correlational analysis so key drivers were not identified. The CEO had no ideal of the true impact of recognition or any other of the survey factors. To get an accurate picture of what you survey is telling you, it is important to get some perspective on your survey results by comparing your findings to industry, regional or national benchmarks. Dodgy survey design gives dodgy results Many survey findings are seriously compromised by poor methodology or poor survey design. So I’ve come up with “ten deadly sins of surveying employees”. Let’s start with some mentioned above and work through the rest. 1. Not doing a “key driver” analysis You will miss a key advantage of conducting a survey if you don’t find out the pattern of statistical relationships – exactly what factors are driving satisfaction or whatever other criteria you wish to understand (e.g. communication satisfaction, retention, engagement). In my experience it is not common for communication surveys to have key driver analysis. Unless you know that executive, manager and change communication are key drivers and that the intranet and publications are not (which is typically the case) you won’t know what to concentrate your efforts on. My Communication World article “Finding the Right Direction” (Nov-Dec 2004) discusses the key drivers of internal communication. 2. Not including all the main factors If you want to find the key drivers of satisfaction or engagement you can’t get an accurate picture if you omit possible key variables (e.g. senior executives’ communication). My preference is to do some qualitative research before you design your survey. If you do a few focus groups of typical employees in typical jobs in typical locations you will have an excellent idea of what all the main factors are likely to be. You can then ensure these areas are well-covered in your survey. 3. Using survey results for bonus decisions When employees know their survey scores will influence their (or their manager’s) bonus or balance scorecard results, they artificially raise their ratings and you don’t get an accurate picture. 4. Not benchmarking your findings If you don’t know how your results compare (e.g. to industry, local regional or national norms) you won’t know if you have a relatively good or poor score in regard to the various factors. For example, a 70% positive score on “understanding job expectations” might sound okay. But here in 5. Relying on single items You should use at least three or four items for each construct you wish to measure. Using one item is not sufficient to provide satisfactory results, as some respondents can interpret individual items in different ways. I get called in by clients wanting help to improve communication on the basis of a poor employee survey score. Often they are relying on one survey item that is potentially ambiguous. For example, “communication here is good” could relate to internal or external communication, manager communication, formal channels, all of the above, or to other things. 6. Keeping the survey too short If you are going to have a few items for each construct you are going to end up with a longer survey than you expect. But there is no evidence that people will not complete long surveys. They will if the content is relevant to them and layout is clear. Comprehensive internal communication surveys need to have at least 80 items if you want to cover a typical 20 areas of investigation. Employee surveys probably need to be twice as long if you want a thorough survey. Years ago I piloted a long survey with a number of groups in a Big Four accounting firm. I was convinced the survey of 250 items (agreed by a group of a dozen partners) was too long. But the employees (who had all been in pre-survey focus groups) told me emphatically that this was what needed. The good survey response showed they were right. Incidentally, in my experience the response rate is most influenced by not communicating findings or taking action (or explaining why action is not taken). I accept that you can “over-survey” your employees such that they get “survey-fatigue”. More on this later. 7. Changing response scales unnecessarily Changing the response scales throughout a survey confuses and annoys respondents. Mostly you should be able to use a standard five point Likert scale (strongly agree, agree, neutral, disagree, strongly disagree) with all items expressed positively. You can vary the wording of the “neutral” if you wish (for example, “neither satisfied nor dissatisfied”). I accept that academics suggest a mix of positively and negatively worded items. But I think it is better for employees not to be confused as can happen if they are doing the survey in what is not their native language. 8. Asking double-barrelled questions Each item should have only one proposition. Otherwise the results will be unclear. This is a frequent error made. The common examples are to refer to reward and recognition and pay and benefits. Asking employees to rate an executive or management team is the same. It is much better to ask employees to rate the CEO and (separately) their divisional chief (or senior VP). 9. Disenfranchising employees
In many organizations you will disenfranchise some employees if you conduct a survey only electronically. Even if they have kiosk terminals, some may have limited computer skills and avoid doing surveys. This is a particular problem for communication surveys assessing channels. 10. Inadequate sample I’ve left the most difficult issue until last. I get asked about response rates often. But the number of surveys you need (the sample) depends on the number of completed surveys compared to the population (i.e. the total number in the group you are measuring) not the response rate. Generally social researchers aim to get a 95% confidence level (or co-efficient). This means you’re 95% sure that you would get the same result if you surveyed everyone in the population. If you get 400 completed surveys back you will always achieve this for whatever population you are measuring. The problem comes with groups of fewer than 400 employees. For example, you need to get 80 surveys in if you have 100 in the population, 24 if you have 25 in the group. Otherwise you won’t achieve the same level of confidence in your findings. That’s not to say the results will be inaccurate, but that you won’t have the same degree of confidence. (You have greater than a one in 20 chance of getting inaccurate results.) To avoid survey fatigue you should not unnecessarily survey employees. Some time ago I had a call from a communicator in a bank. She had 6,500 surveys back from 10,000 employees in one division and wanted to know if the 65% response rate was good enough. As she was not going to report the findings below divisional level I explained she’d wasted the time of 6,000 employees. She only needed 400 completed surveys to get an acceptable confidence level. At about the same time I was working with a telco in which communicators wanted to do some ongoing measurement in a division also with 10,000 employees. I recommended a short emailed survey going to a different 1,000 employees each month for 10 months. (We assumed no more than a 50% response rate so were hoping for at least 400 each month). No survey fatigue here. * * * * * * * * * In my communication work I see dozens of employee survey results each year. Almost always I find problems making some of the findings inaccurate, misleading, meaningless or less useful than they should be. The “ten deadly sins” are remarkably common. The examples I’ve quoted all involved surveys conducted by professional survey providers. It stands to reason that do-it-yourself surveys are likely to have even more problems. It’s a tricky business. Rodney Gray Employee Communication & Surveys Published by the International Association of Business Communicators (IABC) Communication World online in May, 2009. |
- ©2010 Employee Communication & Surveys Pty. Ltd.
- Privacy Policy