The dark side of multi-rater assessments - rating of managers

HR Magazine, May, 1998 by Susan Haworth

Multi-rater tools can be hazardous to organizational health if left in the wrong hands. But when used effectively and judiciously, they can produce dramatic and life-changing results.

The 1990s have been a good decade for 360-degree feedback (a.k.a. multi-rater or multi-source feedback). Once a luxury for senior-level executives, these assessment methods-which collect data from peers, direct reports, supervisors and others - are becoming a must have for managers of all levels.

However, the surge in the popularity of multi-source assessments has led to "knock-offs" and misused applications. Here are the top 10 ways to misuse these tools.

MISTAKE #10: BUYING LOW

If your budget does not allow you to purchase a professionally developed, validated instrument, be wary: You may be faced with yet another example of getting what you pay for.

Any instrument you purchase should include items that are well researched and directly related to job responsibilities and functions. When items are vague, too generic or not linked to job effectiveness, multi-rater exercises become a waste of time and money.

Every skill assessed should be defined by concrete behaviors. For example, organizational skills are important in most every professional job, but how would you know organizational skills if you saw them? Descriptions such as "is able to plan, juggle and keep track of multiple projects and assignments" and "allocates resources effectively to accomplish primary objectives" provide observable examples of this skill.

To minimize confusion and ensure the ease of evaluation, items should define discrete behaviors. Double-barreled items such as "selects and trains employees to meet departmental objectives" may pose a problem for raters if the person being assessed selects well but trains poorly, or vice-versa.

To garner the most from your investment, carefully determine the competencies (skill sets) you want to evaluate. Make sure they are job related and reinforced through other human resource initiatives, such as performance appraisals.

In addition, investing in a consultant to deliver assessment results ensures the consistent quality of the feedback, as well as the perception of neutrality, confidentiality and objectivity. If results are delivered in an abrupt, trite, overcritical, condescending or threatening manner, the employee will focus on the treatment rather than the value of the feedback.

Employees also should have the opportunity to engage in dialogue, evaluate the report and discover its merit over time. This usually requires a high level of trust and comfort with the person delivering the feedback. Therefore, employers who cut corners when developing a product or delivering the results will likely damage not only the credibility of the current effort but future efforts as well.

MISTAKE #9: SELLING HIGH

Remember, 360-degree feedback is not a panacea for individual deficiencies or organizational woes. These tools will not make anyone more inventive or imaginative, but they may improve someone's ability to work with others.

Of course, few things work well in isolation. As a result, multi-rater assessments should be tied to development plans, concrete methods for improving skills and reinforcement for making progress. When introduced as a one-time event, a multi-rater assessment will be perceived as another "We tried it once and it didn't work" initiative.

Establishing trust and building faith in the process may take several years of testing and fine tuning, with ongoing organizational support. By using multi-rater assessments annually (with quarterly development discussions), you can institutionalize the process and encourage employees to take action on their feedback.

MISTAKE #8: KEEP 'EM GUESSING

Graceful introduction of multi-rater assessments is recommended for gaining employee acceptance. Whether from bad experience or bad press, employees may connote multi-rater with multi-problem. Dispelling myths about the use of multi-rater assessments - or any assessment - is time well spent.

Inform employees of how the tools were developed, how they will be used, how raters will be selected and who will have access to the results. Ideally, 360-degree materials are distributed in person (individually or in groups), where questions and concerns can be immediately addressed. Minimally, a letter of explanation accompanying the materials should clearly and positively explain the purpose of the tool and provide questionnaire instructions.

Selecting a liberal number of evaluators per employee will help avoid biased results caused either by ingratiating praise or negative comments from individuals with axes to grind. When possible, peer and direct-report categories should consist of three to five raters. In most cases, the person being assessed should be allowed to select his or her raters, although direct reports and supervisors often are predetermined.

Additionally, the employees being evaluated will want to know who will have access to the feedback. Privacy is important; the specific report need be shared only with the person being evaluated, followed by a discussion between the individual and his/her manager. The development plan that results from the assessment can be part of an HR resource file.