Multi Reviewer Accuracy

Recently, Fortune magazine estimated that 90 percent of Fortune 500 firms utilize some form of multi-rater feedback.  If your organization utilizes a 360 degree review process, how do you maximize its effectiveness?  Like any performance management system, the true value of the system relates to the degree of cultural alignment, relative knowledge of the system, resource availability, and accuracy of results.  In other words, success relates to:

  • How well the system matches the organization it serves;
  • How well employees and supervisors understand and maximize the use of the system;
  • How well the organization does at providing the supporting, yet necessary resources; and
  • How well the evaluation actually captures real performance.

Without these basic factors that align process to results, the system stands little chance of actually improving performance in any type of organization.  Like many things, the focus of controversy typically begins with the simplest and most central element.  Accuracy is that element.  Accuracy increases the legitimacy of the system, actionable information, and validity of the results.

Although controversy surrounds the opinions on the potential success of 360 degree reviews, both sides agree that accuracy must be assured.  This is not surprising since systems typically fail due to poor execution more than any other factor and a core component of execution pertains to how accurately the reviewers measure performance of the coworker, supervisor, or employee.  With a 360 degree system, it is not enough to worry about the knowledge of the supervisor, but all participants need to have a similar level of capability.

What do we know about accuracy from research? Robert Eichinger in his 2004 Perspectives article addresses this issue.  Not surprisingly, his study found that rater accuracy increases with the length of time that a rater has known the individual being evaluated, but reaches an inflection point before diminishing.  Nevertheless, time was the most significant factor for predicting accuracy of the 360-degree review.  When reviewing results for accuracy, those raters that knew the employee for one to three years provided the most accurate results.  The second most accurate group knew the employee for one year or less.  Those relationships that encompassed three to five years and more than five years produced the least accurate results.  The study concluded that the greatest accuracy occurs when a rater had passed initial impressions, but the process had not standardized.

Although it might seem this pattern specifically relates to the 360 degree design, any system must address accuracy.  Among those organizations that attempted to increase accuracy, training and feedback produce the greatest improvements in the short and long term.   It comes back to what we know: reviewers need regular refreshers to ensure their performance.

This entry was posted in Organizations, Performance and tagged , , , . Bookmark the permalink.