Validating Competencies for 360-degree
Feedback
ACTION SURVEYS(TM): Guest article
Opt-in Newsletter of The Survey Company
February 25, 2003

The Survey Company newsletter has a new name! "Action
Surveys" signifies our commitment to the use of surveys
as practical tools for planning and decision making.
Archives and subscription information can be accessed at
http://www.surveycompany.com/resources/newsletter/

TODAY'S TOPIC
We periodically receive inquiries similar to this recent
request:

*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*
I'm interested in learning more about what you have to
offer in terms of surveys/survey items related to
leadership competencies. I'm currently working with a
client whose senior management team has identified
several leadership competencies that they would like a
360 systems designed around. I'm interested in questions
that specifically relate to these competencies, that
have been well researched, tested, and validated. I'm
also interested in reliability estimates for questions
related to a single competency.

Can you provide this type of service/information?

*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*
To get a fresh perspective on this question, I asked
Jerry Talley of Organizational Diagnostics to respond to
Sylvia's inquiry. I asked his permission to include his
informative and practical reply in our newsletter this
month.
*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*

Dear Sylvia:

You asked about valid and reliable questions around
particular leadership competencies. I imagine you are
looking at building an instrument for your client and
want to make sure the questions will provide a solid
basis for decision making.

While your concern for validity and reliability are the
traditional questions to pursue, my experience suggests
another area is more critical for securing the best
outcomes: the decision process at the end of the 360
feedback. Most executives are quite comfortable with
questions that have face validity alone. Where they are
most concerned is with how the data will be used...and
they're right to worry about it.

Feedback data can be easily misused. And even when used
appropriately, it is easy to distort the data collection
process. Agreements up front on appropriate use and how
to ensure the integrity of the data will have a much
stronger impact on the final result than time spent on
crafting the most sophisticated set of questions.

I find that using scales of several questions is the
best way to ensure validity. So a dimension such as
"Exemplifying teamwork" is made up 3-4 questions that
come at the same topic from slightly different
perspectives. For example, I might ask...

1. This person demonstrates by his or her own behavior
that they are able and willing to collaborate with their peers.

2. This person requires that their direct reports work
together collaboratively.

3. This person makes sure that people are rewarded as
much for their teamwork as for their individual
achievements.

4. This person actively solicits a range of opinion,
even when there is an emerging consensus of thought.

Statistical analysis of the data (Cronbach's alpha) will
tell us if these questions are behaving similarly and
can be justifiably taken as an indicator of a single
concept. Prior to administration, those being evaluated
can be queried if these questions capture their notion
of supporting teamwork.

But the real question is how the data will be used.
Will it be public? Or only given to the target? Will
part of their bonus be contingent on their scores? Or
on their improvement in scores? Or will their
improvement plan become part of their performance
contract (independent of the data)? What resources are
available to the target to improve their scores? The
real goal here is not just measurement, but improvement.
What's the mechanism to support change?

If people will be compared to their peers, how will
those comparisons be interpreted? For example, someone
overseeing a sales group has a much lower need for
teamwork than someone overseeing a design department.
Demanding equally high scores would seem foolish.

In summary, I'd focus on negotiating the understandings
for use of the data. Once that structure is in place,
you will probably find that the questions are
continually tweaked and adjusted over the years as new
themes become relevant. For example, right now
supporting efficiency is more important than supporting
innovation; in a year or two it could be the reverse.

JLTalley
650 967 1444
2/20/2003

***************************************
BEYOND 360-FEEDBACK
Foster talent development and organizational alignment
throughout your company by linking 360-feedback to
organizational goals, job profiles, performance
management, career development and succession planning.
Learn more at
http://www.surveycompany.com/employees/360.html.

Send questions to jgainen@surveycompany.com, or
call toll-free, 877-666-2486.

*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*

TELL A FRIEND about ACTION SURVEYS™

If you find Action Surveys helpful, please pass it on.
New readers can subscribe at
http://www.surveycompany.com/subscribe.html.

Archives:
http://www.surveycompany.com/resources/newsletter/

Joanne Gainen, Ph.D.
The Survey Company
Online survey management and analysis
http://www.surveycompany.com

*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*
You are welcome to quote this article or any of our
newsletters, or to create a link on your web site to the
archived version. Just include the following information:

Reprinted (or excerpted) from Action Surveys(tm),
published byThe Survey Company,
http://www.surveycompany.com. We would also appreciate
receiving a copy or the URL where a link may be found.
*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*