Benjamin R. Palmer
Organisational Psychology Research Unit, Centre for Neuropsychology
Swinburne University of Technology
Con Stough
Organisational Psychology Research Unit, Centre for Neuropsychology
Swinburne University of Technology
Email: bpalmer@swin.edu.au
In this paper we discuss the potential utility of multi-rater or 360-degree measured EI over self-reported and ability measured EI. We then discuss the development of a workplace 360 measure of emotional intelligence designed specifically for development purposes. Several research studies testing workplace samples are presented which examine the internal consistency reliability of the rater forms of the 360 instrument; compare the means and standard deviations of self- and other-rated EI; and the relationship between how people rate themselves on the instrument and how they rate others. The research findings demonstrate the utility of multi-rater 360-degree EI assessment instruments, which is discussed specifically in terms of leadership development.
The psychometric properties of the 360-degree Swinburne University Emotional Intelligence Test
Multi-rater or 360-degree assessment instruments are very popular development tools amongst organisational psychologists and managerial consultants, particularly as a medium for enhancing leaders knowledge of how others perceive their performance and/or behaviour at work (Sala, 2003). Such assessment instruments typically comprise behaviourally based declarative statements (as items) and likert-type response scales that are systematically scored by a set of so-called “raters”. Raters can include the subject themselves (or “self-ratings”), and manager(s), peers and direct reports ratings thus the term 360-degree. Besides typical psychometric research studies on the reliability, and factor structure of multi-rater assessment instruments, research in this area has also focused on the level of congruence between raters scores (Atwater & Yammarino, 1992; Church, 1997; Johnstone & Ferstl, 1999); interrater reliability (Greguras & Robie, 1998); the comparative predictive validity of different rater groups (i.e., self-scores vs peer-rated scores; Sala & Dwight, 2002); and the efficacy of multi-factor feedback on performance and/or behavioural improvement (Church, 2000; Sala, 2003). Collectively this research has demonstrated that multi-rater assessment instruments can provide superior performance data, promote participant self-awareness, and facilitate behavioural change. Indeed multi-rater assessment instruments are widely utilised for these reasons amongst the corporate sector, with market research suggesting that all Fortune 500 companies have used multi-rater assessments in some form (London & Smither, 1995).
Equally as popular of late is the construct of emotional intelligence (EI; Salovey & Mayer, 1990) and associated assessment and development methodologies. Indeed when the Harvard Business Review published an article on this topic in 1998 (What makes a leader?; Goleman, 1998a), it quickly became the reviews most requested reprint for the last 40 years (Goleman, 2001). EI refers to abilities to do with emotions including (but not limited to), the ability to perceive, understand, utilise and manage one’s own and others emotions (Mayer & Salovey, 1997). The rise in popularity of EI and its application as a medium for managerial development similarly stems from a growing body of research that has demonstrated that EI underlies or predicts effective leadership (Barling, Slater & Kelloway, 2000; Caruso, Mayer & Salovey, In Press; Palmer, Gardner & Stough, 2003); and associated popular literature that suggests that EI can be developed and that doing so may lead to enhanced interpersonal skills and leadership performance (Goleman, 1998; 2002). Research in this area has also seen the development of 360-degree multi-rater EI assessment scales that have been designed specifically as developmental tools (e.g., Bar-On EQ-360, www.mhs.com; Emotional Competency Inventory, ECI, Boyatzis, Goleman, & Rhee, (2000).
Multi-rater measures of EI may prove to be one of the most valuable methods of measuring leaders EI in the workplace. Critics of the multi-rater method for measuring EI have argued that multi-rater measures may never provide a “true” index of one’s actual EI ability (Mayer et al., 2000). Mayer et al. (2000), have argued that self-ratings (in a multi-rater paradigm) are filtered through the individuals self-concept and as such provide an indication of their beliefs about their EI, (or perceived EI), rather than their actual capacity. Similarly, Mayer et al. (2000) have also argued that ratings from others (in a multi-rater paradigm, e.g., peers, managers and/or direct reports) simply reflect different rater groups’ perceptions of the subjects EI as opposed to their actual EI ability. Indeed research with multi-rater assessment instruments has shown that different rater groups often report different (and sometimes incongruent) perceptions of subjects’ performance and/or behaviour at work (Becton & Schraeder, 2004). However, as noted by London and Smither (1995), “…in the socially constructed world in which employees work, others’ judgements about them (no matter how biased they may be) constitute an important reality”, which in the case of multi-rater EI assessment results, may not reflect one’s actual EI ability, but rather the extent to which the subject demonstrates emotionally intelligent behaviour in the workplace. The question that remains unanswered here is what is more important in terms of leadership effectiveness and/or success and the prediction thereof, the leaders actual EI ability, or the extent to which they demonstrate emotionally intelligent behaviour at work?
In the context of using an EI assessment for leadership development purposes, it is conceivable that a leader may have high levels of EI and achieve high scores on an ability-based assessment of EI yet either; (a) not be effectively using their EI at work for whatever reason; (b) choose not to act in an emotionally intelligent way with the people they work with; or (c) indeed choose to use their EI in a manipulative and interpersonally maladaptive fashion. In this context the ability assessment results may show high EI and little insight is provided to both the practitioner and subject into their leadership effectiveness from an EI standpoint. In contrast, if a multi-rater assessment instrument was utilised and the same leader was provided ratings of their EI from the people they work with (e.g., direct reports, peers, managers) it is hard to imagine the person achieving high scores from others. In this latter case it is logical to anticipate that the leader who has high EI but does not demonstrate emotionally intelligent behaviours at work, will be rated as low in EI by others and much more valuable insight may be provided to both the practitioner and leader in the context of 360-degree feedback and development. In this sense it could be argued that multi-rater measures of EI may be as valuable if not more valuable than ability based measure as a means of providing the basis of development.
Despite this potential utility and the advent of multi-rater EI assessment scales very little has been published on the psychometric properties and developmental utility of them. This paper describes a multi-rater EI assessment scale developed at Swinburne University, namely, the Genos 360 EI Assessment Scale or Genos-360; Palmer & Stough, 2001). Some preliminary psychometric analyses of the Genos 360 are presented and the findings are discussed in terms of the potential utility of the instrument and other 360-EI assessment instruments in leadership development programs.
The Genos-360 assesses five definitive facets of the EI construct determined via a large factor analytic study that involved six of the predominant scales of EI available at the time including: (1) the Mayer, Salovey, Caruso Emotional Intelligence test (MSCEIT; Mayer, Salovey & Caruso, 1999); (2) the Bar-On Emotional Quotient Inventory (Bar-On, 1997); (3) the Trait Meta-Mood Scale (Salovey et al., 1995); (4) the twenty-item Toronto Alexithymia Scale-II (TAS-20; Bagby, Taylor & Parker, 1994); (5) the scale by Schutte et al (1998); and finally, (6) the scale by Tett et al., (1997). From this study (Palmer et al., 2001) five common or definitive facets of the EI construct were determined including; (1) Emotional Recognition and Expression, the ability to perceive and express one’s own emotions ; (2) Understanding Emotions External, the ability to perceive and understand the emotions of others and those inherent in workplace environments (e.g., staff meetings, board rooms etc) ; (3) Emotions Direct Cognitio, the extent to which emotions and emotional information are incorporated into reasoning and decision making ; (4) Emotional Management, the ability to manage both positive and negative moods and emotions within oneself and others ; and (5) Emotional Control, the ability to effectively control strong emotional states experienced at work such as anger, stress, anxiety and frustration . These five common dimensions represent our empirically-based model of emotional intelligence which we designed the Genos-360 to assess.
The Genos-360 comprises both self-report and 360-degree rating forms that ask people to rate the extent to which the subject demonstrates emotionally intelligent behaviour at work according to the five facets of EI previous mentioned. The test itself comprises 64 declarative statements and a five-point rating scale and takes around 12-15 minutes to complete. There are a balanced number of positively and negatively phrased items that help determine inconsistent response patterns and illogical responding. In summary, the test provides insight into cross-situational consistencies in emotionally intelligent behaviour in the workplace and hence one’s underlying level of EI.
In this paper we report the internal consistency of the rater forms of the Genos-360 instrument using a relatively large self-other rater sample comprising managers direct reports and peers and the relationship between how people rate themselves (on the self-rating form) and how they rate others (on the rater forms). We do not report on the congruency between self-ratings and ratings from others, that is, the relationship between how people perceive themselves and how others rate them on the Genos-360. Research has consistently demonstrated a general lack of congruence between self-ratings and ratings made by others, although direct reports, peers and manager ratings have been shown to moderately correlate at times (e.g., r=.35; Sala & Dwight, 2002). A general consensus amongst the multi-rater literature is that different rater groups typically observe different facets of the ratees’ behaviour and that this is reflected in the typical differences found between rater groups (Borman, 1974). Indeed according to contingency theories of leadership, effective leaders typically adjust their behaviour and leadership style according to the situation and/or individual they are leading (Becton & Schraeder, 2004).
In contrast a question we argue is more relevant to EI multi-rater measures is: to what extent does one’s own EI (or perceived EI) influence the way you rate other peoples EI? If there were moderate to high relationships between how people rate their own EI and other peoples’ EI, this may have broader practical ramifications for the use of the instrument. Such a find may suggest that you need high perceived and/or actual EI to rate other peoples’ EI and therefore raters would need to be carefully chosen. Alternatively if little relationship existed between how people rate themselves and others this may indicate that you do not need EI to rate someone else’s EI on the instrument.
MethodParticipants
The sample comprised 54 subjects (42 males and 12 females ranging in age from 30 to 55 years; SD = 6.2 years), and 406 raters comprising 54 managers, 194 peers and 157 direct reports. On average each subject had approximately seven raters, one manager, three direct reports and 3 peers. The subjects were senior level Australian managers from a broad range of industries across both private and public sector organisations within the state of Victoria. The annual salaries of the subjects ranged from a low of $69,000.00 to a high of $300,000.00 (M = $160,000; SD = $57,779).
Means and Standard Deviations
The means, standard deviations, and internal consistency reliability (coefficient alpha α) for each of the dimensions of the test pertaining to the sample are presented in Table 1. For comparative purposes, Table 2 shows the means, standard deviations, and internal consistency reliability (coefficient alpha α) for each of the dimensions of the self-test (self-Genos) pertaining to a second normative population sample.
Table 1 - Means, Standard Deviations, and Reliability Coefficients for the Genos-360| Scale | # items | Mean | Standard Deviation | a |
| Total EI | 64 | 224.53 | 22.91 | .94 |
| Emotional Recognition & Expression | 11 | 39.91 | 5.53 | .79 |
| Understanding of Emotions External | 20 | 74.86 | 9.41 | .93 |
| Emotions Direct Cognition | 12 | 31.57 | 6.63 | .79 |
| Emotional Management | 12 | 43.61 | 6.12 | .85 |
| Emotional Control | 9 | 34.56 | 4.47 | .80 |
Table 2 - Means, Standard Deviations, and Reliability Coefficients for the normative population sample on the self-Genos
| Scale | # items | Mean | Standard Deviation | a |
| Total EI | 64 | 225.37 | 20.14 | .91 |
| Emotional Recognition & Expression | 11 | 38.70 | 5.34 | .78 |
| Understanding of Emotions External | 20 | 76.54 | 7.78 | .86 |
| Emotions Direct Cognition | 12 | 35.19 | 6.14 | .81 |
| Emotional Management | 12 | 42.24 | 5.69 | .81 |
| Emotional Control | 9 | 32.71 | 4.66 | .80 |
As shown in Table 1, full-scale reliability is high as is the reliability for each of the sub-scales for the normative population sample on the self-Genos. The general sample consists of 1522 individuals (984 Females, 487 Males, 51 did not nominate their sex), the ages of individuals who completed the self-Genos ranged from 18 – 72 (Mean: 40.43, SD: 10.39). Of the people who supplied their level of education (n~450): 10% had completed their high school certificate, 31% had completed a tertiary certificate, 30% had completed an undergraduate degree, and 29% had completed a postgraduate degree. As can be seen by comparing and contrasting Table 1 and 2 the means, standard deviations and internal consistency reliability of the rater sample and normative population sample are relatively comparative indicating some consistency between how people rate themselves and how raters rate subjects EI.
360-Degree Rating Congruence
Table 3 details the rating congruence between 360 degree and self-ratings of 171 people having undertaken the Genos-360. This analysis was undertaken to assess for instance if somebody has low self-reported understanding of emotions (UE) of others, is that person more likely to rate others low on this dimension and visa versa. This analysis was undertaken for every one of the five dimensions of the Genos-360, ie if I was low in Emotional Control, would I then be likely to rate others low on Emotional Control?
Table 3 - Intercorrelations between Emotional Intelligence self and 360 degree ratings| Self Rated Emotional Intelligence |
|||||
| Raters Emotional Intelligence | ERE | UE | EDC | EM | EC |
| ERE | 0.404** | 0.332 | 0.194* | 0.057 | -.009 |
| UE | 0.298** | 0.356** | 0.227** | 0.136 | 0.118 |
| EDC | 0.184* | -0.028 | 0.045 | -0.151* | -0.153* |
| EM | 0.135 | 0.145 | 0.064 | 0.177* | 0.198** |
| EC | -0.058 | 0.018 | -0.123 | 0.238** | 0.280** |
Based on the random sample of 171 people taken from the Genos database a small effect is evident, accounting for not more than about 0-16% of the variance (relationship).
To summarize:
Self report ERE correlates at 0.40 with ratings of others ERE (16%)
Self report UE correlates at 0.36 with ratings of UE (13%)
Self report EDC correlates at 0.05 with ratings of UE (.25%)
Self report EM correlates at 0.18 with ratings of EM (2.0%)
Self report EC correlates at 0.28 with ratings of EC (7.8%)
The highest correlation between how someone rates themselves and how they rate other people was found between the variables that you would expect, Emotional Recognition and Expression (ERE) and Understanding Emotions (UE). Despite this, there seems to be no systematic pattern related to ones own EI self-ratings and their ratings of others.
ConclusionThe findings presented in this study show that EI can be reliably rated by others and that the means and standard deviations of a rater sample are similar to those of a self-rating sample. The findings also show that little relationship exists between how people rate themselves and how they rate others. Thus people with self-rated low EI can typically rate others as high or visa versa. This finding suggests that one does not need insight into EI or indeed perhaps actual EI in order to provide a valid perception of how they perceive someone else’s emotionally intelligent behaviour in the workplace. While this data provides some initial insight into the how people rate other peoples emotionally intelligent behaviours in the workplace many questions remain unanswered and need to be further addressed. These questions include; what is the relationship between self and other ratings; are higher levels of congruence between self and other ratings associated with various management and leadership outcomes (such as performance and leadership effectiveness as has been shown in other multi-rater research); what are better predictors of leadership effectiveness and performance, that is, self-rated, direct-report rated, peer-rated or management rated EI; and when compared with self-ratings only, can ratings from others enhance self-awareness and behavioural change and/or emotional development as has been previously shown in research (e.g., Church, 2000).
Research with multi-rater assessment instruments has shown that (a) self-ratings are typically poor predictors of performance (Church, 2000; Harris & Schaubroech, 1988; Sala & Dwight, 2002); (b) self-ratings are typically positively biased (Church, 1997; Podsakoff & Organ, 1986; Sala, 2003); and (c) that ratings from others can serve an important role in providing insights into the extent to which others perceive certain behaviours and/or levels of performance associated with a leader (Sala, 2002). At the outset we argued that multi-rater or 360 degree measures of EI may play an important role in providing development practitioners and the leaders they work with, with additional insight into the extent to which others perceive them to be displaying emotionally intelligent behaviours in the workplace. An important next step in this area will be to demonstrate the efficacy of this approach over and above simply self-rated insight in EI leadership development programs.
ReferencesAtwater, L.E., & Yammarino, F.J. (1992). Does self-other agreement on leadership perceptions moderate the validity of leadership and performance predictions? Personnel Psychology, 45, 141-164.
Bagby, R.M., Taylor, G.J., & Parker J.D.A. (1994). The twenty-item Toronto alexithymia scale – II: Convergent, discriminant, and concurrent validity. Journal of Psychosomatic Research, 38, 33-40.
Barling, J., Slater, F., Kelloway, E. K. (2000). Transformational leadership and emotional intelligence: an exploratory study. Leadership and Organization Development Journal, 21, 157-161
Bar-On, R. (1997). Emotional Quotient Inventory (EQ-I): Technical Manual. Toronto, Canada : Multi-Health Systems.
Becton, J.B., & Schraeder, M. (2004). Participant input into rater selection: Potential effects on the quality and acceptance of ratings in the context of 360-degree feedback.
Borman, W. (1974). The rating of individuals in organisations: An alternative approach. Organisational Behaviour and Human Decision Processes, 12, 105-124.
Caruso, D., Mayer, J.D., & Salovey, P. (in press). Emotional intelligence and emotional leadership. In R. Riggio & S. Murphy (Eds.), Multiple intelligences and leadership. Mahwah, NJ : Lawrence Erlbaum Associates.
Church, A.H. (1997). Managerial self-awareness in high performing individuals in organisations. Journal of Applied Psychology, 82, 281-292.
Chruch, A.H. (2000). Do higher performing managers actually receive better ratings? A validation of multi-rater assessment methodology. Consulting Psychology Journal: Practice and Research, 54, 166-172.
Gerguras, G.J., & Robie, C. (1998). A new look at within-source interrater reliability of 360-degree feedback ratings. Journal of Applied Psychology, 83, 960-968.
Goleman, D. (1998a) What makes a leader?. Harvard Business Review, 76, 93-104.
Goleman, D. (1998b). Working with emotional intelligence . Bloomsbury Publishing, London.
Goleman, D. (2001). Emotional Intelligence: Issues in Paradigm Building. In C. Cherniss and D.Goleman (Eds). The Emotionally Intelligence Workplace. New York : Jossey-Bass.
Johnstone, J.W. & Ferstl, K.L. (1999). The effects of inter-rater and self other agreement on performance improvement following upward feedback. Personnel Psychology, 52, 217-303.
London, M., & Smither, J.W. (1995). Can multi-source feedback change perceptions of goal accomplishment, self-evaluations, and performance-related outcomes? Personnel Psychology, 48, 803-839.
Mayer, J., Caruso, D.R., & Salovey, P. (2000). Selecting a measure of emotional intelligence: The case for ability scales. In R. Bar-On & J.D.A., Parker, (Eds). The handbook of emotional intelligence. (pp.320-342) San Francisco : Jossey-Bass Inc.
Mayer, J. & Salovey, P. (1997). What is emotional intelligence? In P. Salovey & D. Sluyter (Eds.). Emotional development and emotional intelligence: Implications for educators. (pp3-31). New York : Basicbooks, Inc.
Mayer, J.D., Salovey, P., & Caruso, D. (1999). Instruction Manual for the MSCEIT: Mayer Salovey-Caruso Emotional Intelligence Test, Multi-Health Systems, Toronto.
Palmer, B.R., Gardner, L., & Stough, C. (2003). Measuring emotional intelligence in the workplace with the Swinburne University Emotional Intelligence Test. Published in the Proceedings of the First International Conference on Contemporary Management (ICCM): Emotional Intelligence in Organisations, Sep. 2003
Palmer, B., & Stough, C. (2001). Workplace SUEIT: Swinburne University Emotional Intelligence Test – Descriptive Report , Organisational Psychology Research Unit, Swinburne University, AU
Sala, F. (2003). Executive blind spots: Discrepancies between self- and other-ratings. Consulting Psychology Journal: Practice and Research, 55, 222-229.
Sala, F. & Dwight, S.A. (2002). Predicting executive performance with multi-rater surveys: Whom you ask makes a difference. Consulting Psychology Journal: Practice and Research, 54, 166-172.
Salovey, P. & Mayer, J.D. (1990). Emotional intelligence. Imagination, Cognition and Personality, 9, 185-211.
Salovey, P., Mayer, J.D., Goldman, S.L., Turvey, C., & Palfai, T.P. (1995). Emotional attention, clarity, and repair: Exploring emotional intelligence using the trait meta-mood scale. In J.W. Pennebaker (Ed.), Emotion, Disclosure and Health (pp.125-154). Washington, D.C. : American Psychological Association.
Schutte, N.S., Malouff, J.M., Hall, L.E., Haggerty, D.J., Cooper, J.T., Golden, C.J., Dornheim, L. (1998). Development and validation of a measure of emotional intelligence. Personality and Individual Differences, 25, 167-177.
Tett, R., Wang, A., Thomas, M., Griebler, J., & Linkovich, T. (April 4, 1997). Testing a Model of Emotional Intelligence . Paper presented at the 1997 Annual Convention of the Southeastern Psychological Association, Atlanta, GA.