Original Research

An investigation of gender-based differences in assessment instruments: A test of measurement invariance

Renier Steyn, Gideon P. de Bruin

Received: 01 June 2019; Accepted: 02 Nov. 2019; Published: 18 Mar. 2020

Copyright: © 2020. The Author(s). Licensee: AOSIS.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Orientation: Practitioners and researchers often assume that the psychometric instruments they use are invariant and that they therefore measure similar constructs in a comparable manner across men and women respondents. This assumption is, however, rarely tested, leading to an undetected bias in research findings or an adverse impact because of the presence of non-invariance.

Research purpose: After presenting essential information about measurement invariance (MI) and arguing for the testing thereof, this research aims to reveal the prevalence of MI across several frequently used psychometric instruments credulously used based on the assumption the revenant constructs are measured equivalently across gender exists.

Motivation for the study: Firstly, this study aims to increase awareness regarding MI, a property that can be tested statistically. Secondly, the research aims to make practitioners aware of the presence of bias in psychometric instruments, specifically to identify instruments that could be included in investigations which attempt to understand gender matters in the workplace.

Research approach/design and method: Cross-sectional survey data, pertaining to seven standard instruments, related to innovative work behaviour, were analysed. Pairwise, multigroup confirmatory factor analyses with robust maximum likelihood estimation were used to examine configural, metric, intercept and strict invariance, as well as the equivalence of the latent means.

Main findings: The findings were binary, with four of the instruments showing MI at an equal latent means level, whilst three instruments were non-invariant at the configural level. Measurement invariance was either accepted completely or rejected completely.

Practical/managerial implications: The serratedness of findings, even when using well-recognised and frequently used psychometric instruments, exposes the prevalence of non-invariance in some instruments, thereby necessitating the standard testing for MI. These findings also specify the instruments that are MI (in terms of gender), which allow other researchers and practitioners to use these instruments with more confidence when measuring and comparing men and women respondents in their studies.

Contribution/value-add: This research demonstrates the ease with which MI testing can be performed and alerts researchers to do MI testing when conducting cross-group studies, as the prevalence of measurement non-invariance is high.

Keywords: gender; measurement invariance; bias; adverse effect; group differences; innovative work behaviour.

Introduction

Gender is a prominent, salient variable within the organisational workplace. In many studies, the perceptions of men and women are compared, or measures of perceptions are used in models to test hypotheses related to gender differences (Cropley & Cropley, 2017; Eagly, 1997; Eagly & Wood, 1999; Koch, D’Mello, & Sackett, 2015). Differential outcomes based on gender are often reported (Eagly & Karau, 2002; Ismail & Nakkache, 2015). Often, these differential outcomes are explained from a sociological perspective, where differences are attributed to gender-specific roles, attributions, stereotypical expectations, performance or attitudes (Hatlevik, Scherer, & Christophersen, 2017). In many of these studies, it is assumed that measures of perceptions are accurate and equally valid for men and women. Examples of such ‘naïve’ studies are plentiful (Eagly, Johannesen-Schmidt, & Van Engen, 2003; Selvarajan, Slattery, & Stringer, 2015; Tabvuma, Georgellis, & Lang, 2015; Wang & Gorenstein, 2015; Yi, Ribbens, Fu, & Cheng, 2015). In all of the aforementioned studies, it is assumed that measures of perceptions are not gender-specific. It was found that in none of these studies the authors tested for the possibility that the measurement characteristics might differ depending on gender.

When considering differences between groups (e.g. men and women), it could be meaningful to go beyond the sociological explanations (Hatlevik et al., 2017) and (firstly) question the assumption that individual test items and/or the entire scale operate equally across the groups (Millsap, 2011; Tsaousis & Kazi, 2013; Vandenberg & Lance, 2000). The concern, as raised above, is that this assumption is hardly ever tested explicitly, and according to Tsaousis and Kazi (2013), this omission renders all such comparative studies’ findings highly questionable. Do different groups of respondents interpret a given measure in a conceptually similar manner? Stated more operationally, are the relationships between manifest indicator variables (scale items, subscales) and the underlying construct the same across groups (Bialosiewicz, Murphy, & Berry, 2013)? Should the construct not be measured equivalently, it will cause bias in the inferences drawn and therefore threaten the validity of the comparisons made (Hatlevik et al., 2017). These potential deviations from equivalence are referred to as measurement non-invariance (Holland & Wainer, 1993). To rule out the possibility that variations in the functioning of a scale result in biased interpretations of results, testing for measurement invariance (MI) can assist with clarity in this regard (Hatlevik et al., 2017).

The academic community is certainly not naïve with regard to the possibility of gender-based MI, and some studies do include tests of MI (Kuhn & Holling, 2009; Van Zyl, 2016; Zampetakis, Bakatsaki, Litos, Kafetsios, & Moustakis, 2017). This is a relatively new trend, however, with most studies failing to test for MI across groups of interest prior to making comparisons (Tsaousis & Kazi, 2013). This article aims to contribute to the literature and the practice of gender-based research by critically analysing the present-day call for the testing of (gender-based) measuring invariance in studies where group (gender) differences are investigated. This will firstly be done by discussing MI, the levels of analyses, as well as the analyses themselves. The most significant contribution of this article lies in the testing of MI across several scales, using the same set of respondents. Presenting tests of MI across several scales, using a common pool of respondents, is envisaged to present rich information on the prevalence, as well as the extent of MI in these scales, which all measure elements within the domain of organisational behaviour. The findings of this research allow for a critical analysis of broad statements that declare that ‘investigating measurement invariance should now become a routine part of research into the structure of group differences’ (Borsboom, 2006, p. 176).

Literature review

Essential to psychometric assessment is that decisions will follow from the scores created, where individuals are categorised based on cut-off scores and afforded or denied opportunities based on this categorisation (Cohen, Swerdlik, & Sturman, 2013; Gregory, 2011). It is, however, possible that the administration or the nature of a measurement results in scores that systematically prevent accurate or impartial decisions (Cohen et al., 2013), excluding specific groups of individuals. When a score is not based on individuals’ standing on the construct, but rather on the individuals’ group membership, the measurement is considered to be biased (Berry, 2015; Fontaine, 2008).

Group differences, bias and measurement invariance

An absence of measurement bias against distinct groups is a prerequisite for the use of a given measure in research or workplace assessment (Lee, Lee, Wells, & Sireci, 2016). The Constitution of the Republic of South Africa (Act 106 of 1996), and particularly the Bill of Rights, as well as the Employment Equity Act (Act 55 of 1998) promotes equity, equal opportunity and fair treatment, specifically referring to the fact that:

[P]sychometric testing and other similar assessments of an employee are prohibited unless the test or assessment being used has been scientifically shown to be valid and reliable, can be applied fairly to all employees, is not biased against any employee. (p. 2)

Differences in psychometric test results between groups (men and women) may be the result of substantive differences between the groups, or they may stem from biased measurements (Berry, Poortinga, Breugelmans, Chasiotis, & Sam, 2011). Three types of bias which may result in differences between groups are commonly identified, namely, construct, method and item bias:

• Construct bias may exist when a measured construct is specific to a particular group, or when items related to the construct are underrepresented in the instrument for one group, and where identical indicators (items) cannot be used across groups, as the repertoire of behaviour associated with the construct differs substantively (Fontaine, 2008; Harzing, 2006).
• Method bias relates to variation in the scores which results from inadequacy in the instructions of the instrument, response styles that elicit a specific way of responding in one group (e.g. more acquiescent responding) or group characteristics that differ along group lines on how to respond to the instrument (e.g. motivation to respond in line with a specific stereotype) (Fontaine, 2008; Libbrecht, Beuckelaer, Lievens, & Rockstuhl, 2014).
• Item bias, of particular interest in this article, materialises when an item systematically has a higher or lower score than expected in a specific group, given other indicators of the construct (Fontaine, 2008). Stated differently, the scores of two persons may differ, not in terms of their standing on the latent construct but rather depending on their group status (Berry, 2015; Fontaine, 2008).

Detecting group differences based on construct heterogeneity (when the construct is not defined or when it measures in the same manner [Casper, Vaziri, Wayne, DeHauw, & Greenhaus, 2017; Van Zyl, 2016]), as well as measurement bias, can be addressed through testing for MI. Measurement invariance is concerned with identifying items or instruments to which individuals respond in a similar manner and asking whether respondents from different groups interpret a given measure in a conceptually similar manner (Vandenberg & Lance, 2000).

Observed mean score comparisons are based on the assumption of invariant or equivalent measurement across groups (Vandenberg & Lance, 2000). Although rarely tested (Tsaousis & Kazi, 2013), these assumptions are routinely and straightforwardly testable as extensions to the basic confirmatory factor analysis (CFA) framework. If not tested, violations of measurement equivalence assumptions threaten substantive interpretations of the results and equate to an inability to demonstrate reliability and validity (Bialosiewicz et al., 2013; Vandenberg & Lance, 2000).

It is the task of the researcher to ensure that the interpretation of the data is not distorted because of inequivalence (Berry et al., 2011). Having determined MI, researchers can compare the occurrence, antecedents and consequences of the latent factor scores across groups (Van de Schoot, Lugtig, & Hox, 2012). Testing for MI has important implications when decisions about individuals in a diverse workplace are to be made (Borsboom, 2006; Cheung & Rensvold, 2002). The meaningful interpretations of test scores across groups (nations, ethnic groups, age and gender) do not only serve organisational goals but also serve to safeguard against the possibility of bias or a so-called adverse impact because of non-invariance (Van Zyl, 2016; Whitman, Kraus & Van Rooy, 2014).

Measurement invariance assessment

The existing literature on MI is abundant (Van de Schoot, Schmidt, & De Beuckelaer, 2015). However, in spite of Vandenberg and Lance’s (2000) seminal work, in which the authors review and synthesise MI literature, the terminology and practices are still not aligned, and some confusion in the meaning of terms and uniformity in practices prevails. Presented below is an attempt to integrate some of the MI literature.

Testing for MI usually occurs within the CFA framework (Vandenberg & Lance, 2000), and the process of assessing MI essentially involves the testing of a series of increasingly restrictive equality constraints hypotheses (Bialosiewicz et al., 2013). It makes sense to discuss the levels of MI within the context of increasingly restrictive models:

• Conceptual equivalence (functional equivalence, construct bias) refers to a situation where the domain or trait makes sense in all the groups that are compared (Berry et al., 2011). When a measured construct is specific to a particular group, it would therefore be impossible to find a comparable operational pattern of relationships with other constructs, across the groups (Fontaine, 2008). Although no statistical tests directly test conceptual equivalence, Berry et al. (2011) stated that evidence of configural invariance supports claims regarding conceptual equivalence. Within the context of gender, postpartum depression items and the management of prostate cancer items may serve as examples of conceptual inequivalence or uniqueness.
• Configural invariance (structural invariance, pattern invariance, the baseline model) involves testing whether or not the same items measure corresponding constructs across groups (Bialosiewicz et al., 2013). In an exploratory factor analysis, the same items would load on the same factors, across groups. It simply implies that the zero or non-zero loadings on the factors (constructs) are the same across groups (Selig, Card, & Little, 2008). This could likewise be done through fitting the same CFA model to each group separately, running the CFA for each group and comparing the fit indexes. Alternatively, and using a comprehensive statistic, running a multiple group analysis without any equality constraints will provide the requisite information regarding configural fit (Van de Schoot et al., 2012). Once this baseline model is established, further testing of invariance can follow, as these results suggest some conceptual equivalence (Berry et al., 2011). No further comparisons, or further tests of MI between groups, are warranted should this level of invariance fail to materialise (Gunn, 2016).
• Metric invariance (weak invariance, loading invariance) builds upon configural invariance by requiring that in addition to the constructs being measured by the same items, the factor loadings of those items must be equivalent across groups (Bialosiewicz et al., 2013). Indicators (items) that are central to the construct in one group (men) should also be central in the other group (women), and those that are less central in one group (men) should also feature less prominently in the other group (women) (Selig et al., 2008). Attaining invariance of factor loadings therefore suggests that the construct has the same meaning to participants across groups (Bialosiewicz et al., 2013). To test for MI at a metric level, a model where only the factor loadings are equal across groups (although the intercepts are allowed to differ between groups) should be run (Van de Schoot et al., 2012). If there is no significant difference in model fit, then there is evidence to suggest that the factor loadings are invariant across groups. Attaining metric invariance suggests that group comparisons of factor variances and covariances are defensible. However, it does not justify the comparisons of group means (Bialosiewicz et al., 2013). (A violation of metric invariance implies non-uniform bias [Barendse, Albers, Oort, & Timmerman, 2015; Fontaine, 2008].)
• Scalar invariance (strong, full-score equivalence) builds upon metric invariance by requiring that the item intercepts also be equivalent across groups. To assess scalar invariance, we compare the fit of the scalar model with the fit of the metric model (Bialosiewicz et al., 2013), which requires the running of a model where the loadings and intercepts are constrained to be equal (Van de Schoot et al., 2012). If there is no significant difference in model fit, then there is evidence to suggest intercept invariance. Item intercepts reflect the starting value of the scale on which the factor is based and then, given equivalent slopes (matric invariance), equivalent intercepts justify comparisons of the latent means across groups (Bialosiewicz et al., 2013; Van de Schoot et al., 2012). (Non-invariance of intercepts may be indicative of uniform bias [Barendse et al., 2015; Fontaine, 2008].) Once scalar invariance is established, there is sufficient evidence to claim that the indicators are measuring the same underlying construct and that any observed differences in the construct relate to veridical differences (Selig et al., 2008), thereby corresponding with reality or facts. This should be seen to signify that there are larger forces, such as cultural norms or developmental differences, that are influencing the way in which participants respond to items across groups and that participants are systematically rating items either higher or lower (Bialosiewicz et al., 2013).
• Strict invariance (full-uniqueness, invariant uniqueness, strict factorial invariance) is concerned with residual error equivalence across groups (Bialosiewicz et al., 2013). It relates to the overall error in the prediction of the construct, as well as unique errors specific to particular indicator variables. When testing strict invariance, you are therefore essentially testing whether your residual error is equivalent across administrations (Bialosiewicz et al., 2013), implying a test of test reliability across groups. Strict invariance represents a highly constrained model and is rarely achieved in practice. Most agree that attaining strict invariance is unreasonable (Bialosiewicz et al., 2013; Byrne, 2009; Chen, Carolina, Curran, Bollen, & Kirby, 2009; Vandenberg & Lance, 2000).
• Latent mean invariance (invariant factor means, latent mean analysis) builds upon the preceding levels of invariance, and refers to a test of the null hypothesis of equal factor means across groups (Vandenberg & Lance, 2000), and test for significant differences between groups on the level for the construct of interest (Gygi, Fux, Grob, & Hagmann-von Arx, 2016). Once latent mean invariance is achieved, it can be stated that not only does the different groups (e.g. men and women) perceive the items in the same manner but also their scores on the constructs are similar.

All tests for MI usually occur within a CFA framework (Vandenberg & Lance, 2000), where increasingly restrictive equality constrained hypotheses are tested (Bialosiewicz et al., 2013). The parameters for the decision-making regarding these hypotheses are integrated below.

Analysis guidelines for measurement invariance

Measurement invariance refers to the statistical property of a measurement instrument, which indicates that the same underlying construct(s) is being measured across groups. This will be evident when the relationship between the manifest variables (the observed variables such as scale items and the subscale scores) and the underlying construct (the latent variable(s)) is the same across groups (Van de Schoot et al., 2015). The aforementioned makes sense when we assume that psychometric instruments comprised a single or several sets of items, which, when combined, are intended to assess a latent construct or constructs. Within the framework of CFA, it signifies that the common factor model holds across groups (Bialosiewicz et al., 2013). The guidelines presented below align with the assessment of independent clusters CFA (IC-CFA) models.

The most commonly used test to check global model fit is the chi-square test (χ²) (Millsap, 2011; Van de Schoot et al., 2012). It is a test of perfect fit of the ‘actual covariance and mean structure, and the covariance and mean structure implied by the hypothesized model’ (Millsap, 2011, p. 93). The χ²-statistic is dependent on the sample size, resulting in rejections of reasonable models if the samples are large and failure to reject poor models if the samples are small (Van de Schoot et al., 2012). Although highly desirable, it can be expected that the hypothesis of perfect fit for models would be rejected in larger samples, and for this reason, the χ²-statistic is no longer relied upon as a basis for acceptance or rejection of a model fit (Schermelleh-Engel, Moosbrugger, & Müller, 2003; Vandenberg, 2006). However, a statistically significant difference in χ² between a less constrained (e.g. metric invariance) and a more constrained model (e.g. scalar invariance) can be deemed as evident of a deteriorating model fit.

Kline (2010) suggests three more types of fit indices that can be used to assess the fit of a model:

• Firstly, the comparative indices compare the fit of the model under consideration with the fit of the baseline model. Examples of comparative indices are the Tucker–Lewis Index (TLI) and Comparative Fit Index (CFI). Awang (2012), Hair, Black, Babin and Anderson (2010) and Van de Schoot et al. (2012) stated that the model fit is acceptable when the TLI or CFI is > 0.90, whilst others (Schreiber et al., 2006) set the margin at > 0.95. Tucker–Lewis Index or CFI values are acceptable if they are > 0.90, and better if they are > 0.95. (Van de Schoot et al., 2012). Tucker–Lewis Index can also become > 1.0, which should be interpreted as overfitting the model, making it more complex than what was initially required. When comparing less constrained models to more constrained models within the MI context, Vandenberg and Lance (2000) noted that:

  [C]hanges in CFI of -0.01 or less indicate that the invariance hypothesis should not be rejected, but when the differences lie between -0.01 and -0.02, the researcher should be aware that differences exist. Definite differences between models exist when the change in CFI is greater than -0.02. (p. 46)

• Secondly, there are absolute indices that examine closeness of fit, with the most used method being the Root Mean Square Error of Approximation (RMSEA) (Vandenberg & Lance, 2000). Some suggest that the cut-off value for RMSEA should be < 0.08, suggesting < 0.05 as better (Van de Schoot et al., 2012), whilst others suggest that the cut-off value for RMSEA should be < 0.06 (Schreiber et al., 2006). Awang (2012) and Hair et al. (2010) suggested that the model fit is acceptable when RMSEA is < 0.05. As no critical values for the change of RMSEA following the application of more constrained models could be located, the same principles as in the case of CFI could be followed where sequential model fits are compared.
• Thirdly, there are information theoretic indices, such as the Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC). Akaike Information Criterion and BIC can be used to compare competing models (Van de Schoot et al., 2012) but should not be applied for the testing of a single model (Schreiber et al., 2006). Lower AIC or BIC values indicate a favourable trade-off between fit and complexity. Whilst the calculation of the actual AIC and BIC values is a very complex matter, the rule of thumb is to simply choose the model with the lowest AIC or BIC value as the best fitting model (Schreiber et al., 2006; Van de Schoot et al., 2012).

The indices used in this study as well as the parameters used when evaluating different models will be discussed in the ‘Methods’ section of this article, under the heading ‘Analysis’.

Methods

The aim of this empirical study was to assess the level of MI across seven instruments, using the same pool of respondents, and then comparing the three assessments of MI of perceptions about the self (in the organisation) with the three measures on perceptions about an organisation (as an entity beyond the self). The discussion which will follow the uniformity of the presence of gender-based MI in general, as well as the possible differences in MI reporting on the self, versus reporting on organisation is presented.

Population and sampling

The target population comprised men and women who were employed within organisations with more than 60 employees and who could report on their perceptions about their own behaviour, as well as on their respective organisations. Companies with more than 60 employees were targeted, as it was presumed that such organisations would have formalised organisational features which might make reporting more uniform. Organisations were included in the study based on their managers’ willingness to grant permission to participate, resulting in a convenient sample of organisations. Respondents (employees) were randomly selected from personnel lists provided by the participating organisations. The selection of respondents was therefore as random as possible, given the operational realities of recruiting respondents. The data were collected in compliance with the requirements of the research ethics committee, and standard matters such as specifying the aim of the study, anonymity, confidentiality, the right to withdraw were discussed.

Measurement instrument

In total, seven instruments were administered. These instruments were selected as they were found to be well-suited for inclusion in a predictive model of innovative work behaviour. The first three measured perceptions about organisation (as an entity beyond the self) and the rest measured perceptions about the self (in the organisation). Only brief descriptions of the instruments are provided below, so as to keep the article as concise as possible and in light of the fact that most of these instruments should be well-known to the reader:

• Brief Corporate Entrepreneurship Assessment Instrument (Hornsby, Kuratko, & Zahra, 2002; Strydom, 2013). Hornsby et al. (2002) are important authors with regard to the conceptualisation and measurement of an organisational climate associated with innovation in the workplace. They developed a 48-item questionnaire to assess the five factors that influence innovation in the workplace: level of management support, work discretion or autonomy, rewards and reinforcement, time availability and organisational boundaries (Hornsby et al. 2002). Strydom (2013) developed a brief version of the instrument, using only 20 items – four per factor. Strydom (2013) reported alphas of 0.731, 0.825, 0.742, 0.689 and 0.574 for the subscales and a reliability coefficient of 0.810 for the entire instrument. Strydom (2013) also provided information on the predictive validity of the instrument.
• Human Resource Practices Scale (HRPS). The HRPS (Nyawose, 2009) was developed on a rational basis by examining the literature on different human resource management practices. Seven human resource management practices (HRMP) were measured in this study, and the questionnaire consisted of 21 items. The HRPS has a hierarchical structure, with each of the seven factors consisting of three items. The factors are training and development, remuneration, performance management, supervisor support, staffing, diversity management and communication. Nyawose (2009) reported reliabilities varying from 0.74 to 0.93, whilst Steyn (2012) reported Cronbach’s alphas of 0.74 to 0.88. Nyawose (2009) and Steyn (2012) reported results pertaining to the predictive validity of the HRPS.
• Multifactor Leadership Questionnaire. The Multifactor Leadership Questionnaire (MLQ) (Avolio, Bass, & Jung, 1995, 1999) is one of the most frequently used measures of leadership styles (Lowe, Kroeck, & Sivasubramaniam, 1996) and measures transformational, transactional and laissez-faire leadership styles, using 21 items. Dumdum, Lowe and Avolio (2002) reported acceptable reliability and validity for the MLQ in their meta-analysis. Remarkably, Eagly et al. (2003) published a study entitled Transformational, Transactional, and Laissez-Faire Leadership Styles: A Meta-Analysis Comparing Women and Men, without making any reference to MI.
• Employee Engagement (Utrecht Work Engagement Scale [UWES-9]). This nine-item instrument measures three dimensions of employee engagement, namely, vigour, dedication and absorption (Schaufeli, Bakker, & Salanova, 2006). Schaufeli and Bakker (2003) reported that the ‘Cronbach’s α of all nine items varies from 0.85 to 0.94 (median = 0.91) across the nine national samples. The α-value for the total database is 0.90’ (p. 33). Schaufeli and Bakker (2003) also reported that the suggested -three-factor structure of engagement is confirmed (-cross-samples from different countries) and that the construct is related to other constructs in the expected manner. De Bruin and Henn (2013) could not replicate the three-factor structure and reported ‘the presence of a very strong general factor and, in comparison, two weak group factors’ (p. 788).
• Organisational Commitment Scale. The Organisational Commitment Scale (OCS) was developed by Allen and Meyer (1990) to assess affective, continuance and normative commitment, with eight items per dimension. Only the items of affective commitment were used, as it is common practice to interpret the sections of the test separately, and as affective commitment is an effective predictor of many organisational variables (Lamba & Choudhary, 2013; Wright & Kehoe, 2007). Allen and Meyer (1990) reported an internal consistency coefficient of 0.86 for the affective commitment section and commented that the ‘relationship between (though) commitment measures … and the antecedent variables … was, for the most part, consistent with prediction’ (p. 13). This points to convergent and discriminant validity. Steyn (2012) reported a Cronbach’s alpha coefficient of 0.82 for the eight items. Steyn, Bezuidenhout, and Grobler (2017) reported relationships between affective commitment and antecedent variables consistent with what was expected.
• Innovative Work Behaviour. The 14 innovative work behaviour (IWB) items present elements descriptive of individual innovation, namely, opportunity exploration, generativity, information investigation, championing and application (Kleysen & Street, 2001). Hebenstreit (2003) reported an alpha of 0.948 when using all the items. Lu and Li (2010) could not replicate a five-factor structure and reported Cronbach’s alpha values of 0.860 for the two factors they extracted. Wojtczuk-Turek and Turek (2013) also reported on a two-factor solution, with values of 0.880 and 0.890. Although empirical support for the theorised structure was mixed, Kleysen and Street (2001) suggested the use of the items as a single measure of innovation behaviour, as did Hebenstreit (2003). In this study, the original five elements were used in the measurement model, which was tested.
• Individual Proactiveness. The 17-item instrument was developed to ‘investigate a personal disposition towards proactive behavior, defined as the relatively stable tendency to effect environmental change’ (Bateman & Crant, 1993, p. 103). It was established through factor analyses that the 17-item instrument is a unidimensional scale with sound psychometric properties, including coefficient alphas varying from 0.85 (Crant & Bateman, 2002) to 0.93 (Crant, 1996). Information on discriminant and predictive validity is also provided by the developers (see Bateman & Crant, 1993; Seibert, Kraimer, & Crant, 2001). Conceptually, the instrument seems well accepted by researchers as a test of proactiveness and a predictor of important organisational outcomes (see Thomas, Whitman, & Viswesvaran, 2010).

Analysis

Descriptive statistics on all seven measures were calculated using SPSS (Version 25.0) (IBM Corp, 2017). These included per gender means and standard deviations, as well as kurtosis and skewness. Within the context of SPSS, significant deviations from normality occur when the skewness/standard error of skewness or kurtosis/standard error of kurtosis has an absolute value > 2 (Weinberg & Abramowitz, 2008). Field (2009, p. 139) seemed more lenient and is comfortable with values below 3.29, also warning against using skew and kurtosis tests in large samples (N > 200), suggesting a visual inspection of the distribution rather than using significance tests because of their sensitivity. In this study, the skewness or kurtosis > 3 will be interpreted as a deviation from normality.

Measurement invariance pertaining to gender was tested for in each of the seven instruments. Following the recommendations of Vandenberg and Lance (2000), pairwise multigroup confirmatory factor analyses with robust maximum likelihood estimation (Wu, Li, & Zumbo, 2007) were used to examine configural, metric, intercept and strict invariance as well as, finally, the equivalence of the latent means. The analyses were performed using the lavaan package (Rosseel, 2012) in R (R Core Team, 2013).

As a non-significant χ²-statistic is highly unlikely given the sample size (Schermelleh-Engel et al., 2003; Vandenberg, 2006), it will not be used as a decisive indicator of model fit. The χ²-statistic will only be reported should it be non-significant. Akaike Information Criterion and BIC, CFI, RMSEA, as well as changes in CFI (∆CFI) and changes in RMSEA (∆RMSEA), were used to test for model fit. The selection of the particular indices was also informed by the standard outputs of the statistical software used.

Following on the discussion presented earlier in this text, the models with the lowest AIC and BIC values were judged as the best fitting models. Comparative Fit Index values > 0.90 were judged as adequate, and lenient RMSEA values < 0.08 as acceptable. With regard to both ∆CFI and ∆RMSEA, a change > 0.01 was seen as an indicator of a deteriorating model, with changes > 0.02 seen as a clear sign of differences between the models.

Ethical considerations

Ethical approval for the study was granted from the University of South Africa’s Graduate School of Business Leadership Research Ethics Review Committee (Reference number: 2014_SBL_018_CA).

Results

Demographics

Respondents from 52 organisations participated in the study. In total, 3180 questionnaires were completed. More men (55.3%) than women (47.7%) completed the questionnaires. All the major race or ethnic groups were represented, with approximately 58% black people, 24% white people and 8% each for Asian and people of mixed ethnicity. The mean age of the respondents was 37.8 years (standard deviation 9.1). For tenure, the mean was 9.0 years (standard deviation 7.5). Because of the large number of organisations included in the study, and the diversity in terms of gender, race, age and tenure, the respondents were heterogeneous and relatively free from any particular context, which some authors (Els, Mostert, & Brouwers, 2016) deemed appropriate to assess bias and equivalence.

Descriptive statistics

In Table 1, descriptive statistics per gender are presented. Data for 3143 respondents were available, with a split of 56.4% men and 43.6% women. In all but one instance (IWB – women), the skewness was negative, and in only one case, it fell within the threshold of being < 3 (IWB – men). Inspection of the distributions also showed high levels of negative skewness. Although skewness was found in most cases, it was not deemed as problematic, as social desirability, acquiescence and leniency are common in organisational research response patterns, with all these elements contributing to negative skewness (Spector & Brannick, 2009). Unlike skewness, several of the kurtosis values were within the normality range, with the clear exception being individual proactiveness (IPA), for both men and women. Despite these deviations, it was decided to continue with further analyses, given the appropriate and adequate sample size, and particularly the similarity in the distributions across gender.

It is noteworthy that skewness and kurtosis values were specific to particular instruments – and not to gender.

Reliability

The reliability coefficients for the instruments are presented in the last column of Table 1. Apart from the very low coefficients for the three items of the measures of Laissez–Faire leadership, the coefficients were moderate to high, with similar values across gender, and female respondents provided mostly more reliable responses.

Mean differences

Mean and mean differences were calculated, whilst remaining cognisant of the fact that means may differ because of bias rather than substantive differences between men and women. These are presented in Table 2. These differences should be considered as valid only once the instruments are proven to be measurement invariant.

From Table 2, it can be observed that the mean scores differed significantly (p < 0.01) on two instruments, OCS and IWB. When considering the Cohen’s d-values, which is an expression of the difference in terms of standard deviation units, the difference was 10.6% of one standard deviation for OCS and 17.5% of one standard deviation for IWB. These differences have a small practical effect size.

Gender-based measurement invariance

All calculated χ²-statistics were significant, necessitating the rejection of the hypotheses of perfect fit for all models, across all instruments. This result was not interpreted negatively, given the large sample size (Schermelleh-Engel et al., 2003; Vandenberg, 2006).

In Table 3, the MI results pertaining to each instrument are presented. When applying both CFI and RMSEA, four of the seven instruments showed configural MI (Brief Corporate Entrepreneurship Assessment Instrument [BCEAI], HRMP, leadership styles and IWB), whilst three instruments (IPA, UWES-9 and OCS) did not meet the CFI > 0.90 and RMSEA < 0.08 criteria. The last-mentioned three instruments were therefore deemed as non-MI and the first four instruments were deemed as MI, at a configural level.

Further analyses were therefore performed for BCEAI, HRMP, leadership styles and IWB. When considering higher levels of MI, acceptable CFI and RMSEA, as well as small ∆CFI and ∆RMSEA, were considered. All of BCEAI, HRMP, leadership styles and IWB met the CFI and RMSEA as well as small ∆CFI and ∆RMSEA criteria for metric, intercept and strict invariance, as well as ultimately the equivalence of the latent means.

For BCEAI, HRMP, leadership styles and IWB, the BIC statistic also reflects that the best fit is at the highest level, that is, equal latent means. The AIC statistic does not follow the same pattern, and only in the case of HRMP, the AIC and BIC statistics overlap. Only HRPM therefore revealed a consistent fit picture for all the selected indices. Brief Corporate Entrepreneurship Assessment Instrument, leadership styles and IWB met five of the six criteria for fit at equal latent means level.

Discussion

From the literature reviewed, it is evident that testing for MI when doing cross-group comparisons is important and that the concepts associated with MI and the measurement thereof are straightforward and performed without much effort. Although these analyses are sometimes performed, it is not a routine practice, even in journals specifically focussing on gender matters. Without denying the presence of MI in some studies focussing on gender, other fields of research have made great progress in this regard. Within cross-cultural research, for example, MI has become almost a standard reporting point at conferences such as the International Association for Cross-Cultural Psychology (Roland-Lévy, Denoux, Voyer, Boski, & Gabrenya, 2016). This article lobbies for conducting MI by those interested in making cross-gender comparisons and presents the reader with an introduction to the topic as well as with some guidelines on the interpretation of MI outputs.

The large number of organisations included in the study and the diversity of the respondents suggest a heterogeneous sample, which was relatively free from any particular context and therefore deemed appropriate to assess bias and equivalence (Els et al., 2016). More men (56.4%) than women (43.6%) respondents completed the questionnaires. This could be seen as a result of the random selection of respondents within organisations and is reflective of the workforce in South Africa (see Statistics South Africa, 2018), which comprises more men than women.

When testing for the normality of the distribution of the data, it was found that most scales showed statistically significant negative skewness, which was confirmed when inspecting the distribution curves. This was, however, not deemed problematic, as this phenomenon often occurs when surveys are used within organisations assessing positively perceived constructs (Spector & Brannick, 2009). All but one of the kurtosis values were within the normality range, and in that case, the statistic was similar for both men and women. Most telling was the fact that the distributions for males and females were similar across the different measurements, regarding both skewness and kurtosis. With regard to reliability, the coefficients ranged from low to high, with most being acceptable. As in the case of the distribution statistics, the reported coefficients for men and women were similar and mirrored each other.

When comparing observed means scores, significant differences were found for OCS and IWB, with men scoring higher than women on both measures. These differences had a small practical effect size. However, it could be asked whether these (small) differences are substantive or based on bias in the instruments?

The question of MI becomes pertinent in answering the above-mentioned question. The results of the MI tests are reported in Table 3. For three instruments, namely, UWES-9, IPA and OCS, the MI analysed showed that, at a configural level, men and women differed in their understanding of the construct. The same items did not measure corresponding constructs across groups (Bialosiewicz et al., 2013), or stated differently, the same items did not load on the same factors across groups. No further analyses are warranted (Gunn, 2016) should this level of non-invariance be detected. It should thus be concluded that the UWES-9, IPA and the OCS function in such a way that the domain or trait does not make sense in the same manner for men and women. This is a very serious ‘indictment’ of the integrity of these instruments, as the UWES-9 (Schaufeli et al., 2006) and the OCS (Allen & Meyer, 1990) are frequently used in research across the globe.

As stated before, in all cases, the χ²-test of perfect fit for all models was rejected. However, when considering the CFI, RMSEA, ∆CFI, ∆RMSEA and BIC, the criteria for equivalence of the latent means were met for the other four instruments, namely, BCEAI, HRMP, leadership styles and IWB. This signifies that these instruments met the MI criteria at the configural, metric, intercept and strict invariance levels. On these measures, men and women did not differ in the way they perceived the construct (configural MI), the way the items and the constructs relate (metric MI), the absolute weighting of the constructs (intercept MI), the errors associated with the measurements (strict MI) as well as their mean scores on the latent constructs (equivalence of the latent means). Regarding BCEAI, HRMP, leadership styles and IWB, users may therefore use these scales knowing that gender comparisons of factor variances and covariances are defensible (metric MI), mean differences comparison are secure (intercept MI), reliability is similar (strict MI) and scores for men and women on the latent constructs are similar (equivalence of the latent means).

Much ‘neater’ results would have been achieved should this reporting have excluded AIC. Akaike Information Criterion indicated – only for HRMP – that the optimal model was at the level of equivalence of the latent means. For the other qualifying instruments (BCEAI, leadership styles and IWB) scores varied, showing lower levels of fit. However, MI at equivalence of the latent means was accepted, given the overwhelming evidence to that effect. Clearly, the number and selection of parameters may also influence the reports. Cherry-picking of parameters is discouraged as this may result in falsification. Therefore, specifying parameters before analyses commence is recommended.

Interesting to note was the fact that in all the cases where configural invariance was achieved, the more constrained models were also satisfactory tested. Stated differently, in none of the cases where configural invariance was attained, it was not eventually followed by providing confirmatory information on MI at the equal latent means level. The additional constraints did not seem to influence the initially obtained statistics considerably. Future research with other instruments could explore this matter further.

Given the results of the MI tests, it could be stated that the observed mean differences (see Table 2) between men and women on OCS may be because of measurement bias, as men and women perceive the concept differently. As far as the observed differences in IWB are concerned, it can be concluded that men show more IWB than women and that they perceive the concept in the same manner. At a latent variable level, however, the means of men and women on IWB are equivalent (see Table 3). Men and women also perceive BCEAI, HRMP and leadership styles in a similar manner, and their standing on these constructs does not differ – not at an observed or a latent mean level. These instruments may be used with confidence when comparing men and women.

Conclusion

In this article, the call for testing of gender-based MI in studies where men and women are compared is answered through discussing the topic of MI and the rationale of testing in an unassuming matter, which allows for easy access to the topic. In this article, the interpretation of the different statistics created with the lavaan package (Rosseel, 2012) in R (R Core Team, 2013) is also operationalised, allying it to seven different instruments. This should make the problems associated with the interpretation of MI results clear to prospective researchers who want to test for MI. Researchers interested in the structure of group differences therefore have no reason not to investigate MI as a routine part of research, as called for by Borsboom (2006). The most significant contribution of this article is the complexity created when testing MI across several scales (using the same set of respondents). The different configurations of the results regarding MI add depth to the discussion on MI and the decision rules that are prescribed. The serratedness of results emphasised the necessity of testing for MI when comparing groups and the structures that underline such measurements. It is recommended that MI be used as a standard procedure, as these tests can be performed and interpreted with ease. Further research on how or why configural invariance ‘inevitably’ leads to equal latent means is encouraged. The findings also allow prospective users of instruments tested in this research to apply those instruments which showed MI confidently, knowing that these are equivalent across gender.

Acknowledgements

The authors are appreciative for the assistance of the 2014 Masters of Business Leadership class for their assistance in collecting the data.

The authors declare that they have no financial or personal relationships that may have inappropriately influenced them in writing this article.

R.S. was the primary author and contributed to the conceptualisation of the research, data collection and writing up of the article. G.P.d.B. was the supervisor of R.S.; he guided him through the process and took a leading role in the statistical analysis.

This research was supported by the UNISA Formal Qualifications grant (Ref No. 215037914).

Data sharing is not applicable to this article, given the specifications of the ethical clearance.

The views and opinions expressed in this article are that of the authors’ and not an official position of the institution or the funder.

References