Work-related Personality Inventory Manual

5. Validity

The validity of a questionnaire provides an indication of the extent to which a test actually measures the construct that it is intended to measure. In other words: does a personality questionnaire really measure personality? In the case of the WPI: does the questionnaire actually measure the work-related personality of a person? To be more specific, the validity indicates whether each factor or scale measures what it purports to measure.

Three types of validity can be distinguished. The external validity indicates to what extent the results of a test can be generalized. This mainly depends on the quality of the norm groups, which we have discussed in detail in Chapter 3. The internal validity of a questionnaire indicates to what extent a causal relation can be assumed, and whether alternative hypotheses can be dismissed. In personality questionnaires, it is very hard to actually prove this. By describing the different aspects of validity of a questionnaire this can nevertheless be made plausible.

The last type of validity is test validity; this type of validity will predominantly be discussed in this chapter. Test validity entails criterium validity and construct validity. Criterium validity indicates to what extent the results from a questionnaire are congruent with a criterium, for example the results from a different questionnaire. When the criterium is measured at the same time as the questionnaire, we talk about concurrent validity, when the criterium is measured at a future time point we talk about predictive validity.

When thinking about construct validity, researchers thinks about what constructs contribute to what is measured. Concerning the WPI, we ask ourselves what contributes to someone’s work-related personality, and what certainly does not contribute. Constructs that do not contribute to work-related personality are therefore constructs that we do not want to measure. This is called discriminant validity.

The distinction between criterium- and construct validity is not always clear. Criterium validity focuses on what is predicted. In terms of IQ-test, an employer is not interested in whether a candidate can reproduce a certain figure, but in his or her alleged intelligence of which this is an indication. In terms of construct validity one actually is interested in the test itself, and a possible correlation with a different instrument is merely meant as support that the test indeed measures what it is supposed to measure.

5.1 Criterium validity

To support the criterium validity (split up into concurrent and predictive validity) three studies have taken place that will be discussed below. For all tests a significance level of 5% was used.

5.1.1. Concurrent validity: Study with the FFPI

As mentioned before, the concurrent validity can be indicated by concurrently administering the WPI and an instrument that is intended to measure the same construct(s), in this case the FFPI.

A large study was conducted at an ICT company in which approximately 2700 employees were asked to take tests and questionnaires. The goal of this study was to gather data for a norm group for capacity tests that the company was going to use. The WPI and the FFPI were also offered as a combined questionnaire. To exclude possible effects of the order in which the questionnaires were taken, this order was determined at random. To be sure that the employees filled out the questionnaire in a serious way, they could win prizes if they participated in the study. In total, 354 employees filled out the WPI and the FFPI.

In this study, the earlier version of the WPI, the ProSiD-PI 25, was used. In comparison with the WPI, some of the scales and factors have different names and there are four items that are not included in this version[1]. Partly on the basis of this study we have chosen to alter the names of some scales and factors. In the presentation of the results we use the division into scales and factors and the naming of the WPI.

The FFPI
The Five Factor Personality Inventory (Hendriks, Hofstee & de Raad, 1999) is a personality questionnaire which can determine someone’s scores on five broad dimensions. The dimensions of the FFPI are: Extraversion, Agreeableness, Conscientiousness, Emotional Stability and (intellectual) Autonomy. The FFPI consists of 100 short concretely formulated behavioural items.

The FFPI stems from the lexical approach to personality, where the goal is to come to a parsimonious but as complete as possible model for describing the most important differences in behaviour. The Abridged Big-Five dimensional Circumplex-model (AB5C-model) (Hofstee & de Raad, 1991; Hofstee, de Raad & Goldberg, 1992) was the starting point of the development of the FFPI. This AB5C-model is a combination of the previously discussed FFM and the interpersonal circumplex model, in which variables are arranged circularly according to their loadings on two orthogonal factors (Hendriks, Hofstee & de Raad, 1999).

The interpersonal circumplex model was introduced by Leary and his employees in the early 1950’s. To most psychologist, this model will be known as the Interpersonal Checklist (ICL) by LaForge and Suczek (LaForge & Suczek, 1955), which is often called ‘Leary’s rose’. The model is mostly represented as a circular arrangement of sixteen categories of interpersonal behaviour, positioned relative to two orthogonal axes. This circular arrangement is more than a convenient presentation. Behind this simplicity lies a substantive and psychometrically interesting regularity. The sixteen categories, often combined per two creating eight octants, are not arbitrarily chosen clusters. They are categories of behaviour that were found, first by intuition and later in numerous empirical studies by several different researchers, to be important aspects of interpersonal behaviour. The nature, the number of categories and the position relative to the main axes were confirmed. All this led to the conclusion that the circumplex structure was not just a coincidental configuration. From the start on the circumplex model has served as a theoretical framework for a great number of studies, including the FFPI.

Expectations
Given that the FFPI is a well-constructed and validated personality questionnaire and that one of the underlying theories of the FFPI is the FFM, we believe the FFPI to be a good instrument to study the criterium validity of the WPI. We investigated whether the factors of the WPI correlate highly with the corresponding factors from the FFPI. On the scale level, we expect a high correlation with those factors of the FFPI of which the description are most similar to the meaning of the scales of the WPI.

Description of the psychometric qualities and results
The sample consisted of 353 people, 55 females (15.6%) and 298 males (84.4%). The average age was 34.5 years old, ranging from 21 to 61 (the age of 8 respondents was unknown).

Both the correlations between the factors of the FFPI and the factors of the WPI as well as the correlations between the factors of the FFPI and the scales of the WPI were calculated. When comparing a questionnaire with a criterium, a correlation of 0.2 is considered to be low; a correlation of 0.3 is considered to be moderate/average; and a correlation of 0.5 is considered to be high (Cohen, 1992). An attenuation correction is shown for factor correlations above .40. Attenuation is the phenomenon that the correlation between two variables decreases when the reliability of the variables is lower. This means that an estimate is given of the correlation in the hypothetical case of no attenuation. These correlations were based on the standardized scores. It is not possible to investigate the correlations between the scales of the FFPI and the scales of the WPI because the FFPI does not work with scales in the way the WPI does. However, we do know which concepts fall under the factors of the FFPI, see Appendix 9. In this way, we can assert whether the scales of the WPI correlate with the factors of the FFPI in an interpretable way.

The reliability of the factors of FFPI in this sample range from 0.80 (Cronbach’s alpha, Autonomy) and .87 (Extraversion). The reliability of the scales of the WPI in this sample ranged from .82 to .94. The reliability (stratified alpha) of the factors of the WPI ranged between .95 and .97. In calculating the reliability of the factors we have assumed that the total error variance is the same as in the Advice group which was used for standardization.  

Table 5.1 shows the means of the standardized factor scores and their corresponding standard deviations. The distributions are quite similar to the distributions in the norm group.

Table 5.1. Mean stenscore and standard deviation for each factor
FactorMeanStandard deviation
Influence5.82.1
Sociability4.72.1
Exuberance5.41.8
Structure4.82.4
Stability6.22.0

From Table 5.1 it becomes clear that each factor of the WPI corresponds highly with one factor of the FFPI, except for the factor Influence; this factor correlates highly with three factors of the WPI.

Table 5.2. Correlations between the factors of the FFPI and the WPI
WPI (N = 353)FFPI (N = 353 )
ExtraversionAgreeablenessConscientiousnessStabilityAutonomy
Stability.27**.11*.13*.69**(.85).19**
Structure-.24**.35**.76**(1.00).10-.17**
Influence.57**(.69)-.46**(-.58).01.07.40**(.51)
Sociability.77**(.91).09-.04.07.19**
Exuberance.26**.09.15**.20**.56**(.72)
Correlation larger than .30 are in bold, for these correlations the attenuation correction is in brackets.
** Significant at the .01 level (2- sided)
* Significant at the .05 level (2- sided)

With correlations of these magnitudes we can conclude that the factors are more or less tapping the same constructs and thus have the same meaning. To show this, we have undertaken attenuation corrections. When calculating the correlation between two instruments one has to take the unreliability of the two separate instruments into account. Attenuation correction is a way to correct for this.

– The factor Stability had a correlation of .69 with Emotional Stability of the FFPI, after attenuation correction this correlation was .85. These factors both include related scales, they are both intended to measure the emotional stability of a person.
– The factor Structure had a correlation of .76 with Conscientiousness of the FFPI, after attenuation correction this correlation was 0.997. The factor Structure included the scale Orderliness and related scales, while under the FFPI factor Conscientiousness concepts fall that are related to structure.
– The factor Influence had a correlation of .40 with Autonomy of the FFPI, after attenuation correction this correlation was .51. Both factors include similar scales that focus on the ‘self’, for example Self representation and Dominance. Influence also correlates with Extraversion (.57, after attenuation correction .69). Extraversion is intended to measure the degree of talkativeness, a concept that can also be found in the scales of Self representation and Dominance. Finally, Influence had a negative correlation of -.46 (after attenuation correction -.58) with Agreeableness of the FFPI. This can be explained by the fact that Agreeableness is directly opposed to scales such as Competition, Dominance and Self presentation.
– The factor Sociability had a correlation of .77 with Extraversion of the FFPI, after attenuation correction .91. The name of the WPI factor Extraversion has been changed into Sociability. Because the definition of this factor has remained the same, it is not surprising that this factor correlates highly with Extraversion of the FFPI.
– The factor Exuberance had a correlation of .56, after attenuation correction .72, with the FFPI factor Autonomy. Both factors include aspects of independency. This factor also has a fairly high with three other factors of the FFPI. On the scale level, we will show that this factor correlates in an interpretable way with the FFPI.

It is striking that when we look at all the correlations of the factors, four factors of the WPI correlate with one of the factors of the FFPI. Only for the factor Influence of the WPI, there are more correlations, however, these are – as described above – explainable. We have to note that the reverse is not always the case. The factors of the FFPI that correlate highly with the WPI, correlate highly with other factors of the WPI as well. This will lead to correlations between the factors of the FFPI and the scales of the WPI, mostly in an explainable way.

An overview of all the correlations between the factors of the FFPI and the scales of the WPI is presented in Appendix 10. The most important and most notable results are presented in Table 5.3, for every factor of the FFPI. All correlations larger than .50 with a significance level smaller than 0.01 are presented in the first column with the corresponding correlations, where the highest correlations are presented first. In the second and third column, all other significant correlations are shown, in the order of the magnitude of the correlation. We have included descriptions of the FFPI factors in Appendix 9.

Table 5.3. Correlations between scales of the WPI and factors of the FFPI (N = 353)
FFPI factorSignificant correlations with  scales of the WPI >.50.Significant* positive correlations with scales of the WPI up to .50.Significant* negative correlations of the scales of the WPI up to .50.
ExtraversionLeisure contact (.75)
Need for contact (.72) Friendliness (.67)
Self-presentation (.68)
Dominance (.54)
Self-disclosure (.53)
Energy (.45)
Attentiveness (.39)
Status (.37)
Self-confidence (.34)
Adaptability (.33)
Optimism (.31)
Trust (.30)
Resilience (.20)
Competition (.14)
Originality (.14)
Regularity (-.30)
Decisiveness (-.27)
Precision (-.17)
AgreeablenessSelf-presentation (-.52)Attentiveness (.41)
Conformity (.38)
Frustration-tolerance (.32) Decisiveness (.30)
Precision (.29)
Perseverance (.23)
Personal growth (.16)
Optimism (.15)
Trust (.14)
Dominance (-.31)
Competition (-.30)
Status (-.26)
Leisure contact (-.19)
Self-confidence (-.17)
ConscientiousnessOrderliness (.66) Perseverance (.60) Precision (.55)  Decisiveness (.50)Regularity (.49)
Conformity (.41)
Energy (.24)
Optimism (.14)
Originality (-.19)
Adaptability (-.14)
StabilityFrustration-tolerance (.64) Resilience (.52)
Optimism (.51)
Self-confidence (.47) Adaptability (.22)
Conformity (.21)
Friendliness (.19)
Decisiveness (.17)
Trust (.17)
Energy (.17)
Leisure contact (.17)
Perseverance (.15)
Attentiveness (-.16)
AutonomyDominance (.63)Originality (.50)
Adaptability (.42)
Self-confidence (.40)
Energy (.38)
Independence (.35)
Personal growth (.33)
Leisure contact (.30) Perseverance (.27)
Attentiveness (.26)
Self-presentation (.22)
Status (.19)
Competition (.18)
Optimism (.16)
Resilience (.16)
Regularity (-.43)
Conformity (-.36)
*Significant at the .01 level

The scales of the WPI all correlate in an explainable way with the factors of the FFPI. For all the correlations it holds that roughly the same constructs are described by the scales and factors. The positive correlation between Conformity with the factor Conscientiousness of the FFPI can be explained by the fact that this factor is defined as “orderliness”. An interesting finding is that Attentiveness has a negative correlation with the Stability factor of the FFPI. This could mean that less stable people are more inclined to take care of others. Almost all scales show a significant correlation with the factors of the FFPI. The scales Dominance, Self-presentation, Need for contact, Leisure contact, Self-disclosure, Friendliness, Perseverance, Orderliness, Precision, Decisiveness, Optimism, Frustration tolerance and Resilience even have correlations higher than .50 with factors of the FFPI. Hereby not only the validity of the factors but of the scales as well, is shown.

Conclusions
The Five Factor Model is the underlying theory of the FFPI as well as the WPI. From the described results on the relation between the factors of the FFPI with the factors and scales of the WPI we have gained insights in the psychological constructs that are measured by the questionnaires. The results closely match our expectations. Each factor of the WPI correlates highly with a theoretically similar factor of the FFPI, except for Influence, that correlated highly with three of the FFPI factors. The scales of the WPI correlate highly and in an interpretable way with the factors of the FFPI as well.
Given that the FFPI is a well-constructed and validated personality questionnaire, this study has been a good contribution to the validity of both the FFPI and the factors and the majority of the scales of the WPI.

5.1.2. Predictive validity

To assert the predictive validity of the WPI we have conducted two studies in which certain employment criteria were measured some time after completion of the WPI. Subsequently, we have tested the predictive value of the WPI. Again, a significance level of 5% was used.

5.1.2.1.  Predictive validity: nurse study with 360 degrees feedback reviews

Ixly has conducted a study in the period between September 2005 and December 2008 on student-nurses following the BBL version of nursing school. The goal of the research was: to assess the predictive value of several personal characteristics of student-nurses for the degree of success in their education. We chose to only include the students that were enrolled in the education program with an apprenticeship agreement. This was done because the curriculum of these forms of education resembles the practical job situations more than the curriculum of the more theoretical forms of education. Moreover, in practical situations – more so than in a more theoretical context – a wider range of personal characteristics is required in order to function effectively.
The personality characteristics were measured in the beginning of the educational program by means of the WPI. The degree of success in the education was measured by means of 360 degrees reviews. The content of the 360 degrees feedback questionnaire was created by an experienced psychologist and adapted to the practical situation in consultation with the participating hospitals. The students took the 360 degrees questionnaire themselves and at least two up to ten persons that had worked with the student were invited to complete the questionnaire about the student. The people that reviewed the student were classified into five categories: managers, work supervisors, trainers/teachers, certified nurses and third or fourth year student nurses.
The student was rated on fourteen competencies, each of them measured by five behavioural indicators. During the study, it became clear that administrating the questionnaire took quite a lot of time, for the trainers as well as the students. Because of this, not all the participating students completed a 360 degrees questionnaire during the education program. In total, 80 students completed 360 degrees questionnaires that could be analysed.

Sample
The total research sample included 204 student-nurses, enrolled in the nurse education program, either on secondary level (MBO) or a higher level (HBO) (35 were male, 160 female, for 8 students no information on gender was provided). When we started the analyses, 80 students had completed the 360 degrees questionnaire (12 male and 67 female, for 1 student information on gender was not provided). The average age of this sample was 24.6 years old (ranging from 18 to 44 years). Of these students, 58 were enrolled in the program on secondary level and 14 on the higher level (for 8 of the students, the level was unknown).

Expectations
Since the study was exploratory in nature, no specific hypotheses were formulated. According to Bloemers and van der Molen, the predictive value that can be found for instruments, lies mostly between the 0.20 and 0.50 (Bloemers and van der Molen, 2004). Research conducted by Schmidt and Hunter shows that for personality instruments, relationships in the order of 0.31 are found (Schmidt and Hunter, 1998). The general expectation is therefore that a number of personality traits will show a correlation of about 0.30 with almost all competencies.

Results
The reliabilities (Cronbach’s alpha) of the scales of the WPI in this sample were between 0.79 (Independence) and 0.94 (Attentiveness). The reliability of the factors were between 0.91 (Influence) and 0.96 (Exuberance).

First, the factor scores of this sample were compared with the factor scores of the reference population. With a t-test, we checked whether there were significant differences in the factor scores between the sample and reference population. Analyses showed that the factor scores of Influence and Exuberance differed significantly. The student-nurses score significantly lower on these factors than the reference population (see Table 5.4). In this light, we can conclude that student-nurses are less ambitious and less driven than the reference population.

Table 5.4. Mean stenscore in this sample in comparison with the sten scores of the norm group (N = 80)
FactorMean stenscore*t-valuep-value
Influence5.4 (1.5)2.59.01
Sociability6.2 (2.0)-0.89.37
Exuberance5.3 (1.9)2.85.00
Structure5.7 (2.6)-0.11.92
Stability5.8 (1.8)1.62.21
* The corresponding standard deviations are in brackets

We chose to calculate, for every student, the average rating for every behavioural indicator of a competency. In this calculation, only the ratings of the respondents were used, not the self-rating of the student. The reliability of the competencies ranged from 0.89 (Relational qualities) to 0.95 (Adaptability). The magnitude of these reliabilities is partly due to the average rating that was used per behavioural indicator.
A factor analysis (PCA, varimax rotation) was conducted on the average ratings per behavioural indicator. The results showed that the first factor explained 65% of the variance. Therefore, we have initially created a total score, based on all competencies. This total score was used to calculate the correlations with the standardized scale and factor scores of the WPI. The scales Self-disclosure (r = 0.38, p < 0.01) and Optimism (r = 0.37, p < 0.01) were significantly correlated with the total score of the 360 degrees feedback questionnaire. In addition, the factor Sociability showed a positive correlation with the 360 degrees feedback questionnaire (r = 0.28, p < 0.01).
In addition to this analysis, correlations were calculated between the ratings on the fourteen separate competencies and the factor scores of the WPI (Table 5.5). Six significant relations were found. Five of these relations included the factor Sociability of the WPI. The competencies Adaptability, Ability to influence, Emotional stability, Drive and Relational qualities all correlated significantly with the factor Sociability. Quality of work showed a relation with the scales of the factor Stability.

Table 5.5. Correlations competencies 360test and standardized factor scores WPI (N = 53)
Factors WPI
Competencies 360InfluenceSociabilityExuberanceStructureStability
Adaptability.04.30*.06.03.04
Analytical qualities.07.27.11-.003.19
Ability to influence.04.33*.14.05.22
Basic communicative skills.09.26.07.02.02
Creativity/innovation.04.26.06-.07.08
Discipline.03.17.10.03.14
Emotional stability.11.31*.14.06.26
Exuberance.06.28*.26.19.26
Knowledge-.02.17.08.02.11
Quality of work-.004.16.13.05.18
Quantity of work.02.24.21.24.32*
Organizational qualities.05.24.11.04.24
Relational qualities.02.30*.08.14.15
Nursing skills.08.20.07.02.12
* Significant at the .05 level (2- sided)

Conclusions
Based on the results of the analyses we can conclude that there is an association between the competencies as measured by 360 degrees method and the personality traits as measured by the WPI. The sizes of the associations are of the magnitude that may be expected on the basis of other studies (see Schmidt and Hunter, 1998). This result adds to the criterion validity of the WPI.

Five of the fourteen competencies necessary for the work of a nurse (i.e.: Adaptability, Ability to influence, Emotional stability, Exuberance and Relational qualities) all show a significant correlation with the factor Sociability. This result is explainable, given that the factor Sociability included the scales Need for contact, Leisure contact, Friendliness and Attentiveness. These are all personality traits that may be helpful as a nurse. From this we can conclude that when student-nurses score higher on the factor Sociability, that they can be expected to be successful in their job in the hospital.

In addition to the score on Sociability, it appears that the score on Stability is important in the selection of student nurses as well. The competency Quantity of work shows a significant correlation with the factor Stability. Noteworthy is the fact that the competency Emotional stability does not correlate significantly with the factor Stability. A possible explanation for this is that the lack of stability for student nurses is most notable in the quantity of work and less notable for the environment by means of the behavioural indicators that measure Emotional stability. Finally, Ability to influence, Emotional stability, Exuberance and Organizational qualities show relatively high correlations with the factor Stability.

Suggestions for future research
For the generalizability of the results, future research should formulate hypotheses in advance that can be tested by the analyses. The hypotheses may be derived from the results as found in this study. The hypotheses would then be:
Sociability is an important factor for the successful functioning of (student)nurses
Stability is an important factor for the quantity of work and is reported by the individual rather than that it is perceivable by the individual’s environment (co-workers).

5.1.2.2 Predictive validity: performance study within a competitive employment agency

In 2010, for a period of 6 months, a study has been performed at a competitive employment agency (hereafter called Agency X) on the relationship between personality, job satisfaction and job performance.

Method
The full employee base of Agency X (284 employees) was invited to take the WPI; an employee satisfaction survey and the Career values questionnaire via e-mail. Performance measures, i.e. individual turnover and supervisor rating, on all persons were available. First, the mean scale and factor scores for different job positions were investigated. Subsequently, using linear regression, we have tested several hypotheses in order to assess the predictive value of the WPI on three job criteria.

Sample
189 employees have gone through the entire research process. Of these, 105 were female, 84 were male. 78 employees had a contract for a definite period, 106 employees for an indefinite period and for 5 employees this was unknown. At Agency X, two general job positions could be distinguished: 54 recruiters and 73 account managers; the other 62 employees were either staff or support staff. Table 5.6 describes the sample.

Table 5.6. Description of the sample of Agency X
189 employees
Mean age 24,5 year
84 Male105 Female
54 Recruiters73 Account managers62 Other functions
78 Definite contract106 Indefinite contract

Expectations
Hypotheses on the relationships between personality, performance and job satisfaction were formulated prior to the analyses. Hypotheses were formulated on the basis of the expectations of the HR consultants at Agency X.

  1. The WPI scales Leisure contact and Need for contact have a positive relation performance.
  2. The WPI scales Decisiveness and Regularity have a negative relationship with performance.

Results
The mean factor and scale scores for both function groups are presented in Table 5.7. Remarkably high scores for both function groups are the scores on the scale Competition and the factor Influence. Remarkably low scores for both groups are the scores on the scale Decisiveness. Differences in mean scale scores are most apparent for the scales Independence, Self-confidence, Structure and Originality. 

Table 5.7. Mean stenscores of the factors and scales for both function groups of Bureau X (norm group used: Advice)
Recruiter (N = 54)Account manager (N = 73)
MSDMSD
Influence7.061.587.961.60
Sociability5.541.625.631.57
Exuberance5.261.555.911.45
Structure3.811.713.531.81
Stability5.601.626.001.56
MSDMSD
Adaptability5.081.435.801.35
Independence5.601.466.531.35
Perseverance5.151.505.051.42
Competition8.161.578.801.57
Energy5.611.445.881.45
Frustration-tolerance5.201.805.511.78
Resilience6.041.506.241.63
Attentiveness4.661.574.391.54
Dominance5.881.366.441.31
Status5.991.446.671.57
Precision4.371.574.261.21
Conformity5.301.424.881.71
Self-disclosure5.501.675.311.71
Orderliness4.241.704.331.83
Originality4.971.415.751.55
Decisiveness4.071.704.001.60
Optimism5.561.605.381.33
Personal growth5.451.725.571.91
Regularity4.801.854.471.46
Need for contact6.271.766.341.33
Leisure contact6.191.496.761.65
Self-presentation6.661.847.431.50
Trust4.801.924.851.92
Friendliness5.811.315.881.38
Self-confidence5.591.346.421.47

Linear regression
Based on the average scores on the scales and factors in Table 5.7, a profile of the average scores of the persons within a function group can be made. However, this does not tell us anything about the predictability of the performance and satisfaction on the basis of the WPI. For this, linear regression is used.

Based on the assumptions that underlie regression analyses, we have performed several explorative analyses on the dependent variables (y1) Turnover, (y2) Supervisor rating and (y3) Job satisfaction. The assumptions underlying regression analyses are:

  1. Linear relationship between X and Y
  2. All pairs of observations (X, Y) are independent of each other
  3. The residuals are normally distributed
  4. The variances of the residuals are equal, independent of X (Homoscedasticity)

In the following tables, B denotes the regression coefficient, or in other words the extent to which the scale contributes to the dependent variable.

Table 5.8. Linear regression (x) WPI scales and (y1) turnover per function for Agency X (Norm group Advice)*
Recruiter**
BSEBetatSig.
(Constant)24.4727.7863.054.004
Decisiveness3.6111.3190.3362.620.012
Orderliness-3.2781.304-0.382-2.983.005
Account manager
BSEBetatSig.
No variables included according to stepwise inclusion of variables.
* Stepwise inclusion of (x)
** R2 = .247
Table 5.9. Linear regression (x) WPI scales and (y2) supervisor rating per function for Agency X (Norm group Advice)*
Recruiter**
BSEBetatSig.
(Constant)0.4810.2082.317.031
Adaptability0.1510.0420.8623.561.002
Competition-0.0910.035-0.623-2.575.018
Account manager***
BSEBetatSig.
(Constant)0.1770.1161.53.135
Decisiveness0.0750.0280.4052.659.012
* Stepwise inclusion of (x)
** R2 = .388
*** R2 = .164
Table 5.10. Linear regression (x) WPI scales and (y3) satisfaction per function for Bureau X (Norm group Advice)*
Recruiter**
BSEBetatSig.
(Constant)58.3624.65012.552.000
Adaptability2.1040.7160.3882.937.005
Independence-1.6740.616-0.315-2.718.009
Resilience1.5100.6390.2922.364.022
Need for contact1.1490.5080.262.259.028
Account manager***
BSEBetatSig.
(Constant)49.6434.70510.552.000
Optimism4.4280.8630.5345.132.000
Conformity1.6790.6370.2612.636.010
Orderliness-1.3445.98-0.223-2.249.028
* Stepwise inclusion of (x)
** R2 = .440
*** R2 = .398

Conclusions
Results from the study at Agency X show that a clear personality profile can be distilled of the two main functions within the organization (Table 5.8). The high average competitiveness of the employees is a striking but expected trend within this organization. Following the same reasoning, the average low score on Decisiveness is a trend that shows that employees within this competitive organization are able (or should be able to) switch quickly and that this way of working fits with the current employee base.
In terms of the predictive value of the results of the WPI on the measures of performance a couple of scales stood out. Quite remarkable was that the score of a recruiter on the scale Competition had a negative effect on the supervisor’s rating of performance. From conversations with the HR consultants of Agency X it became clear that while a competitive attitude during the selection process seemed to be an advantage, this was not always seen as a positive attitude during working in teams.
Remarkable is the positive effect of Decisiveness as well, while the average employee of Agency X scored below average on this scale. A different important aspect of the functioning is the satisfaction of the employee him/herself. From our previous analyses, it became clear that for both functions different scales contributed to this satisfaction. For recruiters, this was mainly the case for Adaptability while for account managers this was mainly the case for Optimism.

The goal of this study was to assess the predictive value of the WPI by means of proposed hypotheses. It appears that several scales of the WPI are predictive of job criteria. This contributes greatly to the predictive validity of the WPI.

5.2 Construct validity

5.2.1. Construct validity: Factor and scale structure

The most important indication of the construct validity of the WPI are the results from the MGM (Multiple Group Method) which we have discussed earlier in the paragraph ‘Structure of the WPI’.
For both the Advice and Selection group an MGM analysis was performed. The background characteristics of these groups are described in the chapter Norms (paragraph 3.1). The analysis was performed on the data from the weighed norm groups.

When an MGM analysis is applied to the both weighed groups, the structure does not appear to differ significantly from the structure that was found in the construction phase of the questionnaire (on the unweighed groups). This holds for the scales as well as for the factors. The results of the MGM analyses on the division of items in scales are included in Appendix 1. The results of the MGM analyses on the division of scales into factors for both norm groups are presented in the following two tables. Notable results and differences between the groups will be discussed.

The presentation of the results of the MGM analyses (Table 5.1 and Table 5.2) is made easier by the use of colours. Three colours are used, each with their own meaning. Grey means that the item/scale correlates the highest with its own scale/factor (item-rest correlation) and that the difference in height with the other correlations is significant (as calculated with the t-test for differences between dependent correlations (Steiger (1980)). Yellow means that there are more correlations of the item/ scale that correlate with other scales/factors (respectively), but not in a significant way. The colour red means that the item/scale has a higher correlation with a different scale/factor than with its own scale/factor and that the difference in the height of the correlation is also significant. In the presented tables there are no red cells.

In the MGM on the division of scales into factors, there are 20 scales that correlate significantly the highest with their own scale (grey) for the Advice group, and 16 scales for the Selection group (see Tables 5.11 and 5.12). It appears that no other scale correlates higher with a different factor than that it was theoretically assigned to (red). In the Advice group, there are five scales that correlate in a meaningful way with more than one factor but without a significance difference, for the Selection group this was the case for nine scales. Three of these scales overlap between the Advice and Selection group.

Information on the results of the MGM analysis based on the division of items in scales is reported in Appendix 1. For both norm groups, there is no item that has a significantly higher correlation with a scale other than the scale it was assigned to. In the Advice group, 18 of the 276 items correlated with scales other than its own scale but without a significant difference in correlation heights (yellow). In the Selection group, 44 of the 276 items were in yellow cells. All other items correlated significantly higher with their own scale than with other scales.

When both groups are compared, it appears that 17 items correlate with more than one scale – but not significantly (yellow) – in both groups. One item appears to show a yellow cell in the Selection group, but not in the Advice group. There are 27 items in the Selection group that correlate with more than one scale – but not significantly (yellow) – while these items were in grey cells in the Advice group. The fact that there are fewer items in the Selection group that differ significantly from each other in terms of their correlations is die to smaller number of candidates in this group. Because the research population is smaller, significant results are less likely to occur.

Because the MGM results show relatively few differences, and because no item or scale correlates significantly with a different scale or factor, respectively, than their own scale or factor, these analyses can be viewed as solid support for the stability of the structure of the WPI. The division of items into scales and scales into factors is supported empirically.

MGM in an independent group
Since the Advice and Selection group were not only used to test the structure of the WPI, but were also used as input for the development of the questionnaire, an additional MGM analysis was run on an independent group. Background characteristics of this group can be found in the paragraph ‘Study with the FFPI’. The results of this group are compared with the advice norm group, because the WPI was administered in an advice setting. It appeared that the results MGM of the independent group only differed slightly from the results of the Advice group. The MGM results for the analyses on both the scales as on the factors are presented in Appendix 1. Notable results and differences between the independent group and the Advice group (the MGM from the previous paragraph) will be discussed.

Table 5.11. MGM results for the Advice group (N = 712)
FactorScaleExuberanceStabilityInfluenceSociabilityStructure
ExuberanceAdaptability.69.59.51.48-.03
Perseverance.54a.41.27.32.51a
Energy.71.53.47.49.27
Independence.37a.12.34a.08-.10
Originality.57.42.48.37-.03
Personal growth.63.35.48.34.16
StabilityFrustration-tolerance.29.59.10.33.24
Resilience.40.68.29.25.06
Optimism.51.63.33.52.12
Self-confidence.60a.64a.53.40.08
InfluenceCompetition.39.16.61.09-.11
Dominance.63a.51.63a.48-.10
Status.47.31.57.21.05
Self-presentation.38.26.67.47-.24
SociabilityNeed for contact.35.37.37.65.10
Friendliness.52.50.37.71.14
Leisure contact.52.56.52.62.01
Trust.14.21.10.48-.08
Self-disclosure.26.20.33.59-.06
Attentiveness.33.20.03.53.30
StructureConformity.03.19-.14.20.36
Precision.35.17-.01.09.55
Orderliness.33.24.06.19.50
Regularity-.31a-.32a-.30a-.17.33a
Decisiveness.20.23-.03-.02.45
a The numbers concern the yellow cells, the other cells are coloured grey.
Table 5.12. MGM results for the Selection group (N = 369)
FactorScaleExuberanceStabilityInfluenceSociabilityStructure
ExuberanceAdaptability.72.60.38.54.17
Perseverance.65.53.28.43.51
Energy.71.64.39.59.41
Independence.23a.03.20a.05-.07
Originality.58.49.42.45.15
Personal growth.59.39.50.43.30
StabilityFrustration-tolerance.39.62.06.40.38
Resilience.44.70.21.40.28
Optimism.59a.58a.31.62a.23
Self-confidence.57.64.43.45.32
InfluenceCompetition.30.08.64.12-.10
Dominance.58a.46.57a.53a.01
Status.40.23.58.22.12
Self-presentation.32.21.66.42-.21
SociabilityNeed for contact.44.45.39.66.17
Friendliness.59.60.34.68.34
Leisure contact.52.62a.48.60a.12
Trust.27.31.19.46.03
Self-disclosure.19.16.29.49-.06
Attentiveness.48a.38.05.55a.40
StructureConformity.29.35a-.01.34a.42a
Precision.36.32-.01.16.60
Orderliness.44a.43a.09.27.51a
Regularity-.19-.21a-.23a-.14.31a
Decisiveness.25.38a-.03.14.45a
a The numbers concern the yellow cells, the other cells are coloured grey.

From the results of the MGM analyses on this independent research group on the item level (Appendix 11) it became clear that no single item correlated higher with a different scale other than the scale it was assigned to (red). 241 of the 272 items correlated higher with their own scales than with any other scale. The remaining 31 items correlated with scales other than their own scale, but not in a significant way. After comparison with the Advice group it appeared that 9 of the yellow labelled items received a yellow label in the Advice group as well. We concluded that the chosen makeup of the questionnaire on the item level was upheld in the independent group.

From the results of the MGM analyses on this independent research group on the factor level it appeared that 19 of the 25 scales correlated higher with their own factor than with any other factor. Five scales correlated with other factors, but not in a significant way. There was one scale that correlated significantly higher with a different factor than with its own designated factor. However, this scale did correlate significantly higher with its own factor than with any other factor in the Advice group. In any independent group, especially when it concerns a specific organization as is the case here, there will be differences with the norm group. Since the difference only concerns one scale, it seems that the items and scales of the WPI show a very stable structure.

Correlations between factors
To check whether the factors have a shared component (or components), a correlation analysis was done on the factor level as well as a factor analysis in both research groups (Advice and Selection). The correlations are reported in Table 5.13.

Table 5.13. Correlations between factors
Correlations Advice group (N = 712) / Selection group (N = 369)*
FactorInfluenceSociabilityExuberanceStructureStability
Influence
Sociability.39 / .44
Exuberance.59 / .55.48 / .56
Structure-.12 / -.06.10 / .16.18 / .26
Stability.39 / .35.47 / .51.56 / .60.16 / .33
* Every correlation is presented for the Advice Group First, for the Selection Group behind the / sign

The results of the rotated component matrices of the performed factor analyses are as follows (Table 5.14 and 5.15).

Table 5.14. Rotated component matrix for  the Advice group (N = 712)Table 5.15. Rotated component matrix for  the Selection group (N = 369)
ComponentComponent
1212
Influence.8-.4Influence.8-.4
Sociability.7.1Sociability.8.2
Exuberance.9.1Exuberance.8.3
Structure.11.0Structure.1.9
Stability.8.2Stability.7.5

Conclusion
For both the advice and selection group, as well as the independent group, MGM analyses showed that the WPI has a solid internal structure. The majority of the items and scales correlate significantly higher with its “own” designated scales and factors, respectively, than with other scales or factors. A few items and scales show correlations with other scales or factors, but not in a significant way. Furthermore, all factors showed a shared component, with the exception of the factor Structure. Future research should shed more light on what this component is. However, it is clear that all factors have a meaning, independent from one another.

5.2.2. Construct validity: relationship with background variables

 

To assert whether the factor and scale scores have a relationship with the background variables, we have analysed for all the variables, whether the mean scores on each scale and each factor for the different categories of these variables differed significantly from each other. For both the Advice and Selection group this was done by means of ANOVA (Table 5.16 through 5.19). The data was weighed during these analyses. The weighed N of the Advice group is 712 (unweighed N = 5629), for the Selection group the N is 369 (unweighed N = 1514).

In addition, the eta (see page 18 for an overview of the statistical terms) was calculated. Eta can be used as a measure of association for variables with more than two categories. When the difference is significant in a two sided test on the 5%-level, this is indicated by a * next to the eta of the corresponding factor/scale.

Finally, results of a study are reported in which job sector differences in the mean sten scores of some factors and scales were identified (Table 5.10). The significance of the differences is determined by T tests.

Gender
The mean scale and factor sten scores for males and females are represented in Table 5.16.

Table 5.16. Mean stenscores by gender including eta-value
Advice group (N = 712)Selection group (N = 369)
Factoreta*MaleFemaleeta*MaleFemale
Influence.17*5.85.1.16*5.85.1
Sociability.14*5.25.9.11*5.35.8
Exuberance.045.65.4.035.55.6
Structure.09*5.35.7.035.45.6
Stability.08*5.75.3.105.75.3
Scale
Status.10*5.75.3.065.65.4
Dominance.13*5.75.2.13*5.75.2
Competition.23*5.94.9.22*5.95.0
Self-presentation.08*5.65.3.105.75.3
Need for contact.08*5.45.7.075.45.7
Leisure contact.015.55.5.015.55.5
Self-disclosure.08*5.45.7.025.55.6
Trust.10*5.35.7.025.55.5
Friendliness.12*5.35.8.15*5.25.9
Attentiveness.23*5.16.0.22*5.16.0
Energy.005.55.5.055.45.6
Personal growth.015.55.5.065.45.7
Perseverance.065.45.6.045.45.6
Adaptability.035.65.4.035.55.6
Originality.13*5.75.2.12*5.75.2
Independence.075.65.3.045.45.6
Orderliness.10*5.35.7.075.45.7
Precision.09*5.35.7.015.55.5
Regularity.08*5.45.7.085.45.7
Conformity.055.45.6.065.45.6
Decisiveness.025.55.5.105.75.3
Self-confidence.15*5.85.1.20*5.95.0
Optimism.07*5.45.7.055.45.6
Frustration-tolerance.025.55.5.075.65.3
Resilience.17*5.85.1.13*5.75.2
* Significant results of the ANOVA are indicated by a * for the eta values.

In the Advice group, there are 4 factors and 16 scales that differ significantly from each other on the gender variable. In the Selection group, this is the case for 2 factors and 7 scales.

For the background variable gender, the found differences were maximally 1.0 sten; this corresponds to a difference of 0.5 times the standard deviation. The mean difference over all significant scales and factors was 0.56 sten; this corresponds to 0.28 times the standard deviation. The differences reported for the gender variable are differences that are also reported in the literature. Examples are women’s higher scores on Sociability, Attentiveness and Leisure contact and men’s higher scores on Status, Trust and Influence (Beutel & Marini, 1995).

Table 5.17. Mean stenscore by age including eta-value
Advice group (N = 712)Selection group (N = 369)
Factoreta*15-24 years25-44 years45-65 yearseta*15-24 years25-44 years45-65 years
Influence.085.55.75.3.14*5.95.75.1
Sociability.015.55.55.5.065.85.55.4
Exuberance.15*4.65.65.6.085.75.65.3
Structure.085.05.55.6.086.05.55.4
Stability.055.25.55.6.035.75.55.5
Scale
Status.18*6.15.75.0.21*6.25.74.9
Dominance.12*4.95.55.7.075.35.45.7
Competition.11*5.45.75.2.17*5.85.85.0
Self-presentation.075.45.65.3.135.75.75.1
Need for contact.16*6.45.55.2.15*6.05.75.1
Leisure contact.085.05.55.6.025.45.55.5
Self-disclosure.075.15.65.5.065.55.45.7
Trust.055.55.45.6.125.45.35.8
Friendliness.035.45.65.4.19*6.05.85.0
Attentiveness.055.85.45.5.075.85.45.6
Energy.10*5.05.65.5.126.05.65.2
Personal growth.085.45.75.3.20*6.25.84.9
Perseverance.16*4.65.55.7.055.65.65.4
Adaptability.075.15.55.6.055.75.65.4
Originality.09*5.05.55.7.045.55.45.6
Independence.21*4.35.65.7.104.95.55.6
Orderliness.15*4.65.65.7.035.75.55.5
Precision.12*4.85.55.6.075.85.55.4
Regularity.035.45.55.6.055.75.45.6
Conformity.15*6.35.55.2.24*6.45.84.9
Decisiveness.13*4.85.55.7.125.45.35.8
Self-confidence.10*5.05.55.7.025.65.55.5
Optimism.075.15.55.6.015.65.55.5
Frustration-tolerance.035.45.55.6.045.75.55.5
Resilience.045.65.65.4.025.65.55.5
* Significant results of the ANOVA are indicated by a * for the eta values.

Age
The mean scale and sten scores for the different age categories are represented in Table 5.17. In the standardization process, the age variable was divided into three categories (15-24, 25-44 and 45-65 years). For this variable, ANOVA analyses were run as well to check whether differences were significant. Significant differences are marked with a * next to the eta.

In the Advice group, there were 1 factor and 13 scales that differed significantly from each other in terms of age, for the Selection group this was the case for 1 factor and 6 scales. The largest difference in means is 1.5 sten; this corresponds to 0.75 times the standard deviation. The average difference over all the significant scales and factors is 0.98 sten. This corresponds to a difference of 0.49 times the standard deviation. From Table 5.17 it appears that the difference between the two oldest groups (25-44 years and 45-65 years) is generally small. The found significant differences are mainly caused by the mean sten score of the group up to 25 years.

Education
The mean scale and sten scores for the education variable are represented in Table 5.18.

Table 5.18. Mean stenscore by education including eta-values
Advice group (N = 712)Selection group (N = 369)
FactorEducationEducation
eta*LowerSecondHighereta*LowerSecondHigher
Influence.12*5.35.45.9.21*5.25.26.2
Sociability.075.55.45.7.125.35.35.9
Exuberance.10*5.35.45.8.095.45.35.8
Structure.20*6.15.64.9.19*5.85.84.8
Stability.075.55.35.7.045.55.45.6
Scale
Status.035.55.45.6.105.45.35.8
Dominance.13*5.25.45.9.22*5.15.26.2
Competition.11*5.25.45.8.19*5.25.36.1
Self-presentation.12*5.35.35.9.16*5.45.26.0
Need for contact.035.65.45.5.085.45.45.8
Leisure contact.065.45.45.7.15*5.35.36.0
Self-disclosure.14*5.15.45.9.115.35.45.8
Trust.18*5.35.26.1.17*5.25.36.0
Friendliness.055.75.55.4.035.55.45.6
Attentiveness.10*5.85.45.3.045.45.55.6
Energy.025.45.55.5.045.55.45.6
Personal growth.13*5.25.45.9.085.45.35.8
Perseverance.055.75.55.4.055.75.45.5
Adaptability.10*5.35.45.8.075.65.35.6
Originality.065.55.45.7.115.25.45.8
Independence.17*5.15.46.0.095.35.45.8
Orderliness.11*5.85.65.2.115.75.65.2
Precision.15*5.85.65.1.20*5.75.84.9
Regularity.22*6.15.64.9.20*5.85.84.9
Conformity.22*6.15.64.9.14*5.75.75.1
Decisiveness.045.55.45.6.015.55.55.5
Self-confidence.035.55.45.6.045.65.55.5
Optimism.10*5.35.45.8.14*5.25.46.0
Frustration-tolerance.095.65.35.7.015.55.55.5
Resilience.035.65.55.5.115.95.35.4
* Significant results of the ANOVA are indicated by a * for the eta values.

In the Advice group, there were 3 factors and 14 scales that differed significantly from each other in terms of education, for the Selection group this was the case for 2 factors and 9 scales. The largest difference in means is 1.2 sten, which corresponds to a difference of 0.6 times the standard deviation. The average distance over all significant scales and factors is 0.77 sten: this corresponds to a difference of 0.39 times the standard deviation.

‘Stereotypical’ differences in scores are found for the different educational levels. Examples are the fact that people with lower educational levels have a higher need for Structure and that people with higher educational levels find Personal growth more important.

Table 5.19. Mean sten score by job status including eta-values
Advice group

(N=712)

Selection group

(N=369)

Factoreta*non-UWVUWVeta*non-UWVUWV
Influence.065.54.7.095.54.4
Sociability.025.55.3.045.55.0
Exuberance.035.55.2.085.54.6
Structure.055.56.1.075.56.4
Stability.025.55.2.085.54.6
Scale
Status.055.54.9.045.55.1
Dominance.065.54.9.12*5.54.1
Competition.055.54.9.075.54.8
Self-presentation.045.55.0.075.54.7
Need for contact.025.55.3.065.54.8
Leisure contact.025.55.2.075.54.7
Self-disclosure.025.55.2.025.55.3
Trust.035.55.2.055.55.0
Friendliness.015.55.6.035.55.2
Attentiveness.035.55.8.035.55.9
Energy.025.55.2.065.54.9
Personal growth.045.55.0.065.54.8
Perseverance.015.55.7.025.55.3
Adaptability.025.55.2.085.54.6
Originality.025.55.3.085.54.7
Independence.025.55.3.045.55.0
Orderliness.035.55.8.005.55.5
Precision.025.55.8.055.56.0
Regularity.065.56.2.135.46.9
Conformity.055.56.1.085.56.4
Decisiveness.005.55.5.015.55.4
Self-confidence.035.55.2.075.54.8
Optimism.045.55.0.12*5.54.1
Frustration-tolerance.015.55.6.025.55.3
Resilience.015.55.4.035.55.2
* Significant results of the ANOVA are indicated by a * for the eta values.

Two significant differences were found for the job status variable (Table 5.19). People not registered at the UWV scored significantly higher on Dominance and Optimism than people registered at the UWV in the Selection group.

Job sector
Using t-tests, we have checked which scales and factors show significant differences in means between the different job sectors. The distribution and sizes of the different sectors in the sample are represented in Table 3.4. The significant differences in mean scores are represented in Table 5.20, 5.21, 5.22 and 5.23. This study was performed on a sample of 712 people in the Advice group and 369 people in the Selection group.

Table 5.20. Differences in mean sten scores on the factor level between sectors in the Advice group (N=719)
FactorSector ISten ISector JSten JDifference (I-J)p*SDp**Effect size***
StructureBusiness and administration6.12Commercial services4.991.13.001.640.69
StructureBusiness and administration6.12Public administration, Safety and Law5.071.06.001.590.66
* Due to the large number of significance tests performed in this study, a correction of the significance level is needed. Only significant differences after the Bonferonni correction are presented.
** Pooled standard deviation.
*** The difference between mean scores divided by the pooled standard deviation.

On the factor level, only the factor Structure shows significant differences between the different job sectors. The sector ‘Business and Administration’ differs significantly from ‘Commercial services’ and ‘Public administration, Safety and Law’ on this factor.

Table 5.21. Differences in mean sten scores on the factor level between sectors in the Selection group (N=1067)
FactorSector ISten ISector JSten JDifference(I-J)p*SDp**Effect size***
InfluenceCommercial services6.68Health, Wellbeing and Personal Care5.92.75.001.300.58
InfluenceCommercial services6.68Public administration, Safety and Law5.90.78.001.290.61
InfluenceEngineering and Production6.94Public administration, Safety and Law5.901.04.001.350.74
InfluenceEngineering and Production6.94Health, Wellbeing and Personal Care5.921.01.001.330.74
ExuberanceCommercial services6.44Public administration, Safety and Law5.78.66.00
* Due to the large number of significance tests performed in this study, a correction of the significance level is needed. Only significant differences after the Bonferonni correction are presented.
** Pooled standard deviation.
*** The difference between mean scores divided by the pooled standard deviation.

In the Selection group, significant differences were only found for the factors Influence and Exuberance (see Table 5.21).

Table 5.22. Significant differences in mean sten scores on the scale level between sectors in the Advice group
ScaleSector ISector JDifference (I-J)p
RegularityBusiness and administrationPublic administration, Safety and Law1.50.00
DominanceBusiness and administrationPublic administration, Safety and Law-1.53.00
PrecisenessBusiness and administrationPublic administration, Safety and Law1.54.00
CompetitionHealth, Wellbeing and Personal CareCommercial services-1.09.00
Self-confidenceHealth, Wellbeing and Personal CarePublic administration, Safety and Law-1.07.00
RegularityBusiness and administrationCommercial services1.20.00
DominanceBusiness and administrationCommercial services-1.22.00
PrecisenessBusiness and administrationCommercial services1.22.00


Conclusion

The most important question that needs to be answered is whether the WPI can be used sector independently for selection and advice purposes. In other words: how large is the effect of the sector on the scale and factor scores? In answering this question, it is not important how one sector relates to the other sectors, but more how a sector relates to the total norm group. The distribution of people over the different sectors for both norm groups is represented in Appendix 12, as well as the mean factor and scale scales and the corresponding effect difference with the total group. From this it becomes clear that people in some sectors score significantly higher on some scales and factors than the people in the total norm group, but that this does not outweigh the information that the groups contain.

Despite the large numbers of people in the norm groups, we found only a limited amount of significant effects for the background variables gender, age, job status and education. In terms of effect size (Cohen, 1992), the significant differences for the background variables are not of such an order that separate norm groups should be created. It can be interesting to include this information in the interpretation of the results since it concerns real differences, but this is not absolutely necessary. The study on differences between sectors has shown a number of results that can be considered as real, given the nature of the sectors. This contributes to the construct validity of the WPI, and indicates why separate norm groups are not desirable.

5.2.3. Construct validity: Cultural Bias

Since people with different cultural backgrounds live and work in the Netherlands, it is necessary that the WPI is a culturally fair questionnaire. Culturally fair means that no unfair biases occur in individual outcomes, and that only real differences between individuals are evident in relation to the entire work force. After all, these differences have a significant meaning for the Dutch labour market.

An extensive study was done to investigate the cultural bias of the WPI. The results of this study are described in Appendix 13.

The most important finding is that significant differences for only two scales were found; this was the case for Status and Trust. The native Dutch respondents have a lower score on Status and a higher score on Trust, compared with their non-native counterparts. This is consistent with the expectations one can derive from the literature on Dutch cultural characteristics. From this we can conclude that the WPI does not show cultural bias and that the scores and texts in the report represent real individual differences.

5.2.4 Construct validity: The WPI and the Career Values questionnaire by Ixly

The Career Values questionnaire (CV) is a personality questionnaire that provides insight into the aspects of work that can motivate a person. The questionnaire was developed for the HRM work field and can be used in both advice and selection contexts. In advice situations, the questionnaire provides insights into what a person finds motivating in a certain job. This way, it is easier for someone to look for a suitable job. In selection contexts, the CV provides insights into the match between the career values of the candidate and the characteristics of the job the candidate is applying for. When there is a match, this tells something about the motivation a candidate has for this specific function (Orga, 2007). The questionnaire that was used in this study (CV ipsative) consists of 190 items which are presented in a forced choice format: two items are presented opposed to each other, with a four point scale in between. The candidate has to indicate which of the two items applies more to him/her. The questionnaire consists of 20 scales. For a description of the scales, see Appendix 14.

This study also contributes to the criterium validity of the WPI. Although the CV is not what the WPI intends to measure, finding no or unsuspected relations between the two would question the criterium validity of the WPI.

The CV and the WPI are complementary during assessment procedures, because the WPI concerns the personality of the candidate while the CV concerns the personal preferences and values in terms of the candidate’s professional career. From experience we know that when someone scores high on a certain scale of the WPI, this does not necessarily have to mean that someone values this specific characteristic in a job. It is interesting to investigate how the results on both questionnaires correlate since one can expect that there are similarities in personal characteristics and the characteristics people find important in their job. The expectation is that when someone scores high or low on a scale of the WPI, this corresponds with certain value on the CV. The reason for this is that people often find aspects of work interesting or fun because they are good at it or because it they with their personality.

Data was available of 1329 people who took both the WPI and the CV questionnaires. All the information came from the Ixly’s dataset and was retrieved at different companies. The age of 761 people from the dataset was known, varying from 16 to 62 years old, with a mean age of 35,4. Of all the people of which the gender was known 539 were male and 669 female. The questionnaire was taken by 1139 people in an advice context, while 190 people took the questionnaire in a selection context. The questionnaires were completed during the beginning of 2004 and mid-2006.

An overview of the mean sten score on the factors of the WPV and their standard deviations are reported in Table 5.24. The mean sten scores and the standard deviations are very similar to the norm group.

Table 5.24. Mean sten scores and corresponding standard  deviations (N = 1329)
FactorMean sten scoreMean standard deviation
Influence52.2
Sociability5.22.2
Exuberance52.2
Structure5.32.5
Stability52.2

Correlations were calculated between the factors and scales of the WPI and the scales of the CV. The reliabilities of the scales of the CV in this sample were between .65 and .89 (Cronbach’s alpha). The reliabilities of the scales of the WPI were between .83 and .94 (α) and the reliabilities of the factors of the WPI were between .96 and .97 (stratified alpha). In calculating the reliability of the factors we have assumed that the total error variance was the same as in the norm group.

The correlations between the factors of the WPI and the CV are reported in Table 5.25. The correlations larger than .30 and smaller than -.30 are in bold. In studies like this, correlations that are larger than .30 (absolute) are considered as of average size and correlation larger than .50 of large size.

Table 5.25. Correlations between the factors of the WPI and the scales of the CV
CV-scales (N =1329)WPI-factors (N = 1329)
InfluenceSociabilityExuberanceStructureStability
Balance private life – work-.23**-.03-.23**.12**-.22**
Financial reward.27**-.03.05.09**-.05
Praise and recognition.14**.16**.13**.16**-.13**
Tangible results.21**.15**.37**.31**.11**
Useful contribution.03.19**.14**.12**.01
Career.68**.08**.40**.04.17**
Quality.24**.11**.39**.45**.15**
Creative thinking.37**.19**.56**-.07*.19**
Influencing.67**.26**.43**-.10**.25**
Enterprising.50**.25**.44**-.13**.24**
Helping others-.06*.39**.13**.23**.06*
Analysing.32**.18**.53**.14**.25**
Developing.30**.24**.53**.10**.23**
Being physically active-.06*-.00-.03.03-.05
Autonomy.28**-.03.27**-.28**-.03
Security and stability-.03.03-.05.44**-.05
Challenging tasks.36**.24**.60**-.02.33**
Cooperation.16**.50**.23**.24**.23**
Attention.63**.21**.23**-.13**.05
Hectic situations.26**.28**.46**.15**.23**
** Significant at p < .01 (2-sided)
* Significant at p < .05 (2-sided)

The factor Influence shows average (.30) to large (>.50) correlations with the following career values: Attention, Influencing, Creative thinking, Enterprising, Developing, Career, Analysing and Challenging tasks. People that score highly on this factor find these career values very important. All of these career values are, in a logical and explainable way, consistent with scoring high on the Influence factor of the WPI.

The factor Sociability shows average to large correlations with the following career values: Helping others and Cooperation. Social people often find it important to be able to help others in their job and social people tend to value the cooperative aspects of work as well.

The factor Exuberance shows average to large correlations with the career values: Influence, Creative thinking, Dynamics, Entrepreneurship, Development, Quality, Career, Analysing, Tangible results and Security and stability. All these career values can be linked, again, in an interpretable way with the factor Exuberance. When we look at scales this factor comprises, it is easy to see that scales such as Self-development, Adaptability and Originality fit well with these career values.

The factor Structure shows average to large correlations with the following career values: Quality, Tangible results and Security and stability. Structured people like to deliver quality and specific, tangible results.

The factor Stability only shows an average relation with the career value Task challenge. These people are stable and therefore need a challenge in their job. The fact that this factor does not correlate with the career value Security and stability can be explained by the reasoning that people who are stable do not need this in their job. The factor Stability of the WPI concerns the (emotional) stability of a person while the career value Security and stability mainly concerns the rewards and the materialistic (monetary) aspects of a job (career). Thus, to what extent are you stable, and to what extent do you need stability, respectively.

The table with all correlations between the WPI and the scale scores of the CV is included in Appendix 15. Below in Table 5.26, an overview is provided of the correlations between the scales of the WPI and the CV that are at least .30. The career values with the highest correlations with the corresponding scales of the WPI are reported first.

Table 5.26 Relationships between the scales of the WPI and CV.
Career Values
WPI scalePositive relationNegative relation
StatusCareer, Attention, Influencing, EnterprisingBalance private life – work, Helping others, Security and stability, Analysing, Being physically active
DominanceInfluencing, EnterprisingSecurity and stability, Balance private life – work
CompetitionCareerBalance private life – work, Helping others
Self-presentationAttention, Influencing, Career, EnterprisingSecurity and stability, Balance private life – work.
Leisure contactCooperation
AttentivenessHelping others
EnergyHectic situations, CareerBalance private life – work, Security and stability
Personal growthDeveloping, Career, Challenging tasksBeing physically active
PerseveranceQuality
AdaptabilityChallenging tasks, Developing, Creative thinkingSecurity and stability, Balance private life – work
OriginalityCreative thinkingSecurity and stability
IndependenceAutonomyCooperation, Helping others
PrecisionQualityEnterprising, Attention, Autonomy, Influencing
RegularitySecurity and stability, Balance private life – work, QualityEnterprising, Influencing, Challenging tasks, Creative thinking
ConformityAutonomy
DecisivenessQuality
Self-confidenceInfluencingSecurity and stability
OptimismSecurity and stability

The scales Self-disclosure, Trust, Friendliness, Orderliness, Optimism, Frustration-tolerance and Resilience are not shown in Table 5.26 because they did not show an average (>.30) or high (>.50) relation with one of the career values. The associations between the CV-scales and WPI-scales are not difficult to explain. Interestingly, Security and stability and Balance private life – work both show mostly negative correlations with the WPI-scales, while the WPI-scales mostly show positive correlations with Enterprising. The WPI is a work-related personality questionnaire and people who find enterprising important in their career, appear to have less of a need for security and stability and a well-balanced private and work life.

The research described here strongly contributes to the construct validity of the WPI. All the relationships found were either expected or easy to explain.

5.3. Concluding remarks on the validity of the WPI

In the research on the validity of the WPI, a distinction was made between predictive validity and construct validity.

Predictive validity
Research focusing on concurrent validity shows that the factors and the majority of the scales of the WPI show high and theoretically justifiable correlations with the factors of the FFPI. The average correlation between the respective factors is .650
Research on student nurses shows that particularly the Sociability and Stability factors (and their underlying scales) are predictive of the academic performance in nursing school. Research at an employment agency has shown that the WPI has predictive power in this context as well. Different criteria of job performance were predicted by scores on the WPI. These two studies show the predictive validity of the WPI in two very different populations.

Construct validity
Research on the internal structure showed that the Multiple Group Method indicated that the questionnaire consists of relatively independent, homogeneous, reliable and stable scales. At the factor level, the reliability is high to very high (>.95). The stability of the scales is .87 on average.
Research on different background characteristics and their relations with WPI scores shows that there only small differences in scores between different subpopulations. Furthermore, we believe that the small differences found reflect real and relevant differences which are a good reflection of the reality. If, for example, different norm groups would be formed for men and women separately, then a male with a low score for Attentiveness would not receive a very large sten score. While in reality, we are still dealing with an inattentive person. Because of this we have chosen to use one single norm group for the different subpopulations.
Finally, a study was conducted investigating the relations between the WPI and the Career Values questionnaire. This study showed that all factors of the WPI had an average to strong relation with one or more scales of the CV questionnaire. Similarly, at the scale level, a number of theoretically justifiable correlations were found between the WPI and the CV. These results also support the construct validity of the WPI.

[1] The problem of the four missing items was solved by repeating the standardization process for the scales and factors to which these items belonged, using the same norm population on which the official norms were based. The correlations between the scales and factors were .97 and .99, which means that the differences were negligible.