# 5. Validity

The validity of a questionnaire provides an indication of the extent to which a test actually measures the construct that it is intended to measure. In other words: does a personality questionnaire really measure personality? In the case of the WPI: does the questionnaire actually measure the work-related personality of a person? To be more specific, the validity indicates whether each factor or scale measures what it purports to measure.

Three types of validity can be distinguished. The external validity indicates to what extent the results of a test can be generalized. This mainly depends on the quality of the norm groups, which we have discussed in detail in Chapter 3. The internal validity of a questionnaire indicates to what extent a causal relation can be assumed, and whether alternative hypotheses can be dismissed. In personality questionnaires, it is very hard to actually prove this. By describing the different aspects of validity of a questionnaire this can nevertheless be made plausible.

The last type of validity is test validity; this type of validity will predominantly be discussed in this chapter. Test validity entails criterium validity and construct validity. Criterium validity indicates to what extent the results from a questionnaire are congruent with a criterium, for example the results from a different questionnaire. When the criterium is measured at the same time as the questionnaire, we talk about concurrent validity, when the criterium is measured at a future time point we talk about predictive validity.

When thinking about construct validity, researchers thinks about what constructs contribute to what is measured. Concerning the WPI, we ask ourselves what contributes to someone’s work-related personality, and what certainly does not contribute. Constructs that do not contribute to work-related personality are therefore constructs that we do not want to measure. This is called discriminant validity.

The distinction between criterium- and construct validity is not always clear. Criterium validity focuses on what is predicted. In terms of IQ-test, an employer is not interested in whether a candidate can reproduce a certain figure, but in his or her alleged intelligence of which this is an indication. In terms of construct validity one actually is interested in the test itself, and a possible correlation with a different instrument is merely meant as support that the test indeed measures what it is supposed to measure.

## 5.1 Criterium validity

To support the criterium validity (split up into concurrent and predictive validity) three studies have taken place that will be discussed below. For all tests a significance level of 5% was used.

#### 5.1.1. Concurrent validity: Study with the FFPI

As mentioned before, the concurrent validity can be indicated by concurrently administering the WPI and an instrument that is intended to measure the same construct(s), in this case the FFPI.

A large study was conducted at an ICT company in which approximately 2700 employees were asked to take tests and questionnaires. The goal of this study was to gather data for a norm group for capacity tests that the company was going to use. The WPI and the FFPI were also offered as a combined questionnaire. To exclude possible effects of the order in which the questionnaires were taken, this order was determined at random. To be sure that the employees filled out the questionnaire in a serious way, they could win prizes if they participated in the study. In total, 354 employees filled out the WPI and the FFPI.

In this study, the earlier version of the WPI, the ProSiD-PI 25, was used. In comparison with the WPI, some of the scales and factors have different names and there are four items that are not included in this version[1]. Partly on the basis of this study we have chosen to alter the names of some scales and factors. In the presentation of the results we use the division into scales and factors and the naming of the WPI.

**The FFPI **

The Five Factor Personality Inventory (Hendriks, Hofstee & de Raad, 1999) is a personality questionnaire which can determine someone’s scores on five broad dimensions. The dimensions of the FFPI are: Extraversion, Agreeableness**, **Conscientiousness**, **Emotional Stability and (intellectual) Autonomy. The FFPI consists of 100 short concretely formulated behavioural items.

The FFPI stems from the lexical approach to personality, where the goal is to come to a parsimonious but as complete as possible model for describing the most important differences in behaviour. The Abridged Big-Five dimensional Circumplex-model (AB5C-model) (Hofstee & de Raad, 1991; Hofstee, de Raad & Goldberg, 1992) was the starting point of the development of the FFPI. This AB5C-model is a combination of the previously discussed FFM and the interpersonal circumplex model, in which variables are arranged circularly according to their loadings on two orthogonal factors (Hendriks, Hofstee & de Raad, 1999).

The interpersonal circumplex model was introduced by Leary and his employees in the early 1950’s. To most psychologist, this model will be known as the Interpersonal Checklist (ICL) by LaForge and Suczek (LaForge & Suczek, 1955), which is often called ‘Leary’s rose’. The model is mostly represented as a circular arrangement of sixteen categories of interpersonal behaviour, positioned relative to two orthogonal axes. This circular arrangement is more than a convenient presentation. Behind this simplicity lies a substantive and psychometrically interesting regularity. The sixteen categories, often combined per two creating eight octants, are not arbitrarily chosen clusters. They are categories of behaviour that were found, first by intuition and later in numerous empirical studies by several different researchers, to be important aspects of interpersonal behaviour. The nature, the number of categories and the position relative to the main axes were confirmed. All this led to the conclusion that the circumplex structure was not just a coincidental configuration. From the start on the circumplex model has served as a theoretical framework for a great number of studies, including the FFPI.

**Expectations**

Given that the FFPI is a well-constructed and validated personality questionnaire and that one of the underlying theories of the FFPI is the FFM, we believe the FFPI to be a good instrument to study the criterium validity of the WPI. We investigated whether the factors of the WPI correlate highly with the corresponding factors from the FFPI. On the scale level, we expect a high correlation with those factors of the FFPI of which the description are most similar to the meaning of the scales of the WPI.

**Description of the psychometric qualities and results**

The sample consisted of 353 people, 55 females (15.6%) and 298 males (84.4%). The average age was 34.5 years old, ranging from 21 to 61 (the age of 8 respondents was unknown).

Both the correlations between the factors of the FFPI and the factors of the WPI as well as the correlations between the factors of the FFPI and the scales of the WPI were calculated. When comparing a questionnaire with a criterium, a correlation of 0.2 is considered to be low; a correlation of 0.3 is considered to be moderate/average; and a correlation of 0.5 is considered to be high (Cohen, 1992). An attenuation correction is shown for factor correlations above .40. Attenuation is the phenomenon that the correlation between two variables decreases when the reliability of the variables is lower. This means that an estimate is given of the correlation in the hypothetical case of no attenuation. These correlations were based on the standardized scores. It is not possible to investigate the correlations between the scales of the FFPI and the scales of the WPI because the FFPI does not work with scales in the way the WPI does. However, we do know which concepts fall under the factors of the FFPI, see Appendix 9. In this way, we can assert whether the scales of the WPI correlate with the factors of the FFPI in an interpretable way.

The reliability of the factors of FFPI in this sample range from 0.80 (Cronbach’s alpha, *Autonomy*) and .87 (*Extraversion*). The reliability of the scales of the WPI in this sample ranged from .82 to .94. The reliability (stratified alpha) of the factors of the WPI ranged between .95 and .97. In calculating the reliability of the factors we have assumed that the total error variance is the same as in the Advice group which was used for standardization. * *

Table 5.1 shows the means of the standardized factor scores and their corresponding standard deviations. The distributions are quite similar to the distributions in the norm group.

Table 5.1. Mean stenscore and standard deviation for each factor | ||

Factor | Mean | Standard deviation |

Influence | 5.8 | 2.1 |

Sociability | 4.7 | 2.1 |

Exuberance | 5.4 | 1.8 |

Structure | 4.8 | 2.4 |

Stability | 6.2 | 2.0 |

From Table 5.1 it becomes clear that each factor of the WPI corresponds highly with one factor of the FFPI, except for the factor *Influence*; this factor correlates highly with three factors of the WPI.

Table 5.2. Correlations between the factors of the FFPI and the WPI | |||||

WPI (N = 353) | FFPI (N = 353 ) | ||||

Extraversion | Agreeableness | Conscientiousness | Stability | Autonomy | |

Stability | .27** | .11* | .13* | .69**(.85) | .19** |

Structure | -.24** | .35** | .76**(1.00) | .10 | -.17** |

Influence | .57**(.69) | -.46**(-.58) | .01 | .07 | .40**(.51) |

Sociability | .77**(.91) | .09 | -.04 | .07 | .19** |

Exuberance | .26** | .09 | .15** | .20** | .56**(.72) |

Correlation larger than .30 are in bold, for these correlations the attenuation correction is in brackets. ** Significant at the .01 level (2- sided) * Significant at the .05 level (2- sided) |

With correlations of these magnitudes we can conclude that the factors are more or less tapping the same constructs and thus have the same meaning. To show this, we have undertaken attenuation corrections. When calculating the correlation between two instruments one has to take the unreliability of the two separate instruments into account. Attenuation correction is a way to correct for this.

– The factor *Stability *had a correlation of .69 with *Emotional* *Stability *of the FFPI, after attenuation correction this correlation was .85. These factors both include related scales, they are both intended to measure the emotional stability of a person.

– The factor *Structure* had a correlation of .76 with *Conscientiousness *of the FFPI, after attenuation correction this correlation was 0.997. The factor *Structure *included the scale *Orderliness *and related scales, while under the FFPI factor *Conscientiousness *concepts fall that are related to structure.

– The factor *Influence *had a correlation of .40 with *Autonomy *of the FFPI, after attenuation correction this correlation was .51. Both factors include similar scales that focus on the ‘self’, for example *Self representation *and *Dominance*. *Influence *also correlates with *Extraversion *(.57, after attenuation correction .69). *Extraversion *is intended to measure the degree of talkativeness, a concept that can also be found in the scales of *Self representation *and *Dominance. *Finally, *Influence *had a negative correlation of -.46 (after attenuation correction -.58) with *Agreeableness *of the FFPI. This can be explained by the fact that *Agreeableness *is directly opposed to scales such as *Competition*, *Dominance *and *Self presentation*.

– The factor *Sociability *had a correlation of .77 with *Extraversion *of the FFPI, after attenuation correction .91. The name of the WPI factor *Extraversion *has been changed into *Sociability*. Because the definition of this factor has remained the same, it is not surprising that this factor correlates highly with *Extraversion* of the FFPI.

– The factor *Exuberance *had a correlation of .56, after attenuation correction .72, with the FFPI factor *Autonomy*. Both factors include aspects of independency. This factor also has a fairly high with three other factors of the FFPI. On the scale level, we will show that this factor correlates in an interpretable way with the FFPI.

It is striking that when we look at all the correlations of the factors, four factors of the WPI correlate with one of the factors of the FFPI. Only for the factor *Influence *of the WPI, there are more correlations, however, these are – as described above – explainable. We have to note that the reverse is not always the case. The factors of the FFPI that correlate highly with the WPI, correlate highly with other factors of the WPI as well. This will lead to correlations between the factors of the FFPI and the scales of the WPI, mostly in an explainable way.

An overview of all the correlations between the factors of the FFPI and the scales of the WPI is presented in Appendix 10. The most important and most notable results are presented in Table 5.3, for every factor of the FFPI. All correlations larger than .50 with a significance level smaller than 0.01 are presented in the first column with the corresponding correlations, where the highest correlations are presented first. In the second and third column, all other significant correlations are shown, in the order of the magnitude of the correlation. We have included descriptions of the FFPI factors in Appendix 9.

Table 5.3. Correlations between scales of the WPI and factors of the FFPI (N = 353) | |||

FFPI factor | Significant correlations with scales of the WPI >.50. | Significant* positive correlations with scales of the WPI up to .50. | Significant* negative correlations of the scales of the WPI up to .50. |

Extraversion | Leisure contact (.75) Need for contact (.72) Friendliness (.67) Self-presentation (.68) Dominance (.54) Self-disclosure (.53) | Energy (.45) Attentiveness (.39) Status (.37) Self-confidence (.34) Adaptability (.33) Optimism (.31) Trust (.30) Resilience (.20) Competition (.14) Originality (.14) | Regularity (-.30) Decisiveness (-.27) Precision (-.17) |

Agreeableness | Self-presentation (-.52) | Attentiveness (.41) Conformity (.38) Frustration-tolerance (.32) Decisiveness (.30) Precision (.29) Perseverance (.23) Personal growth (.16) Optimism (.15) Trust (.14) | Dominance (-.31) Competition (-.30) Status (-.26) Leisure contact (-.19) Self-confidence (-.17) |

Conscientiousness | Orderliness (.66) Perseverance (.60) Precision (.55) Decisiveness (.50) | Regularity (.49) Conformity (.41) Energy (.24) Optimism (.14) | Originality (-.19) Adaptability (-.14) |

Stability | Frustration-tolerance (.64) Resilience (.52) Optimism (.51) | Self-confidence (.47) Adaptability (.22) Conformity (.21) Friendliness (.19) Decisiveness (.17) Trust (.17) Energy (.17) Leisure contact (.17) Perseverance (.15) | Attentiveness (-.16) |

Autonomy | Dominance (.63) | Originality (.50) Adaptability (.42) Self-confidence (.40) Energy (.38) Independence (.35) Personal growth (.33) Leisure contact (.30) Perseverance (.27) Attentiveness (.26) Self-presentation (.22) Status (.19) Competition (.18) Optimism (.16) Resilience (.16) | Regularity (-.43) Conformity (-.36) |

*Significant at the .01 level |

The scales of the WPI all correlate in an explainable way with the factors of the FFPI. For all the correlations it holds that roughly the same constructs are described by the scales and factors. The positive correlation between *Conformity *with the factor *Conscientiousness *of the FFPI can be explained by the fact that this factor is defined as “orderliness”. An interesting finding is that *Attentiveness *has a negative correlation with the *Stability *factor of the FFPI. This could mean that less stable people are more inclined to take care of others. Almost all scales show a significant correlation with the factors of the FFPI. The scales *Dominance, Self-presentation, Need for contact, Leisure contact, Self-disclosure, Friendliness, Perseverance, Orderliness, Precision, Decisiveness, Optimism, Frustration tolerance *and *Resilience *even have correlations higher than .50 with factors of the FFPI. Hereby not only the validity of the factors but of the scales as well, is shown.

**Conclusions**

The Five Factor Model is the underlying theory of the FFPI as well as the WPI. From the described results on the relation between the factors of the FFPI with the factors and scales of the WPI we have gained insights in the psychological constructs that are measured by the questionnaires. The results closely match our expectations. Each factor of the WPI correlates highly with a theoretically similar factor of the FFPI, except for *Influence*, that correlated highly with three of the FFPI factors. The scales of the WPI correlate highly and in an interpretable way with the factors of the FFPI as well.

Given that the FFPI is a well-constructed and validated personality questionnaire, this study has been a good contribution to the validity of both the FFPI and the factors and the majority of the scales of the WPI.

#### 5.1.2. Predictive validity

To assert the predictive validity of the WPI we have conducted two studies in which certain employment criteria were measured some time after completion of the WPI. Subsequently, we have tested the predictive value of the WPI. Again, a significance level of 5% was used.

#### 5.1.2.1. Predictive validity: nurse study with 360 degrees feedback reviews

Ixly has conducted a study in the period between September 2005 and December 2008 on student-nurses following the BBL version of nursing school. The goal of the research was: to assess the predictive value of several personal characteristics of student-nurses for the degree of success in their education. We chose to only include the students that were enrolled in the education program with an apprenticeship agreement. This was done because the curriculum of these forms of education resembles the practical job situations more than the curriculum of the more theoretical forms of education. Moreover, in practical situations – more so than in a more theoretical context – a wider range of personal characteristics is required in order to function effectively.

The personality characteristics were measured in the beginning of the educational program by means of the WPI. The degree of success in the education was measured by means of 360 degrees reviews. The content of the 360 degrees feedback questionnaire was created by an experienced psychologist and adapted to the practical situation in consultation with the participating hospitals. The students took the 360 degrees questionnaire themselves and at least two up to ten persons that had worked with the student were invited to complete the questionnaire about the student. The people that reviewed the student were classified into five categories: managers, work supervisors, trainers/teachers, certified nurses and third or fourth year student nurses.

The student was rated on fourteen competencies, each of them measured by five behavioural indicators. During the study, it became clear that administrating the questionnaire took quite a lot of time, for the trainers as well as the students. Because of this, not all the participating students completed a 360 degrees questionnaire during the education program. In total, 80 students completed 360 degrees questionnaires that could be analysed.

**Sample**

The total research sample included 204 student-nurses, enrolled in the nurse education program, either on secondary level (MBO) or a higher level (HBO) (35 were male, 160 female, for 8 students no information on gender was provided). When we started the analyses, 80 students had completed the 360 degrees questionnaire (12 male and 67 female, for 1 student information on gender was not provided). The average age of this sample was 24.6 years old (ranging from 18 to 44 years). Of these students, 58 were enrolled in the program on secondary level and 14 on the higher level (for 8 of the students, the level was unknown).

**Expectations**

Since the study was exploratory in nature, no specific hypotheses were formulated. According to Bloemers and van der Molen, the predictive value that can be found for instruments, lies mostly between the 0.20 and 0.50 (Bloemers and van der Molen, 2004). Research conducted by Schmidt and Hunter shows that for personality instruments, relationships in the order of 0.31 are found (Schmidt and Hunter, 1998). The general expectation is therefore that a number of personality traits will show a correlation of about 0.30 with almost all competencies.

**Results**

The reliabilities (Cronbach’s alpha) of the scales of the WPI in this sample were between 0.79 (*Independence*) and 0.94 (*Attentiveness*). The reliability of the factors were between 0.91 (*Influence*) and 0.96 (*Exuberance*).

First, the factor scores of this sample were compared with the factor scores of the reference population. With a t-test, we checked whether there were significant differences in the factor scores between the sample and reference population. Analyses showed that the factor scores of *Influence *and *Exuberance *differed significantly. The student-nurses score significantly lower on these factors than the reference population (see Table 5.4). In this light, we can conclude that student-nurses are less ambitious and less driven than the reference population.

Table 5.4. Mean stenscore in this sample in comparison with the sten scores of the norm group (N = 80) | |||

Factor | Mean stenscore* | t-value | p-value |

Influence | 5.4 (1.5) | 2.59 | .01 |

Sociability | 6.2 (2.0) | -0.89 | .37 |

Exuberance | 5.3 (1.9) | 2.85 | .00 |

Structure | 5.7 (2.6) | -0.11 | .92 |

Stability | 5.8 (1.8) | 1.62 | .21 |

* The corresponding standard deviations are in brackets |

We chose to calculate, for every student, the average rating for every behavioural indicator of a competency. In this calculation, only the ratings of the respondents were used, not the self-rating of the student. The reliability of the competencies ranged from 0.89 (*Relational qualities*) to 0.95 (*Adaptability*). The magnitude of these reliabilities is partly due to the average rating that was used per behavioural indicator.

A factor analysis (PCA, varimax rotation) was conducted on the average ratings per behavioural indicator. The results showed that the first factor explained 65% of the variance. Therefore, we have initially created a total score, based on all competencies. This total score was used to calculate the correlations with the standardized scale and factor scores of the WPI. The scales *Self-disclosure *(*r *= 0.38, *p *< 0.01) and *Optimism *(*r *= 0.37, *p *< 0.01) were significantly correlated with the total score of the 360 degrees feedback questionnaire. In addition, the factor *Sociability *showed a positive correlation with the 360 degrees feedback questionnaire (*r *= 0.28, *p < *0.01).

In addition to this analysis, correlations were calculated between the ratings on the fourteen separate competencies and the factor scores of the WPI (Table 5.5). Six significant relations were found. Five of these relations included the factor *Sociability *of the WPI. The competencies *Adaptability*, *Ability to influence*, *Emotional stability, Drive *and *Relational qualities *all correlated significantly with the factor *Sociability*. *Quality of work *showed a relation with the scales of the factor *Stability*.

Table 5.5. Correlations competencies 360test and standardized factor scores WPI (N = 53) | |||||

Factors WPI | |||||

Competencies 360 | Influence | Sociability | Exuberance | Structure | Stability |

Adaptability | .04 | .30* | .06 | .03 | .04 |

Analytical qualities | .07 | .27 | .11 | -.003 | .19 |

Ability to influence | .04 | .33* | .14 | .05 | .22 |

Basic communicative skills | .09 | .26 | .07 | .02 | .02 |

Creativity/innovation | .04 | .26 | .06 | -.07 | .08 |

Discipline | .03 | .17 | .10 | .03 | .14 |

Emotional stability | .11 | .31* | .14 | .06 | .26 |

Exuberance | .06 | .28* | .26 | .19 | .26 |

Knowledge | -.02 | .17 | .08 | .02 | .11 |

Quality of work | -.004 | .16 | .13 | .05 | .18 |

Quantity of work | .02 | .24 | .21 | .24 | .32* |

Organizational qualities | .05 | .24 | .11 | .04 | .24 |

Relational qualities | .02 | .30* | .08 | .14 | .15 |

Nursing skills | .08 | .20 | .07 | .02 | .12 |

* Significant at the .05 level (2- sided) |

**Conclusions**

Based on the results of the analyses we can conclude that there is an association between the competencies as measured by 360 degrees method and the personality traits as measured by the WPI. The sizes of the associations are of the magnitude that may be expected on the basis of other studies (see Schmidt and Hunter, 1998). This result adds to the criterion validity of the WPI.

Five of the fourteen competencies necessary for the work of a nurse (i.e.: Adaptability, Ability to influence, Emotional stability, Exuberance and Relational qualities) all show a significant correlation with the factor *Sociability*. This result is explainable, given that the factor *Sociability *included the scales *Need for contact, Leisure contact, Friendliness *and *Attentiveness*. These are all personality traits that may be helpful as a nurse. From this we can conclude that when student-nurses score higher on the factor *Sociability*, that they can be expected to be successful in their job in the hospital.

In addition to the score on *Sociability, *it appears that the score on *Stability *is important in the selection of student nurses as well. The competency Quantity of work shows a significant correlation with the factor *Stability. *Noteworthy is the fact that the competency Emotional stability does not correlate significantly with the factor *Stability. *A possible explanation for this is that the lack of stability for student nurses is most notable in the quantity of work and less notable for the environment by means of the behavioural indicators that measure Emotional stability. Finally, Ability to influence, Emotional stability, Exuberance and Organizational qualities show relatively high correlations with the factor *Stability*.

**Suggestions for future research**

For the generalizability of the results, future research should formulate hypotheses in advance that can be tested by the analyses. The hypotheses may be derived from the results as found in this study. The hypotheses would then be:

– *Sociability *is an important factor for the successful functioning of (student)nurses

– *Stability *is an important factor for the quantity of work and is reported by the individual rather than that it is perceivable by the individual’s environment (co-workers).

#### 5.1.2.2 Predictive validity: performance study within a competitive employment agency

In 2010, for a period of 6 months, a study has been performed at a competitive employment agency (hereafter called Agency X) on the relationship between personality, job satisfaction and job performance.

**Method**

The full employee base of Agency X (284 employees) was invited to take the WPI; an employee satisfaction survey and the Career values questionnaire via e-mail. Performance measures, i.e. individual turnover and supervisor rating, on all persons were available. First, the mean scale and factor scores for different job positions were investigated. Subsequently, using linear regression, we have tested several hypotheses in order to assess the predictive value of the WPI on three job criteria.

**Sample**

189 employees have gone through the entire research process. Of these, 105 were female, 84 were male. 78 employees had a contract for a definite period, 106 employees for an indefinite period and for 5 employees this was unknown. At Agency X, two general job positions could be distinguished: 54 recruiters and 73 account managers; the other 62 employees were either staff or support staff. Table 5.6 describes the sample.

Table 5.6. Description of the sample of Agency X | |||

189 employees | |||

Mean age 24,5 year | |||

84 Male | 105 Female | ||

54 Recruiters | 73 Account managers | 62 Other functions | |

78 Definite contract | 106 Indefinite contract | ||

**Expectations**

Hypotheses on the relationships between personality, performance and job satisfaction were formulated prior to the analyses. Hypotheses were formulated on the basis of the expectations of the HR consultants at Agency X.

- The WPI scales
*Leisure contact*and*Need for contact*have a positive relation performance. - The WPI scales
*Decisiveness*and*Regularity*have a negative relationship with performance.

**Results**

The mean factor and scale scores for both function groups are presented in Table 5.7. Remarkably high scores for both function groups are the scores on the scale *Competition *and the factor *Influence. *Remarkably low scores for both groups are the scores on the scale *Decisiveness. *Differences in mean scale scores are most apparent for the scales *Independence, Self-confidence, Structure *and *Originality. *

Table 5.7. Mean stenscores of the factors and scales for both function groups of Bureau X (norm group used: Advice) | ||||

Recruiter (N = 54) | Account manager (N = 73) | |||

M | SD | M | SD | |

Influence | 7.06 | 1.58 | 7.96 | 1.60 |

Sociability | 5.54 | 1.62 | 5.63 | 1.57 |

Exuberance | 5.26 | 1.55 | 5.91 | 1.45 |

Structure | 3.81 | 1.71 | 3.53 | 1.81 |

Stability | 5.60 | 1.62 | 6.00 | 1.56 |

M | SD | M | SD | |

Adaptability | 5.08 | 1.43 | 5.80 | 1.35 |

Independence | 5.60 | 1.46 | 6.53 | 1.35 |

Perseverance | 5.15 | 1.50 | 5.05 | 1.42 |

Competition | 8.16 | 1.57 | 8.80 | 1.57 |

Energy | 5.61 | 1.44 | 5.88 | 1.45 |

Frustration-tolerance | 5.20 | 1.80 | 5.51 | 1.78 |

Resilience | 6.04 | 1.50 | 6.24 | 1.63 |

Attentiveness | 4.66 | 1.57 | 4.39 | 1.54 |

Dominance | 5.88 | 1.36 | 6.44 | 1.31 |

Status | 5.99 | 1.44 | 6.67 | 1.57 |

Precision | 4.37 | 1.57 | 4.26 | 1.21 |

Conformity | 5.30 | 1.42 | 4.88 | 1.71 |

Self-disclosure | 5.50 | 1.67 | 5.31 | 1.71 |

Orderliness | 4.24 | 1.70 | 4.33 | 1.83 |

Originality | 4.97 | 1.41 | 5.75 | 1.55 |

Decisiveness | 4.07 | 1.70 | 4.00 | 1.60 |

Optimism | 5.56 | 1.60 | 5.38 | 1.33 |

Personal growth | 5.45 | 1.72 | 5.57 | 1.91 |

Regularity | 4.80 | 1.85 | 4.47 | 1.46 |

Need for contact | 6.27 | 1.76 | 6.34 | 1.33 |

Leisure contact | 6.19 | 1.49 | 6.76 | 1.65 |

Self-presentation | 6.66 | 1.84 | 7.43 | 1.50 |

Trust | 4.80 | 1.92 | 4.85 | 1.92 |

Friendliness | 5.81 | 1.31 | 5.88 | 1.38 |

Self-confidence | 5.59 | 1.34 | 6.42 | 1.47 |

**Linear regression**

Based on the average scores on the scales and factors in Table 5.7, a profile of the average scores of the persons within a function group can be made. However, this does not tell us anything about the predictability of the performance and satisfaction on the basis of the WPI. For this, linear regression is used.

Based on the assumptions that underlie regression analyses, we have performed several explorative analyses on the dependent variables (y^{1}) *Turnover*, (y^{2}) *Supervisor rating* and (y^{3}) *Job satisfaction*. The assumptions underlying regression analyses are:

- Linear relationship between X and Y
- All pairs of observations (X, Y) are independent of each other
- The residuals are normally distributed
- The variances of the residuals are equal, independent of X (Homoscedasticity)

In the following tables, B denotes the regression coefficient, or in other words the extent to which the scale contributes to the dependent variable.

Table 5.8. Linear regression (x) WPI scales and (y^{1}) turnover per function for Agency X (Norm group Advice)* | |||||

Recruiter** | |||||

B | SE | Beta | t | Sig. | |

(Constant) | 24.472 | 7.786 | 3.054 | .004 | |

Decisiveness | 3.611 | 1.319 | 0.336 | 2.620 | .012 |

Orderliness | -3.278 | 1.304 | -0.382 | -2.983 | .005 |

Account manager | |||||

B | SE | Beta | t | Sig. | |

No variables included according to stepwise inclusion of variables. | |||||

* Stepwise inclusion of (x) | |||||

** R^{2} = .247 |

Table 5.9. Linear regression (x) WPI scales and (y^{2}) supervisor rating per function for Agency X (Norm group Advice)* | |||||

Recruiter** | |||||

B | SE | Beta | t | Sig. | |

(Constant) | 0.481 | 0.208 | 2.317 | .031 | |

Adaptability | 0.151 | 0.042 | 0.862 | 3.561 | .002 |

Competition | -0.091 | 0.035 | -0.623 | -2.575 | .018 |

Account manager*** | |||||

B | SE | Beta | t | Sig. | |

(Constant) | 0.177 | 0.116 | 1.53 | .135 | |

Decisiveness | 0.075 | 0.028 | 0.405 | 2.659 | .012 |

* Stepwise inclusion of (x) | |||||

** R^{2} = .388 | |||||

*** R^{2} = .164 |

Table 5.10. Linear regression (x) WPI scales and (y^{3}) satisfaction per function for Bureau X (Norm group Advice)* | |||||

Recruiter** | |||||

B | SE | Beta | t | Sig. | |

(Constant) | 58.362 | 4.650 | 12.552 | .000 | |

Adaptability | 2.104 | 0.716 | 0.388 | 2.937 | .005 |

Independence | -1.674 | 0.616 | -0.315 | -2.718 | .009 |

Resilience | 1.510 | 0.639 | 0.292 | 2.364 | .022 |

Need for contact | 1.149 | 0.508 | 0.26 | 2.259 | .028 |

Account manager*** | |||||

B | SE | Beta | t | Sig. | |

(Constant) | 49.643 | 4.705 | 10.552 | .000 | |

Optimism | 4.428 | 0.863 | 0.534 | 5.132 | .000 |

Conformity | 1.679 | 0.637 | 0.261 | 2.636 | .010 |

Orderliness | -1.344 | 5.98 | -0.223 | -2.249 | .028 |

* Stepwise inclusion of (x) | |||||

** R^{2} = .440 | |||||

*** R^{2} = .398 |

**Conclusions**

Results from the study at Agency X show that a clear personality profile can be distilled of the two main functions within the organization (Table 5.8). The high average competitiveness of the employees is a striking but expected trend within this organization. Following the same reasoning, the average low score on *Decisiveness* is a trend that shows that employees within this competitive organization are able (or should be able to) switch quickly and that this way of working fits with the current employee base.

In terms of the predictive value of the results of the WPI on the measures of performance a couple of scales stood out. Quite remarkable was that the score of a recruiter on the scale *Competition *had a negative effect on the supervisor’s rating of performance. From conversations with the HR consultants of Agency X it became clear that while a competitive attitude during the selection process seemed to be an advantage, this was not always seen as a positive attitude during working in teams.

Remarkable is the positive effect of *Decisiveness *as well, while the average employee of Agency X scored below average on this scale. A different important aspect of the functioning is the satisfaction of the employee him/herself. From our previous analyses, it became clear that for both functions different scales contributed to this satisfaction. For recruiters, this was mainly the case for *Adaptability *while for account managers this was mainly the case for *Optimism*.

The goal of this study was to assess the predictive value of the WPI by means of proposed hypotheses. It appears that several scales of the WPI are predictive of job criteria. This contributes greatly to the predictive validity of the WPI.

## 5.2 Construct validity

#### 5.2.1. Construct validity: Factor and scale structure

The most important indication of the construct validity of the WPI are the results from the MGM (Multiple Group Method) which we have discussed earlier in the paragraph ‘Structure of the WPI’.

For both the Advice and Selection group an MGM analysis was performed. The background characteristics of these groups are described in the chapter Norms (paragraph 3.1). The analysis was performed on the data from the weighed norm groups.

When an MGM analysis is applied to the both weighed groups, the structure does not appear to differ significantly from the structure that was found in the construction phase of the questionnaire (on the unweighed groups). This holds for the scales as well as for the factors. The results of the MGM analyses on the division of items in scales are included in Appendix 1. The results of the MGM analyses on the division of scales into factors for both norm groups are presented in the following two tables. Notable results and differences between the groups will be discussed.

The presentation of the results of the MGM analyses (Table 5.1 and Table 5.2) is made easier by the use of colours. Three colours are used, each with their own meaning. Grey means that the item/scale correlates the highest with its own scale/factor (item-rest correlation) and that the difference in height with the other correlations is significant (as calculated with the t-test for differences between dependent correlations (Steiger (1980)). Yellow means that there are more correlations of the item/ scale that correlate with other scales/factors (respectively), but not in a significant way. The colour red means that the item/scale has a higher correlation with a different scale/factor than with its own scale/factor and that the difference in the height of the correlation is also significant. In the presented tables there are no red cells.

In the MGM on the division of scales into factors, there are 20 scales that correlate significantly the highest with their own scale (grey) for the Advice group, and 16 scales for the Selection group (see Tables 5.11 and 5.12). It appears that no other scale correlates higher with a different factor than that it was theoretically assigned to (red). In the Advice group, there are five scales that correlate in a meaningful way with more than one factor but without a significance difference, for the Selection group this was the case for nine scales. Three of these scales overlap between the Advice and Selection group.

Information on the results of the MGM analysis based on the division of items in scales is reported in Appendix 1. For both norm groups, there is no item that has a significantly higher correlation with a scale other than the scale it was assigned to. In the Advice group, 18 of the 276 items correlated with scales other than its own scale but without a significant difference in correlation heights (yellow). In the Selection group, 44 of the 276 items were in yellow cells. All other items correlated significantly higher with their own scale than with other scales.

When both groups are compared, it appears that 17 items correlate with more than one scale – but not significantly (yellow) – in both groups. One item appears to show a yellow cell in the Selection group, but not in the Advice group. There are 27 items in the Selection group that correlate with more than one scale – but not significantly (yellow) – while these items were in grey cells in the Advice group. The fact that there are fewer items in the Selection group that differ significantly from each other in terms of their correlations is die to smaller number of candidates in this group. Because the research population is smaller, significant results are less likely to occur.

Because the MGM results show relatively few differences, and because no item or scale correlates significantly with a different scale or factor, respectively, than their own scale or factor, these analyses can be viewed as solid support for the stability of the structure of the WPI. The division of items into scales and scales into factors is supported empirically.

**MGM in an independent group**

Since the Advice and Selection group were not only used to test the structure of the WPI, but were also used as input for the development of the questionnaire, an additional MGM analysis was run on an independent group. Background characteristics of this group can be found in the paragraph ‘Study with the FFPI’. The results of this group are compared with the advice norm group, because the WPI was administered in an advice setting. It appeared that the results MGM of the independent group only differed slightly from the results of the Advice group. The MGM results for the analyses on both the scales as on the factors are presented in Appendix 1. Notable results and differences between the independent group and the Advice group (the MGM from the previous paragraph) will be discussed.

Table 5.11. MGM results for the Advice group (N = 712) | |||||||

Factor | Scale | Exuberance | Stability | Influence | Sociability | Structure | |

Exuberance | Adaptability | .69 | .59 | .51 | .48 | -.03 | |

Perseverance | .54^{a} | .41 | .27 | .32 | .51^{a} | ||

Energy | .71 | .53 | .47 | .49 | .27 | ||

Independence | .37^{a} | .12 | .34^{a} | .08 | -.10 | ||

Originality | .57 | .42 | .48 | .37 | -.03 | ||

Personal growth | .63 | .35 | .48 | .34 | .16 | ||

Stability | Frustration-tolerance | .29 | .59 | .10 | .33 | .24 | |

Resilience | .40 | .68 | .29 | .25 | .06 | ||

Optimism | .51 | .63 | .33 | .52 | .12 | ||

Self-confidence | .60^{a} | .64^{a} | .53 | .40 | .08 | ||

Influence | Competition | .39 | .16 | .61 | .09 | -.11 | |

Dominance | .63^{a} | .51 | .63^{a} | .48 | -.10 | ||

Status | .47 | .31 | .57 | .21 | .05 | ||

Self-presentation | .38 | .26 | .67 | .47 | -.24 | ||

Sociability | Need for contact | .35 | .37 | .37 | .65 | .10 | |

Friendliness | .52 | .50 | .37 | .71 | .14 | ||

Leisure contact | .52 | .56 | .52 | .62 | .01 | ||

Trust | .14 | .21 | .10 | .48 | -.08 | ||

Self-disclosure | .26 | .20 | .33 | .59 | -.06 | ||

Attentiveness | .33 | .20 | .03 | .53 | .30 | ||

Structure | Conformity | .03 | .19 | -.14 | .20 | .36 | |

Precision | .35 | .17 | -.01 | .09 | .55 | ||

Orderliness | .33 | .24 | .06 | .19 | .50 | ||

Regularity | -.31^{a} | -.32^{a} | -.30^{a} | -.17 | .33^{a} | ||

Decisiveness | .20 | .23 | -.03 | -.02 | .45 | ||

The numbers concern the yellow cells, the other cells are coloured grey.^{a} | |||||||

Table 5.12. MGM results for the Selection group (N = 369) | ||||||

Factor | Scale | Exuberance | Stability | Influence | Sociability | Structure |

Exuberance | Adaptability | .72 | .60 | .38 | .54 | .17 |

Perseverance | .65 | .53 | .28 | .43 | .51 | |

Energy | .71 | .64 | .39 | .59 | .41 | |

Independence | .23^{a} | .03 | .20^{a} | .05 | -.07 | |

Originality | .58 | .49 | .42 | .45 | .15 | |

Personal growth | .59 | .39 | .50 | .43 | .30 | |

Stability | Frustration-tolerance | .39 | .62 | .06 | .40 | .38 |

Resilience | .44 | .70 | .21 | .40 | .28 | |

Optimism | .59^{a} | .58^{a} | .31 | .62^{a} | .23 | |

Self-confidence | .57 | .64 | .43 | .45 | .32 | |

Influence | Competition | .30 | .08 | .64 | .12 | -.10 |

Dominance | .58^{a} | .46 | .57^{a} | .53^{a} | .01 | |

Status | .40 | .23 | .58 | .22 | .12 | |

Self-presentation | .32 | .21 | .66 | .42 | -.21 | |

Sociability | Need for contact | .44 | .45 | .39 | .66 | .17 |

Friendliness | .59 | .60 | .34 | .68 | .34 | |

Leisure contact | .52 | .62^{a} | .48 | .60^{a} | .12 | |

Trust | .27 | .31 | .19 | .46 | .03 | |

Self-disclosure | .19 | .16 | .29 | .49 | -.06 | |

Attentiveness | .48^{a} | .38 | .05 | .55^{a} | .40 | |

Structure | Conformity | .29 | .35^{a} | -.01 | .34^{a} | .42^{a} |

Precision | .36 | .32 | -.01 | .16 | .60 | |

Orderliness | .44^{a} | .43^{a} | .09 | .27 | .51^{a} | |

Regularity | -.19 | -.21^{a} | -.23^{a} | -.14 | .31^{a} | |

Decisiveness | .25 | .38^{a} | -.03 | .14 | .45^{a} | |

The numbers concern the yellow cells, the other cells are coloured grey.^{a} |

From the results of the MGM analyses on this independent research group on the item level (Appendix 11) it became clear that no single item correlated higher with a different scale other than the scale it was assigned to (red). 241 of the 272 items correlated higher with their own scales than with any other scale. The remaining 31 items correlated with scales other than their own scale, but not in a significant way. After comparison with the Advice group it appeared that 9 of the yellow labelled items received a yellow label in the Advice group as well. We concluded that the chosen makeup of the questionnaire on the item level was upheld in the independent group.

From the results of the MGM analyses on this independent research group on the factor level it appeared that 19 of the 25 scales correlated higher with their own factor than with any other factor. Five scales correlated with other factors, but not in a significant way. There was one scale that correlated significantly higher with a different factor than with its own designated factor. However, this scale did correlate significantly higher with its own factor than with any other factor in the Advice group. In any independent group, especially when it concerns a specific organization as is the case here, there will be differences with the norm group. Since the difference only concerns one scale, it seems that the items and scales of the WPI show a very stable structure.

**Correlations between factors**

To check whether the factors have a shared component (or components), a correlation analysis was done on the factor level as well as a factor analysis in both research groups (Advice and Selection). The correlations are reported in Table 5.13.

Table 5.13. Correlations between factors | |||||

Correlations Advice group (N = 712) / Selection group (N = 369)* | |||||

Factor | Influence | Sociability | Exuberance | Structure | Stability |

Influence | – | ||||

Sociability | .39 / .44 | – | |||

Exuberance | .59 / .55 | .48 / .56 | – | ||

Structure | -.12 / -.06 | .10 / .16 | .18 / .26 | – | |

Stability | .39 / .35 | .47 / .51 | .56 / .60 | .16 / .33 | – |

* Every correlation is presented for the Advice Group First, for the Selection Group behind the / sign |

The results of the rotated component matrices of the performed factor analyses are as follows (Table 5.14 and 5.15).

Table 5.14. Rotated component matrix for the Advice group (N = 712) | Table 5.15. Rotated component matrix for the Selection group (N = 369) | |||||

Component | Component | |||||

1 | 2 | 1 | 2 | |||

Influence | .8 | -.4 | Influence | .8 | -.4 | |

Sociability | .7 | .1 | Sociability | .8 | .2 | |

Exuberance | .9 | .1 | Exuberance | .8 | .3 | |

Structure | .1 | 1.0 | Structure | .1 | .9 | |

Stability | .8 | .2 | Stability | .7 | .5 |

**Conclusion**

For both the advice and selection group, as well as the independent group, MGM analyses showed that the WPI has a solid internal structure. The majority of the items and scales correlate significantly higher with its “own” designated scales and factors, respectively, than with other scales or factors. A few items and scales show correlations with other scales or factors, but not in a significant way. Furthermore, all factors showed a shared component, with the exception of the factor *Structure*. Future research should shed more light on what this component is. However, it is clear that all factors have a meaning, independent from one another.

**5.2.2. Construct validity: relationship with background variables**

* *

To assert whether the factor and scale scores have a relationship with the background variables, we have analysed for all the variables, whether the mean scores on each scale and each factor for the different categories of these variables differed significantly from each other. For both the Advice and Selection group this was done by means of ANOVA (Table 5.16 through 5.19). The data was weighed during these analyses. The weighed N of the Advice group is 712 (unweighed N = 5629), for the Selection group the N is 369 (unweighed N = 1514).

In addition, the eta (see page 18 for an overview of the statistical terms) was calculated. Eta can be used as a measure of association for variables with more than two categories. When the difference is significant in a two sided test on the 5%-level, this is indicated by a * next to the eta of the corresponding factor/scale.

Finally, results of a study are reported in which job sector differences in the mean sten scores of some factors and scales were identified (Table 5.10). The significance of the differences is determined by T tests.

**Gender**

The mean scale and factor sten scores for males and females are represented in Table 5.16.

Table 5.16. Mean stenscores by gender including eta-value | ||||||

Advice group (N = 712) | Selection group (N = 369) | |||||

Factor | eta* | Male | Female | eta* | Male | Female |

Influence | .17* | 5.8 | 5.1 | .16* | 5.8 | 5.1 |

Sociability | .14* | 5.2 | 5.9 | .11* | 5.3 | 5.8 |

Exuberance | .04 | 5.6 | 5.4 | .03 | 5.5 | 5.6 |

Structure | .09* | 5.3 | 5.7 | .03 | 5.4 | 5.6 |

Stability | .08* | 5.7 | 5.3 | .10 | 5.7 | 5.3 |

Scale | ||||||

Status | .10* | 5.7 | 5.3 | .06 | 5.6 | 5.4 |

Dominance | .13* | 5.7 | 5.2 | .13* | 5.7 | 5.2 |

Competition | .23* | 5.9 | 4.9 | .22* | 5.9 | 5.0 |

Self-presentation | .08* | 5.6 | 5.3 | .10 | 5.7 | 5.3 |

Need for contact | .08* | 5.4 | 5.7 | .07 | 5.4 | 5.7 |

Leisure contact | .01 | 5.5 | 5.5 | .01 | 5.5 | 5.5 |

Self-disclosure | .08* | 5.4 | 5.7 | .02 | 5.5 | 5.6 |

Trust | .10* | 5.3 | 5.7 | .02 | 5.5 | 5.5 |

Friendliness | .12* | 5.3 | 5.8 | .15* | 5.2 | 5.9 |

Attentiveness | .23* | 5.1 | 6.0 | .22* | 5.1 | 6.0 |

Energy | .00 | 5.5 | 5.5 | .05 | 5.4 | 5.6 |

Personal growth | .01 | 5.5 | 5.5 | .06 | 5.4 | 5.7 |

Perseverance | .06 | 5.4 | 5.6 | .04 | 5.4 | 5.6 |

Adaptability | .03 | 5.6 | 5.4 | .03 | 5.5 | 5.6 |

Originality | .13* | 5.7 | 5.2 | .12* | 5.7 | 5.2 |

Independence | .07 | 5.6 | 5.3 | .04 | 5.4 | 5.6 |

Orderliness | .10* | 5.3 | 5.7 | .07 | 5.4 | 5.7 |

Precision | .09* | 5.3 | 5.7 | .01 | 5.5 | 5.5 |

Regularity | .08* | 5.4 | 5.7 | .08 | 5.4 | 5.7 |

Conformity | .05 | 5.4 | 5.6 | .06 | 5.4 | 5.6 |

Decisiveness | .02 | 5.5 | 5.5 | .10 | 5.7 | 5.3 |

Self-confidence | .15* | 5.8 | 5.1 | .20* | 5.9 | 5.0 |

Optimism | .07* | 5.4 | 5.7 | .05 | 5.4 | 5.6 |

Frustration-tolerance | .02 | 5.5 | 5.5 | .07 | 5.6 | 5.3 |

Resilience | .17* | 5.8 | 5.1 | .13* | 5.7 | 5.2 |

* Significant results of the ANOVA are indicated by a * for the eta values. |

In the Advice group, there are 4 factors and 16 scales that differ significantly from each other on the gender variable. In the Selection group, this is the case for 2 factors and 7 scales.

For the background variable gender, the found differences were maximally 1.0 sten; this corresponds to a difference of 0.5 times the standard deviation. The mean difference over all significant scales and factors was 0.56 sten; this corresponds to 0.28 times the standard deviation. The differences reported for the gender variable are differences that are also reported in the literature. Examples are women’s higher scores on *Sociability*, *Attentiveness *and *Leisure contact *and men’s higher scores on *Status*, *Trust *and *Influence *(Beutel & Marini, 1995).

Table 5.17. Mean stenscore by age including eta-value | ||||||||

Advice group (N = 712) | Selection group (N = 369) | |||||||

Factor | eta* | 15-24 years | 25-44 years | 45-65 years | eta* | 15-24 years | 25-44 years | 45-65 years |

Influence | .08 | 5.5 | 5.7 | 5.3 | .14* | 5.9 | 5.7 | 5.1 |

Sociability | .01 | 5.5 | 5.5 | 5.5 | .06 | 5.8 | 5.5 | 5.4 |

Exuberance | .15* | 4.6 | 5.6 | 5.6 | .08 | 5.7 | 5.6 | 5.3 |

Structure | .08 | 5.0 | 5.5 | 5.6 | .08 | 6.0 | 5.5 | 5.4 |

Stability | .05 | 5.2 | 5.5 | 5.6 | .03 | 5.7 | 5.5 | 5.5 |

Scale | ||||||||

Status | .18* | 6.1 | 5.7 | 5.0 | .21* | 6.2 | 5.7 | 4.9 |

Dominance | .12* | 4.9 | 5.5 | 5.7 | .07 | 5.3 | 5.4 | 5.7 |

Competition | .11* | 5.4 | 5.7 | 5.2 | .17* | 5.8 | 5.8 | 5.0 |

Self-presentation | .07 | 5.4 | 5.6 | 5.3 | .13 | 5.7 | 5.7 | 5.1 |

Need for contact | .16* | 6.4 | 5.5 | 5.2 | .15* | 6.0 | 5.7 | 5.1 |

Leisure contact | .08 | 5.0 | 5.5 | 5.6 | .02 | 5.4 | 5.5 | 5.5 |

Self-disclosure | .07 | 5.1 | 5.6 | 5.5 | .06 | 5.5 | 5.4 | 5.7 |

Trust | .05 | 5.5 | 5.4 | 5.6 | .12 | 5.4 | 5.3 | 5.8 |

Friendliness | .03 | 5.4 | 5.6 | 5.4 | .19* | 6.0 | 5.8 | 5.0 |

Attentiveness | .05 | 5.8 | 5.4 | 5.5 | .07 | 5.8 | 5.4 | 5.6 |

Energy | .10* | 5.0 | 5.6 | 5.5 | .12 | 6.0 | 5.6 | 5.2 |

Personal growth | .08 | 5.4 | 5.7 | 5.3 | .20* | 6.2 | 5.8 | 4.9 |

Perseverance | .16* | 4.6 | 5.5 | 5.7 | .05 | 5.6 | 5.6 | 5.4 |

Adaptability | .07 | 5.1 | 5.5 | 5.6 | .05 | 5.7 | 5.6 | 5.4 |

Originality | .09* | 5.0 | 5.5 | 5.7 | .04 | 5.5 | 5.4 | 5.6 |

Independence | .21* | 4.3 | 5.6 | 5.7 | .10 | 4.9 | 5.5 | 5.6 |

Orderliness | .15* | 4.6 | 5.6 | 5.7 | .03 | 5.7 | 5.5 | 5.5 |

Precision | .12* | 4.8 | 5.5 | 5.6 | .07 | 5.8 | 5.5 | 5.4 |

Regularity | .03 | 5.4 | 5.5 | 5.6 | .05 | 5.7 | 5.4 | 5.6 |

Conformity | .15* | 6.3 | 5.5 | 5.2 | .24* | 6.4 | 5.8 | 4.9 |

Decisiveness | .13* | 4.8 | 5.5 | 5.7 | .12 | 5.4 | 5.3 | 5.8 |

Self-confidence | .10* | 5.0 | 5.5 | 5.7 | .02 | 5.6 | 5.5 | 5.5 |

Optimism | .07 | 5.1 | 5.5 | 5.6 | .01 | 5.6 | 5.5 | 5.5 |

Frustration-tolerance | .03 | 5.4 | 5.5 | 5.6 | .04 | 5.7 | 5.5 | 5.5 |

Resilience | .04 | 5.6 | 5.6 | 5.4 | .02 | 5.6 | 5.5 | 5.5 |

* Significant results of the ANOVA are indicated by a * for the eta values. |

**Age**

The mean scale and sten scores for the different age categories are represented in Table 5.17. In the standardization process, the age variable was divided into three categories (15-24, 25-44 and 45-65 years). For this variable, ANOVA analyses were run as well to check whether differences were significant. Significant differences are marked with a * next to the eta.

In the Advice group, there were 1 factor and 13 scales that differed significantly from each other in terms of age, for the Selection group this was the case for 1 factor and 6 scales. The largest difference in means is 1.5 sten; this corresponds to 0.75 times the standard deviation. The average difference over all the significant scales and factors is 0.98 sten. This corresponds to a difference of 0.49 times the standard deviation. From Table 5.17 it appears that the difference between the two oldest groups (25-44 years and 45-65 years) is generally small. The found significant differences are mainly caused by the mean sten score of the group up to 25 years.

**Education**

The mean scale and sten scores for the education variable are represented in Table 5.18.

Table 5.18. Mean stenscore by education including eta-values | ||||||||

Advice group (N = 712) | Selection group (N = 369) | |||||||

Factor | Education | Education | ||||||

eta* | Lower | Second | Higher | eta* | Lower | Second | Higher | |

Influence | .12* | 5.3 | 5.4 | 5.9 | .21* | 5.2 | 5.2 | 6.2 |

Sociability | .07 | 5.5 | 5.4 | 5.7 | .12 | 5.3 | 5.3 | 5.9 |

Exuberance | .10* | 5.3 | 5.4 | 5.8 | .09 | 5.4 | 5.3 | 5.8 |

Structure | .20* | 6.1 | 5.6 | 4.9 | .19* | 5.8 | 5.8 | 4.8 |

Stability | .07 | 5.5 | 5.3 | 5.7 | .04 | 5.5 | 5.4 | 5.6 |

Scale | ||||||||

Status | .03 | 5.5 | 5.4 | 5.6 | .10 | 5.4 | 5.3 | 5.8 |

Dominance | .13* | 5.2 | 5.4 | 5.9 | .22* | 5.1 | 5.2 | 6.2 |

Competition | .11* | 5.2 | 5.4 | 5.8 | .19* | 5.2 | 5.3 | 6.1 |

Self-presentation | .12* | 5.3 | 5.3 | 5.9 | .16* | 5.4 | 5.2 | 6.0 |

Need for contact | .03 | 5.6 | 5.4 | 5.5 | .08 | 5.4 | 5.4 | 5.8 |

Leisure contact | .06 | 5.4 | 5.4 | 5.7 | .15* | 5.3 | 5.3 | 6.0 |

Self-disclosure | .14* | 5.1 | 5.4 | 5.9 | .11 | 5.3 | 5.4 | 5.8 |

Trust | .18* | 5.3 | 5.2 | 6.1 | .17* | 5.2 | 5.3 | 6.0 |

Friendliness | .05 | 5.7 | 5.5 | 5.4 | .03 | 5.5 | 5.4 | 5.6 |

Attentiveness | .10* | 5.8 | 5.4 | 5.3 | .04 | 5.4 | 5.5 | 5.6 |

Energy | .02 | 5.4 | 5.5 | 5.5 | .04 | 5.5 | 5.4 | 5.6 |

Personal growth | .13* | 5.2 | 5.4 | 5.9 | .08 | 5.4 | 5.3 | 5.8 |

Perseverance | .05 | 5.7 | 5.5 | 5.4 | .05 | 5.7 | 5.4 | 5.5 |

Adaptability | .10* | 5.3 | 5.4 | 5.8 | .07 | 5.6 | 5.3 | 5.6 |

Originality | .06 | 5.5 | 5.4 | 5.7 | .11 | 5.2 | 5.4 | 5.8 |

Independence | .17* | 5.1 | 5.4 | 6.0 | .09 | 5.3 | 5.4 | 5.8 |

Orderliness | .11* | 5.8 | 5.6 | 5.2 | .11 | 5.7 | 5.6 | 5.2 |

Precision | .15* | 5.8 | 5.6 | 5.1 | .20* | 5.7 | 5.8 | 4.9 |

Regularity | .22* | 6.1 | 5.6 | 4.9 | .20* | 5.8 | 5.8 | 4.9 |

Conformity | .22* | 6.1 | 5.6 | 4.9 | .14* | 5.7 | 5.7 | 5.1 |

Decisiveness | .04 | 5.5 | 5.4 | 5.6 | .01 | 5.5 | 5.5 | 5.5 |

Self-confidence | .03 | 5.5 | 5.4 | 5.6 | .04 | 5.6 | 5.5 | 5.5 |

Optimism | .10* | 5.3 | 5.4 | 5.8 | .14* | 5.2 | 5.4 | 6.0 |

Frustration-tolerance | .09 | 5.6 | 5.3 | 5.7 | .01 | 5.5 | 5.5 | 5.5 |

Resilience | .03 | 5.6 | 5.5 | 5.5 | .11 | 5.9 | 5.3 | 5.4 |

* Significant results of the ANOVA are indicated by a * for the eta values. |

In the Advice group, there were 3 factors and 14 scales that differed significantly from each other in terms of education, for the Selection group this was the case for 2 factors and 9 scales. The largest difference in means is 1.2 sten, which corresponds to a difference of 0.6 times the standard deviation. The average distance over all significant scales and factors is 0.77 sten: this corresponds to a difference of 0.39 times the standard deviation.

‘Stereotypical’ differences in scores are found for the different educational levels. Examples are the fact that people with lower educational levels have a higher need for *Structure *and that people with higher educational levels find *Personal growth *more important.

Table 5.19. Mean sten score by job status including eta-values | ||||||

Advice group (N=712) | Selection group (N=369) | |||||

Factor | eta* | non-UWV | UWV | eta* | non-UWV | UWV |

Influence | .06 | 5.5 | 4.7 | .09 | 5.5 | 4.4 |

Sociability | .02 | 5.5 | 5.3 | .04 | 5.5 | 5.0 |

Exuberance | .03 | 5.5 | 5.2 | .08 | 5.5 | 4.6 |

Structure | .05 | 5.5 | 6.1 | .07 | 5.5 | 6.4 |

Stability | .02 | 5.5 | 5.2 | .08 | 5.5 | 4.6 |

Scale | ||||||

Status | .05 | 5.5 | 4.9 | .04 | 5.5 | 5.1 |

Dominance | .06 | 5.5 | 4.9 | .12* | 5.5 | 4.1 |

Competition | .05 | 5.5 | 4.9 | .07 | 5.5 | 4.8 |

Self-presentation | .04 | 5.5 | 5.0 | .07 | 5.5 | 4.7 |

Need for contact | .02 | 5.5 | 5.3 | .06 | 5.5 | 4.8 |

Leisure contact | .02 | 5.5 | 5.2 | .07 | 5.5 | 4.7 |

Self-disclosure | .02 | 5.5 | 5.2 | .02 | 5.5 | 5.3 |

Trust | .03 | 5.5 | 5.2 | .05 | 5.5 | 5.0 |

Friendliness | .01 | 5.5 | 5.6 | .03 | 5.5 | 5.2 |

Attentiveness | .03 | 5.5 | 5.8 | .03 | 5.5 | 5.9 |

Energy | .02 | 5.5 | 5.2 | .06 | 5.5 | 4.9 |

Personal growth | .04 | 5.5 | 5.0 | .06 | 5.5 | 4.8 |

Perseverance | .01 | 5.5 | 5.7 | .02 | 5.5 | 5.3 |

Adaptability | .02 | 5.5 | 5.2 | .08 | 5.5 | 4.6 |

Originality | .02 | 5.5 | 5.3 | .08 | 5.5 | 4.7 |

Independence | .02 | 5.5 | 5.3 | .04 | 5.5 | 5.0 |

Orderliness | .03 | 5.5 | 5.8 | .00 | 5.5 | 5.5 |

Precision | .02 | 5.5 | 5.8 | .05 | 5.5 | 6.0 |

Regularity | .06 | 5.5 | 6.2 | .13 | 5.4 | 6.9 |

Conformity | .05 | 5.5 | 6.1 | .08 | 5.5 | 6.4 |

Decisiveness | .00 | 5.5 | 5.5 | .01 | 5.5 | 5.4 |

Self-confidence | .03 | 5.5 | 5.2 | .07 | 5.5 | 4.8 |

Optimism | .04 | 5.5 | 5.0 | .12* | 5.5 | 4.1 |

Frustration-tolerance | .01 | 5.5 | 5.6 | .02 | 5.5 | 5.3 |

Resilience | .01 | 5.5 | 5.4 | .03 | 5.5 | 5.2 |

* Significant results of the ANOVA are indicated by a * for the eta values. |

Two significant differences were found for the job status variable (Table 5.19). People not registered at the UWV scored significantly higher on *Dominance *and *Optimism *than people registered at the UWV in the Selection group.

**Job sector**

Using t-tests, we have checked which scales and factors show significant differences in means between the different job sectors. The distribution and sizes of the different sectors in the sample are represented in Table 3.4. The significant differences in mean scores are represented in Table 5.20, 5.21, 5.22 and 5.23. This study was performed on a sample of 712 people in the Advice group and 369 people in the Selection group.

Table 5.20. Differences in mean sten scores on the factor level between sectors in the Advice group (N=719) | ||||||||

Factor | Sector I | Sten I | Sector J | Sten J | Difference (I-J) | p* | SD_{p}** | Effect size*** |

Structure | Business and administration | 6.12 | Commercial services | 4.99 | 1.13 | .00 | 1.64 | 0.69 |

Structure | Business and administration | 6.12 | Public administration, Safety and Law | 5.07 | 1.06 | .00 | 1.59 | 0.66 |

* Due to the large number of significance tests performed in this study, a correction of the significance level is needed. Only significant differences after the Bonferonni correction are presented. ** Pooled standard deviation. *** The difference between mean scores divided by the pooled standard deviation. |

On the factor level, only the factor *Structure *shows significant differences between the different job sectors. The sector ‘Business and Administration’ differs significantly from ‘Commercial services’ and ‘Public administration, Safety and Law’ on this factor.

Table 5.21. Differences in mean sten scores on the factor level between sectors in the Selection group (N=1067) | ||||||||

Factor | Sector I | Sten I | Sector J | Sten J | Difference(I-J) | p* | SD_{p}** | Effect size*** |

Influence | Commercial services | 6.68 | Health, Wellbeing and Personal Care | 5.92 | .75 | .00 | 1.30 | 0.58 |

Influence | Commercial services | 6.68 | Public administration, Safety and Law | 5.90 | .78 | .00 | 1.29 | 0.61 |

Influence | Engineering and Production | 6.94 | Public administration, Safety and Law | 5.90 | 1.04 | .00 | 1.35 | 0.74 |

Influence | Engineering and Production | 6.94 | Health, Wellbeing and Personal Care | 5.92 | 1.01 | .00 | 1.33 | 0.74 |

Exuberance | Commercial services | 6.44 | Public administration, Safety and Law | 5.78 | .66 | .00 | ||

* Due to the large number of significance tests performed in this study, a correction of the significance level is needed. Only significant differences after the Bonferonni correction are presented. ** Pooled standard deviation. *** The difference between mean scores divided by the pooled standard deviation. |

In the Selection group, significant differences were only found for the factors *Influence *and *Exuberance *(see Table 5.21).

Table 5.22. Significant differences in mean sten scores on the scale level between sectors in the Advice group | ||||

Scale | Sector I | Sector J | Difference (I-J) | p |

Regularity | Business and administration | Public administration, Safety and Law | 1.50 | .00 |

Dominance | Business and administration | Public administration, Safety and Law | -1.53 | .00 |

Preciseness | Business and administration | Public administration, Safety and Law | 1.54 | .00 |

Competition | Health, Wellbeing and Personal Care | Commercial services | -1.09 | .00 |

Self-confidence | Health, Wellbeing and Personal Care | Public administration, Safety and Law | -1.07 | .00 |

Regularity | Business and administration | Commercial services | 1.20 | .00 |

Dominance | Business and administration | Commercial services | -1.22 | .00 |

Preciseness | Business and administration | Commercial services | 1.22 | .00 |

Conclusion

The most important question that needs to be answered is whether the WPI can be used sector independently for selection and advice purposes. In other words: how large is the effect of the sector on the scale and factor scores? In answering this question, it is not important how one sector relates to the other sectors, but more how a sector relates to the total norm group. The distribution of people over the different sectors for both norm groups is represented in Appendix 12, as well as the mean factor and scale scales and the corresponding effect difference with the total group. From this it becomes clear that people in some sectors score significantly higher on some scales and factors than the people in the total norm group, but that this does not outweigh the information that the groups contain.

Despite the large numbers of people in the norm groups, we found only a limited amount of significant effects for the background variables gender, age, job status and education. In terms of effect size (Cohen, 1992), the significant differences for the background variables are not of such an order that separate norm groups should be created. It can be interesting to include this information in the interpretation of the results since it concerns real differences, but this is not absolutely necessary. The study on differences between sectors has shown a number of results that can be considered as real, given the nature of the sectors. This contributes to the construct validity of the WPI, and indicates why separate norm groups are not desirable.

#### 5.2.3. Construct validity: Cultural Bias

Since people with different cultural backgrounds live and work in the Netherlands, it is necessary that the WPI is a culturally fair questionnaire. Culturally fair means that no unfair biases occur in individual outcomes, and that only real differences between individuals are evident in relation to the entire work force. After all, these differences have a significant meaning for the Dutch labour market.

An extensive study was done to investigate the cultural bias of the WPI. The results of this study are described in Appendix 13.

The most important finding is that significant differences for only two scales were found; this was the case for *Status *and *Trust*. The native Dutch respondents have a lower score on *Status *and a higher score on *Trust, *compared with their non-native counterparts. This is consistent with the expectations one can derive from the literature on Dutch cultural characteristics. From this we can conclude that the WPI does not show cultural bias and that the scores and texts in the report represent real individual differences.

#### 5.2.4 Construct validity: The WPI and the Career Values questionnaire by Ixly

The Career Values questionnaire (CV) is a personality questionnaire that provides insight into the aspects of work that can motivate a person. The questionnaire was developed for the HRM work field and can be used in both advice and selection contexts. In advice situations, the questionnaire provides insights into what a person finds motivating in a certain job. This way, it is easier for someone to look for a suitable job. In selection contexts, the CV provides insights into the match between the career values of the candidate and the characteristics of the job the candidate is applying for. When there is a match, this tells something about the motivation a candidate has for this specific function (Orga, 2007). The questionnaire that was used in this study (CV ipsative) consists of 190 items which are presented in a forced choice format: two items are presented opposed to each other, with a four point scale in between. The candidate has to indicate which of the two items applies more to him/her. The questionnaire consists of 20 scales. For a description of the scales, see Appendix 14.

This study also contributes to the criterium validity of the WPI. Although the CV is not what the WPI intends to measure, finding no or unsuspected relations between the two would question the criterium validity of the WPI.

The CV and the WPI are complementary during assessment procedures, because the WPI concerns the personality of the candidate while the CV concerns the personal preferences and values in terms of the candidate’s professional career. From experience we know that when someone scores high on a certain scale of the WPI, this does not necessarily have to mean that someone values this specific characteristic in a job. It is interesting to investigate how the results on both questionnaires correlate since one can expect that there are similarities in personal characteristics and the characteristics people find important in their job. The expectation is that when someone scores high or low on a scale of the WPI, this corresponds with certain value on the CV. The reason for this is that people often find aspects of work interesting or fun because they are good at it or because it they with their personality.

Data was available of 1329 people who took both the WPI and the CV questionnaires. All the information came from the Ixly’s dataset and was retrieved at different companies. The age of 761 people from the dataset was known, varying from 16 to 62 years old, with a mean age of 35,4. Of all the people of which the gender was known 539 were male and 669 female. The questionnaire was taken by 1139 people in an advice context, while 190 people took the questionnaire in a selection context. The questionnaires were completed during the beginning of 2004 and mid-2006.

An overview of the mean sten score on the factors of the WPV and their standard deviations are reported in Table 5.24. The mean sten scores and the standard deviations are very similar to the norm group.

Table 5.24. Mean sten scores and corresponding standard deviations (N = 1329) | ||

Factor | Mean sten score | Mean standard deviation |

Influence | 5 | 2.2 |

Sociability | 5.2 | 2.2 |

Exuberance | 5 | 2.2 |

Structure | 5.3 | 2.5 |

Stability | 5 | 2.2 |

Correlations were calculated between the factors and scales of the WPI and the scales of the CV. The reliabilities of the scales of the CV in this sample were between .65 and .89 (Cronbach’s alpha). The reliabilities of the scales of the WPI were between .83 and .94 (α) and the reliabilities of the factors of the WPI were between .96 and .97 (stratified alpha). In calculating the reliability of the factors we have assumed that the total error variance was the same as in the norm group.

The correlations between the factors of the WPI and the CV are reported in Table 5.25. The correlations larger than .30 and smaller than -.30 are in bold. In studies like this, correlations that are larger than .30 (absolute) are considered as of average size and correlation larger than .50 of large size.

Table 5.25. Correlations between the factors of the WPI and the scales of the CV | |||||

CV-scales (N =1329) | WPI-factors (N = 1329) | ||||

Influence | Sociability | Exuberance | Structure | Stability | |

Balance private life – work | -.23** | -.03 | -.23** | .12** | -.22** |

Financial reward | .27** | -.03 | .05 | .09** | -.05 |

Praise and recognition | .14** | .16** | .13** | .16** | -.13** |

Tangible results | .21** | .15** | .37** | .31** | .11** |

Useful contribution | .03 | .19** | .14** | .12** | .01 |

Career | .68** | .08** | .40** | .04 | .17** |

Quality | .24** | .11** | .39** | .45** | .15** |

Creative thinking | .37** | .19** | .56** | -.07* | .19** |

Influencing | .67** | .26** | .43** | -.10** | .25** |

Enterprising | .50** | .25** | .44** | -.13** | .24** |

Helping others | -.06* | .39** | .13** | .23** | .06* |

Analysing | .32** | .18** | .53** | .14** | .25** |

Developing | .30** | .24** | .53** | .10** | .23** |

Being physically active | -.06* | -.00 | -.03 | .03 | -.05 |

Autonomy | .28** | -.03 | .27** | -.28** | -.03 |

Security and stability | -.03 | .03 | -.05 | .44** | -.05 |

Challenging tasks | .36** | .24** | .60** | -.02 | .33** |

Cooperation | .16** | .50** | .23** | .24** | .23** |

Attention | .63** | .21** | .23** | -.13** | .05 |

Hectic situations | .26** | .28** | .46** | .15** | .23** |

** Significant at p < .01 (2-sided)* Significant at p < .05 (2-sided) |

The factor Influence shows average (.30) to large (>.50) correlations with the following career values: *Attention*, *Influencing*, *Creative thinking*, *Enterprising*, *Developing*, *Career*, *Analysing* and *Challenging* tasks. People that score highly on this factor find these career values very important. All of these career values are, in a logical and explainable way, consistent with scoring high on the Influence factor of the WPI.

The factor Sociability shows average to large correlations with the following career values: *Helping others* and Cooperation. Social people often find it important to be able to help others in their job and social people tend to value the cooperative aspects of work as well.

The factor Exuberance shows average to large correlations with the career values: Influence, Creative thinking, Dynamics, Entrepreneurship, Development, Quality, Career, Analysing, Tangible results and Security and stability. All these career values can be linked, again, in an interpretable way with the factor Exuberance. When we look at scales this factor comprises, it is easy to see that scales such as Self-development, Adaptability and Originality fit well with these career values.

The factor Structure shows average to large correlations with the following career values: Quality, Tangible results and Security and stability. Structured people like to deliver quality and specific, tangible results.

The factor Stability only shows an average relation with the career value Task challenge. These people are stable and therefore need a challenge in their job. The fact that this factor does not correlate with the career value Security and stability can be explained by the reasoning that people who are stable do not need this in their job. The factor Stability of the WPI concerns the (emotional) stability of a person while the career value Security and stability mainly concerns the rewards and the materialistic (monetary) aspects of a job (career). Thus, to what extent are you stable, and to what extent do you need stability, respectively.

The table with all correlations between the WPI and the scale scores of the CV is included in Appendix 15. Below in Table 5.26, an overview is provided of the correlations between the scales of the WPI and the CV that are at least .30. The career values with the highest correlations with the corresponding scales of the WPI are reported first.

Table 5.26 Relationships between the scales of the WPI and CV. | ||

Career Values | ||

WPI scale | Positive relation | Negative relation |

Status | Career, Attention, Influencing, Enterprising | Balance private life – work, Helping others, Security and stability, Analysing, Being physically active |

Dominance | Influencing, Enterprising | Security and stability, Balance private life – work |

Competition | Career | Balance private life – work, Helping others |

Self-presentation | Attention, Influencing, Career, Enterprising | Security and stability, Balance private life – work. |

Leisure contact | Cooperation | |

Attentiveness | Helping others | |

Energy | Hectic situations, Career | Balance private life – work, Security and stability |

Personal growth | Developing, Career, Challenging tasks | Being physically active |

Perseverance | Quality | |

Adaptability | Challenging tasks, Developing, Creative thinking | Security and stability, Balance private life – work |

Originality | Creative thinking | Security and stability |

Independence | Autonomy | Cooperation, Helping others |

Precision | Quality | Enterprising, Attention, Autonomy, Influencing |

Regularity | Security and stability, Balance private life – work, Quality | Enterprising, Influencing, Challenging tasks, Creative thinking |

Conformity | Autonomy | |

Decisiveness | Quality | |

Self-confidence | Influencing | Security and stability |

Optimism | Security and stability |

The scales *Self-disclosure*, *Trust*, *Friendliness*, *Orderliness*, *Optimism*, *Frustration-tolerance* and *Resilience* are not shown in Table 5.26 because they did not show an average (>.30) or high (>.50) relation with one of the career values. The associations between the CV-scales and WPI-scales are not difficult to explain. Interestingly, *Security and stability* and *Balance private life – work* both show mostly negative correlations with the WPI-scales, while the WPI-scales mostly show positive correlations with *Enterprising*. The WPI is a work-related personality questionnaire and people who find enterprising important in their career, appear to have less of a need for security and stability and a well-balanced private and work life.

The research described here strongly contributes to the construct validity of the WPI. All the relationships found were either expected or easy to explain.

## 5.3. Concluding remarks on the validity of the WPI

In the research on the validity of the WPI, a distinction was made between predictive validity and construct validity.

**Predictive validity**

Research focusing on concurrent validity shows that the factors and the majority of the scales of the WPI show high and theoretically justifiable correlations with the factors of the FFPI. The average correlation between the respective factors is .650

Research on student nurses shows that particularly the Sociability and Stability factors (and their underlying scales) are predictive of the academic performance in nursing school. Research at an employment agency has shown that the WPI has predictive power in this context as well. Different criteria of job performance were predicted by scores on the WPI. These two studies show the predictive validity of the WPI in two very different populations.

**Construct validity**

Research on the internal structure showed that the Multiple Group Method indicated that the questionnaire consists of relatively independent, homogeneous, reliable and stable scales. At the factor level, the reliability is high to very high (>.95). The stability of the scales is .87 on average.

Research on different background characteristics and their relations with WPI scores shows that there only small differences in scores between different subpopulations. Furthermore, we believe that the small differences found reflect real and relevant differences which are a good reflection of the reality. If, for example, different norm groups would be formed for men and women separately, then a male with a low score for Attentiveness would not receive a very large sten score. While in reality, we are still dealing with an inattentive person. Because of this we have chosen to use one single norm group for the different subpopulations.

Finally, a study was conducted investigating the relations between the WPI and the Career Values questionnaire. This study showed that all factors of the WPI had an average to strong relation with one or more scales of the CV questionnaire. Similarly, at the scale level, a number of theoretically justifiable correlations were found between the WPI and the CV. These results also support the construct validity of the WPI.

[1] The problem of the four missing items was solved by repeating the standardization process for the scales and factors to which these items belonged, using the same norm population on which the official norms were based. The correlations between the scales and factors were .97 and .99, which means that the differences were negligible.