3. Norm development
The WPI uses norm groups for the interpretation of the scores. This means that the scores of a candidate are being compared with the scores of a reference population. The reference population for the WPI is a representation of the work force of the Netherlands. By means of weights we have made sure that the sample population matches the Dutch work force in terms of education, age, gender and working situation as close as possible. For information on the work force we have used data from 2006, from the Central Bureau of Statistics (CBS, 2007). The questionnaire has two norm groups, namely Selection and Advice. In this chapter, we will discuss how the norm groups are formed and we describe the standardization procedure.
3.1 Norm groups
The WPI is used in two different testing contexts, namely in selection and advice situations. Advice situations include all forms of career counselling, consultancy and coaching. Candidates who take the WPI in these kinds of situations will come to different, most likely less socially desirable scores than candidates that take the test in selection situations. With selection situations we mean, for example, assessments in job application procedures. In these situations, people will be more prone to give socially desirable answers because they will think this will get them through the selection. For both situations, independently from each other, but in the same manner, norms were calculated.
The data over which the standardization procedure was run, was collected between 01-03-2004 and 29-12-2006 for the Advice group and between 22-01-2004 and 03-01-2007 for the Selection group. Data was collected for 5629 people in the Advice group, of which 47.9% was male and 52.1% was female. The average age of this group is 36 with a minimum of 15 and a maximum of 62. For the Selection group, information was available on 1514 people, 38.8% male and 61.2% female. The average age of this group is 32 with a minimum of 17 and a maximum of 59. Education levels for both groups vary from lower education to academic education.
The two research groups were weighed for four variables, in such a way that they resembled the Dutch work force in terms of these four variables. In Table 3.1 these variables are described.
|Table 3.1. Description of the background variables used in the weighing procedure|
|Education||The education levels in which the work force is divided are: lower education, secondary education and higher education. The education programs that fall under each of the levels are presented in Appendix 5.|
|Age||The age variable is divided up into three categories. The work force falls within the age range of 15 to 65 years. The three age groups are 15-24 years old, 25-44 years and 45-65.|
|Job status||A distinction is made between the employed and the registered unemployed. The data on the latter group is obtained via the UWV, where job seekers took the WPI.|
All combinations of the four variables result in 36 cells. The weight factors are chosen in such a matter that the distribution of the 36 cells corresponds as much as possible with the relative proportions in the Dutch population (See Appendix 6 for details on this procedure).
The weighted Advice- and Selection group form the norm groups for respectively the Advice- and Selection situation. In Table 3.2 the distribution in terms of the background characteristics are shown for both research groups, for the weighted norm groups and for the Dutch work force.
|Table 3.2. Frequencies of background characteristics in both norm groups|
From Table 3.2 it becomes clear that in both norm groups, especially for the Advice group, the background characteristic working situation has had a large influence in the weighing procedure. In the data on which the norms have been formed there were relatively more women than men, while in the work force this is exactly the opposite. In addition, in data of the Selection group, there was a relatively large group of higher educated people.
The data was collected at 118 different companies from all over the Netherlands. For the Advice norm group, information was gathered from 2775 people. For the Selection norm group, information was gathered for 2708 people. In Table 3.3 the number of persons from every Dutch province are provided for both norm groups. The real distribution of persons over the Dutch provinces is provided as well, as measured by the CBS in 2011.
|Table 3.3. Distribution of regions in both norm groups|
|Advice norm group||Selection norm group||CBS 2011*|
|Unknown / will not say||25||.9||95||3.51||–|
|* CBS Key figures Work force 4th quarter of 2011, requested in 2012|
Distribution of work sectors in norm population
The background characteristics of a sample of 1786 respondents from 118 companies, operating in different work sectors, were successfully retrieved. Because the sample was drawn randomly, it can be rightfully assumed to be a representative sample of the reference population.
A sample of 1786 respondents was drawn from the same companies that provided the respondents on which the norm groups were based. Of these companies the work sector, ranging over 12 different sectors, was retrieved.
Because the sample was drawn randomly, it can be rightfully assumed to be a representative sample of the Dutch work force. Since the same companies were used, it can be reasonably assumed that the distribution of the companies in sectors is the same as in the norm group.
|Table 3.4. Distribution of work sectors in both norm groups|
|Advice||Health, Wellbeing and Personal Care||443||23%|
|Public administration, Safety and Law||284||15%|
|Automation and ICT||201||11%|
|Engineering and Production||194||10%|
|Commerce and Administration||131||7%|
|“Do not know”||88||5%|
|Education, Culture and Science||69||4%|
|Agriculture, Culture and Environment||44||2%|
|Catering and Housekeeping||41||2%|
|Storage and Transport||33||2%|
|“Prefer not to answer”||23||1%|
|Language, Media and Communication||22||1%|
|Tourism and Recreation||7||1%|
|Selection||Health, Wellbeing and Personal Care||741||34%|
|Public administration, Safety and Law||511||23%|
|Engineering and Production||175||8%|
|“Do not know”||125||6%|
|Commerce and Administration||79||4%|
|Education, Culture and Science||70||3%|
|Automation and ICT||46||2%|
|”Prefer not to answer”||28||1%|
|Language, Media and Communication||26||1%|
|Catering and Housekeeping||24||1%|
|Storage and Transport||21||1%|
|Agriculture, Culture and Environment||17||1%|
|Tourism and Recreation||7||1%|
In Table 3.4, the number of respondents in every work sector are represented. In the advice situation, 111 respondents of the total 1894 responded with “I do not know” or “Prefer not to answer”. In the selection situation this was the case for 153 of the 2185 respondents. All the work sectors are represented in the sample.
The sectors ‘Health, Wellbeing and Personal care’, ‘Public administration, Safety and Law’ and ‘Business services’ contain a lot of people not because they are overrepresented, but because a relatively large number of professions fall within these categories. The distribution of people in different sectors is representative of the work force.
Our norm groups for the selection and advice situation are a good representation of the Dutch work force in terms of education, age, gender, work situation and work sector. This, in addition to the size of the norm groups contributes to the reliable standardization of the WPI.
3.2 Establishing norm groups
For each scale and for both weighed groups the raw scale scores are transformed into stenscores. These stenscores are normally distributed scores on a scale ranging from 1 to 10. The theoretical mean is 5.5 with a standard deviation of 2, the deviation of the true scores is fixed to 2. The scale is symmetrical and normally distributed. Standard scores, to which stenscores belong, provide insight in the way a certain scores relate to the mean of all scores. How the stenscores are interpreted will be discussed in the chapter “Application, interpretation and use’.
The stenscores are not directly calculated discretely, but as interval scores and reported discretely. The factor scores are being calculated by summing the standardized scale scores, after which this sum is in turn transformed into stenscores as well. The procedure that is used to transform the raw scale scores into stenscores is described in van der Woud (2007), see Appendix 7.
Standard scores and likelihood intervals
In Appendix 8 the norm tables of the 25 scales and 5 factors are shown. In these tables, for every raw score the corresponding standard score, norm score and latent score is given. In addition, for the norm scores a confidence interval is given and for the latent score a probability interval. In order to avoid confusion, both concepts are discussed in more detail next.
|Clarification of statistical terms (COTAN, 2010)|
|True score||The true score is not observable and is estimated simply by using the observed score X.|
|Standard error of measurement||The standard error of measurement (SE) is estimated by the standard deviation (σx) and the reliability of the scale (rxx). It is assumed that measurement errors are normally distributed. Standard errors of measurement are used to estimate a reliability interval for the true score .|
|Reliability interval||The reliability interval (RI) is situated symmetrically around X and is estimated by the standard error of measurement and a corresponding level of reliability. The value of 1.96 is used for the 95% interval, 1.28 for the 80% interval. The RI indicates the precision of a measurement and can be used to test hypotheses on someone’s true score.|
|True score||In this case, the true score is estimated by means of linear regression. From the formula, it can be derived that the mean μx influences the true score . The higher the reliability of a scale, the larger the share of an individual’s score X on the estimate . With a low reliability, μx will have a larger share in .|
|Standard estimation error||The standard estimation error indicates the variance of the true score. As can be derived from the formulas, the standard estimation error is times smaller than the standard error of measurement.|
|Likelihood interval||The standard estimation error produces a likelihood interval (LI) which is symmetrically distributed around . The LI is important when one want to get an estimation of the level of the measured variable, while taking the reliability of the variable into account..|
η2 = Spredictor / Stotal
|Indicates the explained variance in the sample by the model. η2 usually overestimates the explained variance, but in larger samples this overestimation disappears.|
For both norm groups the average stenscore of every factor is shown in Table 3.5, before and after weighing. The average stenscores of every factor for the categories of the different background variables are given as well.
|Table 3.5. Average stenscore of the factors and background variables|
|Influence||5.0 (2.2)||5.5 (2.2)||5.6 (2.6)||5.5 (2.2)|
|Sociability||5.4 (2.3)||5.5 (2.2)||5.6 (2.2)||5.5 (2.2)|
|Exuberance||5.3 (2.2)||5.5 (2.2)||5.4 (2.2)||5.5 (2.2)|
|Structure||5.8 (2.6)||5.5 (2.4)||5.4 (2.4)||5.5 (2.4)|
|Stability||5.2 (2.2)||5.5 (2.2)||5.2 (2.2)||5.5 (2.2)|
|Total score||5.3 (2.3)||5.5 (2.3)||5.4 (2.2)||5.5 (2.3)|
|*Stenscore with the corresponding standard deviation in brackets.|
The standardization of the WPI has taken place on the basis of a weighted norm group. The norm groups are a good representation of the work force in terms of education, age, work situation and work sector. In Appendix 8, for every scale and for both norm groups a detailed overview is given of the raw scores and their corresponding stenscores. Thanks to the large norm groups and the weighed distribution, the standardization of the WPI is very solid.
 At the time of research, the UWV was known as the Centre for Work and Income (CWI).