Consistency, Self image and Response tendency have been added to the Work-related Personality Questionnaire (WPV) report in order to provide greater insight into the way in which the questions were answered in the report. This article describes these metrics and how they can be interpreted.
Let’s assume you have requested a respondent to complete a personality questionnaire. How sure are you that the profile matches the respondent and that you have not been wrongfooted? A questionnaire can be as reliable and valid as they come, yet the exactitude of the outcome can never be absolutely guaranteed. Bias related to social desirability, response tendency and consistency is always just around the corner. Although research shows that the chance of a distorted profile is relatively small, misrepresentations can occur. How can you be assured of profile accuracy? In order to facilitate this, we have added three metrics to the Work-related Personality Questionnaire report. These metrics reveal the pattern behind the given answers.
Social desirability and personality
During the test interpretation workshops that I moderate, I am often asked if a Work-related Personality Questionnaire is corrected for socially-desirable answers. I reply by saying that this is not necessary because the Work-related Personality Questionnaire contains more than one norm group. In advisory situations, there is little reason for answering in a socially-desirable manner – and it rarely happens. In this case, it is important for the respondent to acquire as objective and reliable an impression of the personality as possible. In selection situations, the motive is different because the respondent is not trying to assess how he or she ‘fits together’. In this case, the motive is to bring across that his or her personality is a perfect fit with the organisational requirement. By making use of a selection norm group, socially-desirable answers are automatically corrected. You know with almost absolute certainty that socially-desirable answers are given in a selection situation.
The ‘information’ behind the reply pattern
What is interesting is to look at a respondent’s reply pattern. Is a respondent replying in a consistent manner or are the answers to two items with a high correlation value far apart? Is the respondent replying in a critical fashion, or does he think highly of himself? And does the respondent make use of extreme reply options of does he steer towards the middle of the road? In order to answer these questions, we have added three metrics to the Work-related Personality Questionnaire: Consistency, Self image and Response tendency. These metrics enable you to access the information behind the reply pattern. Permit me to run you through the metrics.
The name says it all. This metric indicates how consistently the respondent has completed the questionnaire. This helps us to assess whether we can take this profile seriously or if we need to take a more critical look. A very low consistency score suggests that answers have been provided at random. In the norm group, we went in search of item pairs with a high correlation factor. We looked at item pairs within one’s own scale or factor as well as item pairs outside one’s own scale or factor. We found a total of 48 item pairs with a high positive or negative correlation factor. The respondent’s deviation with respect to these item pairs is then compared to the norm group. In the report, a high score in the tables implies a consistent reply pattern. A low score means that the questionnaire has been completed inconsistently.
Suggestions for discussing Consistency
With very low scores of three or under, the results should be interpreted with all due caution. There is a chance that the respondent has answered at random. During the discussion of the report, you can always ask how the respondent filled in the questionnaire. An indication could be the time he took to do so; the average completion time is between 30 and 40 minutes. It can also be worthwhile to ascertain if the respondent fully understood the questions. A final tip is to ask the respondent to describe himself with the help of a number of personality traits. You can then see if his description is in line with the report results.
This metric indicates the degree to which a respondent demonstrates a critical or positive self image. A low score suggests a very self-critical attitude, a high score a positive self image. The score is compiled by looking at the candidate’s competence scores. If the respondent is only shown above-average scores on the competence indicator of the Work-related Personality Questionnaire, then this contributes to a high score and a positive self image. A low score on the competence indicator contributes to a lower score and a more critical self image.
Distortion of Self image
Self image can often be spot on, because positive or extravert personalities tend to complete the questionnaire with a certain zest. They choose extremes more easily than more modest people with a more self-critical attitude. It is also true that the way in which the personality profile is filled in can be too critical or positive. Someone with a critical self image can sometimes be too critical of himself and subsequently score lower on the competence indicator. A highly-successful manager who is a bit of a perfectionist had a highly critical self image – to his detriment! The competence indicator came up with scores that were too low. On the flip side I have worked with someone who scored highly due to his desire to create a positive impression of himself. The Self-image metric provides insight. Moreover, Self image in this context is not the same as Self confidence, despite the obvious correlation. The Self image score can be seen on all scales.
Tips for discussing Self image
The questionnaire can also indicate if the Self image score is in line with reality. During the debriefing it is important to reflect on extremely high or low Self image scores. It is advisable to ask for tangible examples in the case of a high score (+8). Ask the respondent for an example of a situation when things went well as well as for a situation that left room for improvement. You can also ask questions such as: what kind of feedback have you had from your manager in terms of points for improvement, what personality traits of yours actually irritate your colleagues? Ask the respondent with a low Self image score (<3) to quote situations where things went well. Ask for the positive things his manager may have said or for personality traits that colleagues appreciate. Finally, ask if the respondent recognises himself in the positive answers provided or if he thinks differently about them and, if so, why?
The reply scale of the Work-related Personality Questionnaire is a five-point Likert scale that runs from ‘Absolutely disagree’ to ‘Absolutely agree’. The response tendency indicates the deviation per reply option from the answer given by the respondent to that of the norm group. The norm group indicates how often a given reply option is chosen. The number of times each reply option is clicked during completion of the Work-related Personality Questionnaire is counted and compared to the norm group average. The Work-related Personality Questionnaire report shows the deviation as compared to the norm group per reply option in percentages. When the graphical representation shows a negative percentage, that means the respondent clicked the relevant reply option less frequently. A positive percentage indicates that the reply option was chosen more frequently than in the norm group.
In the table above, the candidate has emphasized the extreme options; the first and the fifth point has been chosen over 50% more frequently. That can lead to extreme scores in the graph. When considering the Consistency and Self image metrics, this candidate has answered the questions consistently. The Self image scores are neither extremely positive nor negative. In this case, the conclusion can be drawn that the deviating reply pattern has not led to misrepresentation in the report.
Tips for discussing Response tendency
Generally speaking, response tendency indicates whether someone dares to choose extremes or if he is wary of taking risks. Deviations per reply option of around 25% may be considered ‘normaal’. Deviations above this percentage should be reviewed with all due care. Did the respondent provide more extreme scores than the norm group, so more frequently a 1 and/or 5 option and less frequently a middle-of-the-road option? Ask him how he takes decisions and if his decisions have occasionally backfired. Or did the respondent provide safer scores, so more frequently a 3 option and less frequently a 1 and/or 5 option? Then ask him for situations in which he dared to take a risk and where he had to take a tough decision.
The inclusion of Consistency, Self image and Response tendency should make it easier for advisors to see how candidates complete the questionnaire. In cases where the scores for these metrics fall within the standard deviation, you can be sure that the respondent has completed the questionnaire in a reliable manner. That does not automatically mean, however, that a respondent with scores outside the standard deviation has been dishonest! In this case, it is important to ask the respondent to provide a little more background during the debriefing.
PS: these metrics are currently only available for the Work-related Personality Questionnaire, and not yet for the WPV Compact.