Validation is the cornerstone of accredited assessment. It’s the net that catches errors, compliance problems and shortfalls in the assessment process. A well-run validation will study the questions and activities that have been designed to produce evidence of competence and lead to the recommendation of improvements. That said, there are a number of barriers to an effective validation meeting.
1. Cognitive biases
The list of biases is long, but the awareness falls short. A bias, by its very nature, is difficult to recognise in oneself. In validation, it is essential to remain as objective as feasibly possible. This objectivity helps to pick up on grammar or wording issues, layout problems, mapping faults and more. Typical biases one can observe in a validation meeting range from confirmation bias, and framing effects, to the Dunning-Kruger effect. As long as humans are involved in the process, biases will feature in decision-making, recommendations and observations. The only solution in the moment, is to be vigilant and aware that you carry with you a number of biases that can rear their unwanted heads at any moment. Having multiple people looking over the material goes some way to reducing these effects.
2. Failing to understand or effectively apply benchmark mapping
Mapping assessment activities and questions to the various criteria of a benchmark, such as a unit of competency, poses a number of difficulties depending on when the mapping occurs. If mapping is done carefully during the creation of the assessment tools, then generally the validation process is made much easier. But attempting to map existing materials to a benchmark introduces a number of possible mistakes. Firstly, asking if a question meets a particular criterion is fraught with subjectivity. Generally speaking, a single question will rarely capture all of the required evidence for a performance criterion or knowledge evidence statement. Similarly, just because the question has similar key words to the performance criterion, does not mean it maps perfectly to that benchmark. Finally, ensuring that the performance evidence is satisfactorily captured will require unbiased reasoning on behalf of the validators. The input of a subject matter expert will assist with this, as will a clear understanding of the benchmarks performance standards.
3. Groupthink
Although officially a cognitive bias, groupthink deserves its own heading here, as there is a likely tendency for individuals to follow the decisions of a senior or more experienced validator. This can hamper a group’s ability to fact-check one another and ensure objectivity is followed.
4. Allocation of sufficient time
How long should a validation meeting last? The answer: As long as it needs to. Allocating a specific time to end the meeting can and often does result in attendees rushing as the clock draws close to the allocated hour. Obviously, rushing the scrutiny of assessment tools will raise the risk of errors and missing key issues with the materials. Rushing can also lead to groupthink and other biases in order to finish in the time provided. To avoid this common problem, time should be allocated based on the complexity of the task. If you are reviewing a commonly used assessment of a single unit, then an hour is probably sufficient, but a newly designed tool for a skill set may take all day (including making changes to the materials). It is believed that mornings are best for critical and lateral thinking tasks as individuals have slept and had breakfast – and are therefore in the neurologically best condition. But this can change depending on many factors, so it’s best left to the group to decide on the time of the day they choose to conduct the validation.
Note from the author:
While not an exhaustive list of ‘what can go wrong’ in a validation meeting, these are certainly the more common errors I observe. Having witnessed many validation meetings over the last 15 years, I’ve found these tendencies and mistakes to be the most common – and I include myself in this observation. To avoid the errors and conduct efficient and effective validation meetings, it is good to be aware of the barriers that may lurk under the surface of an otherwise rosy meeting. Good luck!
