Assessment businesses compete by differentiating or cost minimising. Cost minimising is unattractive because high margins are available if you can differentiate. However, the commercial landscape of assessment is dominated by marketing led competitive strategies, claims of difference that don’t withstand scrutiny. (SHL’s remarkable integration of Thurstonian IRT into its OPQ is an exception). To get high margins, vendors need to differentiate from competition.
How can a vendor differentiate for strong value capture in the age of AI in the next 5 years? One approach that will not work is integration of off-the-shelf LLMs into existing workflows. This leads to cost minimization, everyone has access to the same models you do. An assessment vendor doing well with this approach to AI is benefitting from pre-existing conditions, not their A.I. strategy. Their advantages will likely be threatened by companies with more sophisticated A.I. strategies.
To capture markets and value, vendors need to get deeper into custom transformer models which will need to be multimodal. Like all innovation, this comes with risks that need management. Firms need to navigate the utility trade-offs between building their own ground-up models, fine tuning existing models on proprietary data, and using big commercial models off-the-shelf (ideally as big as possible using efficient methods such as LoRA and QLoRA). Each approach has different worker skill and computing requirements.
Don’t trust vendors claiming they’ve succeeded, be wary of marketing led differentiation. Open access, reproducible, peer review remains best albeit flawed verification process (again, SHL excelled here with their technical work on Thurstonian IRT). In summary, marketing led differentiation is hollow and vendors should pursue technical depth and transparency. Others will emulate you, so you’ll need to keep innovating, but you have first mover advantage.
Next page
Last page
Historical roots of AI assessment debates
Return home
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0).
