Speculative and provocative by design, in this section we look into the future at what AI capabilities might lift the quality of assessment in industry. We focus on two future paths, the first is custom-trained and multi-modal models. The second is neuro-symbolic AI, and approach that integrates LLMs with formal rule based systems where the strengths of LLMs and Symbolic AI are drawn on, which we describe with examples from coaching and assessment feedback.
Custom, multi-modal LLMs
Assessment businesses compete by differentiating or cost minimising. Cost minimising is unattractive because high margins are available if you can differentiate. However, the commercial landscape of assessment is dominated by marketing led competitive strategies, claims of difference that don’t withstand scrutiny. To get high margins, vendors need to differentiate from competition.
How can a vendor differentiate for strong value capture in the age of AI in the next 5 years? One approach that will not work is integration of off-the-shelf LLMs into existing workflows. This leads to cost minimization, everyone has access to the same models you do. An assessment vendor doing well with this approach to AI is benefitting from pre-existing conditions, not their A.I. strategy. Their advantages will likely be threatened by companies with more sophisticated A.I. strategies.
To capture markets and value, vendors need to get deeper into custom transformer models which will need to be multimodal (e.g., voice, audio, text). Like all innovation, this comes with risks that need management. Firms need to navigate the utility trade-offs between building their own ground-up models, fine tuning existing models on proprietary data, and using big commercial models off-the-shelf (ideally as big as possible using efficient methods such as LoRA and QLoRA). Each approach has different worker skill and computing requirements.
Be sceptical of vendors claiming they’ve succeeded. Such claims are often marketing led differentiation. Open access, reproducible, peer review remains best, albeit flawed, verification process. In summary, marketing led differentiation is hollow and vendors should pursue technical depth and transparency. Others will emulate you, so you’ll need to keep innovating, but you have first mover advantage.
Neuro-symbolic AI
Along with the incredible capabilities of LLMs come some frustrating features like inappropriate advice and information that’s incorrect. They’re too probabilistic for high stakes domains. We can put guardrails around the LLMs by giving them instructions to operate within constraints. But these instructions are interpreted within the same framework as the challenge we need to resolve, so they don’t guarantee success.
Even retrieval augmented generation (RAG) systems, which search to provide context for more precise LLM generation, do not involve explicitly reasoning with logic. However, the transformer architectures that underpin LLMs only represent one of two principal branches of AI. LLMs represent the neural network approach while symbolic AI, which uses rules and knowledge, is the second (Garcez et al., 2023; Marcus, 2022, Sheth et al., 2023).
While LLMs learn patterns during training, symbolic AI represents known concepts as symbols and reasons about them using knowledge, logic, and decision rules. AI excels at reasoning while scaling LLMs has not resolved their reasoning errors. To address this, the future of AI assessment may blend the neural capabilities of transformers with the reasoning capabilities of symbolic AI, so called neuro-symbolic AI.
What might neuro-symbolic AI assessment look like? Let’s consider two examples, a coaching conversation where an LLM system draws on a symbolic AI system for a conversation that reflects formalised rules, and another example where a symbolic AI system draws on LLMs for a scientifically informed conversation in the context of personality testing and feedback.
Coaching example
Taking the coaching example first, consider the topic of negotiation, an area where there is clear theoretical advice about what to do in a given situation. This advice can be formalized as a knowledge base of if x then y conditions that the learners must acquire. Simulations that follow these rules can then be designed in a symbolic AI system, which can in turn draw on the LLM solely for realistic peripheral conversation around the coded rules of negotiation.
Assessment example
Consider a reverse setup where the LLM leads the symbolic AI system. We start by passing the system candidate assessment results and have the LLM generate an initial interpretation, interpreting the profile against the job role. A symbolic component can then validate insights against a set of rules. It may flag that high agreeableness and a competitive environment are a mismatch and prevent a positive LLM candidate evaluation.
References
Garcez, A. D. A., & Lamb, L. C. (2023). Neurosymbolic ai: The 3 rd wave. Artificial Intelligence Review, 56(11), 12387-12406.
Marcus, G. (2022). Deep learning is hitting a wall. Nautilus, 10, 2022.
Sheth, A., Roy, K., & Gaur, M. (2023). Neurosymbolic AI--Why, What, and How. arXiv preprint arXiv:2305.00813.
Next page
Last page
Historical roots of AI assessment debates
Return home
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0).
