AI systems are increasingly making decisions that were previously reserved for professionals. In the financial sector, in the judiciary, in healthcare. Everywhere, the same question arises: on what basis was an AI decision made? That question is not new, but the urgency is growing. Because a decision that no one can explain is hard to defend.
The European AI Act, which is now being phased in, formalises this explainability: depending on the risk level, transparency and human oversight become legal requirements.
Why explainability is not a luxury
Explainability affects everyone in the chain. The employee wants to understand why certain choices are being made. Why is a consultation with the occupational physician being scheduled? Or why not? The employer asks for substantiation when advice deviates from expectations. And the occupational physician wants to know whether the system reasons the way they would reason — and why it does or why it does not.
Explainability is not a technical detail to leave to the IT department. It goes to the heart of what it means to bear responsibility in an era where machines increasingly co-decide.
Two families of AI, two ways of explaining
Not all AI works the same way. In practice, you encounter two main types.
The first: classical machine learning models. These systems work with fixed variables. They receive structured data as input. Variables such as age, type of complaint, and number of working hours per week are used to predict a score or category.
The second: large language models (LLMs). These systems process unstructured text. They can summarise conversations, answer questions, or formulate advice based on context.
Both can be valuable. But they explain their decisions in fundamentally different ways.
Classification models and SHAP: calculating with factors
A classical machine learning model works like a set of scales. It weighs factors against each other and arrives at an outcome. The question is: which factors weigh the most?
This is where SHAP comes in. SHAP stands for SHapley Additive exPlanations. The technical name matters less than what it delivers... SHAP makes visible which input variables pushed the outcome up or down.
Imagine: a case manager opens a file with a high risk score. The SHAP report shows that two factors were dominant: the duration of a previous absence period and the type of complaint. The case manager now sees not just the score, but the reasoning behind it.
This provides a foothold. In the conversation with the employee, the professional can say: "The system takes into account that you were absent for a prolonged period last year with similar complaints. That doesn't mean the same will happen now, but it is a signal to take seriously."
The strength of SHAP: it is quantifiable, can be displayed visually, and is auditable. The limitation: it explains how the model calculates, not whether that calculation aligns with reality.
Language models and chain-of-thought: reasoning in words
LLMs work differently. They have no fixed variables. They process text and generate text. That makes them flexible, but also harder to see through.
The emerging solution: chain-of-thought reasoning. Here, the model is asked to reason step by step and to make those steps explicit. Instead of just a conclusion, the model shows the path towards it.
An example: an LLM assistant reads an intake form and drafts a summary. Instead of just providing the summary, the model writes: "I see that the employee reports that the complaints started after a reorganisation. I also see that there are sleep problems. Based on this, I recommend a consultation with the occupational physician within five working days."
The professional can now follow along with the reasoning. Is the interpretation correct? Is context missing? The explanation is not a mathematical formula, but a narrative that can be tested.
The strength: natural language, context-sensitive, and accessible to non-technical professionals. The limitation: reasoning can sound convincing and still be incorrect. LLMs can hallucinate. The explanation therefore always requires professional review.
Ask vendors about explainability
When purchasing or evaluating AI tools, it helps to ask specifically about explainability. Not "is it explainable?" — everyone answers that with yes. But:
- How can our occupational physician explain to an employee why this score was produced?
- Which factors are shown, and in what form?
- Can we adapt the explanation to our own language and process?
The answers say a lot about the maturity of a solution.
Stefan Samba — CTO, Triagen
