Explainability for end users

«Explainability» means different things to different recipients — and requires different things:

Recipient	Wants to know	A suitable form
The person affected	why did it turn out this way for me, and what can I do?	a counterfactual: «with three months more employment the answer would have been yes»
The case officer	can I trust this case?	the decisive factors + the confidence + similar cases
The developer	where does it go wrong?	SHAP, error analysis, layer profiles
The auditor/regulator	is the system fair and documented?	a model card, an evaluation per group, logs

Counterfactual explanations are usually best for end users: they are concrete, actionable and do not require understanding the model.

The law in brief. GDPR Article 22 gives a right not to be subject to solely automated decision-making with legal or similarly significant effects, and Articles 13–15 require «meaningful information about the logic» behind such decisions. Exactly how far that stretches is debated, but the direction is clear: automated decisions with significant consequences require human involvement and an intelligible justification.

The EU AI Act sharpens this for high-risk systems (including education, recruitment and credit) with requirements on documentation, human oversight and logging.

Three traps to avoid:

1. Explanations that instil false confidence. A neat saliency map makes the user trust a wrong decision more. The explanation has to be faithful, otherwise it does harm.
2. Explanations that cannot be acted on. «Your application was rejected because of variable x₄₇» helps nobody.
3. Explanations that reveal too much. Exact thresholds in a fraud system are an instruction manual for anyone wanting to get around it.

For AI-grafen that means concretely: when the platform says «you need to revise gradient descent first», the justification should be the actual cause (the mastery value and the prerequisite chain), not a post-hoc construction — and the learner should be able to see their own measurements.

import numpy as np

def counterfactual(model, x, features, direction=1, steps=40):
    """The smallest change in ONE feature that flips the decision. A simple but useful variant."""
    base = model.predict_proba([x])[0, 1]
    suggestions = []
    for i, name in enumerate(features):
        for delta in np.linspace(0, 3 * np.std(X_tr[:, i]), steps)[1:]:
            xp = x.copy(); xp[i] += direction * delta
            if (model.predict_proba([xp])[0, 1] > 0.5) != (base > 0.5):
                suggestions.append({"feature": name, "change": round(float(direction * delta), 2),
                                    "from": round(float(x[i]), 2), "to": round(float(xp[i]), 2)})
                break
    return sorted(suggestions, key=lambda f: abs(f["change"]))[:3]

for f in counterfactual(model, x_application, FEATURES):
    print(f"If {f['feature']} were {f['to']} instead of {f['from']}, the decision would have been different.")

The phrasing matters: concrete, in the user's language, and actionable — not «the SHAP value for feature 12 is −0.43».