May we use this AI tool? An assessment in four steps
An upper-secondary school wants to buy an AI tool that reads pupils' essays, gives feedback and proposes a grade the teacher can approve. The head teacher asks: are we allowed to? This is how the assessment goes — the same four steps the course's final project tests, without writing a line of code.
1. Start with what the tool is used for, not what it is
The EU AI Act (2024/1689) is risk-based. The requirements follow from the use, not the technology. The same language model can be minimal risk when it checks spelling and high-risk when it affects a grade. So describe the use in one sentence before reading a single article: «the tool proposes grades for pupils' essays, and the teacher sets the grade».
Then go through the levels from the top. First: is anything prohibited? Article 5 prohibits, among other things, social scoring, untargeted scraping of facial images and emotion recognition in workplaces and education. If the tool also measured pupils' engagement through the camera, that part would be prohibited — not regulated, prohibited.
2. Is the use listed in annex III? Then it is high-risk
Annex III lists areas where AI systems are high-risk: among them admission to education and evaluation of learning outcomes, recruitment and selection of staff, creditworthiness and access to public benefits. A tool that proposes grades evaluates learning outcomes. It is high-risk, even if the teacher has the final say.
If the tool only gave formative feedback that never affects the grade, the picture would change. Then article 50 remains: pupils must know they are interacting with an AI, and AI-generated content must be markable. That is limited risk — transparency requirements, not high-risk requirements. The line is drawn at the consequence for the individual.
3. GDPR applies at the same time — on other grounds
The AI Act regulates the system; GDPR regulates the personal data. A tool can comply with one and break the other. Essays, grades and logs are personal data, and the processing needs a legal basis under article 6.
Consent is the wrong basis here. A pupil cannot freely refuse a tool the school uses in teaching, and consent under an imbalance of power is not valid. For a school the basis is normally a task in the public interest. Also check purpose limitation: may the vendor use the pupils' texts to train its models? That is a new purpose, and the data processing agreement should state that it does not happen. Because the processing concerns children and systematic assessment, a data protection impact assessment (DPIA) is needed.
4. Who is responsible for what?
For a high-risk system the Act splits responsibility between two roles. The provider, who develops and sells the tool, is responsible for the system: risk management, technical documentation, a quality management system, conformity assessment and CE marking. The deployer, which is the school, is responsible for the use.
- Use the system according to the provider's instructions for use, and ask for them before buying.
- Assign teachers with the competence, time and authority to exercise human oversight.
- Keep the logs the system generates.
- Inform pupils that a high-risk system is used in decisions about them.
- As a public body: carry out a fundamental rights impact assessment before the tool goes live.
5. Where should the human sit?
A grade is significant and hard to change afterwards. The teacher should therefore make the decision with the tool as a proposal (human-in-the-loop), not supervise a system that grades on its own. But a human formally approving is not enough. If teachers approve 99% of proposals in a few seconds per essay they are not reviewing — they are rubber-stamping. Measure how often teachers deviate, and plant essays where the tool is known to be wrong to see whether the review catches them.
The decision
The school may use the tool, but as a high-risk system. Ask the provider for technical information and CE marking, carry out the DPIA and the fundamental rights impact assessment, switch off any emotion analysis, inform the pupils and make sure the teacher sets the grade with time to review. If the school wants to avoid the high-risk requirements, the way is to use the tool only for feedback that does not affect grades. The Act's requirements apply in stages, and the timetable for the high-risk requirements has been subject to proposed changes — check the current timetable on EUR-Lex before planning.