How to Evaluate Whether an AI Tutor Is Helping

A reproducible way to evaluate whether an AI tutor is helping, using baselines, comparison groups, honest attribution and clear stop-or-scale thresholds.

2026-03-05 By DoviLearn® Insights Team

Featured image for How to Evaluate Whether an AI Tutor Is Helping

An AI tutor can feel useful long before anyone can show that it is. Learners engage, usage climbs, and the dashboard looks healthy, yet none of that proves that people are learning more than they would have otherwise. This guide sets out a reproducible way to answer a narrower question: is the tutor actually helping? It describes a baseline-to-outcome measurement plan with leading indicators, honest limits on what you can attribute, and clear thresholds for when to stop or scale.

Start with a baseline, not a launch

Measurement begins before the tutor does. Record where learners stand now on the outcomes you care about, who they are, and how you will define success, so that later numbers have something to be compared against. Keep the human role explicit while you do this. The report recommends keeping humans in the loop and emphasises that AI should support, not replace, the judgement of educators. [c-usedai] A baseline that captures prior attainment and context is what separates a real effect from a hopeful story.

Choose outcomes and a denominator

Decide what counts as learning for this cohort and state the denominator before you collect anything. If completion is the measure, completion out of how many enrolled? If it is assessment gain, measured how and marked by whom? Writing the denominator down stops the quiet habit of reporting only the learners who stayed, which flatters almost any tool.

Editorial illustration representing the article subject. ⚠ AI-generated editorial illustration. It represents the article’s subject and does not document a real event or identify real people.

AI-generated editorial illustration. It represents the article’s subject and does not document a real event or identify real people.

Separate leading indicators from learning

Leading indicators such as time on task, questions asked, or return visits are useful early signals, but they are not learning. Treat them as a prompt to look closer, never as proof. A tutor can raise engagement and still leave attainment untouched, which is exactly why engagement and learning need separate lines on the page.

Be honest about attribution

Most institutions cannot run a clean randomised trial, and that is fine as long as you are candid about it. A simple before-and-after with a comparison group is far stronger than before-and-after alone. The What Works Clearinghouse reviews and summarises the quality of education research and sets standards for study design, comparison groups and attrition. [c-wwc] Borrowing that discipline, name your comparison and report how many learners you lost along the way.

Set stop-or-scale thresholds in advance

Decide now what result would make you scale the tutor, hold it steady, or switch it off. Thresholds agreed before the data arrives protect you from reading whatever the numbers happen to say as encouraging. Pair each threshold with a privacy check, since UNESCO frames generative AI in education around a human-centred approach with data protection built in. UNESCO calls for a human-centred approach to generative AI in education, with data-privacy protection and age limits among its key steps. [c-unescogen]

A worked evaluation plan

Here is the plan filled in. Baseline: a short diagnostic for all enrolled learners. Outcome: assessment gain at eight weeks, plus completion out of the full cohort. Leading indicators: weekly active use and help requests, watched but not scored. Comparison: a parallel group taught without the tutor. Threshold: scale only if the tutor group shows a meaningful gain over the comparison with acceptable attrition in both.

Where DoviLearn® fits

A DoviLearn® Specialist AI Tutor journey can be put through exactly this plan: you can question its measurement design, configure the baseline and thresholds, and evaluate it against a comparison group for the query of whether the tutor is helping. Explore it at the DoviLearn® AI Tutor . Judgement about which outcomes matter and which learners need protection stays with educators and the organisation, and the example shows a process rather than a guaranteed learning, employment or business outcome.