AI Japanese Tutor Progress Tracking: Metrics, Logs and Monthly Self-Tests

Why AI sessions need their own measures

Tests and levels measure what you know. AI conversation practice is about what you can do in the moment, and the two drift apart more than learners expect. A learner who passes a written test can still produce a correction every ten seconds in a role-play, and a learner with a modest vocabulary can get through a shop, a clinic and a phone call with barely a stumble. If the goal is speaking, the measures should come from speaking.

The good news is that an AI Sensei makes measurement cheap. Every session ends with a correction list, every mission has a clear end point, and the same task can be repeated exactly a month later. That repeatability is something no human conversation offers, and it is the basis of every metric below.

Corrections per minute on a repeated task

The single most useful number is how many corrections you receive per minute of your own speaking, on a task you have done before. The repetition matters. A new task always produces more corrections because you meet new vocabulary and new situations, so comparing a fresh mission with an old one tells you about the missions, not about you. Choose three anchor tasks in your first week, note the corrections and roughly how long you spoke, and rerun them once a month.

Sort the corrections into rough types when you log them: particles, verb forms, politeness, word choice, pronunciation. The total falling is good; the mix changing is more interesting. Particle errors usually fade first, politeness errors later, and word choice errors last, because they need the widest exposure. If one type is not falling after two months, that is where the next month's sessions should point.

Do not chase zero. An anchor with no corrections has stopped teaching you; replace it with a harder one.

Topics and situations completed

The second measure is coverage: how many situations can you now handle from the first greeting to the end without asking for help or switching language. This is closer to what you actually want from Japanese than any score. Keep a list of situations, tick each one the first time you finish it unaided, and add a second tick when you can finish it under harder conditions, such as a faster or less cooperative partner.

Situations come in families. Once you can check in at a hotel you are most of the way to checking in at a clinic or a gym, because the shape of the exchange is the same. Log the situations by family so you can see which are covered and which are not, and let the gaps choose the next month's missions. A learner who has ticked every shop situation and no workplace one knows exactly what to do next.

Phrases retained after a month

The third measure catches the thing sessions hide: forgetting. A phrase you used correctly in a role-play feels learned, but the true test is whether you can still produce it a month later without the context that prompted it. After each session, add two or three phrases you want to keep to the log, with the situation in English. A month later, give the Sensei the English situations and try to produce the Japanese.

Score it simply: produced correctly, produced with a correction, or not produced. Ten phrases tested monthly gives a retention figure that is easy to compare. Low retention is not a failure; it tells you the phrases are not being reused, and the fix is to bring them into phone chats and repeated missions rather than to add more.

The monthly self-test

Put the three measures together into one session on a fixed date each month. It takes about half an hour and produces one line in the log. Keep the format identical every time, because the value is in the comparison, not in any single result.

• Rerun the three anchor tasks and log corrections per minute for each
• Ask the Sensei to sort the corrections by type and note the mix
• Tick any situation completed unaided for the first time this month
• Retest ten phrases from last month's log and score them
• Hold a five-minute free chat on a new topic and count requests for help
• Write one line: date, three anchor scores, situations ticked, retention out of ten
• Choose next month's anchors and missions from the weakest column

The free chat on a new topic is the closest thing to a real conversation in the test. Its number, how often you needed a word, is the one to watch over six months.

Keeping a log that survives

Logs die from ambition. A page per session with reflections, new words and goals is abandoned within a fortnight. The log that lasts is one line per session, written in the minute after the corrections appear, plus one longer line per month. Keep it in whatever you already open daily, a notes app or a spreadsheet, and never redesign it.

A session line holds the date, the task, the number of corrections, the minutes you spoke and the one correction you most want to keep. That last item does double duty: it is the seed for the phone rerun tomorrow and the pool from which next month's retention test is drawn. The monthly line holds the self-test numbers. That is the whole system, and it is enough to see three months of change at a glance.

What not to measure

Some numbers are easy to collect and say almost nothing about speaking. Streaks measure attendance, and a streak of easy chats is worth less than three hard sessions a week with rest days. Total minutes measure time, not what happened in it. Word counts from a vocabulary app measure recognition, which is the skill AI conversation is meant to go beyond. Number of sessions rewards short, shallow practice.

Feelings are the trickiest. A session that felt terrible, full of corrections, is often the most productive one, and a session that felt smooth may simply have been too easy. Log the numbers before you log the feeling, and if the two disagree over several weeks, believe the numbers. The exception is motivation: if the log shows progress and you still dread sessions, change the format, because a method you will not keep doing has no metric worth tracking.

When the numbers stop moving

Every learner hits a month where the anchors do not improve. Before changing anything, check the conditions: were the sessions shorter, noisier or on a harder setting than last time? If the conditions match and the numbers are flat, the usual cause is that the practice has become comfortable. Raise the difficulty of the anchors, add a family of situations you have avoided, or ask the Sensei to be a less cooperative partner.

If retention is the flat number, the phrases are not being reused; move them into daily short chats. If the free-talk number is flat while anchors improve, you have been practising scripts rather than conversation, and the fix is more unscripted topics. A plateau in the log is not a verdict; it is the log doing its job and telling you what to change next.