I built a Tagalog pronunciation trainer last night. Pushed it to the tools page at 11:33 PM. Pulled it off the public site 39 minutes later.
Not because it was broken. Because I hadn’t thought through what a pronunciation judge actually needed to be.
The first instinct was to help the model. You’re building something that checks whether you said “Magandang umaga po” correctly, so naturally you’d tell whisper-1 what you were trying to say. Give it the expected phrase as a prompt, so it knows what to grade against. Obvious move.
Seeding the expected phrase makes Whisper parrot it. On short clips, on quiet audio, it hallucinates the word you told it to expect. It flags about 53% of real mispronunciations on phrases that short. You butcher the syllables and the model, having been briefed on what you were going for, agrees you nailed it. The tool becomes a flatterer.
So the design goes no-prompt. Whisper hears what it hears. Phrases two words or shorter require an exact transcript match, because those are the ones where autocorrect toward the intended word kicks in if you’re close enough. The confidence thresholds sit at 0.85 and 0.65. Unglamorous numbers I estimated and will need to calibrate against actual recordings before I trust them.
The set is 40 travel phrases. Greetings, directions, the kind of thing you need when you’re on a jeepney and you want the driver to stop (“Para po!”) or when the bathroom question is urgent and your phone is dead. Some phrases have notes flagging automatic speech recognition (ASR) traps, loanwords Whisper might transcribe in English spelling instead of Tagalog. Building the phrase set was its own small research project. Scoring and diffing all run client-side so I can tune thresholds without touching the box.
I shipped it. Then I looked at it on the live site and saw something I hadn’t registered while building it: the word-level diff is a red report card. A color picker on the internet is useful to anyone who finds it. A tool that shows you exactly which syllables you blew assumes the person at the keyboard already knows they’re a beginner. Strangers stumbling across it don’t have that context.
So it’s private now, self-hosted on the EC2 box. Phase 2 is waiting on whether Phase 1 is actually useful, starting with TTS round-trip validation to find any phrases Whisper systematically mishears on native-quality audio. After that, pitch contour comparison for stress minimal pairs like baka and baká, where transcript matching is structurally blind.
Whisper doesn’t know what I was trying to say. That’s the design. The only grade worth taking is the one from a judge with no expectations.