AI Product / PoC
Pronunciation & speaking-style feedback
A web app for practicing how you speak — intonation, pauses, emphasis, and pace — in Japanese and English. Record a short free-form utterance and it analyzes monotony, hesitations, filler words, and more, visualizes them as word-level overlays alongside a pitch curve, and an LLM gives concrete feedback on exactly what to fix and how.
Key results
- ASR agreement averaging 0.953 in automated evaluation over 1,680 clips (JVS + LibriSpeech)
- False-positive rate of 2.7% for hesitations (mid-word pauses)
- Monotony score separates a flat vs. expressive reading of the same sentence: 0.74 vs. 0.15
Background
Self-studying how you speak runs into three walls: you cannot hear your own voice objectively, you cannot articulate what is wrong, and you do not know how to fix it. Aimed at people who speak on video, this PoC started from the idea that a loop of "speak, auto-segment and replay, analyze and visualize, get concrete feedback" on 5-10 second utterances would let people improve their delivery on their own.
How it works
The browser detects silence and segments utterances automatically; the server then runs Whisper (whisper-large-v3) transcription with word-level timestamps in parallel with pitch (F0) and loudness extraction via essentia.js. From these it computes utterance-level metrics such as monotony and pace, plus word-level ones: emphasis, pauses (distinguishing mid-word hesitations from intentional between-word pauses), and filler words. Results are visualized as overlays on the transcript and an SVG pitch curve, and the features plus transcript go to Llama 3.3 70B to generate concrete improvement advice. For Japanese, kuromoji reconstructs word boundaries, and an experimental pitch-accent check against the kanjium accent dictionary is also included.
Key design points
The practical hurdles are in the audio-analysis groundwork. Octave errors in pitch estimation are corrected with a hybrid of median filtering, continuity smoothing, and median folding. Because Whisper word timestamps leave no gaps, pause detection switched to energy-based silence detection, with guards against hallucination on silent input (fabricating sentences from silence). Pace is computed over actual articulation time excluding silences, and a reliability gate suppresses confident judgments on utterances that are too short. To keep evaluations from fluctuating, these metrics are regression-tested with an automated evaluation harness over multi-speaker corpora.
Results
In automated evaluation over multi-speaker corpora (1,200 Japanese JVS clips plus 480 English LibriSpeech clips, 1,680 in total), ASR agreement averaged 0.953, the false-positive rate for hesitations (mid-word pauses) was 2.7%, and suspected octave errors were 0.2%. In live calibration sessions, the monotony score cleanly separated a flat reading of a sentence from an expressive one — 0.74 versus 0.15 — confirming the intended sensitivity.
Measured results
| Metric | Overall | Japanese (JVS) | English (LibriSpeech) |
|---|---|---|---|
| ASR agreement (mean) | 0.953 | 0.942 | 0.981 |
| Hesitation false-positive rate | 2.7% | 2.8% | 2.7% |
| Reliability-gate pass rate | 98.9% | 100% | 96.2% |
ASR agreement measures how closely Whisper output matches the corpus reference text. The reliability gate suppresses confident judgments on inputs that are too short to evaluate.
Demo & write-up
A demo video and write-up are in preparation and will be published on this page.