I thought the hard problem with voice AI was knowing when to speak.
My setup has a slow lane: a background process that bubbles up notes to Jarvis while we’re talking. Before this week, those notes just sat in a queue until I typed my next message. So I built a gate. The idea was that Jarvis would score each pending note before deciding whether to interrupt the live conversation. High relevance, speak up. Low relevance, stay quiet.
I ran the evaluation on 42 reconstructed Board Room windows and 7 labeled examples. The gate’s “worth interjecting” score averaged 0.58 on sessions where an interruption would have been welcome, and 0.58 on sessions where it wouldn’t have been. No separation. The gate couldn’t tell the difference, so I declined to ship it.
I’d built a gate for a problem that wasn’t there.
I dug into 17 voice sessions with a fresh harness, looking at actual interjection phrasing this time. Twelve verdicts came back good. Five came back stale. Zero came back wrong. Not a single thing Jarvis said was incorrect. But five of them landed flat because the opener didn’t match what the message actually was.
The stale ones all opened with “Hold on” or “Wait.” Those words promise an interruption. They brace the listener. When the note that followed was a gentle background observation (“This just popped into my head…”) rather than urgent news, the listener had already tensed up for something that never arrived. The opener was announcing the wrong kind of message.
I updated the prompt: asides open as “This just popped into my head…” Genuine interruptions still get the urgent openers.
12 good, 5 stale, 0 wrong is actually a fine result. The content is solid. The signal was never “Jarvis doesn’t know when to speak.” It was “Jarvis announces a gentle thought like he’s stopping traffic.”
If I could go back to the morning I started building the gate, I’d say “check the openers first”.