I thought the hard part of building an AI writing detector was the NLP. The tokenizing text, matching patterns, scoring heuristics.
The part that actually took time was calibrating the weights. If I’m being really open, I’m still not satisfied enough with the calibration right now.
Before I wrote a line of code, I had a mental list of AI tells: “in today’s fast-paced world,” the word “delve” where no human would natural this to use it, bullet-point summaries at the end. I figured turning those into a detector would be straightforward. You find the phrases, you flag them, you ship it.
The opener ratio alone took me four iterations to get right. The opener ratio is a heuristic for the first sentence or two of a paragraph: what percentage of a document’s paragraph openers match known AI patterns before you call the document AI-generated? Set it too high and you miss obvious machine output. Set it too low and legitimate human writing starts tripping the alarm. Four tries before it behaved.
By the third iteration I realized the phrases I’d started with were catching only the most formulaic AI output. The stuff that reads like someone used a template. Any model worth using doesn’t write like that anymore. The real signal was somewhere else.
It’s in the rhythm.
Human writing is asymmetric. Some sentences carry a lot of weight. Others are short. Clause structure varies. An idea starts one way and ends somewhere the opening didn’t predict. AI writing does something different, it distributes clause weight evenly. Balances the sentence against itself. Like it’s afraid to lean too hard in any direction. Smooth in a way that, once you’re looking for it, becomes hard to unsee.
Don’t get me wrong, the phrase-level tells are still real. “It is worth noting” is a flag. “Landscape” used as a metaphor for any abstract concept is a flag. But building the hover tooltip that explains why a phrase got flagged made me realize: the phrases are symptoms. The rhythm is the cause.
I added an output stretch curve toward the end. The tool had been producing scores that clustered around the middle of the confidence scale, a lot of “might be AI.” That’s not useful to anyone. The stretch curve pushes outputs toward the ends: either likely human or likely AI, with the uncertain middle reserved for genuinely ambiguous text. Calibrating that took another full pass.
If there’s anything useful from a day of building this, it is that the AI “tells” you’ve probably internalized from thinkpieces and Twitter threads are real, but they’re the easy part. You have to go one level deeper, past the vocabulary and into the sentence structure, before you’re catching anything real.
Run your writing through a detector not to prove it’s human, but because like Gloria Estefan said “the rhythm is gonna get you.”