The work board had fifteen categories. Things like feature, maintenance, research, integration, production-config. I designed them. The agents applied them.
I wired in Jev, TypeSafe’s System One API, to groom the backlog. For each card it would suggest a category at a measured confidence level and flag disagreements for review. I ran it on 136 migrated cards.
65% agreement overall. Reasonable-sounding. A 35% disagreement rate on an open-ended categorization task felt like expected variance, edge cases, two valid reads on the same card.
Then I looked at the 32 high-confidence cases, where both the board’s label and Jev’s suggestion were confident. I went through each one. Jev was right 23 times. The original label was right six times. Three were too close to call.
That’s not variance. That’s the board’s confident labels losing 23 to 6 in a head-to-head.
The “maintenance” category was the most honest data point. Thirty-five cards had that label. Five of them were actually maintenance. The other thirty had drifted there over time because maintenance was vague enough to absorb anything that didn’t fit cleanly elsewhere.
When I looked at how the groomer built its vocabulary, it was reading category names from the cards. Not from a spec. From what was already in the backlog. Which meant every wrong “maintenance” label made “maintenance” look like a valid option for the next ambiguous card. The system was feeding its own errors back as training signal.
The fix was trimming from fifteen to eleven, cutting the categories that overlapped or had no real definition, and replacing where the vocabulary came from. The rubrics now live in a json file. The groomer reads from the spec, not from the corpus.
Fifteen to eleven isn’t a dramatic reduction. But the count wasn’t the problem. I’d let the categories define themselves from how I’d already applied them, wrong calls included. I had to run a migration script to clean up what a self-taught vocabulary had accumulated. That script should have been a spec.