The belief going in was that three AI researchers with different lenses would catch each other’s errors. The skeptic would push back on the optimist. The empiricist would demand numbers the others hand-waved past. If two of them returned with the same finding, that meant something. Agreement was signal.
The first live run of my /research-debate skill told a different story.
First phase of the skill was three AI researchers. Distinct prompts, distinct angles. The skill ran for the first time today, pointed at a question about token cap behavior for my Claude setup. When the reports came back, two of them had the same number from the same source, a blog post that had done its own roundup of the official docs. The third had cited a vendor policy the vendor’s own site now listed as paused, with a banner on the page saying so.
The second phase of the skill ran the AI evaluators. Gave all three reports to each one, told them to attack. They did, but not by comparing the reports against each other. That’s what I’d expected. Instead, they went back to the vendor’s own pages and ran the forecaster script sitting right in the repo. The researchers had read the script’s log file and reasoned from that. The evaluators executed it. The log had been quiet because there was nothing to alert on, not because it was current.
The kills came from that, not from the cross-check.
The thing I’d gotten wrong was what “adversarial” meant in practice. I was imagining a debate. Researchers as witnesses, evaluators as opposing counsel. But three witnesses who all read the same summary blog before trial don’t become three independent witnesses. And opposing counsel doesn’t catch that by asking them to contradict each other. They catch it by finding the primary document themselves.
The evaluators caught something else too. A legacy API token sat in a deployment config file, committed in plaintext. Nothing to do with the research question. One evaluator noticed it while pulling repo context to verify a claim. The token turned out to authenticate nothing (the service was inactive, the security group had zero ingress), but it still should have been gone. It’s now.
The lessons are in the skill file now. These are the things that shake out when a plan meets real life.