01The worry
Our wizard asks a language model to propose subreddits for a topic. The standard objection writes itself: language models make things up, so won’t it send you to communities that don’t exist?
Rather than guess, we ran a fidelity test we could repeat.
02What we measured
We took ten hypotheses spanning deliberately different domains — cold email, indie SaaS, vape addiction, egg freezing, kubernetes, runner’s knee, toddler screen time, novel writing, crypto MEV, chronic insomnia. For each we asked the model for ten subreddit names, then checked all 100 against Reddit itself.
03The results
The single hallucination was r/quittingvaping. The other nine “losses” were real communities the validator correctly filters out for being private or tiny — not the model’s fault.
04So the real hallucination rate was 1%
Only one name in a hundred was invented. Everything else either passed or was a genuine community our own rules dropped for being unreadable or too small to matter.
05Why we still validate every name
A 1% invented-name rate is low, but “low” isn’t “zero”, and a dead link in a research tool erodes trust fast. So every suggested subreddit — model-proposed or not — is checked against Reddit itself before it reaches you. Hallucinations are caught and dropped automatically.
The model earns its place by finding the niche, cross-cutting communities Reddit’s own keyword search misses — r/eggfreezing, r/runninginjuries, r/platformengineering — not by being trusted blindly.
Reproduce it
The test is an idempotent script (scripts/test-suggest-fidelity.mjs) — re-run it whenever the model changes.
How the pipeline works →Frequently asked questions
Sometimes, but rarely. When we asked a language model for 100 subreddit names across ten different domains and checked every one against Reddit itself, exactly one was invented — r/quittingvaping. The other names were real communities, some of which our rules dropped only for being private or too small.
Mostly. In our fidelity test of 100 model-suggested names, 90 were live, public communities with at least 500 members and just one was a hallucination. A handful more were genuine subreddits that happened to be private, restricted, or under 500 members — real, but filtered out by our own rules, not invented by the model.
We check each name against Reddit itself. A real community returns its details; a hallucinated one comes back empty and gets dropped automatically. We run this check on every suggested subreddit — model-proposed or not — before any name reaches you, so dead links never make it through.
Language models generate plausible-sounding text, so a name that fits the pattern of real subreddits can be produced even when no such community exists. That is why we never trust the model blindly. In our test the invented-name rate was 1 percent — low, but not zero, which is exactly why every name is validated against Reddit itself.