AI services · Detection

AI Content Detection: Why a 95% Accurate Detector Still Gets Half Its Accusations Wrong

The accuracy figure quoted on a detection service’s homepage is not the number you need. The number you need is different, it is computable from theirs, and once you have seen it you will never read a detector score the same way again.

Two different questions, and only one of them matters

Detection services advertise accuracy. Accuracy answers this question: given a piece of AI text, how often does the tool spot it?

That is not the question anyone actually has. When a score comes back at 87% and a real person’s work is sitting under it, the question is the opposite one: given that this was flagged, how likely is it that the person actually used AI?

Those two questions have wildly different answers, and the gap between them is where careers and grades get damaged.

How detection works, in one section

Almost every text detector rests on the same idea, and understanding it explains every failure that follows.

A language model, at every word, holds a probability distribution over what comes next. Perplexity measures how surprised the model is by the word that actually appeared. Text where each word is the obvious next choice has low perplexity. Text full of unexpected turns has high perplexity.

Generated text tends to sit in the low-perplexity range, because the model is literally selecting likely words. Human writing wanders more. A second measure, burstiness, looks at variation: humans write a long sentence, then a short one, then a fragment. Generated text is more even.

So the detector is not identifying a machine. It is measuring one thing only: how predictable the writing is. Everything that follows is a consequence of that.

The failure this mechanism guarantees

If the test is predictability, then anyone who writes predictably gets flagged. That is not a bug that better engineering removes. It is what the instrument measures.

Who writes predictably?

  • People writing in a second language. Simpler constructions, a smaller working vocabulary, safer sentence patterns. All of that is low perplexity.
  • Anyone writing to a template. Technical documentation, legal boilerplate, structured reports, lab write-ups.
  • Careful editors. Smoothing out odd phrasing is, mathematically, lowering perplexity.
  • Beginners. Writers who have not yet developed an idiosyncratic voice.

Meanwhile anyone who generates text and then rewrites a few sentences by hand raises perplexity and walks through. The test is easier to defeat deliberately than to pass accidentally, which is the opposite of what you want from an instrument used to make accusations.

What the research actually says

Presented with its criticisms, because this field is contested and any article that gives you only one side is selling something.

The study most often cited is a 2023 paper in Patterns titled “GPT detectors are biased against non-native English writers”. It ran seven detectors over essays by non-native English speakers and over essays by United States students. The reported average false positive rate on the non-native essays was 61.3%, against near-zero on the native-speaker set.

That figure gets quoted constantly without its context, so here is the context.

The objectionWeight it deserves
The sample was small, around 91 texts, and many under 150 wordsFair, and raised by a detection vendor. Short texts give any statistical test less to work with, so the error rate on them is not the error rate on a full essay.
A 2024 review found insufficient evidence to assert the bias universallyFair. The same paper noted the bias is mechanically plausible, because the detectors examined use perplexity.
Vendors report false positive rates below 1%Treat with care. These are self-reported, on self-chosen samples, by the party selling the product.
A 2026 preprint on 135,389 document pairs found detector output shifts with writing style even when authorship is unchangedDirectionally supportive, but it is a preprint and we have not confirmed peer review. Weigh it as a signal, not a settled finding.

The defensible summary is narrower than either camp would like: the exact size of the bias is disputed, the existence of false positives on predictable writing is not, and the mechanism explains why they occur.

The arithmetic that changes everything

Set the bias argument aside entirely. Grant a detector a genuinely good 95% on both measures: it catches 95% of AI text, and it correctly clears 95% of human text. Better than most things claim honestly.

Now run it over 1,000 submissions in a setting where 5% genuinely used AI.

CountWhat the detector does
Actually used AI50Flags 48, misses 2
Did not use AI950Clears 902, flags 48
Total flagged9548 correct, 48 innocent

A 95% accurate detector, applied to a population where most people did nothing wrong, produces accusations that are wrong half the time.

Nothing is broken. The detector performed exactly as advertised. The problem is that there are nineteen times more innocent people than guilty ones, so even a small error rate on the innocent group produces as many false flags as there are true ones. Rare things are hard to detect, no matter how good the test.

Here is the full picture. Each cell is the share of flagged work that genuinely used AI:

If this share actually used AIDetector clears 90% of humans95%98%99%
2%16%28%49%66%
5%33%50%71%83%
10%51%68%84%91%
20%70%83%92%96%
50%91%95%98%99%

Read the top-left corner. Where AI use is genuinely rare and the detector is merely good, roughly five out of six accusations are wrong.

And notice the shape of the fix. To get 90% of flags right in a 5% population, the detector must correctly clear 99.5% of human writing. Not 95%. Not 99%. Between those two lies the entire difference between a usable instrument and a machine for generating false accusations, and no detection service advertises at that level because nothing performs there.

This is not a criticism of any particular service. It is a property of testing for anything uncommon, the same reason medical screening programmes are designed around confirmatory tests rather than acting on a single positive. Detection services are measuring something real. The arithmetic simply means a single score cannot carry the weight people put on it.

Why watermarking does not rescue it

The obvious engineering answer is to stop guessing and have generators mark their own output, either by biasing word choice in a detectable pattern or by attaching signed provenance data to the file.

It is a genuinely better idea than perplexity, and it still does not solve the problem in front of you.

  • It is voluntary. Anyone running an unmarked model, and there are many, is unaffected.
  • Text watermarks are fragile. Paraphrasing, translating out and back, or ordinary editing degrades the signal. Images and video hold a mark far better than prose does.
  • Provenance metadata is strippable. A screenshot, a copy-paste, or a re-encode removes it, and that happens accidentally all the time.
  • Absence proves nothing. A missing watermark cannot distinguish human writing from generated text that lost its mark, which is the case you actually care about.

Watermarking can tell you something useful when it is present. It can never tell you anything when it is absent, and absence is the normal case.

What a detection score is actually good for

Narrower than the marketing suggests, but not nothing.

Reasonable uses

  • Spotting a change in how one person’s writing behaves over time
  • Triage: deciding which of 400 submissions deserves a human read
  • Screening bulk-generated content at scale, where being wrong about one item costs little
  • A prompt to open a conversation

Unreasonable uses

  • Evidence in a disciplinary process
  • Grounds for rejecting work or withholding payment
  • Any decision where the person cannot answer back
  • Anything automated, with no human in the path

If you are going to use one anyway

Most organisations will, so here is how to do it without hurting people.

Publish the rule before you enforce it

State what is permitted, what must be declared, and what happens when something is flagged. A rule invented after an accusation is not a rule.

Never act on a score alone

Treat a flag as the beginning of a question, not the end of one. Draft history, sources, the ability to discuss the work: all of these are better evidence than a percentage.

Estimate your own base rate

Use the table above with a realistic figure for your population. If you genuinely do not know it, you also do not know what your flags mean.

Watch who gets flagged

If second-language writers appear disproportionately in your flagged pile, the instrument is doing what the mechanism predicts, and continuing anyway is a choice.

Questions readers ask

Are AI detectors accurate?

Often accurate in the sense they advertise, and still misleading in the sense you need. The gap is the base rate. Where AI use is rare, even a strong detector produces flags that are wrong as often as right.

Can I prove I wrote something myself?

Better than a detector can disprove it. Draft history, version timestamps, notes and the ability to discuss your own reasoning are all stronger than a score. Keep the drafts, particularly if you write in a second language.

Why does my writing get flagged when I did not use AI?

Most likely because it is predictable in the statistical sense: clear, structured, consistent in rhythm. The detector cannot distinguish disciplined writing from generated writing, because it is not measuring origin at all.

Does editing generated text defeat detection?

Usually, yes, and easily. That asymmetry is the central weakness. The test is harder on honest predictable writers than on anyone deliberately evading it.

Will detection get better?

Detection improves, and so does generation, and generation is the larger investment. Provenance at the point of creation is a more promising direction than guessing after the fact, but it only ever works for marked content.

Should we build detection into our product?

Only with the base-rate arithmetic in front of you and a human in the loop. A detection feature that triggers an automatic consequence will, in most realistic populations, mostly punish innocent users.

The short version

Detectors measure predictability, not origin. That guarantees false positives on second-language and template-driven writing, and it makes the test easy to defeat on purpose. Even granting 95% on both measures, a 5% base rate means half of all flags are wrong, and reaching 90% reliable flags needs 99.5% specificity that nothing delivers. Watermarking helps when present and proves nothing when absent. A score is a reason to look closer. It is not evidence.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top