Why Statistics Can Reveal AI Authorship

When a large language model generates text, it doesn't "think" the way humans do — it selects each word based on probability distributions learned from training data. This fundamentally statistical nature leaves measurable footprints that detection systems can analyze. Two of the most important metrics are perplexity and burstiness.

What Is Perplexity?

In information theory and natural language processing, perplexity measures how "surprised" a language model is by a given piece of text. Formally, it is the exponentiated average negative log-likelihood of the sequence.

Put simply: low perplexity means the text was highly predictable; high perplexity means the word choices were more surprising.

How This Applies to AI Detection

AI-generated text tends to have low perplexity when evaluated by the same type of model that produced it. This makes sense: an LLM is optimized to produce the most statistically likely continuations of text. Human writers, by contrast, make idiosyncratic choices — they use unexpected metaphors, vary their vocabulary in non-optimal ways, and break statistical patterns in ways that feel natural but are linguistically "surprising."

Detection tools exploit this by running candidate text through a reference language model and measuring how predictable the output is. Text that scores unusually low on perplexity is flagged as potentially AI-generated.

What Is Burstiness?

Burstiness refers to the variation in complexity and structure across a piece of writing. Human writers naturally produce "bursty" text — some sentences are long and complex, others are short and punchy. Some paragraphs are dense with information, others are lighter.

AI-generated text tends toward low burstiness: a more uniform distribution of sentence lengths and complexity. The prose is consistently competent but lacks the natural spikes and valleys of human writing rhythm.

Combining Both Metrics

Detection systems that use both metrics together achieve better accuracy than either signal alone. The reasoning is intuitive:

  • Low perplexity + low burstiness → Strong AI signal
  • Low perplexity + high burstiness → Ambiguous (possibly AI with human editing)
  • High perplexity + high burstiness → Strong human signal
  • High perplexity + low burstiness → Unusual; could indicate highly constrained human writing

Limitations of These Metrics

While powerful, perplexity and burstiness-based detection has well-documented limitations:

  1. Model dependency: Perplexity scores depend on which reference model is used. A text generated by GPT-4 may have high perplexity when measured against a smaller, differently-trained model.
  2. Domain effects: Technical or formal writing genres (legal documents, academic papers) naturally have low perplexity and burstiness — not because they're AI-generated, but because of genre conventions.
  3. Adversarial manipulation: Paraphrasing tools can alter word choices to increase perplexity while preserving meaning, defeating detection.
  4. Non-native speakers: Writers who construct sentences more formulaically due to language learning patterns may generate text with statistical properties similar to LLMs.

Beyond Perplexity: Emerging Approaches

Researchers are actively developing detection approaches that go beyond these two metrics:

  • Token probability watermarking: Embedding detectable statistical biases into the output distribution at generation time, which can then be verified without access to the original model.
  • Stylometric analysis: Building richer models of individual writing style to detect deviations.
  • Semantic coherence scoring: Measuring whether argument structure and topic transitions match human reasoning patterns.

The Takeaway

Perplexity and burstiness give us a statistically grounded window into the authorship of text — but they are probabilistic signals, not certainties. Understanding what these metrics measure, and where they fail, is essential for anyone using or building AI text detection systems. The science is advancing rapidly, and the most effective detectors treat these as starting points in a broader analytical framework.