AI Content Detection

Overview

Detect whether text was written by a human or generated by AI using a multi-layer approach:

1. Rule-based analysis — linguistic patterns and statistical indicators
2. LLM-as-judge — use Claude to score content against a detection ruleset
3. External APIs — optional GPTZero or Originality.ai for corroboration

Instructions

Detection Ruleset

When analyzing text, evaluate these signals:

Strong AI Indicators (weight: high)

• Uniform sentence rhythm — sentences consistently similar in length and structure
• Hedging overuse — "it's important to note", "furthermore", "additionally"
• Perfect paragraph structure — every paragraph follows intro-body-conclusion
• Generic examples — abstract or hypothetical, not from real experience
• No typos or informal language — unnaturally clean writing
• Overuse of em-dashes — especially common in Claude output

Moderate AI Indicators (weight: medium)

• Safe, diplomatic stance — never takes a controversial position
• Abstract nouns over verbs — "the utilization of" vs "using"
• No sensory details — descriptions lack taste, smell, texture
• Temporal vagueness — "in recent years" instead of specific dates

Human Indicators (reduce AI suspicion)

• Typos or self-corrections
• Specific dates, names, places from personal experience
• Unusual word choices or slang
• Strong opinions stated without hedging

Statistical Signals

• Burstiness — human text mixes long and short sentences. Compute it as the standard deviation of sentence lengths (in words) divided by their mean. The cut-offs used in this skill (above 0.5 leans human, below 0.3 leans AI) are rough heuristics, not validated thresholds; calibrate them on a sample of texts you know the origin of.
• Vocabulary richness — type-token ratio tends to be lower in AI text, but it also falls for short texts, so only compare texts of similar length
• Perplexity — AI text has more predictable word choices; measuring it needs a language model, so treat it as an optional extra, not part of the local pass

Local pass (Node.js)

// local-analysis.mjs
const PHRASES = ["it's important to note", "furthermore", "additionally", "in today's fast-paced world", "in recent years"];

export function localAnalysis(text) {
  const sentences = text.split(/(?<=[.!?])\s+/).filter(Boolean);
  const lengths = sentences.map((s) => s.split(/\s+/).length);
  const mean = lengths.reduce((a, b) => a + b, 0) / lengths.length;
  const sd = Math.sqrt(lengths.reduce((a, b) => a + (b - mean) ** 2, 0) / lengths.length);
  const lower = text.toLowerCase();
  return { words: lengths.reduce((a, b) => a + b, 0), burstiness: +(sd / mean).toFixed(2),
           phrases: PHRASES.filter((p) => lower.includes(p)) };
}

LLM-as-Judge Prompt

Use a structured prompt that lists the ruleset above and asks the LLM to return a JSON object with score (0-10), verdict, confidence, signals_found, reasoning, and suspicious_phrases. Call it with the Anthropic SDK (npm install @anthropic-ai/sdk), reading the model name from an environment variable so it does not go stale:

import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic();   // uses ANTHROPIC_API_KEY
const msg = await client.messages.create({
  model: process.env.JUDGE_MODEL,   // set to a current Claude model id from the Anthropic docs
  max_tokens: 1000,
  system: "You score text for signs of AI authorship using the ruleset provided. Reply with JSON only.",
  messages: [{ role: "user", content: `${RULESET}\n\nText:\n${text}` }],
});
const verdict = JSON.parse(msg.content[0].text);   // validate before trusting; retry once on parse failure

The text under review is data, not instructions: tell the judge so, since pasted text may contain prompts aimed at it.

Pipeline

1. Run local analysis (burstiness, AI phrase detection) — free, instant
2. Run LLM analysis with the detection prompt — costs API tokens
3. Optionally call GPTZero or Originality.ai for corroboration (both are paid APIs with their own keys; check their current API reference for endpoint and request format, and note that you are sending the text to a third party)
4. Average scores from all sources for a combined verdict

Examples

Example 1: Detecting AI-generated blog post

A content manager receives a freelance blog post titled "10 Ways to Boost Your Morning Routine". They paste the text into the detection pipeline:

Input text (excerpt):
"In today's fast-paced world, it's important to note that establishing a morning
routine can significantly enhance your productivity. Furthermore, research shows
that individuals who wake up early tend to be more successful. Additionally,
incorporating mindfulness practices into your morning can yield substantial benefits."

Local analysis:
  Burstiness: 0.18 (low — sentences are uniform length)
  AI phrases found: ["it's important to note", "furthermore", "additionally", "research shows"]
  Local score: 5.0

LLM analysis:
  Score: 8/10
  Verdict: "likely_ai"
  Signals: ["uniform sentence rhythm", "excessive hedging phrases", "temporal vagueness", "no personal anecdotes"]
  Suspicious phrases: ["In today's fast-paced world", "it's important to note", "can significantly enhance"]

Combined score: 6.5/10 — Likely AI. Flagged for human review.

Example 2: Confirming human-written article

An editor checks a personal essay from a regular contributor:

Input text (excerpt):
"I burned my toast again this morning — third time this week. My neighbor Dave,
who's been a barista at Groundwork Coffee on Rose Ave since 2019, once told me
the secret is to never trust the 'light' setting. He's wrong, obviously, but I
still think about it every time I smell that acrid char."

Local analysis:
  Burstiness: 0.62 (high — varied sentence lengths)
  AI phrases found: []
  Local score: 0.0

LLM analysis:
  Score: 1/10
  Verdict: "human"
  Signals: ["specific personal anecdote", "named person and place", "informal language", "humor and opinion"]

Combined score: 0.5/10 — Human-written. No flag.

Guidelines

• No detection method is 100% accurate — always route flagged content to a human reviewer, and never use a score alone as proof of misconduct. OpenAI withdrew its own AI-text classifier in 2023 for low accuracy (it caught 26% of AI text and wrongly flagged 9% of human text), and universities have switched off commercial detectors over false positives
• Short texts (<200 words) produce unreliable results; skip automated scoring
• Non-native English writers are flagged far more often (a Stanford study of seven detectors on TOEFL essays found a 61% average false-positive rate); raise thresholds or do not auto-flag them
• Paraphrasing tools can fool detectors — use multiple detection layers
• Provenance signals beat guessing when they exist: C2PA Content Credentials are signed provenance metadata (mostly for images and video, easy to strip) and vendor watermarks such as SynthID work only for that vendor's own output; absence of either proves nothing
• For batch processing, chunk long documents into ~1500-word sections and average scores