I Watermarked 100 Texts With SynthID and Asked Local LLMs to Wash It Out. The Detector Still Saw Some of It.

Description and use

Local LLMs can partially remove SynthID watermarks, but the mean‑test detector still flags 23–41% of rewrites. Longer texts and fiction genres are harder to wash.

Problem

Can a local LLM rewrite a SynthID‑watermarked text enough to evade detection while preserving meaning?

Outcome

Rewriting watermarked texts with local LLMs substantially reduces Bayesian detector flags to near clean‑text levels, but a mean‑test detector still flags 23–41% of rewrites. Longer texts and fiction genres retain more watermark evidence. A second rewrite pass does not further reduce watermark traces.

Workflow

• Generate 100 watermarked texts with Qwen3‑4B‑Instruct‑2507 using SynthID config
• Train a Bayesian detector on 1,200 watermarked and 3,480 clean examples
• Generate 100 unwatermarked control texts
• Rewrite each text with qwen3:4b‑instruct, gemma3:12b, and qwen3:14b using a prompt that preserves meaning
• Optionally run a second rewrite pass with gemma3:12b
• Tokenize all texts with Qwen tokenizer
• Run both Bayesian and mean‑test detectors on originals, rewrites, and controls
• Collect flagged percentages, mean g‑values, z‑scores, AUC, and 5‑gram overlap
• Analyze results by model, text length, and genre

Evidence

The standard detector flagged 3% to 15% of the rewrites, depending on the rewriting model.

Source

The mean test flagged 23% to 41%.

Source

The watermark also shows up fast: using only the first 100 tokens of each original, 94% were flagged; at 200 tokens, 100%.

Source

Longer texts are easier to catch: for 750–900 word texts, qwen3:14b rewrites were flagged 89% of the time.

Source

Fiction is the hardest to wash: short fiction rewrites were flagged 70% by mean test and 47% by Bayesian.

Source