I Watermarked 100 Texts With SynthID and Asked Local LLMs to Wash It Out. The Detector Still Saw Some of It.

Описание и применение

Модели локальных LLM могут частично удалить водяной знак SynthID, но детектор по среднему значению g‑значения всё ещё отмечает 23–41 % перезаписанных текстов. Длинные тексты и художественные жанры более устойчивы к удалению.

Задача

Can a local LLM rewrite a SynthID‑watermarked text enough to evade detection while preserving meaning?

Результат

Rewriting watermarked texts with local LLMs substantially reduces Bayesian detector flags to near clean‑text levels, but a mean‑test detector still flags 23–41% of rewrites. Longer texts and fiction genres retain more watermark evidence. A second rewrite pass does not further reduce watermark traces.

Рабочий процесс

• Generate 100 watermarked texts with Qwen3‑4B‑Instruct‑2507 using SynthID config
• Train a Bayesian detector on 1,200 watermarked and 3,480 clean examples
• Generate 100 unwatermarked control texts
• Rewrite each text with qwen3:4b‑instruct, gemma3:12b, and qwen3:14b using a prompt that preserves meaning
• Optionally run a second rewrite pass with gemma3:12b
• Tokenize all texts with Qwen tokenizer
• Run both Bayesian and mean‑test detectors on originals, rewrites, and controls
• Collect flagged percentages, mean g‑values, z‑scores, AUC, and 5‑gram overlap
• Analyze results by model, text length, and genre

Доказательства

The standard detector flagged 3% to 15% of the rewrites, depending on the rewriting model.

Источник

The mean test flagged 23% to 41%.

Источник

The watermark also shows up fast: using only the first 100 tokens of each original, 94% were flagged; at 200 tokens, 100%.

Источник

Longer texts are easier to catch: for 750–900 word texts, qwen3:14b rewrites were flagged 89% of the time.

Источник

Fiction is the hardest to wash: short fiction rewrites were flagged 70% by mean test and 47% by Bayesian.

Источник