Verify Watermark Removal

Verify Watermark Removal

A tool says it removed a watermark from a text. This skill checks whether it did, by measuring rather than by trusting the tool's own report.

The check works because the mark is planted first, with a key the user holds. A detector that knows the key is the strongest detector that can exist for that text, so its score is a ceiling rather than a guess. What the key buys is a scale of the user's own; what keeps a tool from recognising the sample is something else, namely that a freshly generated private sample is a text nobody has seen before. The samples shipped with the tool are published together with their key, so those can be recognised, and the repository says so in samples/README.md. The skill also reports what the rewrite cost the text, because a mark that disappeared together with half the meaning is not a result anyone wants.

This skill measures. It never removes a mark, and it never claims anything about a specific vendor's watermark.

When to Use This Skill

• A "watermark remover", "AI humanizer" or "detector bypass" tool claims it cleaned a text, and the claim needs checking before anyone repeats it.
• Someone is about to pay for such a tool and wants evidence instead of a landing page.
• Several such tools have to be compared on one ruler, with numbers another person can reproduce.
• A rewriting step sits inside a pipeline, and someone needs to know what it costs in meaning, facts and length.
• The user asks how these tools can be tested at all, or why a "99% undetectable" promise cannot be verified from outside.

What This Skill Does

1. Plants a known mark: generates text carrying a statistical watermark of the SynthID-Text class under a key the user chooses, and scores every sample as it writes it, so a text where the mark did not plant is named before anything is handed to a tool.
2. Scores what came back: runs the keyed detector on the returned text and places the score against thresholds that were fixed before any run, giving one of four outcomes.
3. Measures the cost: meaning similarity, facts kept, longest verbatim run, share of words changed, length ratio.
4. Builds a comparison table: turns several recorded runs into one matrix, so tools are ranked on the same measurements instead of on their own claims.
5. Keeps the answer honest: reports the grey zone as an outcome of its own and never turns a passing run into "this text is now undetectable".

How to Use

Basic Usage

Install once. A CPU is enough, Python 3.10 or newer:

git clone https://github.com/Yurakonoplya/unmark-checker && cd unmark-checker
git checkout v0.1.5              # the revision this page describes
pip install --index-url https://download.pytorch.org/whl/cpu torch
pip install -e .                 # no PyPI package: install from the clone
read -rs UNMARK_CHECKER_KEY && export UNMARK_CHECKER_KEY

The checkout pins exactly the revision described here: main moves, and a command that behaves differently from this page is worse than no page.

The key is typed at the read prompt, which echoes nothing and writes nothing to the shell history. Keep it in the environment, never in a file and never in a command that gets logged. The key is the whole basis of the check.

If the tool under test runs on the same machine, start it with env -u UNMARK_CHECKER_KEY <tool> ...: a process that inherits the key could score the sample itself and shape its output to it, which is exactly what the measurement is meant to rule out.

Then four steps.

1. Get a marked sample. The repository ships ready samples in samples/, with their key printed in samples/README.md. That is the fast path. For a test no tool can anticipate, generate your own (about two minutes each on a CPU with the small model):

unmark-checker generate --num 2 --words 100 --scheme shallow \
    --model sshleifer/tiny-gpt2 --out my-samples

Use --model gpt2 instead when the sample has to read as English, for example when it is going into a web form that rejects nonsense.

Every sample is scored as it is written, and the manifest next to the texts records the outcome of that scoring. Hand a tool only the samples the manifest records as mark_present. The command does not do this filtering for you: if the mark planted in none of the samples it says so and exits with code 2, but if it planted in some of them it warns, lists the ones it did not plant in and exits with code 0, leaving every file on disk. check behaves the same way: given such a sample it prints a warning and measures it anyway, and that measurement means nothing, because a mark that was never there cannot be removed.

2. Run the tool under test on the sample, and save exactly what came back, unedited, to a file.

3. Measure, one command:

unmark-checker check --sample my-samples/UM-1A2B3C.txt --returned cleaned.txt

4. Report what the run says, and only that. The outcome is one of four:

Always report the cost numbers next to the outcome: meaning kept, facts kept, longest verbatim run, share of words changed, length ratio. They exist for the three scored outcomes only (mark_present, uncertain, mark_gone). A not_our_text run stops before they are computed and reports four things instead: the outcome, the share of content words that came from the sample, and the word counts of both texts. Do not ask for a meaning or facts number there, and do not report one as zero: it was never measured.