agentlinkops 0.6.13 → 0.6.16

package/skills/content-defingerprinting/scripts/watermarks/README.md

@@ -1,102 +0,0 @@
-# Stage 2b — the invisible-character scan
-
-**Required on every page, every time, before it ships.** It is part of stage 2
-(humanize), and it is the half of stage 2 that a human reviewer physically cannot do.
-
-## What it is
-
-A vendored copy of the Layer A engine from
-[watermarks-remover](https://github.com/guillaumemeyer/watermarks-remover) (MIT),
-pinned in `UPSTREAM.json`, plus `wm-scan.py`, our wrapper that walks content roots
-and reports line numbers.
-
-```bash
-python3 wm-scan.py <path> [<path> ...]     # report, exit 1 on findings
-python3 wm-scan.py --fix <path>            # clean in place, then re-verify
-python3 wm-scan.py --json <path>           # for a gate
-python3 verify-vendor.py                   # prove the engine is the pinned upstream one
-./revendor.sh [ref]                        # refresh from upstream and re-pin
-```
-
-## Why it is required rather than recommended
-
-Because it catches a defect class that survives every other check we have. These
-characters are invisible in the editor, invisible in review, invisible in the
-rendered page, and they ride in on ordinary copy-paste from PDFs, Word and model
-output. What they break is everything that reads our prose as text:
-
-**A zero-width space inside "affidavit" means the banned-phrase scan, the citation
-audits and every `grep` in the repo stop seeing the word.** The page looks fine and
-the checks that are supposed to guard it have quietly stopped applying.
-
-It is not a theoretical risk. First run across the estate, tax-sale, trucking,
-coverage and proxy repos, 2026-08-18:
-
-| repo | finding |
-|---|---|
-| settled | a soft hyphen inside `affi[U+00AD]davit` in rendered prose, plus 4 more in Missouri state data |
-| Tax Sale Atlas | Wingdings/PUA glyphs (`U+F0B7`, `U+F073`) in 5 counties' `sale.json`, including inside an `evidence` string |
-| covered-weight | 1,200+ PUA glyphs across the archived payer-policy extractions |
-| haulhandbook, proxy-compare | clean |
-
-## The archive rule, which is not negotiable
-
-**`--fix` refuses to touch anything under `sources/`, `snapshots/`, `evidence/`,
-`_sources/`, `price-history/` or `raw/`.**
-
-That tree is the evidence every `_source` and `verified_quote` resolves against. The
-marks in it are glyph artifacts of a document *somebody else published*, and they are
-part of what we recorded. Rewriting the archive so a scan goes green is precisely the
-tampering the provenance chain exists to detect — one edited file and one re-forged
-hash is the whole covered-weight story.
-
-So: **the archive is reported and left alone. The QUOTE lifted out of it is the thing
-to check.** A hit inside a quote field is a provenance question before it is a
-formatting one: open the source, see what the character is, and decide. A PDF bullet
-glyph rendered into a quote is noise the reader never saw and can be dropped; a real
-character cannot.
-
-## The load-bearing rule (found by auditing, 2026-08-18)
-
-**`--fix` also holds back any file where the mark sits on a line carrying regex-replacement
-syntax**, tagging it `[LOAD-BEARING? regex literal on that line - review by hand]`.
-
-The estate-wide audit found three of these, and every one is a **sanitizer written with the
-literal characters it strips**:
-
-```js
-.replace(/[<zwsp><zwnj><zwj><bom>]+/g, '')   covered-weight  extract-ehb-chart-rows.mjs:127
-.replace(/&nbsp;| |<zwsp>/g, " ")            proxy-compare   scraper/vendors/proxyscrape.ts:112
-.replace(/<nbsp>/g, " ")                     proxy-compare   scraper/vendors/rayobyte.ts:67
-```
-
-Auto-cleaning those empties the character class and silently breaks the guard, letting through
-exactly the contamination it existed to catch. **A blanket `--fix` across the estate would have
-broken three sanitizers in two repos** — the scanner would have disabled its own counterparts.
-
-This is the argument for classify-before-fix in one example. A hit is not a defect; a hit is a
-question. The three answers that mean "do not touch" are: it is archived evidence, it is a
-sourced quote, or it is load-bearing.
-
-## What we deliberately do NOT run
-
-The upstream project does more than this, and the rest of it is off by decision, not
-by oversight.
-
-| layer | status | why |
-|---|---|---|
-| **A** — invisible Unicode, exotic spaces | **required** | pure contamination, deterministic, zero judgement needed |
-| **Files** — EXIF / XMP on our own generated covers | available | harmless hygiene; our covers are generated by our own script, so there is little to strip |
-| **C2PA manifests** | **detect, never strip** | C2PA is a provenance record. On one of our own covers it is a signal that something upstream generated an image with AI — on sites that say "never an AI image", that is a question to answer, not noise to delete |
-| **B** — rewriting text to defeat statistical watermarks | **opt-in, off** | our humanize stage already rewrites AI-pattern prose on quality grounds. Making "defeat provenance watermarking" an explicit standing step across five YMYL properties is a decision for the owner, not a default |
-| **Pixels** — CtrlRegen image watermark removal | **not installed** | needs a model backend, and it has no use on sites that generate their own covers. Its actual use is stripping watermarks off images we did not make |
-
-If any of those need turning on, that is a conversation, and the reason belongs in
-this file next to the change.
-
-## Keeping it honest
-
-`vendor/` is byte-identical to upstream and must stay that way — `verify-vendor.py`
-proves it, and `revendor.sh` is how it changes. Do not edit it in place. The last time
-a copied source of truth got edited locally in this estate it became a house rule
-nobody agreed to, sitting in a file that looked authoritative.

package/skills/content-defingerprinting/scripts/watermarks/UPSTREAM.json

@@ -1,26 +0,0 @@
-{
-  "upstream": "https://github.com/guillaumemeyer/watermarks-remover",
-  "license": "MIT",
-  "pinned_commit": "8318d4df79de18a8c18c21eecb4b796192f34805",
-  "pinned_commit_date": "2026-08-18T18:28:19-07:00",
-  "vendored_on": "2026-08-18",
-  "vendored_from": "service/scripts/",
-  "modified": false,
-  "layers_vendored": [
-    "A: invisible Unicode / space homoglyphs (deterministic)"
-  ],
-  "layers_deliberately_not_vendored": [
-    "B: statistical token-sampling watermark removal by rewriting (opt-in, needs a decision)",
-    "Files: C2PA / EXIF / XMP stripping (we DETECT and report C2PA, we do not strip it)",
-    "Pixels: CtrlRegen image watermark removal (not used; our covers are our own generated assets)"
-  ],
-  "note": "Files are byte-identical to upstream. Re-vendor with revendor.sh; never edit in place.",
-  "sha256": {
-    "LICENSE.upstream": "c70d55b5722484a6be261f755b5bdfd5335244d0389436eb116aa8357243347c",
-    "clean_text.py": "8cde7a3c75c06687f7403f85364eda3380ace4c0ab5e5eb7d1f89f9e0c50ce04",
-    "common.py": "311d78fb23351d56b9a2601a7af3b8d5bf834e2e55289796e1f18184a3b9f49d",
-    "inspect_text.py": "a61e5c524f72de8122750114f09973cec826641d798f417d811c7e5649218479",
-    "score_stylometry.py": "cd3dae134d5f641120218fd525afd523d4a0bfb52d4d73c0e92dc04e3bcc6d59",
-    "text_unicode.py": "cd21c5700184a6bb4a43089f546531a2692bf140413954ea9b93f4a67f1f3a5e"
-  }
-}

package/skills/content-defingerprinting/scripts/watermarks/revendor.sh

@@ -1,52 +0,0 @@
-#!/usr/bin/env bash
-# Re-vendor the watermarks-remover Layer A engine from upstream.
-#
-#   ./revendor.sh [ref]        # default: main
-#
-# The vendored files are byte-identical to upstream and MUST stay that way. If a
-# behaviour needs changing, change it in wm-scan.py (ours) or send a PR upstream.
-# Editing vendor/ in place produces a house rule nobody agreed to, in a file that
-# looks like somebody else's source of truth. That has happened here before with a
-# banned-phrase list.
-set -euo pipefail
-
-REF="${1:-main}"
-HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-REPO="https://github.com/guillaumemeyer/watermarks-remover"
-FILES=(common.py text_unicode.py inspect_text.py clean_text.py score_stylometry.py)
-TMP="$(mktemp -d)"
-trap 'rm -rf "$TMP"' EXIT
-
-echo "Fetching $REPO @ $REF ..."
-git clone --quiet --depth 1 --branch "$REF" "$REPO" "$TMP/src" 2>/dev/null \
-  || git clone --quiet "$REPO" "$TMP/src"
-git -C "$TMP/src" checkout --quiet "$REF" 2>/dev/null || true
-
-for f in "${FILES[@]}"; do
-  cp "$TMP/src/service/scripts/$f" "$HERE/vendor/$f"
-done
-cp "$TMP/src/LICENSE" "$HERE/vendor/LICENSE.upstream"
-
-SHA="$(git -C "$TMP/src" rev-parse HEAD)"
-DATE="$(git -C "$TMP/src" log -1 --format=%cI)"
-TODAY="$(date +%Y-%m-%d)"
-
-python3 - "$HERE" "$SHA" "$DATE" "$TODAY" <<'PY'
-import hashlib, io, json, os, sys
-here, sha, date, today = sys.argv[1:5]
-m = json.load(io.open(os.path.join(here, 'UPSTREAM.json'), encoding='utf-8'))
-files = {}
-for f in sorted(os.listdir(os.path.join(here, 'vendor'))):
-    if f.startswith('._'):            # AppleDouble shadows on the exFAT volumes
-        continue
-    files[f] = hashlib.sha256(io.open(os.path.join(here, 'vendor', f), 'rb').read()).hexdigest()
-changed = [f for f, h in files.items() if m['sha256'].get(f) != h]
-m.update(pinned_commit=sha, pinned_commit_date=date, vendored_on=today, sha256=files)
-io.open(os.path.join(here, 'UPSTREAM.json'), 'w', encoding='utf-8').write(json.dumps(m, indent=2) + "\n")
-print("pinned %s (%s)" % (sha[:12], date))
-print("changed: %s" % (", ".join(changed) if changed else "nothing"))
-PY
-
-echo
-echo "Now re-run the scan on a repo you know the answer for, before trusting it:"
-echo "  python3 $HERE/wm-scan.py <a path with a known hit>"

package/skills/content-defingerprinting/scripts/watermarks/vendor/LICENSE.upstream

@@ -1,21 +0,0 @@
-MIT License
-
-Copyright (c) 2026 watermarks-remover contributors
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.

package/skills/content-defingerprinting/scripts/watermarks/vendor/clean_text.py

@@ -1,96 +0,0 @@
-#!/usr/bin/env python3
-"""Strip invisible Unicode / normalize space homoglyphs (Layer A)."""
-
-from __future__ import annotations
-
-import argparse
-import json
-import sys
-from pathlib import Path
-
-sys.path.insert(0, str(Path(__file__).resolve().parent))
-
-from common import (
-    backup_path,
-    cleaned_path,
-    eprint,
-    read_text_input,
-    write_text_output,
-)
-from text_unicode import clean_text
-
-
-def main() -> int:
-    p = argparse.ArgumentParser(description=__doc__)
-    p.add_argument("path", nargs="?", default="-", help="Input text file, or - for stdin")
-    p.add_argument("-o", "--output", help="Output path (default: stdout or *.cleaned.*)")
-    p.add_argument("--nfkc", action="store_true", help="Apply Unicode NFKC after scrub")
-    p.add_argument(
-        "--aggressive-homoglyphs",
-        action="store_true",
-        help="Map Cyrillic/fullwidth Latin confusables to ASCII Latin",
-    )
-    p.add_argument(
-        "--no-normalize-spaces",
-        action="store_true",
-        help="Do not rewrite exotic spaces to U+0020",
-    )
-    p.add_argument(
-        "--strip-emoji-glue",
-        action="store_true",
-        help="Paranoid: strip all load-bearing invisibles too (emoji glue, script joiners, flag tags, same-script fillers/selectors, orthographic Cf)",
-    )
-    p.add_argument(
-        "--strip-bidi",
-        action="store_true",
-        help="Also strip legitimate RTL/LTR directional marks and isolates",
-    )
-    p.add_argument("--stats", action="store_true", help="Print stats JSON to stderr")
-    p.add_argument(
-        "--force-text",
-        action="store_true",
-        help="Clean even when the input looks like a binary container "
-        "(this rewrites the bytes and will corrupt the file)",
-    )
-    p.add_argument(
-        "--in-place",
-        action="store_true",
-        help="Overwrite input file (creates .bak backup)",
-    )
-    args = p.parse_args()
-
-    text = read_text_input(args.path, allow_binary=args.force_text)
-    cleaned, stats = clean_text(
-        text,
-        nfkc=args.nfkc,
-        aggressive_homoglyphs=args.aggressive_homoglyphs,
-        normalize_spaces=not args.no_normalize_spaces,
-        strip_emoji_glue=args.strip_emoji_glue,
-        strip_bidi=args.strip_bidi,
-    )
-
-    out = args.output
-    if args.in_place:
-        if args.path in (None, "-"):
-            eprint("--in-place requires a file path")
-            return 2
-        src = Path(args.path)
-        backup_path(src)  # side effect: keep a .bak of the original before overwriting
-        out = str(src)
-    elif out is None and args.path not in (None, "-"):
-        out = str(cleaned_path(Path(args.path)))
-
-    write_text_output(cleaned, out)
-
-    if args.stats:
-        eprint(json.dumps(stats, indent=2, ensure_ascii=False))
-    else:
-        eprint(
-            f"removed={stats['removed_count']} replaced={stats['replaced_count']} "
-            f"len {stats['input_length']}->{stats['output_length']}"
-        )
-    return 0
-
-
-if __name__ == "__main__":
-    raise SystemExit(main())