scamscript 0.4.1__tar.gz → 0.5.0__tar.gz

{scamscript-0.4.1 → scamscript-0.5.0}/CHANGELOG.md

@@ -5,6 +5,18 @@ schema are the things that count as public API.
 
 ## Unreleased
 
+## 0.5.0 - 2026-11-02
+
+### Added
+
+- Content fingerprints and duplicate folding: reprints collapse to the strongest copy in reports,
+  with a folded count and `duplicate_of` in JSON.
+- `--dedupe/--no-dedupe` (on by default) and `--show-duplicates` to render every panel.
+
+### Changed
+
+- Duplicate panels are replaced by a one-line pointer unless `--show-duplicates` is set.
+
 ## 0.4.1 - 2026-11-02
 
 ### Fixed

{scamscript-0.4.1 → scamscript-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: scamscript
-Version: 0.4.1
+Version: 0.5.0
 Summary: Reverse-engineer scam scripts and vishing transcripts into structured counter-playbooks
 Project-URL: Homepage, https://github.com/sjdsjadjajdjsa/scamscript
 Project-URL: Repository, https://github.com/sjdsjadjajdjsa/scamscript
@@ -133,7 +133,8 @@ corpus is at [`docs/sample-report.md`](https://github.com/sjdsjadjajdjsa/scamscr
 | `prune` | Delete old logs |
 
 Useful flags: `--dry-run`, `--json`, `--redact`, `--top N`, `--recursive`, `--timeout`,
-`--retries`, `--max-tokens`, `--concurrency`, `--max-cost`, `--no-color`. `--json` puts one object per script on stdout and all
+`--retries`, `--max-tokens`, `--concurrency`, `--max-cost`, `--dedupe/--no-dedupe`,
+`--show-duplicates`, `--no-color`. `--json` puts one object per script on stdout and all
 Rich output on stderr, so it pipes:
 
 ```bash
@@ -154,6 +155,19 @@ The cap reads `usage.cost` from the response, so it works only where the provide
 says so when none is sent. Spend is printed and logged per run. In-flight requests can overshoot by
 up to the concurrency, so keep headroom on a hard limit.
 
+## Duplicates
+
+Corpora are full of reprints. Each script is fingerprinted on its body: headers, casing, punctuation
+and whitespace are ignored, so copies collapse. The strongest copy is ranked and the rest are listed
+under it; the report counts what was folded and JSON records carry `fingerprint` and `duplicate_of`.
+
+```bash
+scamscript batch corpus/ --output report.md          # folds by default
+scamscript batch corpus/ --no-dedupe --show-duplicates
+```
+
+Only byte-level twins fold. A copy with one changed account number stays separate, by design.
+
 ## How it works
 
 ```text
@@ -262,7 +276,7 @@ docs/           calibration record, sample report, banner, demo
 - Breakdowns are model readings; a bad prompt or a weak model produces confident nonsense. Read
   `opening_hook` and `red_flags` against the source before quoting anything.
 - Redaction is regex-based and will miss names and identifiers without a fixed shape.
-- One script per request; no cross-script deduplication yet, so reprints inflate a ranking.
+- Duplicate folding ignores formatting only; a single changed digit keeps a copy separate.
 - The cost cap depends on provider-reported usage and can overshoot by the in-flight count.
 - Analysis is English-centric: the prompt and schema assume English artifacts.
 - Quote checking compares text, not meaning. A legitimate paraphrase left in quotes is flagged;

{scamscript-0.4.1 → scamscript-0.5.0}/README.md

@@ -97,7 +97,8 @@ corpus is at [`docs/sample-report.md`](https://github.com/sjdsjadjajdjsa/scamscr
 | `prune` | Delete old logs |
 
 Useful flags: `--dry-run`, `--json`, `--redact`, `--top N`, `--recursive`, `--timeout`,
-`--retries`, `--max-tokens`, `--concurrency`, `--max-cost`, `--no-color`. `--json` puts one object per script on stdout and all
+`--retries`, `--max-tokens`, `--concurrency`, `--max-cost`, `--dedupe/--no-dedupe`,
+`--show-duplicates`, `--no-color`. `--json` puts one object per script on stdout and all
 Rich output on stderr, so it pipes:
 
 ```bash
@@ -118,6 +119,19 @@ The cap reads `usage.cost` from the response, so it works only where the provide
 says so when none is sent. Spend is printed and logged per run. In-flight requests can overshoot by
 up to the concurrency, so keep headroom on a hard limit.
 
+## Duplicates
+
+Corpora are full of reprints. Each script is fingerprinted on its body: headers, casing, punctuation
+and whitespace are ignored, so copies collapse. The strongest copy is ranked and the rest are listed
+under it; the report counts what was folded and JSON records carry `fingerprint` and `duplicate_of`.
+
+```bash
+scamscript batch corpus/ --output report.md          # folds by default
+scamscript batch corpus/ --no-dedupe --show-duplicates
+```
+
+Only byte-level twins fold. A copy with one changed account number stays separate, by design.
+
 ## How it works
 
 ```text
@@ -226,7 +240,7 @@ docs/           calibration record, sample report, banner, demo
 - Breakdowns are model readings; a bad prompt or a weak model produces confident nonsense. Read
   `opening_hook` and `red_flags` against the source before quoting anything.
 - Redaction is regex-based and will miss names and identifiers without a fixed shape.
-- One script per request; no cross-script deduplication yet, so reprints inflate a ranking.
+- Duplicate folding ignores formatting only; a single changed digit keeps a copy separate.
 - The cost cap depends on provider-reported usage and can overshoot by the in-flight count.
 - Analysis is English-centric: the prompt and schema assume English artifacts.
 - Quote checking compares text, not meaning. A legitimate paraphrase left in quotes is flagged;

{scamscript-0.4.1 → scamscript-0.5.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "scamscript"
-version = "0.4.1"
+version = "0.5.0"
 description = "Reverse-engineer scam scripts and vishing transcripts into structured counter-playbooks"
 readme = "README.md"
 requires-python = ">=3.12"

{scamscript-0.4.1 → scamscript-0.5.0}/src/scamscript/__init__.py

@@ -1,3 +1,3 @@
 """scamscript: reverse-engineering analyzer for scam call scripts."""
 
-__version__ = "0.4.1"
+__version__ = "0.5.0"

{scamscript-0.4.1 → scamscript-0.5.0}/src/scamscript/analysis.py

@@ -7,7 +7,7 @@ from dataclasses import dataclass
 from pathlib import Path
 from typing import Any
 
-from scamscript import prompts, redact, verify
+from scamscript import dedupe, prompts, redact, verify
 from scamscript.client import ChatClient, ClientError, TruncatedError
 from scamscript.ingest import ScriptFile
 from scamscript.models import BreakdownError, ScriptBreakdown, parse_breakdown
@@ -27,6 +27,8 @@ class Outcome:
     duration: float = 0.0
     redactions: dict[str, int] | None = None
     skipped: bool = False
+    fingerprint: str = ""
+    duplicate_of: str | None = None
     issues: list[verify.Issue] | None = None
     verification: dict[str, int] | None = None
 
@@ -41,6 +43,8 @@ class Outcome:
             "name": self.name,
             "ok": self.ok,
             "skipped": self.skipped,
+            "fingerprint": self.fingerprint,
+            "duplicate_of": self.duplicate_of,
             "attempts": self.attempts,
             "duration_s": round(self.duration, 3),
             "breakdown": self.breakdown.model_dump() if self.breakdown else None,
@@ -63,7 +67,9 @@ def analyze_script(
     redact_identifiers: bool = False,
     verify_quotes: bool = True,
 ) -> Outcome:
-    outcome = Outcome(source=script.path, name=script.name)
+    outcome = Outcome(
+        source=script.path, name=script.name, fingerprint=dedupe.fingerprint(script.text)
+    )
     started = time.monotonic()
 
     text = script.text

{scamscript-0.4.1 → scamscript-0.5.0}/src/scamscript/cli.py

@@ -52,6 +52,8 @@ class Context:
     json_output: bool
     verify_quotes: bool
     strict_quotes: bool
+    dedupe: bool
+    show_duplicates: bool
     console: Console
     err_console: Console
 
@@ -135,6 +137,16 @@ def main(
         "--strict-quotes",
         help="Exit non-zero when a quoted claim is not found in the source.",
     ),
+    dedupe: bool = typer.Option(
+        True,
+        "--dedupe/--no-dedupe",
+        help="Fold identical scripts into the strongest copy in reports.",
+    ),
+    show_duplicates: bool = typer.Option(
+        False,
+        "--show-duplicates",
+        help="Render a full panel for duplicate copies too.",
+    ),
     dry_run: bool = typer.Option(
         False,
         "--dry-run",
@@ -173,6 +185,8 @@ def main(
         json_output=json_output,
         verify_quotes=verify_quotes,
         strict_quotes=strict_quotes,
+        dedupe=dedupe,
+        show_duplicates=show_duplicates,
         console=Console(
             no_color=no_color,
             highlight=False,
@@ -278,6 +292,7 @@ def _run(
     spent = 0.0
     spent_seen = False
     lock = threading.Lock()
+    seen: dict[str, str] = {}
     client = ChatClient(
         api_key=api_key,
         base_url=settings.base_url,
@@ -308,6 +323,11 @@ def _run(
             redact_identifiers=settings.redact,
             verify_quotes=state.verify_quotes,
         )
+        with lock:
+            if state.dedupe and outcome.fingerprint:
+                outcome.duplicate_of = seen.setdefault(outcome.fingerprint, outcome.name)
+                if outcome.duplicate_of == outcome.name:
+                    outcome.duplicate_of = None
         cost = client.last_cost
         if cost:
             with lock:
@@ -320,6 +340,10 @@ def _run(
             _emit_json(outcome.as_record())
         elif outcome.skipped:
             console.print(f"[dim]skipped {outcome.name}: budget reached[/dim]")
+        elif outcome.duplicate_of and not state.show_duplicates:
+            console.print(
+                f"[dim]{outcome.name}: duplicate of {outcome.duplicate_of}[/dim]"
+            )
         elif outcome.breakdown is not None:
             console.print(
                 render_breakdown(
@@ -388,14 +412,17 @@ def _run(
         cost_usd=round(spent, 6),
     )
 
+    collapsed = reporting.collapse(report_entries) if state.dedupe else report_entries
+
     if batch_mode and (successes or skipped):
         if not state.json_output:
-            console.print(
-                render_summary_table(
-                    [(o.name, o.breakdown, o.error, o.skipped) for o in finished]
-                )
+            rows = (
+                [(e.name, e.breakdown, e.error, e.skipped) for e in collapsed]
+                if state.dedupe
+                else [(o.name, o.breakdown, o.error, o.skipped) for o in finished]
             )
-        ranked = reporting.rank(report_entries)
+            console.print(render_summary_table(rows))
+        ranked = reporting.rank(collapsed)
         if top:
             ranked = ranked[:top]
         markdown = reporting.build_markdown(

{scamscript-0.4.1 → scamscript-0.5.0}/src/scamscript/client.py

@@ -63,7 +63,7 @@ class ChatClient:
                 "Authorization": f"Bearer {api_key}",
                 "Content-Type": "application/json",
                 "Accept": "application/json",
-                "User-Agent": "scamscript/0.4.1",
+                "User-Agent": "scamscript/0.5.0",
             },
         )
 

scamscript-0.5.0/src/scamscript/dedupe.py

@@ -0,0 +1,29 @@
+"""Content fingerprints for collapsing reprinted scripts."""
+
+from __future__ import annotations
+
+import hashlib
+import re
+
+NON_WORD = re.compile(r"[^a-z0-9]+")
+HEADER_LINE = re.compile(r"^\s*#")
+
+
+def body(text: str) -> str:
+    """Drop the leading comment block a captured script usually carries."""
+    lines = text.splitlines()
+    index = 0
+    while index < len(lines) and (not lines[index].strip() or HEADER_LINE.match(lines[index])):
+        index += 1
+    return "\n".join(lines[index:])
+
+
+def normalize(text: str) -> str:
+    lowered = body(text).lower()
+    return " ".join(NON_WORD.sub(" ", lowered).split())
+
+
+def fingerprint(text: str) -> str:
+    """Short hash of the normalized body, stable across whitespace and punctuation."""
+    normalized = normalize(text)
+    return hashlib.sha256(normalized.encode("utf-8")).hexdigest()[:16]

{scamscript-0.4.1 → scamscript-0.5.0}/src/scamscript/report.py

@@ -30,6 +30,8 @@ class ReportEntry:
     error: str | None = None
     duration: float = 0.0
     skipped: bool = False
+    fingerprint: str = ""
+    duplicates: list[str] | None = None
     verification: dict[str, int] | None = None
     unverified: list[dict[str, str]] | None = None
 
@@ -44,6 +46,7 @@ def entry_from_outcome(outcome: Any) -> ReportEntry:
         source=str(getattr(outcome, "source", outcome.name)),
         breakdown=outcome.breakdown,
         skipped=getattr(outcome, "skipped", False),
+        fingerprint=getattr(outcome, "fingerprint", ""),
         error=outcome.error,
         duration=getattr(outcome, "duration", 0.0),
         verification=getattr(outcome, "verification", None),
@@ -71,6 +74,8 @@ def entries_from_run(run: dict[str, Any]) -> list[ReportEntry]:
                 source=record.get("source") or "",
                 breakdown=breakdown,
                 skipped=bool(record.get("skipped")),
+                fingerprint=record.get("fingerprint") or "",
+                duplicates=record.get("duplicates") or None,
                 error=None if breakdown else record.get("error") or "no breakdown stored",
                 duration=float(record.get("duration_s") or 0.0),
                 verification=record.get("verification") or None,
@@ -80,6 +85,29 @@ def entries_from_run(run: dict[str, Any]) -> list[ReportEntry]:
     return entries
 
 
+def collapse(entries: list[ReportEntry]) -> list[ReportEntry]:
+    """Keep the strongest copy of each fingerprint and list the rest under it."""
+    groups: dict[str, list[ReportEntry]] = {}
+    for entry in entries:
+        if entry.ok and entry.fingerprint:
+            groups.setdefault(entry.fingerprint, []).append(entry)
+
+    primaries: dict[str, ReportEntry] = {}
+    for key, group in groups.items():
+        if len(group) > 1:
+            best = max(group, key=lambda item: (item.breakdown.sophistication_score, item.breakdown.flag_count))
+            best.duplicates = [item.name for item in group if item is not best]
+            primaries[key] = best
+        else:
+            primaries[key] = group[0]
+
+    return [
+        primaries[entry.fingerprint] if entry.ok and entry.fingerprint else entry
+        for entry in entries
+        if not (entry.ok and entry.fingerprint and primaries[entry.fingerprint] is not entry)
+    ]
+
+
 def rank(entries: list[ReportEntry]) -> list[ReportEntry]:
     return sorted(
         entries,
@@ -118,6 +146,9 @@ def build_markdown(
     lines.append(f"- Failed: {len(failed)}")
     if skipped:
         lines.append(f"- Skipped at budget: {len(skipped)}")
+    folded = sum(len(entry.duplicates or []) for entry in ok)
+    if folded:
+        lines.append(f"- Duplicate copies folded: {folded}")
     unverified_total = sum(len(entry.unverified or []) for entry in ok)
     if unverified_total:
         lines.append(f"- Unverified quotes: {unverified_total}")
@@ -128,13 +159,13 @@ def build_markdown(
     if ok:
         lines.append("## Ranking by sophistication")
         lines.append("")
-        lines.append("| Rank | Script | Attack type | Sophistication | Red flags | Unverified |")
-        lines.append("| ---: | --- | --- | ---: | ---: | ---: |")
+        lines.append("| Rank | Script | Attack type | Sophistication | Red flags | Unverified | Copies |")
+        lines.append("| ---: | --- | --- | ---: | ---: | ---: | ---: |")
         for position, entry in enumerate(ok, start=1):
             lines.append(
                 f"| {position} | {_cell(entry.name)} | {_cell(entry.breakdown.attack_type)} | "
                 f"{entry.breakdown.sophistication_score}/10 | {entry.breakdown.flag_count} | "
-                f"{len(entry.unverified or [])} |"
+                f"{len(entry.unverified or [])} | {1 + len(entry.duplicates or [])} |"
             )
         lines.append("")
         for position, entry in enumerate(ok, start=1):
@@ -174,6 +205,8 @@ def _detail_section(position: int, entry: ReportEntry) -> list[str]:
     lines.append(f"- **Attack type:** {breakdown.attack_type}")
     lines.append(f"- **Sophistication:** {breakdown.sophistication_score}/10")
     lines.append(f"- **Red flags observed:** {breakdown.flag_count}")
+    if entry.duplicates:
+        lines.append(f"- **Identical copies folded:** {len(entry.duplicates)} ({', '.join(_cell(name) for name in entry.duplicates)})")
     counts = entry.verification or {}
     if counts.get("checked"):
         lines.append(

scamscript-0.5.0/tests/fixtures/reprints/copy-header.txt

@@ -0,0 +1,4 @@
+# third copy, header differs only
+CALLER: Good morning, this is the card security desk.
+CALLER: We have blocked two transactions on your card ending 4417.
+CALLER: Read me the last four digits and we will release the block.

scamscript-0.5.0/tests/fixtures/reprints/copy-punctuation.txt

@@ -0,0 +1,11 @@
+# second copy, reformatted
+
+# captured 2026-10-02, first copy
+
+Caller — Good morning, this is the card security desk.
+
+Caller — We have blocked two transactions on your card ending 4417!
+
+Caller — Read me the last four digits and we will release the block.
+
+

scamscript-0.5.0/tests/fixtures/reprints/near-twin.txt

@@ -0,0 +1,5 @@
+# near twin: one extra sentence, different card digits
+CALLER: Good morning, this is the card security desk.
+CALLER: We have blocked two transactions on your card ending 9902.
+CALLER: Read me the last four digits and we will release the block.
+CALLER: If we do not clear it now, the block stays until you visit a branch.

scamscript-0.5.0/tests/fixtures/reprints/original.txt

@@ -0,0 +1,4 @@
+# captured 2026-10-02, first copy
+CALLER: Good morning, this is the card security desk.
+CALLER: We have blocked two transactions on your card ending 4417.
+CALLER: Read me the last four digits and we will release the block.

{scamscript-0.4.1 → scamscript-0.5.0}/tests/test_cli.py

@@ -1,4 +1,6 @@
 import json
+import re
+from pathlib import Path
 
 import pytest
 from typer.testing import CliRunner
@@ -228,7 +230,7 @@ def test_dry_run_without_endpoint(home, tmp_path, monkeypatch):
 def test_version(home):
     result = runner.invoke(cli.app, ["--version"])
     assert result.exit_code == 0
-    assert result.stdout.strip() == "scamscript 0.4.1"
+    assert result.stdout.strip() == "scamscript 0.5.0"
 
 
 def test_json_output_is_one_object_per_script(home, tmp_path, fake):
@@ -409,3 +411,47 @@ def test_max_cost_warns_when_provider_reports_nothing(home, tmp_path, fake):
     )
     assert result.exit_code == 0
     assert "could not be enforced" in result.stdout
+
+
+def _reprints(tmp_path):
+    src = Path(__file__).parent / "fixtures" / "reprints"
+    for item in src.iterdir():
+        (tmp_path / item.name).write_text(item.read_text(), encoding="utf-8")
+
+
+def test_dedupe_folds_reprints(home, tmp_path, fake):
+    _reprints(tmp_path)
+    scores = {"original": 4, "copy-punctuation": 9, "copy-header": 6, "near-twin": 8}
+    for key, score in scores.items():
+        fake.replies[key] = breakdown(score=score)
+    out = tmp_path / "r.md"
+    result = runner.invoke(cli.app, ["--api-key", "k", "batch", str(tmp_path), "--output", str(out)])
+    assert result.exit_code == 0
+    markdown = out.read_text()
+    assert "Duplicate copies folded: 2" in markdown
+    assert len(re.findall(r"^## \d+\. ", markdown, re.M)) == 2
+    assert "| 1 | copy-punctuation.txt |" in markdown
+    assert "near-twin.txt" in markdown
+    assert "copy-header.txt" in markdown
+
+
+def test_no_dedupe_keeps_every_copy(home, tmp_path, fake):
+    _reprints(tmp_path)
+    out = tmp_path / "r.md"
+    result = runner.invoke(
+        cli.app, ["--api-key", "k", "--no-dedupe", "batch", str(tmp_path), "--output", str(out)]
+    )
+    assert result.exit_code == 0
+    text = out.read_text()
+    assert "Duplicate copies folded" not in text
+    assert len(re.findall(r"^\| \d+ \|", text, re.M)) == 4
+
+
+def test_json_marks_duplicates(home, tmp_path, fake):
+    _reprints(tmp_path)
+    result = runner.invoke(cli.app, ["--api-key", "k", "--json", "batch", str(tmp_path)])
+    assert result.exit_code == 0
+    records = [json.loads(line) for line in result.stdout.splitlines() if line.strip()]
+    results = [record for record in records if record.get("event") == "result"]
+    assert all(record["fingerprint"] for record in results)
+    assert sum(1 for record in results if record["duplicate_of"]) == 2

scamscript-0.5.0/tests/test_dedupe.py

@@ -0,0 +1,60 @@
+from pathlib import Path
+
+from scamscript.dedupe import fingerprint, normalize
+from scamscript.report import ReportEntry, collapse
+
+FIXTURES = Path(__file__).parent / "fixtures" / "reprints"
+
+
+def read(name):
+    return (FIXTURES / name).read_text()
+
+
+def test_header_and_punctuation_ignored():
+    assert fingerprint(read("original.txt")) == fingerprint(read("copy-punctuation.txt"))
+    assert fingerprint(read("original.txt")) == fingerprint(read("copy-header.txt"))
+
+
+def test_near_twin_differs():
+    assert fingerprint(read("original.txt")) != fingerprint(read("near-twin.txt"))
+
+
+def test_normalize_collapses_whitespace():
+    assert normalize("a  b\n\tc") == "a b c"
+
+
+def test_empty_and_header_only_stable():
+    assert fingerprint("") == fingerprint("# only a header\n")
+
+
+def entry(name, score, fp="abc"):
+    from scamscript.models import ScriptBreakdown
+
+    breakdown = ScriptBreakdown(
+        attack_type="vishing",
+        persona="p",
+        opening_hook="h",
+        trust_building_moves=["a"],
+        urgency_tactics=["b"],
+        escalation_ladder=["c"],
+        target_demographics="d",
+        red_flags=["e"],
+        defense_summary="f",
+        sophistication_score=score,
+    )
+    return ReportEntry(name=name, source=name, breakdown=breakdown, fingerprint=fp)
+
+
+def test_collapse_keeps_strongest_and_lists_rest():
+    entries = [entry("a", 4), entry("b", 9), entry("c", 6)]
+    collapsed = collapse(entries)
+    assert len(collapsed) == 1
+    assert collapsed[0].name == "b"
+    assert collapsed[0].duplicates == ["a", "c"]
+
+
+def test_collapse_leaves_distinct_and_failed_alone():
+    entries = [entry("a", 5), entry("twin", 8, fp="zzz"), ReportEntry(name="bad", source="bad")]
+    collapsed = collapse(entries)
+    assert [item.name for item in collapsed] == ["a", "twin", "bad"]
+    assert collapsed[0].duplicates is None