memorymaster 3.22.0__tar.gz → 3.23.0__tar.gz

{memorymaster-3.22.0/memorymaster.egg-info → memorymaster-3.23.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: memorymaster
-Version: 3.22.0
+Version: 3.23.0
 Summary: Production-grade memory reliability system for AI coding agents. Lifecycle-managed claims with citations, conflict detection, steward governance, and MCP integration.
 Author: wolverin0
 License: MIT
@@ -52,7 +52,7 @@ Lifecycle-managed claims with citations, conflict detection, steward governance,
 
 [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
 [![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
-[![Tests](https://img.shields.io/badge/tests-2214-green.svg)]()
+[![Tests](https://img.shields.io/badge/tests-2224-green.svg)]()
 [![MCP Tools](https://img.shields.io/badge/MCP%20tools-24-purple.svg)]()
 [![CLI Commands](https://img.shields.io/badge/CLI%20commands-86-orange.svg)]()
 [![PyPI](https://img.shields.io/pypi/v/memorymaster.svg)](https://pypi.org/project/memorymaster/)
@@ -91,7 +91,8 @@ recent PR status, and sensitivity-filter invariants.
 - **Correction mining** (new in v3.21.0): `mine-rules` scans the verbatim transcript archive for user corrections and distills them into rule claims; the Stop hook also mines each session's latest correction automatically
 - **Versioned schema migrations** (new in v3.20.0): `migrate` applies SQLite/Postgres migrations with sha256 drift detection; incremental `export-delta` ships small claim deltas for cheap cross-machine sync
 - **Retrieval quality** (new in v3.22.0): floor-ratio boost gate (`MEMORYMASTER_BOOST_FLOOR_RATIO`) stops fresh-but-wrong claims outranking the true match; `query --explain` shows per-stage score attribution; an opt-in correctness-safe query cache (`MEMORYMASTER_QUERY_CACHE`) with a generation gate
-- **Semantic contradiction probe** (new in v3.22.0): `detect-contradictions` finds claims that genuinely contradict each other (beyond the deterministic same-subject conflict check) via an LLM judge with a Wilson-CI rate and verdict cache
+- **Semantic contradiction probe** (new in v3.22.0, wired as a steward phase in v3.23.0): `detect-contradictions` finds claims that genuinely contradict each other (beyond the deterministic same-subject conflict check) via an LLM judge with a Wilson-CI rate and verdict cache; in v3.23 the same probe runs inside `run-steward` and emits paste-ready `conflicted` proposals
+- **Verbatim archive cleanup** (new in v3.23.0): `verbatim-cleanup` dedups the raw-transcript table and optionally purges pre-#128 junk rows, with a dry-run default and FTS5 mirror sync
 - **Steward governance**: multi-probe validators (filesystem, format, citation, semantic, tool) with proposal review
 - **Conflict resolution**: 5-tier auto (confidence > freshness > citations > LLM > manual)
 - **Auto-redaction** at ingest: JWT, GitHub tokens, Bearer, AWS keys, SSH keys, custom patterns

{memorymaster-3.22.0 → memorymaster-3.23.0}/README.md

@@ -6,7 +6,7 @@ Lifecycle-managed claims with citations, conflict detection, steward governance,
 
 [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
 [![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
-[![Tests](https://img.shields.io/badge/tests-2214-green.svg)]()
+[![Tests](https://img.shields.io/badge/tests-2224-green.svg)]()
 [![MCP Tools](https://img.shields.io/badge/MCP%20tools-24-purple.svg)]()
 [![CLI Commands](https://img.shields.io/badge/CLI%20commands-86-orange.svg)]()
 [![PyPI](https://img.shields.io/pypi/v/memorymaster.svg)](https://pypi.org/project/memorymaster/)
@@ -45,7 +45,8 @@ recent PR status, and sensitivity-filter invariants.
 - **Correction mining** (new in v3.21.0): `mine-rules` scans the verbatim transcript archive for user corrections and distills them into rule claims; the Stop hook also mines each session's latest correction automatically
 - **Versioned schema migrations** (new in v3.20.0): `migrate` applies SQLite/Postgres migrations with sha256 drift detection; incremental `export-delta` ships small claim deltas for cheap cross-machine sync
 - **Retrieval quality** (new in v3.22.0): floor-ratio boost gate (`MEMORYMASTER_BOOST_FLOOR_RATIO`) stops fresh-but-wrong claims outranking the true match; `query --explain` shows per-stage score attribution; an opt-in correctness-safe query cache (`MEMORYMASTER_QUERY_CACHE`) with a generation gate
-- **Semantic contradiction probe** (new in v3.22.0): `detect-contradictions` finds claims that genuinely contradict each other (beyond the deterministic same-subject conflict check) via an LLM judge with a Wilson-CI rate and verdict cache
+- **Semantic contradiction probe** (new in v3.22.0, wired as a steward phase in v3.23.0): `detect-contradictions` finds claims that genuinely contradict each other (beyond the deterministic same-subject conflict check) via an LLM judge with a Wilson-CI rate and verdict cache; in v3.23 the same probe runs inside `run-steward` and emits paste-ready `conflicted` proposals
+- **Verbatim archive cleanup** (new in v3.23.0): `verbatim-cleanup` dedups the raw-transcript table and optionally purges pre-#128 junk rows, with a dry-run default and FTS5 mirror sync
 - **Steward governance**: multi-probe validators (filesystem, format, citation, semantic, tool) with proposal review
 - **Conflict resolution**: 5-tier auto (confidence > freshness > citations > LLM > manual)
 - **Auto-redaction** at ingest: JWT, GitHub tokens, Bearer, AWS keys, SSH keys, custom patterns

{memorymaster-3.22.0 → memorymaster-3.23.0}/memorymaster/__init__.py

@@ -2,4 +2,4 @@
 
 __all__ = ["__version__"]
 
-__version__ = "3.22.0"
+__version__ = "3.23.0"

{memorymaster-3.22.0 → memorymaster-3.23.0}/memorymaster/cli.py

@@ -332,6 +332,7 @@ def build_parser() -> argparse.ArgumentParser:
     steward.add_argument("--probe-failure-threshold", type=int, default=3, help="Open circuit breaker for a probe type after this many timeout/error failures")
     steward.add_argument("--disable-semantic-probe", action="store_true", help="Disable semantic retrieval probe in steward planner")
     steward.add_argument("--disable-tool-probe", action="store_true", help="Disable tool/storage probe in steward planner")
+    steward.add_argument("--disable-contradiction-probe", action="store_true", help="Disable semantic contradiction probe in steward planner (v3.23)")
     steward.add_argument("--allow-sensitive", action="store_true", help="Include sensitive claims in stewardship scan")
     steward.add_argument("--apply", action="store_true", help="Apply proposed status transitions")
     steward.add_argument("--artifact-json", default="artifacts/steward/steward_report.json", help="Path to steward JSON report artifact")
@@ -464,6 +465,12 @@ def build_parser() -> argparse.ArgumentParser:
     mine_rules_cmd.add_argument("--provider", default="claude_cli", help="LLM provider for this run (default: claude_cli)")
     mine_rules_cmd.add_argument("--reset", action="store_true", help="Clear the stored watermark before running (re-scan from the start)")
 
+    verbatim_clean = sub.add_parser("verbatim-cleanup", help="Dedup the verbatim archive + optionally purge pre-#128 capture-bug junk (v3.23)")
+    verbatim_clean.add_argument("--analyze-only", dest="analyze_only", action="store_true", help="Report composition only; do not delete")
+    verbatim_clean.add_argument("--apply", action="store_true", help="Actually delete (default is dry-run)")
+    verbatim_clean.add_argument("--no-dedup", dest="no_dedup", action="store_true", help="Skip the (session_id, content) exact-dup pass")
+    verbatim_clean.add_argument("--purge-junk", dest="purge_junk", action="store_true", help="Also delete rows matching known pre-#128 junk prefixes (wiki-absorb prompt, etc.)")
+
     detect_contra = sub.add_parser("detect-contradictions", help="Find semantic contradictions between topically-similar claims via an LLM judge (v3.22)")
     detect_contra.add_argument("--limit", type=int, default=200, help="Max claims to load for pair sampling")
     detect_contra.add_argument("--sample", type=int, default=50, help="Max candidate pairs to judge this run (caps LLM calls)")

{memorymaster-3.22.0 → memorymaster-3.23.0}/memorymaster/cli_handlers_basic.py

@@ -1228,6 +1228,7 @@ def _handle_run_steward(args: argparse.Namespace, service, parser: argparse.Argu
         probe_failure_threshold=args.probe_failure_threshold,
         enable_semantic_probe=not args.disable_semantic_probe,
         enable_tool_probe=not args.disable_tool_probe,
+        enable_contradiction_probe=not getattr(args, "disable_contradiction_probe", False),
         artifact_path=Path(args.artifact_json),
     )
     elapsed_ms = (time.perf_counter() - t0) * 1000

{memorymaster-3.22.0 → memorymaster-3.23.0}/memorymaster/cli_handlers_curation.py

@@ -397,6 +397,42 @@ def _handle_detect_contradictions(args: argparse.Namespace, service, parser: arg
     return 0
 
 
+def _handle_verbatim_cleanup(args: argparse.Namespace, service, parser: argparse.ArgumentParser, effective_db: str) -> int:
+    from memorymaster.verbatim_cleanup import analyze, cleanup
+    t0 = time.perf_counter()
+    if getattr(args, "analyze_only", False):
+        result = analyze(effective_db)
+    else:
+        result = cleanup(
+            effective_db,
+            dedup=not getattr(args, "no_dedup", False),
+            purge_junk=getattr(args, "purge_junk", False),
+            dry_run=not getattr(args, "apply", False),
+        )
+    elapsed_ms = (time.perf_counter() - t0) * 1000
+    if args.json_output:
+        print(_json_envelope(result, query_ms=elapsed_ms))
+    else:
+        if not result.get("verbatim_present", True):
+            print("verbatim-cleanup: verbatim_memories table not present; nothing to do")
+        elif "before_total" in result:  # cleanup result
+            tag = "(dry-run)" if result["dry_run"] else "(applied)"
+            print(
+                f"verbatim-cleanup {tag}: dedup_deleted={result['dedup_deleted']}, "
+                f"junk_deleted={result['junk_deleted']}, "
+                f"before={result['before_total']} -> after={result['after_total']} "
+                f"({elapsed_ms:.0f}ms)"
+            )
+        else:  # analyze result
+            print(
+                f"verbatim analyze: total={result['total']}, distinct={result['distinct_content']}, "
+                f"duplicate_extras={result['duplicate_extras']}, "
+                f"empty_role={result['empty_role_rows']}, junk_prefix={result['junk_prefix_rows']} "
+                f"({elapsed_ms:.0f}ms)"
+            )
+    return 0
+
+
 def _handle_wiki_breakdown(args: argparse.Namespace, service, parser: argparse.ArgumentParser, effective_db: str) -> int:
     from memorymaster.wiki_engine import breakdown
     t0 = time.perf_counter()
@@ -759,6 +795,7 @@ COMMAND_HANDLERS: dict[str, object] = {
     "mine-transcript": _handle_mine_transcript,
     "mine-rules": _handle_mine_rules,
     "detect-contradictions": _handle_detect_contradictions,
+    "verbatim-cleanup": _handle_verbatim_cleanup,
     "verify-claims": _handle_verify_claims,
     "extract-entities": _handle_extract_entities,
     "entity-stats": _handle_entity_stats,

{memorymaster-3.22.0 → memorymaster-3.23.0}/memorymaster/contradiction_probe.py

@@ -26,6 +26,7 @@ import logging
 import math
 import os
 import sqlite3
+import time
 from datetime import datetime, timezone
 from typing import Any
 
@@ -33,6 +34,7 @@ from memorymaster import llm_budget, llm_provider
 from memorymaster.embeddings import EmbeddingProvider, cosine_similarity, create_best_provider
 from memorymaster.lifecycle import transition_claim
 from memorymaster.models import Claim
+from memorymaster.security import redact_text
 
 logger = logging.getLogger(__name__)
 
@@ -324,3 +326,130 @@ def run_probe(
         lo, hi = wilson_interval(stats["contradictions"], n)
         stats["rate_ci"] = [round(lo, 4), round(hi, 4)]
     return stats
+
+
+# ---------------------------------------------------------------------------
+# Per-claim steward-phase entry point (v3.23)
+# ---------------------------------------------------------------------------
+
+
+def probe_for_claim(
+    service: Any,
+    claim: Any,
+    *,
+    sim_low: float = 0.60,
+    sim_high: float = 0.92,
+    max_pairs: int = 5,
+    peer_limit: int = 20,
+) -> dict[str, Any]:
+    """Per-claim contradiction probe for the steward cycle.
+
+    Finds topically-similar peers for ``claim`` via the existing hybrid
+    retrieval (so we reuse the embedder + ranker + cache), excludes pairs the
+    deterministic resolver / supersession owns, judges remaining candidates
+    against the verdict cache + LLM, and returns a normalized result dict the
+    steward wraps in a ``ProbeResult``.
+
+    Returns ``{passed, reasons: [...], metrics: {...}}``. ``passed`` is False
+    iff at least one contradiction was found.
+    """
+    started = time.monotonic()
+    metrics: dict[str, Any] = {
+        "pairs_checked": 0, "contradictions": 0,
+        "cache_hits": 0, "llm_calls": 0, "errors": 0,
+        "timed_out": False, "duration_ms": 0.0,
+    }
+    reasons: list[dict[str, Any]] = []
+
+    def _done(passed: bool) -> dict[str, Any]:
+        metrics["duration_ms"] = round((time.monotonic() - started) * 1000.0, 3)
+        return {"passed": passed, "reasons": reasons, "metrics": metrics}
+
+    if claim.status in _SKIP_STATUSES or not (claim.text or "").strip():
+        return _done(True)
+
+    try:
+        peers_rows = service.query_rows(
+            query_text=claim.text or "",
+            limit=peer_limit,
+            retrieval_mode="hybrid",
+            include_candidates=True,
+            include_stale=True,
+            include_conflicted=True,
+        )
+    except Exception as exc:
+        reasons.append({
+            "code": "contradiction_probe.peer_fetch.error",
+            "severity": "low",
+            "detail": f"peer fetch failed: {type(exc).__name__}",
+            "evidence": {},
+        })
+        return _done(False)
+
+    candidates: list[tuple[Any, float]] = []
+    for row in peers_rows:
+        peer = row.get("claim")
+        if peer is None or peer.id == claim.id:
+            continue
+        if peer.status in _SKIP_STATUSES:
+            continue
+        if _same_subject_predicate(claim, peer):
+            continue  # deterministic resolver's domain
+        if _already_linked(claim, peer):
+            continue
+        vs = float(row.get("vector_score") or 0.0)
+        # If vectors are present, only accept pairs in the contradiction band.
+        # If absent (pure-lexical hybrid), accept all topical peers.
+        if vs > 0.0 and not (sim_low <= vs < sim_high):
+            continue
+        candidates.append((peer, vs))
+        if len(candidates) >= max_pairs:
+            break
+
+    if not candidates:
+        return _done(True)
+
+    db_path = getattr(service.store, "db_path", None)
+    if not db_path or "://" in str(db_path):
+        return _done(True)  # verdict cache requires SQLite; skip silently on PG
+
+    conn = sqlite3.connect(str(db_path))
+    try:
+        _ensure_verdict_table(conn)
+        model = _model_key()
+        for peer, sim in candidates:
+            metrics["pairs_checked"] += 1
+            verdict = _cache_get(conn, claim.id, peer.id, model)
+            if verdict is not None:
+                metrics["cache_hits"] += 1
+            else:
+                try:
+                    verdict = _judge_llm(claim, peer)
+                except llm_budget.LLMBudgetExceeded:
+                    metrics["timed_out"] = True
+                    break
+                metrics["llm_calls"] += 1
+                if verdict is None:
+                    metrics["errors"] += 1
+                    continue
+                _cache_put(conn, claim.id, peer.id, model, verdict)
+            if not verdict.get("contradicts"):
+                continue
+            verdict_reason = verdict.get("reason", "")
+            _, leaks = redact_text(verdict_reason)
+            if leaks:
+                continue  # drop rather than surface a sensitive judge-reason
+            metrics["contradictions"] += 1
+            reasons.append({
+                "code": "contradiction_probe.semantic_pair",
+                "severity": verdict.get("severity", "medium"),
+                "detail": f"Semantic contradiction with claim {peer.id}: {verdict_reason}",
+                "evidence": {
+                    "peer_claim_id": int(peer.id),
+                    "similarity": round(float(sim), 4),
+                    "from_cache": bool(verdict.get("cached")),
+                },
+            })
+    finally:
+        conn.close()
+    return _done(metrics["contradictions"] == 0)

{memorymaster-3.22.0 → memorymaster-3.23.0}/memorymaster/steward.py

@@ -24,6 +24,7 @@ ProbeType = Literal[
     "deterministic_citation_locator",
     "semantic_probe",
     "tool_probe",
+    "contradiction_probe",
 ]
 
 _TEXT_SUFFIXES = {
@@ -601,6 +602,46 @@ def _run_tool_probe(*, claim: Any, service: MemoryService, timeout_seconds: floa
     )
 
 
+def _run_contradiction_probe(
+    *, claim: Any, service: MemoryService, max_pairs: int, sim_low: float, sim_high: float,
+) -> ProbeResult:
+    """Steward-phase wrapper around contradiction_probe.probe_for_claim.
+
+    Converts the probe's normalized dict output into a ProbeResult/Reasons.
+    Fail-closed: any unexpected error returns a low-severity error reason so
+    the circuit breaker can open the probe rather than crashing the cycle.
+    """
+    from memorymaster import contradiction_probe as cp
+    started = time.monotonic()
+    try:
+        out = cp.probe_for_claim(
+            service, claim, sim_low=sim_low, sim_high=sim_high, max_pairs=max_pairs,
+        )
+    except Exception as exc:
+        return ProbeResult(
+            probe_type="contradiction_probe", passed=False,
+            metrics={
+                "timed_out": False,
+                "duration_ms": round((time.monotonic() - started) * 1000.0, 3),
+                "error": str(exc)[:120],
+            },
+            reasons=[Reason(
+                code="contradiction_probe.error", probe_type="contradiction_probe",
+                severity="low", detail=f"probe crashed: {type(exc).__name__}",
+            )],
+        )
+    return ProbeResult(
+        probe_type="contradiction_probe",
+        passed=bool(out["passed"]),
+        metrics=out["metrics"],
+        reasons=[Reason(
+            code=r["code"], probe_type="contradiction_probe",
+            severity=r.get("severity", "medium"), detail=r.get("detail", ""),
+            evidence=r.get("evidence", {}),
+        ) for r in out["reasons"]],
+    )
+
+
 def plan_claim_probes(
     claim: Any,
     *,
@@ -608,6 +649,7 @@ def plan_claim_probes(
     max_file_bytes: int,
     include_semantic_probe: bool = True,
     include_tool_probe: bool = True,
+    include_contradiction_probe: bool = True,
 ) -> ClaimProbePlan:
     needle = _normalize_text(claim.object_value) or _normalize_text(claim.text)
     if len(needle) > 160:
@@ -628,6 +670,8 @@ def plan_claim_probes(
         probes.append(ProbeSpec(probe_type="semantic_probe", params={"limit": 8}))
     if include_tool_probe:
         probes.append(ProbeSpec(probe_type="tool_probe", params={}))
+    if include_contradiction_probe:
+        probes.append(ProbeSpec(probe_type="contradiction_probe", params={"max_pairs": 5}))
     return ClaimProbePlan(claim_id=claim.id, probes=probes)
 
 
@@ -709,6 +753,12 @@ def _decision_for_claim(
     proposed_status: str | None = None
     replaced_by_claim_id: int | None = None
 
+    contradiction_found = any(
+        result.probe_type == "contradiction_probe" and not result.passed
+        and any(r.code == "contradiction_probe.semantic_pair" for r in result.reasons)
+        for result in probe_results
+    )
+
     if replacement is not None:
         decision = "superseded_candidate"
         proposed_status = "superseded"
@@ -722,17 +772,20 @@ def _decision_for_claim(
                 evidence={"replaced_by_claim_id": replacement.id},
             )
         )
-    elif claim.status == "conflicted" or has_conflict:
+    elif claim.status == "conflicted" or has_conflict or contradiction_found:
         decision = "conflicted"
         proposed_status = "conflicted"
-        reasons.append(
-            Reason(
-                code="relation.conflicting_claim",
-                probe_type="relation",
-                severity="high",
-                detail="Conflicting confirmed claim exists for same tuple.",
+        if has_conflict or claim.status == "conflicted":
+            reasons.append(
+                Reason(
+                    code="relation.conflicting_claim",
+                    probe_type="relation",
+                    severity="high",
+                    detail="Conflicting confirmed claim exists for same tuple.",
+                )
             )
-        )
+        # contradiction_probe.semantic_pair reasons are already in `reasons` via
+        # the probe_results loop above — no need to duplicate.
     else:
         no_match = any(reason.code == "filesystem_grep.no_match" for reason in reasons)
         weak_format = any(
@@ -852,6 +905,7 @@ def _run_cycle(
     probe_failure_threshold: int,
     enable_semantic_probe: bool,
     enable_tool_probe: bool,
+    enable_contradiction_probe: bool,
 ) -> dict[str, Any]:
     before = _status_snapshot(service, allow_sensitive=allow_sensitive)
 
@@ -900,6 +954,7 @@ def _run_cycle(
             max_file_bytes=max_probe_file_bytes,
             include_semantic_probe=enable_semantic_probe,
             include_tool_probe=enable_tool_probe,
+            include_contradiction_probe=enable_contradiction_probe,
         )
         probe_results: list[ProbeResult] = []
         for probe in plan.probes:
@@ -973,6 +1028,14 @@ def _run_cycle(
                     service=service,
                     timeout_seconds=probe_timeout_seconds,
                 )
+            elif probe.probe_type == "contradiction_probe":
+                result = _run_contradiction_probe(
+                    claim=claim,
+                    service=service,
+                    max_pairs=int(probe.params.get("max_pairs", 5)),
+                    sim_low=float(probe.params.get("sim_low", 0.60)),
+                    sim_high=float(probe.params.get("sim_high", 0.92)),
+                )
 
             if result is None:
                 continue
@@ -1447,6 +1510,7 @@ def run_steward(
     probe_failure_threshold: int = 3,
     enable_semantic_probe: bool = True,
     enable_tool_probe: bool = True,
+    enable_contradiction_probe: bool = True,
     artifact_path: Path | str = "artifacts/steward/steward_report.json",
 ) -> dict[str, Any]:
     if mode not in {"manual", "cadence"}:
@@ -1551,6 +1615,7 @@ def run_steward(
                 probe_failure_threshold=probe_failure_threshold,
                 enable_semantic_probe=enable_semantic_probe,
                 enable_tool_probe=enable_tool_probe,
+                enable_contradiction_probe=enable_contradiction_probe,
             )
         cycle_payload["cycle"] = cycle_index
         cycle_payload["started_at"] = cycle_started
@@ -1589,6 +1654,7 @@ def run_steward(
         "probe_failure_threshold": probe_failure_threshold,
         "enable_semantic_probe": enable_semantic_probe,
         "enable_tool_probe": enable_tool_probe,
+        "enable_contradiction_probe": enable_contradiction_probe,
         "max_tool_probes": max_tool_probes,
         "cycles_completed": len(cycles),
         "cycles": cycles,
@@ -1614,6 +1680,7 @@ def run_steward(
                 "probe_failure_threshold": probe_failure_threshold,
                 "enable_semantic_probe": enable_semantic_probe,
                 "enable_tool_probe": enable_tool_probe,
+                "enable_contradiction_probe": enable_contradiction_probe,
             },
             "cadence": {
                 "mode": mode,

memorymaster-3.23.0/memorymaster/verbatim_cleanup.py

@@ -0,0 +1,175 @@
+"""Verbatim archive cleanup (v3.23).
+
+The verbatim_memories table accumulates two distinct kinds of bloat:
+
+1. **Junk from the pre-#128 capture bug**: the Stop hook's store_transcript
+   used to read top-level entry fields (the actual turns were nested under
+   ``message``), so the only rows it stored were a handful of non-conversation
+   metadata lines per transcript — most of them duplicated internal-LLM prompts
+   like "Rewrite ONLY the compiled truth section…". Tens of thousands of rows
+   carrying the same internal prompt text, with empty ``role``.
+
+2. **Duplicate insertions**: even after the capture fix, re-running
+   store_transcript on the same JSONL is dedup-checked per (session_id, content),
+   but cross-session duplicates and pre-WAL-fix collisions persist.
+
+This module reports + reclaims both. It DELETES rows directly (verbatim has no
+foreign keys), and reflushes the FTS5 mirror so search stays consistent. Dry-run
+by default; ``--apply`` to actually delete.
+"""
+from __future__ import annotations
+
+import logging
+import sqlite3
+from typing import Any
+
+logger = logging.getLogger(__name__)
+
+# Known pre-#128 junk content prefixes (internal LLM prompts that the broken
+# capture path mistakenly stored as if they were conversation turns).
+_JUNK_PREFIXES: tuple[str, ...] = (
+    "Rewrite ONLY the compiled truth",
+    "You are a memory curator",
+    "You are an expert at compiling",
+)
+
+
+def _connect(db_path: str) -> sqlite3.Connection:
+    conn = sqlite3.connect(db_path)
+    conn.row_factory = sqlite3.Row
+    conn.execute("PRAGMA journal_mode = WAL")
+    return conn
+
+
+def _verbatim_present(conn: sqlite3.Connection) -> bool:
+    row = conn.execute(
+        "SELECT 1 FROM sqlite_master WHERE type='table' AND name='verbatim_memories'"
+    ).fetchone()
+    return row is not None
+
+
+def _fts_present(conn: sqlite3.Connection) -> bool:
+    row = conn.execute(
+        "SELECT 1 FROM sqlite_master WHERE type='table' AND name='verbatim_fts'"
+    ).fetchone()
+    return row is not None
+
+
+def analyze(db_path: str) -> dict[str, Any]:
+    """Report verbatim composition without touching anything.
+
+    Returns counts: total rows, distinct contents, exact duplicates (per
+    (session_id, content)), pre-#128 junk rows (matching known prefixes), and
+    rows with empty role (legacy capture-bug signature).
+    """
+    if "://" in str(db_path):
+        raise ValueError("verbatim cleanup is SQLite-only")
+    conn = _connect(db_path)
+    try:
+        if not _verbatim_present(conn):
+            return {"verbatim_present": False}
+        total = conn.execute("SELECT COUNT(*) FROM verbatim_memories").fetchone()[0]
+        distinct_content = conn.execute(
+            "SELECT COUNT(DISTINCT content) FROM verbatim_memories"
+        ).fetchone()[0]
+        # Exact duplicates by (session_id, content): the "extra" copies beyond
+        # the first per group are deletable.
+        dup_extras = conn.execute(
+            """SELECT COALESCE(SUM(c - 1), 0) FROM (
+                   SELECT COUNT(*) AS c FROM verbatim_memories
+                   GROUP BY session_id, content HAVING c > 1
+               )"""
+        ).fetchone()[0]
+        empty_role = conn.execute(
+            "SELECT COUNT(*) FROM verbatim_memories WHERE COALESCE(role, '') = ''"
+        ).fetchone()[0]
+        junk = 0
+        for prefix in _JUNK_PREFIXES:
+            junk += conn.execute(
+                "SELECT COUNT(*) FROM verbatim_memories WHERE content LIKE ?",
+                (prefix + "%",),
+            ).fetchone()[0]
+        return {
+            "verbatim_present": True,
+            "total": int(total),
+            "distinct_content": int(distinct_content),
+            "duplicate_extras": int(dup_extras),
+            "empty_role_rows": int(empty_role),
+            "junk_prefix_rows": int(junk),
+            "junk_prefixes": list(_JUNK_PREFIXES),
+        }
+    finally:
+        conn.close()
+
+
+def cleanup(
+    db_path: str,
+    *,
+    dedup: bool = True,
+    purge_junk: bool = False,
+    dry_run: bool = True,
+) -> dict[str, Any]:
+    """Reclaim space. Default is dry-run (counts only); ``dry_run=False`` to
+    actually delete. ``purge_junk`` is opt-in because it targets a known-bad
+    pattern set — review with ``analyze`` first.
+
+    Returns counts of what was (or would be) removed, and the new total.
+    """
+    if "://" in str(db_path):
+        raise ValueError("verbatim cleanup is SQLite-only")
+    conn = _connect(db_path)
+    try:
+        if not _verbatim_present(conn):
+            return {"verbatim_present": False}
+        out: dict[str, Any] = {
+            "dry_run": bool(dry_run),
+            "dedup_deleted": 0,
+            "junk_deleted": 0,
+            "before_total": int(conn.execute("SELECT COUNT(*) FROM verbatim_memories").fetchone()[0]),
+        }
+        has_fts = _fts_present(conn)
+        deleted_ids: list[int] = []
+
+        if dedup:
+            # Identify ids to drop: for each (session_id, content) group, keep
+            # the smallest id (oldest), drop the rest.
+            rows = conn.execute(
+                """SELECT id FROM verbatim_memories WHERE id NOT IN (
+                       SELECT MIN(id) FROM verbatim_memories GROUP BY session_id, content
+                   )"""
+            ).fetchall()
+            ids = [int(r[0]) for r in rows]
+            out["dedup_deleted"] = len(ids)
+            deleted_ids.extend(ids)
+
+        if purge_junk:
+            for prefix in _JUNK_PREFIXES:
+                rows = conn.execute(
+                    "SELECT id FROM verbatim_memories WHERE content LIKE ?",
+                    (prefix + "%",),
+                ).fetchall()
+                ids = [int(r[0]) for r in rows]
+                out["junk_deleted"] += len(ids)
+                deleted_ids.extend(ids)
+
+        if not dry_run and deleted_ids:
+            # Chunked DELETE to keep SQLite parameter binding bounded.
+            BATCH = 500
+            for i in range(0, len(deleted_ids), BATCH):
+                chunk = deleted_ids[i : i + BATCH]
+                placeholders = ",".join("?" for _ in chunk)
+                conn.execute(
+                    f"DELETE FROM verbatim_memories WHERE id IN ({placeholders})", chunk
+                )
+                if has_fts:
+                    conn.execute(
+                        f"DELETE FROM verbatim_fts WHERE rowid IN ({placeholders})", chunk
+                    )
+            conn.commit()
+
+        out["after_total"] = int(
+            conn.execute("SELECT COUNT(*) FROM verbatim_memories").fetchone()[0]
+        )
+        return out
+    finally:
+        conn.close()

{memorymaster-3.22.0 → memorymaster-3.23.0/memorymaster.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: memorymaster
-Version: 3.22.0
+Version: 3.23.0
 Summary: Production-grade memory reliability system for AI coding agents. Lifecycle-managed claims with citations, conflict detection, steward governance, and MCP integration.
 Author: wolverin0
 License: MIT
@@ -52,7 +52,7 @@ Lifecycle-managed claims with citations, conflict detection, steward governance,
 
 [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
 [![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
-[![Tests](https://img.shields.io/badge/tests-2214-green.svg)]()
+[![Tests](https://img.shields.io/badge/tests-2224-green.svg)]()
 [![MCP Tools](https://img.shields.io/badge/MCP%20tools-24-purple.svg)]()
 [![CLI Commands](https://img.shields.io/badge/CLI%20commands-86-orange.svg)]()
 [![PyPI](https://img.shields.io/pypi/v/memorymaster.svg)](https://pypi.org/project/memorymaster/)
@@ -91,7 +91,8 @@ recent PR status, and sensitivity-filter invariants.
 - **Correction mining** (new in v3.21.0): `mine-rules` scans the verbatim transcript archive for user corrections and distills them into rule claims; the Stop hook also mines each session's latest correction automatically
 - **Versioned schema migrations** (new in v3.20.0): `migrate` applies SQLite/Postgres migrations with sha256 drift detection; incremental `export-delta` ships small claim deltas for cheap cross-machine sync
 - **Retrieval quality** (new in v3.22.0): floor-ratio boost gate (`MEMORYMASTER_BOOST_FLOOR_RATIO`) stops fresh-but-wrong claims outranking the true match; `query --explain` shows per-stage score attribution; an opt-in correctness-safe query cache (`MEMORYMASTER_QUERY_CACHE`) with a generation gate
-- **Semantic contradiction probe** (new in v3.22.0): `detect-contradictions` finds claims that genuinely contradict each other (beyond the deterministic same-subject conflict check) via an LLM judge with a Wilson-CI rate and verdict cache
+- **Semantic contradiction probe** (new in v3.22.0, wired as a steward phase in v3.23.0): `detect-contradictions` finds claims that genuinely contradict each other (beyond the deterministic same-subject conflict check) via an LLM judge with a Wilson-CI rate and verdict cache; in v3.23 the same probe runs inside `run-steward` and emits paste-ready `conflicted` proposals
+- **Verbatim archive cleanup** (new in v3.23.0): `verbatim-cleanup` dedups the raw-transcript table and optionally purges pre-#128 junk rows, with a dry-run default and FTS5 mirror sync
 - **Steward governance**: multi-probe validators (filesystem, format, citation, semantic, tool) with proposal review
 - **Conflict resolution**: 5-tier auto (confidence > freshness > citations > LLM > manual)
 - **Auto-redaction** at ingest: JWT, GitHub tokens, Bearer, AWS keys, SSH keys, custom patterns