msgsearch 0.1.0__tar.gz

msgsearch-0.1.0/.claude/agents/corpus-explorer.md

@@ -0,0 +1,22 @@
+---
+name: corpus-explorer
+description: Answers structural questions about chat.db (schema, joins, Apple quirks) without exposing message content.
+tools: Bash, Read, Grep
+---
+
+You answer questions about the *shape* of the iMessage corpus — schema, joins,
+distributions, encoding quirks — never its content.
+
+Rules:
+- Open only `~/msgsearch/chat.db` and only read-only (`file:...?mode=ro`).
+  `~/Library/Messages` is off limits; refuse if asked.
+- Report aggregates, counts, and schema. Never print message bodies. If an example
+  is genuinely necessary to explain an encoding quirk, show a hexdump of the
+  structural bytes, not the decoded text.
+- Known quirks to account for, and to check before trusting any query:
+  `message.date` mixes second- and nanosecond-scale Apple-epoch values; ~86% of
+  rows carry text in `attributedBody` rather than `text`;
+  `associated_message_type != 0` rows are tapbacks, not messages;
+  some messages have no `chat_message_join` row at all.
+
+Return findings as a short table plus the SQL you ran, so it can be re-verified.

msgsearch-0.1.0/.claude/agents/retrieval-critic.md

@@ -0,0 +1,30 @@
+---
+name: retrieval-critic
+description: Adversarially audits a retrieval change. Use after any change to ranking, chunking, embedding, or fusion — before claiming it works.
+tools: Bash, Read, Grep
+---
+
+You are a skeptical IR engineer. Your job is to find the reason the reported
+improvement is fake. Assume it is until proven otherwise.
+
+Check, in order:
+
+1. **Is it measured at all?** Run `python3 eval/bench.py`. Queries with an empty
+   `relevant` list are scored as nothing. If the labeled set is under ~20 queries,
+   say plainly that the result is noise, not signal.
+2. **Test-set contamination.** Was `eval/gold.jsonl` edited in the same change as
+   the retriever? Check `git diff`. Labeling the gold set to match new output is
+   the most common way to fake a win here.
+3. **Metric selection.** Did the report cite only the metric that moved? Show all
+   four (p@10, r@50, MRR, nDCG@10). A gain in p@10 with a drop in r@50 usually
+   means the change narrowed recall rather than improving ranking.
+4. **Degenerate wins.** Does the retriever return tapbacks, duplicates, or the same
+   message under multiple ids? Does it work on short queries and one-word queries?
+   Try a query with no plausible answer and confirm it returns few or no results
+   rather than confident garbage.
+5. **Privacy.** Does the diff introduce anything that prints message bodies to
+   stdout in committed code, or sends text off the machine? This is a hard stop.
+
+Report only defects you actually confirmed by running something. For each, give the
+command you ran and its output. If the change is genuinely sound, say so in one line
+— do not invent findings to seem useful.

msgsearch-0.1.0/.claude/commands/bench.md

@@ -0,0 +1,12 @@
+---
+description: Run the retrieval eval and report the before/after metrics table
+---
+Run the verification loop for this project:
+
+1. `python3 eval/bench.py --against baseline`
+2. If `index.sqlite` is missing, rebuild it first with `python3 search/index.py`.
+3. Report the metrics table verbatim. Do NOT summarize it away or round the numbers.
+4. If any metric regressed, say so plainly and explain which change caused it.
+5. If there are unlabeled gold queries, list them — they measure nothing until labeled.
+
+A retrieval change that does not move these numbers is not an improvement.

msgsearch-0.1.0/.claude/commands/go.md

@@ -0,0 +1,17 @@
+---
+description: Verify, simplify, and commit the current change
+---
+Finish the current change end to end:
+
+1. **Verify** — run `python3 eval/bench.py --against baseline` and paste the table.
+   For non-retrieval changes, run the affected script directly and show real output.
+   Never claim something works without having run it.
+2. **Simplify** — reread the diff. Remove dead code, collapse needless abstraction,
+   and check it matches the conventions in AGENTS.md.
+3. **Record** — if you got anything wrong during this task, append the correction to
+   the "Learned corrections" section of AGENTS.md. This is not optional; it is how
+   the project gets easier over time.
+4. **Commit** — stage and commit with a message describing the *why*. Do not push
+   unless asked.
+
+Confirm no message bodies appear in the diff before committing.

msgsearch-0.1.0/.claude/commands/recon.md

@@ -0,0 +1,8 @@
+---
+description: Re-run structural recon on the message corpus
+---
+Run `python3 explore.py` and interpret the output: message volume, decode coverage,
+date range, top chats. Flag anything that changed since the numbers recorded in
+AGENTS.md, and update AGENTS.md if the corpus shape has shifted.
+
+Never print message bodies.

msgsearch-0.1.0/.claude/hooks/verify-retrieval.py

@@ -0,0 +1,114 @@
+#!/usr/bin/env python3
+"""Stop hook: refuse to finish a retrieval change that made retrieval worse.
+
+Must be launched with .venv/bin/python (settings.json does this). Under the
+system python3, which is an x86_64 build, child processes inherit that
+architecture and the arm64 numpy in the venv fails to load -- the bench then
+exits non-zero and the check silently passes.
+
+AGENTS.md asks an agent to justify retrieval changes with eval/bench.py. This
+turns that request into a wall, because a request is something a model can
+forget and a hook is not.
+
+Fires only when a file that can change ranking was touched, and only when the
+gold set actually has labels -- there is nothing to measure otherwise.
+
+Exit 0 allows the stop. Exit 2 blocks it and sends stderr back to the agent.
+"""
+
+import contextlib
+import hashlib
+import json
+import os
+import re
+import subprocess
+import sys
+import tempfile
+
+ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+PY_BIN = os.path.join(ROOT, ".venv", "bin", "python")
+BASELINE = os.path.join(ROOT, "eval", "results", "baseline.json")
+GOLD = os.path.join(ROOT, "eval", "gold.jsonl")
+
+# Anything in the engine package can move a ranking, as can the eval adapter.
+# Matched as path prefixes so this survives files being added or renamed -- an
+# earlier version listed modules by bare name and broke the moment the project
+# was restructured into src/.
+WATCHED_PREFIXES = ("src/msgsearch/", "eval/retriever")
+
+# A drop smaller than this is noise at the current gold-set size, not a regression.
+TOLERANCE = 0.02
+
+
+def sh(*args):
+    return subprocess.run(args, cwd=ROOT, capture_output=True, text=True).stdout
+
+
+def touched():
+    diff = sh("git", "diff", "HEAD", "--name-only") + sh("git", "diff", "--name-only")
+    files = {line.strip() for line in diff.splitlines() if line.strip()}
+    return sorted(f for f in files if f.startswith(WATCHED_PREFIXES))
+
+
+def labeled_queries():
+    if not os.path.exists(GOLD):
+        return 0
+    with open(GOLD) as f:
+        return sum(1 for line in f if line.strip() and json.loads(line).get("relevant"))
+
+
+def already_blocked_for(state):
+    """Block at most once per code state, so a change the agent cannot fix
+    does not trap it in a loop."""
+    marker = os.path.join(tempfile.gettempdir(), f"msgsearch-hook-{state}")
+    if os.path.exists(marker):
+        return True
+    open(marker, "w").close()
+    return False
+
+
+def main():
+    with contextlib.suppress(Exception):
+        json.load(sys.stdin)
+
+    files = touched()
+    if not files or not os.path.exists(PY_BIN):
+        return 0
+    if labeled_queries() < 3 or not os.path.exists(BASELINE):
+        return 0
+
+    proc = subprocess.run(
+        [PY_BIN, "eval/bench.py", "-r", "eval.retriever", "--against", "baseline"],
+        cwd=ROOT,
+        capture_output=True,
+        text=True,
+        timeout=600,
+    )
+    out = proc.stdout
+    match = re.search(r"^mrr\s+([\d.]+)", out, re.M)
+    if not match:
+        return 0
+
+    mrr = float(match.group(1))
+    base = json.load(open(BASELINE))["metrics"]["mrr"]
+    if mrr >= base - TOLERANCE:
+        return 0
+
+    state = hashlib.sha1((sh("git", "diff", "HEAD") + f"{mrr}").encode()).hexdigest()[:12]
+    if already_blocked_for(state):
+        return 0
+
+    print(
+        f"Retrieval regression: MRR {base:.4f} -> {mrr:.4f} "
+        f"({mrr - base:+.4f}) after editing {', '.join(files)}.\n\n"
+        f"{out.strip()}\n\n"
+        "Either fix it, or revert the change. If the drop is intended and "
+        "justified, say so explicitly and re-snapshot with "
+        "`eval/bench.py -r eval.retriever --save baseline`.",
+        file=sys.stderr,
+    )
+    return 2
+
+
+if __name__ == "__main__":
+    sys.exit(main())

msgsearch-0.1.0/.claude/settings.json

@@ -0,0 +1,52 @@
+{
+  "permissions": {
+    "allow": [
+      "Bash(./.venv/bin/python:*)",
+      "Bash(git add:*)",
+      "Bash(git checkout:*)",
+      "Bash(git commit:*)",
+      "Bash(git diff:*)",
+      "Bash(git log:*)",
+      "Bash(git status:*)",
+      "Bash(python3 -c:*)",
+      "Bash(python3 eval/bench.py:*)",
+      "Bash(python3 explore.py)",
+      "Bash(python3 search/index.py:*)",
+      "Edit(./AGENTS.md)",
+      "Edit(./eval/**)",
+      "Edit(./search/**)",
+      "Read(./**)"
+    ],
+    "deny": [
+      "Read(~/Library/Messages/**)",
+      "Bash(sqlite3 ~/Library/Messages/**)",
+      "Write(~/msgsearch/**)"
+    ]
+  },
+  "hooks": {
+    "PostToolUse": [
+      {
+        "matcher": "Write|Edit",
+        "hooks": [
+          {
+            "type": "command",
+            "command": "python3 -c \"import ast,sys,glob; [ast.parse(open(f).read(),f) for f in glob.glob('**/*.py',recursive=True) if '.venv' not in f]\" || true"
+          }
+        ]
+      }
+    ],
+    "Stop": [
+      {
+        "matcher": "*",
+        "hooks": [
+          {
+            "type": "command",
+            "command": "${CLAUDE_PROJECT_DIR}/.venv/bin/python ${CLAUDE_PROJECT_DIR}/.claude/hooks/verify-retrieval.py",
+            "timeout": 600,
+            "statusMessage": "checking retrieval didn't regress"
+          }
+        ]
+      }
+    ]
+  }
+}

msgsearch-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,68 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+  workflow_dispatch:
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  test:
+    name: py${{ matrix.python-version }} on ${{ matrix.os }}
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest]
+        python-version: ["3.10", "3.11", "3.12", "3.13"]
+        include:
+          - os: macos-latest
+            python-version: "3.12"
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: pip
+
+      # PyTorch is ~1GB and nothing under test needs it: the parsing, chunking
+      # and tagging logic is pure Python, and the modules that do use models
+      # import them lazily. Installing numpy alone keeps CI to a few seconds
+      # while still checking that every module imports.
+      - name: Install
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install "numpy>=2,<3" "ruff==0.15.4"
+          python -m pip install -e . --no-deps
+
+      - name: Lint
+        run: ruff check .
+
+      - name: Format
+        run: ruff format --check .
+
+      - name: Unit tests
+        run: python -m unittest discover -s tests -v
+
+      - name: Every module imports
+        run: |
+          python -c "
+          import msgsearch, msgsearch.cli, msgsearch.config, msgsearch.chunk
+          import msgsearch.contacts
+          import msgsearch.embedder, msgsearch.explore, msgsearch.extract
+          import msgsearch.doctor, msgsearch.index, msgsearch.search, msgsearch.tagging
+          print('msgsearch', msgsearch.__version__)
+          "
+
+      - name: CLI responds
+        run: |
+          msgsearch --version
+          msgsearch --help
+          msgsearch search --help
+          msgsearch doctor --help

msgsearch-0.1.0/.github/workflows/release.yml

@@ -0,0 +1,62 @@
+name: Release
+
+# Publishing is driven by tags, so the released artefact always corresponds to a
+# specific commit that CI has already tested.
+on:
+  push:
+    tags: ["v*"]
+  workflow_dispatch:
+
+jobs:
+  build:
+    name: Build distributions
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Build sdist and wheel
+        run: |
+          python -m pip install --upgrade pip build
+          python -m build
+
+      # A tag that disagrees with pyproject.toml would publish a version nobody
+      # asked for, and PyPI versions cannot be reused once taken.
+      - name: Tag must match the package version
+        if: startsWith(github.ref, 'refs/tags/v')
+        run: |
+          tag="${GITHUB_REF_NAME#v}"
+          version=$(python -c "import tomllib;print(tomllib.load(open('pyproject.toml','rb'))['project']['version'])")
+          echo "tag=$tag  pyproject=$version"
+          test "$tag" = "$version"
+
+      - uses: actions/upload-artifact@v4
+        with:
+          name: distributions
+          path: dist/
+
+  publish:
+    name: Publish to PyPI
+    needs: build
+    runs-on: ubuntu-latest
+    if: startsWith(github.ref, 'refs/tags/v')
+
+    # Trusted Publishing: PyPI verifies this workflow's OIDC identity, so there
+    # is no API token to store, leak or rotate. Configure the publisher once at
+    # https://pypi.org/manage/project/msgsearch/settings/publishing/
+    environment:
+      name: pypi
+      url: https://pypi.org/p/msgsearch
+    permissions:
+      id-token: write
+
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          name: distributions
+          path: dist/
+
+      - uses: pypa/gh-action-pypi-publish@release/v1

msgsearch-0.1.0/.gitignore

@@ -0,0 +1,27 @@
+# Message data and anything derived from it must never be committed. The index is
+# a plaintext, searchable copy of every private thing anyone has ever texted you.
+*.db
+*.db-shm
+*.db-wal
+*.sqlite
+*.sqlite3
+*.npy
+*.npz
+index/
+msgsearch_index/
+
+# Local configuration holds phone numbers and email addresses.
+config_local.py
+
+# Build artefacts
+dist/
+build/
+*.egg-info/
+
+# Environments and build noise
+.venv/
+venv/
+__pycache__/
+*.py[cod]
+.pytest_cache/
+.DS_Store

msgsearch-0.1.0/AGENTS.md

@@ -0,0 +1,66 @@
+# query_messages
+
+Semantic search over ~260k personal iMessages (2018-08 → 2026-09), by meaning
+rather than keyword.
+
+Source of truth for all coding agents; `CLAUDE.md` symlinks here.
+
+## Rules
+
+Kept deliberately short. A rule earns a line here only if breaking it fails
+*silently* or *irreversibly* — anything that fails loudly teaches itself, and
+anything already enforced in code or `settings.json` is not repeated here.
+
+- **Never** touch `~/Library/Messages`. Work against `~/msgsearch/chat.db`,
+  opened `file:...?mode=ro`. (Irreversible.)
+- Message content is private: none sent over the network, none in commits, logs or
+  eval fixtures, and no bodies printed by diagnostic or eval scripts. `search.py`
+  is the single exemption — printing results to the terminal is its entire
+  function. (Irreversible.)
+- `message.date` is Apple-epoch and **mixes scales** — pre-10.13 rows are
+  seconds, later ones nanoseconds. Use `extract.apple_timestamp()`; `explore.py`
+  carries a standalone copy so it can run before anything else exists. (Silent:
+  the wrong scale yields plausible, wrong dates.)
+- ~86% of rows have `text IS NULL` with content in the `attributedBody`
+  typedstream blob. Go through `attributed_body.message_text(text, blob)`.
+  (Silent: reading `message.text` looks fine and loses most of the corpus.)
+
+## Commands
+
+Use `./.venv/bin/python` — it is the arm64 interpreter. The system `python3` is an
+Intel build and cannot install torch.
+
+```bash
+./.venv/bin/msgsearch doctor                     # setup preflight
+./.venv/bin/msgsearch contacts --unresolved      # unnamed handles by volume
+./.venv/bin/msgsearch explore                    # structural recon
+./.venv/bin/msgsearch index                      # rebuild ~/msgsearch/index/
+./.venv/bin/msgsearch search "atria login"       # query it
+
+./.venv/bin/ruff check . && ./.venv/bin/ruff format --check .
+./.venv/bin/python -m unittest discover -s tests
+
+./.venv/bin/python eval/label.py "a query"       # add a labeled gold query
+./.venv/bin/python eval/bench.py -r eval.retriever --against baseline
+```
+
+Code lives in `src/msgsearch/`; install it with `pip install -e ".[dev]"` once.
+
+Retriever variants for `-r`, to score one stage at a time:
+`eval.retriever` (full) · `eval.retriever_bm25` · `eval.retriever_dense` ·
+`eval.retriever_norerank`
+
+## The verification loop
+
+A retrieval change is justified by `eval/bench.py` or it is a guess. Report the
+before/after table; don't claim an improvement without it.
+
+Architecture: `ARCHITECTURE.md`.
+
+## Learned corrections
+
+Append here **only** when an agent actually gets something wrong in practice.
+Nothing speculative. If a rule can instead be enforced in code, a hook, or
+`settings.json`, do that and leave this file alone.
+
+_(empty — nothing has gone wrong yet)_