protonmail-bridge-agent-skill 1.0.1 → 1.1.0

package/README.md

@@ -37,16 +37,20 @@ npx skills add lukeramsden/protonmail-bridge-agent-skill --list
 ./pmail setup                         # account email + Bridge password -> Keychain
 ./pmail sync --all --full             # backfill the whole archive (resumable)
 ./pmail search "invoice"              # ranked full-text search, sub-second
+./pmail search "INV-2026-00" --substring   # substring / attachment-name search
 ./pmail list INBOX --unseen
 ./pmail read 34655
 ./pmail save-attach 34655 1
 ```
 
-All commands print JSON on stdout; progress and errors go to stderr.
+All commands print JSON on stdout; progress and errors go to stderr. `list` and
+`search` return `{mailbox, source, count, messages: [...]}` — use `jq '.messages[]'`
+for the rows (full table in `skills/protonmail-bridge/SKILL.md`).
 
 ## What the skill provides
 
-- **Full-archive FTS search** — SQLite FTS5 with bm25 ranking and match snippets; phrases, `OR`/`NOT`, and filters (`--from`, `--since`, `--unseen`, `--mailbox`)
+- **Full-archive FTS search** — SQLite FTS5 with bm25 ranking and match snippets; phrases, `OR`/`NOT`, prefix `*`, and filters (`--from`, `--since`, `--unseen`, `--mailbox`)
+- **Substring search** — `--substring` does a case-insensitive `LIKE` scan for partial tokens and attachment filenames/MIME types; metadata fields via a covering index (sub-second), bodies opt-in with `--field body`
 - **Resumable backfill** — cursor-persisted batches; interrupt and resume freely; automatic wipe/resync if Bridge's UIDVALIDITY changes
 - **Cache-first reads** — `list`, `read`, and `search` serve from the local cache with an implicit incremental sync at most once per minute; `--live` and `--no-sync` escape hatches
 - **Attachment download** — `save-attach` extracts parts to the cache's attachments directory

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "protonmail-bridge-agent-skill",
-  "version": "1.0.1",
+  "version": "1.1.0",
   "description": "Agent skill: read-only Proton Mail access via Proton Mail Bridge IMAP with a local SQLite FTS cache (pmail CLI, Python stdlib only).",
   "license": "MIT",
   "author": "lukeramsden",

package/skills/protonmail-bridge/SKILL.md

@@ -12,7 +12,8 @@ The CLI is `pmail` in this skill's directory. Run commands from the skill
 directory as `./pmail <command> ...` (or invoke it by its absolute path).
 
 All data commands print **JSON on stdout**; progress and errors go to stderr.
-Pipe through `jq` when you need to reshape output.
+Pipe through `jq` when you need to reshape output — see [Output shapes](#output-shapes)
+for which key holds the rows (`list`/`search` return a wrapper object, not an array).
 
 **If pmail fails or behaves unexpectedly, run the doctor script first:**
 
@@ -41,11 +42,33 @@ pmail mailboxes                        # folders + labels, message counts, cache
 pmail list [mailbox] [--limit N] [--unseen] [--live]
 pmail read <uid> [--mailbox M] [--max-chars N] [--raw]
 pmail search <words...> [--mailbox M] [--from X] [--since YYYY-MM-DD] [--unseen] [--limit N] [--live]
+pmail search <text> --substring [--field subject|from|to|attachment|body|all]...   # LIKE scan, see below
 pmail sync [--mailbox M | --all] [--full] [--skip-all-mail]
 pmail status                           # cache state per mailbox, db size
 pmail save-attach <uid> <index> [--mailbox M]   # index comes from `read` output
 ```
 
+## Output shapes
+
+Every command returns a **single JSON object** (except `mailboxes`, which is
+an array). Rows live under one key; check `count` before concluding that
+nothing matched — `jq '.[]'` on a wrapper object is *not* an empty result, it
+is the wrong selector.
+
+| Command | Top-level keys | Rows | `jq` |
+|---|---|---|---|
+| `list` | `mailbox, source, count, messages` | `messages[]` → `uid, date, from, to, subject, flags, source` | `jq '.messages[]'` |
+| `search` (FTS) | `mailbox, source:"cache-fts", query, count, messages` | `messages[]` → `uid, date, from, subject, flags, snippet, rank` | `jq '.messages[]'` |
+| `search --substring` | `mailbox, source:"cache-substring", query, fields, scanned, elapsed_s, count, messages` | `messages[]` → `uid, date, from, subject, flags, matched[], attachments[], [snippet]` | `jq '.messages[]'` |
+| `search --live` | `mailbox, source:"live", total_hits, count, messages` | `messages[]` → `uid, date, from, to, subject, flags` | `jq '.messages[]'` |
+| `read` | `mailbox, uid, date, from, to, cc, subject, flags, body_source, truncated, attachments, body, source` | one message; `attachments[]` → `index, name, mime, size` (the `index` is what `save-attach` takes) | `jq '.attachments[]'` |
+| `mailboxes` | *(array)* | each → `mailbox, attributes, messages, cached` | `jq '.[]'` |
+| `status` | `db_path, db_size_mb, mailboxes, attachments_saved` | `mailboxes[]` → `mailbox, cached, server_messages, backfill_complete, ...` | `jq '.mailboxes[]'` |
+| `sync` | `synced` | `synced[]` → per-mailbox counts | `jq '.synced[]'` |
+| `save-attach` | `saved, name, mime, size` | `saved` is the absolute path written | — |
+
+`read --raw` is the exception: it writes the raw RFC822 bytes, not JSON.
+
 ## Behavior notes
 
 - **Mailboxes**: `INBOX`, `Archive`, `Sent`, `Drafts`, `Trash`, `Spam`, `Starred`,
@@ -61,9 +84,31 @@ pmail save-attach <uid> <index> [--mailbox M]   # index comes from `read` output
   uncached server-side search.
 - **Search syntax**: cached search is SQLite FTS5 — plain words are AND-ed,
   `"quoted phrases"` and `OR`/`NOT` work. Results are bm25-ranked with snippets
-  (sub-second over an ~80k-message archive). FTS matches whole tokens;
-  `--live` IMAP search matches substrings and also scans headers, so hit counts
-  differ slightly between the two.
+  (sub-second over an ~80k-message archive). FTS matches **whole tokens**: the
+  tokenizer splits on punctuation, so `inv-2026` finds `INV-2026-0042` (tokens
+  `inv`, `2026`, `0042`), but `2026-00` or `0042.pdf` will not. Use a trailing
+  `*` for prefix matches (`inv*`, `ramsd*`). Attachment filenames are **not**
+  in the FTS index. `--live` IMAP search matches substrings and also scans
+  headers, so hit counts differ slightly between the two.
+- **Substring search** (`--substring`): a case-insensitive `LIKE` scan of the
+  cache for text that FTS cannot express — part of a token, a reference number
+  fragment, a domain fragment, or an **attachment filename / MIME type**.
+  By default it scans `subject`, `from`, `to` and `attachment` via a covering
+  index (well under a second on ~80k messages; the index is built once on first
+  use, ~1 minute on a multi-GB cache). Add `--field body` or `--field all` to
+  also scan message bodies — that reads every cached body and takes tens of
+  seconds on a large archive; prefer FTS (with `*` prefixes) for body text
+  whenever possible. Results are newest-first (not ranked); each row lists
+  which `matched` fields hit. `--from`, `--since`, `--unseen`, `--mailbox` and
+  `--limit` all apply. Not combinable with `--live`.
+  ```bash
+  pmail search 'INV-2026-00' --substring                    # subject/from/to/attachment name
+  pmail search '.xlsx' --substring --field attachment       # attachments by extension
+  pmail search 'acme.co' --substring --field all --since 2026-01-01   # includes bodies (slow)
+  ```
+- **Do not query `mail.db` directly** for substring searches: a raw `LIKE` over
+  the table reads every multi-KB row and can take minutes on a large cache, and
+  it bypasses the CLI's guarantees. Use `--substring` instead.
 - **Bodies**: most real-world mail is HTML-only; pmail indexes the stripped text
   of the HTML part so search covers those messages too.
 - **Read-only guarantee**: bodies are fetched with BODY.PEEK and mailboxes are

package/skills/protonmail-bridge/pmail

@@ -618,8 +618,12 @@ def cmd_read(args):
 
 def cmd_search(args):
     db = open_db()
+    if args.substring and args.live:
+        die("--substring scans the local cache and cannot be combined with --live")
     if not args.live:
         implicit_sync(db, args.mailbox, args.no_sync)
+        if args.substring:
+            return search_substring(db, args)
         if cached_count(db, args.mailbox) > 0:
             return search_fts(db, args)
         log("pmail: cache empty for this mailbox; falling back to --live search")
@@ -694,6 +698,117 @@ def search_fts(db, args):
     emit({"mailbox": args.mailbox, "source": "cache-fts", "query": args.query,
           "count": len(out), "messages": out})
 
+# Column per --field name. `attachment` matches the JSON-encoded attachment
+# list (filenames and MIME types). `body` is the slow one: a full scan of
+# every cached body.
+SUBSTRING_FIELDS = {
+    "subject": "m.subject",
+    "from": "m.from_addr",
+    "to": "m.to_addr",
+    "attachment": "m.attachments",
+    "body": "m.body_text",
+}
+SUBSTRING_FAST_FIELDS = ["subject", "from", "to", "attachment"]
+
+# Covering index so a metadata-only substring scan never touches the table
+# rows (which carry multi-KB bodies). Built lazily on first --substring use.
+META_INDEX = """CREATE INDEX IF NOT EXISTS messages_meta ON messages
+  (mailbox, date, uid, from_addr, to_addr, subject, flags, attachments)"""
+
+def ensure_meta_index(db):
+    if db.execute("SELECT 1 FROM sqlite_master WHERE type='index' AND name='messages_meta'"
+                  ).fetchone():
+        return
+    log("pmail: building metadata index for substring search (one-off, may take a minute)")
+    db.execute(META_INDEX)
+    db.commit()
+
+def like_pattern(text):
+    """Escape LIKE metacharacters so the user's text matches literally."""
+    esc = text.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
+    return f"%{esc}%"
+
+def substring_snippet(text, needle, width=80):
+    """Case-insensitive context window around the first match of needle."""
+    if not text:
+        return ""
+    i = text.lower().find(needle.lower())
+    if i < 0:
+        return ""
+    lo, hi = max(0, i - width // 2), min(len(text), i + len(needle) + width // 2)
+    s = ("..." if lo > 0 else "") + text[lo:hi] + ("..." if hi < len(text) else "")
+    return INVISIBLES_RE.sub("", re.sub(r"\s+", " ", s)).strip()
+
+def search_substring(db, args):
+    """Case-insensitive substring scan (SQL LIKE) over cached columns.
+
+    Unlike FTS this matches inside tokens ("INV-2026-0042", part of an
+    attachment filename, a domain fragment). It cannot use the FTS index:
+    metadata fields scan in well under a second on an ~80k-message cache,
+    but `body` reads every cached body and can take tens of seconds."""
+    needle = args.query
+    if not needle:
+        die("--substring needs the text to look for")
+    fields = args.field or SUBSTRING_FAST_FIELDS
+    if "all" in fields:
+        fields = list(SUBSTRING_FIELDS)
+    fields = list(dict.fromkeys(fields))  # dedupe, keep order
+    if "body" in fields:
+        log(f"pmail: substring scan includes body text over "
+            f"{cached_count(db, args.mailbox)} cached message(s); this may take a while")
+    else:
+        ensure_meta_index(db)
+    pattern = like_pattern(needle)
+    match = " OR ".join(f"{SUBSTRING_FIELDS[f]} LIKE ? ESCAPE '\\'" for f in fields)
+    where, params = [f"({match})"], [pattern] * len(fields)
+    where.append("m.mailbox=?"); params.append(args.mailbox)
+    if args.sender:
+        where.append("m.from_addr LIKE ?"); params.append(f"%{args.sender}%")
+    if args.since:
+        try:
+            ts = int(datetime.fromisoformat(args.since).replace(tzinfo=timezone.utc).timestamp())
+            where.append("m.date >= ?"); params.append(ts)
+        except ValueError:
+            die(f"--since must be ISO date (e.g. 2026-01-01), got '{args.since}'")
+    if args.unseen:
+        where.append("m.flags NOT LIKE '%\\Seen%'")
+    sql = f"""
+      SELECT m.uid, m.date, m.from_addr, m.to_addr, m.subject, m.flags,
+             m.attachments, {'m.body_text' if 'body' in fields else 'NULL'}
+      FROM messages m
+      WHERE {' AND '.join(where)}
+      ORDER BY m.date DESC, m.uid DESC LIMIT ?"""
+    params.append(args.limit)
+    t0 = time.time()
+    rows = db.execute(sql, params).fetchall()
+    out = []
+    for uid, date, from_addr, to_addr, subject, flags, attachments, body in rows:
+        atts = json.loads(attachments or "[]")
+        matched = []
+        nl = needle.lower()
+        if "subject" in fields and nl in (subject or "").lower():
+            matched.append("subject")
+        if "from" in fields and nl in (from_addr or "").lower():
+            matched.append("from")
+        if "to" in fields and nl in (to_addr or "").lower():
+            matched.append("to")
+        if "attachment" in fields and any(
+                nl in (a.get("name", "") + " " + a.get("mime", "")).lower() for a in atts):
+            matched.append("attachment")
+        if "body" in fields and body and nl in body.lower():
+            matched.append("body")
+        row = {"uid": uid, "date": iso(date), "from": from_addr, "subject": subject,
+               "flags": flags, "matched": matched,
+               "attachments": [{"index": a.get("index"), "name": a.get("name"),
+                                "mime": a.get("mime")} for a in atts]}
+        if "body" in matched:
+            row["snippet"] = substring_snippet(body, needle)
+        out.append(row)
+    emit({"mailbox": args.mailbox, "source": "cache-substring", "query": needle,
+          "fields": fields, "scanned": cached_count(db, args.mailbox),
+          "elapsed_s": round(time.time() - t0, 2),
+          "count": len(out), "messages": out})
+
 def cmd_sync(args):
     db = open_db()
     m = connect()
@@ -809,6 +924,14 @@ def build_parser():
     sp.add_argument("--since", help="ISO date, e.g. 2026-01-01")
     sp.add_argument("--unseen", action="store_true")
     sp.add_argument("--limit", type=int, default=DEFAULT_LIMIT)
+    sp.add_argument("--substring", action="store_true",
+                    help="case-insensitive substring (LIKE) scan of the cache instead of "
+                         "FTS; matches inside tokens, e.g. part of an attachment filename")
+    sp.add_argument("--field", action="append",
+                    choices=[*SUBSTRING_FIELDS, "all"],
+                    help="with --substring: column(s) to scan (repeatable). Default: "
+                         "subject, from, to, attachment (fast). 'body' or 'all' scans "
+                         "every cached body and is slow")
     sp.add_argument("--live", action="store_true")
     sp.add_argument("--no-sync", action="store_true")
     sp.set_defaults(fn=cmd_search)
@@ -838,6 +961,8 @@ def main():
         args.query = " ".join(args.words)
         if not args.live and not args.query and not (args.sender or args.subject):
             die("search needs at least one word (cached search is text-based)")
+        if args.field and not args.substring:
+            die("--field only applies with --substring")
     args.fn(args)
 
 if __name__ == "__main__":