dirsql-plugin-embeddings 0.1.5__tar.gz → 0.1.7__tar.gz

{dirsql_plugin_embeddings-0.1.5 → dirsql_plugin_embeddings-0.1.7}/PKG-INFO

@@ -1,15 +1,16 @@
 Metadata-Version: 2.4
 Name: dirsql-plugin-embeddings
-Version: 0.1.5
+Version: 0.1.7
 Summary: First-party dirsql plugin: semantic search via an OpenAI-compatible embeddings endpoint.
 Requires-Python: >=3.10
+Requires-Dist: cachetta>=0.6.15
 Requires-Dist: pypdf>=6
 Description-Content-Type: text/markdown
 
 # dirsql-plugin-embeddings
 
 A first-party [`dirsql`](https://github.com/thekevinscott/dirsql) plugin that
-adds **semantic search** over a directory of Markdown files. It is the worked
+adds **semantic search** over a directory of Markdown files and PDFs. It is the worked
 implementation behind the [Search documents by
 meaning](https://thekevinscott.github.io/dirsql/howto/search-by-meaning) how-to,
 swapping that guide's local `model2vec` model for any OpenAI-compatible
@@ -29,13 +30,24 @@ is installed alongside it. The fragment declares:
 
 - the [`sqlite-vec`](https://github.com/asg017/sqlite-vec) extension, for
   `vec_distance_cosine()`;
-- a `documents` table whose `on-file` hook embeds each `**/*.md` file into a
-  TEXT `embedding` column;
+- a `documents` table whose `on-file` hook embeds each `**/*.{md,pdf}` file into
+  a TEXT `embedding` column;
 - a `pre-query` hook that embeds the incoming question and emits the
   nearest-neighbor SQL.
 
 Both hooks are console scripts that call the same embedder.
 
+Markdown is read as UTF-8 text; a `.pdf` is read with
+[pypdf](https://pypdf.readthedocs.io), whose per-page extracted text is joined
+and embedded like any other document. The extension check is case-insensitive
+(`.PDF` is a PDF), though the glob above is not — globset matches case-sensitively,
+so an uppercase-suffixed file needs its own `glob` entry to be picked up at all.
+
+A PDF that cannot be parsed aborts the scan rather than being skipped: the hook
+exits non-zero and dirsql reports the path and the pypdf reason. A *scanned*,
+image-only PDF is not a failure — pypdf yields no text, and the file is indexed
+with an empty `text`, exactly like an empty `.md`.
+
 ## Configuration
 
 The embedder reads three environment variables (point them at any hosted or
@@ -51,7 +63,7 @@ self-managed OpenAI-compatible inference server):
 
 | Script | Hook | Input | Output |
 |---|---|---|---|
-| `dirsql-embeddings-on-file` | `on-file` | a file's absolute path (`argv[1]`) | one-line JSON row array with `path`, `text`, `embedding` |
+| `dirsql-embeddings-on-file` | `on-file` | a file's absolute path (`argv[1]`), text or PDF | one-line JSON row array with `path`, `text`, `embedding` |
 | `dirsql-embeddings-pre-query` | `pre-query` | a raw request body (`argv[1]`) | nearest-neighbor SQL over `documents` |
 
 `pre-query` accepts both a verbatim server body (`{"q": ...}`) and the CLI

{dirsql_plugin_embeddings-0.1.5 → dirsql_plugin_embeddings-0.1.7}/README.md

@@ -1,7 +1,7 @@
 # dirsql-plugin-embeddings
 
 A first-party [`dirsql`](https://github.com/thekevinscott/dirsql) plugin that
-adds **semantic search** over a directory of Markdown files. It is the worked
+adds **semantic search** over a directory of Markdown files and PDFs. It is the worked
 implementation behind the [Search documents by
 meaning](https://thekevinscott.github.io/dirsql/howto/search-by-meaning) how-to,
 swapping that guide's local `model2vec` model for any OpenAI-compatible
@@ -21,13 +21,24 @@ is installed alongside it. The fragment declares:
 
 - the [`sqlite-vec`](https://github.com/asg017/sqlite-vec) extension, for
   `vec_distance_cosine()`;
-- a `documents` table whose `on-file` hook embeds each `**/*.md` file into a
-  TEXT `embedding` column;
+- a `documents` table whose `on-file` hook embeds each `**/*.{md,pdf}` file into
+  a TEXT `embedding` column;
 - a `pre-query` hook that embeds the incoming question and emits the
   nearest-neighbor SQL.
 
 Both hooks are console scripts that call the same embedder.
 
+Markdown is read as UTF-8 text; a `.pdf` is read with
+[pypdf](https://pypdf.readthedocs.io), whose per-page extracted text is joined
+and embedded like any other document. The extension check is case-insensitive
+(`.PDF` is a PDF), though the glob above is not — globset matches case-sensitively,
+so an uppercase-suffixed file needs its own `glob` entry to be picked up at all.
+
+A PDF that cannot be parsed aborts the scan rather than being skipped: the hook
+exits non-zero and dirsql reports the path and the pypdf reason. A *scanned*,
+image-only PDF is not a failure — pypdf yields no text, and the file is indexed
+with an empty `text`, exactly like an empty `.md`.
+
 ## Configuration
 
 The embedder reads three environment variables (point them at any hosted or
@@ -43,7 +54,7 @@ self-managed OpenAI-compatible inference server):
 
 | Script | Hook | Input | Output |
 |---|---|---|---|
-| `dirsql-embeddings-on-file` | `on-file` | a file's absolute path (`argv[1]`) | one-line JSON row array with `path`, `text`, `embedding` |
+| `dirsql-embeddings-on-file` | `on-file` | a file's absolute path (`argv[1]`), text or PDF | one-line JSON row array with `path`, `text`, `embedding` |
 | `dirsql-embeddings-pre-query` | `pre-query` | a raw request body (`argv[1]`) | nearest-neighbor SQL over `documents` |
 
 `pre-query` accepts both a verbatim server body (`{"q": ...}`) and the CLI

dirsql_plugin_embeddings-0.1.5/e2e-attestations/claude-699-read-pdf.json → dirsql_plugin_embeddings-0.1.7/e2e-attestations/claude-700-route-pdf.json

@@ -1,7 +1,7 @@
 {
   "command": "uv run --with sqlite-vec --with-editable . --with-editable ../../packages/python python -m pytest tests/e2e -x -q",
-  "ran_at": 1785327111,
+  "ran_at": 1785338799,
   "exit_code": 0,
-  "commit": "b59ad4bf5dce929e2172cde15888918de9d277fd",
-  "branch": "claude/699-read-pdf"
+  "commit": "fcca643ac74c98dd9a6b525681711c720c303250",
+  "branch": "claude/700-route-pdf"
 }

{dirsql_plugin_embeddings-0.1.5 → dirsql_plugin_embeddings-0.1.7}/pyproject.toml

@@ -14,7 +14,11 @@ readme = "README.md"
 requires-python = ">=3.10"
 # pypdf, not pymupdf: pymupdf is AGPL-3.0, and a runtime dependency of this
 # MIT-licensed package propagates to everyone who installs it.
-dependencies = ["pypdf>=6"]
+#
+# cachetta >=0.6.15 is load-bearing, not a nicety: every earlier release
+# declares `requires-python >=3.12`, which would drag this package's floor up
+# with it. 0.6.15 lowered cachetta's own floor to `>=3.10`, matching ours.
+dependencies = ["pypdf>=6", "cachetta>=0.6.15"]
 
 [tool.hatch.version]
 source = "vcs"

dirsql_plugin_embeddings-0.1.7/src/dirsql_plugin_embeddings/cache.py

@@ -0,0 +1,20 @@
+"""The plugin's one configured cache.
+
+Every cached function derives its own sub-cache from this singleton with
+cachetta's ``cache / 'name'``, which keeps the defaults and appends a
+subdirectory. Naming that subdirectory after the function is what keeps caches
+apart: cachetta hashes the *arguments* of a call, not the identity of the
+function, so two cached functions sharing a directory would collide as soon as
+their argument shapes matched.
+"""
+
+from cachetta import Cachetta
+
+from .config import CACHE_DIR, CACHE_DURATION, CACHE_READ
+
+cache = Cachetta(
+    path=CACHE_DIR,
+    hashed=True,
+    duration=CACHE_DURATION,
+    read=CACHE_READ,
+)

dirsql_plugin_embeddings-0.1.7/src/dirsql_plugin_embeddings/config.py

@@ -0,0 +1,31 @@
+"""Cache configuration: what the plugin caches, where, and for how long.
+
+Annotations are evaluated at runtime (no ``from __future__ import
+annotations``) so a mutated ``X | None`` union in a signature fails at import
+rather than surviving as an inert string.
+"""
+
+import os
+from datetime import timedelta
+from pathlib import Path
+
+PLUGIN_NAME = "dirsql-plugin-embeddings"
+
+CACHE_DIR = Path.home() / ".cache" / PLUGIN_NAME
+
+# Every cache here keys on the source file's mtime, so a hit can never be
+# stale and expiry would only re-do work for a file nothing has touched.
+# cachetta's own default is 7 days.
+CACHE_DURATION = timedelta(days=365)
+
+# Set to "0" to make every call recompute. Writes still happen -- it is reads
+# that would otherwise let one test run's leftovers decide the next one's
+# result, and the hooks are subprocesses, so an inherited environment variable
+# is what reaches them.
+ENV_CACHE_READ = "DIRSQL_EMBEDDINGS_CACHE_READ"
+CACHE_READ_OFF = frozenset({"0"})
+
+# Membership rather than `!= "0"`: CPython interns single-character strings, so
+# for any value the environment can hold, `x == "0"` and `x is "0"` agree --
+# which makes the equality form's identity variant untestable in one direction.
+CACHE_READ = os.environ.get(ENV_CACHE_READ, "1") not in CACHE_READ_OFF

{dirsql_plugin_embeddings-0.1.5 → dirsql_plugin_embeddings-0.1.7}/src/dirsql_plugin_embeddings/dirsql.toml

@@ -12,7 +12,10 @@ hook-timeout = 300
 path = "sqlite_vec"
 entrypoint = "sqlite3_vec_init"
 
+# The brace is globset alternation, not a dirsql `{name}` placeholder: the core
+# rewrites those to `*`, but only when the braces hold a bare identifier, so a
+# comma-separated list reaches globset intact.
 [[table]]
 ddl = "CREATE TABLE documents (path TEXT, text TEXT, embedding TEXT)"
-glob = "**/*.md"
+glob = "**/*.{md,pdf}"
 on-file = "dirsql-embeddings-on-file {path}"

{dirsql_plugin_embeddings-0.1.5 → dirsql_plugin_embeddings-0.1.7}/src/dirsql_plugin_embeddings/on_file/read_content.py

@@ -1,13 +1,22 @@
 """The entry point's single read seam: everything the hook reads comes through
 here, so per-format handling has one place to land without touching `on_file`.
 
+The extension is matched case-insensitively, and more permissively than the
+fragment's glob (which globset matches case-sensitively): the hook is a console
+script runnable against any path, not only the ones a glob selected.
+
 Annotations are evaluated at runtime (no ``from __future__ import
 annotations``) so a mutated ``X | None`` union in a signature fails at import
 rather than surviving as an inert string.
 """
 
+import os
+
+from .read_pdf import read_pdf
 from .read_text import read_text
 
 
 def read_content(path: str) -> str:
+    if os.path.splitext(path)[1].lower() == ".pdf":
+        return read_pdf(path)
     return read_text(path)

dirsql_plugin_embeddings-0.1.7/src/dirsql_plugin_embeddings/on_file/read_pdf.py

@@ -0,0 +1,31 @@
+"""Extract a PDF's text, once per version of the file.
+
+pypdf rather than pymupdf: pymupdf is AGPL-3.0, and a runtime dependency of
+this MIT-licensed package propagates to everyone who installs it.
+
+Extraction is the expensive step of a scan and a scan re-reads every matched
+file, so the result is persisted to disk. The cache key is the pair
+``(path, mtime)``: an edited PDF gets a different key rather than a stale hit.
+
+Annotations are evaluated at runtime (no ``from __future__ import
+annotations``) so a mutated ``X | None`` union in a signature fails at import
+rather than surviving as an inert string.
+"""
+
+import os
+
+from pypdf import PdfReader
+
+from ..cache import cache
+
+
+@cache / "extract"
+def extract(path: str, mtime: float) -> str:
+    # `mtime` is unread on purpose: it is what makes an edited PDF a cache
+    # miss, since cachetta derives the key from the arguments it is called with.
+    reader = PdfReader(path)
+    return "\n".join(page.extract_text() for page in reader.pages)
+
+
+def read_pdf(path: str) -> str:
+    return extract(path, os.path.getmtime(path))

dirsql_plugin_embeddings-0.1.7/tests/conftest.py

@@ -0,0 +1,110 @@
+"""Shared fixtures for the integration and e2e tiers.
+
+`stub_server` brings up a real, local, threaded OpenAI-compatible
+`/v1/embeddings` endpoint. Its embedding is a deterministic keyword-count
+vector, so nearest-neighbor search over the fixtures is reproducible without a
+real model or network.
+
+`make_pdf` builds a real PDF by hand rather than pulling in a writer library:
+these tiers mock nothing, and a one-page catalog with a `BT ... Tj ET` content
+stream is enough for pypdf to extract the text back out.
+
+Cache reads are switched off for both tiers. These tiers run the hooks as
+subprocesses, which no in-process mock can reach, so a cached extraction from an
+earlier run would otherwise decide this one's result. Writes are left alone --
+they are harmless, and it is only the read that leaks state between runs.
+"""
+
+import json
+import os
+import threading
+from http.server import BaseHTTPRequestHandler, HTTPServer
+
+os.environ["DIRSQL_EMBEDDINGS_CACHE_READ"] = "0"
+
+import pytest
+
+# One dimension per keyword; the embedding of a text is the per-keyword count.
+KEYWORDS = [
+    "pasta",
+    "cook",
+    "garlic",
+    "git",
+    "code",
+    "review",
+    "tomato",
+    "plant",
+    "seed",
+]
+
+
+def keyword_vector(text):
+    lowered = text.lower()
+    return [float(lowered.count(keyword)) for keyword in KEYWORDS]
+
+
+class _Handler(BaseHTTPRequestHandler):
+    def do_POST(self):
+        length = int(self.headers["Content-Length"])
+        payload = json.loads(self.rfile.read(length))
+        text = payload["input"][0]
+        body = json.dumps(
+            {"data": [{"index": 0, "embedding": keyword_vector(text)}]}
+        ).encode("utf-8")
+        self.send_response(200)
+        self.send_header("Content-Type", "application/json")
+        self.send_header("Content-Length", str(len(body)))
+        self.end_headers()
+        self.wfile.write(body)
+
+    def log_message(self, *args):  # silence the default stderr access log
+        pass
+
+
+def _make_pdf(text):
+    stream = f"BT /F1 24 Tf 72 700 Td ({text}) Tj ET".encode()
+    objs = [
+        b"<< /Type /Catalog /Pages 2 0 R >>",
+        b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
+        b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] "
+        b"/Resources << /Font << /F1 5 0 R >> >> /Contents 4 0 R >>",
+        b"<< /Length "
+        + str(len(stream)).encode()
+        + b" >>\nstream\n"
+        + stream
+        + b"\nendstream",
+        b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>",
+    ]
+    out = bytearray(b"%PDF-1.4\n")
+    offsets = []
+    for index, body in enumerate(objs, start=1):
+        offsets.append(len(out))
+        out += f"{index} 0 obj\n".encode() + body + b"\nendobj\n"
+    xref = len(out)
+    out += f"xref\n0 {len(objs) + 1}\n".encode()
+    out += b"0000000000 65535 f \n"
+    for offset in offsets:
+        out += f"{offset:010d} 00000 n \n".encode()
+    out += (
+        f"trailer\n<< /Size {len(objs) + 1} /Root 1 0 R >>\n"
+        f"startxref\n{xref}\n%%EOF\n"
+    ).encode()
+    return bytes(out)
+
+
+@pytest.fixture
+def make_pdf():
+    return _make_pdf
+
+
+@pytest.fixture
+def stub_server():
+    server = HTTPServer(("127.0.0.1", 0), _Handler)
+    thread = threading.Thread(target=server.serve_forever, daemon=True)
+    thread.start()
+    host, port = server.server_address
+    try:
+        yield f"http://{host}:{port}"
+    finally:
+        server.shutdown()
+        thread.join()

dirsql_plugin_embeddings-0.1.5/src/dirsql_plugin_embeddings/on_file/read_pdf.py

@@ -1,16 +0,0 @@
-"""Extract a PDF's text.
-
-pypdf rather than pymupdf: pymupdf is AGPL-3.0, and a runtime dependency of
-this MIT-licensed package propagates to everyone who installs it.
-
-Annotations are evaluated at runtime (no ``from __future__ import
-annotations``) so a mutated ``X | None`` union in a signature fails at import
-rather than surviving as an inert string.
-"""
-
-from pypdf import PdfReader
-
-
-def read_pdf(path: str) -> str:
-    reader = PdfReader(path)
-    return "\n".join(page.extract_text() for page in reader.pages)

dirsql_plugin_embeddings-0.1.5/tests/conftest.py

@@ -1,62 +0,0 @@
-"""Shared fixtures for the integration and e2e tiers.
-
-`stub_server` brings up a real, local, threaded OpenAI-compatible
-`/v1/embeddings` endpoint. Its embedding is a deterministic keyword-count
-vector, so nearest-neighbor search over the fixtures is reproducible without a
-real model or network.
-"""
-
-import json
-import threading
-from http.server import BaseHTTPRequestHandler, HTTPServer
-
-import pytest
-
-# One dimension per keyword; the embedding of a text is the per-keyword count.
-KEYWORDS = [
-    "pasta",
-    "cook",
-    "garlic",
-    "git",
-    "code",
-    "review",
-    "tomato",
-    "plant",
-    "seed",
-]
-
-
-def keyword_vector(text):
-    lowered = text.lower()
-    return [float(lowered.count(keyword)) for keyword in KEYWORDS]
-
-
-class _Handler(BaseHTTPRequestHandler):
-    def do_POST(self):
-        length = int(self.headers["Content-Length"])
-        payload = json.loads(self.rfile.read(length))
-        text = payload["input"][0]
-        body = json.dumps(
-            {"data": [{"index": 0, "embedding": keyword_vector(text)}]}
-        ).encode("utf-8")
-        self.send_response(200)
-        self.send_header("Content-Type", "application/json")
-        self.send_header("Content-Length", str(len(body)))
-        self.end_headers()
-        self.wfile.write(body)
-
-    def log_message(self, *args):  # silence the default stderr access log
-        pass
-
-
-@pytest.fixture
-def stub_server():
-    server = HTTPServer(("127.0.0.1", 0), _Handler)
-    thread = threading.Thread(target=server.serve_forever, daemon=True)
-    thread.start()
-    host, port = server.server_address
-    try:
-        yield f"http://{host}:{port}"
-    finally:
-        server.shutdown()
-        thread.join()