dirsql-plugin-embeddings 0.1.6__tar.gz → 0.1.8__tar.gz

{dirsql_plugin_embeddings-0.1.6 → dirsql_plugin_embeddings-0.1.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dirsql-plugin-embeddings
-Version: 0.1.6
+Version: 0.1.8
 Summary: First-party dirsql plugin: semantic search via an OpenAI-compatible embeddings endpoint.
 Requires-Python: >=3.10
 Requires-Dist: cachetta>=0.6.15
@@ -10,7 +10,8 @@ Description-Content-Type: text/markdown
 # dirsql-plugin-embeddings
 
 A first-party [`dirsql`](https://github.com/thekevinscott/dirsql) plugin that
-adds **semantic search** over a directory of Markdown files. It is the worked
+adds **semantic search** over a directory of documents -- Markdown, plain
+text, reStructuredText and PDFs. It is the worked
 implementation behind the [Search documents by
 meaning](https://thekevinscott.github.io/dirsql/howto/search-by-meaning) how-to,
 swapping that guide's local `model2vec` model for any OpenAI-compatible
@@ -30,13 +31,31 @@ is installed alongside it. The fragment declares:
 
 - the [`sqlite-vec`](https://github.com/asg017/sqlite-vec) extension, for
   `vec_distance_cosine()`;
-- a `documents` table whose `on-file` hook embeds each `**/*.md` file into a
-  TEXT `embedding` column;
+- a `documents` table whose `on-file` hook embeds each
+  `**/*.{md,markdown,mdx,rst,txt,pdf}` file into a TEXT `embedding` column;
 - a `pre-query` hook that embeds the incoming question and emits the
   nearest-neighbor SQL.
 
 Both hooks are console scripts that call the same embedder.
 
+Every matched extension except `.pdf` is read as UTF-8 text; a `.pdf` is read with
+[pypdf](https://pypdf.readthedocs.io), whose per-page extracted text is joined
+and embedded like any other document. The extension check is case-insensitive
+(`.PDF` is a PDF), though the glob above is not — globset matches case-sensitively,
+so an uppercase-suffixed file needs its own `glob` entry to be picked up at all.
+
+The glob is an allowlist rather than `**/*` for a reason worth knowing: a hook
+that exits non-zero aborts the entire scan
+([dirsql#697](https://github.com/thekevinscott/dirsql/issues/697)), and reading a
+PNG as UTF-8 does exactly that. Matching everything would mean one image anywhere
+under the root produces no index at all, so the list stays limited to what the
+plugin can actually read.
+
+A PDF that cannot be parsed aborts the scan rather than being skipped: the hook
+exits non-zero and dirsql reports the path and the pypdf reason. A *scanned*,
+image-only PDF is not a failure — pypdf yields no text, and the file is indexed
+with an empty `text`, exactly like an empty `.md`.
+
 ## Configuration
 
 The embedder reads three environment variables (point them at any hosted or
@@ -52,7 +71,7 @@ self-managed OpenAI-compatible inference server):
 
 | Script | Hook | Input | Output |
 |---|---|---|---|
-| `dirsql-embeddings-on-file` | `on-file` | a file's absolute path (`argv[1]`) | one-line JSON row array with `path`, `text`, `embedding` |
+| `dirsql-embeddings-on-file` | `on-file` | a file's absolute path (`argv[1]`), text or PDF | one-line JSON row array with `path`, `text`, `embedding` |
 | `dirsql-embeddings-pre-query` | `pre-query` | a raw request body (`argv[1]`) | nearest-neighbor SQL over `documents` |
 
 `pre-query` accepts both a verbatim server body (`{"q": ...}`) and the CLI

{dirsql_plugin_embeddings-0.1.6 → dirsql_plugin_embeddings-0.1.8}/README.md

@@ -1,7 +1,8 @@
 # dirsql-plugin-embeddings
 
 A first-party [`dirsql`](https://github.com/thekevinscott/dirsql) plugin that
-adds **semantic search** over a directory of Markdown files. It is the worked
+adds **semantic search** over a directory of documents -- Markdown, plain
+text, reStructuredText and PDFs. It is the worked
 implementation behind the [Search documents by
 meaning](https://thekevinscott.github.io/dirsql/howto/search-by-meaning) how-to,
 swapping that guide's local `model2vec` model for any OpenAI-compatible
@@ -21,13 +22,31 @@ is installed alongside it. The fragment declares:
 
 - the [`sqlite-vec`](https://github.com/asg017/sqlite-vec) extension, for
   `vec_distance_cosine()`;
-- a `documents` table whose `on-file` hook embeds each `**/*.md` file into a
-  TEXT `embedding` column;
+- a `documents` table whose `on-file` hook embeds each
+  `**/*.{md,markdown,mdx,rst,txt,pdf}` file into a TEXT `embedding` column;
 - a `pre-query` hook that embeds the incoming question and emits the
   nearest-neighbor SQL.
 
 Both hooks are console scripts that call the same embedder.
 
+Every matched extension except `.pdf` is read as UTF-8 text; a `.pdf` is read with
+[pypdf](https://pypdf.readthedocs.io), whose per-page extracted text is joined
+and embedded like any other document. The extension check is case-insensitive
+(`.PDF` is a PDF), though the glob above is not — globset matches case-sensitively,
+so an uppercase-suffixed file needs its own `glob` entry to be picked up at all.
+
+The glob is an allowlist rather than `**/*` for a reason worth knowing: a hook
+that exits non-zero aborts the entire scan
+([dirsql#697](https://github.com/thekevinscott/dirsql/issues/697)), and reading a
+PNG as UTF-8 does exactly that. Matching everything would mean one image anywhere
+under the root produces no index at all, so the list stays limited to what the
+plugin can actually read.
+
+A PDF that cannot be parsed aborts the scan rather than being skipped: the hook
+exits non-zero and dirsql reports the path and the pypdf reason. A *scanned*,
+image-only PDF is not a failure — pypdf yields no text, and the file is indexed
+with an empty `text`, exactly like an empty `.md`.
+
 ## Configuration
 
 The embedder reads three environment variables (point them at any hosted or
@@ -43,7 +62,7 @@ self-managed OpenAI-compatible inference server):
 
 | Script | Hook | Input | Output |
 |---|---|---|---|
-| `dirsql-embeddings-on-file` | `on-file` | a file's absolute path (`argv[1]`) | one-line JSON row array with `path`, `text`, `embedding` |
+| `dirsql-embeddings-on-file` | `on-file` | a file's absolute path (`argv[1]`), text or PDF | one-line JSON row array with `path`, `text`, `embedding` |
 | `dirsql-embeddings-pre-query` | `pre-query` | a raw request body (`argv[1]`) | nearest-neighbor SQL over `documents` |
 
 `pre-query` accepts both a verbatim server body (`{"q": ...}`) and the CLI

dirsql_plugin_embeddings-0.1.6/e2e-attestations/claude-701-cache-pdf.json → dirsql_plugin_embeddings-0.1.8/e2e-attestations/claude-706-widen-glob.json

@@ -1,7 +1,7 @@
 {
   "command": "uv run --with sqlite-vec --with-editable . --with-editable ../../packages/python python -m pytest tests/e2e -x -q",
-  "ran_at": 1785337806,
+  "ran_at": 1785339217,
   "exit_code": 0,
-  "commit": "43a5ab7f9e72cde7f55e943649f4e24b85d7fe69",
-  "branch": "claude/701-cache-pdf"
+  "commit": "f96d82085395d4e38bd74992a6ae3c7b029d5f60",
+  "branch": "claude/706-widen-glob"
 }

{dirsql_plugin_embeddings-0.1.6 → dirsql_plugin_embeddings-0.1.8}/pyproject.toml

@@ -16,8 +16,8 @@ requires-python = ">=3.10"
 # MIT-licensed package propagates to everyone who installs it.
 #
 # cachetta >=0.6.15 is load-bearing, not a nicety: every earlier release
-# declares `requires-python >=3.12`, which would drag this package's `>=3.11`
-# floor up with it. 0.6.15 lowered it to `>=3.10`.
+# declares `requires-python >=3.12`, which would drag this package's floor up
+# with it. 0.6.15 lowered cachetta's own floor to `>=3.10`, matching ours.
 dependencies = ["pypdf>=6", "cachetta>=0.6.15"]
 
 [tool.hatch.version]

dirsql_plugin_embeddings-0.1.8/src/dirsql_plugin_embeddings/dirsql.toml

@@ -0,0 +1,27 @@
+# Fragment shipped by the dirsql-plugin-embeddings plugin.
+# Discovered via the package's [project.entry-points.dirsql] declaration and
+# injected by the launcher as an ordinary -c flag ("installed = active", #529).
+#
+# The table name here MUST match pre_query.TABLE_NAME (the SQL the pre-query
+# hook prints queries this table).
+[dirsql]
+pre-query = "dirsql-embeddings-pre-query {args}"
+hook-timeout = 300
+
+[[dirsql.extension]]
+path = "sqlite_vec"
+entrypoint = "sqlite3_vec_init"
+
+# The brace is globset alternation, not a dirsql `{name}` placeholder: the core
+# rewrites those to `*`, but only when the braces hold a bare identifier, so a
+# comma-separated list reaches globset intact.
+#
+# An allowlist rather than `**/*`, and not for tidiness: a hook that fails takes
+# the whole scan down (dirsql#697), and reading a PNG as UTF-8 fails. Matching
+# everything would mean one image anywhere under the root yields no index at
+# all. The list is therefore what `read_content` can actually read -- prose, not
+# source -- and widening it is safe only as far as that stays true.
+[[table]]
+ddl = "CREATE TABLE documents (path TEXT, text TEXT, embedding TEXT)"
+glob = "**/*.{md,markdown,mdx,rst,txt,pdf}"
+on-file = "dirsql-embeddings-on-file {path}"

{dirsql_plugin_embeddings-0.1.6 → dirsql_plugin_embeddings-0.1.8}/src/dirsql_plugin_embeddings/on_file/read_content.py

@@ -1,13 +1,22 @@
 """The entry point's single read seam: everything the hook reads comes through
 here, so per-format handling has one place to land without touching `on_file`.
 
+The extension is matched case-insensitively, and more permissively than the
+fragment's glob (which globset matches case-sensitively): the hook is a console
+script runnable against any path, not only the ones a glob selected.
+
 Annotations are evaluated at runtime (no ``from __future__ import
 annotations``) so a mutated ``X | None`` union in a signature fails at import
 rather than surviving as an inert string.
 """
 
+import os
+
+from .read_pdf import read_pdf
 from .read_text import read_text
 
 
 def read_content(path: str) -> str:
+    if os.path.splitext(path)[1].lower() == ".pdf":
+        return read_pdf(path)
     return read_text(path)

{dirsql_plugin_embeddings-0.1.6 → dirsql_plugin_embeddings-0.1.8}/tests/conftest.py

@@ -5,6 +5,10 @@
 vector, so nearest-neighbor search over the fixtures is reproducible without a
 real model or network.
 
+`make_pdf` builds a real PDF by hand rather than pulling in a writer library:
+these tiers mock nothing, and a one-page catalog with a `BT ... Tj ET` content
+stream is enough for pypdf to extract the text back out.
+
 Cache reads are switched off for both tiers. These tiers run the hooks as
 subprocesses, which no in-process mock can reach, so a cached extraction from an
 earlier run would otherwise decide this one's result. Writes are left alone --
@@ -57,6 +61,42 @@ class _Handler(BaseHTTPRequestHandler):
         pass
 
 
+def _make_pdf(text):
+    stream = f"BT /F1 24 Tf 72 700 Td ({text}) Tj ET".encode()
+    objs = [
+        b"<< /Type /Catalog /Pages 2 0 R >>",
+        b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
+        b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] "
+        b"/Resources << /Font << /F1 5 0 R >> >> /Contents 4 0 R >>",
+        b"<< /Length "
+        + str(len(stream)).encode()
+        + b" >>\nstream\n"
+        + stream
+        + b"\nendstream",
+        b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>",
+    ]
+    out = bytearray(b"%PDF-1.4\n")
+    offsets = []
+    for index, body in enumerate(objs, start=1):
+        offsets.append(len(out))
+        out += f"{index} 0 obj\n".encode() + body + b"\nendobj\n"
+    xref = len(out)
+    out += f"xref\n0 {len(objs) + 1}\n".encode()
+    out += b"0000000000 65535 f \n"
+    for offset in offsets:
+        out += f"{offset:010d} 00000 n \n".encode()
+    out += (
+        f"trailer\n<< /Size {len(objs) + 1} /Root 1 0 R >>\n"
+        f"startxref\n{xref}\n%%EOF\n"
+    ).encode()
+    return bytes(out)
+
+
+@pytest.fixture
+def make_pdf():
+    return _make_pdf
+
+
 @pytest.fixture
 def stub_server():
     server = HTTPServer(("127.0.0.1", 0), _Handler)

dirsql_plugin_embeddings-0.1.6/src/dirsql_plugin_embeddings/dirsql.toml

@@ -1,18 +0,0 @@
-# Fragment shipped by the dirsql-plugin-embeddings plugin.
-# Discovered via the package's [project.entry-points.dirsql] declaration and
-# injected by the launcher as an ordinary -c flag ("installed = active", #529).
-#
-# The table name here MUST match pre_query.TABLE_NAME (the SQL the pre-query
-# hook prints queries this table).
-[dirsql]
-pre-query = "dirsql-embeddings-pre-query {args}"
-hook-timeout = 300
-
-[[dirsql.extension]]
-path = "sqlite_vec"
-entrypoint = "sqlite3_vec_init"
-
-[[table]]
-ddl = "CREATE TABLE documents (path TEXT, text TEXT, embedding TEXT)"
-glob = "**/*.md"
-on-file = "dirsql-embeddings-on-file {path}"