dirsql-plugin-embeddings 0.1.15__tar.gz → 0.1.16__tar.gz

{dirsql_plugin_embeddings-0.1.15 → dirsql_plugin_embeddings-0.1.16}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: dirsql-plugin-embeddings
-Version: 0.1.15
+Version: 0.1.16
 Summary: First-party dirsql plugin: semantic search over files.
 Requires-Python: >=3.10
 Requires-Dist: cachetta>=0.7
@@ -26,10 +26,15 @@ uvx --with dirsql-plugin-embeddings dirsql "
   SELECT path
   FROM (SELECT path, embed(content ->> 'abstract') AS emb
         FROM './arxiv-firehose/data/**/metadata.json')
+  WHERE emb IS NOT NULL
   ORDER BY vec_distance_cosine(emb, embed('local private models'))
   LIMIT 10"
 ```
 
+`WHERE emb IS NOT NULL` is not optional bookkeeping: a file that is unreadable
+or not valid UTF-8 has NULL content, so its distance is NULL, and SQLite sorts
+NULLs *first* ascending — without the guard those files take the top slots.
+
 For the common case — one glob, one question, top-k paths — the package is
 also its own command, generating and running exactly that SQL:
 

{dirsql_plugin_embeddings-0.1.15 → dirsql_plugin_embeddings-0.1.16}/README.md

@@ -13,10 +13,15 @@ uvx --with dirsql-plugin-embeddings dirsql "
   SELECT path
   FROM (SELECT path, embed(content ->> 'abstract') AS emb
         FROM './arxiv-firehose/data/**/metadata.json')
+  WHERE emb IS NOT NULL
   ORDER BY vec_distance_cosine(emb, embed('local private models'))
   LIMIT 10"
 ```
 
+`WHERE emb IS NOT NULL` is not optional bookkeeping: a file that is unreadable
+or not valid UTF-8 has NULL content, so its distance is NULL, and SQLite sorts
+NULLs *first* ascending — without the guard those files take the top slots.
+
 For the common case — one glob, one question, top-k paths — the package is
 also its own command, generating and running exactly that SQL:
 

dirsql_plugin_embeddings-0.1.16/changelog.d/2026-08-12-null-embedding-guard.md

@@ -0,0 +1,3 @@
+**Fixed**
+
+- **Files that cannot be embedded no longer take the top of the ranking (and no longer break the search).** A matched file that is unreadable, not valid UTF-8, or deleted mid-scan has NULL `content`, so `embed(content)` is NULL — and SQLite sorts NULLs *first* ascending, so those files occupied the top-k slots before anything else went wrong. The generated search SQL now carries `WHERE emb IS NOT NULL`, which fixes the ranking and the failure it led to (a `TypeError` formatting a NULL distance, or, depending on the sqlite-vec build, `Error reading 1st vector: ... found NULL` from SQLite itself). The documented hand-written idiom carries the same guard. (#817)

dirsql_plugin_embeddings-0.1.16/e2e-attestations/claude-817-null-embedding-guard.json

@@ -0,0 +1,7 @@
+{
+  "command": "uv run --with-editable ../../packages/python python -m pytest tests/e2e/ -x -q",
+  "ran_at": 1786558054,
+  "exit_code": 0,
+  "commit": "f1af2bf7bf4c21dd62350b48b4d441356047ef59",
+  "branch": "claude/817-null-embedding-guard"
+}

{dirsql_plugin_embeddings-0.1.15 → dirsql_plugin_embeddings-0.1.16}/src/dirsql_plugin_embeddings/search/sql.py

@@ -25,9 +25,14 @@ def embed_call(argument, model):
 def build_search_sql(glob, query, limit, model=None):
     outer = embed_call(quote(query), model)
     inner = embed_call("content", model)
+    # A matched file that is unreadable, not valid UTF-8, or deleted mid-scan
+    # has NULL content, so its embedding is NULL too. Dropping those rows is
+    # what keeps the ranking honest: a NULL distance sorts FIRST ascending in
+    # SQLite, so unrankable files would otherwise take the top-k slots.
     return (
         f"SELECT path, vec_distance_cosine(emb, {outer}) AS distance"
         f" FROM (SELECT path, {inner} AS emb FROM {quote(normalize_glob(glob))})"
+        f" WHERE emb IS NOT NULL"
         f" ORDER BY distance LIMIT {int(limit):d}"
     )