dirsql-plugin-embeddings 0.1.15__tar.gz → 0.1.17__tar.gz

{dirsql_plugin_embeddings-0.1.15 → dirsql_plugin_embeddings-0.1.17}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: dirsql-plugin-embeddings
-Version: 0.1.15
+Version: 0.1.17
 Summary: First-party dirsql plugin: semantic search over files.
 Requires-Python: >=3.10
 Requires-Dist: cachetta>=0.7
@@ -26,10 +26,15 @@ uvx --with dirsql-plugin-embeddings dirsql "
   SELECT path
   FROM (SELECT path, embed(content ->> 'abstract') AS emb
         FROM './arxiv-firehose/data/**/metadata.json')
+  WHERE emb IS NOT NULL
   ORDER BY vec_distance_cosine(emb, embed('local private models'))
   LIMIT 10"
 ```
 
+`WHERE emb IS NOT NULL` is not optional bookkeeping: a file that is unreadable
+or not valid UTF-8 has NULL content, so its distance is NULL, and SQLite sorts
+NULLs *first* ascending — without the guard those files take the top slots.
+
 For the common case — one glob, one question, top-k paths — the package is
 also its own command, generating and running exactly that SQL:
 

{dirsql_plugin_embeddings-0.1.15 → dirsql_plugin_embeddings-0.1.17}/README.md

@@ -13,10 +13,15 @@ uvx --with dirsql-plugin-embeddings dirsql "
   SELECT path
   FROM (SELECT path, embed(content ->> 'abstract') AS emb
         FROM './arxiv-firehose/data/**/metadata.json')
+  WHERE emb IS NOT NULL
   ORDER BY vec_distance_cosine(emb, embed('local private models'))
   LIMIT 10"
 ```
 
+`WHERE emb IS NOT NULL` is not optional bookkeeping: a file that is unreadable
+or not valid UTF-8 has NULL content, so its distance is NULL, and SQLite sorts
+NULLs *first* ascending — without the guard those files take the top slots.
+
 For the common case — one glob, one question, top-k paths — the package is
 also its own command, generating and running exactly that SQL:
 

dirsql_plugin_embeddings-0.1.17/changelog.d/2026-08-12-null-embedding-guard.md

@@ -0,0 +1,3 @@
+**Fixed**
+
+- **Files that cannot be embedded no longer take the top of the ranking (and no longer break the search).** A matched file that is unreadable, not valid UTF-8, or deleted mid-scan has NULL `content`, so `embed(content)` is NULL — and SQLite sorts NULLs *first* ascending, so those files occupied the top-k slots before anything else went wrong. The generated search SQL now carries `WHERE emb IS NOT NULL`, which fixes the ranking and the failure it led to (a `TypeError` formatting a NULL distance, or, depending on the sqlite-vec build, `Error reading 1st vector: ... found NULL` from SQLite itself). The documented hand-written idiom carries the same guard. (#817)

dirsql_plugin_embeddings-0.1.17/changelog.d/2026-08-12-worker-semaphore-leak.md

@@ -0,0 +1 @@
+**Fixed** the `resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown: {'/loky-…'}` warning that every embedding run printed on stderr. model2vec's encode loop constructs a (disabled) tqdm bar whose default write lock allocates a multiprocessing semaphore, and the dirsql core tears the worker down with a kill, so the semaphore was never released. The worker now installs a process-local tqdm write lock at startup, so no semaphore exists to leak.

dirsql_plugin_embeddings-0.1.17/e2e-attestations/claude-815-loky-semaphore.json

@@ -0,0 +1,7 @@
+{
+  "command": "uv run --with-editable ../../packages/python --with-editable . python -m pytest tests/e2e -q",
+  "ran_at": 1786559931,
+  "exit_code": 0,
+  "commit": "b63619a59bf6138234daa5c08d2e29981729642d",
+  "branch": "claude/815-loky-semaphore"
+}

dirsql_plugin_embeddings-0.1.17/e2e-attestations/claude-817-null-embedding-guard.json

@@ -0,0 +1,7 @@
+{
+  "command": "uv run --with-editable ../../packages/python python -m pytest tests/e2e/ -x -q",
+  "ran_at": 1786558054,
+  "exit_code": 0,
+  "commit": "f1af2bf7bf4c21dd62350b48b4d441356047ef59",
+  "branch": "claude/817-null-embedding-guard"
+}

dirsql_plugin_embeddings-0.1.17/src/dirsql_plugin_embeddings/embedding/progress.py

@@ -0,0 +1,24 @@
+import os
+import sys
+import threading
+
+from tqdm import tqdm
+
+
+def stderr_is_tty():
+    return sys.stderr.isatty()
+
+
+def configure():
+    # Hugging Face hub download bars honor this variable; setdefault keeps an
+    # explicit user choice in force.
+    if not stderr_is_tty():
+        os.environ.setdefault("HF_HUB_DISABLE_PROGRESS_BARS", "1")
+    # tqdm's first bar (model2vec's encode loop constructs one even when
+    # disabled) lazily allocates a multiprocessing RLock -- a named semaphore
+    # registered with the resource tracker. The dirsql core tears this worker
+    # down with a kill, not a graceful exit, so that semaphore is never
+    # unregistered and the tracker prints a "leaked semaphore" warning after
+    # every run. A single worker process needs no cross-process bar
+    # coordination: a process-local lock means no semaphore exists at all.
+    tqdm.set_lock(threading.RLock())

{dirsql_plugin_embeddings-0.1.15 → dirsql_plugin_embeddings-0.1.17}/src/dirsql_plugin_embeddings/search/sql.py

@@ -25,9 +25,14 @@ def embed_call(argument, model):
 def build_search_sql(glob, query, limit, model=None):
     outer = embed_call(quote(query), model)
     inner = embed_call("content", model)
+    # A matched file that is unreadable, not valid UTF-8, or deleted mid-scan
+    # has NULL content, so its embedding is NULL too. Dropping those rows is
+    # what keeps the ranking honest: a NULL distance sorts FIRST ascending in
+    # SQLite, so unrankable files would otherwise take the top-k slots.
     return (
         f"SELECT path, vec_distance_cosine(emb, {outer}) AS distance"
         f" FROM (SELECT path, {inner} AS emb FROM {quote(normalize_glob(glob))})"
+        f" WHERE emb IS NOT NULL"
         f" ORDER BY distance LIMIT {int(limit):d}"
     )
 

dirsql_plugin_embeddings-0.1.15/src/dirsql_plugin_embeddings/embedding/progress.py

@@ -1,13 +0,0 @@
-import os
-import sys
-
-
-def stderr_is_tty():
-    return sys.stderr.isatty()
-
-
-def configure():
-    # Hugging Face hub download bars honor this variable; setdefault keeps an
-    # explicit user choice in force.
-    if not stderr_is_tty():
-        os.environ.setdefault("HF_HUB_DISABLE_PROGRESS_BARS", "1")