ph-text-index 0.1.0__tar.gz

ph_text_index-0.1.0/.gitignore

@@ -0,0 +1,31 @@
+# Build and environment
+.venv/
+dist/
+build/
+*.egg-info/
+__pycache__/
+*.py[cod]
+jjt/
+w2/
+
+# Tooling caches
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.coverage
+# Written by the guest's subprocess collectors and combined at the end of the
+# run; see `conftest.GUEST_COVERAGE_RC`.
+.coverage-guest*
+htmlcov/
+# Dropped at the repo root by pytest-textual-snapshot when a snapshot test
+# fails. The reference snapshots under `__snapshots__/` are the committed
+# expectation; this is the diff viewer for a run that did not match one.
+snapshot_report.html
+
+# Reference checkouts of the upstream projects this port reads from. Vendored
+# locally so the plans' citations are verifiable; never part of this repo.
+sources/
+
+# Local scratch
+.ph/
+*.local.yaml

ph_text_index-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Charles Tabor
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

ph_text_index-0.1.0/PKG-INFO

@@ -0,0 +1,342 @@
+Metadata-Version: 2.5
+Name: ph-text-index
+Version: 0.1.0
+Summary: pH plugin: `text_index` and `text_search`, semantic retrieval over documents on a turbovec index.
+Project-URL: Homepage, https://github.com/chastabor/pH
+Project-URL: Repository, https://github.com/chastabor/pH
+Project-URL: Documentation, https://github.com/chastabor/pH/blob/main/docs/README.md
+Project-URL: Issues, https://github.com/chastabor/pH/issues
+Author: Charles Tabor
+License-Expression: MIT
+License-File: LICENSE
+Keywords: agent,embeddings,llm,retrieval,semantic-search
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: MacOS
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development
+Classifier: Typing :: Typed
+Requires-Python: >=3.12
+Requires-Dist: numpy>=1.20
+Requires-Dist: ph-core==0.1.0
+Requires-Dist: turbovec>=1.0
+Provides-Extra: local
+Requires-Dist: sentence-transformers>=3.0; extra == 'local'
+Requires-Dist: torch>=2.2; extra == 'local'
+Provides-Extra: nomic
+Requires-Dist: einops>=0.7; extra == 'nomic'
+Requires-Dist: sentence-transformers>=3.0; extra == 'nomic'
+Requires-Dist: torch>=2.2; extra == 'nomic'
+Description-Content-Type: text/markdown
+
+# ph-text-index
+
+*`text_index` and `text_search`: semantic retrieval over the agent's own
+documents, on a local [turbovec][turbovec] index.*
+
+`grep` finds the string you typed. This finds the passage you meant — and hands
+back the `path:start-end` it came from, so the model can answer from the passage
+or `read` the file around it.
+
+```bash
+phern --profile llama --provider llama --model <model> \
+   --patch '{insert: [{id: text-index, name: text-index},
+                      {id: text-index-local, name: text-index-local}]}' \
+   -p "index docs/ then tell me how the workspace seam decides a containment tier"
+```
+
+## Two rows, and why
+
+| row | what it is |
+|---|---|
+| `text-index` | the seam: `ctx.text_index`, the chunker, the turbovec store, and the two tools |
+| `text-index-local` | the provider: a `sentence-transformers` model, loaded on first use |
+
+The split is the shape `subprocess`/`subprocess-local` and
+`code-runtime`/`code-runtime-python` already have in this tree, and it pays for
+itself three times:
+
+* a deployment with an embeddings **endpoint** — llama.cpp's `/v1/embeddings`, a
+  provider's API — writes its own row and keeps the tools;
+* the **tests** run the real turbovec index against a deterministic stub
+  embedder, so the suite proves chunking, paging and persistence without
+  downloading a model;
+* the row that pulls in **torch** is one a profile can leave out.
+
+**The tools appear only once an embedder is claimed.** `TextIndexSeam.register`
+registers them on the provider's own scope, so a profile with the seam and no
+provider advertises nothing at all — the rule `subagent-task` states, for the
+same reason: a tool named in every prompt and refused on every call spends
+context teaching the model a capability the deployment does not have.
+
+## The two tools
+
+**`text_index(paths, glob?, forget?)`** reads each document through `ctx.fs`,
+cuts it into passages, embeds them and stores them. Re-indexing a file replaces
+its passages, so running it again after an edit is correct. `forget: true`
+removes instead.
+
+**Nothing watches the filesystem** — no daemon, no watcher. The index changes
+when `text_index` runs and at no other moment, so run it again after edits.
+
+That is cheap now because **git or jj is asked first** (`ph.seams.changes`): a
+document the version control vouches for is not read, not re-cut, and — the part
+that matters — **not re-embedded**. This loop has no content digest of its own,
+so before the filter every call re-embedded the whole corpus: 20 s of MiniLM for
+this repo's `docs/`, or 185 s under nomic. Which backend answers is the
+workspace provider's to state; a tree with no version control behaves exactly as
+it did before, which is a test rather than a hope. A document past `max_bytes` is *skipped and
+reported*, not an error — a call over a directory should not fail because it
+found a minified bundle, and a caller who never learns what was skipped cannot
+tell a quiet corpus from a quiet failure.
+
+**`text_search(query, k?, paths?)`** returns the passages with their line
+ranges and scores. `paths` restricts the search to a subtree.
+
+An earlier version of this paragraph claimed a filtered search costs *less* than
+an unfiltered one, because turbovec filters inside the SIMD kernel. Measured,
+that is wrong: the kernel does, but building the id list on the Python side
+dominates it by roughly fifteen times (0.11 ms against a 0.22 ms unfiltered
+search on a 10 000-chunk index, and selectivity changes nothing). Use `paths`
+for **precision**, not for speed.
+
+## Everything is read through `ctx.fs`
+
+Every indexed byte arrives via `ctx.fs.read`, so `fs/read-intent` fires,
+`permissions-fs` decides, the workspace tier bounds the path and every
+registered screen gets its say — the same door `read` goes through, and for the
+reason `tool-attach` insists on it (I-9). It matters more here, because
+indexing is a *bulk* read: a tool that walked the tree with `Path.open` would be
+an exfiltration primitive with a glob argument.
+
+The corollary: the index is **per-deployment, not per-agent**. Two agents share
+`$PH_CACHE/text-index/<embedder digest>` unless a profile says otherwise, so a
+passage one indexed is retrievable by another. Right for a documentation corpus,
+wrong for anything private — `path:` in the row's config is how you separate
+them.
+
+## Chunking
+
+Paragraphs, not a fixed character stride. A stride is simpler and reliably
+splits the one sentence that answers the query across two chunks, so neither
+retrieves; blank lines are where the document's own author already said "new
+idea". Every chunk carries its 1-based line span, because a chunk that does not
+know its place in the file is a wall of prose the agent must then go and locate.
+`overlap_chars` carries the previous chunk's tail forward, which is the
+concession to a paragraph whose meaning depends on the one before it.
+
+## On disk
+
+`index.tvim` is turbovec's own format, written with `sync()` — incremental, one
+fsync per call, crash-safe at any byte. `chunks.json` is the sidecar: path, line
+span and the passage text.
+
+**The sidecar holds the text on purpose.** Storing only pointers would keep the
+file small and make every hit a second tool call before the model learns whether
+the hit was any good — answered from a file that may have changed since it was
+indexed. It is also the ceiling: the sidecar is rewritten whole on every save,
+which is fine for a corpus of documents and wrong for millions.
+
+A crash between the two writes can leave them diverged, so vectors are committed
+first and `open` reconciles by trusting the sidecar and dropping any id the index
+cannot answer for — a chunk nobody can retrieve is invisible, where a vector
+with no text would surface as a hit this row could not describe.
+
+### Calibration
+
+turbovec's TQ+ calibration is worth 2.5 to 8.7 points of R@10, and upstream is
+emphatic about the one way to get it wrong: the sample must be a uniform random,
+representative draw of what the index will hold, and a clustered prefix "fits a
+calibration that actively destroys recall". An incremental indexer has no such
+draw at the moment it would have to commit one — the first document is the most
+clustered prefix there is.
+
+So it is committed in exactly the situation upstream describes: the index is
+**empty** and the incoming batch is large enough (1 024 rows) to sample from,
+in which case a random sample of that batch *is* a representative draw and
+calibrating before the add is the documented order. Otherwise the index stays
+uncalibrated, which is plain TurboQuant — good rather than wrong. `calibrate:
+false` turns even that off.
+
+## Using it from the RLM
+
+Nothing to do: under Code Mode every registered tool is in the generated SDK
+listing, so these arrive as `await tools.text_index(...)` and
+`await tools.text_search(...)` with no work from this package. The
+[`rlm-indexed`](../ph-app/src/ph_app/profiles.py) profile is `rlm-stable` plus
+this bundle and `ph-code-graph`'s:
+
+```bash
+phern --profile rlm-indexed --provider llama --model <model> --mode tui
+```
+
+This package registers a `ph.bundles` entry point so that profile is *hidden*
+on an install without this distribution rather than offered and then failing at
+mount — `available_profiles()` gates on bundle resolution, and the refusal names
+the package to install.
+
+The bundle carries **both** rows, and that matters: the seam registers no tools
+until an embedder is claimed, so a bundle shipping only `text-index` would mount
+a service and advertise nothing, with no error anywhere. A deployment bringing
+its own embedder disables `text-index-local` and mounts its own provider row.
+
+## Installing
+
+`sentence-transformers` is a **hard dependency**, so installing this package is
+how a deployment gets a working index. That is most of a gigabyte of torch, in
+every environment that resolves the package, CI included. Moving it to an
+extra is one line in `pyproject.toml`; the cost of that is that
+`pip install ph-text-index` no longer gives you something that runs. The
+seam/provider split keeps the choice cheap either way — nothing but
+`text-index-local` imports it.
+
+## Switching models
+
+The default is **`sentence-transformers/all-MiniLM-L6-v2`** — 384-dimensional,
+about 90 MB, symmetric so it needs no prefixes, and fast enough on a CPU that
+indexing a documentation tree is a coffee break. It is the *small* choice, not
+the best one.
+
+The index directory is keyed by a digest of the embedder's identity — model
+name **and** both prefixes, because all three move the vector space — so
+switching is safe to try: a new model gets its own index and switching back
+finds the old one intact. Pointing `path:` at a fixed directory and then
+changing the model raises `IndexMismatch` rather than returning neighbours
+computed in a space nothing shares.
+
+### A worked upgrade: `nomic-embed-text-v1.5` — measured
+
+Run against this repository's `docs/seams` (30 documents, 235 passages), same
+queries, same chunking:
+
+| | MiniLM-L6-v2 | nomic-embed-text-v1.5 |
+|---|---|---|
+| dimensions | 384 | 768 |
+| load, cold | 19 s | 26 s |
+| index 235 passages | **5.4 s** | 41.7 s |
+| *"stop an agent writing in my own checkout"* | workspace.md (0.319) | workspace.md (**0.599**) |
+| *"what confines a shell command on linux"* | shell.md (0.529) | shell.md (**0.753**) |
+
+On a harder corpus — all 57 files of `docs/`, 539 passages, eight paraphrase
+queries scored on whether the top hit was the right *file* — nomic got **3/8**
+against MiniLM's **1/8**, at 185 s of indexing against 20 s.
+
+Read that honestly in both directions. nomic retrieves better and separates hits
+far more confidently; it also costs roughly **8× the indexing time**, and 3/8 is
+not good retrieval in absolute terms. Some of those misses are the scoring being
+crude (a filename substring), but not all of them: a corpus of long design notes
+is genuinely hard, and *narrowing the corpus* helped more than changing the model
+— MiniLM found `workspace.md` over `docs/seams` and missed it over all of
+`docs/`. Use `paths=` before reaching for a bigger model.
+
+**MiniLM stays the default** for that reason: the cheap thing is good enough for
+a scoped corpus, and the expensive thing does not rescue an unscoped one.
+
+
+
+768-dimensional, 8192-token context, materially better retrieval than MiniLM.
+Three things it needs, and each is a field rather than a special case:
+
+```bash
+pip install 'ph-text-index[nomic]'   # einops, which its remote code imports
+```
+
+```yaml
+- id: text-index-local
+  config:
+    model: nomic-ai/nomic-embed-text-v1.5
+    # Asymmetric: it wants to know whether it is embedding a question or a
+    # passage. Both prefixes are part of the index identity, so this gets its
+    # own directory automatically.
+    queryPrefix: "search_query: "
+    documentPrefix: "search_document: "
+    # Its config.json carries an `auto_map` pointing at nomic-ai/nomic-bert-2048,
+    # so sentence-transformers will not load it without this. Read the next
+    # paragraph before setting it.
+    trustRemoteCode: true
+```
+
+**`trustRemoteCode` is a trust decision, not a compatibility flag.** It
+downloads Python from the model's repository — here from a *second* repository
+via `auto_map` — and executes it in the harness process. It is off by default
+and nothing infers it from a load failure, because "retry with arbitrary code
+execution enabled" is not a fallback. It is deliberately *not* part of the index
+identity: it governs what may load, not where a vector lands, so granting or
+revoking it does not invalidate an index.
+
+A model whose weights are safetensors and whose architecture `transformers`
+already knows needs none of this — `BAAI/bge-base-en-v1.5` and the `e5` family
+are drop-in with prefixes alone.
+
+**And the `[nomic]` extra is not optional politeness.** Without `einops` the
+model downloads its weights *and* its remote code successfully and then dies at
+import:
+
+```
+ImportError: This modeling file requires the following packages that were not
+found in your environment: einops
+```
+
+A missing transitive dependency that only the model's own code knows about, and
+that no amount of pre-downloading would have caught. It is the reason
+provisioning **loads** the model rather than fetching it — see below.
+
+## Where the weights live, and installing on purpose
+
+`$PH_CACHE/models`, set by the row (`cache:` overrides it). Left to
+`sentence-transformers` they would go to `$HF_HOME` or `~/.cache/huggingface` —
+outside all three of pH's roots, so a gigabyte of weights would sit somewhere
+`phern doctor` never mentions and `rm -rf $PH_CACHE` would not reclaim. Rebuildable
+and large is the lifecycle `$PH_CACHE` names (Q1), and the runtime venv is there
+for the same reason.
+
+**Provision before the agent needs it**, three ways, in increasing order of
+"nobody is watching":
+
+```
+/text-index status     # is the model ready? costs nothing, downloads nothing
+/text-index install    # fetch and load it now, and say what happened
+```
+
+A **command**, not a tool: a person asks the harness to do this, and routing it
+through a model turn would put the model in the log as having decided it. It
+costs no turn. `phern doctor` answers the same question without mounting an agent —
+the section reports `model: not loaded — /text-index install`.
+
+For an **unattended** run — a daemon, a scheduled tick, `phern -p` in CI — there is
+nobody to type either:
+
+```yaml
+- id: text-index-local
+  config:
+    preload: true
+```
+
+That loads the model at **mount** and refuses the mount with a `MountRefusal`
+sentence if it cannot, which is the cookbook's own rule: refuse at mount, not at
+first use, because by then the agent is running and "refuse to start" has already
+been disobeyed. Off by default, because a person at a TUI would rather the
+harness start now and pay for the model when they use it.
+
+All three call the same thing, and it **loads** the model rather than fetching
+it — the `einops` failure above is exactly why. Whatever upstream says on
+failure is passed through verbatim, because that sentence names the fix.
+
+## The skill
+
+The package installs a `text-search` skill, so an RLM under progressive
+disclosure sees one line in its catalog and can read the page when it needs
+it — how to scope a search, that a score is not a confidence, that a passage can
+be an argument the document goes on to reject.
+
+Registered by the row rather than found by a directory scan, so it arrives
+exactly when the tools do and leaves with them: `skills-progressive` ships an
+empty `paths` on purpose, because scanning a well-known directory would make
+"install a skill" mean "drop a file somewhere", and a skill is something a
+distribution installs deliberately (I7).
+
+[turbovec]: https://pypi.org/project/turbovec/