@ninjaxtools/slopdex 0.9.0 → 0.11.0

package/docs/implementation.md

@@ -7,12 +7,12 @@ For installation, command examples, configuration, and result interpretation, se
 | Module | Responsibility |
 | --- | --- |
 | `src/cli.ts` | Argument parsing, configuration, automatic refresh/recovery, diagnostics, and output selection. |
-| `src/code-index.ts` | Index lifecycle, file preparation, Git/working-tree reconciliation, embedding/summary caching, and search facade. |
+| `src/code-index.ts` | Index lifecycle, file preparation, Git/working-tree reconciliation, embedding/description caching, and search facade. |
 | `src/parser/` | Language dispatch, native Tree-sitter extraction, callable identity, and recoverable diagnostics. |
 | `src/source-policy.ts`, `src/gitignore.ts` | Supported paths, built-in/config exclusions, nested ignore rules. |
 | `src/git/repository.ts` | Git commits, trees, blobs, diffs, ancestry, and working-tree changes. |
-| `src/embeddings/`, `src/summaries/` | Provider requests and provider profiles. |
-| `src/storage/database.ts` | SQLite schema, migrations, transactions, metadata, and vector queries. |
+| `src/embeddings/`, `src/descriptions/` | Provider requests and provider profiles. |
+| `src/storage/database.ts` | SQLite schema, durable artifact caches, transactions, metadata, and vector queries. |
 | `src/search/` | Analysis scoring selection and cross-index neighbor discovery. |
 | `src/analysis/cohesion.ts` | Physical distance, gap scores, aggregate affinity, and groups. |
 | `src/format.ts` | Human-readable search, cluster, and cohesion output. |
@@ -44,18 +44,20 @@ The CLI reconciles an index before most commands. Library callers choose when to
 
 Git updates resolve the target commit, validate ancestry against the checkpoint, reconcile committed blobs, and optionally overlay current working-tree files when the target equals HEAD. Overlays account for staged, unstaged, untracked, renamed, and deleted paths. The saved checkpoint remains the committed base. Historical/other-branch targets are committed-only. Explicit `updateFiles` operations do not advance the checkpoint.
 
-File preparation and provider calls precede database updates. Generation checks detect concurrent index changes; working-tree updates also verify source and ignore-rule stability. SQLite transactions apply related file, function, vector, summary, diagnostic, and checkpoint changes together. Failed initialization removes its incomplete index artifacts.
+Tree-sitter results and each valid provider result are committed to content-addressed cache tables immediately. Generation checks detect concurrent logical index changes; working-tree updates also verify source and ignore-rule stability. A separate SQLite transaction atomically applies related file, function, diagnostic, description-reference, and checkpoint changes. Failed or aborted initialization retains the valid database and its cache rows so the next invocation resumes without repeating completed work.
 
-Storage uses Node's `node:sqlite` and `sqlite-vec`. Writable connections enable WAL, foreign keys, and normal synchronization. The schema contains:
+Storage uses Node's `node:sqlite` and `sqlite-vec`. Writable connections enable WAL, foreign keys, and full synchronization. The schema contains:
 
-- `metadata`: repository root, embedding and summary profiles, generation, checkpoint, schema version, and feature/scan state.
+- `metadata`: repository root, embedding and description profiles, generation, checkpoint, schema version, and feature/scan state.
 - `files`: paths, content hashes, blob IDs, source mode, previous paths, language, and size.
-- `functions`: identity, names, signatures, locations, source, provenance, and embedding/summary references.
-- `embeddings` and `summary_embeddings`: cached vectors, with summary text in the latter.
+- `functions`: identity, names, signatures, locations, source, provenance, and embedding/description references.
+- `embeddings`: code, description, and query vectors keyed by embedding profile, operation, and exact input.
+- `description_cache`: generated description text keyed independently by description profile and complete source context.
+- `parse_cache`: successful Tree-sitter extraction results keyed by parser strategy, path, and file-content hash.
 - `callable_provenance`: first-seen committed source identity.
 - `indexing_errors`: diagnostics associated with files.
 
-Current schema version is `4`. Supported older versions migrate automatically; the diagnostics migration marks a full rescan pending. Metadata validation rejects incompatible roots, embedding profiles, and unsupported schemas. CLI `--force-reindex` recreates incompatible indexes, preserving enabled OpenAI summary settings for the same repository where possible. Git divergence reconciliation is a separate operation controlled by `--rebuild-on-divergence`.
+Current schema version is `6`. Earlier schemas are intentionally incompatible and require `--force-reindex`. Metadata validation rejects incompatible roots, embedding profiles, and unsupported schemas. For a schema-6 index, `--force-reindex` clears logical index state while retaining content-addressed caches; incompatible older databases are recreated. Enabled OpenAI description settings are preserved for the same repository where possible. Git divergence reconciliation is a separate operation controlled by `--rebuild-on-divergence`.
 
 ### Diagnostics
 
@@ -63,28 +65,30 @@ Diagnostics cover parse errors, parser exceptions, extraction failures, read fai
 
 Diagnostics commit with their corresponding file update. Updates retry failed files even when their Git blobs are unchanged; successful replacement, deletion, and exclusion clear failures. Standalone readers inspect saved diagnostics without constructing an embedding provider. The CLI's exit handler reports remaining failures for source and target indexes; version exits before registering that handler.
 
-## Embeddings and summaries
+## Embeddings and descriptions
 
 Embedding profiles consist of provider, model, dimensions, and strategy version. Cross-index analysis requires matching profiles. Vectors are validated and normalized before storage/search.
 
-- OpenAI defaults to `text-embedding-3-large`, 3072 dimensions, strategy `callable-v1`. Inputs are truncated to 8192 `cl100k_base` tokens.
-- Jina defaults to `jina-embeddings-v4`, 1024 dimensions, strategy `callable-v1:code-query-passage`. Requests distinguish `code.passage` documents from `code.query` queries and enable truncation.
+- OpenAI defaults to `text-embedding-3-large`, 3072 dimensions, strategy `callable-v2`. Inputs are truncated to 8192 `cl100k_base` tokens.
+- Jina defaults to `jina-embeddings-v4`, 1024 dimensions, strategy `callable-v2:code-query-passage`. Requests distinguish `code.passage` documents from `code.query` queries and enable truncation.
+
+Embedding inputs identify language, callable kind, qualified symbol, signature, documentation, and source. For Python, a function's first-statement docstring is included in a separate `documentation` section in addition to remaining part of the callable source.
 
 Purpose generation uses OpenAI's Responses API with `gpt-5.6-sol` and strategy `callable-purpose-v1`. The prompt asks for one to three sentences describing responsibility and visible relationships, using repository name, path, callable source, and full file context. Requests use `store: false`. Generated text is embedded with the configured embedding provider.
 
-Summary inputs include contextual and profile information so file-context, path, or model changes invalidate relevant cached results. Unchanged inputs reuse summaries and vectors. `useSummaries` persists the profile and enabled state; later updates attach summaries automatically. Summary generation and embedding preparation finish before the corresponding database transaction, preventing partially updated callable records on provider failure. Deleting a function removes it from summary search.
+Description inputs include contextual and profile information so file-context, path, or model changes invalidate relevant cached results. Description text identity is independent of the embedding profile, allowing an embedding-model change to reuse generation output while producing the required new vector. Every validated description and vector is cached before indexing continues. `useDescriptions` persists the profile and enabled state; `disableDescriptions` turns automatic updates and description search/scoring off without deleting cached artifacts. Function references are attached only by the final logical transaction, so provider failure cannot expose partially updated callable records. Deleting a function removes it from description search but retains reusable cache rows.
 
 ## Similarity and analysis
 
-`search` embeds a query and searches code vectors; `search-summary` searches summary vectors. Analysis uses code-only cosine similarity unless all indexed callables have enabled summary embeddings. Cross-index analysis requires completeness on both sides. Summary-generator models may differ across indexes even though embedding profiles must match.
+`search` embeds a query and searches code vectors; `search-description` searches description vectors. Analysis uses code-only cosine similarity unless all indexed callables have enabled description embeddings. Cross-index analysis requires completeness on both sides. Description-generator models may differ across indexes even though embedding profiles must match.
 
-When summaries are complete:
+When descriptions are complete:
 
 ```text
-similarity = 0.5 * codeSimilarity + 0.5 * summarySimilarity
+similarity = 0.5 * codeSimilarity + 0.5 * descriptionSimilarity
 ```
 
-Both component scores and the average are computed in one SQLite query. Name, line-count, and path exclusions plus similarity bounds apply before ranking/limiting. Range upper bounds are exclusive. Combined JSON includes component scores; analysis metadata records mode, weights, and summary profiles.
+Both component scores and the average are computed in one SQLite query. Name, line-count, and path exclusions plus similarity bounds apply before ranking/limiting. Range upper bounds are exclusive. Combined JSON includes component scores; analysis metadata records mode, weights, and description profiles.
 
 Cross-search selects sources, queries neighbors per source, and deduplicates unordered same-index pairs unless symmetric results are requested. Self-matches are excluded in same-index queries. Same-file exclusion uses canonical roots and file identity to handle aliases. Cluster formatting builds connected components from emitted matches and sorts by member count, then name; transitive connectivity does not imply all-to-all similarity.
 
@@ -100,7 +104,7 @@ separationWeight = 1 - exp(-physicalDistance / 2)
 cohesionGap = semanticWeight * separationWeight
 ```
 
-Affinity ratios and mean distance are weighted by `semanticWeight`. Summary metrics use all qualifying edges before output limiting. File reports aggregate internal, same-folder, and external affinity for selected-source files. Pairs rank by gap and tie-breakers; groups are connected components of reported pairs. Source/test classification is a path heuristic, not a dependency or call-graph analysis.
+Affinity ratios and mean distance are weighted by `semanticWeight`. Cohesion metrics use all qualifying edges before output limiting. File reports aggregate internal, same-folder, and external affinity for selected-source files. Pairs rank by gap and tie-breakers; groups are connected components of reported pairs. Source/test classification is a path heuristic, not a dependency or call-graph analysis.
 
 ## Library API
 
@@ -124,12 +128,12 @@ try {
 }
 ```
 
-Exports include `CodeIndex`, `crossSearch`, `analyzeCohesion`, `cohesionLocation`, `JinaEmbeddingProvider`, `OpenAISummaryProvider`, error types, and the contracts in `src/types.ts`. Standalone update/search helpers wrap the corresponding index methods.
+Exports include `CodeIndex`, `crossSearch`, `analyzeCohesion`, `cohesionLocation`, `JinaEmbeddingProvider`, `OpenAIDescriptionProvider`, error types, and the contracts in `src/types.ts`. Standalone update/search helpers wrap the corresponding index methods.
 
 - Use `updateFromWorkingTree()` when Git is unavailable. Unlike the CLI, the library does not automatically refresh before queries or fall back from Git.
 - Set `sourceFilter.nameRegex` for source-only cross-search/cohesion filtering; combine it with `path` and a filter type (`all`, `changed-since`, or `uncommitted`). `changed-since` also accepts `uncommitted: true`.
 - Top-level analysis `nameRegex` filters both sources and candidates. Query `SimilaritySearchOptions.nameRegex` filters result names before limiting.
-- Call `await index.useSummaries()`, then `await index.searchSummary({ query: "maintain the repository index" })`. Select a model via `summaryProvider: new OpenAISummaryProvider({ model: "gpt-5.6-sol" })` in index options. Custom providers implement `EmbeddingProvider` or `SummaryProvider`.
+- Call `await index.useDescriptions()`, then `await index.searchDescription({ query: "maintain the repository index" })`. Select a model via `descriptionProvider: new OpenAIDescriptionProvider({ model: "gpt-5.6-sol" })` in index options. Custom providers implement `EmbeddingProvider` or `DescriptionProvider`.
 - Inspect failures through `index.indexErrors()` or exported `readIndexErrors(indexPath)` without a provider. Records use `IndexingError`.
 - Public cohesion reports retain indexed-function data; the CLI presents compact function references and includes source bodies only with `--include-source`.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ninjaxtools/slopdex",
-  "version": "0.9.0",
+  "version": "0.11.0",
   "description": "Tree-sitter callable-level semantic indexing, search, and duplicate discovery for Python, JavaScript, TypeScript, Rust, Go, Java, and C",
   "license": "MIT",
   "repository": {