@ninjaxtools/slopdex 0.6.0 → 0.8.0

package/.agents/skills/slopdex/SKILL.md

@@ -1,11 +1,11 @@
 ---
 name: slopdex
-description: Use when indexing TypeScript or JavaScript code, running semantic function search, finding duplicate function candidates, or analyzing physical code cohesion with the slopdex command-line tool.
+description: Use when indexing Python, JavaScript, JSX, TypeScript, TSX, Rust, Go, Java, or C code, running semantic function search, finding duplicate function candidates, or analyzing physical code cohesion with the slopdex command-line tool.
 ---
 
 # Slopdex CLI
 
-Use `slopdex` to index named JavaScript and TypeScript callables, search them by meaning, identify similar or duplicated functions, and find related functions scattered across a repository.
+Use `slopdex` to index named Python, JavaScript, JSX, TypeScript, TSX, Rust, Go, Java, and C callables with Tree-sitter, search them by meaning, identify similar or duplicated functions, and find related functions scattered across a repository. Languages are detected by extension and can coexist in one index.
 
 ## Default Workflow
 
@@ -39,6 +39,10 @@ Do not expose API keys in commands, output, configuration files, or commits.
 
 ## Indexing
 
+Root and nested `.gitignore` files restrict indexing, including tracked files. Working-tree updates use current ignore files; historical or committed-only Git updates use ignore files from the selected commit. A normal refresh removes newly ignored files from the index. Explicit updates reject ignored paths.
+
+Indexing records parse, callable-extraction, file-read, and file-size failures in SQLite while retaining healthy callables. Commands warn on stderr when unresolved diagnostics remain. Inspect them with `slopdex index-errors --format summary` (or JSON by default); this reads saved errors without refreshing the index or calling providers. Records include paths, source locations, recoverable function names, messages, and available source text. `status` reports `indexingErrorCount` and `failedFileCount`. Failed files are retried during updates; fixes, deletions, and exclusions clear their diagnostics. `--ignore-errors` silences diagnostic warnings without discarding the records.
+
 If a CLI command cannot find its source index, Slopdex prints a notice to stderr and automatically creates and populates it from committed `HEAD`, then overlays working-tree changes. Missing cross-search target indexes are initialized from the target repository's `HEAD` and working tree as well.
 
 Index the current committed snapshot and working-tree overlay:
@@ -211,7 +215,15 @@ Cross-search excludes one-line callables by default. Raise `--min-lines` for mor
 slopdex cross-search --min-lines 4 --threshold 0.9
 ```
 
-Filter both source and matching candidates by qualified callable name with a JavaScript regular expression:
+Filter source symbols by qualified callable name while searching the whole eligible index:
+
+```bash
+slopdex cross-search -e '^(User|Session)\.' --source-path src --threshold 0.9
+```
+
+`-e` / `--regexp` uses a case-sensitive JavaScript regex. In cross-search and cohesion, it restricts sources only. Combine it with `--source-path`, `--changed-since`, `--uncommitted`, and other analysis options. When both Git filters are present, sources must have changed since the commit and belong to an uncommitted file. In `search` and `search-summary`, `-e` filters result names before applying `--limit`.
+
+To filter both source and matching candidates, use `--regex`:
 
 ```bash
 slopdex cross-search --regex '^(User|Session)\.' --threshold 0.9
@@ -253,7 +265,7 @@ Restrict source functions to a file or recursive directory while still matching
 slopdex cross-search --source-path src/services --format summary --threshold 0.9
 ```
 
-`--source-path` restricts only source functions. It can be combined with `--changed-since` or `--uncommitted`.
+`--source-path` restricts only source functions. It can be combined with `-e`, `--changed-since`, and `--uncommitted`; all supplied restrictions must match.
 
 Restrict source functions to additions, modifications, and moves relative to a commit, including current working-tree changes:
 
@@ -320,7 +332,8 @@ Prefer JSON or JSONL when another command will consume the results. Prefer summa
 --include-source                    Include callable source in cohesion JSON
 --cross-file-only                   Exclude matches from the source file
 --min-lines <number>               Minimum cross-search callable length
---regex <regex>                    Match qualified callable names
+-e, --regexp <regex>               Match qualified symbols (analysis: sources only)
+--regex <regex>                    Match both analysis source and candidate names
 --target-config <path>             Target repository configuration file
 ```
 

package/README.md

@@ -1,13 +1,6 @@
 # slopdex
 
-Slopdex indexes named functions in TypeScript and JavaScript repositories. It uses embeddings to search code by meaning, find similar implementations, and measure whether related functions are stored near each other.
-
-Supported file types are `.ts`, `.tsx`, `.mts`, `.cts`, `.js`, `.jsx`, `.mjs`, and `.cjs`.
-
-## Requirements
-
-- Node.js 24 or later
-- An OpenAI or Jina AI API key
+Slopdex uses Tree-sitter to index named functions in Python, JavaScript, JSX, TypeScript, TSX, Rust, Go, Java, and C repositories. It uses embeddings to search code by meaning, find similar implementations, and measure whether related functions are stored near each other.
 
 ## Install
 
@@ -31,6 +24,45 @@ slopdex search "validate an authenticated session" --format summary --limit 10
 
 ## Analysis Examples
 
+### Search By Function Purpose
+
+Enable optional purpose summaries for every indexed callable:
+
+```bash
+slopdex use-summaries
+slopdex search-summary "keep the repository index synchronized" --limit 10 --format summary
+```
+
+Summaries describe a function's responsibility and role in its codebase, using its repository name, path, source, and surrounding file context. Generation uses the OpenAI Responses API with **`gpt-5.6-sol`** by default and requires `OPENAI_API_KEY`, including when Jina supplies the embeddings.
+
+`use-summaries` stores each summary and its embedding in SQLite and enables a persistent repository-index setting. Subsequent indexing operations automatically generate summaries for new and changed files, including changes to surrounding context and file paths. Unchanged inputs reuse cached summaries and vectors; deleted functions disappear from summary search. Running `use-summaries` again is a no-op when summaries are already complete. Summary generation and embeddings are saved atomically, so a failed request does not leave partially updated callables.
+
+`search-summary` uses a separate summary embedding store with the configured embedding provider and supports the same query, limit, threshold, and output options as `search`. JSON results include the summary; `--format summary` prints it alongside each match. `status` reports `summariesEnabled`, `summaryCount`, and `summaryProfile`.
+
+To choose a different summary model, run `slopdex use-summaries --summary-model <model-id>` or set `summaryModel` in `.slopdex/config.json`. The chosen model is persisted for future updates. Existing indexes migrate automatically; summaries remain disabled until enabled explicitly.
+
+### Combined Code And Purpose Analysis
+
+Once summaries are enabled and every indexed callable has a summary embedding, `cross-search` and `cohesion` automatically combine implementation and purpose similarity:
+
+```text
+similarity = 0.5 * codeSimilarity + 0.5 * summarySimilarity
+```
+
+```bash
+slopdex use-summaries
+slopdex cross-search --threshold 0.8 --format json
+slopdex cohesion --threshold 0.8 --format summary
+```
+
+Both cosine scores and their average are calculated in one SQLite query. Thresholds (including half-open ranges), ranking, and neighbor/result limits apply to the combined score. Cohesion uses these combined neighbors for reciprocity, gap scores, file affinities, and groups.
+
+For cross-repository search, both indexes must have complete, enabled summaries. If either index lacks them, the entire analysis uses code-only similarity. Cohesion likewise uses code-only scoring if its summary index is incomplete or disabled. The existing embedding profiles must match across repositories; summary-generator models may differ.
+
+Combined JSON matches and cohesion pairs expose `codeSimilarity` and `summarySimilarity` alongside `similarity`. Cross-search rows include a `scoring` object with `similarityMode`, `similarityWeights`, and both summary-generator profiles. Cohesion records the mode and weights in `parameters` and the summary-generator profile in `repository.summaryProfile`. The mode is `"code-summary-average"` with weights `{ "code": 0.5, "summary": 0.5 }`, or `"code"` with weights `{ "code": 1, "summary": 0 }`. Text output labels combined scores.
+
+Compare analysis results only when the similarity mode and weights match, as well as the embedding profile, summary-generator profiles, thresholds, and analysis scope. Enabling summaries can change rankings and cohesion metrics. `search` continues to use only code vectors, and `search-summary` uses only summary vectors.
+
 ### Duplicate Analysis
 
 Compare functions in different files and include functions that span at least four lines:
@@ -77,7 +109,7 @@ Cohesion: 184 functions analyzed, 37 semantic edges
 
 The summary divides semantic relationships into the same file, the same folder, and different folders. Mean distance increases when related functions are stored farther apart. The gap score combines semantic similarity with path distance and ranks pairs for review. `reciprocal` means both functions are among each other's nearest semantic matches.
 
-Cohesion does not have a universal pass threshold. Compare results only when the embedding profile, threshold, neighbor count, and source scope are the same.
+Cohesion does not have a universal pass threshold. Compare results only when the similarity mode and weights, embedding and summary-generator profiles, threshold, neighbor count, and source scope are the same.
 
 ## Limit Analysis Scope
 
@@ -104,10 +136,39 @@ slopdex cohesion --uncommitted --format summary
 
 These options restrict the source functions. Slopdex still compares them with the full index.
 
+Filter source symbols by qualified name with `-e <regex>` (or `--regexp <regex>`):
+
+```bash
+slopdex cross-search -e '^UserService\.' --format summary
+slopdex cohesion -e 'validate|authenticate' --source-path src/auth
+```
+
+The regex uses case-sensitive JavaScript syntax and matches qualified names such as `UserService.authenticate`. It filters only the source symbols for cross-search and cohesion; matching candidates are still drawn from the whole eligible index, including symbols that do not match the regex. It works with code-only and combined code/summary scoring.
+
+Source restrictions combine by intersection:
+
+```bash
+slopdex cross-search -e 'validate' \
+  --source-path src \
+  --changed-since origin/main \
+  --uncommitted \
+  --min-lines 4 \
+  --cross-file-only
+```
+
+With both Git filters, a source must have changed since the specified commit and belong to an uncommitted file. Path, name, and minimum-line filters narrow the selection further. Existing target restrictions such as `--min-lines` and `--cross-file-only` continue to apply. The separate `--regex` option filters **both** analysis sources and matching candidates.
+
 ## Output And Filters
 
 `search` supports `json` and `summary` output. `cross-search` supports `json`, `summary`, and `clusters` output. Use `--format` to select one.
 
+For `search` and `search-summary`, `-e` restricts result symbols before the result limit is applied:
+
+```bash
+slopdex search "validate session" -e '^Session\.' --limit 10
+slopdex search-summary "persist user data" -e 'save|persist' --limit 5
+```
+
 Use `--threshold` with a minimum similarity or a range:
 
 ```bash
@@ -121,6 +182,34 @@ Run `slopdex --help` for all commands and options.
 
 ## Indexing And Data
 
+### Supported Languages
+
+Languages are detected automatically from file extensions. A single index can contain multiple languages; search, purpose summaries, cross-search, and cohesion work across all of them.
+
+| Language | Extensions | Extracted callables |
+| --- | --- | --- |
+| Python | `.py`, `.pyw` | Functions, async functions, class methods, constructors, generators, and bound lambdas; includes decorators in source |
+| JavaScript | `.js`, `.mjs`, `.cjs` | Functions, generators, methods, constructors, and named function expressions/arrows |
+| JSX | `.jsx` | JavaScript callables, including components returning JSX |
+| TypeScript | `.ts`, `.mts`, `.cts` | Typed functions, methods, constructors, and named function expressions/arrows |
+| TSX | `.tsx` | TypeScript callables, including generic components returning JSX |
+| Rust | `.rs` | Functions, `impl` methods/associated functions, trait default methods, and `let`-bound closures |
+| Go | `.go` | Functions, receiver methods, and function literals bound to variables or assignments |
+| Java | `.java` | Methods, constructors (including compact record constructors), and variable-bound lambdas |
+| C | `.c`, `.h` | Function definitions, including static/inline functions and functions returning pointers |
+
+Qualified names include enclosing classes, functions, and explicit modules. Go methods include their receiver type (`Store[T].Get`); Rust trait implementations include the type and trait (`<Store<T> as Read>.read`). Definitions retain their source, signature, line/column locations, and stable identity for incremental updates. Declarations without bodies and anonymous callbacks are omitted. Extraction is syntactic: Rust macros and C macros are not expanded, and C preprocessor branches are indexed as written. `.h` files use the C grammar.
+
+Tree-sitter grammars ship as package dependencies; no language server or project compiler configuration is required. Syntax errors produce warnings while recoverable callables are still indexed. Existing indexes pick up newly supported files during the next normal refresh.
+
+Default exclusions cover dependency and build directories: `.git`, `.slopdex`, `node_modules`, `dist`, `build`, `coverage`, `vendor`, `generated`, `.venv`, `venv`, `__pycache__`, `.tox`, `.mypy_cache`, `.pytest_cache`, and `target`. Optional `include` and `exclude` glob arrays in `.slopdex/config.json` further restrict the indexed files.
+
+Slopdex also respects root and nested `.gitignore` files using the [`ignore`](https://www.npmjs.com/package/ignore) package. Patterns, directory rules, anchoring, escapes, and `!` exceptions follow Git ignore semantics. A nested rule cannot re-include files beneath an excluded parent directory. These exclusions apply even to tracked files and when Git is unavailable; explicit `update-files` requests for ignored files are rejected.
+
+Working-tree indexing and HEAD overlays use the current `.gitignore` files. Historical snapshots and committed-only Git indexing use the ignore files stored in the selected commit. Normal refreshes remove previously indexed files that become ignored and discover files that become eligible again, including when only ignore rules change. Configuration includes and negated ignore rules cannot override the built-in exclusions.
+
+### Index Updates
+
 Before each analysis command, Slopdex updates its index from the current Git commit and the staged, unstaged, and untracked files in the working tree. The index is created at `.slopdex/index.sqlite` by default.
 
 Add `.slopdex/` to the repository's `.gitignore` so the local index is not committed.
@@ -129,6 +218,27 @@ Slopdex sends extracted function source to the configured embedding provider. Th
 
 Whole-repository cohesion analysis compares neighbors for every selected function. Use `--source-path`, `--changed-since`, or `--uncommitted` to reduce the scope in large repositories.
 
+### Inspecting Indexing Errors
+
+Slopdex persists file and function indexing diagnostics in the SQLite `indexing_errors` table. Parse errors, parser exceptions, callable-extraction failures, file-read failures, and files exceeding `maxFileSize` retain references instead of silently disappearing. Healthy functions in partially parsed files and other healthy files remain searchable; malformed callables are omitted from semantic search and listed in the diagnostics.
+
+```bash
+slopdex index-errors
+slopdex index-errors --format summary
+slopdex index-errors --index /path/to/another/index.sqlite
+```
+
+This command reads the saved diagnostics without refreshing the index or calling an embedding/summary provider, so it works without API credentials. JSON includes the path, language, error code and message, file/function scope, recovered qualified name when available, line/column ranges, available source text, and Git/working-tree provenance. If Tree-sitter cannot identify a function, the unparsed region is still retained as a file diagnostic.
+
+Every invocation warns on stderr while saved errors remain, including cached runs and cross-search target indexes. Suppress this warning with `--ignore-errors`:
+
+```bash
+slopdex cross-search --ignore-errors
+slopdex index-errors --format summary --ignore-errors
+```
+
+Silencing warnings does not delete diagnostics. `status` reports `indexingErrorCount` and `failedFileCount`; `functionCount` counts searchable callables. Normal updates retry failed files, including unchanged Git blobs. Errors clear when a file is successfully indexed, deleted, or excluded. Existing indexes migrate automatically and rescan once on their next full update to detect previously unreported parse failures. Diagnostics are committed atomically with the corresponding file update.
+
 ## Library
 
 The package also exports the index and analysis APIs:
@@ -156,12 +266,32 @@ index.close();
 
 Exports also include `crossSearch`, `analyzeCohesion`, `JinaEmbeddingProvider`, and standalone functions for index updates and searches.
 
+Library callers can set `sourceFilter.nameRegex` on cross-search/cohesion options for source-only filtering, together with `path` and the Git filter type. A `changed-since` filter also accepts `uncommitted: true`. Query searches accept `nameRegex` in `SimilaritySearchOptions` to filter result symbols. Cohesion reports record source restrictions in `parameters.sourceFilter` and label filtered analyses as `selected-sources`.
+
+Use `index.indexErrors()` to inspect diagnostics through an open index, or the exported `readIndexErrors(indexPath)` to inspect a saved database without a provider. Error records use the exported `IndexingError` type.
+
+For summary search, call `await index.useSummaries()` after updating the index, then `await index.searchSummary({ query: "maintain the repository index" })`. Optionally pass `summaryProvider: new OpenAISummaryProvider({ model: "gpt-5.6-sol" })` when opening an index. Custom providers implement the exported `SummaryProvider` interface. The package also exports standalone `useSummaries` and `searchSummary` helpers.
+
 ## Development
 
 ```bash
 npm run check
 ```
 
+To cross-check the shared-language fixtures against a built sibling checkout of `treesitter-index`, run:
+
+```bash
+npm run check:parser-parity
+# Or provide another reference executable:
+npm run check:parser-parity -- /path/to/treesitter-index
+```
+
+The default reference is `../treesitter-index/target/debug/treesitter-index`. The check covers eight shared languages; that project has no C grammar. Callable regression tests also run as part of `npm run check` without requiring the sibling checkout.
+
+The projects index different information: `treesitter-index` includes declarations, types, imports, and `.pyi` stubs, while Slopdex indexes callable implementations (including nested callables and bound closures). Slopdex also supports `.pyw` and C. Its native Node grammar versions are pinned for compatibility with `tree-sitter@0.21`; the reference uses newer Python and Rust grammars. In particular, Rust `unsafe extern` blocks and async closures currently produce syntax-recovery warnings in Slopdex, so those cases are not covered by the passing parity fixtures.
+
+After upgrading parser behavior, explicitly reparse previously indexed files with `slopdex update-files <path...>` to refresh their symbols, signatures, and embeddings even when the file contents are unchanged.
+
 ## Configuration
 
 Configuration is optional. Without `.slopdex/config.json`, Slopdex uses OpenAI's `text-embedding-3-large` model with 3072 dimensions and the built-in source exclusions.