@ninjaxtools/slopdex 0.19.0 → 0.21.0

package/.gitignore

@@ -0,0 +1,2 @@
+/node_modules
+

package/README.md

@@ -4,40 +4,79 @@ This readme is written by a human.
 
 # slopdex
 
-<img src="sloppy-dexter.png" width="280" align="right" alt="Dexter, the sloppy slime" />
+<picture>
+  <source media="(max-width: 600px)" srcset="sloppy-dexter.png" width="600" />
+  <img src="sloppy-dexter.png" width="280" align="right" alt="Dexter, the sloppy slime" />
+</picture>
 
-Slopdex helps with doing analysis on codebases that contain a lot of AI generated code.
+Slopdex helps with doing analysis on codebases that contain a lot of AI generated ~~slop~~ slime.
 
 The main supported functions are
 
 - `slopdex search <query>`
-   <br/>which finds code similar to the query
+   <br/>find code/docs similar to the query
 - `slopdex cross-search`
-   <br/>which finds clusters of similar code
+   <br/>find clusters of similar code/docs
+- `slopdex describe <query>`
+   <br/>explain code relevant to a task
+- `slopdex map [PATH]...`
+   <br/>show a map/skeleton of the code structure
 
-`cross-search` helps with finding duplicated code, and with `--cohesion` helps with identifying similar code that is not necessarily duplicated but spread out across the codebase, which could indicate that a refactoring could make it more cohesive.
+`search` does a vector searche of code, and markdown, and when enabled generated code descriptions.
 
-## Getting started
+`cross-search` also does a vector search but compares all functions with each other (scope can be limited with additional options) and helps with finding duplicated code, or code that is not necessarily duplicated but spread out across the codebase (with `--cohesion`).
+
+`describe` does a `search` first, then passes the result through the configured LLM model to create a tailored description.
+
+`map` shows a map/skeleton of the code, excluding implementation details like function bodies. This can be helpful to get a concise map of the code to allow an LLM to explore the codebase incrementally.
 
-An embedding-provider API key is required. You can use either OpenAI or Jina.
+## Installation
 
-```bash
+Install from npm:
+
+```sh
 npm install -g @ninjaxtools/slopdex
+```
 
-export OPENAI_API_KEY="your-api-key"
-# Or
-# export JINA_API_KEY="your-api-key" # and pass --provider jina
+You can also download a binary from one of the releases: https://github.com/ninjaxtools/slopdex/releases
 
-slopdex search "validate an authenticated session"
+Or you can build from source:
+
+```sh
+cargo install --path . --locked
+```
+
+## Getting started
+
+Semantic search requires an embedding-provider API key from OpenAI or Jina. `slopdex map` doesn't
+use vector search and works locally without a provider or API key.
+
+```console
+$ export OPENAI_API_KEY="your-api-key"
+$ slopdex search "validate an authenticated session"
+...
 ```
 
 The index tracks the current git commit and is created or updated on every command and stored in `.slopdex/index.sqlite`. Add `.slopdex/` to your repository's `.gitignore` to prevent it from being committed.
 
-### Search code
+### Code map/skeleton
+
+```console
+$ slopdex map src docs
+$ slopdex map src -g '*.rs' -k fns -e 'refresh|search'
+$ slopdex map -g '*.md' -e '^Guide\.Setup' -i --format json
+$ slopdex map --no-reindex
+```
+
+The `map` command can be used to generate a source code skeleton that strips most of the code implementation but retains many of the structurally useful information that can act as an index into the source code. The resulting index output is most useful to conserve tokens and improve the agentic programming experience.
+
+See the [selector reference](docs/reference.md#shared-selectors).
+
+### Search
 
-By default only vector embeddings of code is used for search.
+By default all available indexes are searched and ranked together.
 
-```bash
+```console
 $ slopdex search "keep the repository index synchronized"
 0.4284  tests/languages.test.ts :: refresh
 0.4200  src/cli.ts :: refreshIndex
@@ -45,35 +84,65 @@ $ slopdex search "keep the repository index synchronized"
 ...
 ```
 
+To search code/docs individually:
+
+```console
+$ slopdex search-code "configure the embedding provider"
+$ slopdex search-md "configure the embedding provider" 
+```
+
 You can also enable optional description generation which will automatically generate file and function descriptions with a configured LLM provider and include them in the search.
 
 I use OpenCode Go usually with DeepSeek or Muse Spark, which are fairly good low-cost models. If you
 sign up for OpenCode Go through [this link](https://opencode.ai/go?ref=RAR3Z744DZ), we both receive
 $5 in credit.
 
-```bash
-export OPENAI_API_KEY="your-api-key"
+```console
+$ export OPENAI_API_KEY="your-api-key"
 # Or
 # export OPENCODE_API_KEY="your-api-key" # for OpenCode Zen/Go descriptions
 # Or
 # opencode auth login                    # use stored credentials instead of env variables
 
-slopdex descriptions enable
-slopdex search-description "keep the repository index synchronized"
+$ slopdex descriptions enable
+$ slopdex search-descriptions "keep the repository index synchronized"
+```
+
+You can list and configure one of OpenCode's models like this (or use the interactive config):
+
+```console
+$ slopdex models opencode-go
+$ slopdex config model opencode-go/gpt-5.6-luna
+$ slopdex config fallback-model opencode-go/muse-spark-1.3-contributor
+```
+
+The fallback-model is used if the main model reports an error, and if the fallback-model reports an error the main model is tried again.
+
+File and function descriptions are cached. Function descriptions will only be regenerated when functions change. File descriptions need to be regenerated explcitily with the `reindex-files` command.
+
+### Interactive Config
+
+To configure all configurable settings interactively run:
+
+```console
+$ slopdex config
 ```
 
-You can list and configure one of OpenCode's models like this:
+### Search and describe
+
+The intention of the `describe` command is to use a low-cost model to summarise vector search findings to provide more relevant pre-processed results to a more powerful calling agent. It first performs a vector search and then uses the results to provide a tailored summary/description of relevant code and pre-generated generated file and function descriptions. 
 
-```bash
-slopdex models opencode-go
-slopdex config model opencode-go/gpt-5.6-luna
+```console
+$ slopdex describe "I want to implement a new rpc endpoint"
 ```
 
+For the best matching files (configured with `--describe-full-file-threshold`, default `0.8`) the full file contents are used.
+
 ### Find duplicate code
 
 Compare functions across files, exclude short wrappers, and group matches into clusters:
 
-```bash
+```console
 $ slopdex cross-search --cross-file-only --min-lines 4 --threshold 0.9
 Cluster 1 (3 functions, similarity 0.9124-0.9568)
   src/auth/session.ts:18:1 :: validateSession
@@ -84,66 +153,73 @@ Cluster 1 (3 functions, similarity 0.9124-0.9568)
 
 Using `--cross-file-only` is useful to exclude similar code in the same file.
 
-Review adjacent bands with threshold ranges:
+Cross-search defaults to `--threshold 0.8` (query search defaults to `0.3`).
+Clusters join matches transitively, so a low threshold can connect many groups
+into one large cluster. Use `0.9` for stricter duplicate detection, or explicitly
+pass `--threshold 0.3` for the pre-rewrite default.
 
-```bash
-slopdex cross-search --cross-file-only --min-lines 4 --threshold 0.9
-slopdex cross-search --cross-file-only --min-lines 4 --threshold 0.85-0.9
+You can use threshold ranges as well:
+
+```console
+$ slopdex cross-search --cross-file-only --min-lines 4 --threshold 0.9
+$ slopdex cross-search --cross-file-only --min-lines 4 --threshold 0.85-0.9
 ```
 
 ### Restrict functions used in the cross-search
 
 Only use uncommitted working-tree functions as sources:
 
-```bash
-slopdex cross-search --uncommitted --cross-file-only --min-lines 4 --threshold 0.9
+```console
+$ slopdex cross-search --uncommitted --cross-file-only --min-lines 4 --threshold 0.9
 ```
 
 Only use functions changed since origin/main as sources:
 
-```bash
-slopdex cross-search --changed-since origin/main --threshold 0.9
+```console
+$ slopdex cross-search --changed-since origin/main --threshold 0.9
 ```
 
 Only use matching symbols under src/services as sources:
 
-```bash
-slopdex cross-search --source-path src/services -e '^UserService\.' --threshold 0.9
+```console
+$ slopdex cross-search --source-path src/services -e '^UserService\.' --threshold 0.9
 ```
 
 ### Find related code stored far apart
 
 When code is similar but not actually duplicated, then `--cohesion` can help find similar code that exists far apart in the filesystem tree, which could potentially be refactored to make it more cohesive, by ordering matches from farthest to nearest.
 
-```bash
-slopdex cross-search --cross-file-only --cohesion --threshold 0.8
+```console
+$ slopdex cross-search --cross-file-only --cohesion --threshold 0.8
 ```
 ### Use with agents
 
 Just put this in your `AGENTS.md` file, no skill required:
 
 ```
+- use `treesitter-index -g <glob> <files or directories...>` early to obtain a compact structural skeleton before reading the full file, and then perform targeted reads for implementation details. Line numbers are indicated in square brackets (e.g [5] means line 5, and [5-10] means lines 5 to 10). To filter for specific symbols, use `-e`. To only include specific kinds of symbols use `-k` with `imports`, `fns`, `consts`, `types`, or `classes`. For example: `treesitter-index -g "*.ts" -i -e "manager|main" src`.
 - use semantic code search to find code with: `slopdex search "..." --threshold 0.5`
 - when reviewing uncommitted code avoid introducing duplicates by looking for related matches: `slopdex cross-search --uncommitted --threshold 0.8`
+- don't use `*` as the glob since that circumvents the default ignore rules.
 ```
 
 Any other use, like doing a full `cross-search` is probably better done interactively with the agent, in which case you can just ask the agent to run `slopdex --help` to get usage information.
 
 ### Reranking
 
-Optionally a second-stage reranker can be enabled for `search` and `search-description`:
+Optionally a second-stage reranker can be enabled for all query-search commands:
 
-```bash
-export COHERE_API_KEY="your-api-key"
-slopdex config reranker cohere
+```console
+$ export COHERE_API_KEY="your-api-key"
+$ slopdex config reranker cohere
 # Or: export JINA_API_KEY="your-api-key" && slopdex config reranker jina
 # Or use an LLM: export OPENAI_API_KEY="your-api-key" && slopdex config reranker openai
 ```
 
 ### Compare repositories
 
-```bash
-slopdex cross-search \
+```console
+$ slopdex cross-search \
   --target-root /path/to/other/repo \
   --target-index /path/to/other/repo/.slopdex/index.sqlite \
   --threshold 0.9
@@ -153,10 +229,42 @@ slopdex cross-search \
 
 If some functions can't be indexed a warning is printed. Index errors can be investigated and fixed with the `index-errors` command to ensure the index is complete.
 
-```bash
-slopdex status
-slopdex index-errors --format summary
-slopdex --version
+```console
+$ slopdex status
+$ slopdex index-errors --format summary
+$ slopdex --version
+```
+
+## Related Work
+
+Code similarity search
+
+- [treepeat](https://github.com/dsummersl/treepeat)
+
+Structured code queries:
+
+- [CodeGraphContext](https://github.com/CodeGraphContext/CodeGraphContext)
+- [CoderLM](https://github.com/JaredStewart/coderlm)
+
+Structured code queries and edits:
+
+- [srgn](https://github.com/alexpovel/srgn)
+- [comby](https://github.com/comby-tools/comby)
+- [ast-grep](https://ast-grep.github.io/guide/introduction)
+- [gritql](https://github.com/biomejs/gritql)
+
+Semantic code search with vector embeddings:
+
+- [qmd](https://github.com/tobi/qmd)
+- [grepai](https://github.com/yoanbernabeu/grepai)
+- [cocoindex-code](https://github.com/cocoindex-io/cocoindex-code)
+- [codana](https://github.com/bartolli/codanna)
+- [open-codebase-index](https://github.com/Helweg/open-codebase-index)
+
+## Publishing
+
+```console
+$ cargo release minor --no-publish --execute
 ```
 
 ## Commands

package/binary-install.js

@@ -0,0 +1,348 @@
+const {
+  createWriteStream,
+  existsSync,
+  mkdirSync,
+  mkdtemp,
+  rmSync,
+} = require("fs");
+const { join, sep } = require("path");
+const { spawnSync } = require("child_process");
+const { tmpdir } = require("os");
+
+const https = require("node:https");
+const http = require("node:http");
+
+const tmpDir = tmpdir();
+
+const error = (msg) => {
+  console.error(msg);
+  process.exit(1);
+};
+
+function getProxyForUrl(urlString) {
+  const url = new URL(urlString);
+  const isHttps = url.protocol === "https:";
+
+  const noProxy = process.env.NO_PROXY || process.env.no_proxy || "";
+  if (noProxy === "*") return null;
+  if (noProxy) {
+    const hostname = url.hostname.toLowerCase();
+    const noProxyList = noProxy.split(",").map((s) => s.trim().toLowerCase());
+    for (const entry of noProxyList) {
+      if (hostname === entry || hostname.endsWith("." + entry)) {
+        return null;
+      }
+    }
+  }
+
+  const proxyEnv = isHttps
+    ? process.env.HTTPS_PROXY || process.env.https_proxy
+    : process.env.HTTP_PROXY || process.env.http_proxy;
+
+  if (!proxyEnv) return null;
+
+  const proxyUrl = new URL(proxyEnv);
+
+  let auth = null;
+  if (proxyUrl.username || proxyUrl.password) {
+    auth = `${proxyUrl.username}:${proxyUrl.password}`;
+  }
+
+  return {
+    hostname: proxyUrl.hostname,
+    port: proxyUrl.port || (proxyUrl.protocol === "https:" ? 443 : 80),
+    auth: auth,
+  };
+}
+
+function connectThroughProxy(proxy, target) {
+  return new Promise((resolve, reject) => {
+    const headers = {};
+    if (proxy.auth) {
+      headers["Proxy-Authorization"] =
+        "Basic " + Buffer.from(proxy.auth).toString("base64");
+    }
+
+    const connectReq = http.request({
+      hostname: proxy.hostname,
+      port: proxy.port,
+      method: "CONNECT",
+      path: `${target.hostname}:${target.port || 443}`,
+      headers,
+    });
+    connectReq.on("connect", (res, socket) => {
+      if (res.statusCode === 200) {
+        resolve(socket);
+      } else {
+        reject(new Error(`Proxy CONNECT failed with status ${res.statusCode}`));
+      }
+    });
+    connectReq.on("error", reject);
+    connectReq.end();
+  });
+}
+
+function download(urlString, maxRedirects) {
+  if (maxRedirects === undefined) maxRedirects = 5;
+  return new Promise((resolve, reject) => {
+    if (maxRedirects < 0) {
+      return reject(new Error("Too many redirects"));
+    }
+
+    const parsed = new URL(urlString);
+    const isHttps = parsed.protocol === "https:";
+    const mod = isHttps ? https : http;
+    const proxy = getProxyForUrl(urlString);
+
+    const doRequest = (extraOptions) => {
+      const options = Object.assign(
+        {
+          hostname: parsed.hostname,
+          port: parsed.port || (isHttps ? 443 : 80),
+          path: parsed.pathname + parsed.search,
+          method: "GET",
+          headers: { "User-Agent": "cargo-dist-npm-installer" },
+        },
+        extraOptions || {},
+      );
+
+      if (proxy && !isHttps) {
+        // HTTP through HTTP proxy: request the full URL via the proxy
+        options.hostname = proxy.hostname;
+        options.port = proxy.port;
+        options.path = urlString;
+        if (proxy.auth) {
+          options.headers["Proxy-Authorization"] =
+            "Basic " + Buffer.from(proxy.auth).toString("base64");
+        }
+      }
+
+      const req = mod.request(options, (res) => {
+        if (
+          res.statusCode >= 300 &&
+          res.statusCode < 400 &&
+          res.headers.location
+        ) {
+          res.resume();
+          const nextUrl = new URL(res.headers.location, urlString).toString();
+          return download(nextUrl, maxRedirects - 1).then(resolve, reject);
+        }
+        if (res.statusCode < 200 || res.statusCode >= 300) {
+          res.resume();
+          return reject(new Error(`HTTP ${res.statusCode} from ${urlString}`));
+        }
+        resolve(res);
+      });
+      req.on("error", reject);
+      req.end();
+    };
+
+    if (proxy && isHttps) {
+      connectThroughProxy(proxy, parsed).then(
+        (socket) => doRequest({ socket, agent: false }),
+        reject,
+      );
+    } else {
+      doRequest();
+    }
+  });
+}
+
+class Package {
+  constructor(platform, name, url, filename, zipExt, binaries) {
+    let errors = [];
+    if (typeof url !== "string") {
+      errors.push("url must be a string");
+    } else {
+      try {
+        new URL(url);
+      } catch (e) {
+        errors.push(e);
+      }
+    }
+    if (name && typeof name !== "string") {
+      errors.push("package name must be a string");
+    }
+    if (!name) {
+      errors.push("You must specify the name of your package");
+    }
+    if (binaries && typeof binaries !== "object") {
+      errors.push("binaries must be a string => string map");
+    }
+    if (!binaries) {
+      errors.push("You must specify the binaries in the package");
+    }
+
+    if (errors.length > 0) {
+      let errorMsg =
+        "One or more of the parameters you passed to the Binary constructor are invalid:\n";
+      errors.forEach((error) => {
+        errorMsg += error;
+      });
+      errorMsg +=
+        '\n\nCorrect usage: new Package("my-binary", "https://example.com/binary/download.tar.gz", {"my-binary": "my-binary"})';
+      error(errorMsg);
+    }
+
+    this.platform = platform;
+    this.url = url;
+    this.name = name;
+    this.filename = filename;
+    this.zipExt = zipExt;
+    this.installDirectory = join(__dirname, "node_modules", ".bin_real");
+    this.binaries = binaries;
+
+    if (!existsSync(this.installDirectory)) {
+      mkdirSync(this.installDirectory, { recursive: true });
+    }
+  }
+
+  exists() {
+    for (const binaryName in this.binaries) {
+      const binRelPath = this.binaries[binaryName];
+      const binPath = join(this.installDirectory, binRelPath);
+      if (!existsSync(binPath)) {
+        return false;
+      }
+    }
+    return true;
+  }
+
+  install(suppressLogs = false) {
+    if (this.exists()) {
+      if (!suppressLogs) {
+        console.error(
+          `${this.name} is already installed, skipping installation.`,
+        );
+      }
+      return Promise.resolve();
+    }
+
+    try {
+      rmSync(this.installDirectory, { recursive: true, force: true });
+    } catch {
+      // ignore - directory may not exist
+    }
+
+    mkdirSync(this.installDirectory, { recursive: true });
+
+    if (!suppressLogs) {
+      console.error(`Downloading release from ${this.url}`);
+    }
+
+    return download(this.url)
+      .then((res) => {
+        return new Promise((resolve, reject) => {
+          mkdtemp(`${tmpDir}${sep}`, (err, directory) => {
+            if (err) return reject(err);
+            let tempFile = join(directory, this.filename);
+            const sink = res.pipe(createWriteStream(tempFile));
+            sink.on("error", (err) => reject(err));
+            sink.on("close", () => {
+              if (/\.tar\.*/.test(this.zipExt)) {
+                const result = spawnSync("tar", [
+                  "xf",
+                  tempFile,
+                  // The tarballs are stored with a leading directory
+                  // component; we strip one component in the
+                  // shell installers too.
+                  "--strip-components",
+                  "1",
+                  "-C",
+                  this.installDirectory,
+                ]);
+                if (result.status == 0) {
+                  resolve();
+                } else if (result.error) {
+                  reject(result.error);
+                } else {
+                  reject(
+                    new Error(
+                      `An error occurred untarring the artifact: stdout: ${result.stdout}; stderr: ${result.stderr}`,
+                    ),
+                  );
+                }
+              } else if (this.zipExt == ".zip") {
+                let result;
+                if (this.platform.artifactName.includes("windows")) {
+                  // Windows does not have "unzip" by default on many installations, instead
+                  // we use Expand-Archive from powershell
+                  result = spawnSync("powershell.exe", [
+                    "-NoProfile",
+                    "-NonInteractive",
+                    "-Command",
+                    `& {
+                        param([string]$LiteralPath, [string]$DestinationPath)
+                        Expand-Archive -LiteralPath $LiteralPath -DestinationPath $DestinationPath -Force
+                    }`,
+                    tempFile,
+                    this.installDirectory,
+                  ]);
+                } else {
+                  result = spawnSync("unzip", [
+                    "-q",
+                    tempFile,
+                    "-d",
+                    this.installDirectory,
+                  ]);
+                }
+
+                if (result.status == 0) {
+                  resolve();
+                } else if (result.error) {
+                  reject(result.error);
+                } else {
+                  reject(
+                    new Error(
+                      `An error occurred unzipping the artifact: stdout: ${result.stdout}; stderr: ${result.stderr}`,
+                    ),
+                  );
+                }
+              } else {
+                reject(
+                  new Error(`Unrecognized file extension: ${this.zipExt}`),
+                );
+              }
+            });
+          });
+        });
+      })
+      .then(() => {
+        if (!suppressLogs) {
+          console.error(`${this.name} has been installed!`);
+        }
+      })
+      .catch((e) => {
+        error(`Error fetching release: ${e.message}`);
+      });
+  }
+
+  run(binaryName) {
+    const promise = !this.exists() ? this.install(true) : Promise.resolve();
+
+    promise
+      .then(() => {
+        const [, , ...args] = process.argv;
+
+        const options = { cwd: process.cwd(), stdio: "inherit" };
+
+        const binRelPath = this.binaries[binaryName];
+        if (!binRelPath) {
+          error(`${binaryName} is not a known binary in ${this.name}`);
+        }
+        const binPath = join(this.installDirectory, binRelPath);
+        const result = spawnSync(binPath, args, options);
+
+        if (result.error) {
+          error(result.error);
+        }
+
+        process.exit(result.status);
+      })
+      .catch((e) => {
+        error(e.message);
+      });
+  }
+}
+
+module.exports.Package = Package;