@ninjaxtools/slopdex 0.19.0 → 0.20.0

package/.gitignore

@@ -0,0 +1,2 @@
+/node_modules
+

package/README.md

@@ -4,40 +4,61 @@ This readme is written by a human.
 
 # slopdex
 
-<img src="sloppy-dexter.png" width="280" align="right" alt="Dexter, the sloppy slime" />
+<picture>
+  <source media="(max-width: 600px)" srcset="sloppy-dexter.png" width="600" />
+  <img src="sloppy-dexter.png" width="280" align="right" alt="Dexter, the sloppy slime" />
+</picture>
 
-Slopdex helps with doing analysis on codebases that contain a lot of AI generated code.
+Slopdex helps with doing analysis on codebases that contain a lot of AI generated ~~slop~~ slime.
 
 The main supported functions are
 
 - `slopdex search <query>`
-   <br/>which finds code similar to the query
+   <br/>find code/docs similar to the query
 - `slopdex cross-search`
-   <br/>which finds clusters of similar code
+   <br/>find clusters of similar code/docs
+- `slopdex describe <query>`
+   <br/>explain code relevant to a task
 
-`cross-search` helps with finding duplicated code, and with `--cohesion` helps with identifying similar code that is not necessarily duplicated but spread out across the codebase, which could indicate that a refactoring could make it more cohesive.
+`search` does a vector searche of code, and markdown, and when enabled generated code descriptions.
 
-## Getting started
+`cross-search` also does a vector search but compares all functions with each other (scope can be limited with additional options) and helps with finding duplicated code, or code that is not necessarily duplicated but spread out across the codebase (with `--cohesion`).
 
-An embedding-provider API key is required. You can use either OpenAI or Jina.
+`describe` does a `search` first, then passes the result through the configured LLM model to create a tailored description.
+
+## Installation
 
-```bash
+Install from npm:
+
+```sh
 npm install -g @ninjaxtools/slopdex
+```
 
-export OPENAI_API_KEY="your-api-key"
-# Or
-# export JINA_API_KEY="your-api-key" # and pass --provider jina
+You can also download a binary from one of the releases: https://github.com/ninjaxtools/slopdex/releases
 
-slopdex search "validate an authenticated session"
+Or you can build from source:
+
+```sh
+cargo install --path . --locked
+```
+
+## Getting started
+
+An embedding-provider API key is required. You can use either OpenAI or Jina.
+
+```console
+$ export OPENAI_API_KEY="your-api-key"
+$ slopdex search "validate an authenticated session"
+...
 ```
 
 The index tracks the current git commit and is created or updated on every command and stored in `.slopdex/index.sqlite`. Add `.slopdex/` to your repository's `.gitignore` to prevent it from being committed.
 
-### Search code
+### Search
 
-By default only vector embeddings of code is used for search.
+By default all available indexes are searched and ranked together.
 
-```bash
+```console
 $ slopdex search "keep the repository index synchronized"
 0.4284  tests/languages.test.ts :: refresh
 0.4200  src/cli.ts :: refreshIndex
@@ -45,35 +66,65 @@ $ slopdex search "keep the repository index synchronized"
 ...
 ```
 
+To search code/docs individually:
+
+```console
+$ slopdex search-code "configure the embedding provider"
+$ slopdex search-md "configure the embedding provider" 
+```
+
 You can also enable optional description generation which will automatically generate file and function descriptions with a configured LLM provider and include them in the search.
 
 I use OpenCode Go usually with DeepSeek or Muse Spark, which are fairly good low-cost models. If you
 sign up for OpenCode Go through [this link](https://opencode.ai/go?ref=RAR3Z744DZ), we both receive
 $5 in credit.
 
-```bash
-export OPENAI_API_KEY="your-api-key"
+```console
+$ export OPENAI_API_KEY="your-api-key"
 # Or
 # export OPENCODE_API_KEY="your-api-key" # for OpenCode Zen/Go descriptions
 # Or
 # opencode auth login                    # use stored credentials instead of env variables
 
-slopdex descriptions enable
-slopdex search-description "keep the repository index synchronized"
+$ slopdex descriptions enable
+$ slopdex search-descriptions "keep the repository index synchronized"
+```
+
+You can list and configure one of OpenCode's models like this (or use the interactive config):
+
+```console
+$ slopdex models opencode-go
+$ slopdex config model opencode-go/gpt-5.6-luna
+$ slopdex config fallback-model opencode-go/muse-spark-1.3-contributor
+```
+
+The fallback-model is used if the main model reports an error, and if the fallback-model reports an error the main model is tried again.
+
+File and function descriptions are cached. Function descriptions will only be regenerated when functions change. File descriptions need to be regenerated explcitily with the `reindex-files` command.
+
+### Interactive Config
+
+To configure all configurable settings interactively run:
+
+```console
+$ slopdex config
 ```
 
-You can list and configure one of OpenCode's models like this:
+### Search and describe
+
+The intention of the `describe` command is to use a low-cost model to summarise vector search findings to provide more relevant pre-processed results to a more powerful calling agent. It first performs a vector search and then uses the results to provide a tailored summary/description of relevant code and pre-generated generated file and function descriptions. 
 
-```bash
-slopdex models opencode-go
-slopdex config model opencode-go/gpt-5.6-luna
+```console
+$ slopdex describe "I want to implement a new rpc endpoint"
 ```
 
+For the best matching files (configured with `--describe-full-file-threshold`, default `0.8`) the full file contents are used.
+
 ### Find duplicate code
 
 Compare functions across files, exclude short wrappers, and group matches into clusters:
 
-```bash
+```console
 $ slopdex cross-search --cross-file-only --min-lines 4 --threshold 0.9
 Cluster 1 (3 functions, similarity 0.9124-0.9568)
   src/auth/session.ts:18:1 :: validateSession
@@ -84,39 +135,44 @@ Cluster 1 (3 functions, similarity 0.9124-0.9568)
 
 Using `--cross-file-only` is useful to exclude similar code in the same file.
 
-Review adjacent bands with threshold ranges:
+Cross-search defaults to `--threshold 0.8` (query search defaults to `0.3`).
+Clusters join matches transitively, so a low threshold can connect many groups
+into one large cluster. Use `0.9` for stricter duplicate detection, or explicitly
+pass `--threshold 0.3` for the pre-rewrite default.
+
+You can use threshold ranges as well:
 
-```bash
-slopdex cross-search --cross-file-only --min-lines 4 --threshold 0.9
-slopdex cross-search --cross-file-only --min-lines 4 --threshold 0.85-0.9
+```console
+$ slopdex cross-search --cross-file-only --min-lines 4 --threshold 0.9
+$ slopdex cross-search --cross-file-only --min-lines 4 --threshold 0.85-0.9
 ```
 
 ### Restrict functions used in the cross-search
 
 Only use uncommitted working-tree functions as sources:
 
-```bash
-slopdex cross-search --uncommitted --cross-file-only --min-lines 4 --threshold 0.9
+```console
+$ slopdex cross-search --uncommitted --cross-file-only --min-lines 4 --threshold 0.9
 ```
 
 Only use functions changed since origin/main as sources:
 
-```bash
-slopdex cross-search --changed-since origin/main --threshold 0.9
+```console
+$ slopdex cross-search --changed-since origin/main --threshold 0.9
 ```
 
 Only use matching symbols under src/services as sources:
 
-```bash
-slopdex cross-search --source-path src/services -e '^UserService\.' --threshold 0.9
+```console
+$ slopdex cross-search --source-path src/services -e '^UserService\.' --threshold 0.9
 ```
 
 ### Find related code stored far apart
 
 When code is similar but not actually duplicated, then `--cohesion` can help find similar code that exists far apart in the filesystem tree, which could potentially be refactored to make it more cohesive, by ordering matches from farthest to nearest.
 
-```bash
-slopdex cross-search --cross-file-only --cohesion --threshold 0.8
+```console
+$ slopdex cross-search --cross-file-only --cohesion --threshold 0.8
 ```
 ### Use with agents
 
@@ -131,19 +187,19 @@ Any other use, like doing a full `cross-search` is probably better done interact
 
 ### Reranking
 
-Optionally a second-stage reranker can be enabled for `search` and `search-description`:
+Optionally a second-stage reranker can be enabled for all query-search commands:
 
-```bash
-export COHERE_API_KEY="your-api-key"
-slopdex config reranker cohere
+```console
+$ export COHERE_API_KEY="your-api-key"
+$ slopdex config reranker cohere
 # Or: export JINA_API_KEY="your-api-key" && slopdex config reranker jina
 # Or use an LLM: export OPENAI_API_KEY="your-api-key" && slopdex config reranker openai
 ```
 
 ### Compare repositories
 
-```bash
-slopdex cross-search \
+```console
+$ slopdex cross-search \
   --target-root /path/to/other/repo \
   --target-index /path/to/other/repo/.slopdex/index.sqlite \
   --threshold 0.9
@@ -153,10 +209,42 @@ slopdex cross-search \
 
 If some functions can't be indexed a warning is printed. Index errors can be investigated and fixed with the `index-errors` command to ensure the index is complete.
 
-```bash
-slopdex status
-slopdex index-errors --format summary
-slopdex --version
+```console
+$ slopdex status
+$ slopdex index-errors --format summary
+$ slopdex --version
+```
+
+## Related Work
+
+Code similarity search
+
+- [treepeat](https://github.com/dsummersl/treepeat)
+
+Structured code queries:
+
+- [CodeGraphContext](https://github.com/CodeGraphContext/CodeGraphContext)
+- [CoderLM](https://github.com/JaredStewart/coderlm)
+
+Structured code queries and edits:
+
+- [srgn](https://github.com/alexpovel/srgn)
+- [comby](https://github.com/comby-tools/comby)
+- [ast-grep](https://ast-grep.github.io/guide/introduction)
+- [gritql](https://github.com/biomejs/gritql)
+
+Semantic code search with vector embeddings:
+
+- [qmd](https://github.com/tobi/qmd)
+- [grepai](https://github.com/yoanbernabeu/grepai)
+- [cocoindex-code](https://github.com/cocoindex-io/cocoindex-code)
+- [codana](https://github.com/bartolli/codanna)
+- [open-codebase-index](https://github.com/Helweg/open-codebase-index)
+
+## Publishing
+
+```console
+$ cargo release minor --no-publish --execute
 ```
 
 ## Commands

package/binary-install.js

@@ -0,0 +1,348 @@
+const {
+  createWriteStream,
+  existsSync,
+  mkdirSync,
+  mkdtemp,
+  rmSync,
+} = require("fs");
+const { join, sep } = require("path");
+const { spawnSync } = require("child_process");
+const { tmpdir } = require("os");
+
+const https = require("node:https");
+const http = require("node:http");
+
+const tmpDir = tmpdir();
+
+const error = (msg) => {
+  console.error(msg);
+  process.exit(1);
+};
+
+function getProxyForUrl(urlString) {
+  const url = new URL(urlString);
+  const isHttps = url.protocol === "https:";
+
+  const noProxy = process.env.NO_PROXY || process.env.no_proxy || "";
+  if (noProxy === "*") return null;
+  if (noProxy) {
+    const hostname = url.hostname.toLowerCase();
+    const noProxyList = noProxy.split(",").map((s) => s.trim().toLowerCase());
+    for (const entry of noProxyList) {
+      if (hostname === entry || hostname.endsWith("." + entry)) {
+        return null;
+      }
+    }
+  }
+
+  const proxyEnv = isHttps
+    ? process.env.HTTPS_PROXY || process.env.https_proxy
+    : process.env.HTTP_PROXY || process.env.http_proxy;
+
+  if (!proxyEnv) return null;
+
+  const proxyUrl = new URL(proxyEnv);
+
+  let auth = null;
+  if (proxyUrl.username || proxyUrl.password) {
+    auth = `${proxyUrl.username}:${proxyUrl.password}`;
+  }
+
+  return {
+    hostname: proxyUrl.hostname,
+    port: proxyUrl.port || (proxyUrl.protocol === "https:" ? 443 : 80),
+    auth: auth,
+  };
+}
+
+function connectThroughProxy(proxy, target) {
+  return new Promise((resolve, reject) => {
+    const headers = {};
+    if (proxy.auth) {
+      headers["Proxy-Authorization"] =
+        "Basic " + Buffer.from(proxy.auth).toString("base64");
+    }
+
+    const connectReq = http.request({
+      hostname: proxy.hostname,
+      port: proxy.port,
+      method: "CONNECT",
+      path: `${target.hostname}:${target.port || 443}`,
+      headers,
+    });
+    connectReq.on("connect", (res, socket) => {
+      if (res.statusCode === 200) {
+        resolve(socket);
+      } else {
+        reject(new Error(`Proxy CONNECT failed with status ${res.statusCode}`));
+      }
+    });
+    connectReq.on("error", reject);
+    connectReq.end();
+  });
+}
+
+function download(urlString, maxRedirects) {
+  if (maxRedirects === undefined) maxRedirects = 5;
+  return new Promise((resolve, reject) => {
+    if (maxRedirects < 0) {
+      return reject(new Error("Too many redirects"));
+    }
+
+    const parsed = new URL(urlString);
+    const isHttps = parsed.protocol === "https:";
+    const mod = isHttps ? https : http;
+    const proxy = getProxyForUrl(urlString);
+
+    const doRequest = (extraOptions) => {
+      const options = Object.assign(
+        {
+          hostname: parsed.hostname,
+          port: parsed.port || (isHttps ? 443 : 80),
+          path: parsed.pathname + parsed.search,
+          method: "GET",
+          headers: { "User-Agent": "cargo-dist-npm-installer" },
+        },
+        extraOptions || {},
+      );
+
+      if (proxy && !isHttps) {
+        // HTTP through HTTP proxy: request the full URL via the proxy
+        options.hostname = proxy.hostname;
+        options.port = proxy.port;
+        options.path = urlString;
+        if (proxy.auth) {
+          options.headers["Proxy-Authorization"] =
+            "Basic " + Buffer.from(proxy.auth).toString("base64");
+        }
+      }
+
+      const req = mod.request(options, (res) => {
+        if (
+          res.statusCode >= 300 &&
+          res.statusCode < 400 &&
+          res.headers.location
+        ) {
+          res.resume();
+          const nextUrl = new URL(res.headers.location, urlString).toString();
+          return download(nextUrl, maxRedirects - 1).then(resolve, reject);
+        }
+        if (res.statusCode < 200 || res.statusCode >= 300) {
+          res.resume();
+          return reject(new Error(`HTTP ${res.statusCode} from ${urlString}`));
+        }
+        resolve(res);
+      });
+      req.on("error", reject);
+      req.end();
+    };
+
+    if (proxy && isHttps) {
+      connectThroughProxy(proxy, parsed).then(
+        (socket) => doRequest({ socket, agent: false }),
+        reject,
+      );
+    } else {
+      doRequest();
+    }
+  });
+}
+
+class Package {
+  constructor(platform, name, url, filename, zipExt, binaries) {
+    let errors = [];
+    if (typeof url !== "string") {
+      errors.push("url must be a string");
+    } else {
+      try {
+        new URL(url);
+      } catch (e) {
+        errors.push(e);
+      }
+    }
+    if (name && typeof name !== "string") {
+      errors.push("package name must be a string");
+    }
+    if (!name) {
+      errors.push("You must specify the name of your package");
+    }
+    if (binaries && typeof binaries !== "object") {
+      errors.push("binaries must be a string => string map");
+    }
+    if (!binaries) {
+      errors.push("You must specify the binaries in the package");
+    }
+
+    if (errors.length > 0) {
+      let errorMsg =
+        "One or more of the parameters you passed to the Binary constructor are invalid:\n";
+      errors.forEach((error) => {
+        errorMsg += error;
+      });
+      errorMsg +=
+        '\n\nCorrect usage: new Package("my-binary", "https://example.com/binary/download.tar.gz", {"my-binary": "my-binary"})';
+      error(errorMsg);
+    }
+
+    this.platform = platform;
+    this.url = url;
+    this.name = name;
+    this.filename = filename;
+    this.zipExt = zipExt;
+    this.installDirectory = join(__dirname, "node_modules", ".bin_real");
+    this.binaries = binaries;
+
+    if (!existsSync(this.installDirectory)) {
+      mkdirSync(this.installDirectory, { recursive: true });
+    }
+  }
+
+  exists() {
+    for (const binaryName in this.binaries) {
+      const binRelPath = this.binaries[binaryName];
+      const binPath = join(this.installDirectory, binRelPath);
+      if (!existsSync(binPath)) {
+        return false;
+      }
+    }
+    return true;
+  }
+
+  install(suppressLogs = false) {
+    if (this.exists()) {
+      if (!suppressLogs) {
+        console.error(
+          `${this.name} is already installed, skipping installation.`,
+        );
+      }
+      return Promise.resolve();
+    }
+
+    try {
+      rmSync(this.installDirectory, { recursive: true, force: true });
+    } catch {
+      // ignore - directory may not exist
+    }
+
+    mkdirSync(this.installDirectory, { recursive: true });
+
+    if (!suppressLogs) {
+      console.error(`Downloading release from ${this.url}`);
+    }
+
+    return download(this.url)
+      .then((res) => {
+        return new Promise((resolve, reject) => {
+          mkdtemp(`${tmpDir}${sep}`, (err, directory) => {
+            if (err) return reject(err);
+            let tempFile = join(directory, this.filename);
+            const sink = res.pipe(createWriteStream(tempFile));
+            sink.on("error", (err) => reject(err));
+            sink.on("close", () => {
+              if (/\.tar\.*/.test(this.zipExt)) {
+                const result = spawnSync("tar", [
+                  "xf",
+                  tempFile,
+                  // The tarballs are stored with a leading directory
+                  // component; we strip one component in the
+                  // shell installers too.
+                  "--strip-components",
+                  "1",
+                  "-C",
+                  this.installDirectory,
+                ]);
+                if (result.status == 0) {
+                  resolve();
+                } else if (result.error) {
+                  reject(result.error);
+                } else {
+                  reject(
+                    new Error(
+                      `An error occurred untarring the artifact: stdout: ${result.stdout}; stderr: ${result.stderr}`,
+                    ),
+                  );
+                }
+              } else if (this.zipExt == ".zip") {
+                let result;
+                if (this.platform.artifactName.includes("windows")) {
+                  // Windows does not have "unzip" by default on many installations, instead
+                  // we use Expand-Archive from powershell
+                  result = spawnSync("powershell.exe", [
+                    "-NoProfile",
+                    "-NonInteractive",
+                    "-Command",
+                    `& {
+                        param([string]$LiteralPath, [string]$DestinationPath)
+                        Expand-Archive -LiteralPath $LiteralPath -DestinationPath $DestinationPath -Force
+                    }`,
+                    tempFile,
+                    this.installDirectory,
+                  ]);
+                } else {
+                  result = spawnSync("unzip", [
+                    "-q",
+                    tempFile,
+                    "-d",
+                    this.installDirectory,
+                  ]);
+                }
+
+                if (result.status == 0) {
+                  resolve();
+                } else if (result.error) {
+                  reject(result.error);
+                } else {
+                  reject(
+                    new Error(
+                      `An error occurred unzipping the artifact: stdout: ${result.stdout}; stderr: ${result.stderr}`,
+                    ),
+                  );
+                }
+              } else {
+                reject(
+                  new Error(`Unrecognized file extension: ${this.zipExt}`),
+                );
+              }
+            });
+          });
+        });
+      })
+      .then(() => {
+        if (!suppressLogs) {
+          console.error(`${this.name} has been installed!`);
+        }
+      })
+      .catch((e) => {
+        error(`Error fetching release: ${e.message}`);
+      });
+  }
+
+  run(binaryName) {
+    const promise = !this.exists() ? this.install(true) : Promise.resolve();
+
+    promise
+      .then(() => {
+        const [, , ...args] = process.argv;
+
+        const options = { cwd: process.cwd(), stdio: "inherit" };
+
+        const binRelPath = this.binaries[binaryName];
+        if (!binRelPath) {
+          error(`${binaryName} is not a known binary in ${this.name}`);
+        }
+        const binPath = join(this.installDirectory, binRelPath);
+        const result = spawnSync(binPath, args, options);
+
+        if (result.error) {
+          error(result.error);
+        }
+
+        process.exit(result.status);
+      })
+      .catch((e) => {
+        error(e.message);
+      });
+  }
+}
+
+module.exports.Package = Package;