@tobilu/qmd 2.0.1 → 2.5.1

package/CHANGELOG.md

@@ -2,6 +2,183 @@
 
 ## [Unreleased]
 
+## [2.5.1] - 2026-05-20
+
+### Changes
+
+- Release: publish from GitHub Actions via npm Trusted Publishing/OIDC instead of a long-lived `NPM_TOKEN` secret.
+
+## [2.5.0] - 2026-05-19
+
+### Changes
+
+- Dependencies: update core SQLite/config/chunking packages (`better-sqlite3`, `yaml`, `web-tree-sitter`, `tree-sitter-go`, and `tree-sitter-python`) while keeping incompatible `zod`, `tsx`, and `vitest` majors pinned.
+- Agent skills: add `qmd skills list|get|path` to serve version-matched runtime skill instructions from the installed CLI, and make `qmd skill install` write a stable discovery stub so installed agent skills do not go stale after QMD upgrades.
+- CLI: add `qmd doctor` for index/runtime diagnostics, including SQLite/sqlite-vec versions, embedding fingerprint freshness, mixed-fingerprint detection, safe legacy fingerprint adoption, and content-hash sampling.
+
+### Fixes
+
+- Launcher: prefer runnable TypeScript source in git checkouts even when ignored `dist/` artifacts exist, while packaged installs continue to run `dist/`.
+- GPU: keep node-llama-cpp's documented `gpu: "auto"` initialization as the primary path, then perform no-build packaged CUDA/Vulkan/Metal probes only if auto falls back to CPU.
+- CLI: move GPU/CPU runtime diagnostics out of `qmd status`; use `qmd doctor` for device probing and related environment guidance.
+- CLI: point unexpected command/setup failures toward `qmd doctor` so diagnostics are the default next step when QMD behaves incorrectly.
+- Doctor: explicitly warn when `content_vectors` contains multiple non-empty embedding fingerprint names, with the per-fingerprint document/chunk breakdown.
+- Embed: make the TTY progress line label byte-based input progress explicitly, show embedded chunks as a count, and shorten the displayed model name.
+- Embed: retain per-chunk failure details, retry failed chunks after later successful embeds and again when no other chunks remain, clear recovered errors, and cap retries to avoid endless loops.
+- Tests: expand the container smoke harness to cover npm-global, npx-style, and Bun-global install scenarios, always checking auto and `QMD_FORCE_CPU=1` doctor modes, with opt-in tiny `qmd embed` and GPU probe runs for supported container runtimes.
+- Embedding: fingerprint vector metadata using the active embedding model and formatting/chunking parameters so stale vectors are treated as pending after search semantics change. Legacy `content_vectors` columns are migrated lazily on first vector-health/write use to preserve fast QMD startup.
+
+- Skill: expand the packaged QMD skill with retrieval-first workflows, structured query examples, wiki/source collection guidance, and safe fallbacks when model-backed search is unavailable.
+- Tests: make `bun run test` execute the local unit suite under both Node/Vitest and Bun (`test:node` + `test:bun`) so runtime-specific regressions are caught before CI.
+- Model config: centralize embedding/rerank/generation model resolution so `qmd embed`, `status`, `query`, `vsearch`, `pull`, SDK vector search, and `bench` use the same active `.qmd/index.yaml` model hints and environment fallbacks.
+- GPU/status: `qmd status` now uses the same embedding model identity as `qmd embed` when computing pending embeddings, so URI-backed embeddings are not incorrectly reported as pending under the legacy `embeddinggemma` alias.
+- GPU status: `qmd status` now always shows GPU mode/configuration without unsafe native probing, and CPU-fallback warnings point to `QMD_STATUS_DEVICE_PROBE=1 qmd status` for an actual backend probe. The no-GPU warning is emitted once per process instead of once per LLM instance during benchmarks.
+- GPU: add `QMD_FORCE_CPU=1` / `--no-gpu` to bypass CUDA/Vulkan/Metal probing entirely, and route native llama.cpp stdout noise to stderr so JSON output stays parseable during search/query commands.
+- Snippet line numbers: `qmd_query` (MCP), HTTP `/query`, and `qmd query`
+  (CLI JSON output and snippet headers) now return absolute source-file
+  line numbers instead of chunk-local ones, so the `line` field can be
+  passed back to `qmd_get` as `fromLine` without a separate lookup.
+  Snippet selection remains scoped to the best matching chunk
+  (preserves #149).
+- CLI: `qmd query --full` now emits the full document body in all output
+  formats (json, csv, md, xml), restoring the documented behavior of the
+  flag. Previously it returned only the best matching chunk (~3.6KB max
+  per result). Output payload for `--full` queries is now proportional
+  to total document size.
+- macOS Metal: `qmd query --json` now flushes successful JSON output and uses a safe immediate-exit path on Darwin to avoid ggml Metal finalizer aborts; other commands still dispose LLM contexts/models before the llama runtime. #368
+- Embedding: require complete chunk coverage before treating a document as
+  embedded, remove partial vectors when chunk/session failures leave a
+  document incomplete, and keep `qmd status` pending counts honest after
+  interrupted long embed runs. #637 #378
+- Embedding: `qmd embed -c <collection>` now scopes pending-doc selection
+  to the requested collection instead of embedding global pending work.
+  Scoped `--force` clears only collection-owned vectors, preserves shared
+  hashes referenced by sibling collections, and drops `vectors_vec` only
+  when the scoped clear empties all vectors.
+- Hybrid search: weight RRF lists by query type so original FTS and original vector evidence get the intended 2x boost, instead of accidentally boosting the first lexical expansion. #591
+- MCP: seed llama.cpp/GGML quiet env vars before launching `qmd mcp` so native logs cannot pollute stdio JSON-RPC framing. #593
+- CLI: remove CommonJS `require()` calls from ESM index path normalization so `qmd --index <path>` no longer crashes with `ERR_AMBIGUOUS_MODULE_SYNTAX` on Node 22+. #634
+- Windows CUDA: serialize llama.cpp embedding/reranking contexts by default to avoid intermittent `ggml-cuda.cu:98` crashes in `qmd query`; set `QMD_EMBED_PARALLELISM` to opt back into parallel contexts if your driver is stable. #519
+- MCP: make `qmd mcp --index <name>` use the selected index for both foreground and daemon HTTP servers instead of falling back to the default store. #343
+- Embedding: respect `QMD_EMBED_MODEL` consistently for vector indexing and vector-backed search, with default-model fallback when unset.
+- Config: use one home-directory resolver for YAML config and the default SQLite cache path, avoiding Windows CLI/MCP split-brain when `HOME` is unset.
+- GPU: respect explicit `QMD_LLAMA_GPU=metal|vulkan|cuda` backend overrides instead of always using auto GPU selection. #529
+- Fix: preserve original filename case in `handelize()`. The previous
+  `.toLowerCase()` call made indexed paths unreachable on case-sensitive
+  filesystems (Linux). `qmd update` automatically migrates legacy
+  lowercase paths without re-embedding.
+- CLI: make `qmd status` skip native `node-llama-cpp` device probing by
+  default so status stays safe on machines with broken or unsupported GPU
+  drivers. Set `QMD_STATUS_DEVICE_PROBE=1` to opt in.
+- CLI: lazy-load `node-llama-cpp` so lightweight commands such as
+  `qmd status` do not import native ML dependencies or trigger llama.cpp
+  builds on ARM/no-GPU machines. #491
+- Store: keep content rows referenced by inactive documents during orphan
+  cleanup so `qmd update` preserves soft-deleted tombstones for removed
+  files. #585
+- Packaging: install AST grammar WASM packages as required dependencies so
+  Bun global installs include TypeScript/TSX/JavaScript grammars, and add a
+  `smoke:package-grammars` verification command. #595
+- Launcher: add wrapper smoke coverage for scoped package, npm/npx,
+  Homebrew/Linuxbrew, Bun global symlink layouts, and `$BUN_INSTALL`
+  false-positive runtime selection regressions. #351 #353 #354 #356 #358 #359
+
+## [2.1.0] - 2026-04-05
+
+Code files now chunk at function and class boundaries via tree-sitter,
+clickable editor links land you at the right line from search results,
+and per-collection model configuration means you can point different
+collections at different embedding models. 25+ community PRs fix
+embedding stability, BM25 accuracy, and cross-platform launcher issues.
+
+### Changes
+
+- AST-aware chunking for code files via `web-tree-sitter`. Supported
+  languages: TypeScript/JavaScript, Python, Go, and Rust. Code files
+  are chunked at function, class, and import boundaries instead of
+  arbitrary text positions. Markdown and unknown file types are unchanged.
+  `--chunk-strategy <auto|regex>` flag on `qmd embed` and `qmd query`
+  (default `regex`). SDK: `chunkStrategy` option on `embed()` and
+  `search()`. `qmd status` shows grammar availability.
+- `qmd bench <fixture.json>` command for search quality benchmarks.
+  Measures precision@k, recall, MRR, and F1 across BM25, vector, hybrid,
+  and full pipeline backends. Ships with an example fixture against
+  the eval-docs test collection. #470 (thanks @jmilinovich)
+- `models:` section in `index.yml` lets you configure `embed`, `rerank`,
+  and `generate` model URIs per collection. Resolution order is
+  config > env var (`QMD_EMBED_MODEL`, `QMD_RERANK_MODEL`,
+  `QMD_GENERATE_MODEL`) > built-in default. #502
+  (thanks @JohnRichardEnders)
+- CLI search output now emits clickable OSC 8 terminal hyperlinks when
+  stdout is a TTY. Links resolve `qmd://` paths to absolute filesystem
+  paths and open in editors via URI templates (default:
+  `vscode://file/{path}:{line}:{col}`). Configure with `QMD_EDITOR_URI`
+  or `editor_uri` in the YAML config. #508 (thanks @danmackinlay)
+- `--no-rerank` flag skips the reranking step in `qmd query` — useful
+  when you want fast results or don't have a GPU. Also exposed as
+  `rerank: false` on the MCP `query` tool. #370 (thanks @mvanhorn),
+  #478 (thanks @zestyboy)
+- ONNX conversion script for deploying embedding models via
+  Transformers.js. #399 (thanks @shreyaskarnik)
+- GitHub Actions workflow to build the Nix flake on Linux and macOS.
+
+### Fixes
+
+- Embedding: prevent `qmd embed` from running indefinitely when the
+  embedding loop stalls. #458 (thanks @ccc-fff)
+- Embedding: truncate oversized text before embedding to prevent GGML
+  crash, and bound memory usage during batch embedding. #393
+  (thanks @lskun), #395 (thanks @ProgramCaiCai)
+- Embedding: set explicit embed context size (default 2048, configurable
+  via `QMD_EMBED_CONTEXT_SIZE`) instead of using the model's full
+  window. #500
+- Embedding: error on dimension mismatch instead of silently rebuilding
+  the vec0 table. #501
+- Embedding: handle vec0 `OR REPLACE` limitation in `insertEmbedding`.
+  #456 (thanks @antonio-mello-ai)
+- Embedding: fix model selection when multiple models are configured.
+  #494
+- BM25: correct field weights to include all 3 FTS columns — title,
+  body, and path were not weighted correctly. #462 (thanks @goldsr09)
+- BM25: handle hyphenated tokens in FTS5 lex queries so terms like
+  "real-time" match correctly. #463 (thanks @goldsr09)
+- BM25: preserve underscores in search terms instead of stripping them.
+  #404
+- BM25: use CTE in `searchFTS` to prevent query planner regression with
+  collection filter.
+- Reranker: increase default context size 2048→4096 and make
+  configurable via `QMD_RERANK_CONTEXT_SIZE`. Fix template overhead
+  underestimate 200→512. #453 (thanks @builderjarvis)
+- GPU: catch initialization failures and fall back to CPU instead of
+  crashing.
+- MCP: read version from `package.json` instead of hardcoding. #431
+- MCP: include collection name in status output. #416
+- Multi-get: support brace expansion patterns in glob matching. #424
+- Launcher: prioritize `package-lock.json` to prevent Bun false
+  positive. #385 (thanks @rymalia)
+- Launcher: remove `$BUN_INSTALL` check that caused false Bun detection.
+  #362 (thanks @syedair)
+- Launcher: skip Git Bash path detection on WSL. #371
+  (thanks @oysteinkrog)
+- Model cache: respect `XDG_CACHE_HOME` for model cache directory. #457
+  (thanks @antonio-mello-ai)
+- SQLite: add macOS Homebrew SQLite support for Bun and restore
+  actionable errors. #377 (thanks @serhii12)
+- Pin zod to exact 4.2.1 to fix `tsc` build failure. #382
+  (thanks @rymalia)
+- Preserve dots and original case in `handelize()` — filenames like
+  `MEMORY.md` no longer become `memory-md`. #475 (thanks @alexei-led)
+- Include `line` in `--json` search output so editor integrations can
+  jump directly to `file:line`. #506 (thanks @danmackinlay)
+- Nix: fix paths in flake and make Bun dependency a fixed-output
+  derivation so sandboxed Linux builds work offline. #479
+  (thanks @surma-dump)
+- Sync stale `bun.lock` (`better-sqlite3` 11.x → 12.x). CI and release
+  script now use `--frozen-lockfile` to prevent recurrence. #386
+  (thanks @Mic92)
+- Approve native build scripts in pnpm so `better-sqlite3` and
+  tree-sitter modules compile correctly. Update vitest ^3.0.0 → ^3.2.4.
+
 ## [2.0.1] - 2026-03-10
 
 ### Changes

package/README.md

@@ -318,6 +318,7 @@ const result = await store.update({
 // Generate vector embeddings
 const embedResult = await store.embed({
   force: false,           // true to re-embed everything
+  chunkStrategy: "auto",  // "regex" (default) or "auto" (AST for code files)
   onProgress: ({ current, total, collection }) => {
     console.log(`Embedding ${current}/${total}`)
   },
@@ -564,8 +565,27 @@ qmd embed
 
 # Force re-embed everything
 qmd embed -f
+
+# Enable AST-aware chunking for code files (TS, JS, Python, Go, Rust)
+qmd embed --chunk-strategy auto
+
+# Also works with query for consistent chunk selection
+qmd query "auth flow" --chunk-strategy auto
 ```
 
+**AST-aware chunking** (`--chunk-strategy auto`) uses tree-sitter to chunk code
+files at function, class, and import boundaries instead of arbitrary text
+positions. This produces higher-quality chunks and better search results for
+codebases. Markdown and other file types always use regex-based chunking
+regardless of strategy.
+
+The default is `regex` (existing behavior). Use `--chunk-strategy auto` to
+opt in. Run `qmd status` to verify which grammars are available.
+
+> **Note:** Tree-sitter grammars are optional dependencies. If they are not
+> installed, `--chunk-strategy auto` falls back to regex-only chunking
+> automatically. Tested on both Node.js and Bun.
+
 ### Context Management
 
 Context adds descriptive metadata to collections and paths, helping search understand your content.
@@ -644,7 +664,13 @@ qmd get <file>[:line]  # Get document, optionally starting at line
 
 ### Output Format
 
-Default output is colorized CLI format (respects `NO_COLOR` env):
+Default output is colorized CLI format (respects `NO_COLOR` env).
+
+When stdout is a TTY, result paths are emitted as clickable terminal hyperlinks (OSC 8). Clicking a path opens the file in your editor using an editor URI template.
+
+When stdout is not a TTY (for example piped to another command or redirected to a file), QMD emits plain text paths with no escape sequences.
+
+TTY example:
 
 ```
 docs/guide.md:42 #a1b2c3
@@ -666,6 +692,27 @@ Discussion about code quality and craftsmanship
 in the development process.
 ```
 
+Configure the editor link target with `QMD_EDITOR_URI` (or `editor_uri` in config):
+
+```sh
+# VS Code (default)
+export QMD_EDITOR_URI="vscode://file/{path}:{line}:{col}"
+
+# Cursor
+export QMD_EDITOR_URI="cursor://file/{path}:{line}:{col}"
+
+# Zed
+export QMD_EDITOR_URI="zed://file/{path}:{line}:{col}"
+
+# Sublime Text
+export QMD_EDITOR_URI="subl://open?url=file://{path}&line={line}"
+```
+
+Template placeholders:
+- `{path}` absolute filesystem path (URI-encoded)
+- `{line}` 1-based line number
+- `{col}` or `{column}` 1-based column number
+
 - **Path**: Collection-relative path (e.g., `docs/guide.md`)
 - **Docid**: Short hash identifier (e.g., `#a1b2c3`) - use with `qmd get #a1b2c3`
 - **Title**: Extracted from document (first heading or filename)
@@ -750,6 +797,9 @@ llm_cache       -- Cached LLM responses (query expansion, rerank scores)
 | Variable | Default | Description |
 |----------|---------|-------------|
 | `XDG_CACHE_HOME` | `~/.cache` | Cache directory location |
+| `QMD_LLAMA_GPU` | `auto` | Force llama.cpp GPU backend (`metal`, `vulkan`, `cuda`) or disable GPU with `false` |
+| `QMD_FORCE_CPU` | unset | Set to `1`/`true` to force CPU mode before any CUDA/Vulkan/Metal probing. Equivalent CLI flag: `--no-gpu`. |
+| `QMD_EMBED_PARALLELISM` | automatic | Override embedding/reranking context parallelism (1-8). Windows CUDA defaults to `1` because parallel CUDA contexts can crash with `ggml-cuda.cu:98`; use Vulkan or raise this only if your driver is stable. |
 
 ## How It Works
 
@@ -813,6 +863,19 @@ The squared distance decay means a heading 200 tokens back (score ~30) still bea
 
 **Code Fence Protection:** Break points inside code blocks are ignored—code stays together. If a code block exceeds the chunk size, it's kept whole when possible.
 
+**AST-Aware Chunking (Code Files):**
+
+For supported code files, QMD also parses the source with [tree-sitter](https://tree-sitter.github.io/) and adds AST-derived break points that are merged with the regex scores above:
+
+| AST Node | Score | Languages |
+|----------|-------|-----------|
+| Class / interface / struct / impl / trait | 100 | All |
+| Function / method | 90 | All |
+| Type alias / enum | 80 | All |
+| Import / use declaration | 60 | All |
+
+Supported for `.ts`, `.tsx`, `.js`, `.jsx`, `.py`, `.go`, and `.rs` files. Enable with `--chunk-strategy auto`. Markdown and other file types always use regex chunking.
+
 ### Query Flow (Hybrid)
 
 ```

package/bin/qmd

@@ -15,9 +15,54 @@ done
 # to avoid native module ABI mismatches (e.g., better-sqlite3 compiled for bun vs node)
 DIR="$(cd -P "$(dirname "$SOURCE")/.." && pwd)"
 
-# Check if we were installed with bun (look for bun.lock or bun-lockb)
-if [ -f "$DIR/bun.lock" ] || [ -f "$DIR/bun.lockb" ] || [ -n "$BUN_INSTALL" ]; then
-  exec bun "$DIR/dist/cli/qmd.js" "$@"
+# MCP stdio reserves stdout exclusively for JSON-RPC frames. node-llama-cpp
+# / llama.cpp / ggml can write native logs directly to stdout before JS-level
+# log handlers are attached, so seed the native quiet env before Node/Bun imports
+# the CLI and its LLM modules. Preserve explicit user values when provided.
+if [ "$1" = "mcp" ]; then
+  export LLAMA_LOG_LEVEL="${LLAMA_LOG_LEVEL:-error}"
+  export GGML_LOG_LEVEL="${GGML_LOG_LEVEL:-error}"
+  export GGML_BACKEND_SILENT="${GGML_BACKEND_SILENT:-1}"
+fi
+
+JS="$DIR/dist/cli/qmd.js"
+TS="$DIR/src/cli/qmd.ts"
+
+# In published packages, bin/qmd must run dist/. In a git checkout, however,
+# dist/ is often ignored and can be stale after git reset or branch switches.
+# Prefer source mode only for checkouts so ./bin/qmd reflects the checked-out
+# source without changing packaged/runtime behavior.
+if [ -e "$DIR/.git" ] && [ -f "$TS" ]; then
+  if [ -f "$DIR/bun.lock" ] || [ -f "$DIR/bun.lockb" ]; then
+    if command -v bun >/dev/null 2>&1; then
+      exec bun "$TS" "$@"
+    fi
+  fi
+  if [ -f "$DIR/node_modules/tsx/dist/cli.mjs" ]; then
+    exec node "$DIR/node_modules/tsx/dist/cli.mjs" "$TS" "$@"
+  fi
+fi
+
+if [ ! -f "$JS" ]; then
+  echo "qmd is not built: missing $JS" >&2
+  echo "Run: bun install && bun run build" >&2
+  echo "Or:  npm install && npm run build" >&2
+  echo "After building, run: qmd doctor" >&2
+  exit 1
+fi
+
+# Detect the package manager that installed dependencies by checking lockfiles.
+# $BUN_INSTALL is intentionally NOT checked — it only indicates that bun exists
+# on the system, not that it was used to install this package (see #361).
+#
+# package-lock.json takes priority: if it exists, npm installed the native
+# modules for Node.  The repo ships bun.lock, so without this check, source
+# builds that use npm would be incorrectly routed to bun, causing ABI
+# mismatches with better-sqlite3 / sqlite-vec (see #381).
+if [ -f "$DIR/package-lock.json" ]; then
+  exec node "$JS" "$@"
+elif [ -f "$DIR/bun.lock" ] || [ -f "$DIR/bun.lockb" ]; then
+  exec bun "$JS" "$@"
 else
-  exec node "$DIR/dist/cli/qmd.js" "$@"
+  exec node "$JS" "$@"
 fi

package/dist/ast.d.ts

@@ -0,0 +1,65 @@
+/**
+ * AST-aware chunking support via web-tree-sitter.
+ *
+ * Provides language detection, AST break point extraction for supported
+ * code file types, and a stub for future symbol extraction.
+ *
+ * All functions degrade gracefully: parse failures or unsupported languages
+ * return empty arrays, falling back to regex-only chunking.
+ *
+ * ## Dependency Note
+ *
+ * Grammar packages (tree-sitter-typescript, etc.) are listed as
+ * optionalDependencies with pinned versions. They ship native prebuilds
+ * and source files (~72 MB total) but QMD only uses the .wasm files
+ * (~5 MB). If install size becomes a concern, the .wasm files can be
+ * bundled directly in the repo (e.g. assets/grammars/) and resolved
+ * via import.meta.url instead of require.resolve(), eliminating the
+ * grammar packages entirely.
+ */
+import type { BreakPoint } from "./store.js";
+export type SupportedLanguage = "typescript" | "tsx" | "javascript" | "python" | "go" | "rust";
+/**
+ * Detect language from file path extension.
+ * Returns null for unsupported or unknown extensions (including .md).
+ */
+export declare function detectLanguage(filepath: string): SupportedLanguage | null;
+export declare function formatGrammarLoadError(language: SupportedLanguage, err: unknown): string;
+/**
+ * Parse a source file and return break points at AST node boundaries.
+ *
+ * Returns an empty array for unsupported languages, parse failures,
+ * or grammar loading failures. Never throws.
+ *
+ * @param content - The file content to parse.
+ * @param filepath - The file path (used for language detection).
+ * @returns Array of BreakPoint objects suitable for merging with regex break points.
+ */
+export declare function getASTBreakPoints(content: string, filepath: string): Promise<BreakPoint[]>;
+/**
+ * Check which tree-sitter grammars are available.
+ * Returns a status object for each supported language.
+ */
+export declare function getASTStatus(): Promise<{
+    available: boolean;
+    languages: {
+        language: SupportedLanguage;
+        available: boolean;
+        error?: string;
+    }[];
+}>;
+/**
+ * Metadata about a code symbol within a chunk.
+ * Stubbed for Phase 2 — always returns empty array in Phase 1.
+ */
+export interface SymbolInfo {
+    name: string;
+    kind: string;
+    signature?: string;
+    line: number;
+}
+/**
+ * Extract symbol metadata for code within a byte range.
+ * Stubbed for Phase 2 — returns empty array.
+ */
+export declare function extractSymbols(_content: string, _language: string, _startPos: number, _endPos: number): SymbolInfo[];