npm - better-dsh - Versions diffs - 0.2.5 → 0.3.0 - Mend

better-dsh 0.2.5 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (999) hide show

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/user-guide/skills/optional/research/research-qmd.md DELETED Viewed

@@ -1,459 +0,0 @@
----
-title: "Qmd — Hybrid local search over notes, docs, and transcripts"
-sidebar_label: "Qmd"
-description: "Hybrid local search over notes, docs, and transcripts"
----
-{/* This page is auto-generated from the skill's SKILL.md by website/scripts/generate-skill-docs.py. Edit the source SKILL.md, not this page. */}
-# Qmd
-Hybrid local search over notes, docs, and transcripts.
-## Skill metadata
-| | |
-|---|---|
-| Source | Optional — install with `hermes skills install official/research/qmd` |
-| Path | `optional-skills/research\qmd` |
-| Version | `1.0.0` |
-| Author | Hermes Agent + Teknium |
-| License | MIT |
-| Platforms | macos, linux |
-| Tags | `Search`, `Knowledge-Base`, `RAG`, `Notes`, `MCP`, `Local-AI` |
-| Related skills | [`obsidian`](/docs/user-guide/skills/bundled/note-taking/note-taking-obsidian), [`hermes-agent`](/docs/user-guide/skills/bundled/autonomous-ai-agents/autonomous-ai-agents-hermes-agent), [`arxiv`](/docs/user-guide/skills/bundled/research/research-arxiv) |
-## Reference: full SKILL.md
-:::info
-The following is the complete skill definition that Hermes loads when this skill is triggered. This is what the agent sees as instructions when the skill is active.
-:::
-# QMD — Query Markup Documents
-Local, on-device search engine for personal knowledge bases. Indexes markdown
-notes, meeting transcripts, documentation, and any text-based files, then
-provides hybrid search combining keyword matching, semantic understanding, and
-LLM-powered reranking — all running locally with no cloud dependencies.
-Created by [Tobi Lütke](https://github.com/tobi/qmd). MIT licensed.
-## When to Use
-- User asks to search their notes, docs, knowledge base, or meeting transcripts
-- User wants to find something across a large collection of markdown/text files
-- User wants semantic search ("find notes about X concept") not just keyword grep
-- User has already set up qmd collections and wants to query them
-- User asks to set up a local knowledge base or document search system
-- Keywords: "search my notes", "find in my docs", "knowledge base", "qmd"
-## Prerequisites
-### Node.js >= 22 (required)
-```bash
-# Check version
-node --version  # must be >= 22
-# macOS — install or upgrade via Homebrew
-brew install node@22
-# Linux — use NodeSource or nvm
-curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -
-sudo apt-get install -y nodejs
-# or with nvm:
-nvm install 22 && nvm use 22
-```
-### SQLite with Extension Support (macOS only)
-macOS system SQLite lacks extension loading. Install via Homebrew:
-```bash
-brew install sqlite
-```
-### Install qmd
-```bash
-npm install -g @tobilu/qmd
-# or with Bun:
-bun install -g @tobilu/qmd
-```
-First run auto-downloads 3 local GGUF models (~2GB total):
-| Model | Purpose | Size |
-|-------|---------|------|
-| embeddinggemma-300M-Q8_0 | Vector embeddings | ~300MB |
-| qwen3-reranker-0.6b-q8_0 | Result reranking | ~640MB |
-| qmd-query-expansion-1.7B | Query expansion | ~1.1GB |
-### Verify Installation
-```bash
-qmd --version
-qmd status
-```
-## Quick Reference
-| Command | What It Does | Speed |
-|---------|-------------|-------|
-| `qmd search "query"` | BM25 keyword search (no models) | ~0.2s |
-| `qmd vsearch "query"` | Semantic vector search (1 model) | ~3s |
-| `qmd query "query"` | Hybrid + reranking (all 3 models) | ~2-3s warm, ~19s cold |
-| `qmd get <docid>` | Retrieve full document content | instant |
-| `qmd multi-get "glob"` | Retrieve multiple files | instant |
-| `qmd collection add <path> --name <n>` | Add a directory as a collection | instant |
-| `qmd context add <path> "description"` | Add context metadata to improve retrieval | instant |
-| `qmd embed` | Generate/update vector embeddings | varies |
-| `qmd status` | Show index health and collection info | instant |
-| `qmd mcp` | Start MCP server (stdio) | persistent |
-| `qmd mcp --http --daemon` | Start MCP server (HTTP, warm models) | persistent |
-## Setup Workflow
-### 1. Add Collections
-Point qmd at directories containing your documents:
-```bash
-# Add a notes directory
-qmd collection add ~/notes --name notes
-# Add project docs
-qmd collection add ~/projects/myproject/docs --name project-docs
-# Add meeting transcripts
-qmd collection add ~/meetings --name meetings
-# List all collections
-qmd collection list
-```
-### 2. Add Context Descriptions
-Context metadata helps the search engine understand what each collection
-contains. This significantly improves retrieval quality:
-```bash
-qmd context add qmd://notes "Personal notes, ideas, and journal entries"
-qmd context add qmd://project-docs "Technical documentation for the main project"
-qmd context add qmd://meetings "Meeting transcripts and action items from team syncs"
-```
-### 3. Generate Embeddings
-```bash
-qmd embed
-```
-This processes all documents in all collections and generates vector
-embeddings. Re-run after adding new documents or collections.
-### 4. Verify
-```bash
-qmd status   # shows index health, collection stats, model info
-```
-## Search Patterns
-### Fast Keyword Search (BM25)
-Best for: exact terms, code identifiers, names, known phrases.
-No models loaded — near-instant results.
-```bash
-qmd search "authentication middleware"
-qmd search "handleError async"
-```
-### Semantic Vector Search
-Best for: natural language questions, conceptual queries.
-Loads embedding model (~3s first query).
-```bash
-qmd vsearch "how does the rate limiter handle burst traffic"
-qmd vsearch "ideas for improving onboarding flow"
-```
-### Hybrid Search with Reranking (Best Quality)
-Best for: important queries where quality matters most.
-Uses all 3 models — query expansion, parallel BM25+vector, reranking.
-```bash
-qmd query "what decisions were made about the database migration"
-```
-### Structured Multi-Mode Queries
-Combine different search types in a single query for precision:
-```bash
-# BM25 for exact term + vector for concept
-qmd query $'lex: rate limiter\nvec: how does throttling work under load'
-# With query expansion
-qmd query $'expand: database migration plan\nlex: "schema change"'
-```
-### Query Syntax (lex/BM25 mode)
-| Syntax | Effect | Example |
-|--------|--------|---------|
-| `term` | Prefix match | `perf` matches "performance" |
-| `"phrase"` | Exact phrase | `"rate limiter"` |
-| `-term` | Exclude term | `performance -sports` |
-### HyDE (Hypothetical Document Embeddings)
-For complex topics, write what you expect the answer to look like:
-```bash
-qmd query $'hyde: The migration plan involves three phases. First, we add the new columns without dropping the old ones. Then we backfill data. Finally we cut over and remove legacy columns.'
-```
-### Scoping to Collections
-```bash
-qmd search "query" --collection notes
-qmd query "query" --collection project-docs
-```
-### Output Formats
-```bash
-qmd search "query" --json        # JSON output (best for parsing)
-qmd search "query" --limit 5     # Limit results
-qmd get "#abc123"                # Get by document ID
-qmd get "path/to/file.md"       # Get by file path
-qmd get "file.md:50" -l 100     # Get specific line range
-qmd multi-get "journals/*.md" --json  # Batch retrieve by glob
-```
-## MCP Integration (Recommended)
-qmd exposes an MCP server that provides search tools directly to
-Hermes Agent via the native MCP client. This is the preferred
-integration — once configured, the agent gets qmd tools automatically
-without needing to load this skill.
-### Option A: Stdio Mode (Simple)
-Add to `~/.hermes/config.yaml`:
-```yaml
-mcp_servers:
-  qmd:
-    command: "qmd"
-    args: ["mcp"]
-    timeout: 30
-    connect_timeout: 45
-```
-This registers tools: `mcp_qmd_search`, `mcp_qmd_vsearch`,
-`mcp_qmd_deep_search`, `mcp_qmd_get`, `mcp_qmd_status`.
-**Tradeoff:** Models load on first search call (~19s cold start),
-then stay warm for the session. Acceptable for occasional use.
-### Option B: HTTP Daemon Mode (Fast, Recommended for Heavy Use)
-Start the qmd daemon separately — it keeps models warm in memory:
-```bash
-# Start daemon (persists across agent restarts)
-qmd mcp --http --daemon
-# Runs on http://localhost:8181 by default
-```
-Then configure Hermes Agent to connect via HTTP:
-```yaml
-mcp_servers:
-  qmd:
-    url: "http://localhost:8181/mcp"
-    timeout: 30
-```
-**Tradeoff:** Uses ~2GB RAM while running, but every query is fast
-(~2-3s). Best for users who search frequently.
-### Keeping the Daemon Running
-#### macOS (launchd)
-```bash
-cat > ~/Library/LaunchAgents/com.qmd.daemon.plist << 'EOF'
-<?xml version="1.0" encoding="UTF-8"?>
-<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
-  "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
-<plist version="1.0">
-<dict>
-  <key>Label</key>
-  <string>com.qmd.daemon</string>
-  <key>ProgramArguments</key>
-  <array>
-    <string>qmd</string>
-    <string>mcp</string>
-    <string>--http</string>
-    <string>--daemon</string>
-  </array>
-  <key>RunAtLoad</key>
-  <true/>
-  <key>KeepAlive</key>
-  <true/>
-  <key>StandardOutPath</key>
-  <string>/tmp/qmd-daemon.log</string>
-  <key>StandardErrorPath</key>
-  <string>/tmp/qmd-daemon.log</string>
-</dict>
-</plist>
-EOF
-launchctl load ~/Library/LaunchAgents/com.qmd.daemon.plist
-```
-#### Linux (systemd user service)
-```bash
-mkdir -p ~/.config/systemd/user
-cat > ~/.config/systemd/user/qmd-daemon.service << 'EOF'
-[Unit]
-Description=QMD MCP Daemon
-After=network.target
-[Service]
-ExecStart=qmd mcp --http --daemon
-Restart=on-failure
-RestartSec=10
-Environment=PATH=/usr/local/bin:/usr/bin:/bin
-[Install]
-WantedBy=default.target
-EOF
-systemctl --user daemon-reload
-systemctl --user enable --now qmd-daemon
-systemctl --user status qmd-daemon
-```
-### MCP Tools Reference
-Once connected, these tools are available as `mcp_qmd_*`:
-| MCP Tool | Maps To | Description |
-|----------|---------|-------------|
-| `mcp_qmd_search` | `qmd search` | BM25 keyword search |
-| `mcp_qmd_vsearch` | `qmd vsearch` | Semantic vector search |
-| `mcp_qmd_deep_search` | `qmd query` | Hybrid search + reranking |
-| `mcp_qmd_get` | `qmd get` | Retrieve document by ID or path |
-| `mcp_qmd_status` | `qmd status` | Index health and stats |
-The MCP tools accept structured JSON queries for multi-mode search:
-```json
-{
-  "searches": [
-    {"type": "lex", "query": "authentication middleware"},
-    {"type": "vec", "query": "how user login is verified"}
-  ],
-  "collections": ["project-docs"],
-  "limit": 10
-}
-```
-## CLI Usage (Without MCP)
-When MCP is not configured, use qmd directly via terminal:
-```
-terminal(command="qmd query 'what was decided about the API redesign' --json", timeout=30)
-```
-For setup and management tasks, always use terminal:
-```
-terminal(command="qmd collection add ~/Documents/notes --name notes")
-terminal(command="qmd context add qmd://notes 'Personal research notes and ideas'")
-terminal(command="qmd embed")
-terminal(command="qmd status")
-```
-## How the Search Pipeline Works
-Understanding the internals helps choose the right search mode:
-1. **Query Expansion** — A fine-tuned 1.7B model generates 2 alternative
-   queries. The original gets 2x weight in fusion.
-2. **Parallel Retrieval** — BM25 (SQLite FTS5) and vector search run
-   simultaneously across all query variants.
-3. **RRF Fusion** — Reciprocal Rank Fusion (k=60) merges results.
-   Top-rank bonus: #1 gets +0.05, #2-3 get +0.02.
-4. **LLM Reranking** — qwen3-reranker scores top 30 candidates (0.0-1.0).
-5. **Position-Aware Blending** — Ranks 1-3: 75% retrieval / 25% reranker.
-   Ranks 4-10: 60/40. Ranks 11+: 40/60 (trusts reranker more for long tail).
-**Smart Chunking:** Documents are split at natural break points (headings,
-code blocks, blank lines) targeting ~900 tokens with 15% overlap. Code
-blocks are never split mid-block.
-## Best Practices
-1. **Always add context descriptions** — `qmd context add` dramatically
-   improves retrieval accuracy. Describe what each collection contains.
-2. **Re-embed after adding documents** — `qmd embed` must be re-run when
-   new files are added to collections.
-3. **Use `qmd search` for speed** — when you need fast keyword lookup
-   (code identifiers, exact names), BM25 is instant and needs no models.
-4. **Use `qmd query` for quality** — when the question is conceptual or
-   the user needs the best possible results, use hybrid search.
-5. **Prefer MCP integration** — once configured, the agent gets native
-   tools without needing to load this skill each time.
-6. **Daemon mode for frequent users** — if the user searches their
-   knowledge base regularly, recommend the HTTP daemon setup.
-7. **First query in structured search gets 2x weight** — put the most
-   important/certain query first when combining lex and vec.
-## Troubleshooting
-### "Models downloading on first run"
-Normal — qmd auto-downloads ~2GB of GGUF models on first use.
-This is a one-time operation.
-### Cold start latency (~19s)
-This happens when models aren't loaded in memory. Solutions:
-- Use HTTP daemon mode (`qmd mcp --http --daemon`) to keep warm
-- Use `qmd search` (BM25 only) when models aren't needed
-- MCP stdio mode loads models on first search, stays warm for session
-### macOS: "unable to load extension"
-Install Homebrew SQLite: `brew install sqlite`
-Then ensure it's on PATH before system SQLite.
-### "No collections found"
-Run `qmd collection add <path> --name <name>` to add directories,
-then `qmd embed` to index them.
-### Embedding model override (CJK/multilingual)
-Set `QMD_EMBED_MODEL` environment variable for non-English content:
-```bash
-export QMD_EMBED_MODEL="your-multilingual-model"
-```
-## Data Storage
-- **Index & vectors:** `~/.cache/qmd/index.sqlite`
-- **Models:** Auto-downloaded to local cache on first run
-- **No cloud dependencies** — everything runs locally
-## References
-- [GitHub: tobi/qmd](https://github.com/tobi/qmd)
-- [QMD Changelog](https://github.com/tobi/qmd/blob/main/CHANGELOG.md)