ko-qmd 2.8.3-ko.4 → 2.8.3-ko.6

package/CHANGELOG.md

@@ -2,6 +2,152 @@
 
 ## [Unreleased]
 
+## [2.8.3-ko.6] - 2026-09-29
+
+Korean queries are handled better before and during reranking. A query written
+as "X 말고 Y" now searches for Y alone, and a Hangul query skips LLM expansion,
+which steadies hard-query results and cuts hybrid latency from about 1s to tens
+of ms per query. The reranker is now shown the chunk that best matches the
+query's Hangul syllable bigrams, led by the note title: held-out full MRR goes
+0.6589 → 0.7260 on a goldset generated from a real vault. Scoped vector search on
+small indexes runs one KNN (vector stage 250–1300ms → about 25ms). The README
+gains an English edition and an FAQ.
+
+- `QMD_RERANK_CACHE=<path>` keeps rerank scores in their own sqlite file instead of the index's
+  `llm_cache`, so a store over a rebuilt index reuses them. The key carries the query, the rerank
+  model, the doc-token cap and the chunk text, not the scoring code in `src/llm.ts`, so the file is
+  safe across index rebuilds with the same reranker code. It is not trimmed to 1000 rows. With it
+  set, a second `scripts/bench-ko.sh` run on ko-vault gives the same lines and its `full` backend
+  spends 3.7 s reranking instead of 300 s (the whole run drops from 5:17 to 0:56). Unset, nothing
+  changes. A fallback result (no rerank context could be created, every score 0.5) is no longer
+  cached in either place.
+
+- The reranker now sees a better chunk of a long Korean note. The chunk is chosen by the
+  two-syllable pairs of the query's Hangul words of three or more syllables, instead of whole
+  words, so `임베딩은` still finds a chunk about `임베딩`. Generic two-syllable words such as 방법
+  and 경우 no longer steer the choice. A query word with no Hangul (an English term, say) counts double. When the chosen chunk
+  does not contain the note title, `# <title>` is put in front of it. An autonomous loop
+  (`scripts/autoresearch/`) found this over 28 experiments on a goldset generated from a private
+  vault. The goldset's held-out split, which the loop never saw, goes from full MRR 0.6589 to
+  0.7260 (32 queries). ko-vault seam and hard lines are unchanged. The bm25, vector and hybrid
+  results do not read the rerank input and are identical on both sets (`BASELINE.md`
+  2026-09-28). An intent word now counts 1 point in chunk selection (was 0.5), keeping it at half a
+  query word now that a non-Hangul query word counts 2; no goldset carries intent, so no bench
+  number moves.
+
+- The reranker's input (the query string, which chunk of each candidate is sent and the document
+  text) is now built in `src/rerank-input.ts` for both `hybridQuery` and `structuredSearch`, which
+  had copied the chunk-scoring loop. Results are unchanged.
+
+- A Korean query of the form "X 말고 Y" now searches for Y alone. `hybridQuery` (CLI `qmd query`,
+  plain MCP `query`) and the `lex`/`vec` lines of `structuredSearch` (MCP `searches`, typed lines)
+  drop everything up to the last whitespace-separated 말고, 빼고, 제외하고 or 제외한. The rerank
+  query follows. `hyde` lines, `intent` and `qmd search`/`vsearch` stay as written. Before this
+  change every list also searched for X, the concept the user ruled out, so a note about X often
+  took rank 1. `아닌` and bare `제외` are not markers, because they also state conditions or act as
+  nouns. On the ko-vault seam form, the `neg` queries' full MRR goes 0.4500 → 0.6944. Hard
+  `full_r1` goes 0.5111 → 0.6111 and hard `full_mrr` 0.7472 → 0.8287. No query outside `neg`
+  changed its results, and that run is the gate's new reference (`BASELINE.md` 2026-09-27,
+  "a Korean negation clause is stripped before search").
+
+- `structuredSearch` (MCP `searches`, SDK `search({ queries })`, typed `lex:`/`vec:` lines) now
+  counts a relaxed lex list, one that matched only through the any-word retry, at half its slot's
+  weight, as `hybridQuery` already did. At a flat 2.0 for the first list, a lex line that matched
+  some of a Korean question's words outranked the vector list, and a decoy sharing those words took
+  rank 1. This one weight was the whole gap between the two paths on the Korean ko-vault queries:
+  re-fusing each path's lists with the other's weights reproduced its top 10 for 85 of 85 queries.
+  On the seam form, hard `hybrid_r1` goes 0.3111 → 0.3778 and `hybrid_r5` 0.8996 → 0.9211, with
+  `full_r1` unchanged at 0.5111, and that run is the gate's new reference (`BASELINE.md` 2026-09-27).
+
+- A query with any Hangul is no longer expanded by the LLM on the hybrid path (`hybridQuery`). That
+  path serves CLI `qmd query`, including an explicit `expand:` line, as well as a plain MCP `query`
+  and SDK `search({ query })`; `qmd vsearch` still expands. On the ko-vault goldset the
+  expansion model wrapped Korean questions in English sentence templates and stray Chinese, and its
+  sampling alone moved hard rank 1 by up to five of 30 queries between runs. Without it, hard
+  `hybrid_r1` reads 0.3778 (0.2611–0.3611 with expansion), `full_r1` holds at 0.5111, the easy
+  queries lose nothing, and the median hybrid query drops from about 1 s to 20 ms. The Korean
+  queries now return the same results on every run. English-only queries are still expanded, and a
+  caller who wants variants for a Korean question can pass its own `searches`
+  (`BASELINE.md` 2026-09-26).
+
+- `scripts/bench-ko.sh` benches the seam form by default. Every one-line goldset query reaches
+  `qmd bench` as `lex: <q>` + `vec: <q>`, the shape REST/MCP callers send, which skips LLM query
+  expansion. `KO_FORM=plain` keeps the old path, and the `RESULT-HARD` line now ends with
+  `form=<seam|plain>`. The `dogfood.sh` gate refuses a bench whose form differs from its baseline
+  line's form. On the seam form it gates hard `hybrid_r1` as well as `full_r1`, against a reference
+  line with `tol=0`, since the seam form gave the same lines in every run. The reason is the plain
+  form's hard wobble, which turned out to be the expansion sampler alone. Replaying recorded
+  expansions reproduces every query's top 10, and every query that moved between same-commit runs
+  had drawn a different expansion. The draw moved `hybrid_r1` by up to five of 30 queries and once
+  put `full_r1` 0.0007 above the old gate's floor (`BASELINE.md` 2026-09-26). The gate also runs
+  the bench with any exported `KO_BENCH`/`KO_CORPUS`/`KO_FORM` removed, and `bench-ko.sh` refuses a
+  `KO_BENCH` inside `tmp/bench-ko/`, where the run clears its outputs.
+
+- `scripts/bench-ko.sh` takes `KO_CORPUS` (a corpus directory holding `wiki/` and
+  `distractors/`) and `KO_BENCH` (a goldset) so an A/B runs through the same isolated harness
+  and stdout lines. Overrides resolve against the caller's cwd and are checked before the
+  previous run's output is removed. The first use settled a question: three everyday-language
+  question lines per document, decoys included, did not move hard rank 1 on the REST-shaped
+  goldset (hybrid_r1 0.3111 → 0.2778, full_r1 flat at 0.5111; hard hit@5 up 2 of 30). Index-time
+  summaries are therefore not being built. The same runs point the `hybrid_r1` wobble at LLM
+  query expansion rather than re-embedding (`BASELINE.md` 2026-09-25).
+- ko-vault goldset: 69 near-topic distractor documents and 30 hard queries (`sem-hard` /
+  `multi` / `neg`) join the fixture; `bench-ko.sh` indexes both directories (97 files) and
+  prints a `RESULT-HARD hybrid_r1=… full_r1=… n=30` line next to `RESULT`. The `dogfood.sh`
+  gate now also fails when the hard `full_r1` falls below the baseline minus the `tol=` recorded
+  on that baseline line, and fails closed when `RESULT-HARD` or its `tol=` goes missing.
+  `full_r1` is gated rather than `hybrid_r1` because it held one value across eight same-commit
+  runs, while `hybrid_r1` moved by about 3.5 queries of 30.
+  Fixture invariants (no title-term leakage, no excluded concept among expected files, no basename
+  collisions) are enforced by `test/ko-bench-fixture.test.ts`.
+
+- Collection- or filter-scoped vector search is faster on indexes of up to
+  4096 vectors. It now post-filters one sqlite-vec KNN over the whole table,
+  which is exact because vec0 KNN is a brute-force scan, instead of computing
+  `vec_distance_cosine` row by row. On the KB goldset (1047 of 3389 vectors,
+  M3 Max) the vector stage dropped from 250–1300ms to about 25ms, and a cold
+  capped query (cap 128, candidateLimit 15) went from p90 2000ms to 906ms with
+  identical rankings. Larger indexes keep the exact scan.
+
+## [2.8.3-ko.5] - 2026-09-24
+
+Rerank gets an input cap. Truncating each document to 128 tokens keeps KB
+goldset quality (Hit@5 0.97 / Recall@5 0.77) while cutting cold rerank p90 from
+3.6s to 0.9s on an M3 Max. The cap can be set daemon-wide
+(`QMD_RERANK_MAX_DOC_TOKENS`) or per request (`rerankMaxDocTokens` on REST
+`/query` and SDK `search()`). The README is now a Korean guide to ko-qmd itself,
+and the English reference lives in `docs/REFERENCE.md`.
+
+### Changes
+
+- README.md is rewritten in Korean around ko-qmd itself: what differs from
+  upstream, install, quick start, MCP/REST/SDK, environment variables and the
+  maintenance guide that used to live in `README.ko.md` (now removed). The
+  upstream-derived English reference (full CLI, MCP tool parameters, SDK API,
+  score fusion) moved to `docs/REFERENCE.md`, with install and import names
+  switched to `ko-qmd`.
+- REST `POST /query` and SDK `search()` take `rerankMaxDocTokens`, a
+  per-request rerank doc token cap. For that request it overrides the daemon's
+  `QMD_RERANK_MAX_DOC_TOKENS`, and it never raises the context budget. A caller
+  like the KB seam can then turn a capped rerank on for one machine without
+  changing the daemon's env. The rerank cache key follows the effective cap
+  (request, else env). Invalid values (non-integer, ≤ 0, strings) are ignored.
+- `QMD_RERANK_MAX_DOC_TOKENS` caps the tokens of each document sent to the
+  reranker, below the context budget. Unset keeps today's behaviour. On the KB
+  goldset (40 queries, candidateLimit 15), a cap of 128 keeps quality at
+  Hit@5 0.97 / Recall@5 0.77 against 0.95 / 0.76 uncapped. It cuts cold p90 from
+  3623ms to 897ms on an M3 Max and from 4720ms to 2192ms on another Apple Silicon
+  Mac. The cap is part of the rerank cache key, so capped and uncapped scores
+  never mix. Listed in `qmd doctor`.
+- `scripts/dogfood.sh` warms the embedding model with one vec query after
+  every daemon restart (deploy and `--restore`). A fresh daemon's first vec
+  query took 15.5s, so the first KB seam search after a restart hit its 20s
+  limit and fell back to rg. A failed warm-up is a warning, not a failed restart.
+- `scripts/dogfood.sh` waits for the *restarted* daemon. `launchctl kickstart -k`
+  returns while the old process is still shutting down and answering `/health`,
+  so the restart looked done ~9s before the new process listened. A `/health`
+  whose `uptime` predates the kickstart is now treated as the old process.
+
 ## [2.8.3-ko.4] - 2026-09-24
 
 Korean documents are now chunked at their real token budget, and Hangul
@@ -49,7 +195,7 @@ it to `scope`.
 - Plain Hangul lex terms that spell a common technical loanword (`서치`, `웹`,
   `코어`, `마이그레이션`, about 80 entries in a built-in table) also match the
   Latin spelling, so `하이브리드 서치` finds a note that only says
-  `hybrid-search` and `레몬 웹 코어` reaches `lemon-web-core`. The lookup runs
+  `hybrid-search` and `리액트 웹 코어` reaches `react-web-core`. The lookup runs
   on particle-stripped stems too (`서치를`), single-syllable entries (`웹`, `훅`)
   are bridged when they stand alone (not `웹을`), and terms with no entry produce the same FTS5 query as before.
   Query-side only; no re-index. On the ko-vault bench the eleven new loanword

package/README.en.md

@@ -0,0 +1,184 @@
+# ko-qmd — local hybrid search for Korean Markdown notes
+
+[![npm](https://img.shields.io/npm/v/ko-qmd?label=npm%20ko-qmd)](https://www.npmjs.com/package/ko-qmd)
+[![license: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+
+[한국어](README.md) · **English**
+
+**ko-qmd is an on-device search engine for Korean Markdown notes — Obsidian vaults, wikis, meeting
+notes and docs folders — that finds them by keyword or by natural-language question.**
+It combines BM25 full-text search, vector search and LLM reranking, and matches Korean query words
+by their stem even when a particle or verb ending is attached. Every model runs locally through
+[node-llama-cpp](https://github.com/withcatai/node-llama-cpp) (GGUF, no network calls), so your notes
+never leave the machine. AI agents such as Claude Code and Claude Desktop use it through the CLI, an
+MCP server, a REST endpoint or the TypeScript SDK.
+It is the Korean distribution of [tobi/qmd](https://github.com/tobi/qmd) (MIT).
+
+- Package: `ko-qmd` on npm. The binary and MCP server keep the upstream name, `qmd`
+- Version: `<upstream version>-ko.N` (e.g. `2.8.3-ko.5`)
+- Full reference (every CLI option, MCP tool parameters, SDK API, scoring): [docs/REFERENCE.md](docs/REFERENCE.md)
+- Changes: [CHANGELOG.md](CHANGELOG.md) · [FAQ](#faq)
+
+```mermaid
+flowchart LR
+  Q[Query] --> H[Korean handling<br/>particle/ending strip · loanword bridge]
+  H --> FTS[BM25<br/>FTS5 + syllable bigrams]
+  Q --> VS[Vector search<br/>Qwen3-Embedding]
+  FTS --> RRF[RRF fusion]
+  VS --> RRF
+  RRF --> RR[Rerank<br/>Qwen3-Reranker]
+  RR --> OUT[Results]
+```
+
+## What differs from upstream qmd
+
+### Korean keyword search (lex)
+
+- **Particle and ending stripping** — an unquoted Hangul word also matches its stem: one trailing particle
+  (`검색을` → `검색`), a two-particle chain (`청킹에서의` → `청킹`), verb and nominalizing endings
+  (`토큰화하는` → `토큰화`, `검색하기` → `검색`), `한`/`된` endings (`필요한` → `필요`) and the 하↔해
+  contraction. Quoted phrases, Hanja and kana are left as written.
+- **Loanword bridge** — about 80 built-in Hangul spellings of technical terms (`서치`, `웹`, `코어`,
+  `마이그레이션`, …) also search their English spelling, so `하이브리드 서치` finds a `hybrid-search` note.
+  Only the query changes; no re-index is needed.
+- **Mixed-script token split** — `SKILL.md계약의핵심` becomes `skill` AND `md` AND `계약의핵심`.
+- **Long-query relaxation** — a query of three or more words that matches nothing with AND is retried with
+  OR, so a natural-language sentence never returns zero hits.
+- **Syllable-bigram index** — Hangul runs are indexed as syllable bigrams too, so a word inside a
+  compound written without spaces still matches.
+- **Negation strip** — "X 말고 Y" ("Y, not X") searches for Y alone.
+
+### Models and chunking
+
+| Role | Default model | Upstream |
+|---|---|---|
+| Embedding | `Qwen3-Embedding-0.6B` (Q8_0) | embeddinggemma-300M |
+| Rerank | `Qwen3-Reranker-0.6B` (Q8_0) | same |
+| Query expansion | `qmd-query-expansion-1.7B` | same |
+
+- Korean documents are chunked by their real token budget: Hangul costs far more tokens per character
+  than English (measured on Qwen3-Embedding: 1.64 chars/token for Korean, 6.08 for English). English
+  chunk boundaries match upstream.
+- Hybrid RRF weights are tuned for Korean, and a query containing Hangul skips LLM query expansion on
+  the hybrid path (English queries are still expanded).
+- The rerank blend lets the reranker actually change the top result, and the reranker is shown the
+  chunk that best matches the query's Hangul syllable bigrams, led by the note title.
+
+### Scope, tools and daemon
+
+- **Path and time scope** — CLI `--path` / `--since` / `--until`, the same fields over REST and MCP,
+  `scope` in the SDK. Ranking happens inside the filtered corpus.
+- **`qmd grep`** — exact string or regex match over indexed bodies, for spellings the tokenizer never
+  produced.
+- **Daemon** — re-indexes changed files before a search (once per collection every 30 s), optional query
+  log (`QMD_QUERY_LOG=1`), raw `vec_score` / `fts_score` with `rerank:false`, and a per-document
+  rerank token cap (`QMD_RERANK_MAX_DOC_TOKENS`, or `rerankMaxDocTokens` per request).
+
+## Install
+
+```sh
+npm install -g ko-qmd            # or pin: ko-qmd@2.8.3-ko.5
+```
+
+- It cannot sit beside upstream `@tobilu/qmd` (same binary name): `npm uninstall -g @tobilu/qmd` first.
+- Requires Node.js 22+ or Bun; on macOS, Homebrew SQLite (for extension loading). Models download to
+  `~/.cache/qmd/models/` on first use.
+
+## Quick start
+
+```sh
+qmd collection add ~/notes --name notes          # register and index a folder
+qmd context add qmd://notes "Personal dev notes" # description returned with results
+qmd embed                                         # vector embeddings (first time, then changes only)
+
+qmd query "하이브리드 서치 설계 결정"               # fusion + rerank (recommended)
+qmd search "리랭크"                               # BM25 only (fast, no model)
+qmd vsearch "검색 지연을 줄인 방법"                 # vector only
+qmd grep "QMD_QUERY_LOG"                          # exact body match
+qmd get "#abc123"                                 # fetch by docid
+```
+
+Common options: `-c <collection>` · `-n <count>` · `--min-score <score>` · `--full` ·
+`--intent "<what you are after>"` · `--no-rerank` · `--format json|csv|md|xml|files`.
+
+## Use from AI agents
+
+```sh
+qmd mcp                          # stdio MCP server (Claude Desktop, Claude Code, …)
+qmd mcp --http                   # HTTP on port 8181 — models load once, sessions share them
+qmd mcp --http --daemon          # background; stop with qmd mcp stop
+```
+
+The HTTP daemon serves `POST /mcp` (MCP Streamable HTTP), `POST /query` (REST) and `GET /health`.
+
+```sh
+curl -s http://127.0.0.1:8181/query -H 'Content-Type: application/json' -d '{
+  "searches": [{"type": "vec", "query": "리랭크 지연"}, {"type": "lex", "query": "리랭크 지연"}],
+  "collections": ["notes"], "limit": 10
+}'
+```
+
+```ts
+const { createStore } = await import('ko-qmd')   // ESM only — require() does not work
+
+const store = await createStore({ dbPath: './index.sqlite', configPath: `${process.env.HOME}/.config/qmd/index.yml` })
+const results = await store.search({
+  queries: [{ type: 'vec', query: '리랭크 지연' }, { type: 'lex', query: '리랭크 지연' }],
+  collection: 'notes', limit: 10,
+})
+await store.close()
+```
+
+Tool list, request fields and SDK API: [docs/REFERENCE.md](docs/REFERENCE.md). Search guidance for
+agents, including a section on Korean queries: [skills/qmd/SKILL.md](skills/qmd/SKILL.md).
+
+## Benchmark
+
+`bash scripts/bench-ko.sh` runs the `test/fixtures/ko-vault/` fixture (97 documents, 93 queries).
+On its first 52 queries, bm25 recall@5 went from 0.6250 on upstream 2.8.3 to 0.9519 in this
+distribution. Every run's numbers are in [BASELINE.md](test/fixtures/ko-vault/BASELINE.md).
+
+## FAQ
+
+### What is ko-qmd?
+A CLI, MCP server and SDK that search Korean Markdown notes on your own machine. It adds Korean
+particle and ending handling, a Hangul syllable-bigram index, a Korean-friendly default embedding
+model (Qwen3-Embedding) and tuned fusion weights on top of upstream qmd.
+
+### Why not use upstream qmd for Korean?
+Upstream BM25 treats a word with a particle attached (`검색을`, `청킹에서의`) as a different token from
+its stem, so it misses notes that say `검색`. On the ko-vault benchmark (52 queries) bm25 recall@5 is
+0.6250 upstream and 0.9519 here.
+
+### Can it search an Obsidian vault?
+Yes. Register the vault folder with `qmd collection add <vault path> --name <name>` and it indexes the
+`*.md` files inside. It only reads your files.
+
+### Does it need internet access or an API key?
+No. The embedding, rerank and query-expansion models download once on first use and run offline after
+that.
+
+### How do I connect it to Claude Code or another agent?
+Start the MCP server with `qmd mcp` (stdio) or `qmd mcp --http --daemon` (HTTP, shared models). See
+[Use from AI agents](#use-from-ai-agents).
+
+### Does it work for English notes too?
+Yes. English queries and documents behave as upstream: same chunk boundaries, same query expansion.
+Korean and English notes can live in one index.
+
+## Development
+
+```sh
+bun install
+bun src/cli/qmd.ts <command>     # run from source
+bun run lint && bun run test:types
+npx vitest run --reporter=verbose test/
+npm run build                    # dist/ — never `bun build --compile` (it breaks sqlite-vec)
+```
+
+`develop` is the working branch (PRs go there); `main` is the release branch. Release, dogfood and
+upstream-sync procedures are documented in Korean in [README.md](README.md#개발).
+
+## License
+
+MIT. Original copyright Tobi Lutke ([LICENSE](LICENSE)); this distribution uses the same license.