ko-qmd 2.8.3-ko.3 → 2.8.3-ko.5

package/CHANGELOG.md

@@ -2,6 +2,174 @@
 
 ## [Unreleased]
 
+## [2.8.3-ko.5] - 2026-09-24
+
+Rerank gets an input cap. Truncating each document to 128 tokens keeps KB
+goldset quality (Hit@5 0.97 / Recall@5 0.77) while cutting cold rerank p90 from
+3.6s to 0.9s on an M3 Max. The cap can be set daemon-wide
+(`QMD_RERANK_MAX_DOC_TOKENS`) or per request (`rerankMaxDocTokens` on REST
+`/query` and SDK `search()`). The README is now a Korean guide to ko-qmd itself,
+and the English reference lives in `docs/REFERENCE.md`.
+
+### Changes
+
+- README.md is rewritten in Korean around ko-qmd itself: what differs from
+  upstream, install, quick start, MCP/REST/SDK, environment variables and the
+  maintenance guide that used to live in `README.ko.md` (now removed). The
+  upstream-derived English reference (full CLI, MCP tool parameters, SDK API,
+  score fusion) moved to `docs/REFERENCE.md`, with install and import names
+  switched to `ko-qmd`.
+- REST `POST /query` and SDK `search()` take `rerankMaxDocTokens`, a
+  per-request rerank doc token cap. For that request it overrides the daemon's
+  `QMD_RERANK_MAX_DOC_TOKENS`, and it never raises the context budget. A caller
+  like the KB seam can then turn a capped rerank on for one machine without
+  changing the daemon's env. The rerank cache key follows the effective cap
+  (request, else env). Invalid values (non-integer, ≤ 0, strings) are ignored.
+- `QMD_RERANK_MAX_DOC_TOKENS` caps the tokens of each document sent to the
+  reranker, below the context budget. Unset keeps today's behaviour. On the KB
+  goldset (40 queries, candidateLimit 15), a cap of 128 keeps quality at
+  Hit@5 0.97 / Recall@5 0.77 against 0.95 / 0.76 uncapped. It cuts cold p90 from
+  3623ms to 897ms on an M3 Max and from 4720ms to 2192ms on another Apple Silicon
+  Mac. The cap is part of the rerank cache key, so capped and uncapped scores
+  never mix. Listed in `qmd doctor`.
+- `scripts/dogfood.sh` warms the embedding model with one vec query after
+  every daemon restart (deploy and `--restore`). A fresh daemon's first vec
+  query took 15.5s, so the first KB seam search after a restart hit its 20s
+  limit and fell back to rg. A failed warm-up is a warning, not a failed restart.
+- `scripts/dogfood.sh` waits for the *restarted* daemon. `launchctl kickstart -k`
+  returns while the old process is still shutting down and answering `/health`,
+  so the restart looked done ~9s before the new process listened. A `/health`
+  whose `uptime` predates the kickstart is now treated as the old process.
+
+## [2.8.3-ko.4] - 2026-09-24
+
+Korean documents are now chunked at their real token budget, and Hangul
+spellings of technical loanwords (`서치`, `웹 코어`) reach notes written in Latin
+script. This release also syncs tobi/qmd up to `04e4dbd` — metadata filtering,
+the oxlint fence and the REST JSON body guard — and adds `qmd grep`, `path`/
+`since`/`until` scoping on MCP and REST, and a daemon that keeps the text index
+fresh on its own. Existing indexes need one `qmd embed` run to pick up the new
+chunking, and SDK callers that passed `filter` for path/time scoping must rename
+it to `scope`.
+
+### Breaking
+
+- SDK: ko-qmd's path/time narrowing option is now `scope` (was `filter`) on
+  `search()`, `searchLex()` and `searchVector()`, and the trailing `scope`
+  parameter of `searchFTS`/`searchVec`. `filter` is upstream's metadata filter.
+  CLI `--path/--since/--until` and REST/MCP `path/since/until` are unchanged,
+  and a request may pass both — a result has to satisfy each.
+
+### Upstream sync
+
+- Merged tobi/qmd `main` up to `04e4dbd` (24 commits past v2.8.3): oxlint
+  fence (`bun run lint` is oxlint; the tsc check is `bun run test:types`),
+  metadata filtering, separator-aware FTS terms (quoted `"PIO-1384"` now
+  matches), REST JSON body guard, Windows index names, unreadable-gguf
+  diagnostics and the store-local embedding tokenizer. ko-qmd behaviour is
+  kept: the Hangul query branch runs ahead of the separator split, a metadata
+  filter gets the same anti-starvation scan as `--path`, and bm25 on the
+  ko-vault bench is identical per query.
+
+### Added
+
+- Added Oxlint lint fence.
+- Document metadata and metadata filtering. Markdown documents can opt into typed metadata through a namespaced frontmatter block (`qmd.metadata` with strings, numbers, booleans, or flat homogeneous arrays), and every search surface — CLI `search`/`vsearch`/`query` via `--filter <json>`, the SDK's `filter` option on `search()`/`searchLex()`/`searchVector()`, the MCP `query` tool, and HTTP `POST /query` and `/search` — accepts one shared recursive filter AST discriminated by `operator`: `and`/`or`/`not` logical groups, `eq`/`ne`/`gt`/`gte`/`lt`/`lte` comparisons, `in`/`nin`/`all` membership, and `exists` presence. Every returned result satisfies the filter (applied before RRF fusion and reranking); like collection filtering, highly selective filters remain best-effort for top-K completeness. Frontmatter stays ordinary searchable content — no chunking, embedding, snippet, or line-number changes — and documents without `qmd.metadata` behave exactly as before. JSON/SDK/MCP/HTTP results now include each document's indexed metadata, and `qmd status` reports how many documents still need metadata extraction (a normal `qmd update` backfills existing indexes).
+
+### Fixed
+
+- Embedding generation and legacy fingerprint adoption now tokenize documents
+  with the store-selected embedding model instead of the global default. This
+  keeps chunk boundaries aligned with the model that creates and verifies the
+  stored vectors without initializing an unrelated provider.
+
+### Changes
+
+- Plain Hangul lex terms that spell a common technical loanword (`서치`, `웹`,
+  `코어`, `마이그레이션`, about 80 entries in a built-in table) also match the
+  Latin spelling, so `하이브리드 서치` finds a note that only says
+  `hybrid-search` and `레몬 웹 코어` reaches `lemon-web-core`. The lookup runs
+  on particle-stripped stems too (`서치를`), single-syllable entries (`웹`, `훅`)
+  are bridged when they stand alone (not `웹을`), and terms with no entry produce the same FTS5 query as before.
+  Query-side only; no re-index. On the ko-vault bench the eleven new loanword
+  queries go from bm25_r5 0.0000 to 1.0000 while the original 52 hold at
+  0.9519 per query.
+- REST `POST /query` and the MCP `query` tool attach `vec_score` (cosine
+  similarity) and `fts_score` (normalized BM25) to each result on the
+  `rerank:false` path, where `score` is the 1/rank fusion position and cannot
+  be thresholded. Each is the max over the sub-queries that returned the hit
+  and is omitted, never 0, when that backend did not return it. The daemon
+  query log (`QMD_QUERY_LOG`) records the same two fields per result; the row
+  schema stays `v: 1` and `rerank:true` responses are unchanged.
+- Korean documents are chunked at their real token budget instead of the
+  English-tuned 3.0 chars/token: `estimateCharsPerToken` blends CJK and other
+  scripts as a harmonic mean (measured ko 1.64 / en 6.08 chars per token on
+  Qwen3-Embedding-0.6B-Q8_0, 2026-09-21), and indexing plus both search-path
+  best-chunk selections size chunks with it. A 20KB Korean document now
+  first-passes into ~900-token chunks instead of being split ~855 and Korean
+  rerank chunks shrink from ~3600 to ~1440 chars. Non-CJK text keeps the ratio
+  each call site already used — 3.0 when indexing, 4.0 on the search paths — so
+  English chunk boundaries do not move, and the ko-vault bench holds
+  (bm25_r5=0.9519).
+- Re-embedding a document whose chunk count shrank now prunes its stale
+  higher-`seq` vectors before inserting, so one `qmd embed` run converges
+  instead of waiting for a second run. The embedding fingerprint gains a
+  `chunk_chars_estimator` term, so `qmd status` reports re-embedding as needed
+  for existing indexes; run `qmd embed` once to pick up the new chunking.
+
+- `search`, `vsearch` and `query` take `--path`, `--since` and `--until`, so a
+  search can be narrowed to part of a collection or to what changed recently.
+  `--path` matches globs against `collection/path` and is repeatable, with a
+  `!` prefix to exclude; `--since` / `--until` take a span (`7d`) or a date.
+  The filter narrows the corpus rather than the result list — excluded
+  documents are gone before ranking, on the vector path as well as BM25, so a
+  narrow filter returns its own best matches instead of whatever survived the
+  global top-K. A span that does not parse is an error, and a filter that
+  leaves nothing to search says so instead of looking like "no match".
+
+- `qmd query --explain` now lists every expansion sub-query that placed the
+  document, with that sub-query's own rank, contribution, backend score and the
+  text it ran. It previously showed the top three contributions collapsed onto
+  one line without the query text, which could not distinguish a document that
+  ranked first in one sub-query and nowhere else from one that ranked middling
+  everywhere — two situations that call for different fixes.
+
+- `qmd grep <pattern>` matches document bodies exactly, by string or regular
+  expression, with no ranking and no LLM: every matching line, grouped by file,
+  with line numbers that `qmd get <file>:<n>:<count>` accepts. It is the escape
+  hatch for a query that finds nothing because tokenization never produced the
+  term the document contains — the failure this fork keeps hitting with Korean
+  text — and for finding an identifier or error string verbatim. It searches
+  exactly what the index holds and honours collection scoping, `--path`,
+  `--since` and `--until`. Smart case: an uppercase letter in the pattern makes
+  it case-sensitive, and Hangul has no case so Korean patterns are unaffected.
+  `-F` searches for a literal string. An MCP `grep` tool exposes the same thing
+  to agents, which is the reason it exists: a client whose only tools are
+  `query`, `get` and `multi_get` has no other way to match a string exactly.
+
+- The MCP `query` tool and the REST `/query` endpoint take the same `path`,
+  `since` and `until` parameters, so an agent can scope a search to a folder or
+  to recent changes without shelling out. A span that does not parse comes back
+  as an error rather than an unfiltered search, and an empty result under a
+  filter reports how many documents the filter left — an agent reading a bare
+  "No results" would otherwise conclude the corpus has nothing on the topic.
+
+- The HTTP and stdio MCP daemon keeps the text index current on its own. Before
+  each search it checks the collections that search covers — at most once every
+  30 seconds each — and re-indexes the ones whose files changed, so a note
+  written while the daemon runs is findable on the next query without `qmd
+  update`. It updates keyword search only; vectors still wait for `qmd embed`.
+  It does not run a collection's `update:` hook, does not clear the LLM cache
+  the way `qmd update` does, and does not touch a collection whose folder has
+  vanished or emptied, since reindexing would deactivate every document in it.
+
+- The daemon's query log now covers the MCP `query` tool, not just the REST
+  endpoints. A search an agent runs through MCP over HTTP lands in the same
+  `queries-YYYY-MM.jsonl` with `via: "mcp"`, the same result path spelling and
+  the same `X-QMD-*` header handling, so a dogfooding session's real traffic is
+  in one file. A plain `query` argument is recorded as one search of type
+  `auto`. stdio MCP calls stay unlogged — they carry no request to annotate.
+
 ## [2.8.3-ko.3] - 2026-09-18
 
 The reranker now decides which document ranks first. The blend's retrieval-position