kernel-lore-mcp 0.3.0__tar.gz → 0.3.2__tar.gz

{kernel_lore_mcp-0.3.0 → kernel_lore_mcp-0.3.2}/CHANGELOG.md

@@ -10,6 +10,75 @@ release tags move them into a dated section. Release process in
 
 ## [Unreleased]
 
+## [0.3.2] — 2026-04-22
+
+### Added
+
+- Added `lore_fix_status`, a bug-centric fix-correlation tool that
+  follows `Reported-by:`, `Link:`, and `Closes:` evidence across
+  separate lore threads and upgrades to an authoritative `merged`
+  verdict when the git sidecar can confirm it.
+- Added a normalized trailer-reference side index in `over.db` for
+  `reported_by`, `fixes`, `link`, and `closes`, plus backfill support
+  and reader-side exact lookups for syzbot hashes, lore message-ids,
+  SHA prefixes, lowercased URLs, and normalized trailer payloads.
+- Added human-friendly `since` / `until` parsing for MCP tools and the
+  Rust query router: raw nanoseconds remain supported, but callers can
+  now also pass ISO dates, RFC3339 timestamps, and relative windows
+  such as `90d`.
+
+### Changed
+
+- `lore_activity`, `lore_author_profile`, `lore_file_timeline`,
+  `lore_path_mentions`, `lore_eq`, `lore_in_list`, `lore_count`,
+  `lore_substr_subject`, `lore_substr_trailers`, and `lore_regex` now
+  accept both legacy `*_unix_ns` bounds and new human-readable
+  `since` / `until` fields, with shared validation.
+- `lore_search` query parsing now understands `since:` and `until:`
+  bounds in human formats instead of requiring raw integer nanoseconds
+  only.
+
+### Fixed
+
+- Upper-bound date filtering is now carried consistently through the
+  Rust reader, over.db index helpers, PyO3 boundary, and MCP tool
+  layer instead of being post-filtered inconsistently or silently
+  ignored on some paths.
+
+## [0.3.1] — 2026-04-22
+
+### Added
+
+- `/status` and `kernel-lore-mcp status` now surface live writer state:
+  `writer_lock_present`, `sync_active`, and a `sync` block populated
+  from `kernel-lore-sync`'s machine-readable heartbeat file.
+- Added `kernel_lore_mcp_writer_lock_present` and
+  `kernel_lore_mcp_sync_active` gauges to `/metrics`.
+- Added `scripts/bench/stress_mcp_multiprocess.py`, the async +
+  multiprocess soak harness used to pressure-test a hosted box over a
+  longer run.
+
+### Changed
+
+- `kernel-lore-sync` now writes `state/sync.json` while it runs and
+  emits explicit operator-facing stage logs for `bm25_commit`,
+  `tid_rebuild`, `path_vocab_rebuild`, `generation_bump`, and
+  `save_manifest`.
+- Inline BM25 uses conservative explicit writer defaults instead of
+  Tantivy's auto-thread selection. New env knobs:
+  `KLMCP_BM25_WRITER_THREADS` and `KLMCP_BM25_WRITER_MEMORY_MB`.
+- `rate_limited` and `query_timeout` now return load-aware in-band
+  retry hints. Retry guidance rises when a writer-heavy sync stage is
+  active on the same box.
+
+### Fixed
+
+- Writer activity detection no longer confuses “lockfile exists” with
+  “writer lock is actively held”; status probes now test the live flock
+  state.
+- The “ingest phase done, then silence for minutes” sync log shape is
+  gone; BM25 finalization is visible instead of looking hung.
+
 ## [0.3.0] — 2026-04-21
 
 ### Added

{kernel_lore_mcp-0.3.0 → kernel_lore_mcp-0.3.2}/Cargo.lock

@@ -2638,7 +2638,7 @@ dependencies = [
 
 [[package]]
 name = "kernel_lore_mcp"
-version = "0.3.0"
+version = "0.3.2"
 dependencies = [
  "aho-corasick",
  "anyhow",

{kernel_lore_mcp-0.3.0 → kernel_lore_mcp-0.3.2}/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "kernel_lore_mcp"
-version = "0.3.0"
+version = "0.3.2"
 edition = "2024"
 rust-version = "1.85"
 authors = ["Michael Bommarito <michael@bommaritollc.com>"]

{kernel_lore_mcp-0.3.0 → kernel_lore_mcp-0.3.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: kernel-lore-mcp
-Version: 0.3.0
+Version: 0.3.2
 Classifier: Development Status :: 2 - Pre-Alpha
 Classifier: Intended Audience :: Developers
 Classifier: License :: OSI Approved :: MIT License
@@ -78,26 +78,24 @@ kernel-lore-sync \
     --include '/wireguard/*' --include '/linux-cifs/*'
 # Drop --include to mirror all ~390 lists. Plan the disk + time.
 
-# 3. (optional, recommended) build the path-mention index. Tiny, fast.
-python -c 'from kernel_lore_mcp import _core; \
-           print(_core.rebuild_path_vocab("'"$KLMCP_DATA_DIR"'"))'
-
-# 4. confirm freshness + which capabilities are provisioned
+# 3. confirm freshness + which capabilities are provisioned
 kernel-lore-mcp status --data-dir "$KLMCP_DATA_DIR"
 # Look at `capabilities`: each over_db / bm25 / path_vocab / embedding /
 # maintainers / git_sidecar boolean tells you which tools will actually
-# return data on this deployment.
+# return data on this deployment. While a sync is active, the same
+# status output also shows `writer_lock_present`, `sync_active`, and
+# the current sync stage.
 
-# 4b. inspect shard/index health; add --heal to repair unborn shard HEADs
+# 3b. inspect shard/index health; add --heal to repair unborn shard HEADs
 #     and remove unrecoverable shard repos so the next sync reclones them
 kernel-lore-doctor --data-dir "$KLMCP_DATA_DIR"
 
-# 5. verify the MCP surface — zero API cost
+# 4. verify the MCP surface — zero API cost
 git clone --depth 1 https://github.com/mjbommar/kernel-lore-mcp.git
 cd kernel-lore-mcp && ./scripts/agentic_smoke.sh local
 # PASS: 7/7 tools, 5/5 resource templates, 5/5 prompts (the
 # `REQUIRED_*` subset from src/kernel_lore_mcp/_surface_manifest.py;
-# the live server registers 24 tools in total).
+# the live server registers 25 tools in total).
 ```
 
 Then pick your agent and copy its snippet from
@@ -148,15 +146,14 @@ Want production-grade systemd deployment (single `klmcp-sync.timer`
 replacing the pre-v0.2.0 grokmirror + ingest pair)?
 [`docs/ops/runbook.md`](./docs/ops/runbook.md) §1 onwards.
 
-## Status — v0.3.0 (2026-04-21)
+## Status — v0.3.2 (2026-04-22)
 
-Current release: the hosted-readiness line that would previously have
-been cut as `0.2.3` now lands in `v0.3.0`. The main additions on top of
-`0.2.2` are: sync self-healing for poisoned shard repos, hosted-mode
-regex gating, generation-bound `lore_corpus_stats` caching, automatic
-path-vocab rebuild during sync, explicit `local`/`hosted` deployment
-profiles, structured slow-path profiling logs, and a repeatable
-HTTP/MCP adversarial-load harness plus public-launch checklist.
+Current release: `v0.3.2`, the follow-on patch after the hosted
+readiness and same-box sync hardening line. The focus is better bug
+workflow ergonomics and safer query scoping: a first-class
+`lore_fix_status` tool, indexed trailer-reference correlation for
+syzbot / lore / `Fixes:` joins, and human-readable `since` / `until`
+bounds across both tools and `lore_search`.
 
 Shipped:
 
@@ -168,7 +165,7 @@ Shipped:
   manifest fetch, gix smart-HTTP clone-or-fetch (rayon-fanned
   across shards), ingest, tid rebuild, generation bump — all
   under one writer lock so there's no trigger/debounce race.
-- Full MCP surface: **24 tools** (search, primitives, sampling-
+- Full MCP surface: **25 tools** (search, primitives, sampling-
   backed summarize/classify/explain, authoritative `merged` /
   `picked_up` verdicts via git-sidecar, `lore_corpus_stats` for
   coverage transparency, `lore_author_footprint` for address-

{kernel_lore_mcp-0.3.0 → kernel_lore_mcp-0.3.2}/README.md

@@ -43,26 +43,24 @@ kernel-lore-sync \
     --include '/wireguard/*' --include '/linux-cifs/*'
 # Drop --include to mirror all ~390 lists. Plan the disk + time.
 
-# 3. (optional, recommended) build the path-mention index. Tiny, fast.
-python -c 'from kernel_lore_mcp import _core; \
-           print(_core.rebuild_path_vocab("'"$KLMCP_DATA_DIR"'"))'
-
-# 4. confirm freshness + which capabilities are provisioned
+# 3. confirm freshness + which capabilities are provisioned
 kernel-lore-mcp status --data-dir "$KLMCP_DATA_DIR"
 # Look at `capabilities`: each over_db / bm25 / path_vocab / embedding /
 # maintainers / git_sidecar boolean tells you which tools will actually
-# return data on this deployment.
+# return data on this deployment. While a sync is active, the same
+# status output also shows `writer_lock_present`, `sync_active`, and
+# the current sync stage.
 
-# 4b. inspect shard/index health; add --heal to repair unborn shard HEADs
+# 3b. inspect shard/index health; add --heal to repair unborn shard HEADs
 #     and remove unrecoverable shard repos so the next sync reclones them
 kernel-lore-doctor --data-dir "$KLMCP_DATA_DIR"
 
-# 5. verify the MCP surface — zero API cost
+# 4. verify the MCP surface — zero API cost
 git clone --depth 1 https://github.com/mjbommar/kernel-lore-mcp.git
 cd kernel-lore-mcp && ./scripts/agentic_smoke.sh local
 # PASS: 7/7 tools, 5/5 resource templates, 5/5 prompts (the
 # `REQUIRED_*` subset from src/kernel_lore_mcp/_surface_manifest.py;
-# the live server registers 24 tools in total).
+# the live server registers 25 tools in total).
 ```
 
 Then pick your agent and copy its snippet from
@@ -113,15 +111,14 @@ Want production-grade systemd deployment (single `klmcp-sync.timer`
 replacing the pre-v0.2.0 grokmirror + ingest pair)?
 [`docs/ops/runbook.md`](./docs/ops/runbook.md) §1 onwards.
 
-## Status — v0.3.0 (2026-04-21)
+## Status — v0.3.2 (2026-04-22)
 
-Current release: the hosted-readiness line that would previously have
-been cut as `0.2.3` now lands in `v0.3.0`. The main additions on top of
-`0.2.2` are: sync self-healing for poisoned shard repos, hosted-mode
-regex gating, generation-bound `lore_corpus_stats` caching, automatic
-path-vocab rebuild during sync, explicit `local`/`hosted` deployment
-profiles, structured slow-path profiling logs, and a repeatable
-HTTP/MCP adversarial-load harness plus public-launch checklist.
+Current release: `v0.3.2`, the follow-on patch after the hosted
+readiness and same-box sync hardening line. The focus is better bug
+workflow ergonomics and safer query scoping: a first-class
+`lore_fix_status` tool, indexed trailer-reference correlation for
+syzbot / lore / `Fixes:` joins, and human-readable `since` / `until`
+bounds across both tools and `lore_search`.
 
 Shipped:
 
@@ -133,7 +130,7 @@ Shipped:
   manifest fetch, gix smart-HTTP clone-or-fetch (rayon-fanned
   across shards), ingest, tid rebuild, generation bump — all
   under one writer lock so there's no trigger/debounce race.
-- Full MCP surface: **24 tools** (search, primitives, sampling-
+- Full MCP surface: **25 tools** (search, primitives, sampling-
   backed summarize/classify/explain, authoritative `merged` /
   `picked_up` verdicts via git-sidecar, `lore_corpus_stats` for
   coverage transparency, `lore_author_footprint` for address-

{kernel_lore_mcp-0.3.0 → kernel_lore_mcp-0.3.2}/TODO.md

@@ -10,12 +10,41 @@ Status markers:
 - `[DEFER]` — decided to defer past v1; must have a `--> docs/...`
   pointer explaining why
 
-## Current release track (2026-04-21)
-
-We are working straight to `v0.3.0` now. There is **no standalone
-`0.2.3` release tag** planned, but the would-be `0.2.3` hardening
-line is still tracked separately here because it is the first block
-of work that must land before a public hosted launch.
+## Current release track (2026-04-22)
+
+`v0.3.0` shipped on 2026-04-21. The active patch line is
+`v0.3.2`: bug-workflow correlation and human time-bound ergonomics
+after the same-box hosted soak and sync-under-load hardening on
+`server6`.
+
+### 0.3.2 target — bug correlation + human time bounds
+
+- [x] Add a first-class `lore_fix_status` tool for bug-centric
+  correlation across separate lore threads, with authoritative
+  `merged` only when the git sidecar can confirm it.
+- [x] Materialize a normalized trailer-reference index for
+  `reported_by`, `fixes`, `link`, and `closes`, and expose exact
+  reader lookups over syzbot hashes, lore mids, SHA prefixes, and
+  normalized trailer values.
+- [x] Accept human-friendly `since` / `until` values across the
+  low-level tools and query router, while keeping raw
+  `*_unix_ns` compatibility for existing clients.
+
+### 0.3.1 target — sync-under-load visibility + safer inline BM25
+
+- [x] `kernel-lore-sync` writes live machine-readable progress under
+  `state/sync.json`, and `/status` + `kernel-lore-mcp status` surface
+  `writer_lock_present`, `sync_active`, and the current sync stage.
+- [x] Sync logs the post-ingest work explicitly:
+  `bm25_commit`, `tid_rebuild`, `path_vocab_rebuild`,
+  `generation_bump`, and `save_manifest` are no longer silent gaps.
+- [x] Inline BM25 is safer-by-default and operator-tunable:
+  default BM25 writer thread count is conservative, and env knobs exist
+  for thread count + memory budget.
+- [x] `rate_limited` / `query_timeout` retry hints are load-aware and
+  rise when a live writer-heavy sync stage is active.
+- [x] The async + multiprocess soak harness used against `server6`
+  lives in-tree and is referenced by the launch docs.
 
 ### 0.2.3 carry-forward — land these in `v0.3.0` first
 

{kernel_lore_mcp-0.3.0 → kernel_lore_mcp-0.3.2}/docs/ops/public-launch-checklist.md

@@ -61,6 +61,21 @@ Pass criteria:
   histogram summaries so we can distinguish queueing from tool-body
   work.
 
+For a longer soak on a live box, also run:
+
+```sh
+uv run python scripts/bench/stress_mcp_multiprocess.py \
+    --base-url http://127.0.0.1:8080 \
+    --scenario mixed_hot \
+    --processes 4 \
+    --concurrency-per-process 16 \
+    --duration-seconds 600 \
+    --json-out /tmp/klmcp-soak.json
+```
+
+Use this when you want the “can we make it ugly over 10 minutes?”
+answer, not just the short adversarial gate.
+
 ## 5. Operator-log readability
 
 - Hosted default logs are readable without hand-filtering.

{kernel_lore_mcp-0.3.0 → kernel_lore_mcp-0.3.2}/docs/ops/runbook.md

@@ -48,6 +48,10 @@ mkdir -p "$KLMCP_DATA_DIR"
     --include '/netdev/*'
 # Drop --include entirely to mirror all 390 shards (will take hours
 # and ~100+ GB of disk on the first run).
+# Keep BM25 deferred on a serving box unless you have measured that the
+# overlap is acceptable. Inline `--with-bm25` is the heavier path;
+# prefer `kernel-lore-ingest --rebuild-bm25` off-peak if prose freshness
+# matters.
 
 # 0A.5b — over.db has already been written incrementally by the
 # sync in 0A.4 (because we passed --with-over). If you skipped that
@@ -63,6 +67,8 @@ mkdir -p "$KLMCP_DATA_DIR"
 # 0A.6 — confirm the index is live (no HTTP needed)
 ./.venv/bin/kernel-lore-mcp status --data-dir "$KLMCP_DATA_DIR"
 # Expect: {"generation": >= 1, "freshness_ok": true, ...}
+# While a sync is active, status also reports `writer_lock_present`,
+# `sync_active`, and the current sync stage from `state/sync.json`.
 
 # 0A.6b — inspect shard/index health. If a prior run left poisoned shard
 # repos behind, --heal repairs unborn HEADs in place and removes

kernel_lore_mcp-0.3.2/docs/plans/2026-04-22-v0.3.1-plan.md

@@ -0,0 +1,106 @@
+# v0.3.1 plan
+
+**Date:** 2026-04-22
+**Motivation:** the 10-minute hosted soak on `server6` showed that
+`v0.3.0` is fine for serving-only load on similar hardware, but
+co-locating `serve` with `kernel-lore-sync --with-bm25` still creates
+an operator-visibility and client-backoff problem.
+
+## Findings that changed the patch line
+
+Two concrete runs shaped this release:
+
+- **Serving only, no sync overlap:** 10-minute `mixed_hot` soak,
+  4 processes x 16 async workers, delivered ~66 req/s with `/status`
+  staying healthy and only a small number of `rate_limited` outcomes.
+- **Same box, live sync overlap:** the same soak during a real sync
+  dropped to ~33 req/s and produced ~1,000 client-side timeouts.
+  `/status` itself timed out repeatedly, and operator logs stopped at
+  `ingest phase done` even though the process was still busy inside the
+  BM25 finalization path.
+
+That means the main `v0.3.1` job is not “invent a new architecture.”
+It is:
+
+1. tell the truth about what sync is doing,
+2. make inline BM25 safer by default,
+3. give callers a realistic retry hint when the box is clearly under
+   writer pressure,
+4. commit the soak harness we used to find the problem in the first
+   place.
+
+## Scope
+
+### 1. Live sync-state visibility
+
+- `kernel-lore-sync` writes a machine-readable `state/sync.json`
+  heartbeat while a run is active.
+- `/status` and `kernel-lore-mcp status` surface:
+  `writer_lock_present`, `sync_active`, and a `sync` block with the
+  current stage + heartbeat.
+- Status caching shortens while a writer is active so operators do not
+  stare at a 30-second-old snapshot during a live incident.
+
+**Done when:** an operator can tell “the box is currently in
+`bm25_commit`” from `/status` alone.
+
+### 2. BM25 finalization is visible and configurable
+
+- BM25 writer defaults become explicitly conservative:
+  `KLMCP_BM25_WRITER_THREADS=1` and
+  `KLMCP_BM25_WRITER_MEMORY_MB=128` unless overridden.
+- `kernel-lore-sync` logs explicit stage boundaries for:
+  `bm25_commit`, `tid_rebuild`, `path_vocab_rebuild`,
+  `generation_bump`, and `save_manifest`.
+- CLI help and operator docs explicitly warn that inline BM25 is the
+  heavier path and that co-located serving boxes should prefer deferred
+  rebuilds when practical.
+
+**Done when:** “ingest phase done” is no longer the last meaningful log
+line before a long quiet period, and operators can tune BM25 writer
+parallelism without rebuilding.
+
+### 3. Load-aware retry guidance
+
+- `rate_limited` and `query_timeout` errors continue to return
+  MCP-friendly in-band retry hints.
+- Retry hints rise when a live writer/sync is active, especially during
+  `bm25_commit`, instead of always saying “retry in 5s”.
+- The goal is coarse honesty, not fake precision. We are telling the
+  caller “back off more aggressively because the box is doing writer
+  work.”
+
+**Done when:** a caller hitting overload during `bm25_commit` sees a
+longer retry hint than a normal moderate-tool saturation event.
+
+### 4. Repeatable soak tooling lands in-tree
+
+- Ship the async + multiprocess stress harness used against `server6`
+  as `scripts/bench/stress_mcp_multiprocess.py`.
+- Reference it from the public-launch checklist as the longer-running
+  soak companion to the adversarial harness.
+
+**Done when:** reproducing the hosted soak requires no local scratch
+script or shell-history archaeology.
+
+## Explicit non-goals for v0.3.1
+
+- This patch release does **not** claim that same-box
+  `serve` + `sync --with-bm25` is fully public-launch-safe on modest
+  hardware.
+- The recommended posture for a public box remains:
+  keep the default deferred-BM25 sync path, or rebuild BM25 off-peak /
+  off-box if you need prose freshness.
+- If we want a hard guarantee that heavy writer work never disturbs
+  serving traffic, that is a `v0.3.x`/`v0.4.0` architecture item:
+  stronger isolation, not just better defaults and telemetry.
+
+## Acceptance gate
+
+Before tagging `v0.3.1`:
+
+- targeted Python tests for status / retry / observability pass,
+- `cargo test --bin kernel-lore-sync` passes,
+- release docs name the new sync-state + BM25 controls,
+- versions match in `Cargo.toml`, `pyproject.toml`, and
+  `src/kernel_lore_mcp/__init__.py`.

{kernel_lore_mcp-0.3.0 → kernel_lore_mcp-0.3.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "kernel-lore-mcp"
-version = "0.3.0"
+version = "0.3.2"
 description = "MCP server exposing fast, structured search over lore.kernel.org to LLM developer tools."
 readme = "README.md"
 license = { text = "MIT" }