@staix/vault-mem 0.3.3 → 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -2,6 +2,22 @@
2
2
 
3
3
  ## [Unreleased]
4
4
 
5
+ ## [0.4.0] - 2026-11-04
6
+
7
+ - Search, `vmem_search`, spec grep and writer duplicate hints match a Hangul or other CJK term of 2+ characters inside particle and compound forms (`사업단` finds `사업단은`, `사업단장`, `해외사업단`) through a side FTS5 bigram table. Latin terms keep token matching and their ranking; standalone-word and title/description hits rank before substring-only body hits. The index file name gains a schema version (`-links2-schema2.sqlite`), so the first run builds a new index once and older vmem versions keep their own file (#59).
8
+
9
+ - `vmem_patch` takes `edits` (1 to 32 `{oldString, newString}` pairs) as an alternative to the single pair, and `vmem patch --edits file|-` takes the same JSON array. Each `oldString` must match exactly once in the original note, edits must not overlap or start in frontmatter, and all apply or none do, with one receipt, one log event and the 256 KiB bound checked on the combined result. Several rows of one note no longer need one SHA round trip per edit (#63).
10
+
11
+ - Every MCP write tool takes the optional `log` event: `vmem_patch`, `vmem_append`, `vmem_set_frontmatter`, `vmem_move` and `vmem_delete` join `vmem_create`, `vmem_link` and `vmem_supersede` (move and delete use it on the confirm call). CLI writers keep `--log JSON` as the one form, now shown in each command's usage and checked against the same closed schema as the MCP field. A given event is validated and screened like `log append` before any note changes, so an invalid event (bad JSON, unknown key, bad type, unknown project, screened secret) refuses the whole write instead of saving the note and logging nothing; the event is no longer part of the move/delete fingerprint (#60).
12
+
13
+ - `vmem_list` takes a `cursor` and returns the filtered `total`, a per-row `mtime` from the index and `nextCursor`, so a full vault can be paged in stable path order. New read-only `vmem stats <vault>` and `vmem_stats` count notes by type, domain and status, notes modified in the last 7 and 30 days, and log events by type over a bounded window (default 30 days), without writing or locking the vault or its log. `vmem status` and `vmem doctor` no longer print the obsolete `stage: "foundation"` field; scripts that read it should drop that check (#62).
14
+
15
+ - `vmem manifest --json` lists read-only MCP tools as `{name, readOnly: true}` (taken from each tool's `readOnlyHint`), so installers can approve new read tools without a hard-coded list; other tools stay plain names. The release workflow states N-1 per minor line; patch releases carry no N-1 line (#106).
16
+
17
+ - Add MCP `vmem_read_file` and `vmem_list_files`, bounded read-only access to `ops/`, `reports/`, `self/`, `templates/`, `AGENTS.md` and `README.md` with `vmem_read` paging and limits; `notes/`, `log`, dot paths, symlinked components, non-UTF-8 and over-limit files are refused with the tool to use instead. Add `vmem_report`, the MCP form of `vmem report` with the same receipt and log event (#61).
18
+
19
+ - N-1: 0.3.3 runs beside this release. The derived index moves to `<sha>-links2-schema2.sqlite` (spec caches likewise), so 0.3.3 keeps its own `-links2.sqlite` and the first 0.4.0 run builds its index once. The config format, note format and log line format are unchanged, so each version reads what the other writes. MCP and CLI changes are additive for 0.3.3 clients: new tools (`vmem_read_file`, `vmem_list_files`, `vmem_report`, `vmem_stats`), optional `log` and `edits` arguments, and `vmem_list` adds `total`, `nextCursor` and per-row `mtime` beside `notes`. Two behaviors differ: `vmem status` no longer prints `stage`, and a write with an invalid log event is refused instead of saved without its event.
20
+
5
21
  ## [0.3.3] - 2026-11-02
6
22
 
7
23
  - Polish the spec read surface: scope fish parent completions to the first subcommand, preserve positional help tokens after `--`, document spec MCP tools and startup source registration, clarify shared field/error guidance, reuse body section helpers, and keep literal snippets on code-point boundaries (#103).
package/README.md CHANGED
@@ -87,9 +87,9 @@ bun bin/vmem.js links personal "Note or alias" --limit 50 --json
87
87
 
88
88
  All four commands require a registered vault id. They read only flat, regular `notes/*.md` files (case-sensitive `.md` extension), skip subdirectories and symlink files, and refuse a replaced vault/notes directory. Notes are limited to 2 MiB each. They never write note content, registry configuration or `.obsidian/`. `rebuild` rebuilds the derived cache; other commands scan metadata on every invocation and rehash new or metadata-changed files. Content-identical changes only update metadata; deleted/renamed paths leave the index. SQLite transactions make each scan atomic. A concurrently changing note aborts that scan, so retry after its writer finishes. No watcher, model calls or embeddings run.
89
89
 
90
- The cache is `$VMEM_HOME/indexes/<canonical-vault-path-sha256>-links<N>.sqlite` using the resolved home, not necessarily the environment variable. `N` is the link-resolution version (currently 2), so two installed vmem versions that resolve links differently keep separate caches instead of rebuilding each other's links; 0.2.4 and older use the unversioned `<sha256>.sqlite`, and a first run after upgrading builds the new file once. Cache files use mode 0600 on creation. The location must be outside every registered vault, including through symlinks; linked cache directories/files are refused. It contains note text and is disposable private derived state. Removing it while commands are stopped allows the next read to build it again. SQLite uses its own transaction lock and a five-second busy timeout. There is no automatic migration or live vault cutover.
90
+ The cache is `$VMEM_HOME/indexes/<canonical-vault-path-sha256>-links<N>-schema<S>.sqlite` using the resolved home, not necessarily the environment variable. `N` is the link-resolution version (currently 2) and `S` the index schema version (currently 2, which added the CJK bigram table), so two installed vmem versions that resolve links or store the index differently keep separate caches instead of rebuilding each other's; 0.2.4 and older use the unversioned `<sha256>.sqlite`, 0.3.0 through 0.3.3 use `<sha256>-links2.sqlite`, and a first run after upgrading builds the new file once with no manual step. Cache files use mode 0600 on creation. The location must be outside every registered vault, including through symlinks; linked cache directories/files are refused. It contains note text and is disposable private derived state. Removing it while commands are stopped allows the next read to build it again. SQLite uses its own transaction lock and a five-second busy timeout. There is no automatic migration or live vault cutover.
91
91
 
92
- Search uses FTS5 `unicode61` over title, description and body with BM25 weights 5:2:1, ascending score and path tie-breaking. Whitespace-separated query terms are literal AND terms, not a raw FTS expression; quote the query as one shell argument. `--type` and `--domain` are exact scalar frontmatter filters. Results include a bounded snippet and metadata, not the full body. Search queries and note targets accept at most 1024 UTF-8 bytes. Search/links default to 50 results, with an enforced 1..500 range. Show defaults to 16384 UTF-8 bytes and permits 1..65536, returning `truncated` and the full note SHA-256. `--section "Exact H2"` selects one real H2 through the next H1/H2, including its H3 children; code-fenced headings are ignored and duplicate/missing names refuse. Truncated reads include bounded `headings` (at most 64 names/8KiB metadata; `headingsTruncated` when omitted) and `nextOffset`. `--offset N` is a UTF-8 byte offset within the selected section or whole note; use the returned `nextOffset` to page even a single section larger than 64 KiB, and compare note hashes across pages. Invalid UTF-8 boundaries and zero-progress byte budgets refuse. Link string fields are capped at 1024 bytes in command results. Oversized arguments fail instead of silently raising a bound.
92
+ Search uses FTS5 `unicode61` over title, description and body with BM25 weights 5:2:1, ascending score and path tie-breaking. Whitespace-separated query terms are literal AND terms, not a raw FTS expression; quote the query as one shell argument. A term whose letters and digits form one run of 2 or more Hangul, Han, Hiragana or Katakana characters (surrounding punctuation is ignored) matches wherever it occurs in the title, description or body, including particle and compound forms: `사업단` finds `사업단은`, `사업단장` and `해외사업단`. That matching uses a side FTS5 table of overlapping character bigrams (Unicode NFC), so a 2-character term is one indexed lookup and a longer term a phrase of consecutive bigrams; it is substring matching, not morphological analysis. Every other term, including Latin words, single CJK characters and terms mixing CJK with Latin letters or digits (`AI교육`), keeps `unicode61` token matching, and a query without CJK terms ranks exactly as before. With a CJK term, notes where every CJK term is a standalone word or occurs in the title or description rank before notes that contain it only inside a longer body word; within each group BM25 over the bigram table (5:2:1) and path break ties, and the snippet is a body window around the first CJK term. `--type` and `--domain` are exact scalar frontmatter filters. Results include a bounded snippet and metadata, not the full body. Search queries and note targets accept at most 1024 UTF-8 bytes. Search/links default to 50 results, with an enforced 1..500 range. Show defaults to 16384 UTF-8 bytes and permits 1..65536, returning `truncated` and the full note SHA-256. `--section "Exact H2"` selects one real H2 through the next H1/H2, including its H3 children; code-fenced headings are ignored and duplicate/missing names refuse. Truncated reads include bounded `headings` (at most 64 names/8KiB metadata; `headingsTruncated` when omitted) and `nextOffset`. `--offset N` is a UTF-8 byte offset within the selected section or whole note; use the returned `nextOffset` to page even a single section larger than 64 KiB, and compare note hashes across pages. Invalid UTF-8 boundaries and zero-progress byte budgets refuse. Link string fields are capped at 1024 bytes in command results. Oversized arguments fail instead of silently raising a bound.
93
93
 
94
94
  Link resolution uses NFC and case-insensitive filename stems only, as Obsidian and `vmem lint` L01 do. Frontmatter `aliases` (string or string array; legacy `alias` accepted) are not link targets; `show`, `links <target>` and `vmem_read` still find a note by alias. A link that only an alias matches is a missing-note link, so the writer refuses a new one and org promotion drops it to text. An optional `notes/` prefix and `.md` suffix are accepted. `[[target|label]]`, `[[target#heading]]`, `[[target#^block]]`, `![[target]]` and same-note anchors work. ATX/Setext headings and end-of-line block ids are checked; fenced code is skipped. Missing/ambiguous targets and missing anchors remain explicit with a reason. Stems that differ only by case or Unicode normalization are ambiguous. Links returns outgoing links in source order, with one-based body-relative `line` values (frontmatter excluded); it does not write backlinks or repair dead links.
95
95
 
@@ -104,6 +104,7 @@ All mutations use `applyWrite(home, request)`: per-vault `mutation.lock`, curren
104
104
  vmem new personal example-finding --type insight --domain research --topics research --description "A distinct supporting observation."
105
105
  vmem new personal another-finding --type insight --domain research --topics research --description "An independently sourced result." --body body.md
106
106
  vmem patch personal example-finding --old "This note argues that" --new "This finding states that" --expected-sha <sha256>
107
+ vmem patch personal example-finding --edits edits.json --expected-sha <sha256> # [{"oldString": "...", "newString": "..."}, ...]
107
108
  vmem append personal example-finding --section Evidence --markdown "Additional evidence."
108
109
  vmem frontmatter personal example-finding --set '{"confidence":"likely"}'
109
110
  vmem link personal example-finding another-finding --relation supports
@@ -117,11 +118,11 @@ vmem tidy personal --connections
117
118
  vmem tidy personal --all --json # migration sanitize preview (see Migration tidy)
118
119
  ```
119
120
 
120
- Syntax follows indexed reads: the vault id comes first. `new` supports `--title`, `--source`, `--confidence`, `--status`, `--extra JSON` and `--body file|-`; omitted body gets an H1/Relevant Notes/Topics skeleton. Patch accepts nonempty old/new strings up to 8192 bytes and requires exactly one occurrence (including overlapping occurrences). Append uses an existing section or inserts it before Topics; it refuses targeting or introducing a Topics H2. `frontmatter` supports `--set JSON`, `--unset key,key` and `--expected-sha`; changing `topics` also replaces only the body Topics wiki-list items in the same write, preserving other content and line endings; setting an equal topics value leaves the body unchanged; append also supports `--expected-sha`. Link relations are `supports`, `contradicts`, `extends`, `applies-to`, `context-for`; links are bidirectional unless `--one-way`. Supersede sets `status`/`superseded_by` on the old note and adds a reverse link in the next note. Report permits plain paths below `reports/` or `ops/`, is schema-exempt and still screened. Inputs are bounded to 256 KiB, follow no file symlinks and accept stdin with `-`.
121
+ Syntax follows indexed reads: the vault id comes first. `new` supports `--title`, `--source`, `--confidence`, `--status`, `--extra JSON` and `--body file|-`; omitted body gets an H1/Relevant Notes/Topics skeleton. Patch accepts nonempty old/new strings up to 8192 bytes and requires exactly one occurrence (including overlapping occurrences). `--edits file|-` (MCP `edits`) instead takes a JSON array of 1 to 32 `{oldString, newString}` pairs with the same bounds: each `oldString` must match exactly once in the original note, edits must not overlap and none may start in frontmatter. They apply all-or-nothing, so any failing edit (or a stale `--expected-sha`) refuses the whole patch; a success gives one receipt and one log event, and the 256 KiB bound applies to the combined result. Use exactly one form per call; both or neither is refused. Append uses an existing section or inserts it before Topics; it refuses targeting or introducing a Topics H2. `frontmatter` supports `--set JSON`, `--unset key,key` and `--expected-sha`; changing `topics` also replaces only the body Topics wiki-list items in the same write, preserving other content and line endings; setting an equal topics value leaves the body unchanged; append also supports `--expected-sha`. Link relations are `supports`, `contradicts`, `extends`, `applies-to`, `context-for`; links are bidirectional unless `--one-way`. Supersede sets `status`/`superseded_by` on the old note and adds a reverse link in the next note. Report permits plain paths below `reports/` or `ops/`, is schema-exempt and still screened. Inputs are bounded to 256 KiB, follow no file symlinks and accept stdin with `-`.
121
122
 
122
123
  `--dry-run` plans and validates any writer without creating locks, temporary files, directories or caches. `check` always takes this path, even without `--dry-run`. Move, delete and tidy first return a preview with planned path/hash changes and a fingerprint; confirmation requires that fingerprint. It covers canonical operation input, the canonical vault path/identity and policy, expected SHAs, every scanned note and all touched files, so even a new inbound reference after preview forces another preview. Move rewrites visible wiki/heading/block/embed targets and wiki references in frontmatter across the vault, preserving explicit `|label` text, anchors and code. Alias-only links resolve to no note and are left as written. The move source may be an existing note with a legacy file name (lint L13, for example a dot in the stem); the destination must be a valid stem. Tidy folds Connections sections into Relevant Notes. Delete refuses when it would newly break inbound links or MOC topics; fix those references first.
123
124
 
124
- Validation returns named issues with severity and no matched secret text: V01 schema/enums/description; V02 existing MOC topics; V03 newly unresolved links; V04 flat ASCII note paths (maximum 200 bytes) and safe report paths; V05 final Topics placement/content (error when the write introduces a placement or content problem the note did not already have, or on create; warning when every problem was already present), with explicit append-to-Topics refusal; V06 pinned agent-mem secret regex parity and denied sources/input/snapshot paths; V07 email/phone/configured PII patterns when denied; V08 create duplicates (stem/NFKC H1); V09 256 KiB prospective note size / 16 MiB report size, exact patch and expected-SHA/drift; V10 supersession target integrity. Unchanged pre-existing broken links warn, including after a move. `possibleDuplicates` lists candidate paths for warnings from a disposable in-memory FTS5 query of title terms, with BM25 <= -0.000001, up to three paths; no disk cache is written for that check. `LOCKED`, `IO`, `ROLLBACK`, `INDEX` and `CLEANUP` identify operational failures. CLI returns exit 1 when `ok` is false.
125
+ Validation returns named issues with severity and no matched secret text: V01 schema/enums/description; V02 existing MOC topics; V03 newly unresolved links; V04 flat ASCII note paths (maximum 200 bytes) and safe report paths; V05 final Topics placement/content (error when the write introduces a placement or content problem the note did not already have, or on create; warning when every problem was already present), with explicit append-to-Topics refusal; V06 pinned agent-mem secret regex parity and denied sources/input/snapshot paths; V07 email/phone/configured PII patterns when denied; V08 create duplicates (stem/NFKC H1); V09 256 KiB prospective note size / 16 MiB report size, exact patch and expected-SHA/drift; V10 supersession target integrity. Unchanged pre-existing broken links warn, including after a move. `possibleDuplicates` lists candidate paths for warnings from a disposable in-memory FTS5 query of title terms, matched like search terms (CJK substrings included), with BM25 <= -0.000001, up to three paths; no disk cache is written for that check. `LOCKED`, `IO`, `ROLLBACK`, `INDEX` and `CLEANUP` identify operational failures. CLI returns exit 1 when `ok` is false.
125
126
 
126
127
  A successful receipt reports `applied`, planned path/hash/byte changes, issues and `indexUpdated`. Unrelated malformed legacy YAML is skipped by duplicate hints and by writer index refresh, with a V01 warning naming the unchanged note; normal read/rebuild commands retain strict parsing. Move/tidy still require parseable inputs and name the offending path on refusal. Reads allocate each note's stat size plus one growth-probe byte, capped at the 2 MiB read limit plus one. A failed index refresh rolls back the file changes. Rollback first verifies each file still equals this operation's committed result; external edits are preserved. Failed/refused rollback retains the original backup and reports its relative recovery path. All files are staged before replacement and backups permit rollback of caught multi-file failures. Each individual rename is atomic; a machine/process crash between multiple renames can leave a partially completed batch. The retained lock and `.vmem-<uuid>.tmp` files require inspection and manual recovery after confirming the writer stopped. They are never automatically removed by registry recovery. Uncoordinated editors do not honor the lock; the pipeline rechecks SHAs before replacement, but this does not provide a filesystem transaction against concurrent external edits. Existing notes may be read up to the index's 2 MiB limit; prospective notes are limited to 256 KiB; report operations under `reports/` and `ops/` allow up to 16 MiB for both prospective content and existing-file reads (including staging and rollback rechecks), with at most 10000 flat notes per mutation snapshot. File/directory fsync has the platform guarantees described above. Successful changed mutations now append an event as described below; log failure is reported separately from note success.
127
128
 
@@ -132,6 +133,7 @@ A successful receipt reports `applied`, planned path/hash/byte changes, issues a
132
133
  vmem log append personal EXTRACT - synthetic:source "A synthetic event" --dest notes/example.md
133
134
  vmem log tail personal --limit 50 --json
134
135
  vmem log grep personal "EXTRACT|CONNECT" --type EXTRACT --limit 20
136
+ vmem patch personal example-finding --old "argues" --new "states" --log '{"type":"EXTRACT","project":"-","source":"synthetic:source","note":"Reworded claim"}'
135
137
  ```
136
138
 
137
139
  Events keep the canonical local-time format `YYYY-MM-DD HH:MM TYPE project source → dest — note` (two spaces between fields, one LF at the end). Destination is optional. Allowed new types: `INGEST ROUTE EXTRACT CONNECT DIGEST LEARN LINT TASK GRAPH SYNC RETHINK SOURCE`. Project must be `-` or a lowercase alphanumeric/hyphen id; when `registry` is configured, ids must exist in its bounded YAML `projects`/nested `sub_projects` arrays. Registry files are limited to 1 MiB, eight nesting levels and 10000 unique ids; invalid or unavailable configuration fails closed. Invalid types/fields/projects use L09 issues, the 1000-byte line bound uses V09, and secret/PII refusals use V06/V07. Fields cannot inject pipes, control characters or two-space/column separators. Validation and screening run before a file or lock is created. The 1000 UTF-8-byte bound includes the LF and attribution.
@@ -140,10 +142,21 @@ Every MCP log append and CLI call with a nonempty `AGENTHUB_PEER_ID`, `CLAUDECOD
140
142
 
141
143
  `appendLog` validates the registered canonical vault and write/channel policy, acquires a separate `log.lock` (bounded two-second contention wait), then makes one O_APPEND write and fsyncs the log. New log files are mode 0600 and their parent directory is flushed. Linked/nonregular files and an incomplete existing final line are refused. Cooperating appenders share the lock; a short write is truncated back only when the same file still has the expected size, and its tail still equals our exact prefix, then flushed. Error-after-write cases inspect at most one event's bytes. Failed/ambiguous writes or cleanup are explicit; the receipt reports `appended` (possibly null when unknown), `synced`, `bytesWritten` (possibly null) and `rolledBack` when a partial append was removed. Before reporting sync, the current log and vault identities are rechecked; rotation produces LOG_DRIFT and an unknown current-path append state. A synced event can coexist with a cleanup error. No existing event is rewritten. A crash can retain `log.lock`, and failed short-write recovery can retain an incomplete line; inspect the file and stopped writer before manual recovery. There is no automatic cleanup or new TTY requirement.
142
144
 
143
- Successful changed mutations default to EXTRACT, link to CONNECT, move to ROUTE, and supersede/tidy to RETHINK. Default events use project `-`, affected paths and the operation name rather than note text. Library callers can supply `WriteRequest.log` to override the event or `false` to disable it; CLI writers accept `--log JSON` for overrides. Preview/check/no-op/refused mutations never append. The append is bound to the mutation's resolved vault identity/policy, so re-registration cannot route its event to another vault; the same binding covers the tidy LINT event and both SYNC events of org promotion, which are refused with LOG_DRIFT instead. A bound append compares the registry before any policy or path check, so a re-registration always reads as LOG_DRIFT. Tidy and org promotion also bind every note write to the vaults resolved at the start of the run: if a vault id is re-registered before a write, that write is refused with VAULT_DRIFT and nothing is written to either vault. After a committed mutation, log append is independent: `ok`/`applied` describe the note operation, `log` holds its append receipt, and `logged` is true only when the event was appended and synced. A log error does not roll back a successfully saved note. CLI exits 1 for a log failure even when the note receipt says `ok: true`; inspect the receipt before retrying to avoid duplicate mutations/events. An override is validated at append time, not during a note-only dry run.
145
+ Successful changed mutations default to EXTRACT, link to CONNECT, move to ROUTE, and supersede/tidy to RETHINK. Default events use project `-`, affected paths and the operation name rather than note text. Library callers can supply `WriteRequest.log` to override the event or `false` to disable it. Every MCP write tool (`vmem_create`, `vmem_patch`, `vmem_append`, `vmem_set_frontmatter`, `vmem_link`, `vmem_supersede`, `vmem_move`, `vmem_delete`) takes the same optional `log` object (`type`, `project`, `source`, `note`, optional `dest`); move and delete use it on the confirm call. The CLI writers `new`, `patch`, `append`, `frontmatter`, `link`, `supersede`, `move`, `delete` and `report` take the same object as `--log JSON`, checked against the same closed schema (unknown keys, a missing field or a bad type are refused); omitting it keeps the default event. `tidy --log` is the unrelated redaction rule. Preview/check/no-op/refused mutations never append. The append is bound to the mutation's resolved vault identity/policy, so re-registration cannot route its event to another vault; the same binding covers the tidy LINT event and both SYNC events of org promotion, which are refused with LOG_DRIFT instead. A bound append compares the registry before any policy or path check, so a re-registration always reads as LOG_DRIFT. Tidy and org promotion also bind every note write to the vaults resolved at the start of the run: if a vault id is re-registered before a write, that write is refused with VAULT_DRIFT and nothing is written to either vault. After a committed mutation, log append is independent: `ok`/`applied` describe the note operation, `log` holds its append receipt, and `logged` is true only when the event was appended and synced. A log error does not roll back a successfully saved note. CLI exits 1 for a log failure even when the note receipt says `ok: true`; inspect the receipt before retrying to avoid duplicate mutations/events. A given event is validated and screened like `log append` (type, registry project id, fields, secret/PII screening, line bound) before any lock or note change, also on dry runs and previews: an invalid event refuses the whole write with its L09/V06/V07/V09 issue and nothing changes. The event is not part of the move/delete fingerprint.
144
146
 
145
147
  Readers never create a home, log or lock, and work with `write: none`. They skip blank/comment lines, return the last matching 1..200 records (default 50), accept an optional canonical TYPE and non-backtracking regex subset up to 256 characters (`|`, `^`, `$`, `.`, character classes, `\d`/`\s`/`\w` and their inverses, escaped literals, and atom `*`/`+`/`?`; groups, backreferences and brace quantifiers are refused), and scan a fixed descriptor-size snapshot. Streaming is capped at 32 MiB; individual historical lines over 4096 bytes and incomplete final lines are omitted with warnings, and the serialized line array stays within 128 KiB. Above 20000 data lines, the manual-archival warning is preserved. Read failure returns `dataLines: null` rather than claiming a count. Golden synthetic fixtures and concurrent/fault persistence tests cover the format and actual append state; no live vault log was read or modified for this increment.
146
148
 
149
+ # Vault statistics
150
+
151
+ ```sh
152
+ vmem stats personal --json # log window: last 30 days
153
+ vmem stats personal --days 7 --json
154
+ ```
155
+
156
+ `vmem stats <vault>` and `vmem_stats` report note counts (total, by `type`, `domain` and frontmatter `status`; empty values count as `(none)`), notes whose indexed mtime falls in the last 7 and 30 local calendar days (today included), and log event counts by type dated on or after `log.since`, the first local date of the `--days` window (1..3650, default 30). Known types always appear, unknown ones count as `(other)`. The derived index cache is refreshed as by every read command and the log is streamed with the bounded log reader; no vault file or log is written and no vault or log lock is taken. When the log cannot be read, `log.total` and `log.byType` are null, `log.issues` names the cause and the CLI exits 1.
157
+
158
+ `vmem_list` pages the whole vault: pass the returned `nextCursor` as `cursor` to get the next page in path order. The cursor is the previous page's last path (exclusive), so a note added or removed between pages does not repeat or skip the others; a cursor outside the `prefix` filter is refused.
159
+
147
160
  # MCP server
148
161
 
149
162
  `vmem mcp` serves newline-delimited JSON-RPC 2.0 (MCP tools, protocol 2025-06-18 and earlier) on stdio. stdout carries protocol messages only; it needs no TTY and runs in agent sessions. Requests are limited to 64 KiB per frame and results to 256 KiB (a larger result becomes a structured `LIMIT` refusal). Arguments are validated against each tool's bounded schema (`additionalProperties: false`, string/array/integer caps); invalid arguments or unknown tools return JSON-RPC `-32602`, unknown methods `-32601`, malformed frames `-32700`/`-32600`. At startup it loads every module a tool can run, the `@staix/agent-mem` root included, so a server keeps its own code and tool list when its install is replaced or a newer release is installed beside it; restart the host session to pick up the new release.
@@ -160,26 +173,32 @@ The vault set is read once at startup: every tool's `vault` property is an enum
160
173
  | `vmem_search` | `query`<=1000, `vault`, `limit`<=50, `type`, `domain` |
161
174
  | `vmem_read` | `path` (stem, alias or `notes/` path), `vault`, `maxBytes`<=65536; returns `{path, frontmatter, body, sha256, truncated}` |
162
175
  | `vmem_read_many` | `paths`<=20 (8 KiB each), `vault` |
163
- | `vmem_list` | `vault`, `prefix`, `limit`<=500 |
176
+ | `vmem_list` | `vault`, `prefix`, `limit`<=500, `cursor`; rows in path order with `mtime`, the filtered `total` and `nextCursor` while more remain |
164
177
  | `vmem_links` | `path`, `vault`, `direction` out/in/both, `limit`<=200; returns `{out, dead, in}` |
165
178
  | `vmem_mocs` | `vault`; MOC notes with incoming link counts |
166
179
  | `vmem_check` | `vault`, `path`, `text`<=256 KiB; validation only |
167
180
  | `vmem_log_tail` | `vault`, `n`<=200, `type`, `grep`<=256 |
168
181
  | `vmem_create` | `stem`, `type`, `domain`, `description`, `topics[]`<=32, `title`, `body`<=65536, `source`, `confidence`, `status`, `extra`, `log` |
169
- | `vmem_patch` | `path`, `oldString`<=8192, `newString`<=8192, `expectedSha` |
170
- | `vmem_append` | `path`, `section`<=120, `markdown`<=16384, `expectedSha` |
171
- | `vmem_set_frontmatter` | `path`, `set`, `unset[]`, `expectedSha` |
182
+ | `vmem_patch` | `path`, `oldString`<=8192 and `newString`<=8192, or `edits[]` 1..32 of `{oldString, newString}` (all-or-nothing), `expectedSha`, `log` |
183
+ | `vmem_append` | `path`, `section`<=120, `markdown`<=16384, `expectedSha`, `log` |
184
+ | `vmem_set_frontmatter` | `path`, `set`, `unset[]`, `expectedSha`, `log` |
172
185
  | `vmem_link` | `from`, `to`, `relation`, `context`, `bidirectional` (default true), `log` |
173
186
  | `vmem_supersede` | `old`, `next`, `log` |
174
187
  | `vmem_log_append` | `type`, `project`, `source`, `dest`, `note`<=500 |
175
- | `vmem_move` | `from`, `to`, `confirm`, `fingerprint` (two-step) |
176
- | `vmem_delete` | `path`, `confirm`, `fingerprint` (two-step; the preview lists backlinks) |
188
+ | `vmem_move` | `from`, `to`, `confirm`, `fingerprint` (two-step), `log` (on confirm) |
189
+ | `vmem_delete` | `path`, `confirm`, `fingerprint` (two-step; the preview lists backlinks), `log` (on confirm) |
177
190
  | `vmem_promote` | `path` (in the personal vault), `to` (enum of org vaults), `topics[]`, `again`, `confirm`, `fingerprint` (two-step; never pushes) |
178
191
  | `vmem_skills_mine` | `vault`, `days`, `project`, `allProjects`, `vocab`, `skills[]`, `minSize`, `minNovelty` (read-only; see Skill mining) |
192
+ | `vmem_stats` | `vault`, `days`<=3650 (default 30); read-only counts, see Vault statistics |
179
193
  | `vmem_promote_from_amem` | `id`, `project`, `stem`, `type`, `domain`, `topics[]`, `description`, `title`, `body`<=65536, `linkTo`, `confirm`, `fingerprint` (two-step) |
194
+ | `vmem_read_file` | `path`, `vault`, `maxBytes`<=65536, `offset`<=2097152; returns `{path, sha256, truncated, body, offset?, nextOffset?}` (read-only; allowed roots below the table) |
195
+ | `vmem_list_files` | `dir`, `vault`; one level, <=500 entries `{name, path, type, bytes?}` (read-only) |
196
+ | `vmem_report` | `path` under `reports/` or `ops/`, `body`<=65536, `dryRun`, `log`; the receipt of `vmem report` |
180
197
 
181
198
  Move and delete without `confirm` write nothing and return the preview and `fingerprint`; `confirm: true` requires the fingerprint recomputed on the current state, otherwise the call is refused with `V09`. `vmem_promote_from_amem` is described under Promotion from agent-mem. `vmem_promote` is described under Org promotion and sync. Graph/lint tools are listed below. Note paths in read results are relative to `notes/`.
182
199
 
200
+ File reads and reports: `vmem_read_file` and `vmem_list_files` read only these roots: `ops/`, `reports/`, `self/`, `templates/`, `AGENTS.md` and `README.md`. They refuse `notes/` (use `vmem_read`), `log` (use `vmem_log_tail`), absolute paths, `.` or `..` segments, dot paths such as `.git`, any symlinked component, denied paths, non-UTF-8 files and files over 2 MiB. Paging, `maxBytes` and `offset` match `vmem_read`; listings leave out dot, linked, special and denied entries. Neither tool takes a lock, touches the index or writes a file. `vmem_report` is `vmem report` over MCP: the same path rules, screening, receipt and default `EXTRACT` event (a `log` override sets another event, for example `LINT` or `GRAPH`), with `write: mcp` required; MCP events always carry the agent attribution.
201
+
183
202
  # Host setup
184
203
 
185
204
  ```sh
@@ -196,7 +215,7 @@ Apply is idempotent and touches only the `vmem` entry: other servers and setting
196
215
 
197
216
  `vmem completion bash|zsh|fish` prints a completion script generated from the same command table as `--help`.
198
217
 
199
- `vmem manifest --json` prints `{name, version, configVersion, mcp: {server, tools}, commands}`: the package version, the config format it writes, the MCP tool names and the CLI command names from the same table, for tools that would otherwise hard-code them.
218
+ `vmem manifest --json` prints `{name, version, configVersion, mcp: {server, tools}, commands}`: the package version, the config format it writes, the MCP tools and the CLI command names from the same table, for tools that would otherwise hard-code them. A read-only tool (its `readOnlyHint` annotation) is listed as `{name, readOnly: true}`, every other tool as its plain name.
200
219
 
201
220
  # Promotion from agent-mem
202
221
 
@@ -291,7 +310,7 @@ vmem sync team --confirm --fingerprint <sha>
291
310
 
292
311
  - `bun run parity:release` checks same-snapshot frontmatter bytes, incremental notes/FTS/links against clean rebuild, and two consecutive synthetic Python-oracle lint/graph comparisons. External Python reference paths can be supplied with `--lint-reference` and `--graph-reference`; `--output` records aggregate evidence.
293
312
  - `bun run release:verify` enforces committed runtime/method/external-parity evidence freshness and the declared synthetic thresholds at the explicit release gate. Ordinary tests validate evidence shape and internal sample consistency; routine source edits and version bumps do not require fresh benchmark evidence until release validation.
294
- - `bun run bench:release --output <file.json>` measures a generated 5000-note corpus in the authorized Linux dgx02 container. It records actual MCP search round-trip timings (including metadata refresh), kernel RSS high-water, raw samples and environment; it exits nonzero when a synthetic threshold fails. Synthetic write timings are diagnostic and do not establish representative private-vault acceptance.
313
+ - `bun run bench:release --output <file.json>` measures a generated 5000-note corpus in the authorized Linux dgx02 container. It records actual MCP search round-trip timings (including metadata refresh), kernel RSS high-water, the derived index size after rebuild (`indexBytes`), raw samples and environment; it exits nonzero when a synthetic threshold fails. Synthetic write timings are diagnostic and do not establish representative private-vault acceptance.
295
314
  - [Readiness evidence](docs/release-readiness.md), [release workflow](docs/release-workflow.md) and [cutover checklist](docs/cutover-checklist.md) distinguish verified results from outstanding private/native/integration/deployment gates. Live cutover remains blocked.
296
315
 
297
316
 
@@ -362,7 +381,7 @@ vmem spec validate --repo /absolute/checkout --require-fresh --json
362
381
 
363
382
  `get` returns frontmatter, resolved outgoing links, backlinks, children, path, SHA-256 and computed freshness. Body reads are optional (`--body`, or body paging/section options) and use byte offsets with bounded UTF-8 pages. `graph` accepts subtree, ancestors or neighbors and an optional edge kind. `grep` searches FTS terms by default or a literal substring with `--literal`; filters are type, status, tag, parent, depends-on and covers-path. Use `--` before a positional id/query beginning with a hyphen, for example `vmem spec grep --literal -- "- [ ]"`. Results carry a continuation `nextOffset` when more matching hits exist. Snapshot path/hash filtering occurs before search limit/offset, so another scanner's cache rows cannot shorten pages. Rerun a command to refresh its snapshot. Large responses refuse with guidance or mark bounded collections as truncated.
364
383
 
365
- `validate` exits 1 on validation errors; stale scopes become errors only with `--require-fresh`. `--repo` validates a checkout without consulting a home/config file or creating a registry/cache, and cannot be combined with `--source` or `--worktree`. The four read tools are listed by `vmem manifest --json`; downstream hosts with frozen tool allow-lists need to add their names separately.
384
+ `validate` exits 1 on validation errors; stale scopes become errors only with `--require-fresh`. `--repo` validates a checkout without consulting a home/config file or creating a registry/cache, and cannot be combined with `--source` or `--worktree`. The four read tools are listed by `vmem manifest --json` as `{name, readOnly: true}`; hosts that approve from a frozen allow-list instead of the manifest need to add their names separately.
366
385
 
367
386
  # Spec validation and freshness library
368
387
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@staix/vault-mem",
3
- "version": "0.3.3",
3
+ "version": "0.4.0",
4
4
  "description": "Validated Obsidian-compatible knowledge vaults for agents and people",
5
5
  "license": "MIT",
6
6
  "type": "module",
package/src/cli.ts CHANGED
@@ -1,4 +1,4 @@
1
- import { runSpecCommand } from "./mcp-spec.ts";
1
+ import { runSpecCommand, specTools } from "./mcp-spec.ts";
2
2
  import { tidyVault, TIDY_RULES } from "./tidy.ts";
3
3
  import { originCheck } from "./sync.ts";
4
4
  import { registryIds } from "./vault-log.ts";
@@ -9,12 +9,13 @@ import { isWriteCommand, runWriteCommand } from "./write-commands.ts";
9
9
  import { isReadCommand, runReadCommand } from "./read-commands.ts";
10
10
  import { version } from "../package.json";
11
11
  import { COMMANDS, completionScript, help } from "./commands-table.ts";
12
- import { VMEM_TOOL_NAMES } from "./mcp-vault.ts";
12
+ import { VMEM_TOOL_NAMES, vaultTools } from "./mcp-vault.ts";
13
13
  import { CONFIG_VERSION } from "./config.ts";
14
14
  import { runSetup, type SetupMode } from "./setup.ts";
15
15
  import { resolveHome } from "./home.ts";
16
16
  import { addVault, addSpec, removeSpec, specStatus, effectivePii, loadRegistry, recoverRegistry, registryStatus, removeVault, setVaultPii } from "./registry.ts";
17
17
  import { piiPolicy } from "./pii.ts";
18
+ import { vaultStats } from "./vault-stats.ts";
18
19
  function parse(args: string[], allowed: string[], values: string[]) {
19
20
  const flags: Record<string, string | boolean> = {}, positional: string[] = [];
20
21
  for (let i = 0; i < args.length; i++) {
@@ -45,7 +46,10 @@ export function run(args: string[], env: NodeJS.ProcessEnv = process.env): { tex
45
46
  if (args[0] === "manifest") {
46
47
  const { flags, positional } = parse(args.slice(1), ["--json"], []);
47
48
  if (positional.length) throw new Error("manifest: unexpected arguments");
48
- const manifest = { name: "@staix/vault-mem", version, configVersion: CONFIG_VERSION, mcp: { server: "vmem", tools: [...VMEM_TOOL_NAMES] }, commands: COMMANDS.map(c => c.name) };
49
+ // Read-only tools are published as { name, readOnly: true } so installers can approve them without a hard-coded list.
50
+ const readOnly = new Set([...vaultTools([]), ...specTools()].filter(t => t.annotations.readOnlyHint).map(t => t.name));
51
+ const tools = VMEM_TOOL_NAMES.map(name => readOnly.has(name) ? { name, readOnly: true } : name);
52
+ const manifest = { name: "@staix/vault-mem", version, configVersion: CONFIG_VERSION, mcp: { server: "vmem", tools }, commands: COMMANDS.map(c => c.name) };
49
53
  return { text: JSON.stringify(manifest, null, flags["--json"] ? undefined : 2), code: 0 };
50
54
  }
51
55
  if (args[0] === "setup") {
@@ -68,7 +72,7 @@ export function run(args: string[], env: NodeJS.ProcessEnv = process.env): { tex
68
72
  const group = args[0] === "vaults" || args[0] === "specs";
69
73
  const command = group ? `${args[0]} ${args[1] ?? ""}` : args[0];
70
74
  const tail = args.slice(group ? 2 : 1);
71
- const options = command === "vaults add" ? ["--kind", "--write", "--pii", "--remote", "--registry"] : command === "vaults pii" ? ["--set"] : command === "specs add" ? ["--write", "--pii"] : [];
75
+ const options = command === "vaults add" ? ["--kind", "--write", "--pii", "--remote", "--registry"] : command === "vaults pii" ? ["--set"] : command === "specs add" ? ["--write", "--pii"] : command === "stats" ? ["--days"] : [];
72
76
  const { flags, positional } = parse(tail, ["--json", ...options, ...(command === "vaults remove" || command === "specs remove" || command === "vaults pii" || command === "recover" ? ["--confirm"] : [])], options);
73
77
  const home = resolveHome(env);
74
78
  let result: unknown, code = 0;
@@ -105,10 +109,19 @@ export function run(args: string[], env: NodeJS.ProcessEnv = process.env): { tex
105
109
  });
106
110
  const sources = command === "doctor" && config.specs !== undefined ? specStatus(home.home, env) : undefined;
107
111
  const healthy = vaults.every(v => v.healthy && (!v.checks || Object.values(v.checks).every(check => check.ok))) && (!sources || sources.every(source => source.healthy));
108
- result = { ...home, version, stage: "foundation", healthy, vaults, ...(config.specs !== undefined ? { specs: sources ?? config.specs } : {}) };
112
+ result = { ...home, version, healthy, vaults, ...(config.specs !== undefined ? { specs: sources ?? config.specs } : {}) };
109
113
  if (command === "doctor" && !healthy) code = 1;
110
114
  break;
111
115
  }
116
+ case "stats": {
117
+ if (positional.length !== 1) throw new Error("stats: requires one vault id; use vmem stats <vault> [--days 1..3650]");
118
+ const days = flags["--days"];
119
+ if (days !== undefined && !/^[0-9]+$/.test(String(days))) throw new Error("stats: --days expects an integer 1..3650; omit it for the 30-day default");
120
+ const stats = vaultStats(home.home, positional[0]!, days === undefined ? {} : { days: Number(days) });
121
+ result = stats;
122
+ if (stats.log.issues.length) code = 1;
123
+ break;
124
+ }
112
125
  case "recover":
113
126
  if (positional.length) throw new Error("recover: unexpected arguments");
114
127
  result = recoverRegistry(home.home, flags["--confirm"] === true);
@@ -19,16 +19,16 @@ export const COMMANDS = [
19
19
  { name: "specs list", usage: "specs list [--json]", mutates: false },
20
20
  { name: "specs add", usage: "specs add <id> <absolute-path> [--write mcp|cli|none] [--pii allow|deny] [--json]", mutates: true },
21
21
  { name: "specs remove", usage: "specs remove <id> [--confirm] [--json]", mutates: "confirm" },
22
- { name: "new", usage: "new <vault> <stem> --type TYPE --domain DOMAIN --topics a,b --description TEXT [--title TEXT] [--source TEXT|JSON] [--confidence TEXT] [--status TEXT] [--body file|-] [--extra JSON] [--dry-run] [--json]", mutates: true },
23
- { name: "patch", usage: "patch <vault> <note> --old TEXT --new TEXT [--expected-sha SHA] [--dry-run] [--json]", mutates: true },
24
- { name: "append", usage: "append <vault> <note> --section SECTION --markdown TEXT [--expected-sha SHA] [--dry-run] [--json]", mutates: true },
25
- { name: "frontmatter", usage: "frontmatter <vault> <note> [--set JSON] [--unset key,key] [--expected-sha SHA] [--dry-run] [--json]", mutates: true },
26
- { name: "link", usage: "link <vault> <from> <to> --relation R [--context TEXT] [--one-way] [--dry-run] [--json]", mutates: true },
27
- { name: "supersede", usage: "supersede <vault> <old> <next> [--dry-run] [--json]", mutates: true },
28
- { name: "move", usage: "move <vault> <from> <to> [--confirm --fingerprint SHA] [--dry-run] [--json]", mutates: "confirm" },
29
- { name: "delete", usage: "delete <vault> <note> [--confirm --fingerprint SHA] [--dry-run] [--json]", mutates: "confirm" },
22
+ { name: "new", usage: "new <vault> <stem> --type TYPE --domain DOMAIN --topics a,b --description TEXT [--title TEXT] [--source TEXT|JSON] [--confidence TEXT] [--status TEXT] [--body file|-] [--extra JSON] [--log JSON] [--dry-run] [--json]", mutates: true },
23
+ { name: "patch", usage: "patch <vault> <note> [--old TEXT --new TEXT] [--edits file|-] [--expected-sha SHA] [--log JSON] [--dry-run] [--json]", mutates: true },
24
+ { name: "append", usage: "append <vault> <note> --section SECTION --markdown TEXT [--expected-sha SHA] [--log JSON] [--dry-run] [--json]", mutates: true },
25
+ { name: "frontmatter", usage: "frontmatter <vault> <note> [--set JSON] [--unset key,key] [--expected-sha SHA] [--log JSON] [--dry-run] [--json]", mutates: true },
26
+ { name: "link", usage: "link <vault> <from> <to> --relation R [--context TEXT] [--one-way] [--log JSON] [--dry-run] [--json]", mutates: true },
27
+ { name: "supersede", usage: "supersede <vault> <old> <next> [--log JSON] [--dry-run] [--json]", mutates: true },
28
+ { name: "move", usage: "move <vault> <from> <to> [--confirm --fingerprint SHA] [--log JSON] [--dry-run] [--json]", mutates: "confirm" },
29
+ { name: "delete", usage: "delete <vault> <note> [--confirm --fingerprint SHA] [--log JSON] [--dry-run] [--json]", mutates: "confirm" },
30
30
  { name: "tidy", usage: "tidy <vault> [--connections] [--escapes] [--enums] [--keys] [--sources] [--topics] [--secrets] [--log] [--all] [--map file] [--confirm --fingerprint SHA] [--dry-run] [--json]", mutates: "confirm" },
31
- { name: "report", usage: "report <vault> <reports/path|ops/path> --body file|- [--dry-run] [--json]", mutates: true },
31
+ { name: "report", usage: "report <vault> <reports/path|ops/path> --body file|- [--log JSON] [--dry-run] [--json]", mutates: true },
32
32
  { name: "check", usage: "check <vault> <file|-> [--path stem] [--json]", mutates: false },
33
33
  { name: "log tail", usage: "log tail <vault> [--limit 1..200] [--type TYPE] [--json]", mutates: false },
34
34
  { name: "log grep", usage: "log grep <vault> <pattern> [--limit 1..200] [--type TYPE] [--json]", mutates: false },
@@ -40,6 +40,7 @@ export const COMMANDS = [
40
40
  { name: "setup", usage: "setup <claude|codex|kimi|pi|qwen|jcode|agy|all> [--root PATH] [--apply|--undo] [--json]", mutates: "apply" },
41
41
  { name: "manifest", usage: "manifest [--json]", mutates: false },
42
42
  { name: "completion", usage: "completion <bash|zsh|fish>", mutates: false },
43
+ { name: "stats", usage: "stats <vault> [--days 1..3650] [--json]", mutates: false },
43
44
  ] as const;
44
45
  const RELATIONS = ["supports", "contradicts", "extends", "applies-to", "context-for"] as const;
45
46
  const LOG_EVENT_TYPES = ["INGEST", "ROUTE", "EXTRACT", "CONNECT", "DIGEST", "LEARN", "LINT", "TASK", "GRAPH", "SYNC", "RETHINK", "SOURCE"] as const;
package/src/log-format.ts CHANGED
@@ -7,6 +7,9 @@ import type { LogContext, LogEvent } from "./log-types.ts";
7
7
  import type { WriteIssue } from "./write-types.ts";
8
8
 
9
9
  export const LOG_TYPES = ["INGEST", "ROUTE", "EXTRACT", "CONNECT", "DIGEST", "LEARN", "LINT", "TASK", "GRAPH", "SYNC", "RETHINK", "SOURCE"] as const;
10
+ const bounded = (maxLength: number) => ({ type: "string", minLength: 1, maxLength });
11
+ /** Closed schema of a write's log event, shared by the MCP write tools and CLI `--log JSON`. */
12
+ export const LOG_EVENT_SCHEMA = { type: "object", properties: { type: { type: "string", enum: [...LOG_TYPES] }, project: bounded(64), source: bounded(512), dest: bounded(512), note: bounded(500) }, required: ["type", "project", "source", "note"], additionalProperties: false };
10
13
  export const AGENT_MARKERS = ["AGENTHUB_PEER_ID", "CLAUDECODE", "CLAUDE_CODE_SESSION_ID", "CODEX_THREAD_ID", "PI_SESSION_ID"] as const;
11
14
  const EVENT_KEYS = new Set(["type", "project", "source", "dest", "note"]);
12
15
  const CONTEXT_KEYS = new Set(["vault", "config", "channel", "env", "now", "projectIds"]);
package/src/log-types.ts CHANGED
@@ -4,7 +4,11 @@ export interface LogEvent { type: string; project: string; source: string; dest?
4
4
  export interface LogRequest { vault: string; event: LogEvent; channel?: "cli" | "mcp" }
5
5
  export interface LogContext { vault: VaultSpec; config: VmemConfig; channel?: "cli" | "mcp"; env?: NodeJS.ProcessEnv; now?: Date; projectIds?: ReadonlySet<string> }
6
6
  export interface LogReceipt { ok: boolean; appended: boolean | null; synced: boolean; bytesWritten: number | null; issues: WriteIssue[]; rolledBack?: boolean }
7
- export interface LogReadOptions { vault: string; limit?: number; grep?: string; type?: string }
7
+ export interface LogReadOptions {
8
+ vault: string; limit?: number; grep?: string; type?: string;
9
+ /** In-process callers only (never CLI/MCP input): called with every complete data line, before the type and grep filters. */
10
+ visit?: (line: string) => void;
11
+ }
8
12
  export interface LogReadResult { ok: boolean; lines: string[]; dataLines: number | null; warnings: string[]; issues: WriteIssue[] }
9
13
 
10
14
  export interface LogBinding { vault: VaultSpec; denyPaths: string[]; piiPatterns: string[]; dev: number; ino: number }
package/src/mcp-spec.ts CHANGED
@@ -5,10 +5,13 @@ import { validateSpecRepo } from "./spec-validate.ts";
5
5
  import { resolveHome } from "./home.ts";
6
6
  import { loadRegistry } from "./registry.ts";
7
7
 
8
+ /** Spec tool definitions; every spec tool is read-only. */
9
+ export const specTools = (): McpTool[] => SPEC_COMMANDS.map(item => ({ name: item.name, description: item.description, inputSchema: item.inputSchema,
10
+ annotations: { readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: false } }));
11
+
8
12
  export function createSpecMcp(home: string, env: NodeJS.ProcessEnv = process.env): { tools: McpTool[]; handlers: Record<string, McpHandler> } {
9
13
  const registered = loadRegistry(home).specs ?? [];
10
- const tools: McpTool[] = SPEC_COMMANDS.map(item => ({ name: item.name, description: item.description, inputSchema: item.inputSchema,
11
- annotations: { readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: false } }));
14
+ const tools = specTools();
12
15
  const handlers: Record<string, McpHandler> = {};
13
16
  for (const item of SPEC_COMMANDS) handlers[item.name] = args => {
14
17
  const invalid = argumentError(args, item.inputSchema);
package/src/mcp-vault.ts CHANGED
@@ -8,11 +8,14 @@ import { parseFrontmatter } from "./frontmatter.ts";
8
8
  import type { VaultSpec } from "./config.ts";
9
9
  import { loadRegistry, registryStatus } from "./registry.ts";
10
10
  import { VaultIndex, MAX_NOTE_BYTES } from "./vault-index.ts";
11
+ import { listVaultFiles, readVaultFile } from "./vault-files.ts";
11
12
  import { appendLog, readLog } from "./vault-log.ts";
13
+ import { STATS_MAX_DAYS, vaultStats } from "./vault-stats.ts";
12
14
  import { applyWrite } from "./writer.ts";
13
15
  import { notePath } from "./write-operations.ts";
14
16
  import type { LogEvent } from "./log-types.ts";
15
- import type { WriteOp, WriteReceipt, WriteRequest } from "./write-types.ts";
17
+ import { LOG_EVENT_SCHEMA } from "./log-format.ts";
18
+ import type { PatchEdit, WriteOp, WriteReceipt, WriteRequest } from "./write-types.ts";
16
19
  import type { McpHandler, McpTool } from "./mcp.ts";
17
20
 
18
21
  const string = (maxLength: number, minLength = 1) => ({ type: "string", minLength, maxLength });
@@ -24,12 +27,12 @@ const tool = (name: string, description: string, inputSchema: Record<string, unk
24
27
  const RELATIONS = ["supports", "contradicts", "extends", "applies-to", "context-for"];
25
28
  const LOG_TYPES = ["INGEST", "ROUTE", "EXTRACT", "CONNECT", "DIGEST", "LEARN", "LINT", "TASK", "GRAPH", "SYNC", "RETHINK", "SOURCE"];
26
29
  const note = string(1024), sha = { type: "string", minLength: 64, maxLength: 64 };
27
- const logEvent = object({ type: { type: "string", enum: LOG_TYPES }, project: string(64), source: string(512), dest: string(512), note: string(500) }, ["type", "project", "source", "note"]);
30
+ const logEvent = LOG_EVENT_SCHEMA;
28
31
  const scalar = { type: ["string", "number", "boolean", "null", "array"], maxLength: 4096, maxItems: 64, items: { type: ["string", "number", "boolean"], maxLength: 1024 } };
29
32
 
30
33
  /** Tool names for the implemented read, write, log, analysis and promotion providers. */
31
34
  export const VMEM_TOOL_NAMES = ["vmem_status", "vmem_search", "vmem_read", "vmem_read_many", "vmem_list", "vmem_links", "vmem_mocs", "vmem_check", "vmem_log_tail",
32
- "vmem_create", "vmem_patch", "vmem_append", "vmem_set_frontmatter", "vmem_link", "vmem_supersede", "vmem_log_append", "vmem_move", "vmem_delete", "vmem_promote_from_amem", "vmem_lint", "vmem_graph_build", "vmem_graph_neighbors", "vmem_promote", "vmem_skills_mine", ...SPEC_COMMANDS.map(item => item.name)] as const;
35
+ "vmem_create", "vmem_patch", "vmem_append", "vmem_set_frontmatter", "vmem_link", "vmem_supersede", "vmem_log_append", "vmem_move", "vmem_delete", "vmem_promote_from_amem", "vmem_lint", "vmem_graph_build", "vmem_graph_neighbors", "vmem_promote", "vmem_skills_mine", "vmem_read_file", "vmem_list_files", "vmem_report", "vmem_stats", ...SPEC_COMMANDS.map(item => item.name)] as const;
33
36
 
34
37
  /** Tool definitions for a fixed vault set: `vault` is an enum of the registered ids. */
35
38
  export function vaultTools(ids: readonly string[], orgIds: readonly string[] = []): McpTool[] {
@@ -41,20 +44,20 @@ export function vaultTools(ids: readonly string[], orgIds: readonly string[] = [
41
44
  tool("vmem_search", "Full-text (BM25) search over indexed notes. Literal AND terms.", object({ query: string(1000), vault, limit: integer(1, 50), type: string(64), domain: string(64) }, ["query"])),
42
45
  tool("vmem_read", "Read one note by stem, alias or notes/ path: frontmatter, body, sha256. Bounded by maxBytes; section selects an exact H2, offset pages UTF-8 byte ranges, and truncated reads list H2 headings/nextOffset.", object({ path: note, vault, maxBytes: integer(1, 65536), section: string(1024), offset: integer(0, MAX_NOTE_BYTES) }, ["path"])),
43
46
  tool("vmem_read_many", "Read up to 20 notes (8 KiB each).", object({ paths: { type: "array", minItems: 1, maxItems: 20, items: note }, vault }, ["paths"])),
44
- tool("vmem_list", "List indexed notes under notes/, optionally by stem prefix.", object({ vault, prefix: string(256, 0), limit: integer(1, 500) })),
47
+ tool("vmem_list", "List indexed notes under notes/ in path order, optionally by stem prefix, with each note's mtime and the filtered total. Pass the returned nextCursor as cursor for the next page.", object({ vault, prefix: string(256, 0), limit: integer(1, 500), cursor: string(1024) })),
45
48
  tool("vmem_links", "Outgoing links (with resolution) and/or incoming backlinks of a note.", object({ path: note, vault, direction: { type: "string", enum: ["out", "in", "both"] }, limit: integer(1, 200) }, ["path"])),
46
49
  tool("vmem_mocs", "MOC notes with incoming link counts.", object({ vault })),
47
50
  tool("vmem_check", "Validate a prospective note text without writing.", object({ vault, path: string(200), text: string(262144, 0) }, ["text"])),
48
51
  tool("vmem_log_tail", "Read the newest vault log lines, optionally filtered.", object({ vault, n: integer(1, 200), type: { type: "string", enum: LOG_TYPES }, grep: string(256) })),
49
52
  tool("vmem_create", "Create a validated note. Requires a vault with write=mcp.", object({ vault, stem: string(200), type: string(64), domain: string(64), topics: { type: "array", maxItems: 32, items: string(200) }, description: string(1000), title: string(300), body: string(65536, 0), source: { type: ["string", "array"], minLength: 1, maxLength: 512, minItems: 1, maxItems: 32, items: string(512) }, confidence: string(32), status: string(32), created: string(32, 0), extra: { type: "object", maxProperties: 32, additionalProperties: scalar }, log: logEvent }, ["stem", "type", "domain", "description"]), "write"),
50
- tool("vmem_patch", "Replace one exact string in a note.", object({ vault, path: note, oldString: string(8192), newString: string(8192, 0), expectedSha: sha }, ["path", "oldString", "newString"]), "write"),
51
- tool("vmem_append", "Append markdown to a section of a note.", object({ vault, path: note, section: string(120), markdown: string(16384), expectedSha: sha }, ["path", "section", "markdown"]), "write"),
52
- tool("vmem_set_frontmatter", "Set or unset frontmatter keys of a note.", object({ vault, path: note, set: { type: "object", maxProperties: 32, additionalProperties: scalar }, unset: { type: "array", maxItems: 32, items: string(64) }, expectedSha: sha }, ["path"]), "write"),
53
+ tool("vmem_patch", "Replace exact strings in a note: one oldString/newString pair, or edits (1..32 pairs, each matching once in the original text, non-overlapping) applied all-or-nothing with one receipt and one log event.", object({ vault, path: note, oldString: string(8192), newString: string(8192, 0), edits: { type: "array", minItems: 1, maxItems: 32, items: object({ oldString: string(8192), newString: string(8192, 0) }, ["oldString", "newString"]) }, expectedSha: sha, log: logEvent }, ["path"]), "write"),
54
+ tool("vmem_append", "Append markdown to a section of a note.", object({ vault, path: note, section: string(120), markdown: string(16384), expectedSha: sha, log: logEvent }, ["path", "section", "markdown"]), "write"),
55
+ tool("vmem_set_frontmatter", "Set or unset frontmatter keys of a note.", object({ vault, path: note, set: { type: "object", maxProperties: 32, additionalProperties: scalar }, unset: { type: "array", maxItems: 32, items: string(64) }, expectedSha: sha, log: logEvent }, ["path"]), "write"),
53
56
  tool("vmem_link", "Add a typed relation link between two notes (bidirectional by default).", object({ vault, from: note, to: note, relation: { type: "string", enum: RELATIONS }, context: string(500), bidirectional: { type: "boolean" }, log: logEvent }, ["from", "to", "relation"]), "write"),
54
57
  tool("vmem_supersede", "Mark a note superseded by another.", object({ vault, old: note, next: note, log: logEvent }, ["old", "next"]), "write"),
55
58
  tool("vmem_log_append", "Append one validated event line to the vault log.", object({ vault, type: { type: "string", enum: LOG_TYPES }, project: string(64), source: string(512), dest: string(512), note: string(500) }, ["type", "project", "source", "note"]), "write"),
56
- tool("vmem_move", "Two-step rename with link rewrites: without confirm returns preview and fingerprint; then confirm=true with that fingerprint.", object({ vault, from: note, to: note, ...twoStep }, ["from", "to"]), "confirm"),
57
- tool("vmem_delete", "Two-step delete: without confirm returns preview (or refusal), backlinks and fingerprint; then confirm=true with that fingerprint.", object({ vault, path: note, ...twoStep }, ["path"]), "confirm"),
59
+ tool("vmem_move", "Two-step rename with link rewrites: without confirm returns preview and fingerprint; then confirm=true with that fingerprint (pass log on the confirm call).", object({ vault, from: note, to: note, ...twoStep, log: logEvent }, ["from", "to"]), "confirm"),
60
+ tool("vmem_delete", "Two-step delete: without confirm returns preview (or refusal), backlinks and fingerprint; then confirm=true with that fingerprint (pass log on the confirm call).", object({ vault, path: note, ...twoStep, log: logEvent }, ["path"]), "confirm"),
58
61
  tool("vmem_promote_from_amem", "Two-step promotion of one durable agent-mem entry (id from amem_vault_candidates) into a vault note: without confirm returns the full note text, possible duplicates and fingerprint; confirm=true with that fingerprint creates the note, logs LEARN and records the agent-mem vault link. linkTo links an existing note instead.",
59
62
  object({ vault, id: { type: "string", minLength: 32, maxLength: 36 }, stem: string(120), type: string(64), domain: string(64), topics: { type: "array", maxItems: 32, items: string(200) }, description: string(1000), title: string(300), body: string(65536, 0), linkTo: note, project: string(400), ...twoStep }, ["id"]), "confirm"),
60
63
  tool("vmem_lint", "Lint a private derived snapshot; write=true writes a screened report.", object({ vault, write: { type: "boolean" }, dryRun: { type: "boolean" } }), "write"),
@@ -64,6 +67,10 @@ export function vaultTools(ids: readonly string[], orgIds: readonly string[] = [
64
67
  object({ path: note, to: orgVault, topics: { type: "array", maxItems: 32, items: string(200) }, again: { type: "boolean" }, ...twoStep }, ["path", "to"]), "confirm"),
65
68
  tool("vmem_skills_mine", "Read-only skill-candidate mining over amem curated entries, amem capture rows and vault notes: dictionary matching, deterministic communities and novelty scoring. Returns counts, masked candidates and a markdown report; writes nothing.",
66
69
  object({ vault, days: integer(1, 3650), project: string(400), allProjects: { type: "boolean" }, vocab: string(4096), skills: { type: "array", maxItems: 16, items: string(4096) }, minSize: integer(1, 10000), minNovelty: { type: "number", minimum: 0, maximum: 1 } })),
70
+ tool("vmem_read_file", "Read one UTF-8 file outside notes/: ops/, reports/, self/, templates/, AGENTS.md or README.md. Same bounds and paging as vmem_read (maxBytes, offset, nextOffset); notes/ use vmem_read, the log vmem_log_tail. Never locks or writes.", object({ vault, path: string(1024), maxBytes: integer(1, 65536), offset: integer(0, MAX_NOTE_BYTES) }, ["path"])),
71
+ tool("vmem_list_files", "List one directory level under ops/, reports/, self/ or templates/ (up to 500 entries; dot, linked and denied entries omitted). Never locks or writes.", object({ vault, dir: string(1024) }, ["dir"])),
72
+ tool("vmem_report", "Write a screened report file under reports/ or ops/ (the MCP form of `vmem report`): same path rules, screening, write policy (write=mcp), receipt and EXTRACT log event; log overrides it, for example a LINT or GRAPH event. dryRun changes nothing.", object({ vault, path: string(200), body: string(65536, 0), dryRun: { type: "boolean" }, log: logEvent }, ["path", "body"]), "write"),
73
+ tool("vmem_stats", "Read-only vault statistics: note counts by type, domain and status, notes modified in the last 7 and 30 days, and log event counts by type over the last `days` (default 30). Writes no vault file or log and takes no vault or log lock.", object({ vault, days: integer(1, STATS_MAX_DAYS) })),
67
74
  ];
68
75
  }
69
76
  type Issue = { code: string; severity: "error"; message: string };
@@ -122,6 +129,9 @@ export function createVaultMcp(home: string, env: NodeJS.ProcessEnv = process.en
122
129
  ...(Array.isArray(args.skills) ? { skills: args.skills as string[] } : {}), ...(typeof args.days === "number" ? { days: args.days } : {}),
123
130
  ...(typeof args.minSize === "number" ? { minSize: args.minSize } : {}), ...(typeof args.minNovelty === "number" ? { minNovelty: args.minNovelty } : {}) });
124
131
  },
132
+ vmem_read_file: args => readVaultFile(pick(args), loadRegistry(home).denyPaths ?? [], str(args.path)!, (args.maxBytes as number | undefined) ?? 16384, args.offset as number | undefined),
133
+ vmem_list_files: args => listVaultFiles(pick(args), loadRegistry(home).denyPaths ?? [], str(args.dir)!),
134
+ vmem_report: args => write(args, { op: "report", path: str(args.path)!, text: str(args.body)! }, args.dryRun === true ? { dryRun: true } : {}),
125
135
  vmem_lint: args => args.write === true ? writeAnalysisReports(home,{vault:pick(args).id,kind:"lint",channel:"mcp",expectedVault:pick(args),dryRun:args.dryRun === true},env) : lintVault(analysisInput(home,pick(args).id,{expectedVault:pick(args)}).snapshot),
126
136
  vmem_graph_build: args => writeAnalysisReports(home,{vault:pick(args).id,kind:"graph",workspace:args.workspace === true,channel:"mcp",expectedVault:pick(args),dryRun:args.dryRun === true},env),
127
137
  vmem_graph_neighbors: args => graphNeighbors(buildGraph(analysisInput(home,pick(args).id,{expectedVault:pick(args)}).snapshot),str(args.path)!, (args.hops as 1|2|undefined) ?? 1, (args.limit as number|undefined) ?? 50),
@@ -132,7 +142,8 @@ export function createVaultMcp(home: string, env: NodeJS.ProcessEnv = process.en
132
142
  try { const r = index.show(path, 8192); return document(r); }
133
143
  catch (error) { return { path, ...refusal("V04", error instanceof Error ? error.message : "unreadable") }; }
134
144
  }) })),
135
- vmem_list: args => read(args, index => ({ notes: index.list(str(args.prefix) ?? "", (args.limit as number | undefined) ?? 100) })),
145
+ vmem_list: args => read(args, index => index.list(str(args.prefix) ?? "", (args.limit as number | undefined) ?? 100, str(args.cursor))),
146
+ vmem_stats: args => vaultStats(home, pick(args).id, { expectedVault: pick(args), ...(typeof args.days === "number" ? { days: args.days } : {}) }),
136
147
  vmem_links: args => read(args, index => {
137
148
  const direction = str(args.direction) ?? "both", limit = (args.limit as number | undefined) ?? 50;
138
149
  const out = direction === "in" ? [] : index.links(str(args.path)!, limit);
@@ -152,7 +163,7 @@ export function createVaultMcp(home: string, env: NodeJS.ProcessEnv = process.en
152
163
  if (Array.isArray(args.topics)) frontmatter.topics = (args.topics as string[]).map(v => v.startsWith("[[") ? v : `[[${v}]]`);
153
164
  return write(args, { op: "create", path: str(args.stem)!, frontmatter, ...(str(args.body) !== undefined ? { body: str(args.body) } : {}) });
154
165
  },
155
- vmem_patch: args => write(args, { op: "patch", path: str(args.path)!, oldString: str(args.oldString)!, newString: str(args.newString)!, expectedSha: str(args.expectedSha) }),
166
+ vmem_patch: args => write(args, { op: "patch", path: str(args.path)!, ...(args.oldString !== undefined ? { oldString: str(args.oldString) } : {}), ...(args.newString !== undefined ? { newString: str(args.newString) } : {}), ...(args.edits !== undefined ? { edits: args.edits as PatchEdit[] } : {}), expectedSha: str(args.expectedSha) }),
156
167
  vmem_append: args => write(args, { op: "append", path: str(args.path)!, section: str(args.section)!, markdown: str(args.markdown)!, expectedSha: str(args.expectedSha) }),
157
168
  vmem_set_frontmatter: args => write(args, { op: "frontmatter", path: str(args.path)!, set: (args.set as Record<string, unknown> | undefined) ?? {}, ...(args.unset ? { unset: args.unset as string[] } : {}), expectedSha: str(args.expectedSha) }),
158
169
  vmem_link: args => write(args, { op: "link", from: str(args.from)!, to: str(args.to)!, relation: args.relation as Extract<WriteOp, { op: "link" }>["relation"], ...(str(args.context) ? { context: str(args.context) } : {}), bidirectional: args.bidirectional !== false }),
@@ -0,0 +1,74 @@
1
+ import { closeSync, constants, fstatSync, lstatSync, openSync, readdirSync, readSync, realpathSync } from "node:fs";
2
+ import { createHash } from "node:crypto";
3
+ import { join } from "node:path";
4
+ import type { VaultSpec } from "./config.ts";
5
+ import { isDenied } from "./platform.ts";
6
+ import { MAX_NOTE_BYTES, MAX_READ_BYTES, MAX_RESULTS, readPage } from "./vault-index.ts";
7
+
8
+ /** Vault paths readable outside notes/: these directories and root files. Reads never lock, index or write. */
9
+ export const FILE_ROOTS = ["ops/", "reports/", "self/", "templates/", "AGENTS.md", "README.md"] as const;
10
+ const DIRS = new Set(["ops", "reports", "self", "templates"]), FILES = new Set(["AGENTS.md", "README.md"]);
11
+ const ROOTS = "ops/, reports/, self/, templates/, AGENTS.md or README.md";
12
+ function fail(message: string, code = "V04"): never { throw Object.assign(new Error(message), { code }); }
13
+
14
+ /** Checks `path` against the allowed roots and returns its location; every component must be a plain directory or file. */
15
+ function locate(vault: VaultSpec, denyPaths: string[], path: string, kind: "file" | "dir"): string {
16
+ const parts = path.split("/");
17
+ if (path.startsWith("/") || path.includes("\\") || path.includes("\0") || parts.some(part => !part || part === "." || part === ".."))
18
+ fail(`path: absolute, empty, . or .. segments are refused; pass a vault-relative path under ${ROOTS}`);
19
+ if (parts.some(part => part.startsWith("."))) fail(`path: dot paths such as .git are not readable through vmem; read a path under ${ROOTS}`);
20
+ if (parts[0] === "notes") fail("path: notes/ is read with vmem_read or vmem_read_many and listed with vmem_list");
21
+ if (parts[0] === "log") fail("path: the vault log is read with vmem_log_tail");
22
+ if (!(DIRS.has(parts[0]!) || (kind === "file" && parts.length === 1 && FILES.has(parts[0]!))))
23
+ fail(`path: outside the readable roots; use a path under ${ROOTS}`);
24
+ const root = realpathSync(vault.path);
25
+ if (root !== vault.path) fail(`vault ${vault.id}: registered path changed; re-register it with vmem vaults add and restart the MCP server`);
26
+ if (isDenied(join(root, path), denyPaths)) fail("path: denied by the built-in secret rules or registry denyPaths; nothing read", "V06");
27
+ let current = root;
28
+ for (let i = 0; i < parts.length; i++) {
29
+ current = join(current, parts[i]!);
30
+ let stat: ReturnType<typeof lstatSync>;
31
+ try { stat = lstatSync(current); }
32
+ catch (error) { if ((error as NodeJS.ErrnoException).code === "ENOENT") fail(`path: not found; list the parent directory with vmem_list_files`); throw error; }
33
+ // ponytail: every link is refused, not only links leaving the vault; matches the writer's plain-path rule.
34
+ if (stat.isSymbolicLink()) fail("path: symlinked components are refused because a link can leave the vault; read the link target's own vault path");
35
+ const last = i === parts.length - 1, wantDir = !last || kind === "dir";
36
+ if (wantDir ? !stat.isDirectory() : !stat.isFile()) fail(wantDir ? "path: not a directory; list its parent with vmem_list_files" : "path: not a regular file; list it with vmem_list_files");
37
+ }
38
+ return current;
39
+ }
40
+
41
+ /** Bounded UTF-8 read with vmem_read's limits and paging: maxBytes 1..64 KiB, files up to 2 MiB, byte offsets on character boundaries. */
42
+ export function readVaultFile(vault: VaultSpec, denyPaths: string[], path: string, maxBytes = 16384, offset?: number) {
43
+ if (!Number.isInteger(maxBytes) || maxBytes < 1 || maxBytes > MAX_READ_BYTES) fail(`max-bytes: expected integer 1..${MAX_READ_BYTES}`);
44
+ if (offset !== undefined && (!Number.isSafeInteger(offset) || offset < 0 || offset > MAX_NOTE_BYTES)) fail(`offset: expected integer 0..${MAX_NOTE_BYTES}`);
45
+ const fd = openSync(locate(vault, denyPaths, path, "file"), constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK);
46
+ let bytes: Buffer;
47
+ try {
48
+ const before = fstatSync(fd);
49
+ if (!before.isFile()) fail("path: not a regular file; list it with vmem_list_files");
50
+ if (before.size > MAX_NOTE_BYTES) fail(`file: exceeds the ${MAX_NOTE_BYTES}-byte read bound; read it outside vmem or split it`, "V09");
51
+ const buffer = Buffer.alloc(before.size + 1); let size = 0;
52
+ while (size < buffer.length) { const count = readSync(fd, buffer, size, buffer.length - size, null); if (!count) break; size += count; }
53
+ const after = fstatSync(fd);
54
+ if (size !== before.size || after.size !== before.size || after.mtimeMs !== before.mtimeMs) fail("file: changed during read; retry once it is stable", "V09");
55
+ bytes = buffer.subarray(0, size);
56
+ } finally { closeSync(fd); }
57
+ const text = bytes.toString("utf8");
58
+ if (!Buffer.from(text).equals(bytes)) fail("file: not valid UTF-8 text; vmem_read_file reads text files only", "V01");
59
+ const start = offset ?? 0, page = readPage(bytes.subarray(start, start + maxBytes + 4), maxBytes, start, bytes.length);
60
+ return { path, sha256: createHash("sha256").update(bytes).digest("hex"), truncated: page.truncated, body: page.content,
61
+ ...(offset !== undefined ? { offset } : {}), ...(page.nextOffset !== undefined ? { nextOffset: page.nextOffset } : {}) };
62
+ }
63
+
64
+ /** One directory level under an allowed root, sorted by name; dot, linked, special and denied entries are left out. */
65
+ export function listVaultFiles(vault: VaultSpec, denyPaths: string[], dir: string) {
66
+ const clean = dir.replace(/\/$/, ""), location = locate(vault, denyPaths, clean, "dir");
67
+ // ponytail: readdirSync loads the whole directory before the 500-entry cut; stream with opendir if vault dirs grow huge.
68
+ const entries = readdirSync(location, { withFileTypes: true })
69
+ .filter(entry => !entry.name.startsWith(".") && (entry.isFile() || entry.isDirectory()) && !isDenied(join(location, entry.name), denyPaths))
70
+ .sort((a, b) => a.name < b.name ? -1 : a.name > b.name ? 1 : 0);
71
+ return { dir: clean, truncated: entries.length > MAX_RESULTS, entries: entries.slice(0, MAX_RESULTS).map(entry => entry.isDirectory()
72
+ ? { name: entry.name, path: `${clean}/${entry.name}`, type: "dir" as const }
73
+ : { name: entry.name, path: `${clean}/${entry.name}`, type: "file" as const, bytes: lstatSync(join(location, entry.name)).size }) };
74
+ }
@@ -16,7 +16,40 @@ export const MAX_RESULTS = 500;
16
16
  * share the unversioned `<digest>.sqlite`; the first run of a newer version builds its own file once.
17
17
  */
18
18
  export const LINK_RESOLUTION = 2;
19
- const indexName = (root: string) => `${digest(root)}-links${LINK_RESOLUTION}.sqlite`;
19
+ /** Bump when tables or FTS columns change. Named per value like LINK_RESOLUTION. 2 = CJK bigram table. */
20
+ export const INDEX_SCHEMA = 2;
21
+ const indexName = (root: string) => `${digest(root)}-links${LINK_RESOLUTION}-schema${INDEX_SCHEMA}.sqlite`;
22
+ /**
23
+ * unicode61 keeps each Hangul/CJK run as one token, so `사업단` misses `사업단은`. A side table holds the
24
+ * overlapping bigrams of every CJK run of 2+ characters, runs split by a `0` token; a CJK query term then matches
25
+ * one bigram (2 characters) or a phrase of consecutive bigrams, which is exact substring matching. A separate
26
+ * table keeps bm25 of the unicode61 table, whose document length spans all columns, unchanged for Latin queries.
27
+ */
28
+ export const GRAMS_COLUMNS = "path UNINDEXED, title, description, body";
29
+ const CJK = "\\p{sc=Hangul}\\p{sc=Han}\\p{sc=Hiragana}\\p{sc=Katakana}";
30
+ const CJK_RUN = new RegExp(`[${CJK}]{2,}`, "gu"), CJK_TERM = new RegExp(`^[${CJK}]{2,}$`, "u");
31
+ function bigrams(run: string): string[] {
32
+ const chars = [...run];
33
+ return chars.slice(1).map((char, i) => chars[i]! + char);
34
+ }
35
+ export function grams(text: string): string {
36
+ return [...text.normalize("NFC").matchAll(CJK_RUN)].map(([run]) => `${bigrams(run).join(" ")} 0`).join(" ");
37
+ }
38
+ export const gramPhrase = (run: string) => `"${bigrams(run).join(" ")}"`;
39
+ /**
40
+ * Literal AND terms: users cannot inject FTS operators, columns or SQL. A term whose letters and digits form one
41
+ * CJK run of 2+ characters (`runs`) matches as a substring through the bigram table; any other term (`tokens`,
42
+ * quoted) keeps unicode61 token matching.
43
+ */
44
+ export function splitTerms(words: string[]): { tokens: string[]; runs: string[] } {
45
+ const tokens: string[] = [], runs: string[] = [];
46
+ for (const term of words) {
47
+ const letters = term.normalize("NFC").match(/[\p{L}\p{N}]+/gu);
48
+ if (letters?.length === 1 && CJK_TERM.test(letters[0]!)) runs.push(letters[0]!);
49
+ else tokens.push(`"${term.replaceAll('"', '""')}"`);
50
+ }
51
+ return { tokens, runs };
52
+ }
20
53
  export const MAX_READ_BYTES = 64 * 1024;
21
54
  export interface IndexedNote extends LinkNote { hash: string; description: string; type: string; domain: string }
22
55
  export interface NoteReadResult {
@@ -62,6 +95,11 @@ function aliases(value: unknown): string[] {
62
95
  function metadata(stat: NonNullable<ReturnType<typeof lstatSync>>): string {
63
96
  return [stat.dev, stat.ino, stat.size, stat.mtimeMs, stat.ctimeMs].join(":");
64
97
  }
98
+ /** The mtime (epoch ms) stored in a `metadata()` stamp, or null when the stamp has another shape. */
99
+ function stampMtime(stamp: string): number | null {
100
+ const value = Number(stamp.split(":")[3]);
101
+ return stamp.split(":").length === 5 && Number.isFinite(value) ? value : null;
102
+ }
65
103
  /** Adapter file I/O gets path-specific guidance without rewriting unrelated Git or binding errors. */
66
104
  function specFileIO<T>(path: string, action: string, operation: () => T): T {
67
105
  try { return operation(); }
@@ -78,7 +116,7 @@ function boundedText(value: string, bytes: number): { content: string; truncated
78
116
  while (end > 0 && (buffer[end]! & 0xc0) === 0x80) end--;
79
117
  return { content: buffer.subarray(0, end).toString("utf8"), truncated: true };
80
118
  }
81
- function readPage(prefix: Buffer, maxBytes: number, offset: number, totalBytes: number): Pick<NoteReadResult, "content" | "truncated" | "nextOffset"> {
119
+ export function readPage(prefix: Buffer, maxBytes: number, offset: number, totalBytes: number): Pick<NoteReadResult, "content" | "truncated" | "nextOffset"> {
82
120
  if (offset > totalBytes) throw new Error("offset: exceeds requested note or section");
83
121
  if (prefix.length && (prefix[0]! & 0xc0) === 0x80) throw new Error("offset: must be a UTF-8 character boundary");
84
122
  let end = Math.min(maxBytes, prefix.length);
@@ -165,6 +203,7 @@ export class VaultIndex<Source extends VaultSpec | SpecSourceSpec = VaultSpec> {
165
203
  title TEXT NOT NULL, description TEXT NOT NULL, body TEXT NOT NULL,
166
204
  type TEXT NOT NULL, domain TEXT NOT NULL, aliases TEXT NOT NULL, source TEXT NOT NULL);
167
205
  CREATE VIRTUAL TABLE IF NOT EXISTS fts USING fts5(path UNINDEXED, title, description, body, tokenize='unicode61');
206
+ CREATE VIRTUAL TABLE IF NOT EXISTS grams USING fts5(${GRAMS_COLUMNS}, tokenize='unicode61');
168
207
  CREATE TABLE IF NOT EXISTS analysis_notes (path TEXT PRIMARY KEY, source TEXT NOT NULL, hash TEXT NOT NULL, metadata TEXT NOT NULL);
169
208
  CREATE TABLE IF NOT EXISTS links (source TEXT NOT NULL, ordinal INTEGER NOT NULL, data TEXT NOT NULL, PRIMARY KEY(source, ordinal));`);
170
209
  // Dry runs reuse the derived cache via a readonly connection, then refresh only changed notes in memory.
@@ -182,7 +221,7 @@ export class VaultIndex<Source extends VaultSpec | SpecSourceSpec = VaultSpec> {
182
221
  if (size.count > 10000 || size.bytes > 64 * 1024 * 1024) throw new Error("analysis: cached snapshot exceeds bound");
183
222
  for (const row of disk.query<Omit<StoredNote,"aliases"> & {aliases:string}, []>("SELECT * FROM notes").all()) {
184
223
  this.db.query("INSERT INTO notes VALUES (?,?,?,?,?,?,?,?,?,?)").run(row.path,row.hash,row.metadata,row.title,row.description,row.body,row.type,row.domain,row.aliases,row.source);
185
- this.db.query("INSERT INTO fts VALUES (?,?,?,?)").run(row.path,row.title,row.description,row.body);
224
+ this.indexText(row.path, row.title, row.description, row.body);
186
225
  }
187
226
  for (const row of disk.query<{source:string;ordinal:number;data:string}, []>("SELECT * FROM links").all()) this.db.query("INSERT INTO links VALUES (?,?,?)").run(row.source,row.ordinal,row.data);
188
227
  this.db.run(`PRAGMA user_version = ${Number(disk.query<{ user_version: number }, []>("PRAGMA user_version").get()?.user_version ?? 0)}`);
@@ -202,6 +241,15 @@ export class VaultIndex<Source extends VaultSpec | SpecSourceSpec = VaultSpec> {
202
241
  }
203
242
  }
204
243
  close(): void { this.db.close(); }
244
+ // ponytail: grams stores its bigram text (about 2.3x the CJK bytes); contentless with stable rowids would save it.
245
+ private indexText(path: string, title: string, description: string, body: string): void {
246
+ this.db.query("INSERT INTO fts VALUES (?,?,?,?)").run(path, title, description, body);
247
+ this.db.query("INSERT INTO grams VALUES (?,?,?,?)").run(path, grams(title), grams(description), grams(body));
248
+ }
249
+ private unindex(path: string): void {
250
+ this.db.query("DELETE FROM fts WHERE path=?").run(path);
251
+ this.db.query("DELETE FROM grams WHERE path=?").run(path);
252
+ }
205
253
  private stored(): LinkNote[] {
206
254
  return this.db.query<Omit<LinkNote, "aliases"> & { aliases: string }, []>("SELECT path,title,aliases,body FROM notes ORDER BY path").all().map(row => ({ ...row, aliases: JSON.parse(row.aliases) as string[] }));
207
255
  }
@@ -215,7 +263,7 @@ export class VaultIndex<Source extends VaultSpec | SpecSourceSpec = VaultSpec> {
215
263
  if (!this.adapter && (realpathSync(this.root) !== this.root || realpathSync(notesDir) !== notesDir || !lstatSync(notesDir).isDirectory())) throw new Error(`vault ${this.vault.id}: path or notes directory changed`);
216
264
  const indexed = this.adapter ? new Set(this.db.query<{path:string}, []>("SELECT path FROM notes").all().map(row => row.path)) : undefined;
217
265
  const previous = new Map(this.db.query<{ path: string; hash: string; metadata: string }, []>(this.adapter ? "SELECT path,hash,metadata FROM analysis_notes" : options.captureInvalidFrontmatter ? "SELECT path,hash,metadata FROM notes UNION SELECT path,hash,metadata FROM analysis_notes" : "SELECT path,hash,metadata FROM notes").all().map(note => [note.path, note]));
218
- if (rebuild) this.db.run("DELETE FROM fts");
266
+ if (rebuild) this.db.run("DELETE FROM fts; DELETE FROM grams;");
219
267
  const live = new Set<string>(), present = new Set<string>();
220
268
  let changed = 0, analysisBytes = 0;
221
269
  const skipped: string[] = [], touched = new Set<string>(), keys = new Set<string>();
@@ -298,15 +346,16 @@ export class VaultIndex<Source extends VaultSpec | SpecSourceSpec = VaultSpec> {
298
346
  }
299
347
  const { data, body } = document;
300
348
  touch(path);
349
+ const title = text(data.title) || path.slice(0, -3), description = text(data.description);
301
350
  this.db.query("INSERT OR REPLACE INTO notes VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)").run(path, hash, stamp,
302
- text(data.title) || path.slice(0, -3), text(data.description), body, text(data.type), text(data.domain), JSON.stringify(aliases(data.aliases ?? data.alias)), source);
303
- this.db.query("DELETE FROM fts WHERE path=?").run(path);
304
- this.db.query("INSERT INTO fts(path,title,description,body) SELECT path,title,description,body FROM notes WHERE path=?").run(path);
351
+ title, description, body, text(data.type), text(data.domain), JSON.stringify(aliases(data.aliases ?? data.alias)), source);
352
+ this.unindex(path);
353
+ this.indexText(path, title, description, body);
305
354
  changed++;
306
355
  }
307
356
  for (const row of this.db.query<{path:string}, []>("SELECT path FROM analysis_notes").all()) if (!present.has(row.path)) this.db.query("DELETE FROM analysis_notes WHERE path=?").run(row.path);
308
357
  let removed = 0;
309
- for (const path of indexed ?? previous.keys()) if (!live.has(path)) { touch(path); this.db.query("DELETE FROM notes WHERE path=?").run(path); this.db.query("DELETE FROM fts WHERE path=?").run(path); removed++; }
358
+ for (const path of indexed ?? previous.keys()) if (!live.has(path)) { touch(path); this.db.query("DELETE FROM notes WHERE path=?").run(path); this.unindex(path); removed++; }
310
359
  if (this.adapter) {
311
360
  // Spec links are resolved by frontmatter id in buildSpecGraph, never by vault paths.
312
361
  this.db.run("DELETE FROM links");
@@ -345,13 +394,22 @@ export class VaultIndex<Source extends VaultSpec | SpecSourceSpec = VaultSpec> {
345
394
  search(query: string, options: SearchOptions = {}, snapshot?: ReadonlyMap<string, string>, snapshotOffset = 0, allSnapshotRows = false): SearchResult[] {
346
395
  const limit = resultLimit(options.limit);
347
396
  if (!query.trim() || Buffer.byteLength(query) > 1024) throw new Error("search: query must contain 1..1024 bytes");
348
- // Literal AND terms: users cannot inject FTS operators, columns or SQL.
349
- const terms = query.trim().split(/\s+/).map(term => `"${term.replaceAll('"', '""')}"`).join(" AND ");
397
+ const { tokens, runs } = splitTerms(query.trim().split(/\s+/));
350
398
  const hashColumn = snapshot ? ",n.hash AS sourceHash" : "";
351
- return this.db.query<SearchResult & { sourceHash?: string }, [string, string | null, string | null, string | null, string | null, number]>(`SELECT n.path, n.title, substr(n.description,1,512) AS description, n.type, n.domain,
399
+ const filters = [options.type ?? null, options.type ?? null, options.domain ?? null, options.domain ?? null, snapshot ? 10000 : limit];
400
+ // Latin-only queries run the unicode61 query unchanged. With a CJK term the bigram table drives the match, and a
401
+ // note where every CJK term is a standalone word or sits in the title/description ranks before substring-only hits.
402
+ const strong = runs.map(() => "(n.path IN (SELECT path FROM fts WHERE fts MATCH ?) OR n.path IN (SELECT path FROM grams WHERE grams MATCH ?))").join(" AND ");
403
+ const rows = !runs.length
404
+ ? this.db.query<SearchResult & { sourceHash?: string }, (string | number | null)[]>(`SELECT n.path, n.title, substr(n.description,1,512) AS description, n.type, n.domain,
352
405
  bm25(fts,0,5,2,1) AS score, snippet(fts,3,'','',' ... ',32) AS snippet${hashColumn} FROM fts JOIN notes n ON n.path=fts.path
353
- WHERE fts MATCH ? AND (? IS NULL OR n.type=?) AND (? IS NULL OR n.domain=?) ORDER BY score,n.path LIMIT ?`).all(terms, options.type ?? null, options.type ?? null, options.domain ?? null, options.domain ?? null, snapshot ? 10000 : limit)
354
- .filter(row => !snapshot || snapshot.get(row.path) === row.sourceHash)
406
+ WHERE fts MATCH ? AND (? IS NULL OR n.type=?) AND (? IS NULL OR n.domain=?) ORDER BY score,n.path LIMIT ?`).all(tokens.join(" AND "), ...filters)
407
+ : this.db.query<SearchResult & { sourceHash?: string }, (string | number | null)[]>(`SELECT n.path, n.title, substr(n.description,1,512) AS description, n.type, n.domain,
408
+ bm25(grams,0,5,2,1) AS score, substr(n.body, max(1, instr(n.body, ?) - 48), 160) AS snippet${hashColumn} FROM grams JOIN notes n ON n.path=grams.path
409
+ WHERE grams MATCH ? ${tokens.length ? "AND n.path IN (SELECT path FROM fts WHERE fts MATCH ?)" : ""} AND (? IS NULL OR n.type=?) AND (? IS NULL OR n.domain=?)
410
+ ORDER BY NOT (${strong}),score,n.path LIMIT ?`).all(runs[0]!, runs.map(gramPhrase).join(" AND "), ...(tokens.length ? [tokens.join(" AND ")] : []), ...filters.slice(0, 4),
411
+ ...runs.flatMap(run => [`"${run}"`, `{title description} : ${gramPhrase(run)}`]), filters[4]!);
412
+ return rows.filter(row => !snapshot || snapshot.get(row.path) === row.sourceHash)
355
413
  .slice(snapshot ? snapshotOffset : 0, (snapshot ? snapshotOffset : 0) + (snapshot && allSnapshotRows ? 10000 : limit))
356
414
  .map(({ sourceHash: _, ...row }) => ({ ...row, title: boundedText(row.title, 512).content, snippet: boundedText(row.snippet, 1024).content, type: boundedText(row.type, 256).content, domain: boundedText(row.domain, 256).content }));
357
415
  }
@@ -397,13 +455,51 @@ export class VaultIndex<Source extends VaultSpec | SpecSourceSpec = VaultSpec> {
397
455
  }
398
456
  return { path, hash: note.hash, ...readPage(Buffer.from(note.prefix), maxBytes, start, note.bytes), ...metadata, ...headingMetadata(sections) };
399
457
  }
400
- /** Indexed notes (paths relative to notes/) ordered by path, optionally filtered by a literal stem prefix. */
401
- list(prefix = "", limit = 50): { path: string; title: string; type: string; domain: string; description: string }[] {
458
+ /**
459
+ * Indexed notes (paths relative to notes/) ordered by path, optionally filtered by a literal stem prefix.
460
+ * `cursor` is the previous page's `nextCursor` (its last path, exclusive); `total` counts the filtered set.
461
+ */
462
+ list(prefix = "", limit = 50, cursor?: string): { notes: { path: string; title: string; type: string; domain: string; description: string; mtime: string | null }[]; total: number; nextCursor?: string } {
402
463
  if (Buffer.byteLength(prefix) > 1024) throw new Error("prefix: expected <=1024 bytes");
403
- const key = prefix.replace(/^notes\//i, "");
404
- return this.db.query<{ path: string; title: string; type: string; domain: string; description: string }, [number, string, number]>(
405
- "SELECT path,title,type,domain,substr(description,1,512) AS description FROM notes WHERE substr(path,1,?)=? ORDER BY path LIMIT ?"
406
- ).all(key.length, key, resultLimit(limit)).map(row => ({ ...row, title: boundedText(row.title, 512).content, type: boundedText(row.type, 256).content, domain: boundedText(row.domain, 256).content, description: boundedText(row.description, 512).content }));
464
+ const key = prefix.replace(/^notes\//i, ""), max = resultLimit(limit);
465
+ if (cursor !== undefined && (!cursor || Buffer.byteLength(cursor) > 1024 || !cursor.startsWith(key))) throw new Error("cursor: expected the nextCursor of a previous page listed with the same prefix; list again without cursor to restart");
466
+ const rows = this.db.query<{ path: string; title: string; type: string; domain: string; description: string; metadata: string }, [number, string, string, number]>(
467
+ "SELECT path,title,type,domain,substr(description,1,512) AS description,metadata FROM notes WHERE substr(path,1,?)=? AND path>? ORDER BY path LIMIT ?"
468
+ ).all(key.length, key, cursor ?? "", max + 1);
469
+ const total = this.db.query<{ count: number }, [number, string]>("SELECT count(*) AS count FROM notes WHERE substr(path,1,?)=?").get(key.length, key)!.count;
470
+ const notes = rows.slice(0, max).map(({ metadata: stamp, ...row }) => {
471
+ const mtime = stampMtime(stamp);
472
+ return { ...row, title: boundedText(row.title, 512).content, type: boundedText(row.type, 256).content, domain: boundedText(row.domain, 256).content, description: boundedText(row.description, 512).content, mtime: mtime === null ? null : new Date(mtime).toISOString() };
473
+ });
474
+ return { notes, total, ...(rows.length > max ? { nextCursor: notes.at(-1)!.path } : {}) };
475
+ }
476
+ /**
477
+ * Note counts by type, domain and frontmatter `status` ("(none)" when empty), and how many notes have an mtime at or
478
+ * after each cutoff (epoch ms). Streams rows; returns no note text.
479
+ */
480
+ noteStats(cutoffs: readonly number[]): { total: number; byType: Record<string, number>; byDomain: Record<string, number>; byStatus: Record<string, number>; modified: number[] } {
481
+ const add = (map: Map<string, number>, value: string, count = 1) => { const key = value ? boundedText(value, 256).content : "(none)"; map.set(key, (map.get(key) ?? 0) + count); };
482
+ const sorted = (map: Map<string, number>) => Object.fromEntries([...map].sort(([a], [b]) => a < b ? -1 : a > b ? 1 : 0));
483
+ const grouped = (column: "type" | "domain") => {
484
+ const map = new Map<string, number>();
485
+ for (const row of this.db.query<{ value: string; count: number }, []>(`SELECT ${column} AS value, count(*) AS count FROM notes GROUP BY ${column}`).all()) add(map, row.value, row.count);
486
+ return sorted(map);
487
+ };
488
+ // Status is not a column: stream each stored source once, parsing only its frontmatter.
489
+ const byStatus = new Map<string, number>(), modified = cutoffs.map(() => 0);
490
+ let total = 0;
491
+ const rows = this.db.prepare<{ metadata: string; source: string }, []>("SELECT metadata,source FROM notes");
492
+ try {
493
+ for (const row of rows.iterate()) {
494
+ total++;
495
+ let status: unknown;
496
+ try { status = parseFrontmatter(row.source).data.status; } catch { /* counted as (none) */ }
497
+ add(byStatus, text(status));
498
+ const mtime = stampMtime(row.metadata);
499
+ cutoffs.forEach((cutoff, i) => { if (mtime !== null && mtime >= cutoff) modified[i]!++; });
500
+ }
501
+ } finally { rows.finalize(); }
502
+ return { total, byType: grouped("type"), byDomain: grouped("domain"), byStatus: sorted(byStatus), modified };
407
503
  }
408
504
  /** Links that resolve to the target note, ordered by source. */
409
505
  backlinks(target: string, limit = 50): ResolvedLink[] {
package/src/vault-log.ts CHANGED
@@ -67,6 +67,16 @@ function context(home: string, id: string, channel: "cli" | "mcp" | undefined, w
67
67
  if (isDenied(join(vault.path, "log"), config.denyPaths)) refuse("V06", "log path is denied");
68
68
  return { vault, config, channel, ...(write && vault.registry ? { projectIds: registryIds(vault.registry, config.denyPaths) } : {}) };
69
69
  }
70
+ /** Validate and screen an event exactly as `appendLog` would, writing nothing. Returns the blocking issues (empty when valid). */
71
+ export function checkLogEvent(home: string, request: LogRequest, env: NodeJS.ProcessEnv = process.env): WriteIssue[] {
72
+ try {
73
+ const ctx = context(home, request.vault, request.channel, true);
74
+ return formatLogEvent(request.event, { ...ctx, env }).issues.filter(issue => issue.severity === "error");
75
+ } catch (caught) {
76
+ if (caught instanceof LogRefused) return caught.issues;
77
+ throw caught;
78
+ }
79
+ }
70
80
  /** One O_APPEND write per event. A separate cooperative lock permits safe short-write rollback. */
71
81
  /** Bind a later append to the vault spec, path policy and root identity resolved now (refused with LOG_DRIFT if they change). */
72
82
  export function logBinding(vault: VaultSpec, config: VmemConfig): LogBinding {
@@ -189,7 +199,7 @@ export function readLog(home: string, options: LogReadOptions): LogReadResult {
189
199
  const result: LogReadResult = { ok: false, lines: [], dataLines: null, warnings: [], issues: [] };
190
200
  let fd: number | undefined;
191
201
  try {
192
- if (!options || typeof options !== "object" || typeof options.vault !== "string" || Object.keys(options).some(key => !["vault", "limit", "grep", "type"].includes(key))) refuse("L09", "invalid log read options");
202
+ if (!options || typeof options !== "object" || typeof options.vault !== "string" || Object.keys(options).some(key => !["vault", "limit", "grep", "type", "visit"].includes(key)) || (options.visit !== undefined && typeof options.visit !== "function")) refuse("L09", "invalid log read options");
193
203
  const limit = options.limit ?? 50;
194
204
  if (!Number.isInteger(limit) || limit < 1 || limit > 200) refuse("L09", "log limit must be an integer 1..200");
195
205
  if (options.type !== undefined && !(LOG_TYPES as readonly string[]).includes(options.type)) refuse("L09", "unknown log type filter");
@@ -222,6 +232,7 @@ export function readLog(home: string, options: LogReadOptions): LogReadResult {
222
232
  } else {
223
233
  const line = pending.endsWith("\r") ? pending.slice(0, -1) : pending;
224
234
  const type = /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}\s+(\S+)/.exec(line)?.[1];
235
+ options.visit?.(line);
225
236
  if ((!options.type || options.type === type) && (!filter || filter(line))) {
226
237
  result.lines.push(line); resultBytes += Buffer.byteLength(JSON.stringify(line)) + 1;
227
238
  while (result.lines.length > limit || resultBytes > MAX_LOG_RESULT_BYTES) resultBytes -= Buffer.byteLength(JSON.stringify(result.lines.shift()!)) + 1;
@@ -0,0 +1,50 @@
1
+ import { localDate } from "./local-date.ts";
2
+ import { LOG_TYPES } from "./log-format.ts";
3
+ import { loadRegistry } from "./registry.ts";
4
+ import { VaultIndex } from "./vault-index.ts";
5
+ import { readLog } from "./vault-log.ts";
6
+ import type { WriteIssue } from "./write-types.ts";
7
+ import type { VaultSpec } from "./config.ts";
8
+
9
+ export const STATS_DEFAULT_DAYS = 30;
10
+ export const STATS_MAX_DAYS = 3650;
11
+ export interface VaultStats {
12
+ vault: string;
13
+ /** Local calendar date the windows end on (today included). */
14
+ asOf: string;
15
+ notes: { total: number; byType: Record<string, number>; byDomain: Record<string, number>; byStatus: Record<string, number> };
16
+ /** Notes whose indexed mtime falls within the last 7 and 30 local calendar days, today included. */
17
+ modified: { last7Days: number; last30Days: number };
18
+ /** Log events dated on or after `since` (local date), by type; types outside the known set count as "(other)". */
19
+ log: { days: number; since: string; total: number | null; byType: Record<string, number> | null; warnings: string[]; issues: WriteIssue[] };
20
+ }
21
+ /**
22
+ * Read-only vault statistics over the derived index (refreshed like every read tool) and the log streamed by `readLog`:
23
+ * no vault file or log is written and no vault or log lock is taken.
24
+ */
25
+ export function vaultStats(home: string, id: string, options: { days?: number; now?: Date; expectedVault?: VaultSpec } = {}): VaultStats {
26
+ const days = options.days ?? STATS_DEFAULT_DAYS, now = options.now ?? new Date();
27
+ if (!Number.isInteger(days) || days < 1 || days > STATS_MAX_DAYS) throw new Error(`stats: days must be an integer 1..${STATS_MAX_DAYS}; omit it for the ${STATS_DEFAULT_DAYS}-day default`);
28
+ const registry = loadRegistry(home), vault = registry.vaults.find(v => v.id === id);
29
+ if (!vault) throw Object.assign(new Error("stats: unknown vault id; run vmem vaults list for the registered ids"), { code: "V04" });
30
+ // An MCP server reads the vault set fixed at startup; a re-registered id would otherwise count a different vault.
31
+ if (options.expectedVault && JSON.stringify(options.expectedVault) !== JSON.stringify(vault)) throw new Error("stats: the vault registration changed since the MCP server started; restart the MCP server");
32
+ // Local midnight that opens a window of n calendar days ending today.
33
+ const start = (n: number) => new Date(now.getFullYear(), now.getMonth(), now.getDate() - (n - 1));
34
+ const index = new VaultIndex(home, vault, registry.vaults.map(v => v.path));
35
+ let notes: ReturnType<VaultIndex["noteStats"]>;
36
+ try { index.scan(); notes = index.noteStats([start(7).getTime(), start(30).getTime()]); } finally { index.close(); }
37
+ const since = localDate(start(days)), byType: Record<string, number> = Object.fromEntries(LOG_TYPES.map(type => [type, 0]));
38
+ let total = 0;
39
+ const log = readLog(home, { vault: vault.id, limit: 1, visit: line => {
40
+ const match = /^(\d{4}-\d{2}-\d{2}) \d{2}:\d{2}\s+(\S+)/.exec(line);
41
+ if (!match || match[1]! < since) return;
42
+ const type = (LOG_TYPES as readonly string[]).includes(match[2]!) ? match[2]! : "(other)";
43
+ total++; byType[type] = (byType[type] ?? 0) + 1;
44
+ } });
45
+ const { modified: [last7Days = 0, last30Days = 0], ...counts } = notes;
46
+ return {
47
+ vault: vault.id, asOf: localDate(now), notes: counts, modified: { last7Days, last30Days },
48
+ log: { days, since, total: log.ok ? total : null, byType: log.ok ? byType : null, warnings: log.warnings, issues: log.issues },
49
+ };
50
+ }
@@ -5,7 +5,10 @@ import { loadRegistry } from "./registry.ts";
5
5
  import { basename } from "node:path";
6
6
  import { applyWrite, MAX_WRITE_BYTES } from "./writer.ts";
7
7
  import { notePath } from "./write-operations.ts";
8
- import type { WriteOp, WriteRequest } from "./write-types.ts";
8
+ import type { PatchEdit, WriteOp, WriteRequest } from "./write-types.ts";
9
+ import type { LogEvent } from "./log-types.ts";
10
+ import { LOG_EVENT_SCHEMA } from "./log-format.ts";
11
+ import { argumentError } from "./mcp.ts";
9
12
  import { parseTidyMap, tidyVault, TIDY_RULES } from "./tidy.ts";
10
13
  const COMMANDS = new Set(["new", "patch", "append", "frontmatter", "link", "supersede", "move", "delete", "tidy", "report", "check"]);
11
14
  export function isWriteCommand(command: string): boolean { return COMMANDS.has(command); }
@@ -29,15 +32,24 @@ function input(file: string, home: string): string {
29
32
  return text;
30
33
  } finally { if (file !== "-") closeSync(fd); }
31
34
  }
35
+ /** `--log JSON` checked against the same closed schema as the MCP `log` argument; registry and screening checks follow in applyWrite. */
36
+ function logOption(text: string | undefined): { log?: LogEvent } {
37
+ if (text === undefined) return {};
38
+ let value: unknown;
39
+ try { value = JSON.parse(text); } catch { throw new Error("--log: invalid JSON; expected {\"type\", \"project\", \"source\", \"note\"[, \"dest\"]}"); }
40
+ const error = argumentError(value, LOG_EVENT_SCHEMA, "--log");
41
+ if (error) throw new Error(`${error}; expected {"type", "project", "source", "note"[, "dest"]} as for log append`);
42
+ return { log: value as LogEvent };
43
+ }
32
44
  export function runWriteCommand(command: string, args: string[], home: string, env: NodeJS.ProcessEnv = process.env): { text: string; code: number } {
33
45
  const valueOptions: Record<string, string[]> = {
34
46
  new: ["--type", "--domain", "--topics", "--description", "--title", "--source", "--confidence", "--status", "--extra", "--body"],
35
- patch: ["--old", "--new", "--expected-sha"], append: ["--section", "--markdown", "--expected-sha"],
47
+ patch: ["--old", "--new", "--edits", "--expected-sha"], append: ["--section", "--markdown", "--expected-sha"],
36
48
  frontmatter: ["--set", "--unset", "--expected-sha"], link: ["--relation", "--context"],
37
49
  move: ["--fingerprint"], delete: ["--fingerprint"], tidy: ["--fingerprint", "--map"], report: ["--body"], check: ["--path"], supersede: [],
38
50
  };
39
- // For tidy, --log selects the log-redaction rule; tidy always writes its own LINT event.
40
- const values = valueOptions[command] ? [...valueOptions[command]!, ...(command === "tidy" ? [] : ["--log"])] : undefined;
51
+ // For tidy, --log selects the log-redaction rule; tidy always writes its own LINT event. Check never appends an event.
52
+ const values = valueOptions[command] ? [...valueOptions[command]!, ...(["tidy", "check"].includes(command) ? [] : ["--log"])] : undefined;
41
53
  if (!values) throw new Error("unknown writer command");
42
54
  const twoStep = ["move", "delete", "tidy"].includes(command);
43
55
  const booleans = ["--json", "--dry-run", ...(twoStep ? ["--confirm"] : []), ...(command === "link" ? ["--one-way"] : []), ...(command === "tidy" ? TIDY_RULES.map(rule => `--${rule}`).concat("--all") : [])];
@@ -80,7 +92,16 @@ export function runWriteCommand(command: string, args: string[], home: string, e
80
92
  if (optional("--topics") !== undefined) frontmatter.topics = required("--topics").split(",").filter(Boolean).map(value => value.startsWith("[[") ? value : `[[${value}]]`);
81
93
  operation = { op: "create", path: target, frontmatter, ...(optional("--body") ? { body: input(required("--body"), home) } : {}) }; break;
82
94
  }
83
- case "patch": operation = { op: "patch", path: target, oldString: required("--old"), newString: required("--new"), expectedSha: optional("--expected-sha") }; break;
95
+ case "patch": {
96
+ // --edits takes the same JSON array as the MCP `edits` argument.
97
+ if (optional("--edits") === undefined) { operation = { op: "patch", path: target, oldString: required("--old"), newString: required("--new"), expectedSha: optional("--expected-sha") }; break; }
98
+ if (optional("--old") !== undefined || optional("--new") !== undefined) throw new Error("patch: use --old/--new or --edits, not both");
99
+ const text = input(required("--edits"), home);
100
+ let edits: unknown;
101
+ try { edits = JSON.parse(text); } catch { throw new Error("--edits: invalid JSON; expected an array of {oldString, newString}"); }
102
+ if (!Array.isArray(edits)) throw new Error("--edits: expected a JSON array of {oldString, newString}");
103
+ operation = { op: "patch", path: target, edits: edits as PatchEdit[], expectedSha: optional("--expected-sha") }; break;
104
+ }
84
105
  case "append": operation = { op: "append", path: target, section: required("--section"), markdown: required("--markdown"), expectedSha: optional("--expected-sha") }; break;
85
106
  case "frontmatter": operation = { op: "frontmatter", path: target, set: optional("--set") ? object(required("--set"), "set") : {}, unset: optional("--unset")?.split(","), expectedSha: optional("--expected-sha") }; break;
86
107
  case "link": operation = { op: "link", from: target, to: positional[2]!, relation: required("--relation") as Extract<WriteOp, { op: "link" }>["relation"], context: optional("--context"), bidirectional: !flags["--one-way"] }; break;
@@ -92,7 +113,7 @@ export function runWriteCommand(command: string, args: string[], home: string, e
92
113
  case "check": operation = { op: "check", path: notePath(optional("--path") ?? (target === "-" ? "checked-note" : basename(target))), text: input(target, home) }; break;
93
114
  default: throw new Error("unknown writer command");
94
115
  }
95
- const request: WriteRequest = { vault, operation, channel: "cli", dryRun: flags["--dry-run"] === true, confirm: flags["--confirm"] === true, fingerprint: optional("--fingerprint"), ...(optional("--log") ? { log: object(required("--log"), "log") as unknown as import("./log-types.ts").LogEvent } : {}) };
116
+ const request: WriteRequest = { vault, operation, channel: "cli", dryRun: flags["--dry-run"] === true, confirm: flags["--confirm"] === true, fingerprint: optional("--fingerprint"), ...logOption(optional("--log")) };
96
117
  const result = applyWrite(home, request, { env });
97
118
  return { text: JSON.stringify(result, null, flags["--json"] ? undefined : 2), code: result.ok && result.log?.ok !== false ? 0 : 1 };
98
119
  }
@@ -2,7 +2,7 @@ import { isDeepStrictEqual } from "node:util";
2
2
  import { normalizeSource } from "./source.ts";
3
3
  import { parseFrontmatter, emitFrontmatter } from "./frontmatter.ts";
4
4
  import { normalizeTarget, proseLines, rewriteLinkTargets, extractLinks, resolveLinks, type LinkNote } from "./links.ts";
5
- import type { PlannedChange, WriteIssue, WriteOp } from "./write-types.ts";
5
+ import type { PatchEdit, PlannedChange, WriteIssue, WriteOp } from "./write-types.ts";
6
6
  export class WriteRefusal extends Error {
7
7
  constructor(readonly issues: WriteIssue[]) { super(issues[0]?.message ?? "write refused"); }
8
8
  }
@@ -149,11 +149,28 @@ export function planOperation(operation: WriteOp, notes: Map<string, string>, re
149
149
  }
150
150
  case "patch": {
151
151
  const { path, source } = existing(operation.path);
152
- if (typeof operation.oldString !== "string" || !operation.oldString || typeof operation.newString !== "string" || Buffer.byteLength(operation.oldString) > 8192 || Buffer.byteLength(operation.newString) > 8192) refuse("V09", "patch strings must be bounded and oldString nonempty", path);
153
- const first = source.indexOf(operation.oldString);
154
- if (first < 0 || source.indexOf(operation.oldString, first + 1) >= 0) refuse("V09", "oldString must match exactly once", path);
155
- if(first < (document(source,path).frontmatter?.length ?? 0))refuse("V01","patch cannot target frontmatter; use vmem_set_frontmatter or the frontmatter command",path);
156
- set(path, source.slice(0, first) + operation.newString + source.slice(first + operation.oldString.length));
152
+ const single = operation.oldString !== undefined || operation.newString !== undefined;
153
+ if (single === (operation.edits !== undefined)) refuse("V01", "patch takes either oldString/newString or edits, not both or neither; send one form", path);
154
+ const edits: unknown[] = single ? [{ oldString: operation.oldString, newString: operation.newString }] : Array.isArray(operation.edits) ? operation.edits : [];
155
+ if (edits.length < 1 || edits.length > 32) refuse("V09", "edits must list 1 to 32 replacements; split a larger change into several patches", path);
156
+ const frontmatterEnd = document(source, path).frontmatter?.length ?? 0;
157
+ // Every edit is located in the original text, so earlier edits cannot create or remove a later match.
158
+ const spans = edits.map((edit, index) => {
159
+ const at = single ? "" : `edits[${index}]: `;
160
+ if (!edit || typeof edit !== "object" || Array.isArray(edit) || Object.keys(edit).some(key => key !== "oldString" && key !== "newString")) refuse("V01", `${at}an edit is exactly {oldString, newString}`, path);
161
+ const { oldString, newString } = edit as Partial<PatchEdit>;
162
+ if (typeof oldString !== "string" || !oldString || typeof newString !== "string" || Buffer.byteLength(oldString) > 8192 || Buffer.byteLength(newString) > 8192) refuse("V09", `${at}patch strings must be bounded and oldString nonempty`, path);
163
+ const first = source.indexOf(oldString);
164
+ if (first < 0 || source.indexOf(oldString, first + 1) >= 0) refuse("V09", single ? "oldString must match exactly once" : `${at}oldString must match exactly once in the original note; re-read the note and widen the string`, path);
165
+ if (first < frontmatterEnd) refuse("V01", `${at}patch cannot target frontmatter; use vmem_set_frontmatter or the frontmatter command`, path);
166
+ return { start: first, end: first + oldString.length, text: newString, index };
167
+ }).sort((a, b) => a.start - b.start);
168
+ let after = "", cursor = 0;
169
+ for (const span of spans) {
170
+ if (span.start < cursor) refuse("V09", `edits[${span.index}]: oldString overlaps another edit; merge overlapping edits into one`, path);
171
+ after += source.slice(cursor, span.start) + span.text; cursor = span.end;
172
+ }
173
+ set(path, after + source.slice(cursor));
157
174
  break;
158
175
  }
159
176
  case "append": { const { path, source } = existing(operation.path); set(path, appendSection(source, operation.section, operation.markdown, path)); break; }
@@ -1,10 +1,12 @@
1
1
  import type { PiiSurface } from "./config.ts";
2
2
  import type { LogEvent, LogReceipt } from "./log-types.ts";
3
3
  import type { VaultSpec, VmemConfig } from "./config.ts";
4
+ export interface PatchEdit { oldString: string; newString: string }
4
5
  export type WriteOp =
5
6
  | { op: "check"; path: string; text: string }
6
7
  | { op: "create"; path: string; frontmatter: Record<string, unknown>; body?: string }
7
- | { op: "patch"; path: string; oldString: string; newString: string; expectedSha?: string }
8
+ /** One `oldString`/`newString` pair or `edits` (1..32 pairs), never both; edits apply all-or-nothing against the original text. */
9
+ | { op: "patch"; path: string; oldString?: string; newString?: string; edits?: PatchEdit[]; expectedSha?: string }
8
10
  | { op: "append"; path: string; section: string; markdown: string; expectedSha?: string }
9
11
  | { op: "frontmatter"; path: string; set: Record<string, unknown>; unset?: string[]; expectedSha?: string }
10
12
  | { op: "link"; from: string; to: string; relation: "supports" | "contradicts" | "extends" | "applies-to" | "context-for"; context?: string; bidirectional?: boolean }
package/src/writer.ts CHANGED
@@ -1,4 +1,4 @@
1
- import { appendLog, matchesBinding } from "./vault-log.ts";
1
+ import { appendLog, checkLogEvent, matchesBinding } from "./vault-log.ts";
2
2
  import type { LogEvent } from "./log-types.ts";
3
3
  import { closeSync, constants, fstatSync, fsyncSync, lstatSync, mkdirSync, openSync, readSync, readdirSync, realpathSync, renameSync, rmdirSync, unlinkSync, writeFileSync } from "node:fs";
4
4
  import { createHash, randomUUID } from "node:crypto";
@@ -6,7 +6,7 @@ import { dirname, join } from "node:path";
6
6
  import { hostname } from "node:os";
7
7
  import { Database } from "bun:sqlite";
8
8
  import { loadRegistry, syncDirectory } from "./registry.ts";
9
- import { VaultIndex, MAX_NOTE_BYTES } from "./vault-index.ts";
9
+ import { VaultIndex, MAX_NOTE_BYTES, GRAMS_COLUMNS, gramPhrase, grams, splitTerms } from "./vault-index.ts";
10
10
  import { parseFrontmatter } from "./frontmatter.ts";
11
11
  import { canonicalPath, isDenied } from "./platform.ts";
12
12
  import { moveSourcePath, notePath, planOperation, refuse, WriteRefusal } from "./write-operations.ts";
@@ -102,14 +102,20 @@ function duplicateWarnings(request: WriteRequest, notes: Map<string, string>, ch
102
102
  // Disposable FTS preserves dry-run's no-disk-write contract, including on a fresh home.
103
103
  const db = new Database(":memory:");
104
104
  try {
105
- db.run("CREATE VIRTUAL TABLE candidates USING fts5(path UNINDEXED,title,description,body,tokenize='unicode61')");
106
- const insert = db.query("INSERT INTO candidates VALUES (?,?,?,?)");
105
+ db.run(`CREATE VIRTUAL TABLE candidates USING fts5(path UNINDEXED,title,description,body,tokenize='unicode61');
106
+ CREATE VIRTUAL TABLE grams USING fts5(${GRAMS_COLUMNS},tokenize='unicode61');`);
107
+ const insert = db.query("INSERT INTO candidates VALUES (?,?,?,?)"), insertGrams = db.query("INSERT INTO grams VALUES (?,?,?,?)");
107
108
  for (const [path, source] of notes) {
108
109
  let value: ReturnType<typeof parseFrontmatter>;
109
110
  try { value = parseFrontmatter(source); } catch { continue; } // Unrelated legacy YAML is not a duplicate candidate.
110
- insert.run(path, typeof value.data.title === "string" ? value.data.title : path, typeof value.data.description === "string" ? value.data.description : "", value.body);
111
+ const title = typeof value.data.title === "string" ? value.data.title : path, description = typeof value.data.description === "string" ? value.data.description : "";
112
+ insert.run(path, title, description, value.body); insertGrams.run(path, grams(title), grams(description), grams(value.body));
111
113
  }
112
- const hits = db.query<{ path: string; score: number }, [string]>("SELECT path,bm25(candidates,0,5,2,1) AS score FROM candidates WHERE candidates MATCH ? ORDER BY score,path LIMIT 3").all(terms.map(term => `"${term}"`).join(" AND "));
114
+ // Same term split as VaultIndex.search: CJK terms match substrings through the bigram table.
115
+ const { tokens, runs } = splitTerms(terms);
116
+ const hits = runs.length
117
+ ? db.query<{ path: string; score: number }, string[]>(`SELECT path,bm25(grams,0,5,2,1) AS score FROM grams WHERE grams MATCH ? ${tokens.length ? "AND path IN (SELECT path FROM candidates WHERE candidates MATCH ?)" : ""} ORDER BY score,path LIMIT 3`).all(runs.map(gramPhrase).join(" AND "), ...(tokens.length ? [tokens.join(" AND ")] : []))
118
+ : db.query<{ path: string; score: number }, [string]>("SELECT path,bm25(candidates,0,5,2,1) AS score FROM candidates WHERE candidates MATCH ? ORDER BY score,path LIMIT 3").all(tokens.join(" AND "));
113
119
  return hits.filter(hit => hit.score <= -0.000001).map(hit => ({ code: "V08", severity: "warning", path: hit.path, message: "possible duplicate: FTS title terms match an existing note" }));
114
120
  } finally { db.close(); }
115
121
  }
@@ -136,6 +142,11 @@ export function applyWrite(home: string, request: WriteRequest, options: { env?:
136
142
  root = realpathSync(vault.path);
137
143
  if (root !== vault.path || isDenied(root, config.denyPaths)) refuse("V06", "vault path denied or changed");
138
144
  if (config.vaults.some(value => canonicalPath(home) === value.path || canonicalPath(home).startsWith(`${value.path}/`))) refuse("V04", "VMEM_HOME must be outside registered vaults");
145
+ // A given event is checked like `log append` before any lock or note change, so a bad event never leaves an unlogged write.
146
+ if (request.log !== undefined && request.log !== false && request.operation.op !== "check") {
147
+ const logIssues = checkLogEvent(home, { vault: request.vault, event: request.log, channel: request.channel }, options.env);
148
+ if (logIssues.length) throw new WriteRefusal(logIssues.map(issue => ({ ...issue, message: `log event refused, nothing written: ${issue.message}; fix the event fields (rules as for \`log append\`) or omit the event for the default line` })));
149
+ }
139
150
  const twoStep = ["move", "delete", "tidy"].includes(request.operation.op);
140
151
  const preview = request.operation.op === "check" || request.dryRun === true || (twoStep && request.confirm !== true);
141
152
  if (!preview) {
@@ -159,7 +170,8 @@ export function applyWrite(home: string, request: WriteRequest, options: { env?:
159
170
  for (const change of changes) if (isDenied(join(root, change.path), config.denyPaths)) refuse("V06", "planned mutation path is denied", change.path);
160
171
  issues = validateWrite(changes, { vault, config, notes, operation: request.operation, ...(options.piiSurface ? { piiSurface: options.piiSurface } : {}) });
161
172
  if (!issues.some(issue => issue.severity === "error")) issues.push(...duplicateWarnings(request, notes, changes));
162
- const input = { vault: request.vault, channel: request.channel ?? "cli", operation: request.operation, expectedShas: request.expectedShas ?? {}, log: request.log ?? null };
173
+ // The log event is not part of the fingerprint: move/delete take it on the confirm call only.
174
+ const input = { vault: request.vault, channel: request.channel ?? "cli", operation: request.operation, expectedShas: request.expectedShas ?? {} };
163
175
  const identity = lstatSync(root);
164
176
  // The identity every later commit check uses must still be the bound root, not a directory swapped in meanwhile.
165
177
  if (options.binding && (Number(identity.dev) !== options.binding.dev || Number(identity.ino) !== options.binding.ino)) refuse("VAULT_DRIFT", "vault root changed since the run started; nothing written");