@staix/vault-mem 0.3.3 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +16 -0
- package/README.md +33 -14
- package/package.json +1 -1
- package/src/cli.ts +18 -5
- package/src/commands-table.ts +10 -9
- package/src/log-format.ts +3 -0
- package/src/log-types.ts +5 -1
- package/src/mcp-spec.ts +5 -2
- package/src/mcp-vault.ts +22 -11
- package/src/vault-files.ts +74 -0
- package/src/vault-index.ts +115 -19
- package/src/vault-log.ts +12 -1
- package/src/vault-stats.ts +50 -0
- package/src/write-commands.ts +27 -6
- package/src/write-operations.ts +23 -6
- package/src/write-types.ts +3 -1
- package/src/writer.ts +19 -7
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,22 @@
|
|
|
2
2
|
|
|
3
3
|
## [Unreleased]
|
|
4
4
|
|
|
5
|
+
## [0.4.0] - 2026-11-04
|
|
6
|
+
|
|
7
|
+
- Search, `vmem_search`, spec grep and writer duplicate hints match a Hangul or other CJK term of 2+ characters inside particle and compound forms (`사업단` finds `사업단은`, `사업단장`, `해외사업단`) through a side FTS5 bigram table. Latin terms keep token matching and their ranking; standalone-word and title/description hits rank before substring-only body hits. The index file name gains a schema version (`-links2-schema2.sqlite`), so the first run builds a new index once and older vmem versions keep their own file (#59).
|
|
8
|
+
|
|
9
|
+
- `vmem_patch` takes `edits` (1 to 32 `{oldString, newString}` pairs) as an alternative to the single pair, and `vmem patch --edits file|-` takes the same JSON array. Each `oldString` must match exactly once in the original note, edits must not overlap or start in frontmatter, and all apply or none do, with one receipt, one log event and the 256 KiB bound checked on the combined result. Several rows of one note no longer need one SHA round trip per edit (#63).
|
|
10
|
+
|
|
11
|
+
- Every MCP write tool takes the optional `log` event: `vmem_patch`, `vmem_append`, `vmem_set_frontmatter`, `vmem_move` and `vmem_delete` join `vmem_create`, `vmem_link` and `vmem_supersede` (move and delete use it on the confirm call). CLI writers keep `--log JSON` as the one form, now shown in each command's usage and checked against the same closed schema as the MCP field. A given event is validated and screened like `log append` before any note changes, so an invalid event (bad JSON, unknown key, bad type, unknown project, screened secret) refuses the whole write instead of saving the note and logging nothing; the event is no longer part of the move/delete fingerprint (#60).
|
|
12
|
+
|
|
13
|
+
- `vmem_list` takes a `cursor` and returns the filtered `total`, a per-row `mtime` from the index and `nextCursor`, so a full vault can be paged in stable path order. New read-only `vmem stats <vault>` and `vmem_stats` count notes by type, domain and status, notes modified in the last 7 and 30 days, and log events by type over a bounded window (default 30 days), without writing or locking the vault or its log. `vmem status` and `vmem doctor` no longer print the obsolete `stage: "foundation"` field; scripts that read it should drop that check (#62).
|
|
14
|
+
|
|
15
|
+
- `vmem manifest --json` lists read-only MCP tools as `{name, readOnly: true}` (taken from each tool's `readOnlyHint`), so installers can approve new read tools without a hard-coded list; other tools stay plain names. The release workflow states N-1 per minor line; patch releases carry no N-1 line (#106).
|
|
16
|
+
|
|
17
|
+
- Add MCP `vmem_read_file` and `vmem_list_files`, bounded read-only access to `ops/`, `reports/`, `self/`, `templates/`, `AGENTS.md` and `README.md` with `vmem_read` paging and limits; `notes/`, `log`, dot paths, symlinked components, non-UTF-8 and over-limit files are refused with the tool to use instead. Add `vmem_report`, the MCP form of `vmem report` with the same receipt and log event (#61).
|
|
18
|
+
|
|
19
|
+
- N-1: 0.3.3 runs beside this release. The derived index moves to `<sha>-links2-schema2.sqlite` (spec caches likewise), so 0.3.3 keeps its own `-links2.sqlite` and the first 0.4.0 run builds its index once. The config format, note format and log line format are unchanged, so each version reads what the other writes. MCP and CLI changes are additive for 0.3.3 clients: new tools (`vmem_read_file`, `vmem_list_files`, `vmem_report`, `vmem_stats`), optional `log` and `edits` arguments, and `vmem_list` adds `total`, `nextCursor` and per-row `mtime` beside `notes`. Two behaviors differ: `vmem status` no longer prints `stage`, and a write with an invalid log event is refused instead of saved without its event.
|
|
20
|
+
|
|
5
21
|
## [0.3.3] - 2026-11-02
|
|
6
22
|
|
|
7
23
|
- Polish the spec read surface: scope fish parent completions to the first subcommand, preserve positional help tokens after `--`, document spec MCP tools and startup source registration, clarify shared field/error guidance, reuse body section helpers, and keep literal snippets on code-point boundaries (#103).
|
package/README.md
CHANGED
|
@@ -87,9 +87,9 @@ bun bin/vmem.js links personal "Note or alias" --limit 50 --json
|
|
|
87
87
|
|
|
88
88
|
All four commands require a registered vault id. They read only flat, regular `notes/*.md` files (case-sensitive `.md` extension), skip subdirectories and symlink files, and refuse a replaced vault/notes directory. Notes are limited to 2 MiB each. They never write note content, registry configuration or `.obsidian/`. `rebuild` rebuilds the derived cache; other commands scan metadata on every invocation and rehash new or metadata-changed files. Content-identical changes only update metadata; deleted/renamed paths leave the index. SQLite transactions make each scan atomic. A concurrently changing note aborts that scan, so retry after its writer finishes. No watcher, model calls or embeddings run.
|
|
89
89
|
|
|
90
|
-
The cache is `$VMEM_HOME/indexes/<canonical-vault-path-sha256>-links<N>.sqlite` using the resolved home, not necessarily the environment variable. `N` is the link-resolution version (currently 2), so two installed vmem versions that resolve links differently keep separate caches instead of rebuilding each other's
|
|
90
|
+
The cache is `$VMEM_HOME/indexes/<canonical-vault-path-sha256>-links<N>-schema<S>.sqlite` using the resolved home, not necessarily the environment variable. `N` is the link-resolution version (currently 2) and `S` the index schema version (currently 2, which added the CJK bigram table), so two installed vmem versions that resolve links or store the index differently keep separate caches instead of rebuilding each other's; 0.2.4 and older use the unversioned `<sha256>.sqlite`, 0.3.0 through 0.3.3 use `<sha256>-links2.sqlite`, and a first run after upgrading builds the new file once with no manual step. Cache files use mode 0600 on creation. The location must be outside every registered vault, including through symlinks; linked cache directories/files are refused. It contains note text and is disposable private derived state. Removing it while commands are stopped allows the next read to build it again. SQLite uses its own transaction lock and a five-second busy timeout. There is no automatic migration or live vault cutover.
|
|
91
91
|
|
|
92
|
-
Search uses FTS5 `unicode61` over title, description and body with BM25 weights 5:2:1, ascending score and path tie-breaking. Whitespace-separated query terms are literal AND terms, not a raw FTS expression; quote the query as one shell argument. `--type` and `--domain` are exact scalar frontmatter filters. Results include a bounded snippet and metadata, not the full body. Search queries and note targets accept at most 1024 UTF-8 bytes. Search/links default to 50 results, with an enforced 1..500 range. Show defaults to 16384 UTF-8 bytes and permits 1..65536, returning `truncated` and the full note SHA-256. `--section "Exact H2"` selects one real H2 through the next H1/H2, including its H3 children; code-fenced headings are ignored and duplicate/missing names refuse. Truncated reads include bounded `headings` (at most 64 names/8KiB metadata; `headingsTruncated` when omitted) and `nextOffset`. `--offset N` is a UTF-8 byte offset within the selected section or whole note; use the returned `nextOffset` to page even a single section larger than 64 KiB, and compare note hashes across pages. Invalid UTF-8 boundaries and zero-progress byte budgets refuse. Link string fields are capped at 1024 bytes in command results. Oversized arguments fail instead of silently raising a bound.
|
|
92
|
+
Search uses FTS5 `unicode61` over title, description and body with BM25 weights 5:2:1, ascending score and path tie-breaking. Whitespace-separated query terms are literal AND terms, not a raw FTS expression; quote the query as one shell argument. A term whose letters and digits form one run of 2 or more Hangul, Han, Hiragana or Katakana characters (surrounding punctuation is ignored) matches wherever it occurs in the title, description or body, including particle and compound forms: `사업단` finds `사업단은`, `사업단장` and `해외사업단`. That matching uses a side FTS5 table of overlapping character bigrams (Unicode NFC), so a 2-character term is one indexed lookup and a longer term a phrase of consecutive bigrams; it is substring matching, not morphological analysis. Every other term, including Latin words, single CJK characters and terms mixing CJK with Latin letters or digits (`AI교육`), keeps `unicode61` token matching, and a query without CJK terms ranks exactly as before. With a CJK term, notes where every CJK term is a standalone word or occurs in the title or description rank before notes that contain it only inside a longer body word; within each group BM25 over the bigram table (5:2:1) and path break ties, and the snippet is a body window around the first CJK term. `--type` and `--domain` are exact scalar frontmatter filters. Results include a bounded snippet and metadata, not the full body. Search queries and note targets accept at most 1024 UTF-8 bytes. Search/links default to 50 results, with an enforced 1..500 range. Show defaults to 16384 UTF-8 bytes and permits 1..65536, returning `truncated` and the full note SHA-256. `--section "Exact H2"` selects one real H2 through the next H1/H2, including its H3 children; code-fenced headings are ignored and duplicate/missing names refuse. Truncated reads include bounded `headings` (at most 64 names/8KiB metadata; `headingsTruncated` when omitted) and `nextOffset`. `--offset N` is a UTF-8 byte offset within the selected section or whole note; use the returned `nextOffset` to page even a single section larger than 64 KiB, and compare note hashes across pages. Invalid UTF-8 boundaries and zero-progress byte budgets refuse. Link string fields are capped at 1024 bytes in command results. Oversized arguments fail instead of silently raising a bound.
|
|
93
93
|
|
|
94
94
|
Link resolution uses NFC and case-insensitive filename stems only, as Obsidian and `vmem lint` L01 do. Frontmatter `aliases` (string or string array; legacy `alias` accepted) are not link targets; `show`, `links <target>` and `vmem_read` still find a note by alias. A link that only an alias matches is a missing-note link, so the writer refuses a new one and org promotion drops it to text. An optional `notes/` prefix and `.md` suffix are accepted. `[[target|label]]`, `[[target#heading]]`, `[[target#^block]]`, `![[target]]` and same-note anchors work. ATX/Setext headings and end-of-line block ids are checked; fenced code is skipped. Missing/ambiguous targets and missing anchors remain explicit with a reason. Stems that differ only by case or Unicode normalization are ambiguous. Links returns outgoing links in source order, with one-based body-relative `line` values (frontmatter excluded); it does not write backlinks or repair dead links.
|
|
95
95
|
|
|
@@ -104,6 +104,7 @@ All mutations use `applyWrite(home, request)`: per-vault `mutation.lock`, curren
|
|
|
104
104
|
vmem new personal example-finding --type insight --domain research --topics research --description "A distinct supporting observation."
|
|
105
105
|
vmem new personal another-finding --type insight --domain research --topics research --description "An independently sourced result." --body body.md
|
|
106
106
|
vmem patch personal example-finding --old "This note argues that" --new "This finding states that" --expected-sha <sha256>
|
|
107
|
+
vmem patch personal example-finding --edits edits.json --expected-sha <sha256> # [{"oldString": "...", "newString": "..."}, ...]
|
|
107
108
|
vmem append personal example-finding --section Evidence --markdown "Additional evidence."
|
|
108
109
|
vmem frontmatter personal example-finding --set '{"confidence":"likely"}'
|
|
109
110
|
vmem link personal example-finding another-finding --relation supports
|
|
@@ -117,11 +118,11 @@ vmem tidy personal --connections
|
|
|
117
118
|
vmem tidy personal --all --json # migration sanitize preview (see Migration tidy)
|
|
118
119
|
```
|
|
119
120
|
|
|
120
|
-
Syntax follows indexed reads: the vault id comes first. `new` supports `--title`, `--source`, `--confidence`, `--status`, `--extra JSON` and `--body file|-`; omitted body gets an H1/Relevant Notes/Topics skeleton. Patch accepts nonempty old/new strings up to 8192 bytes and requires exactly one occurrence (including overlapping occurrences). Append uses an existing section or inserts it before Topics; it refuses targeting or introducing a Topics H2. `frontmatter` supports `--set JSON`, `--unset key,key` and `--expected-sha`; changing `topics` also replaces only the body Topics wiki-list items in the same write, preserving other content and line endings; setting an equal topics value leaves the body unchanged; append also supports `--expected-sha`. Link relations are `supports`, `contradicts`, `extends`, `applies-to`, `context-for`; links are bidirectional unless `--one-way`. Supersede sets `status`/`superseded_by` on the old note and adds a reverse link in the next note. Report permits plain paths below `reports/` or `ops/`, is schema-exempt and still screened. Inputs are bounded to 256 KiB, follow no file symlinks and accept stdin with `-`.
|
|
121
|
+
Syntax follows indexed reads: the vault id comes first. `new` supports `--title`, `--source`, `--confidence`, `--status`, `--extra JSON` and `--body file|-`; omitted body gets an H1/Relevant Notes/Topics skeleton. Patch accepts nonempty old/new strings up to 8192 bytes and requires exactly one occurrence (including overlapping occurrences). `--edits file|-` (MCP `edits`) instead takes a JSON array of 1 to 32 `{oldString, newString}` pairs with the same bounds: each `oldString` must match exactly once in the original note, edits must not overlap and none may start in frontmatter. They apply all-or-nothing, so any failing edit (or a stale `--expected-sha`) refuses the whole patch; a success gives one receipt and one log event, and the 256 KiB bound applies to the combined result. Use exactly one form per call; both or neither is refused. Append uses an existing section or inserts it before Topics; it refuses targeting or introducing a Topics H2. `frontmatter` supports `--set JSON`, `--unset key,key` and `--expected-sha`; changing `topics` also replaces only the body Topics wiki-list items in the same write, preserving other content and line endings; setting an equal topics value leaves the body unchanged; append also supports `--expected-sha`. Link relations are `supports`, `contradicts`, `extends`, `applies-to`, `context-for`; links are bidirectional unless `--one-way`. Supersede sets `status`/`superseded_by` on the old note and adds a reverse link in the next note. Report permits plain paths below `reports/` or `ops/`, is schema-exempt and still screened. Inputs are bounded to 256 KiB, follow no file symlinks and accept stdin with `-`.
|
|
121
122
|
|
|
122
123
|
`--dry-run` plans and validates any writer without creating locks, temporary files, directories or caches. `check` always takes this path, even without `--dry-run`. Move, delete and tidy first return a preview with planned path/hash changes and a fingerprint; confirmation requires that fingerprint. It covers canonical operation input, the canonical vault path/identity and policy, expected SHAs, every scanned note and all touched files, so even a new inbound reference after preview forces another preview. Move rewrites visible wiki/heading/block/embed targets and wiki references in frontmatter across the vault, preserving explicit `|label` text, anchors and code. Alias-only links resolve to no note and are left as written. The move source may be an existing note with a legacy file name (lint L13, for example a dot in the stem); the destination must be a valid stem. Tidy folds Connections sections into Relevant Notes. Delete refuses when it would newly break inbound links or MOC topics; fix those references first.
|
|
123
124
|
|
|
124
|
-
Validation returns named issues with severity and no matched secret text: V01 schema/enums/description; V02 existing MOC topics; V03 newly unresolved links; V04 flat ASCII note paths (maximum 200 bytes) and safe report paths; V05 final Topics placement/content (error when the write introduces a placement or content problem the note did not already have, or on create; warning when every problem was already present), with explicit append-to-Topics refusal; V06 pinned agent-mem secret regex parity and denied sources/input/snapshot paths; V07 email/phone/configured PII patterns when denied; V08 create duplicates (stem/NFKC H1); V09 256 KiB prospective note size / 16 MiB report size, exact patch and expected-SHA/drift; V10 supersession target integrity. Unchanged pre-existing broken links warn, including after a move. `possibleDuplicates` lists candidate paths for warnings from a disposable in-memory FTS5 query of title terms, with BM25 <= -0.000001, up to three paths; no disk cache is written for that check. `LOCKED`, `IO`, `ROLLBACK`, `INDEX` and `CLEANUP` identify operational failures. CLI returns exit 1 when `ok` is false.
|
|
125
|
+
Validation returns named issues with severity and no matched secret text: V01 schema/enums/description; V02 existing MOC topics; V03 newly unresolved links; V04 flat ASCII note paths (maximum 200 bytes) and safe report paths; V05 final Topics placement/content (error when the write introduces a placement or content problem the note did not already have, or on create; warning when every problem was already present), with explicit append-to-Topics refusal; V06 pinned agent-mem secret regex parity and denied sources/input/snapshot paths; V07 email/phone/configured PII patterns when denied; V08 create duplicates (stem/NFKC H1); V09 256 KiB prospective note size / 16 MiB report size, exact patch and expected-SHA/drift; V10 supersession target integrity. Unchanged pre-existing broken links warn, including after a move. `possibleDuplicates` lists candidate paths for warnings from a disposable in-memory FTS5 query of title terms, matched like search terms (CJK substrings included), with BM25 <= -0.000001, up to three paths; no disk cache is written for that check. `LOCKED`, `IO`, `ROLLBACK`, `INDEX` and `CLEANUP` identify operational failures. CLI returns exit 1 when `ok` is false.
|
|
125
126
|
|
|
126
127
|
A successful receipt reports `applied`, planned path/hash/byte changes, issues and `indexUpdated`. Unrelated malformed legacy YAML is skipped by duplicate hints and by writer index refresh, with a V01 warning naming the unchanged note; normal read/rebuild commands retain strict parsing. Move/tidy still require parseable inputs and name the offending path on refusal. Reads allocate each note's stat size plus one growth-probe byte, capped at the 2 MiB read limit plus one. A failed index refresh rolls back the file changes. Rollback first verifies each file still equals this operation's committed result; external edits are preserved. Failed/refused rollback retains the original backup and reports its relative recovery path. All files are staged before replacement and backups permit rollback of caught multi-file failures. Each individual rename is atomic; a machine/process crash between multiple renames can leave a partially completed batch. The retained lock and `.vmem-<uuid>.tmp` files require inspection and manual recovery after confirming the writer stopped. They are never automatically removed by registry recovery. Uncoordinated editors do not honor the lock; the pipeline rechecks SHAs before replacement, but this does not provide a filesystem transaction against concurrent external edits. Existing notes may be read up to the index's 2 MiB limit; prospective notes are limited to 256 KiB; report operations under `reports/` and `ops/` allow up to 16 MiB for both prospective content and existing-file reads (including staging and rollback rechecks), with at most 10000 flat notes per mutation snapshot. File/directory fsync has the platform guarantees described above. Successful changed mutations now append an event as described below; log failure is reported separately from note success.
|
|
127
128
|
|
|
@@ -132,6 +133,7 @@ A successful receipt reports `applied`, planned path/hash/byte changes, issues a
|
|
|
132
133
|
vmem log append personal EXTRACT - synthetic:source "A synthetic event" --dest notes/example.md
|
|
133
134
|
vmem log tail personal --limit 50 --json
|
|
134
135
|
vmem log grep personal "EXTRACT|CONNECT" --type EXTRACT --limit 20
|
|
136
|
+
vmem patch personal example-finding --old "argues" --new "states" --log '{"type":"EXTRACT","project":"-","source":"synthetic:source","note":"Reworded claim"}'
|
|
135
137
|
```
|
|
136
138
|
|
|
137
139
|
Events keep the canonical local-time format `YYYY-MM-DD HH:MM TYPE project source → dest — note` (two spaces between fields, one LF at the end). Destination is optional. Allowed new types: `INGEST ROUTE EXTRACT CONNECT DIGEST LEARN LINT TASK GRAPH SYNC RETHINK SOURCE`. Project must be `-` or a lowercase alphanumeric/hyphen id; when `registry` is configured, ids must exist in its bounded YAML `projects`/nested `sub_projects` arrays. Registry files are limited to 1 MiB, eight nesting levels and 10000 unique ids; invalid or unavailable configuration fails closed. Invalid types/fields/projects use L09 issues, the 1000-byte line bound uses V09, and secret/PII refusals use V06/V07. Fields cannot inject pipes, control characters or two-space/column separators. Validation and screening run before a file or lock is created. The 1000 UTF-8-byte bound includes the LF and attribution.
|
|
@@ -140,10 +142,21 @@ Every MCP log append and CLI call with a nonempty `AGENTHUB_PEER_ID`, `CLAUDECOD
|
|
|
140
142
|
|
|
141
143
|
`appendLog` validates the registered canonical vault and write/channel policy, acquires a separate `log.lock` (bounded two-second contention wait), then makes one O_APPEND write and fsyncs the log. New log files are mode 0600 and their parent directory is flushed. Linked/nonregular files and an incomplete existing final line are refused. Cooperating appenders share the lock; a short write is truncated back only when the same file still has the expected size, and its tail still equals our exact prefix, then flushed. Error-after-write cases inspect at most one event's bytes. Failed/ambiguous writes or cleanup are explicit; the receipt reports `appended` (possibly null when unknown), `synced`, `bytesWritten` (possibly null) and `rolledBack` when a partial append was removed. Before reporting sync, the current log and vault identities are rechecked; rotation produces LOG_DRIFT and an unknown current-path append state. A synced event can coexist with a cleanup error. No existing event is rewritten. A crash can retain `log.lock`, and failed short-write recovery can retain an incomplete line; inspect the file and stopped writer before manual recovery. There is no automatic cleanup or new TTY requirement.
|
|
142
144
|
|
|
143
|
-
Successful changed mutations default to EXTRACT, link to CONNECT, move to ROUTE, and supersede/tidy to RETHINK. Default events use project `-`, affected paths and the operation name rather than note text. Library callers can supply `WriteRequest.log` to override the event or `false` to disable it; CLI writers
|
|
145
|
+
Successful changed mutations default to EXTRACT, link to CONNECT, move to ROUTE, and supersede/tidy to RETHINK. Default events use project `-`, affected paths and the operation name rather than note text. Library callers can supply `WriteRequest.log` to override the event or `false` to disable it. Every MCP write tool (`vmem_create`, `vmem_patch`, `vmem_append`, `vmem_set_frontmatter`, `vmem_link`, `vmem_supersede`, `vmem_move`, `vmem_delete`) takes the same optional `log` object (`type`, `project`, `source`, `note`, optional `dest`); move and delete use it on the confirm call. The CLI writers `new`, `patch`, `append`, `frontmatter`, `link`, `supersede`, `move`, `delete` and `report` take the same object as `--log JSON`, checked against the same closed schema (unknown keys, a missing field or a bad type are refused); omitting it keeps the default event. `tidy --log` is the unrelated redaction rule. Preview/check/no-op/refused mutations never append. The append is bound to the mutation's resolved vault identity/policy, so re-registration cannot route its event to another vault; the same binding covers the tidy LINT event and both SYNC events of org promotion, which are refused with LOG_DRIFT instead. A bound append compares the registry before any policy or path check, so a re-registration always reads as LOG_DRIFT. Tidy and org promotion also bind every note write to the vaults resolved at the start of the run: if a vault id is re-registered before a write, that write is refused with VAULT_DRIFT and nothing is written to either vault. After a committed mutation, log append is independent: `ok`/`applied` describe the note operation, `log` holds its append receipt, and `logged` is true only when the event was appended and synced. A log error does not roll back a successfully saved note. CLI exits 1 for a log failure even when the note receipt says `ok: true`; inspect the receipt before retrying to avoid duplicate mutations/events. A given event is validated and screened like `log append` (type, registry project id, fields, secret/PII screening, line bound) before any lock or note change, also on dry runs and previews: an invalid event refuses the whole write with its L09/V06/V07/V09 issue and nothing changes. The event is not part of the move/delete fingerprint.
|
|
144
146
|
|
|
145
147
|
Readers never create a home, log or lock, and work with `write: none`. They skip blank/comment lines, return the last matching 1..200 records (default 50), accept an optional canonical TYPE and non-backtracking regex subset up to 256 characters (`|`, `^`, `$`, `.`, character classes, `\d`/`\s`/`\w` and their inverses, escaped literals, and atom `*`/`+`/`?`; groups, backreferences and brace quantifiers are refused), and scan a fixed descriptor-size snapshot. Streaming is capped at 32 MiB; individual historical lines over 4096 bytes and incomplete final lines are omitted with warnings, and the serialized line array stays within 128 KiB. Above 20000 data lines, the manual-archival warning is preserved. Read failure returns `dataLines: null` rather than claiming a count. Golden synthetic fixtures and concurrent/fault persistence tests cover the format and actual append state; no live vault log was read or modified for this increment.
|
|
146
148
|
|
|
149
|
+
# Vault statistics
|
|
150
|
+
|
|
151
|
+
```sh
|
|
152
|
+
vmem stats personal --json # log window: last 30 days
|
|
153
|
+
vmem stats personal --days 7 --json
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
`vmem stats <vault>` and `vmem_stats` report note counts (total, by `type`, `domain` and frontmatter `status`; empty values count as `(none)`), notes whose indexed mtime falls in the last 7 and 30 local calendar days (today included), and log event counts by type dated on or after `log.since`, the first local date of the `--days` window (1..3650, default 30). Known types always appear, unknown ones count as `(other)`. The derived index cache is refreshed as by every read command and the log is streamed with the bounded log reader; no vault file or log is written and no vault or log lock is taken. When the log cannot be read, `log.total` and `log.byType` are null, `log.issues` names the cause and the CLI exits 1.
|
|
157
|
+
|
|
158
|
+
`vmem_list` pages the whole vault: pass the returned `nextCursor` as `cursor` to get the next page in path order. The cursor is the previous page's last path (exclusive), so a note added or removed between pages does not repeat or skip the others; a cursor outside the `prefix` filter is refused.
|
|
159
|
+
|
|
147
160
|
# MCP server
|
|
148
161
|
|
|
149
162
|
`vmem mcp` serves newline-delimited JSON-RPC 2.0 (MCP tools, protocol 2025-06-18 and earlier) on stdio. stdout carries protocol messages only; it needs no TTY and runs in agent sessions. Requests are limited to 64 KiB per frame and results to 256 KiB (a larger result becomes a structured `LIMIT` refusal). Arguments are validated against each tool's bounded schema (`additionalProperties: false`, string/array/integer caps); invalid arguments or unknown tools return JSON-RPC `-32602`, unknown methods `-32601`, malformed frames `-32700`/`-32600`. At startup it loads every module a tool can run, the `@staix/agent-mem` root included, so a server keeps its own code and tool list when its install is replaced or a newer release is installed beside it; restart the host session to pick up the new release.
|
|
@@ -160,26 +173,32 @@ The vault set is read once at startup: every tool's `vault` property is an enum
|
|
|
160
173
|
| `vmem_search` | `query`<=1000, `vault`, `limit`<=50, `type`, `domain` |
|
|
161
174
|
| `vmem_read` | `path` (stem, alias or `notes/` path), `vault`, `maxBytes`<=65536; returns `{path, frontmatter, body, sha256, truncated}` |
|
|
162
175
|
| `vmem_read_many` | `paths`<=20 (8 KiB each), `vault` |
|
|
163
|
-
| `vmem_list` | `vault`, `prefix`, `limit`<=500 |
|
|
176
|
+
| `vmem_list` | `vault`, `prefix`, `limit`<=500, `cursor`; rows in path order with `mtime`, the filtered `total` and `nextCursor` while more remain |
|
|
164
177
|
| `vmem_links` | `path`, `vault`, `direction` out/in/both, `limit`<=200; returns `{out, dead, in}` |
|
|
165
178
|
| `vmem_mocs` | `vault`; MOC notes with incoming link counts |
|
|
166
179
|
| `vmem_check` | `vault`, `path`, `text`<=256 KiB; validation only |
|
|
167
180
|
| `vmem_log_tail` | `vault`, `n`<=200, `type`, `grep`<=256 |
|
|
168
181
|
| `vmem_create` | `stem`, `type`, `domain`, `description`, `topics[]`<=32, `title`, `body`<=65536, `source`, `confidence`, `status`, `extra`, `log` |
|
|
169
|
-
| `vmem_patch` | `path`, `oldString`<=8192
|
|
170
|
-
| `vmem_append` | `path`, `section`<=120, `markdown`<=16384, `expectedSha` |
|
|
171
|
-
| `vmem_set_frontmatter` | `path`, `set`, `unset[]`, `expectedSha` |
|
|
182
|
+
| `vmem_patch` | `path`, `oldString`<=8192 and `newString`<=8192, or `edits[]` 1..32 of `{oldString, newString}` (all-or-nothing), `expectedSha`, `log` |
|
|
183
|
+
| `vmem_append` | `path`, `section`<=120, `markdown`<=16384, `expectedSha`, `log` |
|
|
184
|
+
| `vmem_set_frontmatter` | `path`, `set`, `unset[]`, `expectedSha`, `log` |
|
|
172
185
|
| `vmem_link` | `from`, `to`, `relation`, `context`, `bidirectional` (default true), `log` |
|
|
173
186
|
| `vmem_supersede` | `old`, `next`, `log` |
|
|
174
187
|
| `vmem_log_append` | `type`, `project`, `source`, `dest`, `note`<=500 |
|
|
175
|
-
| `vmem_move` | `from`, `to`, `confirm`, `fingerprint` (two-step) |
|
|
176
|
-
| `vmem_delete` | `path`, `confirm`, `fingerprint` (two-step; the preview lists backlinks) |
|
|
188
|
+
| `vmem_move` | `from`, `to`, `confirm`, `fingerprint` (two-step), `log` (on confirm) |
|
|
189
|
+
| `vmem_delete` | `path`, `confirm`, `fingerprint` (two-step; the preview lists backlinks), `log` (on confirm) |
|
|
177
190
|
| `vmem_promote` | `path` (in the personal vault), `to` (enum of org vaults), `topics[]`, `again`, `confirm`, `fingerprint` (two-step; never pushes) |
|
|
178
191
|
| `vmem_skills_mine` | `vault`, `days`, `project`, `allProjects`, `vocab`, `skills[]`, `minSize`, `minNovelty` (read-only; see Skill mining) |
|
|
192
|
+
| `vmem_stats` | `vault`, `days`<=3650 (default 30); read-only counts, see Vault statistics |
|
|
179
193
|
| `vmem_promote_from_amem` | `id`, `project`, `stem`, `type`, `domain`, `topics[]`, `description`, `title`, `body`<=65536, `linkTo`, `confirm`, `fingerprint` (two-step) |
|
|
194
|
+
| `vmem_read_file` | `path`, `vault`, `maxBytes`<=65536, `offset`<=2097152; returns `{path, sha256, truncated, body, offset?, nextOffset?}` (read-only; allowed roots below the table) |
|
|
195
|
+
| `vmem_list_files` | `dir`, `vault`; one level, <=500 entries `{name, path, type, bytes?}` (read-only) |
|
|
196
|
+
| `vmem_report` | `path` under `reports/` or `ops/`, `body`<=65536, `dryRun`, `log`; the receipt of `vmem report` |
|
|
180
197
|
|
|
181
198
|
Move and delete without `confirm` write nothing and return the preview and `fingerprint`; `confirm: true` requires the fingerprint recomputed on the current state, otherwise the call is refused with `V09`. `vmem_promote_from_amem` is described under Promotion from agent-mem. `vmem_promote` is described under Org promotion and sync. Graph/lint tools are listed below. Note paths in read results are relative to `notes/`.
|
|
182
199
|
|
|
200
|
+
File reads and reports: `vmem_read_file` and `vmem_list_files` read only these roots: `ops/`, `reports/`, `self/`, `templates/`, `AGENTS.md` and `README.md`. They refuse `notes/` (use `vmem_read`), `log` (use `vmem_log_tail`), absolute paths, `.` or `..` segments, dot paths such as `.git`, any symlinked component, denied paths, non-UTF-8 files and files over 2 MiB. Paging, `maxBytes` and `offset` match `vmem_read`; listings leave out dot, linked, special and denied entries. Neither tool takes a lock, touches the index or writes a file. `vmem_report` is `vmem report` over MCP: the same path rules, screening, receipt and default `EXTRACT` event (a `log` override sets another event, for example `LINT` or `GRAPH`), with `write: mcp` required; MCP events always carry the agent attribution.
|
|
201
|
+
|
|
183
202
|
# Host setup
|
|
184
203
|
|
|
185
204
|
```sh
|
|
@@ -196,7 +215,7 @@ Apply is idempotent and touches only the `vmem` entry: other servers and setting
|
|
|
196
215
|
|
|
197
216
|
`vmem completion bash|zsh|fish` prints a completion script generated from the same command table as `--help`.
|
|
198
217
|
|
|
199
|
-
`vmem manifest --json` prints `{name, version, configVersion, mcp: {server, tools}, commands}`: the package version, the config format it writes, the MCP
|
|
218
|
+
`vmem manifest --json` prints `{name, version, configVersion, mcp: {server, tools}, commands}`: the package version, the config format it writes, the MCP tools and the CLI command names from the same table, for tools that would otherwise hard-code them. A read-only tool (its `readOnlyHint` annotation) is listed as `{name, readOnly: true}`, every other tool as its plain name.
|
|
200
219
|
|
|
201
220
|
# Promotion from agent-mem
|
|
202
221
|
|
|
@@ -291,7 +310,7 @@ vmem sync team --confirm --fingerprint <sha>
|
|
|
291
310
|
|
|
292
311
|
- `bun run parity:release` checks same-snapshot frontmatter bytes, incremental notes/FTS/links against clean rebuild, and two consecutive synthetic Python-oracle lint/graph comparisons. External Python reference paths can be supplied with `--lint-reference` and `--graph-reference`; `--output` records aggregate evidence.
|
|
293
312
|
- `bun run release:verify` enforces committed runtime/method/external-parity evidence freshness and the declared synthetic thresholds at the explicit release gate. Ordinary tests validate evidence shape and internal sample consistency; routine source edits and version bumps do not require fresh benchmark evidence until release validation.
|
|
294
|
-
- `bun run bench:release --output <file.json>` measures a generated 5000-note corpus in the authorized Linux dgx02 container. It records actual MCP search round-trip timings (including metadata refresh), kernel RSS high-water, raw samples and environment; it exits nonzero when a synthetic threshold fails. Synthetic write timings are diagnostic and do not establish representative private-vault acceptance.
|
|
313
|
+
- `bun run bench:release --output <file.json>` measures a generated 5000-note corpus in the authorized Linux dgx02 container. It records actual MCP search round-trip timings (including metadata refresh), kernel RSS high-water, the derived index size after rebuild (`indexBytes`), raw samples and environment; it exits nonzero when a synthetic threshold fails. Synthetic write timings are diagnostic and do not establish representative private-vault acceptance.
|
|
295
314
|
- [Readiness evidence](docs/release-readiness.md), [release workflow](docs/release-workflow.md) and [cutover checklist](docs/cutover-checklist.md) distinguish verified results from outstanding private/native/integration/deployment gates. Live cutover remains blocked.
|
|
296
315
|
|
|
297
316
|
|
|
@@ -362,7 +381,7 @@ vmem spec validate --repo /absolute/checkout --require-fresh --json
|
|
|
362
381
|
|
|
363
382
|
`get` returns frontmatter, resolved outgoing links, backlinks, children, path, SHA-256 and computed freshness. Body reads are optional (`--body`, or body paging/section options) and use byte offsets with bounded UTF-8 pages. `graph` accepts subtree, ancestors or neighbors and an optional edge kind. `grep` searches FTS terms by default or a literal substring with `--literal`; filters are type, status, tag, parent, depends-on and covers-path. Use `--` before a positional id/query beginning with a hyphen, for example `vmem spec grep --literal -- "- [ ]"`. Results carry a continuation `nextOffset` when more matching hits exist. Snapshot path/hash filtering occurs before search limit/offset, so another scanner's cache rows cannot shorten pages. Rerun a command to refresh its snapshot. Large responses refuse with guidance or mark bounded collections as truncated.
|
|
364
383
|
|
|
365
|
-
`validate` exits 1 on validation errors; stale scopes become errors only with `--require-fresh`. `--repo` validates a checkout without consulting a home/config file or creating a registry/cache, and cannot be combined with `--source` or `--worktree`. The four read tools are listed by `vmem manifest --json`;
|
|
384
|
+
`validate` exits 1 on validation errors; stale scopes become errors only with `--require-fresh`. `--repo` validates a checkout without consulting a home/config file or creating a registry/cache, and cannot be combined with `--source` or `--worktree`. The four read tools are listed by `vmem manifest --json` as `{name, readOnly: true}`; hosts that approve from a frozen allow-list instead of the manifest need to add their names separately.
|
|
366
385
|
|
|
367
386
|
# Spec validation and freshness library
|
|
368
387
|
|
package/package.json
CHANGED
package/src/cli.ts
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { runSpecCommand } from "./mcp-spec.ts";
|
|
1
|
+
import { runSpecCommand, specTools } from "./mcp-spec.ts";
|
|
2
2
|
import { tidyVault, TIDY_RULES } from "./tidy.ts";
|
|
3
3
|
import { originCheck } from "./sync.ts";
|
|
4
4
|
import { registryIds } from "./vault-log.ts";
|
|
@@ -9,12 +9,13 @@ import { isWriteCommand, runWriteCommand } from "./write-commands.ts";
|
|
|
9
9
|
import { isReadCommand, runReadCommand } from "./read-commands.ts";
|
|
10
10
|
import { version } from "../package.json";
|
|
11
11
|
import { COMMANDS, completionScript, help } from "./commands-table.ts";
|
|
12
|
-
import { VMEM_TOOL_NAMES } from "./mcp-vault.ts";
|
|
12
|
+
import { VMEM_TOOL_NAMES, vaultTools } from "./mcp-vault.ts";
|
|
13
13
|
import { CONFIG_VERSION } from "./config.ts";
|
|
14
14
|
import { runSetup, type SetupMode } from "./setup.ts";
|
|
15
15
|
import { resolveHome } from "./home.ts";
|
|
16
16
|
import { addVault, addSpec, removeSpec, specStatus, effectivePii, loadRegistry, recoverRegistry, registryStatus, removeVault, setVaultPii } from "./registry.ts";
|
|
17
17
|
import { piiPolicy } from "./pii.ts";
|
|
18
|
+
import { vaultStats } from "./vault-stats.ts";
|
|
18
19
|
function parse(args: string[], allowed: string[], values: string[]) {
|
|
19
20
|
const flags: Record<string, string | boolean> = {}, positional: string[] = [];
|
|
20
21
|
for (let i = 0; i < args.length; i++) {
|
|
@@ -45,7 +46,10 @@ export function run(args: string[], env: NodeJS.ProcessEnv = process.env): { tex
|
|
|
45
46
|
if (args[0] === "manifest") {
|
|
46
47
|
const { flags, positional } = parse(args.slice(1), ["--json"], []);
|
|
47
48
|
if (positional.length) throw new Error("manifest: unexpected arguments");
|
|
48
|
-
|
|
49
|
+
// Read-only tools are published as { name, readOnly: true } so installers can approve them without a hard-coded list.
|
|
50
|
+
const readOnly = new Set([...vaultTools([]), ...specTools()].filter(t => t.annotations.readOnlyHint).map(t => t.name));
|
|
51
|
+
const tools = VMEM_TOOL_NAMES.map(name => readOnly.has(name) ? { name, readOnly: true } : name);
|
|
52
|
+
const manifest = { name: "@staix/vault-mem", version, configVersion: CONFIG_VERSION, mcp: { server: "vmem", tools }, commands: COMMANDS.map(c => c.name) };
|
|
49
53
|
return { text: JSON.stringify(manifest, null, flags["--json"] ? undefined : 2), code: 0 };
|
|
50
54
|
}
|
|
51
55
|
if (args[0] === "setup") {
|
|
@@ -68,7 +72,7 @@ export function run(args: string[], env: NodeJS.ProcessEnv = process.env): { tex
|
|
|
68
72
|
const group = args[0] === "vaults" || args[0] === "specs";
|
|
69
73
|
const command = group ? `${args[0]} ${args[1] ?? ""}` : args[0];
|
|
70
74
|
const tail = args.slice(group ? 2 : 1);
|
|
71
|
-
const options = command === "vaults add" ? ["--kind", "--write", "--pii", "--remote", "--registry"] : command === "vaults pii" ? ["--set"] : command === "specs add" ? ["--write", "--pii"] : [];
|
|
75
|
+
const options = command === "vaults add" ? ["--kind", "--write", "--pii", "--remote", "--registry"] : command === "vaults pii" ? ["--set"] : command === "specs add" ? ["--write", "--pii"] : command === "stats" ? ["--days"] : [];
|
|
72
76
|
const { flags, positional } = parse(tail, ["--json", ...options, ...(command === "vaults remove" || command === "specs remove" || command === "vaults pii" || command === "recover" ? ["--confirm"] : [])], options);
|
|
73
77
|
const home = resolveHome(env);
|
|
74
78
|
let result: unknown, code = 0;
|
|
@@ -105,10 +109,19 @@ export function run(args: string[], env: NodeJS.ProcessEnv = process.env): { tex
|
|
|
105
109
|
});
|
|
106
110
|
const sources = command === "doctor" && config.specs !== undefined ? specStatus(home.home, env) : undefined;
|
|
107
111
|
const healthy = vaults.every(v => v.healthy && (!v.checks || Object.values(v.checks).every(check => check.ok))) && (!sources || sources.every(source => source.healthy));
|
|
108
|
-
result = { ...home, version,
|
|
112
|
+
result = { ...home, version, healthy, vaults, ...(config.specs !== undefined ? { specs: sources ?? config.specs } : {}) };
|
|
109
113
|
if (command === "doctor" && !healthy) code = 1;
|
|
110
114
|
break;
|
|
111
115
|
}
|
|
116
|
+
case "stats": {
|
|
117
|
+
if (positional.length !== 1) throw new Error("stats: requires one vault id; use vmem stats <vault> [--days 1..3650]");
|
|
118
|
+
const days = flags["--days"];
|
|
119
|
+
if (days !== undefined && !/^[0-9]+$/.test(String(days))) throw new Error("stats: --days expects an integer 1..3650; omit it for the 30-day default");
|
|
120
|
+
const stats = vaultStats(home.home, positional[0]!, days === undefined ? {} : { days: Number(days) });
|
|
121
|
+
result = stats;
|
|
122
|
+
if (stats.log.issues.length) code = 1;
|
|
123
|
+
break;
|
|
124
|
+
}
|
|
112
125
|
case "recover":
|
|
113
126
|
if (positional.length) throw new Error("recover: unexpected arguments");
|
|
114
127
|
result = recoverRegistry(home.home, flags["--confirm"] === true);
|
package/src/commands-table.ts
CHANGED
|
@@ -19,16 +19,16 @@ export const COMMANDS = [
|
|
|
19
19
|
{ name: "specs list", usage: "specs list [--json]", mutates: false },
|
|
20
20
|
{ name: "specs add", usage: "specs add <id> <absolute-path> [--write mcp|cli|none] [--pii allow|deny] [--json]", mutates: true },
|
|
21
21
|
{ name: "specs remove", usage: "specs remove <id> [--confirm] [--json]", mutates: "confirm" },
|
|
22
|
-
{ name: "new", usage: "new <vault> <stem> --type TYPE --domain DOMAIN --topics a,b --description TEXT [--title TEXT] [--source TEXT|JSON] [--confidence TEXT] [--status TEXT] [--body file|-] [--extra JSON] [--dry-run] [--json]", mutates: true },
|
|
23
|
-
{ name: "patch", usage: "patch <vault> <note> --old TEXT --new TEXT [--expected-sha SHA] [--dry-run] [--json]", mutates: true },
|
|
24
|
-
{ name: "append", usage: "append <vault> <note> --section SECTION --markdown TEXT [--expected-sha SHA] [--dry-run] [--json]", mutates: true },
|
|
25
|
-
{ name: "frontmatter", usage: "frontmatter <vault> <note> [--set JSON] [--unset key,key] [--expected-sha SHA] [--dry-run] [--json]", mutates: true },
|
|
26
|
-
{ name: "link", usage: "link <vault> <from> <to> --relation R [--context TEXT] [--one-way] [--dry-run] [--json]", mutates: true },
|
|
27
|
-
{ name: "supersede", usage: "supersede <vault> <old> <next> [--dry-run] [--json]", mutates: true },
|
|
28
|
-
{ name: "move", usage: "move <vault> <from> <to> [--confirm --fingerprint SHA] [--dry-run] [--json]", mutates: "confirm" },
|
|
29
|
-
{ name: "delete", usage: "delete <vault> <note> [--confirm --fingerprint SHA] [--dry-run] [--json]", mutates: "confirm" },
|
|
22
|
+
{ name: "new", usage: "new <vault> <stem> --type TYPE --domain DOMAIN --topics a,b --description TEXT [--title TEXT] [--source TEXT|JSON] [--confidence TEXT] [--status TEXT] [--body file|-] [--extra JSON] [--log JSON] [--dry-run] [--json]", mutates: true },
|
|
23
|
+
{ name: "patch", usage: "patch <vault> <note> [--old TEXT --new TEXT] [--edits file|-] [--expected-sha SHA] [--log JSON] [--dry-run] [--json]", mutates: true },
|
|
24
|
+
{ name: "append", usage: "append <vault> <note> --section SECTION --markdown TEXT [--expected-sha SHA] [--log JSON] [--dry-run] [--json]", mutates: true },
|
|
25
|
+
{ name: "frontmatter", usage: "frontmatter <vault> <note> [--set JSON] [--unset key,key] [--expected-sha SHA] [--log JSON] [--dry-run] [--json]", mutates: true },
|
|
26
|
+
{ name: "link", usage: "link <vault> <from> <to> --relation R [--context TEXT] [--one-way] [--log JSON] [--dry-run] [--json]", mutates: true },
|
|
27
|
+
{ name: "supersede", usage: "supersede <vault> <old> <next> [--log JSON] [--dry-run] [--json]", mutates: true },
|
|
28
|
+
{ name: "move", usage: "move <vault> <from> <to> [--confirm --fingerprint SHA] [--log JSON] [--dry-run] [--json]", mutates: "confirm" },
|
|
29
|
+
{ name: "delete", usage: "delete <vault> <note> [--confirm --fingerprint SHA] [--log JSON] [--dry-run] [--json]", mutates: "confirm" },
|
|
30
30
|
{ name: "tidy", usage: "tidy <vault> [--connections] [--escapes] [--enums] [--keys] [--sources] [--topics] [--secrets] [--log] [--all] [--map file] [--confirm --fingerprint SHA] [--dry-run] [--json]", mutates: "confirm" },
|
|
31
|
-
{ name: "report", usage: "report <vault> <reports/path|ops/path> --body file|- [--dry-run] [--json]", mutates: true },
|
|
31
|
+
{ name: "report", usage: "report <vault> <reports/path|ops/path> --body file|- [--log JSON] [--dry-run] [--json]", mutates: true },
|
|
32
32
|
{ name: "check", usage: "check <vault> <file|-> [--path stem] [--json]", mutates: false },
|
|
33
33
|
{ name: "log tail", usage: "log tail <vault> [--limit 1..200] [--type TYPE] [--json]", mutates: false },
|
|
34
34
|
{ name: "log grep", usage: "log grep <vault> <pattern> [--limit 1..200] [--type TYPE] [--json]", mutates: false },
|
|
@@ -40,6 +40,7 @@ export const COMMANDS = [
|
|
|
40
40
|
{ name: "setup", usage: "setup <claude|codex|kimi|pi|qwen|jcode|agy|all> [--root PATH] [--apply|--undo] [--json]", mutates: "apply" },
|
|
41
41
|
{ name: "manifest", usage: "manifest [--json]", mutates: false },
|
|
42
42
|
{ name: "completion", usage: "completion <bash|zsh|fish>", mutates: false },
|
|
43
|
+
{ name: "stats", usage: "stats <vault> [--days 1..3650] [--json]", mutates: false },
|
|
43
44
|
] as const;
|
|
44
45
|
const RELATIONS = ["supports", "contradicts", "extends", "applies-to", "context-for"] as const;
|
|
45
46
|
const LOG_EVENT_TYPES = ["INGEST", "ROUTE", "EXTRACT", "CONNECT", "DIGEST", "LEARN", "LINT", "TASK", "GRAPH", "SYNC", "RETHINK", "SOURCE"] as const;
|
package/src/log-format.ts
CHANGED
|
@@ -7,6 +7,9 @@ import type { LogContext, LogEvent } from "./log-types.ts";
|
|
|
7
7
|
import type { WriteIssue } from "./write-types.ts";
|
|
8
8
|
|
|
9
9
|
export const LOG_TYPES = ["INGEST", "ROUTE", "EXTRACT", "CONNECT", "DIGEST", "LEARN", "LINT", "TASK", "GRAPH", "SYNC", "RETHINK", "SOURCE"] as const;
|
|
10
|
+
const bounded = (maxLength: number) => ({ type: "string", minLength: 1, maxLength });
|
|
11
|
+
/** Closed schema of a write's log event, shared by the MCP write tools and CLI `--log JSON`. */
|
|
12
|
+
export const LOG_EVENT_SCHEMA = { type: "object", properties: { type: { type: "string", enum: [...LOG_TYPES] }, project: bounded(64), source: bounded(512), dest: bounded(512), note: bounded(500) }, required: ["type", "project", "source", "note"], additionalProperties: false };
|
|
10
13
|
export const AGENT_MARKERS = ["AGENTHUB_PEER_ID", "CLAUDECODE", "CLAUDE_CODE_SESSION_ID", "CODEX_THREAD_ID", "PI_SESSION_ID"] as const;
|
|
11
14
|
const EVENT_KEYS = new Set(["type", "project", "source", "dest", "note"]);
|
|
12
15
|
const CONTEXT_KEYS = new Set(["vault", "config", "channel", "env", "now", "projectIds"]);
|
package/src/log-types.ts
CHANGED
|
@@ -4,7 +4,11 @@ export interface LogEvent { type: string; project: string; source: string; dest?
|
|
|
4
4
|
export interface LogRequest { vault: string; event: LogEvent; channel?: "cli" | "mcp" }
|
|
5
5
|
export interface LogContext { vault: VaultSpec; config: VmemConfig; channel?: "cli" | "mcp"; env?: NodeJS.ProcessEnv; now?: Date; projectIds?: ReadonlySet<string> }
|
|
6
6
|
export interface LogReceipt { ok: boolean; appended: boolean | null; synced: boolean; bytesWritten: number | null; issues: WriteIssue[]; rolledBack?: boolean }
|
|
7
|
-
export interface LogReadOptions {
|
|
7
|
+
export interface LogReadOptions {
|
|
8
|
+
vault: string; limit?: number; grep?: string; type?: string;
|
|
9
|
+
/** In-process callers only (never CLI/MCP input): called with every complete data line, before the type and grep filters. */
|
|
10
|
+
visit?: (line: string) => void;
|
|
11
|
+
}
|
|
8
12
|
export interface LogReadResult { ok: boolean; lines: string[]; dataLines: number | null; warnings: string[]; issues: WriteIssue[] }
|
|
9
13
|
|
|
10
14
|
export interface LogBinding { vault: VaultSpec; denyPaths: string[]; piiPatterns: string[]; dev: number; ino: number }
|
package/src/mcp-spec.ts
CHANGED
|
@@ -5,10 +5,13 @@ import { validateSpecRepo } from "./spec-validate.ts";
|
|
|
5
5
|
import { resolveHome } from "./home.ts";
|
|
6
6
|
import { loadRegistry } from "./registry.ts";
|
|
7
7
|
|
|
8
|
+
/** Spec tool definitions; every spec tool is read-only. */
|
|
9
|
+
export const specTools = (): McpTool[] => SPEC_COMMANDS.map(item => ({ name: item.name, description: item.description, inputSchema: item.inputSchema,
|
|
10
|
+
annotations: { readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: false } }));
|
|
11
|
+
|
|
8
12
|
export function createSpecMcp(home: string, env: NodeJS.ProcessEnv = process.env): { tools: McpTool[]; handlers: Record<string, McpHandler> } {
|
|
9
13
|
const registered = loadRegistry(home).specs ?? [];
|
|
10
|
-
const tools
|
|
11
|
-
annotations: { readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: false } }));
|
|
14
|
+
const tools = specTools();
|
|
12
15
|
const handlers: Record<string, McpHandler> = {};
|
|
13
16
|
for (const item of SPEC_COMMANDS) handlers[item.name] = args => {
|
|
14
17
|
const invalid = argumentError(args, item.inputSchema);
|
package/src/mcp-vault.ts
CHANGED
|
@@ -8,11 +8,14 @@ import { parseFrontmatter } from "./frontmatter.ts";
|
|
|
8
8
|
import type { VaultSpec } from "./config.ts";
|
|
9
9
|
import { loadRegistry, registryStatus } from "./registry.ts";
|
|
10
10
|
import { VaultIndex, MAX_NOTE_BYTES } from "./vault-index.ts";
|
|
11
|
+
import { listVaultFiles, readVaultFile } from "./vault-files.ts";
|
|
11
12
|
import { appendLog, readLog } from "./vault-log.ts";
|
|
13
|
+
import { STATS_MAX_DAYS, vaultStats } from "./vault-stats.ts";
|
|
12
14
|
import { applyWrite } from "./writer.ts";
|
|
13
15
|
import { notePath } from "./write-operations.ts";
|
|
14
16
|
import type { LogEvent } from "./log-types.ts";
|
|
15
|
-
import
|
|
17
|
+
import { LOG_EVENT_SCHEMA } from "./log-format.ts";
|
|
18
|
+
import type { PatchEdit, WriteOp, WriteReceipt, WriteRequest } from "./write-types.ts";
|
|
16
19
|
import type { McpHandler, McpTool } from "./mcp.ts";
|
|
17
20
|
|
|
18
21
|
const string = (maxLength: number, minLength = 1) => ({ type: "string", minLength, maxLength });
|
|
@@ -24,12 +27,12 @@ const tool = (name: string, description: string, inputSchema: Record<string, unk
|
|
|
24
27
|
const RELATIONS = ["supports", "contradicts", "extends", "applies-to", "context-for"];
|
|
25
28
|
const LOG_TYPES = ["INGEST", "ROUTE", "EXTRACT", "CONNECT", "DIGEST", "LEARN", "LINT", "TASK", "GRAPH", "SYNC", "RETHINK", "SOURCE"];
|
|
26
29
|
const note = string(1024), sha = { type: "string", minLength: 64, maxLength: 64 };
|
|
27
|
-
const logEvent =
|
|
30
|
+
const logEvent = LOG_EVENT_SCHEMA;
|
|
28
31
|
const scalar = { type: ["string", "number", "boolean", "null", "array"], maxLength: 4096, maxItems: 64, items: { type: ["string", "number", "boolean"], maxLength: 1024 } };
|
|
29
32
|
|
|
30
33
|
/** Tool names for the implemented read, write, log, analysis and promotion providers. */
|
|
31
34
|
export const VMEM_TOOL_NAMES = ["vmem_status", "vmem_search", "vmem_read", "vmem_read_many", "vmem_list", "vmem_links", "vmem_mocs", "vmem_check", "vmem_log_tail",
|
|
32
|
-
"vmem_create", "vmem_patch", "vmem_append", "vmem_set_frontmatter", "vmem_link", "vmem_supersede", "vmem_log_append", "vmem_move", "vmem_delete", "vmem_promote_from_amem", "vmem_lint", "vmem_graph_build", "vmem_graph_neighbors", "vmem_promote", "vmem_skills_mine", ...SPEC_COMMANDS.map(item => item.name)] as const;
|
|
35
|
+
"vmem_create", "vmem_patch", "vmem_append", "vmem_set_frontmatter", "vmem_link", "vmem_supersede", "vmem_log_append", "vmem_move", "vmem_delete", "vmem_promote_from_amem", "vmem_lint", "vmem_graph_build", "vmem_graph_neighbors", "vmem_promote", "vmem_skills_mine", "vmem_read_file", "vmem_list_files", "vmem_report", "vmem_stats", ...SPEC_COMMANDS.map(item => item.name)] as const;
|
|
33
36
|
|
|
34
37
|
/** Tool definitions for a fixed vault set: `vault` is an enum of the registered ids. */
|
|
35
38
|
export function vaultTools(ids: readonly string[], orgIds: readonly string[] = []): McpTool[] {
|
|
@@ -41,20 +44,20 @@ export function vaultTools(ids: readonly string[], orgIds: readonly string[] = [
|
|
|
41
44
|
tool("vmem_search", "Full-text (BM25) search over indexed notes. Literal AND terms.", object({ query: string(1000), vault, limit: integer(1, 50), type: string(64), domain: string(64) }, ["query"])),
|
|
42
45
|
tool("vmem_read", "Read one note by stem, alias or notes/ path: frontmatter, body, sha256. Bounded by maxBytes; section selects an exact H2, offset pages UTF-8 byte ranges, and truncated reads list H2 headings/nextOffset.", object({ path: note, vault, maxBytes: integer(1, 65536), section: string(1024), offset: integer(0, MAX_NOTE_BYTES) }, ["path"])),
|
|
43
46
|
tool("vmem_read_many", "Read up to 20 notes (8 KiB each).", object({ paths: { type: "array", minItems: 1, maxItems: 20, items: note }, vault }, ["paths"])),
|
|
44
|
-
tool("vmem_list", "List indexed notes under notes
|
|
47
|
+
tool("vmem_list", "List indexed notes under notes/ in path order, optionally by stem prefix, with each note's mtime and the filtered total. Pass the returned nextCursor as cursor for the next page.", object({ vault, prefix: string(256, 0), limit: integer(1, 500), cursor: string(1024) })),
|
|
45
48
|
tool("vmem_links", "Outgoing links (with resolution) and/or incoming backlinks of a note.", object({ path: note, vault, direction: { type: "string", enum: ["out", "in", "both"] }, limit: integer(1, 200) }, ["path"])),
|
|
46
49
|
tool("vmem_mocs", "MOC notes with incoming link counts.", object({ vault })),
|
|
47
50
|
tool("vmem_check", "Validate a prospective note text without writing.", object({ vault, path: string(200), text: string(262144, 0) }, ["text"])),
|
|
48
51
|
tool("vmem_log_tail", "Read the newest vault log lines, optionally filtered.", object({ vault, n: integer(1, 200), type: { type: "string", enum: LOG_TYPES }, grep: string(256) })),
|
|
49
52
|
tool("vmem_create", "Create a validated note. Requires a vault with write=mcp.", object({ vault, stem: string(200), type: string(64), domain: string(64), topics: { type: "array", maxItems: 32, items: string(200) }, description: string(1000), title: string(300), body: string(65536, 0), source: { type: ["string", "array"], minLength: 1, maxLength: 512, minItems: 1, maxItems: 32, items: string(512) }, confidence: string(32), status: string(32), created: string(32, 0), extra: { type: "object", maxProperties: 32, additionalProperties: scalar }, log: logEvent }, ["stem", "type", "domain", "description"]), "write"),
|
|
50
|
-
tool("vmem_patch", "Replace
|
|
51
|
-
tool("vmem_append", "Append markdown to a section of a note.", object({ vault, path: note, section: string(120), markdown: string(16384), expectedSha: sha }, ["path", "section", "markdown"]), "write"),
|
|
52
|
-
tool("vmem_set_frontmatter", "Set or unset frontmatter keys of a note.", object({ vault, path: note, set: { type: "object", maxProperties: 32, additionalProperties: scalar }, unset: { type: "array", maxItems: 32, items: string(64) }, expectedSha: sha }, ["path"]), "write"),
|
|
53
|
+
tool("vmem_patch", "Replace exact strings in a note: one oldString/newString pair, or edits (1..32 pairs, each matching once in the original text, non-overlapping) applied all-or-nothing with one receipt and one log event.", object({ vault, path: note, oldString: string(8192), newString: string(8192, 0), edits: { type: "array", minItems: 1, maxItems: 32, items: object({ oldString: string(8192), newString: string(8192, 0) }, ["oldString", "newString"]) }, expectedSha: sha, log: logEvent }, ["path"]), "write"),
|
|
54
|
+
tool("vmem_append", "Append markdown to a section of a note.", object({ vault, path: note, section: string(120), markdown: string(16384), expectedSha: sha, log: logEvent }, ["path", "section", "markdown"]), "write"),
|
|
55
|
+
tool("vmem_set_frontmatter", "Set or unset frontmatter keys of a note.", object({ vault, path: note, set: { type: "object", maxProperties: 32, additionalProperties: scalar }, unset: { type: "array", maxItems: 32, items: string(64) }, expectedSha: sha, log: logEvent }, ["path"]), "write"),
|
|
53
56
|
tool("vmem_link", "Add a typed relation link between two notes (bidirectional by default).", object({ vault, from: note, to: note, relation: { type: "string", enum: RELATIONS }, context: string(500), bidirectional: { type: "boolean" }, log: logEvent }, ["from", "to", "relation"]), "write"),
|
|
54
57
|
tool("vmem_supersede", "Mark a note superseded by another.", object({ vault, old: note, next: note, log: logEvent }, ["old", "next"]), "write"),
|
|
55
58
|
tool("vmem_log_append", "Append one validated event line to the vault log.", object({ vault, type: { type: "string", enum: LOG_TYPES }, project: string(64), source: string(512), dest: string(512), note: string(500) }, ["type", "project", "source", "note"]), "write"),
|
|
56
|
-
tool("vmem_move", "Two-step rename with link rewrites: without confirm returns preview and fingerprint; then confirm=true with that fingerprint.", object({ vault, from: note, to: note, ...twoStep }, ["from", "to"]), "confirm"),
|
|
57
|
-
tool("vmem_delete", "Two-step delete: without confirm returns preview (or refusal), backlinks and fingerprint; then confirm=true with that fingerprint.", object({ vault, path: note, ...twoStep }, ["path"]), "confirm"),
|
|
59
|
+
tool("vmem_move", "Two-step rename with link rewrites: without confirm returns preview and fingerprint; then confirm=true with that fingerprint (pass log on the confirm call).", object({ vault, from: note, to: note, ...twoStep, log: logEvent }, ["from", "to"]), "confirm"),
|
|
60
|
+
tool("vmem_delete", "Two-step delete: without confirm returns preview (or refusal), backlinks and fingerprint; then confirm=true with that fingerprint (pass log on the confirm call).", object({ vault, path: note, ...twoStep, log: logEvent }, ["path"]), "confirm"),
|
|
58
61
|
tool("vmem_promote_from_amem", "Two-step promotion of one durable agent-mem entry (id from amem_vault_candidates) into a vault note: without confirm returns the full note text, possible duplicates and fingerprint; confirm=true with that fingerprint creates the note, logs LEARN and records the agent-mem vault link. linkTo links an existing note instead.",
|
|
59
62
|
object({ vault, id: { type: "string", minLength: 32, maxLength: 36 }, stem: string(120), type: string(64), domain: string(64), topics: { type: "array", maxItems: 32, items: string(200) }, description: string(1000), title: string(300), body: string(65536, 0), linkTo: note, project: string(400), ...twoStep }, ["id"]), "confirm"),
|
|
60
63
|
tool("vmem_lint", "Lint a private derived snapshot; write=true writes a screened report.", object({ vault, write: { type: "boolean" }, dryRun: { type: "boolean" } }), "write"),
|
|
@@ -64,6 +67,10 @@ export function vaultTools(ids: readonly string[], orgIds: readonly string[] = [
|
|
|
64
67
|
object({ path: note, to: orgVault, topics: { type: "array", maxItems: 32, items: string(200) }, again: { type: "boolean" }, ...twoStep }, ["path", "to"]), "confirm"),
|
|
65
68
|
tool("vmem_skills_mine", "Read-only skill-candidate mining over amem curated entries, amem capture rows and vault notes: dictionary matching, deterministic communities and novelty scoring. Returns counts, masked candidates and a markdown report; writes nothing.",
|
|
66
69
|
object({ vault, days: integer(1, 3650), project: string(400), allProjects: { type: "boolean" }, vocab: string(4096), skills: { type: "array", maxItems: 16, items: string(4096) }, minSize: integer(1, 10000), minNovelty: { type: "number", minimum: 0, maximum: 1 } })),
|
|
70
|
+
tool("vmem_read_file", "Read one UTF-8 file outside notes/: ops/, reports/, self/, templates/, AGENTS.md or README.md. Same bounds and paging as vmem_read (maxBytes, offset, nextOffset); notes/ use vmem_read, the log vmem_log_tail. Never locks or writes.", object({ vault, path: string(1024), maxBytes: integer(1, 65536), offset: integer(0, MAX_NOTE_BYTES) }, ["path"])),
|
|
71
|
+
tool("vmem_list_files", "List one directory level under ops/, reports/, self/ or templates/ (up to 500 entries; dot, linked and denied entries omitted). Never locks or writes.", object({ vault, dir: string(1024) }, ["dir"])),
|
|
72
|
+
tool("vmem_report", "Write a screened report file under reports/ or ops/ (the MCP form of `vmem report`): same path rules, screening, write policy (write=mcp), receipt and EXTRACT log event; log overrides it, for example a LINT or GRAPH event. dryRun changes nothing.", object({ vault, path: string(200), body: string(65536, 0), dryRun: { type: "boolean" }, log: logEvent }, ["path", "body"]), "write"),
|
|
73
|
+
tool("vmem_stats", "Read-only vault statistics: note counts by type, domain and status, notes modified in the last 7 and 30 days, and log event counts by type over the last `days` (default 30). Writes no vault file or log and takes no vault or log lock.", object({ vault, days: integer(1, STATS_MAX_DAYS) })),
|
|
67
74
|
];
|
|
68
75
|
}
|
|
69
76
|
type Issue = { code: string; severity: "error"; message: string };
|
|
@@ -122,6 +129,9 @@ export function createVaultMcp(home: string, env: NodeJS.ProcessEnv = process.en
|
|
|
122
129
|
...(Array.isArray(args.skills) ? { skills: args.skills as string[] } : {}), ...(typeof args.days === "number" ? { days: args.days } : {}),
|
|
123
130
|
...(typeof args.minSize === "number" ? { minSize: args.minSize } : {}), ...(typeof args.minNovelty === "number" ? { minNovelty: args.minNovelty } : {}) });
|
|
124
131
|
},
|
|
132
|
+
vmem_read_file: args => readVaultFile(pick(args), loadRegistry(home).denyPaths ?? [], str(args.path)!, (args.maxBytes as number | undefined) ?? 16384, args.offset as number | undefined),
|
|
133
|
+
vmem_list_files: args => listVaultFiles(pick(args), loadRegistry(home).denyPaths ?? [], str(args.dir)!),
|
|
134
|
+
vmem_report: args => write(args, { op: "report", path: str(args.path)!, text: str(args.body)! }, args.dryRun === true ? { dryRun: true } : {}),
|
|
125
135
|
vmem_lint: args => args.write === true ? writeAnalysisReports(home,{vault:pick(args).id,kind:"lint",channel:"mcp",expectedVault:pick(args),dryRun:args.dryRun === true},env) : lintVault(analysisInput(home,pick(args).id,{expectedVault:pick(args)}).snapshot),
|
|
126
136
|
vmem_graph_build: args => writeAnalysisReports(home,{vault:pick(args).id,kind:"graph",workspace:args.workspace === true,channel:"mcp",expectedVault:pick(args),dryRun:args.dryRun === true},env),
|
|
127
137
|
vmem_graph_neighbors: args => graphNeighbors(buildGraph(analysisInput(home,pick(args).id,{expectedVault:pick(args)}).snapshot),str(args.path)!, (args.hops as 1|2|undefined) ?? 1, (args.limit as number|undefined) ?? 50),
|
|
@@ -132,7 +142,8 @@ export function createVaultMcp(home: string, env: NodeJS.ProcessEnv = process.en
|
|
|
132
142
|
try { const r = index.show(path, 8192); return document(r); }
|
|
133
143
|
catch (error) { return { path, ...refusal("V04", error instanceof Error ? error.message : "unreadable") }; }
|
|
134
144
|
}) })),
|
|
135
|
-
vmem_list: args => read(args, index =>
|
|
145
|
+
vmem_list: args => read(args, index => index.list(str(args.prefix) ?? "", (args.limit as number | undefined) ?? 100, str(args.cursor))),
|
|
146
|
+
vmem_stats: args => vaultStats(home, pick(args).id, { expectedVault: pick(args), ...(typeof args.days === "number" ? { days: args.days } : {}) }),
|
|
136
147
|
vmem_links: args => read(args, index => {
|
|
137
148
|
const direction = str(args.direction) ?? "both", limit = (args.limit as number | undefined) ?? 50;
|
|
138
149
|
const out = direction === "in" ? [] : index.links(str(args.path)!, limit);
|
|
@@ -152,7 +163,7 @@ export function createVaultMcp(home: string, env: NodeJS.ProcessEnv = process.en
|
|
|
152
163
|
if (Array.isArray(args.topics)) frontmatter.topics = (args.topics as string[]).map(v => v.startsWith("[[") ? v : `[[${v}]]`);
|
|
153
164
|
return write(args, { op: "create", path: str(args.stem)!, frontmatter, ...(str(args.body) !== undefined ? { body: str(args.body) } : {}) });
|
|
154
165
|
},
|
|
155
|
-
vmem_patch: args => write(args, { op: "patch", path: str(args.path)!, oldString: str(args.oldString)
|
|
166
|
+
vmem_patch: args => write(args, { op: "patch", path: str(args.path)!, ...(args.oldString !== undefined ? { oldString: str(args.oldString) } : {}), ...(args.newString !== undefined ? { newString: str(args.newString) } : {}), ...(args.edits !== undefined ? { edits: args.edits as PatchEdit[] } : {}), expectedSha: str(args.expectedSha) }),
|
|
156
167
|
vmem_append: args => write(args, { op: "append", path: str(args.path)!, section: str(args.section)!, markdown: str(args.markdown)!, expectedSha: str(args.expectedSha) }),
|
|
157
168
|
vmem_set_frontmatter: args => write(args, { op: "frontmatter", path: str(args.path)!, set: (args.set as Record<string, unknown> | undefined) ?? {}, ...(args.unset ? { unset: args.unset as string[] } : {}), expectedSha: str(args.expectedSha) }),
|
|
158
169
|
vmem_link: args => write(args, { op: "link", from: str(args.from)!, to: str(args.to)!, relation: args.relation as Extract<WriteOp, { op: "link" }>["relation"], ...(str(args.context) ? { context: str(args.context) } : {}), bidirectional: args.bidirectional !== false }),
|
|
@@ -0,0 +1,74 @@
|
|
|
1
|
+
import { closeSync, constants, fstatSync, lstatSync, openSync, readdirSync, readSync, realpathSync } from "node:fs";
|
|
2
|
+
import { createHash } from "node:crypto";
|
|
3
|
+
import { join } from "node:path";
|
|
4
|
+
import type { VaultSpec } from "./config.ts";
|
|
5
|
+
import { isDenied } from "./platform.ts";
|
|
6
|
+
import { MAX_NOTE_BYTES, MAX_READ_BYTES, MAX_RESULTS, readPage } from "./vault-index.ts";
|
|
7
|
+
|
|
8
|
+
/** Vault paths readable outside notes/: these directories and root files. Reads never lock, index or write. */
|
|
9
|
+
export const FILE_ROOTS = ["ops/", "reports/", "self/", "templates/", "AGENTS.md", "README.md"] as const;
|
|
10
|
+
const DIRS = new Set(["ops", "reports", "self", "templates"]), FILES = new Set(["AGENTS.md", "README.md"]);
|
|
11
|
+
const ROOTS = "ops/, reports/, self/, templates/, AGENTS.md or README.md";
|
|
12
|
+
function fail(message: string, code = "V04"): never { throw Object.assign(new Error(message), { code }); }
|
|
13
|
+
|
|
14
|
+
/** Checks `path` against the allowed roots and returns its location; every component must be a plain directory or file. */
|
|
15
|
+
function locate(vault: VaultSpec, denyPaths: string[], path: string, kind: "file" | "dir"): string {
|
|
16
|
+
const parts = path.split("/");
|
|
17
|
+
if (path.startsWith("/") || path.includes("\\") || path.includes("\0") || parts.some(part => !part || part === "." || part === ".."))
|
|
18
|
+
fail(`path: absolute, empty, . or .. segments are refused; pass a vault-relative path under ${ROOTS}`);
|
|
19
|
+
if (parts.some(part => part.startsWith("."))) fail(`path: dot paths such as .git are not readable through vmem; read a path under ${ROOTS}`);
|
|
20
|
+
if (parts[0] === "notes") fail("path: notes/ is read with vmem_read or vmem_read_many and listed with vmem_list");
|
|
21
|
+
if (parts[0] === "log") fail("path: the vault log is read with vmem_log_tail");
|
|
22
|
+
if (!(DIRS.has(parts[0]!) || (kind === "file" && parts.length === 1 && FILES.has(parts[0]!))))
|
|
23
|
+
fail(`path: outside the readable roots; use a path under ${ROOTS}`);
|
|
24
|
+
const root = realpathSync(vault.path);
|
|
25
|
+
if (root !== vault.path) fail(`vault ${vault.id}: registered path changed; re-register it with vmem vaults add and restart the MCP server`);
|
|
26
|
+
if (isDenied(join(root, path), denyPaths)) fail("path: denied by the built-in secret rules or registry denyPaths; nothing read", "V06");
|
|
27
|
+
let current = root;
|
|
28
|
+
for (let i = 0; i < parts.length; i++) {
|
|
29
|
+
current = join(current, parts[i]!);
|
|
30
|
+
let stat: ReturnType<typeof lstatSync>;
|
|
31
|
+
try { stat = lstatSync(current); }
|
|
32
|
+
catch (error) { if ((error as NodeJS.ErrnoException).code === "ENOENT") fail(`path: not found; list the parent directory with vmem_list_files`); throw error; }
|
|
33
|
+
// ponytail: every link is refused, not only links leaving the vault; matches the writer's plain-path rule.
|
|
34
|
+
if (stat.isSymbolicLink()) fail("path: symlinked components are refused because a link can leave the vault; read the link target's own vault path");
|
|
35
|
+
const last = i === parts.length - 1, wantDir = !last || kind === "dir";
|
|
36
|
+
if (wantDir ? !stat.isDirectory() : !stat.isFile()) fail(wantDir ? "path: not a directory; list its parent with vmem_list_files" : "path: not a regular file; list it with vmem_list_files");
|
|
37
|
+
}
|
|
38
|
+
return current;
|
|
39
|
+
}
|
|
40
|
+
|
|
41
|
+
/** Bounded UTF-8 read with vmem_read's limits and paging: maxBytes 1..64 KiB, files up to 2 MiB, byte offsets on character boundaries. */
|
|
42
|
+
export function readVaultFile(vault: VaultSpec, denyPaths: string[], path: string, maxBytes = 16384, offset?: number) {
|
|
43
|
+
if (!Number.isInteger(maxBytes) || maxBytes < 1 || maxBytes > MAX_READ_BYTES) fail(`max-bytes: expected integer 1..${MAX_READ_BYTES}`);
|
|
44
|
+
if (offset !== undefined && (!Number.isSafeInteger(offset) || offset < 0 || offset > MAX_NOTE_BYTES)) fail(`offset: expected integer 0..${MAX_NOTE_BYTES}`);
|
|
45
|
+
const fd = openSync(locate(vault, denyPaths, path, "file"), constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK);
|
|
46
|
+
let bytes: Buffer;
|
|
47
|
+
try {
|
|
48
|
+
const before = fstatSync(fd);
|
|
49
|
+
if (!before.isFile()) fail("path: not a regular file; list it with vmem_list_files");
|
|
50
|
+
if (before.size > MAX_NOTE_BYTES) fail(`file: exceeds the ${MAX_NOTE_BYTES}-byte read bound; read it outside vmem or split it`, "V09");
|
|
51
|
+
const buffer = Buffer.alloc(before.size + 1); let size = 0;
|
|
52
|
+
while (size < buffer.length) { const count = readSync(fd, buffer, size, buffer.length - size, null); if (!count) break; size += count; }
|
|
53
|
+
const after = fstatSync(fd);
|
|
54
|
+
if (size !== before.size || after.size !== before.size || after.mtimeMs !== before.mtimeMs) fail("file: changed during read; retry once it is stable", "V09");
|
|
55
|
+
bytes = buffer.subarray(0, size);
|
|
56
|
+
} finally { closeSync(fd); }
|
|
57
|
+
const text = bytes.toString("utf8");
|
|
58
|
+
if (!Buffer.from(text).equals(bytes)) fail("file: not valid UTF-8 text; vmem_read_file reads text files only", "V01");
|
|
59
|
+
const start = offset ?? 0, page = readPage(bytes.subarray(start, start + maxBytes + 4), maxBytes, start, bytes.length);
|
|
60
|
+
return { path, sha256: createHash("sha256").update(bytes).digest("hex"), truncated: page.truncated, body: page.content,
|
|
61
|
+
...(offset !== undefined ? { offset } : {}), ...(page.nextOffset !== undefined ? { nextOffset: page.nextOffset } : {}) };
|
|
62
|
+
}
|
|
63
|
+
|
|
64
|
+
/** One directory level under an allowed root, sorted by name; dot, linked, special and denied entries are left out. */
|
|
65
|
+
export function listVaultFiles(vault: VaultSpec, denyPaths: string[], dir: string) {
|
|
66
|
+
const clean = dir.replace(/\/$/, ""), location = locate(vault, denyPaths, clean, "dir");
|
|
67
|
+
// ponytail: readdirSync loads the whole directory before the 500-entry cut; stream with opendir if vault dirs grow huge.
|
|
68
|
+
const entries = readdirSync(location, { withFileTypes: true })
|
|
69
|
+
.filter(entry => !entry.name.startsWith(".") && (entry.isFile() || entry.isDirectory()) && !isDenied(join(location, entry.name), denyPaths))
|
|
70
|
+
.sort((a, b) => a.name < b.name ? -1 : a.name > b.name ? 1 : 0);
|
|
71
|
+
return { dir: clean, truncated: entries.length > MAX_RESULTS, entries: entries.slice(0, MAX_RESULTS).map(entry => entry.isDirectory()
|
|
72
|
+
? { name: entry.name, path: `${clean}/${entry.name}`, type: "dir" as const }
|
|
73
|
+
: { name: entry.name, path: `${clean}/${entry.name}`, type: "file" as const, bytes: lstatSync(join(location, entry.name)).size }) };
|
|
74
|
+
}
|
package/src/vault-index.ts
CHANGED
|
@@ -16,7 +16,40 @@ export const MAX_RESULTS = 500;
|
|
|
16
16
|
* share the unversioned `<digest>.sqlite`; the first run of a newer version builds its own file once.
|
|
17
17
|
*/
|
|
18
18
|
export const LINK_RESOLUTION = 2;
|
|
19
|
-
|
|
19
|
+
/** Bump when tables or FTS columns change. Named per value like LINK_RESOLUTION. 2 = CJK bigram table. */
|
|
20
|
+
export const INDEX_SCHEMA = 2;
|
|
21
|
+
const indexName = (root: string) => `${digest(root)}-links${LINK_RESOLUTION}-schema${INDEX_SCHEMA}.sqlite`;
|
|
22
|
+
/**
|
|
23
|
+
* unicode61 keeps each Hangul/CJK run as one token, so `사업단` misses `사업단은`. A side table holds the
|
|
24
|
+
* overlapping bigrams of every CJK run of 2+ characters, runs split by a `0` token; a CJK query term then matches
|
|
25
|
+
* one bigram (2 characters) or a phrase of consecutive bigrams, which is exact substring matching. A separate
|
|
26
|
+
* table keeps bm25 of the unicode61 table, whose document length spans all columns, unchanged for Latin queries.
|
|
27
|
+
*/
|
|
28
|
+
export const GRAMS_COLUMNS = "path UNINDEXED, title, description, body";
|
|
29
|
+
const CJK = "\\p{sc=Hangul}\\p{sc=Han}\\p{sc=Hiragana}\\p{sc=Katakana}";
|
|
30
|
+
const CJK_RUN = new RegExp(`[${CJK}]{2,}`, "gu"), CJK_TERM = new RegExp(`^[${CJK}]{2,}$`, "u");
|
|
31
|
+
function bigrams(run: string): string[] {
|
|
32
|
+
const chars = [...run];
|
|
33
|
+
return chars.slice(1).map((char, i) => chars[i]! + char);
|
|
34
|
+
}
|
|
35
|
+
export function grams(text: string): string {
|
|
36
|
+
return [...text.normalize("NFC").matchAll(CJK_RUN)].map(([run]) => `${bigrams(run).join(" ")} 0`).join(" ");
|
|
37
|
+
}
|
|
38
|
+
export const gramPhrase = (run: string) => `"${bigrams(run).join(" ")}"`;
|
|
39
|
+
/**
|
|
40
|
+
* Literal AND terms: users cannot inject FTS operators, columns or SQL. A term whose letters and digits form one
|
|
41
|
+
* CJK run of 2+ characters (`runs`) matches as a substring through the bigram table; any other term (`tokens`,
|
|
42
|
+
* quoted) keeps unicode61 token matching.
|
|
43
|
+
*/
|
|
44
|
+
export function splitTerms(words: string[]): { tokens: string[]; runs: string[] } {
|
|
45
|
+
const tokens: string[] = [], runs: string[] = [];
|
|
46
|
+
for (const term of words) {
|
|
47
|
+
const letters = term.normalize("NFC").match(/[\p{L}\p{N}]+/gu);
|
|
48
|
+
if (letters?.length === 1 && CJK_TERM.test(letters[0]!)) runs.push(letters[0]!);
|
|
49
|
+
else tokens.push(`"${term.replaceAll('"', '""')}"`);
|
|
50
|
+
}
|
|
51
|
+
return { tokens, runs };
|
|
52
|
+
}
|
|
20
53
|
export const MAX_READ_BYTES = 64 * 1024;
|
|
21
54
|
export interface IndexedNote extends LinkNote { hash: string; description: string; type: string; domain: string }
|
|
22
55
|
export interface NoteReadResult {
|
|
@@ -62,6 +95,11 @@ function aliases(value: unknown): string[] {
|
|
|
62
95
|
function metadata(stat: NonNullable<ReturnType<typeof lstatSync>>): string {
|
|
63
96
|
return [stat.dev, stat.ino, stat.size, stat.mtimeMs, stat.ctimeMs].join(":");
|
|
64
97
|
}
|
|
98
|
+
/** The mtime (epoch ms) stored in a `metadata()` stamp, or null when the stamp has another shape. */
|
|
99
|
+
function stampMtime(stamp: string): number | null {
|
|
100
|
+
const value = Number(stamp.split(":")[3]);
|
|
101
|
+
return stamp.split(":").length === 5 && Number.isFinite(value) ? value : null;
|
|
102
|
+
}
|
|
65
103
|
/** Adapter file I/O gets path-specific guidance without rewriting unrelated Git or binding errors. */
|
|
66
104
|
function specFileIO<T>(path: string, action: string, operation: () => T): T {
|
|
67
105
|
try { return operation(); }
|
|
@@ -78,7 +116,7 @@ function boundedText(value: string, bytes: number): { content: string; truncated
|
|
|
78
116
|
while (end > 0 && (buffer[end]! & 0xc0) === 0x80) end--;
|
|
79
117
|
return { content: buffer.subarray(0, end).toString("utf8"), truncated: true };
|
|
80
118
|
}
|
|
81
|
-
function readPage(prefix: Buffer, maxBytes: number, offset: number, totalBytes: number): Pick<NoteReadResult, "content" | "truncated" | "nextOffset"> {
|
|
119
|
+
export function readPage(prefix: Buffer, maxBytes: number, offset: number, totalBytes: number): Pick<NoteReadResult, "content" | "truncated" | "nextOffset"> {
|
|
82
120
|
if (offset > totalBytes) throw new Error("offset: exceeds requested note or section");
|
|
83
121
|
if (prefix.length && (prefix[0]! & 0xc0) === 0x80) throw new Error("offset: must be a UTF-8 character boundary");
|
|
84
122
|
let end = Math.min(maxBytes, prefix.length);
|
|
@@ -165,6 +203,7 @@ export class VaultIndex<Source extends VaultSpec | SpecSourceSpec = VaultSpec> {
|
|
|
165
203
|
title TEXT NOT NULL, description TEXT NOT NULL, body TEXT NOT NULL,
|
|
166
204
|
type TEXT NOT NULL, domain TEXT NOT NULL, aliases TEXT NOT NULL, source TEXT NOT NULL);
|
|
167
205
|
CREATE VIRTUAL TABLE IF NOT EXISTS fts USING fts5(path UNINDEXED, title, description, body, tokenize='unicode61');
|
|
206
|
+
CREATE VIRTUAL TABLE IF NOT EXISTS grams USING fts5(${GRAMS_COLUMNS}, tokenize='unicode61');
|
|
168
207
|
CREATE TABLE IF NOT EXISTS analysis_notes (path TEXT PRIMARY KEY, source TEXT NOT NULL, hash TEXT NOT NULL, metadata TEXT NOT NULL);
|
|
169
208
|
CREATE TABLE IF NOT EXISTS links (source TEXT NOT NULL, ordinal INTEGER NOT NULL, data TEXT NOT NULL, PRIMARY KEY(source, ordinal));`);
|
|
170
209
|
// Dry runs reuse the derived cache via a readonly connection, then refresh only changed notes in memory.
|
|
@@ -182,7 +221,7 @@ export class VaultIndex<Source extends VaultSpec | SpecSourceSpec = VaultSpec> {
|
|
|
182
221
|
if (size.count > 10000 || size.bytes > 64 * 1024 * 1024) throw new Error("analysis: cached snapshot exceeds bound");
|
|
183
222
|
for (const row of disk.query<Omit<StoredNote,"aliases"> & {aliases:string}, []>("SELECT * FROM notes").all()) {
|
|
184
223
|
this.db.query("INSERT INTO notes VALUES (?,?,?,?,?,?,?,?,?,?)").run(row.path,row.hash,row.metadata,row.title,row.description,row.body,row.type,row.domain,row.aliases,row.source);
|
|
185
|
-
this.
|
|
224
|
+
this.indexText(row.path, row.title, row.description, row.body);
|
|
186
225
|
}
|
|
187
226
|
for (const row of disk.query<{source:string;ordinal:number;data:string}, []>("SELECT * FROM links").all()) this.db.query("INSERT INTO links VALUES (?,?,?)").run(row.source,row.ordinal,row.data);
|
|
188
227
|
this.db.run(`PRAGMA user_version = ${Number(disk.query<{ user_version: number }, []>("PRAGMA user_version").get()?.user_version ?? 0)}`);
|
|
@@ -202,6 +241,15 @@ export class VaultIndex<Source extends VaultSpec | SpecSourceSpec = VaultSpec> {
|
|
|
202
241
|
}
|
|
203
242
|
}
|
|
204
243
|
close(): void { this.db.close(); }
|
|
244
|
+
// ponytail: grams stores its bigram text (about 2.3x the CJK bytes); contentless with stable rowids would save it.
|
|
245
|
+
private indexText(path: string, title: string, description: string, body: string): void {
|
|
246
|
+
this.db.query("INSERT INTO fts VALUES (?,?,?,?)").run(path, title, description, body);
|
|
247
|
+
this.db.query("INSERT INTO grams VALUES (?,?,?,?)").run(path, grams(title), grams(description), grams(body));
|
|
248
|
+
}
|
|
249
|
+
private unindex(path: string): void {
|
|
250
|
+
this.db.query("DELETE FROM fts WHERE path=?").run(path);
|
|
251
|
+
this.db.query("DELETE FROM grams WHERE path=?").run(path);
|
|
252
|
+
}
|
|
205
253
|
private stored(): LinkNote[] {
|
|
206
254
|
return this.db.query<Omit<LinkNote, "aliases"> & { aliases: string }, []>("SELECT path,title,aliases,body FROM notes ORDER BY path").all().map(row => ({ ...row, aliases: JSON.parse(row.aliases) as string[] }));
|
|
207
255
|
}
|
|
@@ -215,7 +263,7 @@ export class VaultIndex<Source extends VaultSpec | SpecSourceSpec = VaultSpec> {
|
|
|
215
263
|
if (!this.adapter && (realpathSync(this.root) !== this.root || realpathSync(notesDir) !== notesDir || !lstatSync(notesDir).isDirectory())) throw new Error(`vault ${this.vault.id}: path or notes directory changed`);
|
|
216
264
|
const indexed = this.adapter ? new Set(this.db.query<{path:string}, []>("SELECT path FROM notes").all().map(row => row.path)) : undefined;
|
|
217
265
|
const previous = new Map(this.db.query<{ path: string; hash: string; metadata: string }, []>(this.adapter ? "SELECT path,hash,metadata FROM analysis_notes" : options.captureInvalidFrontmatter ? "SELECT path,hash,metadata FROM notes UNION SELECT path,hash,metadata FROM analysis_notes" : "SELECT path,hash,metadata FROM notes").all().map(note => [note.path, note]));
|
|
218
|
-
if (rebuild) this.db.run("DELETE FROM fts");
|
|
266
|
+
if (rebuild) this.db.run("DELETE FROM fts; DELETE FROM grams;");
|
|
219
267
|
const live = new Set<string>(), present = new Set<string>();
|
|
220
268
|
let changed = 0, analysisBytes = 0;
|
|
221
269
|
const skipped: string[] = [], touched = new Set<string>(), keys = new Set<string>();
|
|
@@ -298,15 +346,16 @@ export class VaultIndex<Source extends VaultSpec | SpecSourceSpec = VaultSpec> {
|
|
|
298
346
|
}
|
|
299
347
|
const { data, body } = document;
|
|
300
348
|
touch(path);
|
|
349
|
+
const title = text(data.title) || path.slice(0, -3), description = text(data.description);
|
|
301
350
|
this.db.query("INSERT OR REPLACE INTO notes VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)").run(path, hash, stamp,
|
|
302
|
-
|
|
303
|
-
this.
|
|
304
|
-
this.
|
|
351
|
+
title, description, body, text(data.type), text(data.domain), JSON.stringify(aliases(data.aliases ?? data.alias)), source);
|
|
352
|
+
this.unindex(path);
|
|
353
|
+
this.indexText(path, title, description, body);
|
|
305
354
|
changed++;
|
|
306
355
|
}
|
|
307
356
|
for (const row of this.db.query<{path:string}, []>("SELECT path FROM analysis_notes").all()) if (!present.has(row.path)) this.db.query("DELETE FROM analysis_notes WHERE path=?").run(row.path);
|
|
308
357
|
let removed = 0;
|
|
309
|
-
for (const path of indexed ?? previous.keys()) if (!live.has(path)) { touch(path); this.db.query("DELETE FROM notes WHERE path=?").run(path); this.
|
|
358
|
+
for (const path of indexed ?? previous.keys()) if (!live.has(path)) { touch(path); this.db.query("DELETE FROM notes WHERE path=?").run(path); this.unindex(path); removed++; }
|
|
310
359
|
if (this.adapter) {
|
|
311
360
|
// Spec links are resolved by frontmatter id in buildSpecGraph, never by vault paths.
|
|
312
361
|
this.db.run("DELETE FROM links");
|
|
@@ -345,13 +394,22 @@ export class VaultIndex<Source extends VaultSpec | SpecSourceSpec = VaultSpec> {
|
|
|
345
394
|
search(query: string, options: SearchOptions = {}, snapshot?: ReadonlyMap<string, string>, snapshotOffset = 0, allSnapshotRows = false): SearchResult[] {
|
|
346
395
|
const limit = resultLimit(options.limit);
|
|
347
396
|
if (!query.trim() || Buffer.byteLength(query) > 1024) throw new Error("search: query must contain 1..1024 bytes");
|
|
348
|
-
|
|
349
|
-
const terms = query.trim().split(/\s+/).map(term => `"${term.replaceAll('"', '""')}"`).join(" AND ");
|
|
397
|
+
const { tokens, runs } = splitTerms(query.trim().split(/\s+/));
|
|
350
398
|
const hashColumn = snapshot ? ",n.hash AS sourceHash" : "";
|
|
351
|
-
|
|
399
|
+
const filters = [options.type ?? null, options.type ?? null, options.domain ?? null, options.domain ?? null, snapshot ? 10000 : limit];
|
|
400
|
+
// Latin-only queries run the unicode61 query unchanged. With a CJK term the bigram table drives the match, and a
|
|
401
|
+
// note where every CJK term is a standalone word or sits in the title/description ranks before substring-only hits.
|
|
402
|
+
const strong = runs.map(() => "(n.path IN (SELECT path FROM fts WHERE fts MATCH ?) OR n.path IN (SELECT path FROM grams WHERE grams MATCH ?))").join(" AND ");
|
|
403
|
+
const rows = !runs.length
|
|
404
|
+
? this.db.query<SearchResult & { sourceHash?: string }, (string | number | null)[]>(`SELECT n.path, n.title, substr(n.description,1,512) AS description, n.type, n.domain,
|
|
352
405
|
bm25(fts,0,5,2,1) AS score, snippet(fts,3,'','',' ... ',32) AS snippet${hashColumn} FROM fts JOIN notes n ON n.path=fts.path
|
|
353
|
-
WHERE fts MATCH ? AND (? IS NULL OR n.type=?) AND (? IS NULL OR n.domain=?) ORDER BY score,n.path LIMIT ?`).all(
|
|
354
|
-
.
|
|
406
|
+
WHERE fts MATCH ? AND (? IS NULL OR n.type=?) AND (? IS NULL OR n.domain=?) ORDER BY score,n.path LIMIT ?`).all(tokens.join(" AND "), ...filters)
|
|
407
|
+
: this.db.query<SearchResult & { sourceHash?: string }, (string | number | null)[]>(`SELECT n.path, n.title, substr(n.description,1,512) AS description, n.type, n.domain,
|
|
408
|
+
bm25(grams,0,5,2,1) AS score, substr(n.body, max(1, instr(n.body, ?) - 48), 160) AS snippet${hashColumn} FROM grams JOIN notes n ON n.path=grams.path
|
|
409
|
+
WHERE grams MATCH ? ${tokens.length ? "AND n.path IN (SELECT path FROM fts WHERE fts MATCH ?)" : ""} AND (? IS NULL OR n.type=?) AND (? IS NULL OR n.domain=?)
|
|
410
|
+
ORDER BY NOT (${strong}),score,n.path LIMIT ?`).all(runs[0]!, runs.map(gramPhrase).join(" AND "), ...(tokens.length ? [tokens.join(" AND ")] : []), ...filters.slice(0, 4),
|
|
411
|
+
...runs.flatMap(run => [`"${run}"`, `{title description} : ${gramPhrase(run)}`]), filters[4]!);
|
|
412
|
+
return rows.filter(row => !snapshot || snapshot.get(row.path) === row.sourceHash)
|
|
355
413
|
.slice(snapshot ? snapshotOffset : 0, (snapshot ? snapshotOffset : 0) + (snapshot && allSnapshotRows ? 10000 : limit))
|
|
356
414
|
.map(({ sourceHash: _, ...row }) => ({ ...row, title: boundedText(row.title, 512).content, snippet: boundedText(row.snippet, 1024).content, type: boundedText(row.type, 256).content, domain: boundedText(row.domain, 256).content }));
|
|
357
415
|
}
|
|
@@ -397,13 +455,51 @@ export class VaultIndex<Source extends VaultSpec | SpecSourceSpec = VaultSpec> {
|
|
|
397
455
|
}
|
|
398
456
|
return { path, hash: note.hash, ...readPage(Buffer.from(note.prefix), maxBytes, start, note.bytes), ...metadata, ...headingMetadata(sections) };
|
|
399
457
|
}
|
|
400
|
-
/**
|
|
401
|
-
|
|
458
|
+
/**
|
|
459
|
+
* Indexed notes (paths relative to notes/) ordered by path, optionally filtered by a literal stem prefix.
|
|
460
|
+
* `cursor` is the previous page's `nextCursor` (its last path, exclusive); `total` counts the filtered set.
|
|
461
|
+
*/
|
|
462
|
+
list(prefix = "", limit = 50, cursor?: string): { notes: { path: string; title: string; type: string; domain: string; description: string; mtime: string | null }[]; total: number; nextCursor?: string } {
|
|
402
463
|
if (Buffer.byteLength(prefix) > 1024) throw new Error("prefix: expected <=1024 bytes");
|
|
403
|
-
const key = prefix.replace(/^notes\//i, "");
|
|
404
|
-
|
|
405
|
-
|
|
406
|
-
|
|
464
|
+
const key = prefix.replace(/^notes\//i, ""), max = resultLimit(limit);
|
|
465
|
+
if (cursor !== undefined && (!cursor || Buffer.byteLength(cursor) > 1024 || !cursor.startsWith(key))) throw new Error("cursor: expected the nextCursor of a previous page listed with the same prefix; list again without cursor to restart");
|
|
466
|
+
const rows = this.db.query<{ path: string; title: string; type: string; domain: string; description: string; metadata: string }, [number, string, string, number]>(
|
|
467
|
+
"SELECT path,title,type,domain,substr(description,1,512) AS description,metadata FROM notes WHERE substr(path,1,?)=? AND path>? ORDER BY path LIMIT ?"
|
|
468
|
+
).all(key.length, key, cursor ?? "", max + 1);
|
|
469
|
+
const total = this.db.query<{ count: number }, [number, string]>("SELECT count(*) AS count FROM notes WHERE substr(path,1,?)=?").get(key.length, key)!.count;
|
|
470
|
+
const notes = rows.slice(0, max).map(({ metadata: stamp, ...row }) => {
|
|
471
|
+
const mtime = stampMtime(stamp);
|
|
472
|
+
return { ...row, title: boundedText(row.title, 512).content, type: boundedText(row.type, 256).content, domain: boundedText(row.domain, 256).content, description: boundedText(row.description, 512).content, mtime: mtime === null ? null : new Date(mtime).toISOString() };
|
|
473
|
+
});
|
|
474
|
+
return { notes, total, ...(rows.length > max ? { nextCursor: notes.at(-1)!.path } : {}) };
|
|
475
|
+
}
|
|
476
|
+
/**
|
|
477
|
+
* Note counts by type, domain and frontmatter `status` ("(none)" when empty), and how many notes have an mtime at or
|
|
478
|
+
* after each cutoff (epoch ms). Streams rows; returns no note text.
|
|
479
|
+
*/
|
|
480
|
+
noteStats(cutoffs: readonly number[]): { total: number; byType: Record<string, number>; byDomain: Record<string, number>; byStatus: Record<string, number>; modified: number[] } {
|
|
481
|
+
const add = (map: Map<string, number>, value: string, count = 1) => { const key = value ? boundedText(value, 256).content : "(none)"; map.set(key, (map.get(key) ?? 0) + count); };
|
|
482
|
+
const sorted = (map: Map<string, number>) => Object.fromEntries([...map].sort(([a], [b]) => a < b ? -1 : a > b ? 1 : 0));
|
|
483
|
+
const grouped = (column: "type" | "domain") => {
|
|
484
|
+
const map = new Map<string, number>();
|
|
485
|
+
for (const row of this.db.query<{ value: string; count: number }, []>(`SELECT ${column} AS value, count(*) AS count FROM notes GROUP BY ${column}`).all()) add(map, row.value, row.count);
|
|
486
|
+
return sorted(map);
|
|
487
|
+
};
|
|
488
|
+
// Status is not a column: stream each stored source once, parsing only its frontmatter.
|
|
489
|
+
const byStatus = new Map<string, number>(), modified = cutoffs.map(() => 0);
|
|
490
|
+
let total = 0;
|
|
491
|
+
const rows = this.db.prepare<{ metadata: string; source: string }, []>("SELECT metadata,source FROM notes");
|
|
492
|
+
try {
|
|
493
|
+
for (const row of rows.iterate()) {
|
|
494
|
+
total++;
|
|
495
|
+
let status: unknown;
|
|
496
|
+
try { status = parseFrontmatter(row.source).data.status; } catch { /* counted as (none) */ }
|
|
497
|
+
add(byStatus, text(status));
|
|
498
|
+
const mtime = stampMtime(row.metadata);
|
|
499
|
+
cutoffs.forEach((cutoff, i) => { if (mtime !== null && mtime >= cutoff) modified[i]!++; });
|
|
500
|
+
}
|
|
501
|
+
} finally { rows.finalize(); }
|
|
502
|
+
return { total, byType: grouped("type"), byDomain: grouped("domain"), byStatus: sorted(byStatus), modified };
|
|
407
503
|
}
|
|
408
504
|
/** Links that resolve to the target note, ordered by source. */
|
|
409
505
|
backlinks(target: string, limit = 50): ResolvedLink[] {
|
package/src/vault-log.ts
CHANGED
|
@@ -67,6 +67,16 @@ function context(home: string, id: string, channel: "cli" | "mcp" | undefined, w
|
|
|
67
67
|
if (isDenied(join(vault.path, "log"), config.denyPaths)) refuse("V06", "log path is denied");
|
|
68
68
|
return { vault, config, channel, ...(write && vault.registry ? { projectIds: registryIds(vault.registry, config.denyPaths) } : {}) };
|
|
69
69
|
}
|
|
70
|
+
/** Validate and screen an event exactly as `appendLog` would, writing nothing. Returns the blocking issues (empty when valid). */
|
|
71
|
+
export function checkLogEvent(home: string, request: LogRequest, env: NodeJS.ProcessEnv = process.env): WriteIssue[] {
|
|
72
|
+
try {
|
|
73
|
+
const ctx = context(home, request.vault, request.channel, true);
|
|
74
|
+
return formatLogEvent(request.event, { ...ctx, env }).issues.filter(issue => issue.severity === "error");
|
|
75
|
+
} catch (caught) {
|
|
76
|
+
if (caught instanceof LogRefused) return caught.issues;
|
|
77
|
+
throw caught;
|
|
78
|
+
}
|
|
79
|
+
}
|
|
70
80
|
/** One O_APPEND write per event. A separate cooperative lock permits safe short-write rollback. */
|
|
71
81
|
/** Bind a later append to the vault spec, path policy and root identity resolved now (refused with LOG_DRIFT if they change). */
|
|
72
82
|
export function logBinding(vault: VaultSpec, config: VmemConfig): LogBinding {
|
|
@@ -189,7 +199,7 @@ export function readLog(home: string, options: LogReadOptions): LogReadResult {
|
|
|
189
199
|
const result: LogReadResult = { ok: false, lines: [], dataLines: null, warnings: [], issues: [] };
|
|
190
200
|
let fd: number | undefined;
|
|
191
201
|
try {
|
|
192
|
-
if (!options || typeof options !== "object" || typeof options.vault !== "string" || Object.keys(options).some(key => !["vault", "limit", "grep", "type"].includes(key))) refuse("L09", "invalid log read options");
|
|
202
|
+
if (!options || typeof options !== "object" || typeof options.vault !== "string" || Object.keys(options).some(key => !["vault", "limit", "grep", "type", "visit"].includes(key)) || (options.visit !== undefined && typeof options.visit !== "function")) refuse("L09", "invalid log read options");
|
|
193
203
|
const limit = options.limit ?? 50;
|
|
194
204
|
if (!Number.isInteger(limit) || limit < 1 || limit > 200) refuse("L09", "log limit must be an integer 1..200");
|
|
195
205
|
if (options.type !== undefined && !(LOG_TYPES as readonly string[]).includes(options.type)) refuse("L09", "unknown log type filter");
|
|
@@ -222,6 +232,7 @@ export function readLog(home: string, options: LogReadOptions): LogReadResult {
|
|
|
222
232
|
} else {
|
|
223
233
|
const line = pending.endsWith("\r") ? pending.slice(0, -1) : pending;
|
|
224
234
|
const type = /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}\s+(\S+)/.exec(line)?.[1];
|
|
235
|
+
options.visit?.(line);
|
|
225
236
|
if ((!options.type || options.type === type) && (!filter || filter(line))) {
|
|
226
237
|
result.lines.push(line); resultBytes += Buffer.byteLength(JSON.stringify(line)) + 1;
|
|
227
238
|
while (result.lines.length > limit || resultBytes > MAX_LOG_RESULT_BYTES) resultBytes -= Buffer.byteLength(JSON.stringify(result.lines.shift()!)) + 1;
|
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
import { localDate } from "./local-date.ts";
|
|
2
|
+
import { LOG_TYPES } from "./log-format.ts";
|
|
3
|
+
import { loadRegistry } from "./registry.ts";
|
|
4
|
+
import { VaultIndex } from "./vault-index.ts";
|
|
5
|
+
import { readLog } from "./vault-log.ts";
|
|
6
|
+
import type { WriteIssue } from "./write-types.ts";
|
|
7
|
+
import type { VaultSpec } from "./config.ts";
|
|
8
|
+
|
|
9
|
+
export const STATS_DEFAULT_DAYS = 30;
|
|
10
|
+
export const STATS_MAX_DAYS = 3650;
|
|
11
|
+
export interface VaultStats {
|
|
12
|
+
vault: string;
|
|
13
|
+
/** Local calendar date the windows end on (today included). */
|
|
14
|
+
asOf: string;
|
|
15
|
+
notes: { total: number; byType: Record<string, number>; byDomain: Record<string, number>; byStatus: Record<string, number> };
|
|
16
|
+
/** Notes whose indexed mtime falls within the last 7 and 30 local calendar days, today included. */
|
|
17
|
+
modified: { last7Days: number; last30Days: number };
|
|
18
|
+
/** Log events dated on or after `since` (local date), by type; types outside the known set count as "(other)". */
|
|
19
|
+
log: { days: number; since: string; total: number | null; byType: Record<string, number> | null; warnings: string[]; issues: WriteIssue[] };
|
|
20
|
+
}
|
|
21
|
+
/**
|
|
22
|
+
* Read-only vault statistics over the derived index (refreshed like every read tool) and the log streamed by `readLog`:
|
|
23
|
+
* no vault file or log is written and no vault or log lock is taken.
|
|
24
|
+
*/
|
|
25
|
+
export function vaultStats(home: string, id: string, options: { days?: number; now?: Date; expectedVault?: VaultSpec } = {}): VaultStats {
|
|
26
|
+
const days = options.days ?? STATS_DEFAULT_DAYS, now = options.now ?? new Date();
|
|
27
|
+
if (!Number.isInteger(days) || days < 1 || days > STATS_MAX_DAYS) throw new Error(`stats: days must be an integer 1..${STATS_MAX_DAYS}; omit it for the ${STATS_DEFAULT_DAYS}-day default`);
|
|
28
|
+
const registry = loadRegistry(home), vault = registry.vaults.find(v => v.id === id);
|
|
29
|
+
if (!vault) throw Object.assign(new Error("stats: unknown vault id; run vmem vaults list for the registered ids"), { code: "V04" });
|
|
30
|
+
// An MCP server reads the vault set fixed at startup; a re-registered id would otherwise count a different vault.
|
|
31
|
+
if (options.expectedVault && JSON.stringify(options.expectedVault) !== JSON.stringify(vault)) throw new Error("stats: the vault registration changed since the MCP server started; restart the MCP server");
|
|
32
|
+
// Local midnight that opens a window of n calendar days ending today.
|
|
33
|
+
const start = (n: number) => new Date(now.getFullYear(), now.getMonth(), now.getDate() - (n - 1));
|
|
34
|
+
const index = new VaultIndex(home, vault, registry.vaults.map(v => v.path));
|
|
35
|
+
let notes: ReturnType<VaultIndex["noteStats"]>;
|
|
36
|
+
try { index.scan(); notes = index.noteStats([start(7).getTime(), start(30).getTime()]); } finally { index.close(); }
|
|
37
|
+
const since = localDate(start(days)), byType: Record<string, number> = Object.fromEntries(LOG_TYPES.map(type => [type, 0]));
|
|
38
|
+
let total = 0;
|
|
39
|
+
const log = readLog(home, { vault: vault.id, limit: 1, visit: line => {
|
|
40
|
+
const match = /^(\d{4}-\d{2}-\d{2}) \d{2}:\d{2}\s+(\S+)/.exec(line);
|
|
41
|
+
if (!match || match[1]! < since) return;
|
|
42
|
+
const type = (LOG_TYPES as readonly string[]).includes(match[2]!) ? match[2]! : "(other)";
|
|
43
|
+
total++; byType[type] = (byType[type] ?? 0) + 1;
|
|
44
|
+
} });
|
|
45
|
+
const { modified: [last7Days = 0, last30Days = 0], ...counts } = notes;
|
|
46
|
+
return {
|
|
47
|
+
vault: vault.id, asOf: localDate(now), notes: counts, modified: { last7Days, last30Days },
|
|
48
|
+
log: { days, since, total: log.ok ? total : null, byType: log.ok ? byType : null, warnings: log.warnings, issues: log.issues },
|
|
49
|
+
};
|
|
50
|
+
}
|
package/src/write-commands.ts
CHANGED
|
@@ -5,7 +5,10 @@ import { loadRegistry } from "./registry.ts";
|
|
|
5
5
|
import { basename } from "node:path";
|
|
6
6
|
import { applyWrite, MAX_WRITE_BYTES } from "./writer.ts";
|
|
7
7
|
import { notePath } from "./write-operations.ts";
|
|
8
|
-
import type { WriteOp, WriteRequest } from "./write-types.ts";
|
|
8
|
+
import type { PatchEdit, WriteOp, WriteRequest } from "./write-types.ts";
|
|
9
|
+
import type { LogEvent } from "./log-types.ts";
|
|
10
|
+
import { LOG_EVENT_SCHEMA } from "./log-format.ts";
|
|
11
|
+
import { argumentError } from "./mcp.ts";
|
|
9
12
|
import { parseTidyMap, tidyVault, TIDY_RULES } from "./tidy.ts";
|
|
10
13
|
const COMMANDS = new Set(["new", "patch", "append", "frontmatter", "link", "supersede", "move", "delete", "tidy", "report", "check"]);
|
|
11
14
|
export function isWriteCommand(command: string): boolean { return COMMANDS.has(command); }
|
|
@@ -29,15 +32,24 @@ function input(file: string, home: string): string {
|
|
|
29
32
|
return text;
|
|
30
33
|
} finally { if (file !== "-") closeSync(fd); }
|
|
31
34
|
}
|
|
35
|
+
/** `--log JSON` checked against the same closed schema as the MCP `log` argument; registry and screening checks follow in applyWrite. */
|
|
36
|
+
function logOption(text: string | undefined): { log?: LogEvent } {
|
|
37
|
+
if (text === undefined) return {};
|
|
38
|
+
let value: unknown;
|
|
39
|
+
try { value = JSON.parse(text); } catch { throw new Error("--log: invalid JSON; expected {\"type\", \"project\", \"source\", \"note\"[, \"dest\"]}"); }
|
|
40
|
+
const error = argumentError(value, LOG_EVENT_SCHEMA, "--log");
|
|
41
|
+
if (error) throw new Error(`${error}; expected {"type", "project", "source", "note"[, "dest"]} as for log append`);
|
|
42
|
+
return { log: value as LogEvent };
|
|
43
|
+
}
|
|
32
44
|
export function runWriteCommand(command: string, args: string[], home: string, env: NodeJS.ProcessEnv = process.env): { text: string; code: number } {
|
|
33
45
|
const valueOptions: Record<string, string[]> = {
|
|
34
46
|
new: ["--type", "--domain", "--topics", "--description", "--title", "--source", "--confidence", "--status", "--extra", "--body"],
|
|
35
|
-
patch: ["--old", "--new", "--expected-sha"], append: ["--section", "--markdown", "--expected-sha"],
|
|
47
|
+
patch: ["--old", "--new", "--edits", "--expected-sha"], append: ["--section", "--markdown", "--expected-sha"],
|
|
36
48
|
frontmatter: ["--set", "--unset", "--expected-sha"], link: ["--relation", "--context"],
|
|
37
49
|
move: ["--fingerprint"], delete: ["--fingerprint"], tidy: ["--fingerprint", "--map"], report: ["--body"], check: ["--path"], supersede: [],
|
|
38
50
|
};
|
|
39
|
-
// For tidy, --log selects the log-redaction rule; tidy always writes its own LINT event.
|
|
40
|
-
const values = valueOptions[command] ? [...valueOptions[command]!, ...(
|
|
51
|
+
// For tidy, --log selects the log-redaction rule; tidy always writes its own LINT event. Check never appends an event.
|
|
52
|
+
const values = valueOptions[command] ? [...valueOptions[command]!, ...(["tidy", "check"].includes(command) ? [] : ["--log"])] : undefined;
|
|
41
53
|
if (!values) throw new Error("unknown writer command");
|
|
42
54
|
const twoStep = ["move", "delete", "tidy"].includes(command);
|
|
43
55
|
const booleans = ["--json", "--dry-run", ...(twoStep ? ["--confirm"] : []), ...(command === "link" ? ["--one-way"] : []), ...(command === "tidy" ? TIDY_RULES.map(rule => `--${rule}`).concat("--all") : [])];
|
|
@@ -80,7 +92,16 @@ export function runWriteCommand(command: string, args: string[], home: string, e
|
|
|
80
92
|
if (optional("--topics") !== undefined) frontmatter.topics = required("--topics").split(",").filter(Boolean).map(value => value.startsWith("[[") ? value : `[[${value}]]`);
|
|
81
93
|
operation = { op: "create", path: target, frontmatter, ...(optional("--body") ? { body: input(required("--body"), home) } : {}) }; break;
|
|
82
94
|
}
|
|
83
|
-
case "patch":
|
|
95
|
+
case "patch": {
|
|
96
|
+
// --edits takes the same JSON array as the MCP `edits` argument.
|
|
97
|
+
if (optional("--edits") === undefined) { operation = { op: "patch", path: target, oldString: required("--old"), newString: required("--new"), expectedSha: optional("--expected-sha") }; break; }
|
|
98
|
+
if (optional("--old") !== undefined || optional("--new") !== undefined) throw new Error("patch: use --old/--new or --edits, not both");
|
|
99
|
+
const text = input(required("--edits"), home);
|
|
100
|
+
let edits: unknown;
|
|
101
|
+
try { edits = JSON.parse(text); } catch { throw new Error("--edits: invalid JSON; expected an array of {oldString, newString}"); }
|
|
102
|
+
if (!Array.isArray(edits)) throw new Error("--edits: expected a JSON array of {oldString, newString}");
|
|
103
|
+
operation = { op: "patch", path: target, edits: edits as PatchEdit[], expectedSha: optional("--expected-sha") }; break;
|
|
104
|
+
}
|
|
84
105
|
case "append": operation = { op: "append", path: target, section: required("--section"), markdown: required("--markdown"), expectedSha: optional("--expected-sha") }; break;
|
|
85
106
|
case "frontmatter": operation = { op: "frontmatter", path: target, set: optional("--set") ? object(required("--set"), "set") : {}, unset: optional("--unset")?.split(","), expectedSha: optional("--expected-sha") }; break;
|
|
86
107
|
case "link": operation = { op: "link", from: target, to: positional[2]!, relation: required("--relation") as Extract<WriteOp, { op: "link" }>["relation"], context: optional("--context"), bidirectional: !flags["--one-way"] }; break;
|
|
@@ -92,7 +113,7 @@ export function runWriteCommand(command: string, args: string[], home: string, e
|
|
|
92
113
|
case "check": operation = { op: "check", path: notePath(optional("--path") ?? (target === "-" ? "checked-note" : basename(target))), text: input(target, home) }; break;
|
|
93
114
|
default: throw new Error("unknown writer command");
|
|
94
115
|
}
|
|
95
|
-
const request: WriteRequest = { vault, operation, channel: "cli", dryRun: flags["--dry-run"] === true, confirm: flags["--confirm"] === true, fingerprint: optional("--fingerprint"), ...(optional("--log")
|
|
116
|
+
const request: WriteRequest = { vault, operation, channel: "cli", dryRun: flags["--dry-run"] === true, confirm: flags["--confirm"] === true, fingerprint: optional("--fingerprint"), ...logOption(optional("--log")) };
|
|
96
117
|
const result = applyWrite(home, request, { env });
|
|
97
118
|
return { text: JSON.stringify(result, null, flags["--json"] ? undefined : 2), code: result.ok && result.log?.ok !== false ? 0 : 1 };
|
|
98
119
|
}
|
package/src/write-operations.ts
CHANGED
|
@@ -2,7 +2,7 @@ import { isDeepStrictEqual } from "node:util";
|
|
|
2
2
|
import { normalizeSource } from "./source.ts";
|
|
3
3
|
import { parseFrontmatter, emitFrontmatter } from "./frontmatter.ts";
|
|
4
4
|
import { normalizeTarget, proseLines, rewriteLinkTargets, extractLinks, resolveLinks, type LinkNote } from "./links.ts";
|
|
5
|
-
import type { PlannedChange, WriteIssue, WriteOp } from "./write-types.ts";
|
|
5
|
+
import type { PatchEdit, PlannedChange, WriteIssue, WriteOp } from "./write-types.ts";
|
|
6
6
|
export class WriteRefusal extends Error {
|
|
7
7
|
constructor(readonly issues: WriteIssue[]) { super(issues[0]?.message ?? "write refused"); }
|
|
8
8
|
}
|
|
@@ -149,11 +149,28 @@ export function planOperation(operation: WriteOp, notes: Map<string, string>, re
|
|
|
149
149
|
}
|
|
150
150
|
case "patch": {
|
|
151
151
|
const { path, source } = existing(operation.path);
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
if(
|
|
156
|
-
|
|
152
|
+
const single = operation.oldString !== undefined || operation.newString !== undefined;
|
|
153
|
+
if (single === (operation.edits !== undefined)) refuse("V01", "patch takes either oldString/newString or edits, not both or neither; send one form", path);
|
|
154
|
+
const edits: unknown[] = single ? [{ oldString: operation.oldString, newString: operation.newString }] : Array.isArray(operation.edits) ? operation.edits : [];
|
|
155
|
+
if (edits.length < 1 || edits.length > 32) refuse("V09", "edits must list 1 to 32 replacements; split a larger change into several patches", path);
|
|
156
|
+
const frontmatterEnd = document(source, path).frontmatter?.length ?? 0;
|
|
157
|
+
// Every edit is located in the original text, so earlier edits cannot create or remove a later match.
|
|
158
|
+
const spans = edits.map((edit, index) => {
|
|
159
|
+
const at = single ? "" : `edits[${index}]: `;
|
|
160
|
+
if (!edit || typeof edit !== "object" || Array.isArray(edit) || Object.keys(edit).some(key => key !== "oldString" && key !== "newString")) refuse("V01", `${at}an edit is exactly {oldString, newString}`, path);
|
|
161
|
+
const { oldString, newString } = edit as Partial<PatchEdit>;
|
|
162
|
+
if (typeof oldString !== "string" || !oldString || typeof newString !== "string" || Buffer.byteLength(oldString) > 8192 || Buffer.byteLength(newString) > 8192) refuse("V09", `${at}patch strings must be bounded and oldString nonempty`, path);
|
|
163
|
+
const first = source.indexOf(oldString);
|
|
164
|
+
if (first < 0 || source.indexOf(oldString, first + 1) >= 0) refuse("V09", single ? "oldString must match exactly once" : `${at}oldString must match exactly once in the original note; re-read the note and widen the string`, path);
|
|
165
|
+
if (first < frontmatterEnd) refuse("V01", `${at}patch cannot target frontmatter; use vmem_set_frontmatter or the frontmatter command`, path);
|
|
166
|
+
return { start: first, end: first + oldString.length, text: newString, index };
|
|
167
|
+
}).sort((a, b) => a.start - b.start);
|
|
168
|
+
let after = "", cursor = 0;
|
|
169
|
+
for (const span of spans) {
|
|
170
|
+
if (span.start < cursor) refuse("V09", `edits[${span.index}]: oldString overlaps another edit; merge overlapping edits into one`, path);
|
|
171
|
+
after += source.slice(cursor, span.start) + span.text; cursor = span.end;
|
|
172
|
+
}
|
|
173
|
+
set(path, after + source.slice(cursor));
|
|
157
174
|
break;
|
|
158
175
|
}
|
|
159
176
|
case "append": { const { path, source } = existing(operation.path); set(path, appendSection(source, operation.section, operation.markdown, path)); break; }
|
package/src/write-types.ts
CHANGED
|
@@ -1,10 +1,12 @@
|
|
|
1
1
|
import type { PiiSurface } from "./config.ts";
|
|
2
2
|
import type { LogEvent, LogReceipt } from "./log-types.ts";
|
|
3
3
|
import type { VaultSpec, VmemConfig } from "./config.ts";
|
|
4
|
+
export interface PatchEdit { oldString: string; newString: string }
|
|
4
5
|
export type WriteOp =
|
|
5
6
|
| { op: "check"; path: string; text: string }
|
|
6
7
|
| { op: "create"; path: string; frontmatter: Record<string, unknown>; body?: string }
|
|
7
|
-
|
|
8
|
+
/** One `oldString`/`newString` pair or `edits` (1..32 pairs), never both; edits apply all-or-nothing against the original text. */
|
|
9
|
+
| { op: "patch"; path: string; oldString?: string; newString?: string; edits?: PatchEdit[]; expectedSha?: string }
|
|
8
10
|
| { op: "append"; path: string; section: string; markdown: string; expectedSha?: string }
|
|
9
11
|
| { op: "frontmatter"; path: string; set: Record<string, unknown>; unset?: string[]; expectedSha?: string }
|
|
10
12
|
| { op: "link"; from: string; to: string; relation: "supports" | "contradicts" | "extends" | "applies-to" | "context-for"; context?: string; bidirectional?: boolean }
|
package/src/writer.ts
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { appendLog, matchesBinding } from "./vault-log.ts";
|
|
1
|
+
import { appendLog, checkLogEvent, matchesBinding } from "./vault-log.ts";
|
|
2
2
|
import type { LogEvent } from "./log-types.ts";
|
|
3
3
|
import { closeSync, constants, fstatSync, fsyncSync, lstatSync, mkdirSync, openSync, readSync, readdirSync, realpathSync, renameSync, rmdirSync, unlinkSync, writeFileSync } from "node:fs";
|
|
4
4
|
import { createHash, randomUUID } from "node:crypto";
|
|
@@ -6,7 +6,7 @@ import { dirname, join } from "node:path";
|
|
|
6
6
|
import { hostname } from "node:os";
|
|
7
7
|
import { Database } from "bun:sqlite";
|
|
8
8
|
import { loadRegistry, syncDirectory } from "./registry.ts";
|
|
9
|
-
import { VaultIndex, MAX_NOTE_BYTES } from "./vault-index.ts";
|
|
9
|
+
import { VaultIndex, MAX_NOTE_BYTES, GRAMS_COLUMNS, gramPhrase, grams, splitTerms } from "./vault-index.ts";
|
|
10
10
|
import { parseFrontmatter } from "./frontmatter.ts";
|
|
11
11
|
import { canonicalPath, isDenied } from "./platform.ts";
|
|
12
12
|
import { moveSourcePath, notePath, planOperation, refuse, WriteRefusal } from "./write-operations.ts";
|
|
@@ -102,14 +102,20 @@ function duplicateWarnings(request: WriteRequest, notes: Map<string, string>, ch
|
|
|
102
102
|
// Disposable FTS preserves dry-run's no-disk-write contract, including on a fresh home.
|
|
103
103
|
const db = new Database(":memory:");
|
|
104
104
|
try {
|
|
105
|
-
db.run(
|
|
106
|
-
|
|
105
|
+
db.run(`CREATE VIRTUAL TABLE candidates USING fts5(path UNINDEXED,title,description,body,tokenize='unicode61');
|
|
106
|
+
CREATE VIRTUAL TABLE grams USING fts5(${GRAMS_COLUMNS},tokenize='unicode61');`);
|
|
107
|
+
const insert = db.query("INSERT INTO candidates VALUES (?,?,?,?)"), insertGrams = db.query("INSERT INTO grams VALUES (?,?,?,?)");
|
|
107
108
|
for (const [path, source] of notes) {
|
|
108
109
|
let value: ReturnType<typeof parseFrontmatter>;
|
|
109
110
|
try { value = parseFrontmatter(source); } catch { continue; } // Unrelated legacy YAML is not a duplicate candidate.
|
|
110
|
-
|
|
111
|
+
const title = typeof value.data.title === "string" ? value.data.title : path, description = typeof value.data.description === "string" ? value.data.description : "";
|
|
112
|
+
insert.run(path, title, description, value.body); insertGrams.run(path, grams(title), grams(description), grams(value.body));
|
|
111
113
|
}
|
|
112
|
-
|
|
114
|
+
// Same term split as VaultIndex.search: CJK terms match substrings through the bigram table.
|
|
115
|
+
const { tokens, runs } = splitTerms(terms);
|
|
116
|
+
const hits = runs.length
|
|
117
|
+
? db.query<{ path: string; score: number }, string[]>(`SELECT path,bm25(grams,0,5,2,1) AS score FROM grams WHERE grams MATCH ? ${tokens.length ? "AND path IN (SELECT path FROM candidates WHERE candidates MATCH ?)" : ""} ORDER BY score,path LIMIT 3`).all(runs.map(gramPhrase).join(" AND "), ...(tokens.length ? [tokens.join(" AND ")] : []))
|
|
118
|
+
: db.query<{ path: string; score: number }, [string]>("SELECT path,bm25(candidates,0,5,2,1) AS score FROM candidates WHERE candidates MATCH ? ORDER BY score,path LIMIT 3").all(tokens.join(" AND "));
|
|
113
119
|
return hits.filter(hit => hit.score <= -0.000001).map(hit => ({ code: "V08", severity: "warning", path: hit.path, message: "possible duplicate: FTS title terms match an existing note" }));
|
|
114
120
|
} finally { db.close(); }
|
|
115
121
|
}
|
|
@@ -136,6 +142,11 @@ export function applyWrite(home: string, request: WriteRequest, options: { env?:
|
|
|
136
142
|
root = realpathSync(vault.path);
|
|
137
143
|
if (root !== vault.path || isDenied(root, config.denyPaths)) refuse("V06", "vault path denied or changed");
|
|
138
144
|
if (config.vaults.some(value => canonicalPath(home) === value.path || canonicalPath(home).startsWith(`${value.path}/`))) refuse("V04", "VMEM_HOME must be outside registered vaults");
|
|
145
|
+
// A given event is checked like `log append` before any lock or note change, so a bad event never leaves an unlogged write.
|
|
146
|
+
if (request.log !== undefined && request.log !== false && request.operation.op !== "check") {
|
|
147
|
+
const logIssues = checkLogEvent(home, { vault: request.vault, event: request.log, channel: request.channel }, options.env);
|
|
148
|
+
if (logIssues.length) throw new WriteRefusal(logIssues.map(issue => ({ ...issue, message: `log event refused, nothing written: ${issue.message}; fix the event fields (rules as for \`log append\`) or omit the event for the default line` })));
|
|
149
|
+
}
|
|
139
150
|
const twoStep = ["move", "delete", "tidy"].includes(request.operation.op);
|
|
140
151
|
const preview = request.operation.op === "check" || request.dryRun === true || (twoStep && request.confirm !== true);
|
|
141
152
|
if (!preview) {
|
|
@@ -159,7 +170,8 @@ export function applyWrite(home: string, request: WriteRequest, options: { env?:
|
|
|
159
170
|
for (const change of changes) if (isDenied(join(root, change.path), config.denyPaths)) refuse("V06", "planned mutation path is denied", change.path);
|
|
160
171
|
issues = validateWrite(changes, { vault, config, notes, operation: request.operation, ...(options.piiSurface ? { piiSurface: options.piiSurface } : {}) });
|
|
161
172
|
if (!issues.some(issue => issue.severity === "error")) issues.push(...duplicateWarnings(request, notes, changes));
|
|
162
|
-
|
|
173
|
+
// The log event is not part of the fingerprint: move/delete take it on the confirm call only.
|
|
174
|
+
const input = { vault: request.vault, channel: request.channel ?? "cli", operation: request.operation, expectedShas: request.expectedShas ?? {} };
|
|
163
175
|
const identity = lstatSync(root);
|
|
164
176
|
// The identity every later commit check uses must still be the bound root, not a directory swapped in meanwhile.
|
|
165
177
|
if (options.binding && (Number(identity.dev) !== options.binding.dev || Number(identity.ino) !== options.binding.ino)) refuse("VAULT_DRIFT", "vault root changed since the run started; nothing written");
|