@staix/vault-mem 0.1.3 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +7 -0
- package/README.md +46 -5
- package/package.json +1 -1
- package/src/cli.ts +14 -4
- package/src/commands-table.ts +1 -0
- package/src/config.ts +44 -5
- package/src/log-format.ts +4 -2
- package/src/mcp-vault.ts +1 -1
- package/src/org-promote.ts +7 -5
- package/src/pii.ts +133 -0
- package/src/registry.ts +20 -4
- package/src/tidy.ts +6 -4
- package/src/vault-log.ts +1 -0
- package/src/write-types.ts +3 -0
- package/src/write-validation.ts +9 -40
- package/src/writer.ts +4 -3
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,12 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [0.2.0] - 2026-10-30
|
|
4
|
+
|
|
5
|
+
- Fine-grained PII policy: `vault.pii` can set `allow`, `warn` or `deny` per category (email, phone, the global `piiPatterns`, named per-vault patterns) and per surface (notes, log, promotion), with an allow list of exact addresses, domains and phone numbers. The `allow`/`deny` shorthand keeps its previous meaning.
|
|
6
|
+
- `vmem vaults pii <id>` shows the effective policy; `--set` previews a change and lists every loosened cell, and `--confirm` applies it. `vaults list` and `vmem_status` report the matrix as `piiPolicy`.
|
|
7
|
+
- `vmem tidy --log` redacts only PII that the policy denies on the log surface, so personal vaults keep historical email addresses; secrets are still redacted everywhere.
|
|
8
|
+
- Org promotion reports a `warn` cell in the target policy as a warning instead of refusing.
|
|
9
|
+
|
|
3
10
|
## [0.1.3] - 2026-10-30
|
|
4
11
|
|
|
5
12
|
- `vmem tidy --log` fixes legacy log lines that fail lint L09: a trailing seconds field is dropped, and any other malformed line is kept verbatim behind `# legacy: `.
|
package/README.md
CHANGED
|
@@ -31,10 +31,51 @@ Durability: the temporary file is fsynced before the rename and the home directo
|
|
|
31
31
|
|
|
32
32
|
Crash recovery: the lock file records the writer's pid, host and start time. Nothing is cleaned up automatically. `vmem recover` previews the state; `vmem recover --confirm` removes the lock only when it names this host and a pid that no longer runs, and then removes leftover `.config-<uuid>.tmp` files. A live lock, a lock from another host, or an unreadable (older-format) lock is never removed and the command exits 1; confirm no writer runs, then delete `config.lock` by hand. No other file in the home or any vault is touched. If cleanup fails after an operation (for example the lock cannot be removed), the original error is kept and the cleanup failure, including a retained lock, is appended to it.
|
|
33
33
|
|
|
34
|
-
Vault fields: `id`, absolute `path` (must contain a real `notes/` directory), `kind` (`personal|org`), `write` (`mcp|cli|none`), optional `pii` (`allow|deny
|
|
34
|
+
Vault fields: `id`, absolute `path` (must contain a real `notes/` directory), `kind` (`personal|org`), `write` (`mcp|cli|none`), optional `pii` (the `allow|deny` shorthand or a policy object, see PII policy below; default `allow` for personal and `deny` for org, and `status`/`vaults list` always report the effective shorthand and `piiPolicy` matrix, including for unhealthy entries), optional `remote` (any nonempty string such as an `https://` or `git@host:path` URL; stored only) and optional `registry` (absolute path; stored only). No sync exists yet, so `remote` and `registry` are not validated against a server and doctor does not check them; reachability checks apply once sync is implemented. Policy fields are stored for later note-writing stages, not enforced by the read commands. Writer commands enforce write/channel policy and configured deny/PII screening.
|
|
35
35
|
|
|
36
36
|
The only runtime dependency is the exact first-party pin `@staix/agent-mem@0.11.0`; it is not loaded by registry commands. Public library exports cover configuration, home resolution, registry operations, frontmatter, link resolution and the derived index. No console bin or UI contract is advertised before its implementation.
|
|
37
37
|
|
|
38
|
+
# PII policy
|
|
39
|
+
|
|
40
|
+
`pii` is either the shorthand `allow`/`deny` (every category on every surface) or an object that sets an action per category and, optionally, per surface:
|
|
41
|
+
|
|
42
|
+
```json
|
|
43
|
+
{"email": {"notes": "allow", "log": "deny"}, "phone": "warn", "custom": "deny",
|
|
44
|
+
"patterns": [{"name": "student-id", "regex": "S\\d{6}", "action": {"notes": "deny", "log": "allow"}}],
|
|
45
|
+
"allow": ["owner@example.org", "@example.org", "+82 10 0000 0000"]}
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
- Categories:
|
|
49
|
+
- `email`;
|
|
50
|
+
- `phone`;
|
|
51
|
+
- `custom`, the global `piiPatterns`;
|
|
52
|
+
- named per-vault `patterns` (at most 32; names `[a-z][a-z0-9-]{0,31}`, regexes compiled with the `u` flag).
|
|
53
|
+
- Surfaces:
|
|
54
|
+
- `notes`: note and report writes, frontmatter screening, tidy value echo;
|
|
55
|
+
- `log`: log appends and `tidy --log` on existing lines;
|
|
56
|
+
- `promotion`: the screen `vmem promote` applies when this vault is the target.
|
|
57
|
+
- Actions:
|
|
58
|
+
- `allow`: not screened.
|
|
59
|
+
- `warn`: the write proceeds with a V07 warning, and tidy echo prints the value as `[redacted]`.
|
|
60
|
+
- `deny`: V07 error, and `tidy --log` replaces matches in existing lines.
|
|
61
|
+
- Org promotion refuses only `deny` cells and secrets. The promoted note's write into the target vault is screened with the target's `promotion` cells, not its `notes` cells.
|
|
62
|
+
- `tidy --log` replaces the raw denied matches and leaves allow-listed values intact. Screening also checks the NFKC form. If a denied match is zero-length, or the replaced line still contains a denied match (for example a form only NFKC normalization reveals, or a pattern that matches the `[redacted]` marker), the whole line is redacted (fail closed).
|
|
63
|
+
- An omitted category or surface takes the vault default: `allow` for personal vaults, `deny` for org vaults.
|
|
64
|
+
- `allow` entries (at most 64) are exempt everywhere:
|
|
65
|
+
- an exact address, compared case-insensitively;
|
|
66
|
+
- an `@domain` suffix;
|
|
67
|
+
- a phone number standing alone, compared by its digits. Several numbers run together with spaces are never exempt (fail closed);
|
|
68
|
+
- an exact match of a pattern.
|
|
69
|
+
- Secret detection is separate from this policy and always applies.
|
|
70
|
+
|
|
71
|
+
```sh
|
|
72
|
+
vmem vaults pii personal # effective matrix and allow list
|
|
73
|
+
vmem vaults pii personal --set '{"email":{"log":"deny"}}' # preview: changed cells and every loosened cell
|
|
74
|
+
vmem vaults pii personal --set '{"email":{"log":"deny"}}' --confirm
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
The policy is changed only through the CLI, under the config lock. MCP clients can read it in `vmem_status` but cannot set it.
|
|
78
|
+
|
|
38
79
|
# Indexed reads
|
|
39
80
|
|
|
40
81
|
```sh
|
|
@@ -111,7 +152,7 @@ The vault set is read once at startup: every tool's `vault` property is an enum
|
|
|
111
152
|
|
|
112
153
|
| tool | arguments (caps) |
|
|
113
154
|
|---|---|
|
|
114
|
-
| `vmem_status` | none; vaults with health, write policy
|
|
155
|
+
| `vmem_status` | none; vaults with health, write policy, effective pii and `piiPolicy` (read-only) |
|
|
115
156
|
| `vmem_search` | `query`<=1000, `vault`, `limit`<=50, `type`, `domain` |
|
|
116
157
|
| `vmem_read` | `path` (stem, alias or `notes/` path), `vault`, `maxBytes`<=65536; returns `{path, frontmatter, body, sha256, truncated}` |
|
|
117
158
|
| `vmem_read_many` | `paths`<=20 (8 KiB each), `vault` |
|
|
@@ -192,7 +233,7 @@ vmem sync team --confirm --fingerprint <sha>
|
|
|
192
233
|
`vmem promote` copies one note from a personal vault into the local clone of a writable org vault (`kind: org`, `write` not `none`):
|
|
193
234
|
|
|
194
235
|
- **Transform.** The last prose `## Topics` section (never one inside a code fence) is replaced with the target topics, which must name MOCs that already exist in the org vault (source topics by default, `--topics` to override); a missing section is appended. Body wiki links that do not resolve in the org vault become plain text (their label, if any); wiki-valued frontmatter (`project`, `supersedes`, `superseded_by`, `source_ref`) pointing outside it is dropped. All of this is listed in `droppedLinks`. The org note gets `promoted_from: "<personal>:notes/x.md"` and `promoted_at`.
|
|
195
|
-
- **Screen (fail closed).** It refuses: secrets; PII
|
|
236
|
+
- **Screen (fail closed).** It refuses: secrets; PII denied by the org vault's `promotion` cells (org defaults to `deny`, plus configured `piiPatterns`; a `warn` cell is reported, not refused); `type: person` or `domain: people` records; private correspondence (`source`/`source_ref` from `inbox/`, `outlook:`, `telegram(`, `kakao`); and internal meeting notes (`source` under `meetings/`). These are keyword heuristics; the org clone's `git diff` before `vmem sync` is the review surface. Body links that are dropped are also resolved in the personal vault: a link to a person/people, meeting or correspondence note refuses the promotion (reporting only the category and count), so their names never reach the org vault. A refusal or a preview writes nothing in either vault.
|
|
196
237
|
- **Confirm.** It requires the current fingerprint. Then it creates the org note through `applyWrite`, records `promoted_to: "<org>:notes/x.md"` on the source, and appends `SYNC <project> personal:notes/x.md → <org>:notes/x.md <em-dash> promoted: <type>` to both vault logs (`--log-project` sets the project, default `-`). The note is also linted against the org vault (`lintVault` over the org snapshot with the note overlaid, scoped to that note): any L01/L02/L13 error refuses the preview, and the lint result is returned again after the write. Vault-wide warnings do not block. A second promotion of the same note needs `--again`, and an existing org note is never overwritten. If the source provenance write failed after the org note was created, a rerun detects the org note's matching `promoted_from` and offers a provenance-only preview and confirm.
|
|
197
238
|
- **MCP.** `vmem_promote` does the same over MCP and needs `write: mcp` on both vaults.
|
|
198
239
|
|
|
@@ -234,7 +275,7 @@ Runtime changes intentionally make `release:verify` report stale historical rele
|
|
|
234
275
|
|
|
235
276
|
- **Two steps.** Without `--confirm` it prints per-rule `counts`, a per-note change list (`key: old -> new`, dropped keys with their values, sections added) and `reported` items that were left unchanged; nothing is written. With `--confirm` the fingerprint must match: it covers the rules, the map file, and the before/after sha of every touched file (and of the log). Each note is then written through `applyWrite` (atomic per note, under the vault lock) and one `LINT` event is appended, for example `tidy: escapes 1, enums 13, keys 78, ...`.
|
|
236
277
|
- **Never blocked by one note.** A sanitized note that the writer would still refuse (for example an unknown key) is left unchanged, listed under `skipped` and `reported`, and the rest of the run proceeds. Counts and the `LINT` event cover only notes that are written.
|
|
237
|
-
- **No value leaks.** Every value echoed into a change line or report is screened: secret-looking values, and
|
|
278
|
+
- **No value leaks.** Every value echoed into a change line or report is screened: secret-looking values, and PII values that the `notes` surface warns on or denies, print as `[redacted]`.
|
|
238
279
|
- **Restart-safe.** If a run is interrupted, every note is either fully old or fully new; a rerun previews only the rest. A second `--all` run previews zero changes.
|
|
239
280
|
- **Scope.** Only flat `notes/*.md` and, for `--log`, the vault `log` are touched; `.obsidian/`, `templates/` and `self/` never are. Nothing is committed; commit the receipt paths yourself.
|
|
240
281
|
|
|
@@ -247,6 +288,6 @@ Runtime changes intentionally make `release:verify` report stale historical rele
|
|
|
247
288
|
| `--topics` | append `## Topics` listing the frontmatter topics where no such heading exists; mismatched sections are reported |
|
|
248
289
|
| `--connections` | fold `## Connections (...)` headings into `## Relevant Notes` |
|
|
249
290
|
| `--secrets` | replace V06 secret matches with `[redacted]`: in the body the match itself, in frontmatter the whole field value (the key stays, so the YAML stays valid). A match outside a field value, such as a YAML comment, is reported for manual repair. The preview names note and line, never the value. PII in personal vaults is left alone |
|
|
250
|
-
| `--log` | replace secret-looking substrings
|
|
291
|
+
| `--log` | replace secret-looking substrings in existing log lines with `[redacted]`, and PII values that the vault policy denies on the `log` surface (allow-list entries excepted), under the log lock (refused with V06 when `denyPaths` covers the log). A line that fails the lint L09 structure check loses a trailing seconds field when that makes it valid, and is otherwise kept verbatim behind `# legacy: ` (readers skip `#` lines); no time, zone or TYPE is guessed. Other lines are kept byte for byte. Git history is not rewritten: rotate exposed credentials |
|
|
251
292
|
|
|
252
293
|
`--map <file>` overrides defaults with strict JSON `{"enums": {"confidence|status|type": {"from": "to"}}, "keys": {"key": "drop" | "keep" | {"rename": "key"}}}` (at most 64 KiB and 1000 entries). For `tidy`, `--log` selects the log rule; tidy always writes its own `LINT` event.
|
package/package.json
CHANGED
package/src/cli.ts
CHANGED
|
@@ -7,7 +7,8 @@ import { version } from "../package.json";
|
|
|
7
7
|
import { completionScript, help } from "./commands-table.ts";
|
|
8
8
|
import { runSetup, type SetupMode } from "./setup.ts";
|
|
9
9
|
import { resolveHome } from "./home.ts";
|
|
10
|
-
import { addVault, effectivePii, loadRegistry, recoverRegistry, registryStatus, removeVault } from "./registry.ts";
|
|
10
|
+
import { addVault, effectivePii, loadRegistry, recoverRegistry, registryStatus, removeVault, setVaultPii } from "./registry.ts";
|
|
11
|
+
import { piiPolicy } from "./pii.ts";
|
|
11
12
|
function parse(args: string[], allowed: string[], values: string[]) {
|
|
12
13
|
const flags: Record<string, string | boolean> = {}, positional: string[] = [];
|
|
13
14
|
for (let i = 0; i < args.length; i++) {
|
|
@@ -43,8 +44,8 @@ export function run(args: string[], env: NodeJS.ProcessEnv = process.env): { tex
|
|
|
43
44
|
if (isReadCommand(args[0]!)) return runReadCommand(args[0]!, args.slice(1), resolveHome(env).home);
|
|
44
45
|
const command = args[0] === "vaults" ? `vaults ${args[1] ?? ""}` : args[0];
|
|
45
46
|
const tail = args.slice(args[0] === "vaults" ? 2 : 1);
|
|
46
|
-
const options = command === "vaults add" ? ["--kind", "--write", "--pii", "--remote", "--registry"] : [];
|
|
47
|
-
const { flags, positional } = parse(tail, ["--json", ...options, ...(command === "vaults remove" || command === "recover" ? ["--confirm"] : [])], options);
|
|
47
|
+
const options = command === "vaults add" ? ["--kind", "--write", "--pii", "--remote", "--registry"] : command === "vaults pii" ? ["--set"] : [];
|
|
48
|
+
const { flags, positional } = parse(tail, ["--json", ...options, ...(command === "vaults remove" || command === "vaults pii" || command === "recover" ? ["--confirm"] : [])], options);
|
|
48
49
|
const home = resolveHome(env);
|
|
49
50
|
let result: unknown, code = 0;
|
|
50
51
|
switch (command) {
|
|
@@ -62,7 +63,7 @@ export function run(args: string[], env: NodeJS.ProcessEnv = process.env): { tex
|
|
|
62
63
|
break;
|
|
63
64
|
case "vaults list":
|
|
64
65
|
if (positional.length) throw new Error("vaults list: unexpected arguments");
|
|
65
|
-
result = loadRegistry(home.home).vaults.map(v => ({ ...v, pii: effectivePii(v) }));
|
|
66
|
+
result = loadRegistry(home.home).vaults.map(v => ({ ...v, pii: effectivePii(v), piiPolicy: piiPolicy(v) }));
|
|
66
67
|
break;
|
|
67
68
|
case "vaults add": {
|
|
68
69
|
if (positional.length !== 2) throw new Error("vaults add: requires id and absolute path");
|
|
@@ -71,6 +72,15 @@ export function run(args: string[], env: NodeJS.ProcessEnv = process.env): { tex
|
|
|
71
72
|
result = { applied: true, vault: addVault(home.home, value) };
|
|
72
73
|
break;
|
|
73
74
|
}
|
|
75
|
+
case "vaults pii": {
|
|
76
|
+
if (positional.length !== 1) throw new Error("vaults pii: requires id");
|
|
77
|
+
const raw = flags["--set"];
|
|
78
|
+
if (raw === undefined && flags["--confirm"] === true) throw new Error("vaults pii: --confirm requires --set");
|
|
79
|
+
let value: unknown;
|
|
80
|
+
if (typeof raw === "string") { if (raw === "allow" || raw === "deny") value = raw; else { try { value = JSON.parse(raw); } catch { throw new Error("vaults pii: --set expects allow, deny or a JSON policy object"); } } }
|
|
81
|
+
result = setVaultPii(home.home, positional[0]!, value, flags["--confirm"] === true);
|
|
82
|
+
break;
|
|
83
|
+
}
|
|
74
84
|
case "vaults remove":
|
|
75
85
|
if (positional.length !== 1) throw new Error("vaults remove: requires id");
|
|
76
86
|
result = removeVault(home.home, positional[0]!, flags["--confirm"] === true);
|
package/src/commands-table.ts
CHANGED
|
@@ -12,6 +12,7 @@ export const COMMANDS = [
|
|
|
12
12
|
{ name: "vaults list", usage: "vaults list [--json]", mutates: false },
|
|
13
13
|
{ name: "vaults add", usage: "vaults add <id> <absolute-path> --kind personal|org --write mcp|cli|none [--pii allow|deny] [--remote URL] [--registry PATH] [--json]", mutates: true },
|
|
14
14
|
{ name: "vaults remove", usage: "vaults remove <id> [--confirm] [--json]", mutates: "confirm" },
|
|
15
|
+
{ name: "vaults pii", usage: "vaults pii <id> [--set allow|deny|JSON] [--confirm] [--json]", mutates: "confirm" },
|
|
15
16
|
{ name: "new", usage: "new <vault> <stem> --type TYPE --domain DOMAIN --topics a,b --description TEXT [--title TEXT] [--source TEXT|JSON] [--confidence TEXT] [--status TEXT] [--body file|-] [--extra JSON] [--dry-run] [--json]", mutates: true },
|
|
16
17
|
{ name: "patch", usage: "patch <vault> <note> --old TEXT --new TEXT [--expected-sha SHA] [--dry-run] [--json]", mutates: true },
|
|
17
18
|
{ name: "append", usage: "append <vault> <note> --section SECTION --markdown TEXT [--expected-sha SHA] [--dry-run] [--json]", mutates: true },
|
package/src/config.ts
CHANGED
|
@@ -8,10 +8,17 @@ export interface VaultSpec {
|
|
|
8
8
|
path: string;
|
|
9
9
|
kind: "personal" | "org";
|
|
10
10
|
write: "mcp" | "cli" | "none";
|
|
11
|
-
pii?: "allow" | "deny";
|
|
11
|
+
pii?: "allow" | "deny" | PiiSpec;
|
|
12
12
|
remote?: string;
|
|
13
13
|
registry?: string;
|
|
14
14
|
}
|
|
15
|
+
export type PiiAction = "allow" | "warn" | "deny";
|
|
16
|
+
export const PII_SURFACES = ["notes", "log", "promotion"] as const;
|
|
17
|
+
export type PiiSurface = typeof PII_SURFACES[number];
|
|
18
|
+
export type PiiCell = PiiAction | Partial<Record<PiiSurface, PiiAction>>;
|
|
19
|
+
export interface PiiPattern { name: string; regex: string; action?: PiiCell }
|
|
20
|
+
/** Fine-grained PII policy: per category, optionally per surface; omitted cells take the vault default. */
|
|
21
|
+
export interface PiiSpec { email?: PiiCell; phone?: PiiCell; custom?: PiiCell; patterns?: PiiPattern[]; allow?: string[] }
|
|
15
22
|
export interface VmemConfig { home?: string; vaults: VaultSpec[]; denyPaths?: string[]; piiPatterns?: string[] }
|
|
16
23
|
const controls = /[\x00-\x1f\x7f]/;
|
|
17
24
|
function object(value: unknown, keys: string[], label: string): Record<string, unknown> {
|
|
@@ -38,6 +45,41 @@ function strings(value: unknown, label: string): string[] | undefined {
|
|
|
38
45
|
return text;
|
|
39
46
|
});
|
|
40
47
|
}
|
|
48
|
+
const ACTIONS = new Set(["allow", "warn", "deny"]);
|
|
49
|
+
function cell(value: unknown, label: string): PiiCell {
|
|
50
|
+
if (typeof value === "string") {
|
|
51
|
+
if (!ACTIONS.has(value)) throw new Error(`${label}: expected allow, warn or deny`);
|
|
52
|
+
return value as PiiAction;
|
|
53
|
+
}
|
|
54
|
+
const record = object(value, [...PII_SURFACES], label), result: Partial<Record<PiiSurface, PiiAction>> = {};
|
|
55
|
+
for (const surface of PII_SURFACES) if (record[surface] !== undefined) {
|
|
56
|
+
if (typeof record[surface] !== "string" || !ACTIONS.has(record[surface] as string)) throw new Error(`${label}.${surface}: expected allow, warn or deny`);
|
|
57
|
+
result[surface] = record[surface] as PiiAction;
|
|
58
|
+
}
|
|
59
|
+
return result;
|
|
60
|
+
}
|
|
61
|
+
/** Validate `vault.pii`: the `allow`/`deny` shorthand or a PiiSpec object. */
|
|
62
|
+
export function parsePii(value: unknown): "allow" | "deny" | PiiSpec {
|
|
63
|
+
if (value === "allow" || value === "deny") return value;
|
|
64
|
+
if (typeof value === "string") throw new Error("vault.pii: expected allow, deny or a policy object");
|
|
65
|
+
const raw = object(value, ["email", "phone", "custom", "patterns", "allow"], "vault.pii"), spec: PiiSpec = {};
|
|
66
|
+
for (const key of ["email", "phone", "custom"] as const) if (raw[key] !== undefined) spec[key] = cell(raw[key], `vault.pii.${key}`);
|
|
67
|
+
if (raw.patterns !== undefined) {
|
|
68
|
+
if (!Array.isArray(raw.patterns) || raw.patterns.length > 32) throw new Error("vault.pii.patterns: expected at most 32 patterns");
|
|
69
|
+
const names = new Set<string>();
|
|
70
|
+
spec.patterns = raw.patterns.map((item, index) => {
|
|
71
|
+
const p = object(item, ["name", "regex", "action"], `vault.pii.patterns[${index}]`);
|
|
72
|
+
const name = string(p.name, "vault.pii.patterns.name"), regex = string(p.regex, "vault.pii.patterns.regex");
|
|
73
|
+
if (!/^[a-z][a-z0-9-]{0,31}$/.test(name) || names.has(name)) throw new Error(`vault.pii.patterns: invalid or duplicate name ${name.slice(0, 40)}`);
|
|
74
|
+
if (Buffer.byteLength(regex) > 256) throw new Error("vault.pii.patterns.regex: exceeds 256 bytes");
|
|
75
|
+
try { new RegExp(regex, "u"); } catch { throw new Error(`vault.pii.patterns.${name}: invalid regular expression`); }
|
|
76
|
+
names.add(name);
|
|
77
|
+
return { name, regex, ...(p.action !== undefined ? { action: cell(p.action, `vault.pii.patterns.${name}.action`) } : {}) };
|
|
78
|
+
});
|
|
79
|
+
}
|
|
80
|
+
if (raw.allow !== undefined) spec.allow = strings(raw.allow, "vault.pii.allow");
|
|
81
|
+
return spec;
|
|
82
|
+
}
|
|
41
83
|
export function parseVault(value: unknown): VaultSpec {
|
|
42
84
|
const v = object(value, ["id", "path", "kind", "write", "pii", "remote", "registry"], "vault");
|
|
43
85
|
const id = string(v.id, "vault.id");
|
|
@@ -45,10 +87,7 @@ export function parseVault(value: unknown): VaultSpec {
|
|
|
45
87
|
if (v.kind !== "personal" && v.kind !== "org") throw new Error("vault.kind: expected personal or org");
|
|
46
88
|
if (v.write !== "mcp" && v.write !== "cli" && v.write !== "none") throw new Error("vault.write: expected mcp, cli or none");
|
|
47
89
|
const result: VaultSpec = { id, path: absolute(v.path, "vault.path"), kind: v.kind, write: v.write };
|
|
48
|
-
if (v.pii !== undefined)
|
|
49
|
-
if (v.pii !== "allow" && v.pii !== "deny") throw new Error("vault.pii: expected allow or deny");
|
|
50
|
-
result.pii = v.pii;
|
|
51
|
-
}
|
|
90
|
+
if (v.pii !== undefined) result.pii = parsePii(v.pii);
|
|
52
91
|
if (v.remote !== undefined) result.remote = string(v.remote, "vault.remote");
|
|
53
92
|
if (v.registry !== undefined) result.registry = absolute(v.registry, "vault.registry");
|
|
54
93
|
return result;
|
package/src/log-format.ts
CHANGED
|
@@ -1,4 +1,6 @@
|
|
|
1
1
|
import { screenContent } from "./write-validation.ts";
|
|
2
|
+
import { parsePii } from "./config.ts";
|
|
3
|
+
function validPii(value: unknown): boolean { try { parsePii(value); return true; } catch { return false; } }
|
|
2
4
|
import type { LogContext, LogEvent } from "./log-types.ts";
|
|
3
5
|
import type { WriteIssue } from "./write-types.ts";
|
|
4
6
|
|
|
@@ -31,7 +33,7 @@ export function formatLogEvent(event: LogEvent, context: LogContext): { line?: s
|
|
|
31
33
|
}
|
|
32
34
|
if ((context.channel !== undefined && context.channel !== "cli" && context.channel !== "mcp") ||
|
|
33
35
|
!record(context.vault) || !["personal", "org"].includes(context.vault.kind) ||
|
|
34
|
-
(context.vault.pii !== undefined && context.vault.pii
|
|
36
|
+
(context.vault.pii !== undefined && !validPii(context.vault.pii)) ||
|
|
35
37
|
!record(context.config) || !Array.isArray(context.config.vaults) ||
|
|
36
38
|
(context.config.piiPatterns !== undefined && (!Array.isArray(context.config.piiPatterns) || context.config.piiPatterns.some(value => typeof value !== "string"))) ||
|
|
37
39
|
(context.now !== undefined && !(context.now instanceof Date)) ||
|
|
@@ -51,7 +53,7 @@ export function formatLogEvent(event: LogEvent, context: LogContext): { line?: s
|
|
|
51
53
|
const env = context.env ?? process.env;
|
|
52
54
|
const agent = context.channel === "mcp" || AGENT_MARKERS.some(marker => typeof env[marker] === "string" && env[marker]!.length > 0);
|
|
53
55
|
const line = `${timestamp} ${event.type} ${event.project} ${event.source}${event.dest === undefined ? "" : ` → ${event.dest}`} — ${event.note}${agent ? " (vmem:agent)" : ""}\n`;
|
|
54
|
-
const issues = screenContent(line, context.vault, context.config);
|
|
56
|
+
const issues = screenContent(line, context.vault, context.config, "log");
|
|
55
57
|
if (Buffer.byteLength(line, "utf8") > 1000) issues.push(error("V09", "log line exceeds 1000 UTF-8 bytes including attribution and newline"));
|
|
56
58
|
return issues.some(issue => issue.severity === "error") ? { issues } : { line, issues };
|
|
57
59
|
}
|
package/src/mcp-vault.ts
CHANGED
|
@@ -35,7 +35,7 @@ export function vaultTools(ids: readonly string[], orgIds: readonly string[] = [
|
|
|
35
35
|
const twoStep = { confirm: { type: "boolean" }, fingerprint: sha };
|
|
36
36
|
const orgVault = orgIds.length ? { type: "string", enum: [...orgIds] } : string(32);
|
|
37
37
|
return [
|
|
38
|
-
tool("vmem_status", "Registered vaults with health, write policy and
|
|
38
|
+
tool("vmem_status", "Registered vaults with health, write policy, effective pii and the piiPolicy matrix (read-only).", object({})),
|
|
39
39
|
tool("vmem_search", "Full-text (BM25) search over indexed notes. Literal AND terms.", object({ query: string(1000), vault, limit: integer(1, 50), type: string(64), domain: string(64) }, ["query"])),
|
|
40
40
|
tool("vmem_read", "Read one note by stem, alias or notes/ path: frontmatter, body, sha256. Bounded by maxBytes; section selects an exact H2, offset pages UTF-8 byte ranges, and truncated reads list H2 headings/nextOffset.", object({ path: note, vault, maxBytes: integer(1, 65536), section: string(1024), offset: integer(0, MAX_NOTE_BYTES) }, ["path"])),
|
|
41
41
|
tool("vmem_read_many", "Read up to 20 notes (8 KiB each).", object({ paths: { type: "array", minItems: 1, maxItems: 20, items: note }, vault }, ["paths"])),
|
package/src/org-promote.ts
CHANGED
|
@@ -61,7 +61,8 @@ const MEETING = [/(^|\/)meetings\//i];
|
|
|
61
61
|
* meeting sources. Returns error issues only; any issue refuses promotion.
|
|
62
62
|
*/
|
|
63
63
|
export function screenForOrg(text: string, data: Record<string, unknown>, target: VaultSpec, config: VmemConfig): WriteIssue[] {
|
|
64
|
-
|
|
64
|
+
// Secrets and deny cells refuse; a warn cell in the target policy is reported without refusing.
|
|
65
|
+
const issues: WriteIssue[] = screenContent(text, target, config, "promotion");
|
|
65
66
|
if (data.type === "person" || data.domain === "people") issues.push(issue("SCREEN", "personal_data: person or people records are never promoted"));
|
|
66
67
|
const sources = [...strings(data.source), ...strings(data.source_ref)];
|
|
67
68
|
if (sources.some(value => MEETING.some(pattern => pattern.test(value)))) issues.push(issue("SCREEN", "internal_meeting_notes: source under meetings/"));
|
|
@@ -230,9 +231,10 @@ export function promoteToOrg(home: string, request: OrgPromoteRequest, options:
|
|
|
230
231
|
const text = emitFrontmatter(parseFrontmatter(parsed.source.slice(0, parsed.source.length - parsed.body.length) + body), frontmatter);
|
|
231
232
|
result.target = `${target.id}:${targetPath}`; result.text = text; result.droppedLinks = transformed.dropped;
|
|
232
233
|
const screen = [...screenForOrg(text, parsed.data, target, config), ...screenDroppedLinks(home, source, all, transformed.dropped.filter(link => !link.includes(":")))];
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
|
|
234
|
+
const refusing = screen.filter(found => found.severity === "error");
|
|
235
|
+
result.screen = { passed: refusing.length === 0, issues: screen };
|
|
236
|
+
if (refusing.length) return { ...result, issues: screen };
|
|
237
|
+
const planned = applyWrite(home, { vault: target.id, operation: { op: "create", path: targetPath, frontmatter: {}, body: text }, channel: request.channel ?? "cli", dryRun: true }, { env: options.env, piiSurface: "promotion" });
|
|
236
238
|
const provenanceSet = { promoted_to: prior.length ? [...prior.filter(p => p !== promotedTo), promotedTo] : promotedTo };
|
|
237
239
|
const provenanceText = emitFrontmatter(parsed, provenanceSet);
|
|
238
240
|
const provenancePlan = applyWrite(home, { vault: source.id, operation: { op: "frontmatter", path, set: provenanceSet }, channel: request.channel ?? "cli", dryRun: true }, { env: options.env });
|
|
@@ -245,7 +247,7 @@ export function promoteToOrg(home: string, request: OrgPromoteRequest, options:
|
|
|
245
247
|
if (request.confirm !== true) return { ...result, ok: true, preview: true, issues: [...planned.issues, ...provenancePlan.issues] };
|
|
246
248
|
if (request.fingerprint !== result.fingerprint) return { ...result, issues: [issue("V09", "stale or missing preview fingerprint")] };
|
|
247
249
|
const event = { type: "SYNC", project: request.logProject ?? "-", source: `${source.id}:${path}`, dest: `${target.id}:${targetPath}`, note: `promoted: ${typeof parsed.data.type === "string" ? parsed.data.type : "note"}` };
|
|
248
|
-
const note = applyWrite(home, { vault: target.id, operation: { op: "create", path: targetPath, frontmatter: {}, body: text }, channel: request.channel ?? "cli", log: false }, { env: options.env });
|
|
250
|
+
const note = applyWrite(home, { vault: target.id, operation: { op: "create", path: targetPath, frontmatter: {}, body: text }, channel: request.channel ?? "cli", log: false }, { env: options.env, piiSurface: "promotion" });
|
|
249
251
|
if (!note.ok || !note.applied) return { ...result, note, issues: note.issues };
|
|
250
252
|
const provenance = applyWrite(home, { vault: source.id, operation: { op: "frontmatter", path, set: provenanceSet }, channel: request.channel ?? "cli", log: false }, { env: options.env });
|
|
251
253
|
const logs = { source: null as LogReceipt | null, target: null as LogReceipt | null };
|
package/src/pii.ts
ADDED
|
@@ -0,0 +1,133 @@
|
|
|
1
|
+
import { PII_SURFACES, type PiiAction, type PiiCell, type PiiSurface, type VaultSpec, type VmemConfig } from "./config.ts";
|
|
2
|
+
import type { WriteIssue } from "./write-types.ts";
|
|
3
|
+
|
|
4
|
+
export const PII_CATEGORIES = ["email", "phone", "custom"] as const;
|
|
5
|
+
export type PiiCategory = typeof PII_CATEGORIES[number];
|
|
6
|
+
export type PiiRow = Record<PiiSurface, PiiAction>;
|
|
7
|
+
/** Effective policy: every category and surface resolved, plus named patterns and the allow list. */
|
|
8
|
+
export interface PiiMatrix { email: PiiRow; phone: PiiRow; custom: PiiRow; patterns: { name: string; regex: string; action: PiiRow }[]; allow: string[] }
|
|
9
|
+
|
|
10
|
+
export const EMAIL_PATTERN = /[A-Z0-9.!#$%&'*+/=?^_`{|}~-]+@[A-Z0-9](?:[A-Z0-9.-]*[A-Z0-9])?\.[A-Z]{2,}/i;
|
|
11
|
+
const RANK: Record<PiiAction, number> = { allow: 0, warn: 1, deny: 2 };
|
|
12
|
+
|
|
13
|
+
function row(value: PiiCell | undefined, fallback: PiiAction): PiiRow {
|
|
14
|
+
const out = {} as PiiRow;
|
|
15
|
+
for (const surface of PII_SURFACES) out[surface] = typeof value === "string" ? value : value?.[surface] ?? fallback;
|
|
16
|
+
return out;
|
|
17
|
+
}
|
|
18
|
+
/** Resolve a vault's policy. Omitted cells default to allow for personal and deny for org vaults. */
|
|
19
|
+
export function piiPolicy(vault: VaultSpec): PiiMatrix {
|
|
20
|
+
const fallback: PiiAction = vault.kind === "org" ? "deny" : "allow";
|
|
21
|
+
const pii = vault.pii;
|
|
22
|
+
if (pii === undefined || typeof pii === "string") {
|
|
23
|
+
const action = pii ?? fallback;
|
|
24
|
+
return { email: row(action, action), phone: row(action, action), custom: row(action, action), patterns: [], allow: [] };
|
|
25
|
+
}
|
|
26
|
+
return {
|
|
27
|
+
email: row(pii.email, fallback), phone: row(pii.phone, fallback), custom: row(pii.custom, fallback),
|
|
28
|
+
patterns: (pii.patterns ?? []).map(p => ({ name: p.name, regex: p.regex, action: row(p.action, fallback) })),
|
|
29
|
+
allow: [...(pii.allow ?? [])],
|
|
30
|
+
};
|
|
31
|
+
}
|
|
32
|
+
/** `allow`/`warn`/`deny` when every cell agrees and nothing else is set, else `custom`. */
|
|
33
|
+
export function piiShorthand(vault: VaultSpec): PiiAction | "custom" {
|
|
34
|
+
const m = piiPolicy(vault), cells = PII_CATEGORIES.flatMap(c => PII_SURFACES.map(s => m[c][s]));
|
|
35
|
+
return m.patterns.length || m.allow.length || cells.some(a => a !== cells[0]) ? "custom" : cells[0]!;
|
|
36
|
+
}
|
|
37
|
+
/** Cells (and allow-list or pattern changes) where `next` is more permissive than `prev`. */
|
|
38
|
+
export function loosenedCells(prev: PiiMatrix, next: PiiMatrix): string[] {
|
|
39
|
+
const out: string[] = [];
|
|
40
|
+
for (const c of PII_CATEGORIES) for (const s of PII_SURFACES) if (RANK[next[c][s]] < RANK[prev[c][s]]) out.push(`${c}.${s}: ${prev[c][s]} -> ${next[c][s]}`);
|
|
41
|
+
for (const p of prev.patterns) {
|
|
42
|
+
const n = next.patterns.find(q => q.name === p.name);
|
|
43
|
+
if (!n) { out.push(`patterns.${p.name}: removed`); continue; }
|
|
44
|
+
if (n.regex !== p.regex) out.push(`patterns.${p.name}: regex changed`);
|
|
45
|
+
for (const s of PII_SURFACES) if (RANK[n.action[s]] < RANK[p.action[s]]) out.push(`patterns.${p.name}.${s}: ${p.action[s]} -> ${n.action[s]}`);
|
|
46
|
+
}
|
|
47
|
+
for (const value of next.allow) if (!prev.allow.includes(value)) out.push(`allow: + ${value}`);
|
|
48
|
+
return out;
|
|
49
|
+
}
|
|
50
|
+
|
|
51
|
+
const digitsOf = (value: string) => value.replace(/\D/g, "");
|
|
52
|
+
const TIMESTAMP = /\b[0-9]{4}-(?:0[1-9]|1[0-2])-(?:0[1-9]|[12][0-9]|3[01])[ T](?:[01][0-9]|2[0-3]):[0-5][0-9](?::[0-5][0-9](?:\.[0-9]{1,3})?)?(?:Z|[+-](?:[01][0-9]|2[0-3]):[0-5][0-9])?\b/g;
|
|
53
|
+
/**
|
|
54
|
+
* Phone runs: exact source ranges that hold a 9-15 digit number. `single` is set when the run is exactly one number
|
|
55
|
+
* (the only form the allow list can exempt). An oversized run (numbers separated by spaces) is screened by each
|
|
56
|
+
* contiguous run of digit-bearing groups and stops at the first valid one; it is never exempt (fails closed).
|
|
57
|
+
*/
|
|
58
|
+
export function phoneRuns(text: string): { start: number; end: number; single?: string }[] {
|
|
59
|
+
// A complete ISO calendar timestamp is metadata, not a phone-number candidate; the newline keeps neighbours apart.
|
|
60
|
+
const cleaned = text.replace(TIMESTAMP, match => "\n".repeat(match.length)), out: { start: number; end: number; single?: string }[] = [];
|
|
61
|
+
for (const match of cleaned.matchAll(/(?<![\p{L}\p{N}])\+?\d(?:[\d ().-]{6,}\d)(?![\p{L}\p{N}])/gu)) {
|
|
62
|
+
const digits = digitsOf(match[0]), range = { start: match.index!, end: match.index! + match[0].length };
|
|
63
|
+
if (digits.length >= 9 && digits.length <= 15) { out.push({ ...range, single: digits }); continue; }
|
|
64
|
+
if (digits.length <= 15) continue;
|
|
65
|
+
const groups = match[0].split(/\s+/).map(digitsOf).filter(Boolean);
|
|
66
|
+
search: for (let from = 0; from < groups.length; from++) {
|
|
67
|
+
let total = 0;
|
|
68
|
+
for (let to = from; to < groups.length && total <= 15; to++) { total += groups[to]!.length; if (total >= 9 && total <= 15) { out.push(range); break search; } }
|
|
69
|
+
}
|
|
70
|
+
}
|
|
71
|
+
return out;
|
|
72
|
+
}
|
|
73
|
+
function allowedEmail(value: string, allow: string[]): boolean {
|
|
74
|
+
const lower = value.toLowerCase();
|
|
75
|
+
return allow.some(entry => entry.includes("@") && (entry.startsWith("@") ? lower.endsWith(entry.toLowerCase()) : lower === entry.toLowerCase()));
|
|
76
|
+
}
|
|
77
|
+
function allowedPhone(digits: string, allow: string[]): boolean {
|
|
78
|
+
return allow.some(entry => !entry.includes("@") && digitsOf(entry).length >= 9 && digitsOf(entry) === digits);
|
|
79
|
+
}
|
|
80
|
+
interface Rule { label: string; action: PiiAction; invalid?: boolean; matches: (text: string) => { start: number; end: number }[]; detect: (text: string) => boolean }
|
|
81
|
+
/** The non-allow rules for one surface. `detect` short-circuits; `matches` lists raw non-exempt ranges (zero-length included). */
|
|
82
|
+
function rules(vault: VaultSpec, config: VmemConfig, surface: PiiSurface): Rule[] {
|
|
83
|
+
const m = piiPolicy(vault), out: Rule[] = [];
|
|
84
|
+
const regexRule = (label: string, action: PiiAction, make: () => RegExp, exempt: (value: string) => boolean): Rule => ({
|
|
85
|
+
label, action,
|
|
86
|
+
matches: text => [...text.matchAll(make())].filter(x => !x[0] || !exempt(x[0])).map(x => ({ start: x.index!, end: x.index! + x[0].length })),
|
|
87
|
+
detect: text => { for (const source of [text, text.normalize("NFKC")]) for (const x of source.matchAll(make())) if (!x[0] || !exempt(x[0])) return true; return false; },
|
|
88
|
+
});
|
|
89
|
+
if (m.email[surface] !== "allow") out.push(regexRule("email", m.email[surface], () => new RegExp(EMAIL_PATTERN.source, "gi"), v => allowedEmail(v, m.allow)));
|
|
90
|
+
if (m.phone[surface] !== "allow") {
|
|
91
|
+
const flagged = (text: string) => phoneRuns(text).filter(r => !(r.single && allowedPhone(r.single, m.allow)));
|
|
92
|
+
out.push({ label: "phone", action: m.phone[surface], matches: flagged, detect: text => flagged(text).length > 0 || flagged(text.normalize("NFKC")).length > 0 });
|
|
93
|
+
}
|
|
94
|
+
const custom = (config.piiPatterns ?? []).map((regex, i) => ({ name: `custom-${i + 1}`, regex, action: m.custom[surface], flags: "" }));
|
|
95
|
+
for (const p of [...custom, ...m.patterns.map(p => ({ name: p.name, regex: p.regex, action: p.action[surface], flags: "u" }))]) {
|
|
96
|
+
if (p.action === "allow") continue;
|
|
97
|
+
try { new RegExp(p.regex, p.flags); } catch { out.push({ label: p.name, action: "deny", invalid: true, matches: () => [], detect: () => true }); continue; }
|
|
98
|
+
out.push(regexRule(p.name, p.action, () => new RegExp(p.regex, `${p.flags}g`), v => m.allow.some(a => a.toLowerCase() === v.toLowerCase())));
|
|
99
|
+
}
|
|
100
|
+
return out;
|
|
101
|
+
}
|
|
102
|
+
/** V07 issues for one surface: deny cells are errors, warn cells warnings. Matched values are never returned. */
|
|
103
|
+
export function screenPii(text: string, vault: VaultSpec, config: VmemConfig, surface: PiiSurface): WriteIssue[] {
|
|
104
|
+
const issues: WriteIssue[] = [];
|
|
105
|
+
for (const rule of rules(vault, config, surface)) {
|
|
106
|
+
if (rule.invalid) { if (!issues.some(i => i.message === "invalid PII screening configuration")) issues.push({ code: "V07", severity: "error", message: "invalid PII screening configuration" }); continue; }
|
|
107
|
+
if (!rule.detect(text)) continue;
|
|
108
|
+
const builtin = rule.label === "email" || rule.label === "phone", custom = rule.label.startsWith("custom-");
|
|
109
|
+
const message = rule.action === "deny"
|
|
110
|
+
? builtin ? "PII is refused by the vault policy" : custom ? "configured PII pattern matched" : `PII pattern ${rule.label} matched`
|
|
111
|
+
: `PII flagged by the vault policy (${builtin || custom ? rule.label.replace(/-\d+$/, "") : `pattern ${rule.label}`}, warn)`;
|
|
112
|
+
if (!issues.some(i => i.message === message)) issues.push({ code: "V07", severity: rule.action === "deny" ? "error" : "warning", message });
|
|
113
|
+
}
|
|
114
|
+
return issues;
|
|
115
|
+
}
|
|
116
|
+
/**
|
|
117
|
+
* Replace raw ranges that a deny cell refuses on this surface with `[redacted]` (tidy --log), leaving allow-listed
|
|
118
|
+
* values intact. Fails closed to a fully redacted text when a denied match is zero-length, or when the returned text
|
|
119
|
+
* itself still contains a denied match (for example a form only NFKC normalization reveals, or a pattern that matches
|
|
120
|
+
* the marker).
|
|
121
|
+
*/
|
|
122
|
+
export function redactPii(text: string, vault: VaultSpec, config: VmemConfig, surface: PiiSurface): { text: string; hits: number } {
|
|
123
|
+
const denied = rules(vault, config, surface).filter(rule => rule.action === "deny" && !rule.invalid);
|
|
124
|
+
if (!denied.some(rule => rule.detect(text))) return { text, hits: 0 };
|
|
125
|
+
const ranges = denied.flatMap(rule => rule.matches(text));
|
|
126
|
+
if (ranges.some(r => r.end === r.start)) return { text: "[redacted]", hits: 1 };
|
|
127
|
+
const merged: { start: number; end: number }[] = [];
|
|
128
|
+
for (const r of ranges.sort((a, b) => a.start - b.start || b.end - a.end)) { const last = merged.at(-1); if (last && r.start < last.end) last.end = Math.max(last.end, r.end); else merged.push({ ...r }); }
|
|
129
|
+
let out = "", at = 0;
|
|
130
|
+
for (const r of merged) { out += text.slice(at, r.start) + "[redacted]"; at = r.end; }
|
|
131
|
+
out += text.slice(at);
|
|
132
|
+
return denied.some(rule => rule.detect(out)) ? { text: "[redacted]", hits: 1 } : { text: out, hits: merged.length };
|
|
133
|
+
}
|
package/src/registry.ts
CHANGED
|
@@ -2,7 +2,8 @@ import { closeSync, constants, fsyncSync, mkdirSync, openSync, readdirSync, read
|
|
|
2
2
|
import { hostname } from "node:os";
|
|
3
3
|
import { join } from "node:path";
|
|
4
4
|
import { randomUUID } from "node:crypto";
|
|
5
|
-
import { CONFIG_FILE, MAX_CONFIG_BYTES, parseConfig, parseVault, readConfig, type VaultSpec, type VmemConfig } from "./config.ts";
|
|
5
|
+
import { CONFIG_FILE, MAX_CONFIG_BYTES, parseConfig, parsePii, parseVault, readConfig, type VaultSpec, type VmemConfig } from "./config.ts";
|
|
6
|
+
import { loosenedCells, piiPolicy, piiShorthand } from "./pii.ts";
|
|
6
7
|
export const LOCK_FILE = "config.lock";
|
|
7
8
|
const TEMP = /^\.config-[0-9a-f-]{36}\.tmp$/;
|
|
8
9
|
export interface LockOwner { pid: number; host: string; created: string }
|
|
@@ -11,8 +12,8 @@ export function loadRegistry(home: string): VmemConfig {
|
|
|
11
12
|
if (config.home) throw new Error(`${join(home, CONFIG_FILE)}: redirected home cannot contain another home pointer`);
|
|
12
13
|
return config;
|
|
13
14
|
}
|
|
14
|
-
/** Effective pii
|
|
15
|
-
export function effectivePii(vault: VaultSpec): "allow" | "
|
|
15
|
+
/** Effective pii shorthand (`custom` for a mixed policy); the full matrix is `piiPolicy`. */
|
|
16
|
+
export function effectivePii(vault: VaultSpec): "allow" | "warn" | "deny" | "custom" { return piiShorthand(vault); }
|
|
16
17
|
function message(error: unknown): string { return error instanceof Error ? error.message : String(error); }
|
|
17
18
|
/**
|
|
18
19
|
* Directory fsync makes the rename durable on Linux. On macOS fsync(2) on a directory
|
|
@@ -157,10 +158,25 @@ export function removeVault(home: string, id: string, confirm = false): { applie
|
|
|
157
158
|
};
|
|
158
159
|
return confirm ? mutate(home, remove) : remove(loadRegistry(home));
|
|
159
160
|
}
|
|
161
|
+
/** Show or change a vault's PII policy. Without `confirm` it previews; loosened cells are listed either way. */
|
|
162
|
+
export function setVaultPii(home: string, id: string, value: unknown, confirm = false) {
|
|
163
|
+
const next = value === undefined ? undefined : parsePii(value);
|
|
164
|
+
const change = (config: VmemConfig) => {
|
|
165
|
+
const vault = config.vaults.find(v => v.id === id);
|
|
166
|
+
if (!vault) throw new Error(`${join(home, CONFIG_FILE)}: unknown vault id ${id}`);
|
|
167
|
+
const before = piiPolicy(vault);
|
|
168
|
+
if (next === undefined) return { vault: id, pii: piiShorthand(vault), piiPolicy: before };
|
|
169
|
+
const updated: VaultSpec = { ...vault, pii: next }, after = piiPolicy(updated);
|
|
170
|
+
const loosened = loosenedCells(before, after), changed = JSON.stringify(before) !== JSON.stringify(after);
|
|
171
|
+
if (confirm && changed) vault.pii = next;
|
|
172
|
+
return { vault: id, applied: confirm && changed, changed, loosened, pii: piiShorthand(updated), piiPolicy: after, ...(confirm ? {} : { preview: true }) };
|
|
173
|
+
};
|
|
174
|
+
return confirm && next !== undefined ? mutate(home, change) : change(loadRegistry(home));
|
|
175
|
+
}
|
|
160
176
|
export function registryStatus(home: string) {
|
|
161
177
|
const config = loadRegistry(home);
|
|
162
178
|
return config.vaults.map(vault => {
|
|
163
|
-
const base = { ...vault, pii: effectivePii(vault) };
|
|
179
|
+
const base = { ...vault, pii: effectivePii(vault), piiPolicy: piiPolicy(vault) };
|
|
164
180
|
try {
|
|
165
181
|
const root = realpathSync(vault.path), notes = join(root, "notes");
|
|
166
182
|
if (root !== vault.path || realpathSync(notes) !== notes || !statSync(notes).isDirectory()) throw new Error("changed");
|
package/src/tidy.ts
CHANGED
|
@@ -11,7 +11,8 @@ import { loadRegistry, syncDirectory } from "./registry.ts";
|
|
|
11
11
|
import { normalizeSource, sourceReferences, sourceWorkspaceRoot, missingSourceWarnings } from "./source.ts";
|
|
12
12
|
import { appendLog, LOG_LOCK } from "./vault-log.ts";
|
|
13
13
|
import { appendSection, foldConnections, h2Sections } from "./write-operations.ts";
|
|
14
|
-
import {
|
|
14
|
+
import { SCHEMA_KEYS, SECRET_PATTERNS, TYPED_KEYS, screenContent } from "./write-validation.ts";
|
|
15
|
+
import { redactPii } from "./pii.ts";
|
|
15
16
|
import { applyWrite, sha256, snapshot } from "./writer.ts";
|
|
16
17
|
import type { WriteIssue } from "./write-types.ts";
|
|
17
18
|
|
|
@@ -281,13 +282,14 @@ function normalizeLogLine(line: string): string | undefined {
|
|
|
281
282
|
if (seconds && LOG_STRUCTURE.test(seconds[1]! + seconds[2]!)) return seconds[1]! + seconds[2]! + line.slice(body.length);
|
|
282
283
|
return `# legacy: ${line}`;
|
|
283
284
|
}
|
|
284
|
-
|
|
285
|
+
/** PII is redacted where the vault policy denies it on the log surface; secrets always are. */
|
|
286
|
+
function sanitizeLog(text: string, vault: VaultSpec, config: VmemConfig): { text: string; lines: number; redactions: number; formats: number; changes: string[] } {
|
|
285
287
|
let lines = 0, redactions = 0, formats = 0;
|
|
286
288
|
const changes: string[] = [];
|
|
287
289
|
const out = text.split("\n").map((line, index) => {
|
|
288
290
|
const secret = redact(line);
|
|
289
291
|
let hits = secret.hits, next = secret.text;
|
|
290
|
-
|
|
292
|
+
const pii = redactPii(next, vault, config, "log"); hits += pii.hits; next = pii.text;
|
|
291
293
|
const normalized = normalizeLogLine(next);
|
|
292
294
|
if (normalized !== undefined) { next = normalized; formats++; }
|
|
293
295
|
if (hits || normalized !== undefined) {
|
|
@@ -359,7 +361,7 @@ export function tidyVault(home: string, request: TidyRequest, options: { env?: N
|
|
|
359
361
|
if (rules.includes("log")) {
|
|
360
362
|
const before = readLogFile(root);
|
|
361
363
|
if (before !== null) {
|
|
362
|
-
const sanitized = sanitizeLog(before);
|
|
364
|
+
const sanitized = sanitizeLog(before, vault, config);
|
|
363
365
|
result.log = { lines: sanitized.lines, redactions: sanitized.redactions, formats: sanitized.formats, changes: sanitized.changes };
|
|
364
366
|
if (sanitized.text !== before) { logPlan = { before, ...sanitized }; result.counts.log = sanitized.redactions + sanitized.formats; }
|
|
365
367
|
}
|
package/src/vault-log.ts
CHANGED
|
@@ -78,6 +78,7 @@ export function appendLog(home: string, request: LogRequest, options: { env?: No
|
|
|
78
78
|
}
|
|
79
79
|
const formatted = formatLogEvent(request.event, { ...ctx, env: options.env ?? process.env, now: options.now });
|
|
80
80
|
if (!formatted.line || formatted.issues.some(issue => issue.severity === "error")) { result.issues.push(...formatted.issues); return result; }
|
|
81
|
+
result.issues.push(...formatted.issues); // nonblocking warnings (for example a log-surface PII warn cell)
|
|
81
82
|
const buffer = Buffer.from(formatted.line), root = ctx.vault.path;
|
|
82
83
|
lockFile = join(root, LOG_LOCK);
|
|
83
84
|
const deadline = Date.now() + 2000, sleeper = new Int32Array(new SharedArrayBuffer(4));
|
package/src/write-types.ts
CHANGED
|
@@ -1,3 +1,4 @@
|
|
|
1
|
+
import type { PiiSurface } from "./config.ts";
|
|
1
2
|
import type { LogEvent, LogReceipt } from "./log-types.ts";
|
|
2
3
|
import type { VaultSpec, VmemConfig } from "./config.ts";
|
|
3
4
|
export type WriteOp =
|
|
@@ -32,6 +33,8 @@ export interface ValidationContext {
|
|
|
32
33
|
/** Original flat notes snapshot, paths prefixed with notes/. */
|
|
33
34
|
notes: Map<string, string>;
|
|
34
35
|
operation: WriteOp;
|
|
36
|
+
/** PII surface for content screening; `notes` unless an internal caller (org promotion) sets it. */
|
|
37
|
+
piiSurface?: PiiSurface;
|
|
35
38
|
}
|
|
36
39
|
export interface ChangeReceipt { path: string; beforeSha: string | null; sha256: string | null; bytes: number }
|
|
37
40
|
export interface WriteReceipt {
|
package/src/write-validation.ts
CHANGED
|
@@ -6,6 +6,8 @@ import { parseFrontmatter } from "./frontmatter.ts";
|
|
|
6
6
|
import { extractLinks, normalizeTarget, proseLines, resolveLinks, type Link, type LinkNote, type ResolvedLink } from "./links.ts";
|
|
7
7
|
import { canonicalPath, isDenied } from "./platform.ts";
|
|
8
8
|
import type { PlannedChange, ValidationContext, WriteIssue } from "./write-types.ts";
|
|
9
|
+
import type { PiiSurface } from "./config.ts";
|
|
10
|
+
import { screenPii } from "./pii.ts";
|
|
9
11
|
import { moveSourcePath } from "./write-operations.ts";
|
|
10
12
|
|
|
11
13
|
const TYPES = new Set(["insight", "decision", "observation", "person", "project", "method", "moc", "reference"]);
|
|
@@ -30,50 +32,16 @@ const WIKI_FIELDS = ["topics", "project", "superseded_by", "supersedes", "source
|
|
|
30
32
|
const SECRETS = [/-----BEGIN [A-Z ]*PRIVATE KEY-----/, /\bAKIA[0-9A-Z]{16}\b/, /\bgh[pousr]_[A-Za-z0-9]{30,}/, /\bgithub_pat_[A-Za-z0-9_]{40,}/, /\bsk-[A-Za-z0-9_-]{20,}/,
|
|
31
33
|
/\bxox[abprs]-[A-Za-z0-9-]{10,}/, /\bAIza[0-9A-Za-z_-]{35}\b/, /\beyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}/, /\b(?:password|passwd|secret|api[_-]?key|token)\s*[:=]\s*\S{8,}/i];
|
|
32
34
|
export const SECRET_PATTERNS: readonly RegExp[] = SECRETS;
|
|
33
|
-
|
|
34
|
-
export const EMAIL_PATTERN = EMAIL;
|
|
35
|
+
export { EMAIL_PATTERN } from "./pii.ts";
|
|
35
36
|
const MAX_BYTES = 256 * 1024;
|
|
36
|
-
|
|
37
37
|
function issue(code: string, message: string, severity: WriteIssue["severity"] = "error", path?: string): WriteIssue {
|
|
38
38
|
return { code, severity, message, ...(path ? { path } : {}) };
|
|
39
39
|
}
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
const withoutTimestamps = text.replace(/\b[0-9]{4}-(?:0[1-9]|1[0-2])-(?:0[1-9]|[12][0-9]|3[01])[ T](?:[01][0-9]|2[0-3]):[0-5][0-9](?::[0-5][0-9](?:\.[0-9]{1,3})?)?(?:Z|[+-](?:[01][0-9]|2[0-3]):[0-5][0-9])?\b/g, "\n");
|
|
43
|
-
// The newline replacement is outside the candidate class, so neighbours of a timestamp never merge.
|
|
44
|
-
const digits = (value: string) => value.replace(/\D/g, "").length;
|
|
45
|
-
for (const match of withoutTimestamps.matchAll(/(?<![\p{L}\p{N}])\+?\d(?:[\d ().-]{6,}\d)(?![\p{L}\p{N}])/gu)) {
|
|
46
|
-
const count = digits(match[0]);
|
|
47
|
-
if (count >= 9 && count <= 15) return true;
|
|
48
|
-
if (count <= 15) continue;
|
|
49
|
-
// Oversized run, e.g. two phones separated by spaces: evaluate every contiguous run of
|
|
50
|
-
// whitespace-separated groups alone so each embedded phone is screened (fails closed).
|
|
51
|
-
const groups = match[0].split(/\s+/).filter(Boolean);
|
|
52
|
-
for (let start = 0; start < groups.length; start++) {
|
|
53
|
-
let total = 0;
|
|
54
|
-
for (let end = start; end < groups.length && total <= 15; end++) {
|
|
55
|
-
total += digits(groups[end]!);
|
|
56
|
-
if (total >= 9 && total <= 15) return true;
|
|
57
|
-
}
|
|
58
|
-
}
|
|
59
|
-
}
|
|
60
|
-
return false;
|
|
61
|
-
}
|
|
62
|
-
/** Screen the whole prospective text, including frontmatter, without returning matched values. */
|
|
63
|
-
export function screenContent(text: string, vault: VaultSpec, config: VmemConfig): WriteIssue[] {
|
|
40
|
+
/** Screen the whole prospective text, including frontmatter, without returning matched values. Secrets are always refused; PII follows the vault policy for this surface. */
|
|
41
|
+
export function screenContent(text: string, vault: VaultSpec, config: VmemConfig, surface: PiiSurface = "notes"): WriteIssue[] {
|
|
64
42
|
const issues: WriteIssue[] = [];
|
|
65
43
|
if (SECRETS.some(pattern => pattern.test(text))) issues.push(issue("V06", "secret-looking content is refused"));
|
|
66
|
-
|
|
67
|
-
if (pii === "deny") {
|
|
68
|
-
const normalized = text.normalize("NFKC");
|
|
69
|
-
if (EMAIL.test(normalized) || hasPhone(normalized)) issues.push(issue("V07", "PII is refused by the vault policy"));
|
|
70
|
-
for (const pattern of config.piiPatterns ?? []) {
|
|
71
|
-
try {
|
|
72
|
-
if (new RegExp(pattern).test(text)) { issues.push(issue("V07", "configured PII pattern matched")); break; }
|
|
73
|
-
} catch { issues.push(issue("V07", "invalid PII screening configuration")); break; }
|
|
74
|
-
}
|
|
75
|
-
}
|
|
76
|
-
return issues;
|
|
44
|
+
return [...issues, ...screenPii(text, vault, config, surface)];
|
|
77
45
|
}
|
|
78
46
|
|
|
79
47
|
/** Quoted escapes must not hide semantic metadata from the same content policy. */
|
|
@@ -81,7 +49,8 @@ function screenMetadata(data:Record<string,unknown>,context:ValidationContext,pa
|
|
|
81
49
|
const found=new Map<string,WriteIssue>(),active=new WeakSet<object>();let nodes=0,bytes=0;
|
|
82
50
|
const inspect=(text:string)=>{
|
|
83
51
|
bytes+=Buffer.byteLength(text);if(bytes>MAX_BYTES*4)throw new Error("budget");
|
|
84
|
-
|
|
52
|
+
// One issue per code; an error is never downgraded by a later warning of the same code.
|
|
53
|
+
for(const item of screenContent(text,context.vault,context.config,context.piiSurface)){const prior=found.get(item.code);if(!prior||(prior.severity!=="error"&&item.severity==="error"))found.set(item.code,{...item,path});}
|
|
85
54
|
};
|
|
86
55
|
const visit=(value:unknown,key:string|undefined,depth:number):void=>{
|
|
87
56
|
if(++nodes>10000||depth>32)throw new Error("budget");
|
|
@@ -241,7 +210,7 @@ export function validateWrite(changes: PlannedChange[], context: ValidationConte
|
|
|
241
210
|
else if (change.path.startsWith("notes/")) final.set(change.path, change.after);
|
|
242
211
|
if (change.after === null) continue;
|
|
243
212
|
if (Buffer.byteLength(change.after) > MAX_BYTES) issues.push(issue("V09", "prospective file exceeds 256 KiB", "error", change.path));
|
|
244
|
-
issues.push(...screenContent(change.after, context.vault, context.config).map(value => ({ ...value, path: change.path })));
|
|
213
|
+
issues.push(...screenContent(change.after, context.vault, context.config, context.piiSurface).map(value => ({ ...value, path: change.path })));
|
|
245
214
|
try{issues.push(...screenMetadata(parseFrontmatter(change.after).data,context,change.path));}catch{ /* Invalid note YAML is reported separately; reports remain schema-exempt. */ }
|
|
246
215
|
}
|
|
247
216
|
const sourceRoot=sourceWorkspaceRoot(context.vault);
|
package/src/writer.ts
CHANGED
|
@@ -11,6 +11,7 @@ import { parseFrontmatter } from "./frontmatter.ts";
|
|
|
11
11
|
import { canonicalPath, isDenied } from "./platform.ts";
|
|
12
12
|
import { moveSourcePath, notePath, planOperation, refuse, WriteRefusal } from "./write-operations.ts";
|
|
13
13
|
import { validateWrite } from "./write-validation.ts";
|
|
14
|
+
import type { PiiSurface } from "./config.ts";
|
|
14
15
|
import type { ChangeReceipt, PlannedChange, WriteIssue, WriteReceipt, WriteRequest } from "./write-types.ts";
|
|
15
16
|
export const MUTATION_LOCK = "mutation.lock";
|
|
16
17
|
export const MAX_WRITE_BYTES = 256 * 1024;
|
|
@@ -111,7 +112,7 @@ function duplicateWarnings(request: WriteRequest, notes: Map<string, string>, ch
|
|
|
111
112
|
} finally { db.close(); }
|
|
112
113
|
}
|
|
113
114
|
/** Single mutation entry point. Preview/dry-run perform no writes, including locks and caches. */
|
|
114
|
-
export function applyWrite(home: string, request: WriteRequest, options: { env?: NodeJS.ProcessEnv } = {}): WriteReceipt {
|
|
115
|
+
export function applyWrite(home: string, request: WriteRequest, options: { env?: NodeJS.ProcessEnv; piiSurface?: PiiSurface } = {}): WriteReceipt {
|
|
115
116
|
let lockFd: number | undefined, lockFile: string | undefined, lockIdentity: { ino: number; dev: number } | undefined;
|
|
116
117
|
let index: VaultIndex | undefined;
|
|
117
118
|
const temps = new Set<string>(), createdDirs: string[] = [], committed: PlannedChange[] = [];
|
|
@@ -150,11 +151,11 @@ export function applyWrite(home: string, request: WriteRequest, options: { env?:
|
|
|
150
151
|
}
|
|
151
152
|
changes = planOperation(request.operation, notes, reportBefore);
|
|
152
153
|
for (const change of changes) if (isDenied(join(root, change.path), config.denyPaths)) refuse("V06", "planned mutation path is denied", change.path);
|
|
153
|
-
issues = validateWrite(changes, { vault, config, notes, operation: request.operation });
|
|
154
|
+
issues = validateWrite(changes, { vault, config, notes, operation: request.operation, ...(options.piiSurface ? { piiSurface: options.piiSurface } : {}) });
|
|
154
155
|
if (!issues.some(issue => issue.severity === "error")) issues.push(...duplicateWarnings(request, notes, changes));
|
|
155
156
|
const input = { vault: request.vault, channel: request.channel ?? "cli", operation: request.operation, expectedShas: request.expectedShas ?? {}, log: request.log ?? null };
|
|
156
157
|
const identity = lstatSync(root);
|
|
157
|
-
fingerprint = sha256(canonical({ input, vault, policy: { denyPaths: config.denyPaths ?? [], piiPatterns: config.piiPatterns ?? [] }, root, identity: [identity.dev, identity.ino], files: [...notes].map(([path, text]) => [path, sha256(text)]), changes: receipts(changes) }));
|
|
158
|
+
fingerprint = sha256(canonical({ input, vault, policy: { denyPaths: config.denyPaths ?? [], piiPatterns: config.piiPatterns ?? [], piiSurface: options.piiSurface ?? "notes" }, root, identity: [identity.dev, identity.ino], files: [...notes].map(([path, text]) => [path, sha256(text)]), changes: receipts(changes) }));
|
|
158
159
|
if (twoStep && request.confirm === true && (typeof request.fingerprint !== "string" || request.fingerprint !== fingerprint)) refuse("V09", "stale or missing preview fingerprint");
|
|
159
160
|
if (issues.some(issue => issue.severity === "error")) return result = { ok: false, applied: false, issues, changes: receipts(changes), fingerprint, indexUpdated: false };
|
|
160
161
|
if (preview) return result = { ok: true, applied: false, preview: true, ...(request.operation.op === "link" && !changes.length ? {unchanged:true} : {}), issues, changes: receipts(changes), fingerprint, indexUpdated: false };
|