@vib795/agent-memory 0.4.0 → 0.5.0

package/README.md

@@ -124,6 +124,89 @@ Two things are deliberately dropped on the way through:
 Importing the same file twice updates rather than duplicates, so re-running after a
 change is safe.
 
+### Sharing an export with another person
+
+Scope decides *whose* knowledge travels. It does not decide whether the file can be
+handed to a person, and those are separate questions. Every export is scanned for
+personal identifiers on the way out, and the file carries a receipt naming the rule
+set that ran and what it took.
+
+```
+$ agent-memory export --out carry.json
+Exported 41 global-scope notes to carry.json.
+rule set: pii/v1
+redacted: 6 phone, 2 payment-card, 1 ssn
+withheld: 1 note
+  oncall-roster — 8 identifiers (phone)
+
+Structured identifiers only. Have an agent review this file for names and
+identifying prose before sharing it outside your team.
+```
+
+Notes are **redacted in place**, so the knowledge survives and the identifier does
+not. A note that is *mostly* identifiers — a roster, a contact sheet — is withheld
+whole and named in the receipt, because a redacted skeleton is useless to the reader
+and still re-identifiable from the structure that remains.
+
+**Two layers, and the markers tell you which one acted.**
+
+| | Runs at | Asks | Marker |
+|---|---|---|---|
+| `src/redact.js` | capture | could this authenticate as someone? | `<redacted:kind>` |
+| `src/pii.js` | export | could this identify a person? | `[redacted:kind]` |
+
+They disagree about your own email address on purpose. Capture keeps it, because it
+is already public in every commit you have ever pushed. Export removes it, because it
+is not yours to hand to someone else alongside a hundred notes about how a client
+works.
+
+**What the deterministic layer does not catch.** Names, job titles, postal addresses
+and identifying prose have no shape to match on. They are language, and the honest
+place to judge them is the model already in the conversation — so the receipt says so
+instead of letting a pattern imply a completeness it does not have. Ask your agent to
+review the export before it leaves your team.
+
+## Evals and governance
+
+Disclosure control is a claim until it is measured, so it is measured on every run of
+`npm test`:
+
+```
+[pii/v1] recall by kind
+  email          2/2
+  ip-address     2/2
+  payment-card   3/3
+  phone          4/4
+  ssn            2/2
+[pii/v1] precision 15/15 on the negative corpus
+[pii/v1] 3 known gaps pass through by design: person name, postal address, bare internal identifier
+```
+
+The practices behind that number, which are the transferable part:
+
+- **Both error directions are costed, not just the obvious one.** A miss discloses one
+  identifier. A false hit shreds a sentence, and a tool that mangles ordinary prose
+  gets switched off, which discloses everything. Recall is held at 1.0 because no
+  number of leaked identifiers is acceptable; precision is held at 1.0 against a
+  corpus built to tempt each detector with what it most resembles — a git SHA for a
+  card, a four-part version for an address, a date for a phone number.
+- **Gaps are asserted as gaps.** `test/pii.eval.test.js` proves that names and
+  addresses pass through, so closing one has to be a deliberate act rather than a
+  silent change in what an old receipt meant.
+- **The eval is checked for teeth.** Removing the Luhn validation drops precision to
+  14/15 and fails the build. A green suite that cannot go red is decoration.
+- **Corpora are synthetic.** Reserved ranges only — RFC 5737 addresses, the card
+  networks' published test numbers — so the file that tests for leaks is not one.
+- **Rule sets are versioned.** A receipt means something only against the rules that
+  produced it, so `pii/v1` is stamped into every export and changing a detector
+  changes the name.
+
+For sharing outside your team: keep the receipt with the file, re-export rather than
+hand-editing a file you already sent, and treat `--scope all` as a decision with your
+name on it. What this is not: it is not encryption, access control, or a DLP product.
+It is one command that refuses to hand over identifiers, and it says exactly what it
+checked.
+
 ## Engagements
 
 If you work for more than one client on one machine, knowledge from one of them is
@@ -389,7 +472,7 @@ is generated state, and the committed value is only a placeholder.
 
 Needs Node 22.5 or newer; `doctor` says so plainly if the version is too old.
 
-Run `npm test` for the suite (78 tests, no dependencies). CI runs it on Linux,
+Run `npm test` for the suite (87 tests, no dependencies). CI runs it on Linux,
 macOS and Windows across Node 22 and 24, and separately installs the packed tarball
 and exercises it end to end on all three.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@vib795/agent-memory",
-  "version": "0.4.0",
+  "version": "0.5.0",
   "description": "Durable cross-repo knowledge graph for GitHub Copilot and Claude Code. Markdown source of truth, disposable SQLite index, zero runtime dependencies.",
   "keywords": [
     "github-copilot",

package/src/cli.js

@@ -17,6 +17,7 @@ import { setup as runSetup, unlinkSkills, danglingSkillLinks, SKILLS } from './s
 import { detectTargets, installableTargets } from './targets.js';
 import { join, dirname } from 'node:path';
 import { atomicWrite } from './atomic.js';
+import { redactNodeForExport, buildReceipt, renderReceipt } from './pii.js';
 
 /**
  * One process, one answer.
@@ -581,6 +582,10 @@ const EXPORT_SCOPES = ['global', 'repo', 'all'];
  * `captured_sha` is deliberately dropped. It names a commit that does not exist
  * anywhere else, and a staleness signal that cannot be checked is worse than none:
  * it would either read as current forever or claim the history was rewritten.
+ *
+ * Personal identifiers are removed on the way out, and the export carries a receipt
+ * saying which rule set ran and what it took. Scope decides whose knowledge travels;
+ * the receipt decides whether it can be handed to a person.
  */
 function cmdExport(opts) {
   const scope = opts.scope === true ? 'global' : (opts.scope ?? 'global');
@@ -612,21 +617,42 @@ function cmdExport(opts) {
     source: n.source ?? 'manual',
   }));
 
-  const payload = `${JSON.stringify({ nodes }, null, 2)}\n`;
+  // Export is the only path by which the store leaves this machine, which makes it
+  // the only place disclosure control belongs. It is a different question from the
+  // one capture-time redaction answers, and src/pii.js says why at length.
+  const results = nodes.map((n) => ({ id: n.id, ...redactNodeForExport(n) }));
+  const clean = results.filter((r) => !r.withheld).map((r) => r.node);
+  const receipt = buildReceipt(results, { scanned: nodes.length });
+  const payload = `${JSON.stringify({ nodes: clean, redaction: receipt }, null, 2)}\n`;
+
   if (typeof opts.out === 'string') {
     atomicWrite(opts.out, payload);
     return {
       ok: true,
-      exported: nodes.length,
+      exported: clean.length,
+      withheld: receipt.withheld.length,
       scope,
       out: opts.out,
-      text: `Exported ${nodes.length} ${scope}-scope note${nodes.length === 1 ? '' : 's'} to ${opts.out}.`,
+      redaction: receipt,
+      text: [
+        `Exported ${clean.length} ${scope}-scope note${clean.length === 1 ? '' : 's'} to ${opts.out}.`,
+        ...renderReceipt(receipt),
+      ].join('\n'),
     };
   }
-  // Straight to stdout so it pipes, with the count on stderr where it will not
-  // corrupt the document.
-  process.stderr.write(`${nodes.length} ${scope}-scope notes\n`);
-  return { ok: true, exported: nodes.length, scope, nodes, text: payload.trimEnd() };
+  // Straight to stdout so it pipes, with the count and the receipt on stderr where
+  // they will not corrupt the document.
+  process.stderr.write(`${clean.length} ${scope}-scope notes\n`);
+  for (const line of renderReceipt(receipt)) process.stderr.write(`${line}\n`);
+  return {
+    ok: true,
+    exported: clean.length,
+    withheld: receipt.withheld.length,
+    scope,
+    nodes: clean,
+    redaction: receipt,
+    text: payload.trimEnd(),
+  };
 }
 
 /**

package/src/pii.js

@@ -0,0 +1,269 @@
+/**
+ * Disclosure control for export.
+ *
+ * This is not the same job as src/redact.js and must not be folded into it. That one
+ * runs at capture and asks "could this authenticate as someone?" — its threat is a
+ * credential reaching disk, and it deliberately preserves the operator's own email
+ * because that address is already public in every commit they have ever pushed.
+ *
+ * This one runs at export and asks "could this identify a person?" — its threat is a
+ * human reading the file. The operator's own email is the clearest case of the
+ * difference: safe to store, not yours to hand to someone else along with a hundred
+ * notes about how their client works.
+ *
+ * Two layers, and only one of them lives here. Structured identifiers have shape, so
+ * they are matched and validated deterministically, which makes them testable and the
+ * result reproducible from a named rule set. Names, addresses and identifying prose
+ * have no shape; they are language, and the honest place to judge them is the model
+ * already sitting in the conversation. The receipt says so out loud rather than
+ * letting a regex imply a completeness it does not have.
+ */
+
+/** Bump when a detector changes. A receipt is only reproducible against its rule set. */
+export const RULE_SET = 'pii/v1';
+
+/** Detections in one note before the note is withheld whole rather than redacted. */
+export const DEFAULT_WITHHOLD_COUNT = 8;
+
+/** Share of the text that may be replaced before redaction stops being meaningful. */
+export const DEFAULT_WITHHOLD_RATIO = 0.3;
+
+/** Luhn, the checksum every payment card carries. Turns a digit run into a card. */
+function luhnValid(digits) {
+  let sum = 0;
+  let double = false;
+  for (let i = digits.length - 1; i >= 0; i -= 1) {
+    let d = digits.charCodeAt(i) - 48;
+    if (d < 0 || d > 9) return false;
+    if (double) {
+      d *= 2;
+      if (d > 9) d -= 9;
+    }
+    sum += d;
+    double = !double;
+  }
+  return sum % 10 === 0;
+}
+
+/**
+ * A US SSN has structure the Social Security Administration never issues.
+ * Checking it is what separates an SSN from any other nine digits with dashes.
+ */
+function ssnValid(area, group, serial) {
+  if (area === '000' || area === '666' || area[0] === '9') return false;
+  if (group === '00' || serial === '0000') return false;
+  return true;
+}
+
+function octetsValid(ip) {
+  const parts = ip.split('.');
+  return parts.length === 4 && parts.every((p) => p.length <= 3 && Number(p) <= 255);
+}
+
+/**
+ * Addresses that describe infrastructure rather than a person.
+ *
+ * The privacy interest in an IP is that a routable one, with a timestamp, can be put
+ * to a subscriber by whoever holds the logs. A pod CIDR cannot. Capture-time
+ * redaction already removes these ranges as infrastructure, so scrubbing them again
+ * here would only cost meaning in ordinary engineering notes.
+ */
+function reservedV4(ip) {
+  const [a, b] = ip.split('.').map(Number);
+  if (a === 0 || a === 10 || a === 127 || a >= 224) return true;
+  if (a === 169 && b === 254) return true;
+  if (a === 172 && b >= 16 && b <= 31) return true;
+  if (a === 192 && b === 168) return true;
+  return false;
+}
+
+/**
+ * Every detector is precision-first.
+ *
+ * A false negative leaks one identifier. A false positive shreds a sentence, and a
+ * tool that mangles ordinary prose gets switched off, which leaks everything. So each
+ * pattern here carries a checksum, a structural rule, or a separator requirement that
+ * ordinary technical writing does not satisfy. Bare digit runs are left to the
+ * semantic pass rather than guessed at, because a git SHA, an order number and a
+ * national ID are the same characters.
+ *
+ * Order is load-bearing. A grouped card number is also a valid phone shape, so cards
+ * are consumed before phones get a chance; reordering this array silently changes
+ * what the labels say happened.
+ */
+const DETECTORS = [
+  {
+    kind: 'email',
+    // No self-address exemption. That exemption is correct at capture and wrong here.
+    re: /\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b/g,
+  },
+  {
+    kind: 'ssn',
+    // Only the delimited form. Nine adjacent digits are too many other things.
+    re: /(?<![\w-])(\d{3})-(\d{2})-(\d{4})(?![\w-])/g,
+    validate: (m) => ssnValid(m[1], m[2], m[3]),
+  },
+  {
+    kind: 'payment-card',
+    re: /(?<![\w-])(?:\d[ -]?){12,18}\d(?![\w-])/g,
+    validate: (m) => {
+      const digits = m[0].replace(/\D/g, '');
+      return digits.length >= 13 && digits.length <= 19 && luhnValid(digits);
+    },
+  },
+  {
+    kind: 'phone',
+    // Requires separators or a country code, so `10.2.4` and `v1.2.3` cannot match,
+    // and a bare run of ten digits is left alone on purpose.
+    re: /(?<![\w-])(?:\+\d{1,3}[\s.-]?)?(?:\(\d{2,4}\)|\d{2,4})[\s.-]\d{2,4}[\s.-]\d{2,5}(?:[\s.-]\d{1,5})?(?![\w-])/g,
+    validate: (m) => {
+      const digits = m[0].replace(/\D/g, '');
+      if (digits.length < 10 || digits.length > 15) return false;
+      // One repeated digit is a placeholder or a masked field, never a number.
+      if (/^(\d)\1+$/.test(digits)) return false;
+      // A date reads as three separated numbers too. Reject anything shaped like one
+      // rather than deciding by punctuation, which varies by locale.
+      return !/^\d{4}[.\-/]\d{1,2}[.\-/]\d{1,2}$/.test(m[0].trim());
+    },
+  },
+  {
+    kind: 'ip-address',
+    // Capture-time redaction removes RFC1918 ranges as infrastructure. A routable
+    // address is the one that can identify a subscriber, and it survives to here.
+    re: /(?<![\w.])\d{1,3}(?:\.\d{1,3}){3}(?![\w.])/g,
+    validate: (m) => {
+      if (!octetsValid(m[0])) return false;
+      // Four single-digit groups is a four-part version string (6.0.1.2) far more
+      // often than an address. A deliberate recall gap, measured in the eval and
+      // covered by the semantic pass rather than paid for in shredded prose.
+      if (/^\d(\.\d){3}$/.test(m[0])) return false;
+      return !reservedV4(m[0]);
+    },
+  },
+];
+
+/**
+ * Scan text, returning the redacted form and what fired.
+ *
+ * Fail-closed in the same sense as capture: a non-string is a programming error, and
+ * returning it unscanned would be the one outcome worse than throwing.
+ */
+export function scanText(text) {
+  if (typeof text !== 'string') {
+    throw new TypeError('scanText() requires a string; refusing to export unscanned content');
+  }
+  const counts = new Map();
+  let redactedChars = 0;
+  let out = text;
+
+  for (const { kind, re, validate } of DETECTORS) {
+    out = out.replace(re, (...args) => {
+      const match = args.slice(0, -2);
+      const whole = match[0];
+      if (validate && !validate(match)) return whole;
+      counts.set(kind, (counts.get(kind) || 0) + 1);
+      redactedChars += whole.length;
+      return `[redacted:${kind}]`;
+    });
+  }
+
+  return {
+    text: out,
+    findings: [...counts.entries()].map(([kind, count]) => ({ kind, count })),
+    redactedChars,
+  };
+}
+
+/**
+ * Clean one node for export, or decide it cannot be cleaned.
+ *
+ * Withholding is not a fallback for a broken redactor; it answers a different
+ * question. A note that is mostly identifiers is a contact record rather than
+ * knowledge, and its redacted skeleton is both useless to the reader and still
+ * re-identifiable from the structure that remains. Better to name it in the receipt
+ * and leave it behind.
+ */
+export function redactNodeForExport(node, opts = {}) {
+  const withholdCount = opts.withholdCount ?? DEFAULT_WITHHOLD_COUNT;
+  const withholdRatio = opts.withholdRatio ?? DEFAULT_WITHHOLD_RATIO;
+
+  const clean = { ...node };
+  const counts = new Map();
+  let redactedChars = 0;
+  let textChars = 0;
+
+  for (const field of ['title', 'body']) {
+    if (typeof clean[field] !== 'string') continue;
+    textChars += clean[field].length;
+    const r = scanText(clean[field]);
+    clean[field] = r.text;
+    redactedChars += r.redactedChars;
+    for (const f of r.findings) counts.set(f.kind, (counts.get(f.kind) || 0) + f.count);
+  }
+
+  const findings = [...counts.entries()].map(([kind, count]) => ({ kind, count }));
+  const total = findings.reduce((n, f) => n + f.count, 0);
+  const ratio = textChars ? redactedChars / textChars : 0;
+
+  if (total >= withholdCount || ratio > withholdRatio) {
+    return {
+      node: null,
+      findings,
+      withheld: true,
+      reason:
+        total >= withholdCount ? `${total} identifiers` : `${Math.round(ratio * 100)}% of the text`,
+    };
+  }
+  return { node: clean, findings, withheld: false, reason: null };
+}
+
+/**
+ * Fold per-note results into the receipt that ships with the export.
+ *
+ * The receipt is the governance artifact. An export without one is a file of unknown
+ * provenance, and the reason to write it down is that "we redact PII" is a claim
+ * while "pii/v1 removed four emails and withheld two notes" is evidence.
+ */
+export function buildReceipt(results, { scanned }) {
+  const counts = new Map();
+  const withheld = [];
+  for (const r of results) {
+    for (const f of r.findings) counts.set(f.kind, (counts.get(f.kind) || 0) + f.count);
+    if (r.withheld) {
+      withheld.push({ id: r.id, reason: r.reason, kinds: r.findings.map((f) => f.kind) });
+    }
+  }
+  const redactions = [...counts.entries()]
+    .map(([kind, count]) => ({ kind, count }))
+    .sort((a, b) => b.count - a.count || a.kind.localeCompare(b.kind));
+
+  return {
+    ruleSet: RULE_SET,
+    scanned,
+    exported: scanned - withheld.length,
+    redactions,
+    withheld,
+    // Stated in the artifact rather than only in the docs, because the person who
+    // reads the receipt is the person deciding whether to send the file.
+    semanticReviewRequired:
+      'Structured identifiers only. Names, job titles, addresses and identifying prose are not detected here; have an agent review the export before sharing it outside your team.',
+  };
+}
+
+/** The receipt as lines, for the humans who will read it in a terminal. */
+export function renderReceipt(receipt) {
+  const out = [`rule set: ${receipt.ruleSet}`];
+  if (receipt.redactions.length) {
+    out.push(`redacted: ${receipt.redactions.map((r) => `${r.count} ${r.kind}`).join(', ')}`);
+  } else {
+    out.push('redacted: nothing matched');
+  }
+  if (receipt.withheld.length) {
+    out.push(`withheld: ${receipt.withheld.length} note${receipt.withheld.length === 1 ? '' : 's'}`);
+    for (const w of receipt.withheld) out.push(`  ${w.id} — ${w.reason} (${w.kinds.join(', ')})`);
+  }
+  out.push('');
+  out.push('Structured identifiers only. Have an agent review this file for names and');
+  out.push('identifying prose before sharing it outside your team.');
+  return out;
+}