pi-memory-stone 0.1.4 → 0.1.5

package/README.md

@@ -46,9 +46,10 @@ pi-memory-stone package source
 │   ├── retrieval/                # FTS search, hybrid ranking, injection builder
 │   ├── commands/                 # /memory-* slash commands
 │   ├── tools/                    # LLM-callable tools
+│   ├── vault/                    # Optional Obsidian-compatible markdown vaults
 │   ├── privacy/                  # Secret redaction, sensitive path filtering
 │   └── config/                   # Project identity, settings
-└── test/                         # 51 tests across 6 test files
+└── test/                         # tests across memory, privacy, portable, and vault helpers
 ```
 
 ## How It Works
@@ -116,6 +117,10 @@ Before each agent turn, the extension:
 | `/memory-import <memory-export.json> --preserve-project` | | Import records while preserving exported project IDs |
 | `/memory-import <memory-export.json> --global` | | Import all records as global memories |
 | `/memory-backup [path]` | `/stone-backup` | Copy the SQLite memory database to a timestamped backup file |
+| `/memory-vault-init [--project\|--personal]` | `/stone-vault-init` | Initialize an Obsidian-compatible markdown vault |
+| `/memory-vault-sync [--project\|--personal]` | `/stone-vault-sync` | Generate vault pages from active memory records |
+| `/memory-vault-status [--project\|--personal]` | `/stone-vault-status` | Show vault path, page counts, registry, and last sync |
+| `/memory-vault-capture-url <url> [--project\|--personal]` | `/stone-vault-capture-url` | Capture a web page into the vault as a source page |
 | `/memory-on` | | Enable memory injection for this session |
 | `/memory-off` | | Disable memory injection for this session |
 
@@ -188,6 +193,52 @@ Use a database backup before pruning or hard deletion:
 
 Import defaults are intentionally practical for machine moves: project-scoped records are remapped to the current project. Use `--preserve-project` to keep exported project IDs, or `--global` to import everything as global memory.
 
+## Knowledge Vaults
+
+Memory vaults are optional Obsidian-compatible markdown projections of active memory records. SQLite remains the source of truth; generated pages may be overwritten by `/memory-vault-sync`.
+
+```bash
+# Project-local vault, written only after explicit init or capture request
+/memory-vault-init --project
+/memory-vault-sync --project
+/memory-vault-status --project
+
+# Capture a web page source into the vault
+/memory-vault-capture-url https://example.com/article --project
+
+# Capture resolves known source formats before generic HTML extraction.
+# Examples: GitHub Gist pages are fetched from gist.githubusercontent.com/raw,
+# GitHub blob URLs are fetched from raw.githubusercontent.com, and raw Markdown
+# is preserved as Markdown.
+
+# Natural-language capture also works from normal prompts:
+# "Capture this article into vault https://example.com/article"
+# "Add page to personal vault https://example.com/article"
+
+# Private personal vault for global memories
+/memory-vault-init --personal
+/memory-vault-sync --personal
+```
+
+Default locations:
+
+```txt
+<repo>/.memory-stone/vault/              # project vault
+~/.pi/agent/memory/vaults/personal/      # personal vault
+```
+
+Initial layout:
+
+```txt
+index.md
+WIKI_SCHEMA.md
+records/{decisions,preferences,tasks,error-resolutions,turn-summaries,session-summaries}/
+sources/                       # captured web source pages
+meta/registry.json
+```
+
+URL capture writes curated source notes into `sources/`. Raw provenance packets (manifest, metadata, fetch attempts, original artifact, extracted markdown) are stored outside the Obsidian vault under `.memory-stone/source-packets/` for project vaults or `~/.pi/agent/memory/source-packets/personal/` for personal vaults. Capture extracts HTML articles with Mozilla Readability, converts to Markdown, redacts secrets, and marks extraction quality as `good` or `weak` with warnings.
+
 ## Privacy & Safety
 
 ### Secret Redaction
@@ -260,16 +311,18 @@ npm run typecheck
 
 These scripts are also runnable from a pi package clone installed from git; the required script runners are regular dependencies because pi package installs omit `devDependencies`.
 
-51 tests across 6 test files:
+69 tests across 8 test files:
 
 | Suite | Tests | Focus |
 |---|---|---|
 | `indexing.test.ts` | 1 | Incremental session indexing |
 | `parser.test.ts` | 10 | Turn parsing, file activity detection, error extraction |
-| `portable.test.ts` | 3 | JSON/Markdown export, JSON import, SQLite backup |
-| `privacy.test.ts` | 17 | Secret redaction, sensitive path filtering |
-| `ranking.test.ts` | 16 | Hybrid ranking, cross-project filtering, injection formatting |
-| `session-state.test.ts` | 4 | Injection mode config, session ref selection, manual-only injection |
+| `portable.test.ts` | 5 | JSON/Markdown export, JSON import, SQLite backup |
+| `privacy.test.ts` | 21 | Secret redaction, sensitive path filtering |
+| `ranking.test.ts` | 18 | Hybrid ranking, cross-project filtering, injection formatting |
+| `session-state.test.ts` | 5 | Injection mode config, session ref selection, manual-only injection |
+| `tools.test.ts` | 2 | Tool visibility and forgetting safety |
+| `vault.test.ts` | 7 | Vault path resolution, initialization, sync, URL capture, registry generation |
 
 ## Configuration
 
@@ -302,6 +355,7 @@ For manual-only memory, keep `enabled: true` and set `injectionMode: "manual"`.
 - LLM extraction (decisions, preferences, tasks)
 - Historical backfill (`/memory-backfill`)
 - Embedding-based semantic search
+- Vault backlinks/lint/search/capture commands
 - `/memory-edit`, `/memory-supersede`
 - Rich TUI memory browser
 - Multi-machine sync

package/package.json

@@ -1,10 +1,12 @@
 {
   "name": "pi-memory-stone",
-  "version": "0.1.4",
+  "version": "0.1.5",
   "type": "module",
   "description": "Global pi extension: preserves and retrieves useful memory across pi sessions",
   "license": "MIT",
-  "keywords": ["pi-package"],
+  "keywords": [
+    "pi-package"
+  ],
   "files": [
     "src",
     "skills/**/*",
@@ -12,15 +14,24 @@
     "LICENSE"
   ],
   "pi": {
-    "extensions": ["./src/index.ts"],
-    "skills": ["./skills"]
+    "extensions": [
+      "./src/index.ts"
+    ],
+    "skills": [
+      "./skills"
+    ]
   },
   "scripts": {
     "test": "NODE_OPTIONS='--experimental-sqlite' tsx --test test/*.test.ts",
     "typecheck": "tsc --noEmit"
   },
   "dependencies": {
+    "@mozilla/readability": "^0.6.0",
+    "@types/jsdom": "^28.0.3",
+    "@types/turndown": "^5.0.6",
+    "jsdom": "^29.1.1",
     "tsx": "^4.22.3",
+    "turndown": "^7.2.4",
     "typescript": "^5.9.3"
   },
   "peerDependencies": {

package/src/commands/index.ts

@@ -1,6 +1,6 @@
 /**
  * Commands: /memory-status, /memory-search, /memory-open, /memory-inject, /memory-mode, /memory-last,
- * /memory-export, /memory-import, /memory-backup
+ * /memory-export, /memory-import, /memory-backup, /memory-vault-*
  */
 
 import type { ExtensionAPI, ExtensionCommandContext } from "@earendil-works/pi-coding-agent";
@@ -24,6 +24,15 @@ import {
   isRecordVisibleInProject,
   parseRefArgs,
 } from "../session-state/index.js";
+import {
+  getVaultStatus,
+  initVault,
+  parseVaultScope,
+  syncVault,
+  type VaultScope,
+} from "../vault/index.js";
+import { captureUrlToVault } from "../vault/capture.js";
+import { extractFirstUrl } from "../vault/intent.js";
 
 export function registerCommands(pi: ExtensionAPI): void {
   // ── /memory-status ──────────────────────────────────────────────
@@ -198,6 +207,64 @@ export function registerCommands(pi: ExtensionAPI): void {
     },
   });
 
+  // ── /memory-vault-* ─────────────────────────────────────────────
+
+  pi.registerCommand("memory-vault-init", {
+    description: "Initialize an Obsidian-compatible memory vault",
+    handler: async (args, ctx) => {
+      await handleMemoryVaultInit(args, ctx);
+    },
+  });
+
+  pi.registerCommand("stone-vault-init", {
+    description: "Alias for /memory-vault-init",
+    handler: async (args, ctx) => {
+      await handleMemoryVaultInit(args, ctx);
+    },
+  });
+
+  pi.registerCommand("memory-vault-sync", {
+    description: "Sync active memory records into the initialized markdown vault",
+    handler: async (args, ctx) => {
+      await handleMemoryVaultSync(args, ctx);
+    },
+  });
+
+  pi.registerCommand("stone-vault-sync", {
+    description: "Alias for /memory-vault-sync",
+    handler: async (args, ctx) => {
+      await handleMemoryVaultSync(args, ctx);
+    },
+  });
+
+  pi.registerCommand("memory-vault-status", {
+    description: "Show memory vault path, initialization, and sync status",
+    handler: async (args, ctx) => {
+      await handleMemoryVaultStatus(args, ctx);
+    },
+  });
+
+  pi.registerCommand("memory-vault-capture-url", {
+    description: "Capture a web page URL into the memory vault",
+    handler: async (args, ctx) => {
+      await handleMemoryVaultCaptureUrl(args, ctx);
+    },
+  });
+
+  pi.registerCommand("stone-vault-status", {
+    description: "Alias for /memory-vault-status",
+    handler: async (args, ctx) => {
+      await handleMemoryVaultStatus(args, ctx);
+    },
+  });
+
+  pi.registerCommand("stone-vault-capture-url", {
+    description: "Alias for /memory-vault-capture-url",
+    handler: async (args, ctx) => {
+      await handleMemoryVaultCaptureUrl(args, ctx);
+    },
+  });
+
   // ── /memory-on / /memory-off ────────────────────────────────────
 
   pi.registerCommand("memory-on", {
@@ -481,6 +548,98 @@ async function handleMemoryBackup(args: string, ctx: ExtensionCommandContext): P
   }
 }
 
+async function handleMemoryVaultInit(args: string, ctx: ExtensionCommandContext): Promise<void> {
+  const parsed = parseCommandArgs(args);
+  const scope = parseVaultScopeOrNotify(parsed, ctx);
+  if (!scope) return;
+
+  try {
+    const projectId = getProjectId(ctx.cwd);
+    const result = initVault(scope, projectId, ctx.cwd);
+    ctx.ui.notify(
+      result.created
+        ? `Initialized ${scope} memory vault at ${result.path}`
+        : `${scope} memory vault already initialized at ${result.path}`,
+      "info",
+    );
+  } catch (err) {
+    ctx.ui.notify(`Memory vault init failed: ${err instanceof Error ? err.message : String(err)}`, "warning");
+  }
+}
+
+async function handleMemoryVaultSync(args: string, ctx: ExtensionCommandContext): Promise<void> {
+  const parsed = parseCommandArgs(args);
+  const scope = parseVaultScopeOrNotify(parsed, ctx);
+  if (!scope) return;
+
+  try {
+    const projectId = getProjectId(ctx.cwd);
+    const result = syncVault(scope, projectId, ctx.cwd);
+    ctx.ui.notify(
+      `Synced ${result.records} memory records to ${result.path} (${result.pagesWritten} page${result.pagesWritten === 1 ? "" : "s"} written). Registry: ${result.registryPath}`,
+      "info",
+    );
+  } catch (err) {
+    ctx.ui.notify(`Memory vault sync failed: ${err instanceof Error ? err.message : String(err)}`, "warning");
+  }
+}
+
+async function handleMemoryVaultStatus(args: string, ctx: ExtensionCommandContext): Promise<void> {
+  const parsed = parseCommandArgs(args);
+  const scope = parseVaultScopeOrNotify(parsed, ctx);
+  if (!scope) return;
+
+  const projectId = getProjectId(ctx.cwd);
+  const status = getVaultStatus(scope, projectId, ctx.cwd);
+  const lines: string[] = [];
+  lines.push("📚 Memory Vault Status");
+  lines.push("");
+  lines.push(`  scope: ${scope}`);
+  lines.push(`  path: ${status.path}`);
+  lines.push(`  initialized: ${status.initialized}`);
+  lines.push(`  registry: ${status.registryExists ? "present" : "missing"}`);
+  lines.push(`  markdown pages: ${status.pageCount}`);
+  lines.push(`  synced record pages: ${status.recordPageCount}`);
+  lines.push(`  last sync: ${status.lastSyncedAt ?? "never"}`);
+  ctx.ui.notify(lines.join("\n"), "info");
+}
+
+async function handleMemoryVaultCaptureUrl(args: string, ctx: ExtensionCommandContext): Promise<void> {
+  const parsed = parseCommandArgs(args);
+  const scope = parseVaultScopeOrNotify(parsed, ctx);
+  if (!scope) return;
+
+  const url = extractFirstUrl(args);
+  if (!url) {
+    ctx.ui.notify("Usage: /memory-vault-capture-url <http-url> [--project|--personal]", "warning");
+    return;
+  }
+
+  try {
+    const projectId = getProjectId(ctx.cwd);
+    const result = await captureUrlToVault(scope, projectId, ctx.cwd, url);
+    const warnings = result.warnings.length > 0 ? `\nWarnings: ${result.warnings.join("; ")}` : "";
+    ctx.ui.notify(
+      `Captured ${result.title} into ${scope} memory vault${result.initialized ? " (initialized vault)" : ""}\nQuality: ${result.quality} (${result.qualityScore})${warnings}\nPage: ${result.pagePath}`,
+      "info",
+    );
+  } catch (err) {
+    ctx.ui.notify(`Memory vault URL capture failed: ${err instanceof Error ? err.message : String(err)}`, "warning");
+  }
+}
+
+function parseVaultScopeOrNotify(
+  parsed: ReturnType<typeof parseCommandArgs>,
+  ctx: ExtensionCommandContext,
+): VaultScope | null {
+  const scope = parseVaultScope(parsed);
+  if (!scope) {
+    ctx.ui.notify("Usage: /memory-vault-<init|sync|status> [--project|--personal|--scope project|personal]", "warning");
+    return null;
+  }
+  return scope;
+}
+
 async function handleMemoryForget(args: string, ctx: ExtensionCommandContext): Promise<void> {
   const { softForgetRecord, hardDeleteRecord, getRecord } = await import("../db/index.js");
 
@@ -550,7 +709,7 @@ function parseCommandArgs(args: string): {
     }
 
     const next = parts[i + 1];
-    if (next && !next.startsWith("--") && ["format"].includes(withoutPrefix)) {
+    if (next && !next.startsWith("--") && ["format", "scope"].includes(withoutPrefix)) {
       options.set(withoutPrefix, next);
       i += 1;
     } else {

package/src/index.ts

@@ -9,6 +9,7 @@
  *  - Deterministic turn_summary and file_activity capture on agent_end
  *  - FTS5 search
  *  - /memory-status, /memory-search, /memory-open, /memory-inject, /memory-last commands
+ *  - /memory-vault-* commands and natural-language URL capture
  *  - memory_search, memory_open, memory_remember, memory_forget tools
  *  - Conservative same-project before_agent_start injection
  */
@@ -21,6 +22,8 @@ import { retrieve, buildInjectionPacket, formatInjectionForLlm } from "./retriev
 import { getProjectId, getConfig, clearProjectCache } from "./config/index.js";
 import { closeDb, getRecord, insertInjection } from "./db/index.js";
 import { getMemorySessionState, manualRecordsToRankedResults } from "./session-state/index.js";
+import { captureUrlToVault } from "./vault/capture.js";
+import { parseVaultCaptureIntent } from "./vault/intent.js";
 import { createHash } from "node:crypto";
 
 // ─── Session-scoped state ───────────────────────────────────────────
@@ -31,6 +34,37 @@ const injectedRefsThisSession: Set<string> = new Set();
 /** Whether memory injection is temporarily disabled for this session */
 let sessionEnabled = true;
 
+async function maybeCaptureVaultUrl(
+  prompt: string,
+  projectId: string | null,
+  cwd: string,
+  signal?: AbortSignal,
+): Promise<string | null> {
+  const intent = parseVaultCaptureIntent(prompt);
+  if (!intent) return null;
+
+  try {
+    const result = await captureUrlToVault(intent.scope, projectId, cwd, intent.url, { signal });
+    const warnings = result.warnings.length > 0 ? `\nWarnings: ${result.warnings.join("; ")}` : "";
+    return `Captured web page into ${intent.scope} memory vault${result.initialized ? " (initialized vault)" : ""}: ${result.title}\nQuality: ${result.quality} (${result.qualityScore})${warnings}\nPage: ${result.pagePath}\nSource packet: ${result.sourcePacketPath}`;
+  } catch (err) {
+    const message = err instanceof Error ? err.message : String(err);
+    console.error("[pi-memory-stone] vault URL capture failed:", err);
+    return `Memory vault URL capture failed: ${message}`;
+  }
+}
+
+function vaultCaptureReturn(systemPrompt: string, notice: string) {
+  return {
+    message: {
+      customType: "memory-vault-capture",
+      content: notice,
+      display: true,
+    },
+    systemPrompt: `${systemPrompt}\n\n--- Memory Vault Capture ---\n${notice}\nThe user's vault capture request has already been handled by pi-memory-stone. Briefly confirm the result; do not fetch the same URL again unless the user asks.\n--- End Memory Vault Capture ---`,
+  };
+}
+
 // ─── Extension entry point ─────────────────────────────────────────
 
 export default function (pi: ExtensionAPI) {
@@ -69,18 +103,24 @@ export default function (pi: ExtensionAPI) {
 
   pi.on("before_agent_start", async (event, ctx) => {
     try {
+      const prompt = event.prompt || "";
+      const projectId = getProjectId(ctx.cwd);
+      const vaultCaptureNotice = await maybeCaptureVaultUrl(prompt, projectId, ctx.cwd, ctx.signal);
+
       // Check if memory is enabled
       const config = getConfig(ctx.cwd);
-      if (!config.enabled) return;
+      if (!config.enabled) {
+        return vaultCaptureNotice ? vaultCaptureReturn(event.systemPrompt || "", vaultCaptureNotice) : undefined;
+      }
 
       const sessionState = getMemorySessionState(ctx.sessionManager.getBranch());
       sessionEnabled = sessionState.enabled;
 
-      if (!sessionEnabled) return;
+      if (!sessionEnabled) {
+        return vaultCaptureNotice ? vaultCaptureReturn(event.systemPrompt || "", vaultCaptureNotice) : undefined;
+      }
 
-      const prompt = event.prompt || "";
       const promptHash = createHash("sha256").update(prompt).digest("hex").slice(0, 12);
-      const projectId = getProjectId(ctx.cwd);
       const injectionMode = sessionState.injectionMode ?? config.injectionMode;
 
       const manualRecords = sessionState.manualRefs
@@ -103,10 +143,11 @@ export default function (pi: ExtensionAPI) {
       }
 
       const selectedResults = [...manualResults, ...autoResults];
-      if (selectedResults.length === 0) return;
-
-      const packet = buildInjectionPacket(selectedResults);
-      const formatted = formatInjectionForLlm(packet, config.maxInjectedTokens);
+      let formatted: string | null = null;
+      if (selectedResults.length > 0) {
+        const packet = buildInjectionPacket(selectedResults);
+        formatted = formatInjectionForLlm(packet, config.maxInjectedTokens);
+      }
 
       // Track only search-selected refs. Manually chosen refs are intentionally
       // injected on every turn until /memory-clear-injected is used.
@@ -114,27 +155,36 @@ export default function (pi: ExtensionAPI) {
         injectedRefsThisSession.add(r.record.id);
       }
 
-      insertInjection({
-        session_id: ctx.sessionManager.getSessionId(),
-        turn_entry_id: ctx.sessionManager.getLeafId() ?? undefined,
-        prompt_hash: promptHash,
-        injected_refs: selectedResults.map((r) => r.record.id).join(","),
-        packet: formatted,
-        reasons: selectedResults.map((r) => r.reasons.join(";")).join(" | "),
-      });
-
-      // Inject as a non-context audit custom entry (separate from LLM context)
-      // but also as a system prompt addition for the LLM
-      const systemPromptAddition = [
-        "",
-        "--- Memory Stone Context ---",
-        formatted,
-        "--- End Memory Stone Context ---",
-      ].join("\n");
-
-      return {
-        systemPrompt: (event.systemPrompt || "") + systemPromptAddition,
-      };
+      if (formatted) {
+        insertInjection({
+          session_id: ctx.sessionManager.getSessionId(),
+          turn_entry_id: ctx.sessionManager.getLeafId() ?? undefined,
+          prompt_hash: promptHash,
+          injected_refs: selectedResults.map((r) => r.record.id).join(","),
+          packet: formatted,
+          reasons: selectedResults.map((r) => r.reasons.join(";")).join(" | "),
+        });
+      }
+
+      if (!formatted && !vaultCaptureNotice) return;
+
+      let systemPrompt = event.systemPrompt || "";
+      if (formatted) {
+        // Inject as a non-context audit custom entry (separate from LLM context)
+        // but also as a system prompt addition for the LLM
+        systemPrompt += [
+          "",
+          "--- Memory Stone Context ---",
+          formatted,
+          "--- End Memory Stone Context ---",
+        ].join("\n");
+      }
+
+      if (vaultCaptureNotice) {
+        return vaultCaptureReturn(systemPrompt, vaultCaptureNotice);
+      }
+
+      return { systemPrompt };
     } catch (err) {
       console.error("[pi-memory-stone] before_agent_start handler error:", err);
     }