@cad0p/pi-napkin 0.2.4 → 0.3.0-20260521.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -19,6 +19,13 @@ Then install the pi-napkin extension:
19
19
  pi install npm:@cad0p/pi-napkin
20
20
  ```
21
21
 
22
+ ### Requirements
23
+
24
+ - **bash 4+** — the auto-distill wrapper uses bash arrays, `local -n`, and other bash-4 features. macOS ships bash 3.2 by default; install a newer one via `brew install bash` (the wrapper resolves bash via its `#!/usr/bin/env bash` shebang, so it picks up homebrew's bash if `brew` is on `PATH`).
25
+ - **`timeout(1)` from coreutils** — used to bound the agent's wall-clock budget (`distill.maxDurationMinutes`). Linux distros ship it as `timeout`; macOS ships it as `gtimeout` after `brew install coreutils`. The wrapper detects either binary and falls back fast with a helpful error if neither is present.
26
+ - **git 2.20+** — needed for `git worktree`, `merge-base --is-ancestor`, and `symbolic-ref --short HEAD`.
27
+ - **A `pi` configured with at least one model provider** — auto-distill spawns `pi -p` against the model in `distill.model.{provider,id}`. Manual `/distill` reuses the parent session's provider.
28
+
22
29
  <details>
23
30
  <summary>Pre-release / dev installs</summary>
24
31
 
@@ -206,7 +213,7 @@ Or edit `<vault>/.napkin/config.json` directly:
206
213
 
207
214
  When enabled, on session start the extension:
208
215
 
209
- 1. Runs `ensureVaultReadyForAutoDistill` on the vault (see below) — git-inits if needed and scaffolds `.gitignore` / `.gitattributes`.
216
+ 1. Runs `ensureVaultReadyForDistill` on the vault (see below) — git-inits if needed and scaffolds `.gitignore`.
210
217
  2. Sweeps stale distill worktrees left by crashed pi instances (`cleanupStaleWorktrees`).
211
218
  3. Arms a timer (`intervalMinutes`) that spawns a detached `pi -p` subprocess on tick.
212
219
  4. On shutdown (unless `distill.onShutdown` is false or the session file is already captured), spawns one final distill.
@@ -226,7 +233,7 @@ The distill subprocess runs a prompt that asks the model to:
226
233
  When you enable `distill.enabled: true` on a vault that isn't a git repo, pi-napkin auto-initializes it on the next session start:
227
234
 
228
235
  1. Runs `git init`.
229
- 2. Scaffolds `.gitignore` (excludes `.napkin/distill/` — the per-worktree session fork) and `.gitattributes` (routes `*.md` through the LLM merge driver). Distill worktrees themselves live outside the vault (see [Where worktrees live](#where-worktrees-live)), so `.gitignore` doesn't need to exclude them.
236
+ 2. Scaffolds `.gitignore` (excludes `.napkin/distill/` — the per-worktree session fork). Distill worktrees themselves live outside the vault (see [Where worktrees live](#where-worktrees-live)), so `.gitignore` doesn't need to exclude them. No `.gitattributes` is written — the agent-driven merge architecture has no driver to register.
230
237
  3. Commits everything as `napkin: initial vault commit (auto-distill setup)`.
231
238
  4. Notifies you once, with instructions to undo (`rm -rf <vault>/.git`) or opt out (`distill.enabled: false`).
232
239
 
@@ -250,23 +257,61 @@ Running multiple pi sessions against the same vault (for example, autonomous age
250
257
 
251
258
  ### Worktree per distill
252
259
 
253
- Each auto-distill invocation (interval fire or shutdown) creates its own temporary branch and worktree under `$XDG_CACHE_HOME/napkin-distill/<vault-hash>/<branch-suffix>/` (typically `~/.cache/napkin-distill/…`). pi runs at the **parent session's cwd** (not the worktree) so the system prompt's `Current working directory:` line stays byte-identical to the parent's, preserving prompt-cache hits. Vault writes from the agent's bash tool are routed back to the worktree by a per-distill napkin shim at `<worktree>/.napkin/distill/bin/napkin` that injects `--vault <worktree>` into every napkin invocation. The wrapper's own git operations (`git add` / `commit` / `merge --squash`) all run with `git -C <worktree>` against the isolated checkout. See [Where worktrees live](#where-worktrees-live) for why the worktree dir is external to the vault.
260
+ Each auto-distill invocation (interval fire or shutdown) creates its own temporary branch and worktree under `$XDG_CACHE_HOME/napkin-distill/<vault-hash>/<branch-suffix>/` (typically `~/.cache/napkin-distill/…`). pi runs at the **parent session's cwd** (not the worktree) so the system prompt's `Current working directory:` line stays byte-identical to the parent's, preserving prompt-cache hits. Vault writes from the agent's bash tool are routed back to the worktree by a per-distill napkin shim at `<worktree>/.napkin/distill/bin/napkin` that injects `--vault <worktree>` into every napkin invocation. See [Where worktrees live](#where-worktrees-live) for why the worktree dir is external to the vault.
261
+
262
+ ### How distill resolves conflicts
263
+
264
+ The distill agent owns the full lifecycle: produce content → merge default branch into the distill branch → squash to default → push to origin (if configured) → clean up. The wrapper does NOT invoke a per-file merge driver; the model that wrote the content also resolves any conflicts that surface when the worktree is reconciled with main.
254
265
 
255
- ### LLM merge driver
266
+ Flow:
256
267
 
257
- When the distill commits its changes and merges back to main, conflicts are resolved by a custom git merge driver (`.napkin-distill-merge`) that calls the same model as the distill itself. The driver is registered per-worktree via `git config --local` and routed at the file level by an `*.md merge=napkin-distill-merge` entry in `.gitattributes`.
268
+ 1. **Wrapper** sets up the worktree, installs the per-distill napkin shim, and spawns one `pi -p $PROMPT` invocation under `timeout ${maxDurationMinutes}m`. The prompt instructs the agent to walk steps 1–10 (distill → merge → squash → push → cleanup).
269
+ 2. **Agent** distills content into the worktree, then runs `git -C <worktree> merge --no-edit <default>`. If conflicts surface, it edits each file in place using the conversation history as context.
270
+ 3. **Agent** squash-merges into the vault's default branch (`git -C <vault> merge --squash <distill-branch>` then `git commit`).
271
+ 4. **Agent** pushes to `origin/<default>` if origin is configured. On non-fast-forward failures it recovers via `git pull --no-rebase origin <default>` then re-pushes. It NEVER uses `--force` or `--force-with-lease`.
272
+ 5. **Wrapper** post-validates: no conflict markers in tracked `*.md`, vault HEAD on default branch, push (if attempted) didn't rewrite shared history. Writes an outcome sidecar and force-cleans the worktree + distill branch.
258
273
 
259
- The driver makes three attempts per conflict. After three strikes, the driver exits 1 and git leaves conflict markers — handled by the next step.
274
+ The full prompt template lives at `extensions/distill/distill-prompt.md`. The wrapper bounds the agent's wall-clock with `timeout(1)`; everything else (retry policy, network handling, conflict-resolution shape) is the agent's call.
260
275
 
261
- ### Partial-merge salvage
276
+ ### Outcome classes
262
277
 
263
- If the LLM merge driver fails on some files, we don't abort the whole merge. Files that merged cleanly keep the distill's content; files that failed fall back to main's version via `git checkout main -- <file>`.
278
+ Each distill produces a sidecar at `<vault>/.napkin/distill/errors/<timestamp>-<pid>-<branch>.outcome` whose first line is a machine-readable class:
264
279
 
265
- Think of it as: "save what you can, log what you lost." A per-failure entry is written to `<vault>/.napkin/distill/errors/` so you can inspect what was discarded, and the dangling distill commit SHA is kept for ~2 weeks of git gc grace in case you want to recover it manually.
280
+ | Class | When | UI severity |
281
+ |---|---|---|
282
+ | `merged-content` | Agent produced content, integrated, squashed, and pushed to origin (or origin not configured). | info ✓ |
283
+ | `merged-local` | Agent integrated + squashed, but origin/`<default>` is configured AND local `<default>` is ahead of `origin/<default>` (push didn't land). | warning ⚠ — "distilled but not pushed" |
284
+ | `no-content` | Agent produced nothing (genuine no-op — selective filter), or committed to the distill branch but skipped squash (default branch never moved). | warning ⚠ |
285
+ | `failed:<reason>` | Wrapper validation rejected the agent's output, or the agent didn't complete. | error ✗ |
286
+
287
+ Known `failed:<reason>` codes:
288
+
289
+ | Reason | Meaning | Recovery |
290
+ |---|---|---|
291
+ | `markers-after-agent-exit` | Conflict markers (`<<<<<<<` / `=======` / `>>>>>>>`) found in tracked `*.md` files that the agent did not have at distill start. The agent left an unfinished merge. | The squash commit may already be on `<default>`. Inspect with `git show HEAD`, then `git revert HEAD --no-edit` to undo cleanly. The dangling distill branch is recoverable from `git reflog` for ~2 weeks. |
292
+ | `pre-existing-markers` | Markers were present in the same files BEFORE the agent ran. Validation refuses to misattribute them. | Resolve the pre-existing markers in the vault yourself, then re-run distill. |
293
+ | `internal-validator-error` | The wrapper's post-distill marker validator could NOT run (e.g. `mktemp` failed because of a full disk or locked-down `TMPDIR`), so the vault was never scanned after the agent exited. | Inspect the vault manually for unresolved `<<<<<<< / ======= / >>>>>>>` markers before relying on the squash commit. If clean, the squash is keepable; otherwise `git -C <vault> revert HEAD --no-edit`. Distill content is recoverable from `git reflog` for ~90 days. |
294
+ | `head-not-on-default` | Vault HEAD is not on the default branch after the agent exited (detached HEAD or different branch). | Manually `git checkout <default>` after confirming nothing is in flight. The distill branch lives in `git reflog` if you want to recover its content. |
295
+ | `divergent-history` | After the agent's push, `origin/<default>` and local `<default>` diverged in a way that doesn't look like a fast-forward (unexpected; benign third-party push or — defensively — a force-push). | Inspect `origin/<default>` vs local. If a teammate landed a commit during distill, `git pull --no-rebase` to integrate. |
296
+ | `agent-exit-nonzero` | The agent's `pi -p` invocation exited non-zero with no other diagnostic. | See the error log next to the sidecar for the agent's stderr. |
297
+ | `agent-timeout` | `timeout(1)` killed the agent after `distill.maxDurationMinutes` minutes. | Bump `distill.maxDurationMinutes` if your conversations routinely produce long distills, or check the error log for what the agent was doing when it timed out. |
298
+
299
+ A missing sidecar AND missing error log means the wrapper itself was killed before writing either (SIGKILL, OOM). The JS-side poller surfaces this as "distill terminated abnormally".
300
+
301
+ ### Salvage when validation fails
302
+
303
+ The salvage path is deliberately narrow: the wrapper janitors the **worktree and distill branch**, never the main vault's commit history. If validation fails:
304
+
305
+ - Force-remove the worktree (`git worktree remove --force` + path-safety guard, falling back to `rm -rf` only if the worktree path is a descendant of the resolved cache root).
306
+ - Force-delete the distill branch (`git branch -D`).
307
+ - Write a `failed:<reason>` sidecar with a recovery hint that points at the corrupt squash commit if one landed on `<default>`.
308
+ - Exit 1.
309
+
310
+ The wrapper deliberately does NOT `git reset --hard` or otherwise rewrite the vault's commit history. If the agent's squash landed corrupt content, the user's `git revert HEAD --no-edit` (forward-only) is preferred over a destructive reset that could clobber concurrent edits the user made between distill spawn and validation. The recovery hint in the failed sidecar names the exact command.
266
311
 
267
312
  ### Linear history
268
313
 
269
- Successful distill lifecycles produce exactly one squash commit on `main`, with an LLM-generated summary message. This keeps history clean and makes it easy to revert a bad distill with `git revert <sha>`.
314
+ Successful distill lifecycles produce exactly one squash commit on `main`, with a one-line summary the agent generates from the distill content. This keeps history clean and makes it easy to revert a bad distill with `git revert <sha>`.
270
315
 
271
316
  ## Agent visibility
272
317
 
@@ -308,7 +353,7 @@ If you want auto-distill on an existing Obsidian vault:
308
353
  2. Start a pi session in the vault.
309
354
  3. On session start, pi-napkin prompts to auto-init git if the vault isn't already one.
310
355
 
311
- If your vault IS already a git repo (common for "vault as project"), pi-napkin just adds the `.gitignore` / `.gitattributes` scaffolds and moves on.
356
+ If your vault IS already a git repo (common for "vault as project"), pi-napkin just adds the `.gitignore` scaffold and moves on.
312
357
 
313
358
  ## Troubleshooting
314
359
 
@@ -331,15 +376,18 @@ rm -rf ~/.cache/napkin-distill/
331
376
 
332
377
  Safe — anything valuable is either already committed to main or was never going to commit.
333
378
 
334
- ### Merge driver fails repeatedly
379
+ ### Distill keeps failing (`failed:<reason>`)
380
+
381
+ Outcome sidecar: `<vault>/.napkin/distill/errors/<ISO-timestamp>-<pid>-<branch-hash>.outcome`. First line is the class string (`failed:<reason>`); remaining lines are a human-readable recovery hint.
335
382
 
336
- Error log: `<vault>/.napkin/distill/errors/<ISO-timestamp>-<pid>-<branch-hash>.log`.
383
+ Companion error log (if produced): `<ISO-timestamp>-<pid>-<branch-hash>.log` — wrapper diagnostics + the agent's stderr.
337
384
 
338
- Each log has the conflicted files, the last LLM response, and an optional one-liner describing the failure. Common causes:
385
+ See the [Outcome classes](#outcome-classes) table for what each `<reason>` means and the recommended recovery action. Common patterns:
339
386
 
340
- - Model is rate-limited → retry later.
341
- - LLM returned empty / tiny / huge output → driver sanity-check rejects it. Usually a transient provider issue.
342
- - Conflicted content is genuinely unmergeable (same line added with different content) → the partial-merge salvage falls back to main's version; the distill commit SHA is kept in reflog for ~2 weeks if you want to recover it.
387
+ - `agent-timeout` recurring → bump `distill.maxDurationMinutes` (default 10) or investigate what's taking the agent so long. Log shows the last tool calls before SIGTERM.
388
+ - `markers-after-agent-exit` → the agent's squash commit may already be on `<default>` with literal `<<<<<<<` markers in tracked files. The recovery hint in the sidecar names the exact `git revert HEAD --no-edit` command.
389
+ - `agent-exit-nonzero` → check the model provider (rate limits, auth refresh, network). The agent's stderr is in the `.log` file.
390
+ - `divergent-history` → a teammate likely landed a commit on `origin/<default>` while the agent's distill ran. `git pull --no-rebase` to integrate.
343
391
 
344
392
  ### "vault not a git repo" / "legacy embedded layout"
345
393
 
@@ -356,21 +404,47 @@ The distill wrapper (`extensions/distill/scripts/distill-wrapper.sh`) reads seve
356
404
 
357
405
  | Variable | Purpose |
358
406
  |---|---|
359
- | `NAPKIN_DISTILL_PI_BIN` | Override the `pi` binary path. Tests point this at `/usr/bin/true` (exits 0 quickly) or a stub script so the wrapper completes its lifecycle without contacting a real LLM. |
360
- | `NAPKIN_DISTILL_MERGE_MOCK` | Forwarded to `napkin-distill-merge` (the LLM driver). Set to `fail` to force every conflict to 3-strike, or `ok-after-N` to succeed after N attempts. Used to exercise the partial-merge salvage path. |
361
- | `NAPKIN_DISTILL_SKIP_PI=1` | Skip BOTH the napkin shim install AND the `pi` invocation. Used by tests that pre-stage file changes manually and only want to exercise the wrapper's git lifecycle (commit/merge/squash). |
407
+ | `NAPKIN_DISTILL_PI_BIN` | Override the `pi` binary path. Integration tests under `extensions/distill/test-fixtures/agent-stubs/` point this at a bash stub that simulates a specific agent behavior class (clean-distill, conflict-resolve-clean, agent-timeout, etc.) so the wrapper completes its lifecycle without contacting a real LLM. |
408
+ | `NAPKIN_DISTILL_SKIP_PI=1` | Skip BOTH the napkin shim install AND the `pi` invocation. Used by tests that pre-stage file changes manually and only want to exercise the wrapper's lifecycle (validation, salvage, sidecar emission). |
409
+ | `NAPKIN_DISTILL_NO_RECURSE=1` | Exported by the wrapper into the agent's environment so a nested `pi` won't auto-distill recursively. Tests sometimes set it directly to suppress recursion when invoking the wrapper from inside another distill. |
362
410
  | `NAPKIN_DISTILL_HALT_AFTER_META=1` | Halt right after rewriting `meta.json`'s pid to the wrapper's pid. Lets tests inspect the updated meta without the cleanup trap wiping the worktree. Clears the `EXIT` trap so cleanup is skipped — caller is responsible for tearing down the worktree afterwards. |
363
411
  | `NAPKIN_DISTILL_HALT_AFTER_SHIM=1` | Halt right after the per-distill napkin shim is installed at `<worktree>/.napkin/distill/bin/napkin`. Lets tests inspect the shim contents and PATH injection without the cleanup trap firing. |
364
- | `NAPKIN_DISTILL_FORCE_MERGE_HEAD=1` | Force `MERGE_HEAD` to exist right before the escape-hatch check. Lets tests cover the belt-and-braces "merge did not complete" bail-out path, which isn't reliably triggerable via real driver output (git clears `MERGE_HEAD` on driver exit 0). |
365
- | `NAPKIN_DISTILL_FORCE_MERGE_RC=<n>` | Override the captured merge exit code. Lets tests cover the unexpected-exit-code branch (128 etc.) without contriving a real git failure of that shape. |
412
+ | `NAPKIN_DISTILL_FORCE_CLEANUP=1` | Force the salvage / cleanup path to run unconditionally (even on success). Used to test salvage idempotency. |
413
+ | `NAPKIN_DISTILL_TIMEOUT_KILL_GRACE_SECS=<n>` | Override the `timeout(1) -k` grace window (default 30s) — the delay between SIGTERM and SIGKILL when the agent exceeds `distill.maxDurationMinutes`. Used to keep timeout tests fast. |
366
414
 
367
415
  Production never sets any of these. If you find one in your environment by accident, `unset` it and re-run — the wrapper falls back to its normal behaviour automatically.
368
416
 
369
- ## Future: builder-deleter
417
+ ## Migration from PR #11
370
418
 
371
- Next major feature: a "builder-deleter" janitor that acts on the `supersedes:` frontmatter convention that auto-distill already writes. When a note lists `supersedes: ["old/note.md"]`, the janitor archives the superseded file. Threshold-triggered to avoid running on every distill, git gc as the safety net.
419
+ pi-napkin v0.1.x shipped an LLM-backed git merge driver (`.napkin-distill-merge`). PR #12 (v0.2.x) deleted the driver entirely; the agent now resolves its own conflicts as part of the distill task. New vaults set up by v0.2+ never see the driver. Existing v0.1.x vaults retain inert fragments that are safe to leave in place but cleaner to remove:
420
+
421
+ ```bash
422
+ # In each existing vault that ran v0.1.x distill:
423
+ git config --local --remove-section merge.napkin-distill-merge 2>/dev/null || true
424
+
425
+ # Edit <vault>/.gitattributes and remove this line if present:
426
+ # *.md merge=napkin-distill-merge
427
+ ```
428
+
429
+ Why manual: PR #12 deliberately avoids automatic migration. The orphaned `.gitattributes` rule falls back to git's built-in merge driver once the script is gone (the rule becomes inert, not harmful), so the cost of automating cleanup outweighs the benefit for a project with a small user base. New vaults aren't affected.
430
+
431
+ ## Maintenance
432
+
433
+ ### Verifying the distill flow end-to-end against a real LLM
434
+
435
+ CI uses bash-stub fixtures (`extensions/distill/test-fixtures/agent-stubs/`) to cover the agent-behavior space without burning tokens. For ad-hoc re-validation that the full runtime — wrapper subprocess + agent + JS-side polling — still walks cleanly through a real model, run:
436
+
437
+ ```bash
438
+ bun run verify:e2e
439
+ ```
440
+
441
+ The script (`scripts/verify-e2e.ts`) creates a tmpdir vault with a bare-repo origin, registers the distill extension against a captured mock `ExtensionAPI`, triggers the `/distill` command handler so the production `runDistillWith` path runs (real wrapper subprocess, real 2-second `setInterval` poller), and asserts on the dispatched UI notification's severity + message together with filesystem post-conditions: no conflict markers, HEAD on default branch, agent's squash commit landed, distill branch removed, worktree removed, outcome sidecar with class `merged-content`, and origin advanced. Exits 0 on PASS, 1 on FAIL. Manual-only — not in CI; cost is roughly $0.50 per run.
442
+
443
+ This replaces the earlier prompt-only gate. The strict superset matters because the wrapper↔JS-poller seam — where worktree-teardown and outcome-write race — is invisible to a prompt-only harness.
444
+
445
+ ## Future: builder-deleter
372
446
 
373
- See [features/pi-napkin-distill/builder-deleter](https://github.com/cad0p/pi-napkin/blob/main/features/) (design pending) for the full design.
447
+ Next major feature: a "builder-deleter" janitor that acts on the `supersedes:` frontmatter convention that auto-distill already writes. When a note lists `supersedes: ["old/note.md"]`, the janitor archives the superseded file. Threshold-triggered to avoid running on every distill, git gc as the safety net. Design pending.
374
448
 
375
449
  ## License
376
450