@cad0p/pi-napkin 0.2.4 → 0.3.0-20260521.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +99 -25
- package/extensions/distill/auto-setup.ts +516 -178
- package/extensions/distill/distill-prompt.md +58 -0
- package/extensions/distill/distill-prompt.ts +212 -0
- package/extensions/distill/distill-workspace.ts +115 -211
- package/extensions/distill/health-notify.ts +98 -0
- package/extensions/distill/index.ts +239 -152
- package/extensions/distill/scripts/distill-wrapper.sh +1080 -300
- package/extensions/distill/scripts-paths.ts +0 -9
- package/package.json +5 -2
- package/extensions/distill/scripts/git_retry.sh +0 -65
- package/extensions/distill/scripts/napkin-distill-merge +0 -455
package/README.md
CHANGED
|
@@ -19,6 +19,13 @@ Then install the pi-napkin extension:
|
|
|
19
19
|
pi install npm:@cad0p/pi-napkin
|
|
20
20
|
```
|
|
21
21
|
|
|
22
|
+
### Requirements
|
|
23
|
+
|
|
24
|
+
- **bash 4+** — the auto-distill wrapper uses bash arrays, `local -n`, and other bash-4 features. macOS ships bash 3.2 by default; install a newer one via `brew install bash` (the wrapper resolves bash via its `#!/usr/bin/env bash` shebang, so it picks up homebrew's bash if `brew` is on `PATH`).
|
|
25
|
+
- **`timeout(1)` from coreutils** — used to bound the agent's wall-clock budget (`distill.maxDurationMinutes`). Linux distros ship it as `timeout`; macOS ships it as `gtimeout` after `brew install coreutils`. The wrapper detects either binary and falls back fast with a helpful error if neither is present.
|
|
26
|
+
- **git 2.20+** — needed for `git worktree`, `merge-base --is-ancestor`, and `symbolic-ref --short HEAD`.
|
|
27
|
+
- **A `pi` configured with at least one model provider** — auto-distill spawns `pi -p` against the model in `distill.model.{provider,id}`. Manual `/distill` reuses the parent session's provider.
|
|
28
|
+
|
|
22
29
|
<details>
|
|
23
30
|
<summary>Pre-release / dev installs</summary>
|
|
24
31
|
|
|
@@ -206,7 +213,7 @@ Or edit `<vault>/.napkin/config.json` directly:
|
|
|
206
213
|
|
|
207
214
|
When enabled, on session start the extension:
|
|
208
215
|
|
|
209
|
-
1. Runs `
|
|
216
|
+
1. Runs `ensureVaultReadyForDistill` on the vault (see below) — git-inits if needed and scaffolds `.gitignore`.
|
|
210
217
|
2. Sweeps stale distill worktrees left by crashed pi instances (`cleanupStaleWorktrees`).
|
|
211
218
|
3. Arms a timer (`intervalMinutes`) that spawns a detached `pi -p` subprocess on tick.
|
|
212
219
|
4. On shutdown (unless `distill.onShutdown` is false or the session file is already captured), spawns one final distill.
|
|
@@ -226,7 +233,7 @@ The distill subprocess runs a prompt that asks the model to:
|
|
|
226
233
|
When you enable `distill.enabled: true` on a vault that isn't a git repo, pi-napkin auto-initializes it on the next session start:
|
|
227
234
|
|
|
228
235
|
1. Runs `git init`.
|
|
229
|
-
2. Scaffolds `.gitignore` (excludes `.napkin/distill/` — the per-worktree session fork)
|
|
236
|
+
2. Scaffolds `.gitignore` (excludes `.napkin/distill/` — the per-worktree session fork). Distill worktrees themselves live outside the vault (see [Where worktrees live](#where-worktrees-live)), so `.gitignore` doesn't need to exclude them. No `.gitattributes` is written — the agent-driven merge architecture has no driver to register.
|
|
230
237
|
3. Commits everything as `napkin: initial vault commit (auto-distill setup)`.
|
|
231
238
|
4. Notifies you once, with instructions to undo (`rm -rf <vault>/.git`) or opt out (`distill.enabled: false`).
|
|
232
239
|
|
|
@@ -250,23 +257,61 @@ Running multiple pi sessions against the same vault (for example, autonomous age
|
|
|
250
257
|
|
|
251
258
|
### Worktree per distill
|
|
252
259
|
|
|
253
|
-
Each auto-distill invocation (interval fire or shutdown) creates its own temporary branch and worktree under `$XDG_CACHE_HOME/napkin-distill/<vault-hash>/<branch-suffix>/` (typically `~/.cache/napkin-distill/…`). pi runs at the **parent session's cwd** (not the worktree) so the system prompt's `Current working directory:` line stays byte-identical to the parent's, preserving prompt-cache hits. Vault writes from the agent's bash tool are routed back to the worktree by a per-distill napkin shim at `<worktree>/.napkin/distill/bin/napkin` that injects `--vault <worktree>` into every napkin invocation.
|
|
260
|
+
Each auto-distill invocation (interval fire or shutdown) creates its own temporary branch and worktree under `$XDG_CACHE_HOME/napkin-distill/<vault-hash>/<branch-suffix>/` (typically `~/.cache/napkin-distill/…`). pi runs at the **parent session's cwd** (not the worktree) so the system prompt's `Current working directory:` line stays byte-identical to the parent's, preserving prompt-cache hits. Vault writes from the agent's bash tool are routed back to the worktree by a per-distill napkin shim at `<worktree>/.napkin/distill/bin/napkin` that injects `--vault <worktree>` into every napkin invocation. See [Where worktrees live](#where-worktrees-live) for why the worktree dir is external to the vault.
|
|
261
|
+
|
|
262
|
+
### How distill resolves conflicts
|
|
263
|
+
|
|
264
|
+
The distill agent owns the full lifecycle: produce content → merge default branch into the distill branch → squash to default → push to origin (if configured) → clean up. The wrapper does NOT invoke a per-file merge driver; the model that wrote the content also resolves any conflicts that surface when the worktree is reconciled with main.
|
|
254
265
|
|
|
255
|
-
|
|
266
|
+
Flow:
|
|
256
267
|
|
|
257
|
-
|
|
268
|
+
1. **Wrapper** sets up the worktree, installs the per-distill napkin shim, and spawns one `pi -p $PROMPT` invocation under `timeout ${maxDurationMinutes}m`. The prompt instructs the agent to walk steps 1–10 (distill → merge → squash → push → cleanup).
|
|
269
|
+
2. **Agent** distills content into the worktree, then runs `git -C <worktree> merge --no-edit <default>`. If conflicts surface, it edits each file in place using the conversation history as context.
|
|
270
|
+
3. **Agent** squash-merges into the vault's default branch (`git -C <vault> merge --squash <distill-branch>` then `git commit`).
|
|
271
|
+
4. **Agent** pushes to `origin/<default>` if origin is configured. On non-fast-forward failures it recovers via `git pull --no-rebase origin <default>` then re-pushes. It NEVER uses `--force` or `--force-with-lease`.
|
|
272
|
+
5. **Wrapper** post-validates: no conflict markers in tracked `*.md`, vault HEAD on default branch, push (if attempted) didn't rewrite shared history. Writes an outcome sidecar and force-cleans the worktree + distill branch.
|
|
258
273
|
|
|
259
|
-
The
|
|
274
|
+
The full prompt template lives at `extensions/distill/distill-prompt.md`. The wrapper bounds the agent's wall-clock with `timeout(1)`; everything else (retry policy, network handling, conflict-resolution shape) is the agent's call.
|
|
260
275
|
|
|
261
|
-
###
|
|
276
|
+
### Outcome classes
|
|
262
277
|
|
|
263
|
-
|
|
278
|
+
Each distill produces a sidecar at `<vault>/.napkin/distill/errors/<timestamp>-<pid>-<branch>.outcome` whose first line is a machine-readable class:
|
|
264
279
|
|
|
265
|
-
|
|
280
|
+
| Class | When | UI severity |
|
|
281
|
+
|---|---|---|
|
|
282
|
+
| `merged-content` | Agent produced content, integrated, squashed, and pushed to origin (or origin not configured). | info ✓ |
|
|
283
|
+
| `merged-local` | Agent integrated + squashed, but origin/`<default>` is configured AND local `<default>` is ahead of `origin/<default>` (push didn't land). | warning ⚠ — "distilled but not pushed" |
|
|
284
|
+
| `no-content` | Agent produced nothing (genuine no-op — selective filter), or committed to the distill branch but skipped squash (default branch never moved). | warning ⚠ |
|
|
285
|
+
| `failed:<reason>` | Wrapper validation rejected the agent's output, or the agent didn't complete. | error ✗ |
|
|
286
|
+
|
|
287
|
+
Known `failed:<reason>` codes:
|
|
288
|
+
|
|
289
|
+
| Reason | Meaning | Recovery |
|
|
290
|
+
|---|---|---|
|
|
291
|
+
| `markers-after-agent-exit` | Conflict markers (`<<<<<<<` / `=======` / `>>>>>>>`) found in tracked `*.md` files that the agent did not have at distill start. The agent left an unfinished merge. | The squash commit may already be on `<default>`. Inspect with `git show HEAD`, then `git revert HEAD --no-edit` to undo cleanly. The dangling distill branch is recoverable from `git reflog` for ~2 weeks. |
|
|
292
|
+
| `pre-existing-markers` | Markers were present in the same files BEFORE the agent ran. Validation refuses to misattribute them. | Resolve the pre-existing markers in the vault yourself, then re-run distill. |
|
|
293
|
+
| `internal-validator-error` | The wrapper's post-distill marker validator could NOT run (e.g. `mktemp` failed because of a full disk or locked-down `TMPDIR`), so the vault was never scanned after the agent exited. | Inspect the vault manually for unresolved `<<<<<<< / ======= / >>>>>>>` markers before relying on the squash commit. If clean, the squash is keepable; otherwise `git -C <vault> revert HEAD --no-edit`. Distill content is recoverable from `git reflog` for ~90 days. |
|
|
294
|
+
| `head-not-on-default` | Vault HEAD is not on the default branch after the agent exited (detached HEAD or different branch). | Manually `git checkout <default>` after confirming nothing is in flight. The distill branch lives in `git reflog` if you want to recover its content. |
|
|
295
|
+
| `divergent-history` | After the agent's push, `origin/<default>` and local `<default>` diverged in a way that doesn't look like a fast-forward (unexpected; benign third-party push or — defensively — a force-push). | Inspect `origin/<default>` vs local. If a teammate landed a commit during distill, `git pull --no-rebase` to integrate. |
|
|
296
|
+
| `agent-exit-nonzero` | The agent's `pi -p` invocation exited non-zero with no other diagnostic. | See the error log next to the sidecar for the agent's stderr. |
|
|
297
|
+
| `agent-timeout` | `timeout(1)` killed the agent after `distill.maxDurationMinutes` minutes. | Bump `distill.maxDurationMinutes` if your conversations routinely produce long distills, or check the error log for what the agent was doing when it timed out. |
|
|
298
|
+
|
|
299
|
+
A missing sidecar AND missing error log means the wrapper itself was killed before writing either (SIGKILL, OOM). The JS-side poller surfaces this as "distill terminated abnormally".
|
|
300
|
+
|
|
301
|
+
### Salvage when validation fails
|
|
302
|
+
|
|
303
|
+
The salvage path is deliberately narrow: the wrapper janitors the **worktree and distill branch**, never the main vault's commit history. If validation fails:
|
|
304
|
+
|
|
305
|
+
- Force-remove the worktree (`git worktree remove --force` + path-safety guard, falling back to `rm -rf` only if the worktree path is a descendant of the resolved cache root).
|
|
306
|
+
- Force-delete the distill branch (`git branch -D`).
|
|
307
|
+
- Write a `failed:<reason>` sidecar with a recovery hint that points at the corrupt squash commit if one landed on `<default>`.
|
|
308
|
+
- Exit 1.
|
|
309
|
+
|
|
310
|
+
The wrapper deliberately does NOT `git reset --hard` or otherwise rewrite the vault's commit history. If the agent's squash landed corrupt content, the user's `git revert HEAD --no-edit` (forward-only) is preferred over a destructive reset that could clobber concurrent edits the user made between distill spawn and validation. The recovery hint in the failed sidecar names the exact command.
|
|
266
311
|
|
|
267
312
|
### Linear history
|
|
268
313
|
|
|
269
|
-
Successful distill lifecycles produce exactly one squash commit on `main`, with
|
|
314
|
+
Successful distill lifecycles produce exactly one squash commit on `main`, with a one-line summary the agent generates from the distill content. This keeps history clean and makes it easy to revert a bad distill with `git revert <sha>`.
|
|
270
315
|
|
|
271
316
|
## Agent visibility
|
|
272
317
|
|
|
@@ -308,7 +353,7 @@ If you want auto-distill on an existing Obsidian vault:
|
|
|
308
353
|
2. Start a pi session in the vault.
|
|
309
354
|
3. On session start, pi-napkin prompts to auto-init git if the vault isn't already one.
|
|
310
355
|
|
|
311
|
-
If your vault IS already a git repo (common for "vault as project"), pi-napkin just adds the `.gitignore`
|
|
356
|
+
If your vault IS already a git repo (common for "vault as project"), pi-napkin just adds the `.gitignore` scaffold and moves on.
|
|
312
357
|
|
|
313
358
|
## Troubleshooting
|
|
314
359
|
|
|
@@ -331,15 +376,18 @@ rm -rf ~/.cache/napkin-distill/
|
|
|
331
376
|
|
|
332
377
|
Safe — anything valuable is either already committed to main or was never going to commit.
|
|
333
378
|
|
|
334
|
-
###
|
|
379
|
+
### Distill keeps failing (`failed:<reason>`)
|
|
380
|
+
|
|
381
|
+
Outcome sidecar: `<vault>/.napkin/distill/errors/<ISO-timestamp>-<pid>-<branch-hash>.outcome`. First line is the class string (`failed:<reason>`); remaining lines are a human-readable recovery hint.
|
|
335
382
|
|
|
336
|
-
|
|
383
|
+
Companion error log (if produced): `<ISO-timestamp>-<pid>-<branch-hash>.log` — wrapper diagnostics + the agent's stderr.
|
|
337
384
|
|
|
338
|
-
|
|
385
|
+
See the [Outcome classes](#outcome-classes) table for what each `<reason>` means and the recommended recovery action. Common patterns:
|
|
339
386
|
|
|
340
|
-
-
|
|
341
|
-
-
|
|
342
|
-
-
|
|
387
|
+
- `agent-timeout` recurring → bump `distill.maxDurationMinutes` (default 10) or investigate what's taking the agent so long. Log shows the last tool calls before SIGTERM.
|
|
388
|
+
- `markers-after-agent-exit` → the agent's squash commit may already be on `<default>` with literal `<<<<<<<` markers in tracked files. The recovery hint in the sidecar names the exact `git revert HEAD --no-edit` command.
|
|
389
|
+
- `agent-exit-nonzero` → check the model provider (rate limits, auth refresh, network). The agent's stderr is in the `.log` file.
|
|
390
|
+
- `divergent-history` → a teammate likely landed a commit on `origin/<default>` while the agent's distill ran. `git pull --no-rebase` to integrate.
|
|
343
391
|
|
|
344
392
|
### "vault not a git repo" / "legacy embedded layout"
|
|
345
393
|
|
|
@@ -356,21 +404,47 @@ The distill wrapper (`extensions/distill/scripts/distill-wrapper.sh`) reads seve
|
|
|
356
404
|
|
|
357
405
|
| Variable | Purpose |
|
|
358
406
|
|---|---|
|
|
359
|
-
| `NAPKIN_DISTILL_PI_BIN` | Override the `pi` binary path.
|
|
360
|
-
| `
|
|
361
|
-
| `
|
|
407
|
+
| `NAPKIN_DISTILL_PI_BIN` | Override the `pi` binary path. Integration tests under `extensions/distill/test-fixtures/agent-stubs/` point this at a bash stub that simulates a specific agent behavior class (clean-distill, conflict-resolve-clean, agent-timeout, etc.) so the wrapper completes its lifecycle without contacting a real LLM. |
|
|
408
|
+
| `NAPKIN_DISTILL_SKIP_PI=1` | Skip BOTH the napkin shim install AND the `pi` invocation. Used by tests that pre-stage file changes manually and only want to exercise the wrapper's lifecycle (validation, salvage, sidecar emission). |
|
|
409
|
+
| `NAPKIN_DISTILL_NO_RECURSE=1` | Exported by the wrapper into the agent's environment so a nested `pi` won't auto-distill recursively. Tests sometimes set it directly to suppress recursion when invoking the wrapper from inside another distill. |
|
|
362
410
|
| `NAPKIN_DISTILL_HALT_AFTER_META=1` | Halt right after rewriting `meta.json`'s pid to the wrapper's pid. Lets tests inspect the updated meta without the cleanup trap wiping the worktree. Clears the `EXIT` trap so cleanup is skipped — caller is responsible for tearing down the worktree afterwards. |
|
|
363
411
|
| `NAPKIN_DISTILL_HALT_AFTER_SHIM=1` | Halt right after the per-distill napkin shim is installed at `<worktree>/.napkin/distill/bin/napkin`. Lets tests inspect the shim contents and PATH injection without the cleanup trap firing. |
|
|
364
|
-
| `
|
|
365
|
-
| `
|
|
412
|
+
| `NAPKIN_DISTILL_FORCE_CLEANUP=1` | Force the salvage / cleanup path to run unconditionally (even on success). Used to test salvage idempotency. |
|
|
413
|
+
| `NAPKIN_DISTILL_TIMEOUT_KILL_GRACE_SECS=<n>` | Override the `timeout(1) -k` grace window (default 30s) — the delay between SIGTERM and SIGKILL when the agent exceeds `distill.maxDurationMinutes`. Used to keep timeout tests fast. |
|
|
366
414
|
|
|
367
415
|
Production never sets any of these. If you find one in your environment by accident, `unset` it and re-run — the wrapper falls back to its normal behaviour automatically.
|
|
368
416
|
|
|
369
|
-
##
|
|
417
|
+
## Migration from PR #11
|
|
370
418
|
|
|
371
|
-
|
|
419
|
+
pi-napkin v0.1.x shipped an LLM-backed git merge driver (`.napkin-distill-merge`). PR #12 (v0.2.x) deleted the driver entirely; the agent now resolves its own conflicts as part of the distill task. New vaults set up by v0.2+ never see the driver. Existing v0.1.x vaults retain inert fragments that are safe to leave in place but cleaner to remove:
|
|
420
|
+
|
|
421
|
+
```bash
|
|
422
|
+
# In each existing vault that ran v0.1.x distill:
|
|
423
|
+
git config --local --remove-section merge.napkin-distill-merge 2>/dev/null || true
|
|
424
|
+
|
|
425
|
+
# Edit <vault>/.gitattributes and remove this line if present:
|
|
426
|
+
# *.md merge=napkin-distill-merge
|
|
427
|
+
```
|
|
428
|
+
|
|
429
|
+
Why manual: PR #12 deliberately avoids automatic migration. The orphaned `.gitattributes` rule falls back to git's built-in merge driver once the script is gone (the rule becomes inert, not harmful), so the cost of automating cleanup outweighs the benefit for a project with a small user base. New vaults aren't affected.
|
|
430
|
+
|
|
431
|
+
## Maintenance
|
|
432
|
+
|
|
433
|
+
### Verifying the distill flow end-to-end against a real LLM
|
|
434
|
+
|
|
435
|
+
CI uses bash-stub fixtures (`extensions/distill/test-fixtures/agent-stubs/`) to cover the agent-behavior space without burning tokens. For ad-hoc re-validation that the full runtime — wrapper subprocess + agent + JS-side polling — still walks cleanly through a real model, run:
|
|
436
|
+
|
|
437
|
+
```bash
|
|
438
|
+
bun run verify:e2e
|
|
439
|
+
```
|
|
440
|
+
|
|
441
|
+
The script (`scripts/verify-e2e.ts`) creates a tmpdir vault with a bare-repo origin, registers the distill extension against a captured mock `ExtensionAPI`, triggers the `/distill` command handler so the production `runDistillWith` path runs (real wrapper subprocess, real 2-second `setInterval` poller), and asserts on the dispatched UI notification's severity + message together with filesystem post-conditions: no conflict markers, HEAD on default branch, agent's squash commit landed, distill branch removed, worktree removed, outcome sidecar with class `merged-content`, and origin advanced. Exits 0 on PASS, 1 on FAIL. Manual-only — not in CI; cost is roughly $0.50 per run.
|
|
442
|
+
|
|
443
|
+
This replaces the earlier prompt-only gate. The strict superset matters because the wrapper↔JS-poller seam — where worktree-teardown and outcome-write race — is invisible to a prompt-only harness.
|
|
444
|
+
|
|
445
|
+
## Future: builder-deleter
|
|
372
446
|
|
|
373
|
-
|
|
447
|
+
Next major feature: a "builder-deleter" janitor that acts on the `supersedes:` frontmatter convention that auto-distill already writes. When a note lists `supersedes: ["old/note.md"]`, the janitor archives the superseded file. Threshold-triggered to avoid running on every distill, git gc as the safety net. Design pending.
|
|
374
448
|
|
|
375
449
|
## License
|
|
376
450
|
|