pi-notes 0.1.0 → 0.1.1

package/BENCHMARK.md

@@ -0,0 +1,426 @@
+# Exploratory Notes A/B benchmark
+
+This document reports an exploratory long-horizon evaluation of `pi-notes`.
+It is intentionally more conservative than a product benchmark: the run found
+useful case evidence, but several harness and experimental-design defects mean
+that it does **not** identify a causal productivity or quality effect.
+
+## Executive summary
+
+- Harness: `harness-1789204569100-83132b78`
+- Run window: 2026-09-12 through 2026-09-13
+- Model: local `qwen38-main/qwen3.8-27b`, medium thinking
+- Scenarios: 12, each run once without Notes and once with Notes
+- Arms: 24 total; 23 exited normally and one Notes-present arm timed out
+- Completed pairs: 11; one pair was censored by the timeout
+- Primary conclusion: **hypothesis-generating evidence only**
+
+Across the 11 pairs in which both arms completed, Notes-present was faster in
+7 and slower in 4. The mean paired duration difference was -8.81 minutes and
+the median was -17.38 minutes. Those pooled figures combine two repositories,
+different implementations, and conditionally comparable HostelHawk runs; they
+must not be interpreted as an average Notes treatment effect.
+
+A read-only audit of the retained changes judged Notes-present stronger in six
+scenarios, Notes-absent stronger in two, mixed or inconclusive in three, and
+the timed-out Notes-present scenario unfinished. This audit was not blinded,
+did not execute an independent acceptance suite, and is separate from the
+telemetry results.
+
+The benchmark did establish several actionable facts:
+
+1. Notes was present and active only in the intended arm and was used in every
+   Notes-present scenario.
+2. Agents executed `checkpoint_notes` 55 times: 38 checkpoints committed and
+   17 calls were rejected for invalid payloads.
+3. Fifty provider generations, 2.1% of Notes-present requests, contained only
+   checkpoint work.
+4. All 41 observed compactions succeeded. Notes-present did not show lower
+   repetition or post-compaction rediscovery in this single-run sample.
+5. The current report pairing, dirty-start detection, verification
+   classification, and cache telemetry all require fixes before a controlled
+   rerun.
+
+## Research question
+
+The evaluation asked whether a compact, session-local durable handoff helps a
+local model complete long engineering tasks more efficiently and reliably,
+especially after compaction or a late change in requirements.
+
+The intervention was the Notes tool surface:
+
+- **Notes-absent:** `pi-notes` was omitted entirely. The arm did not use
+  `/notes off`, because that would leave the package and its policy in context.
+- **Notes-present:** `pi-notes` was loaded and active, including
+  `checkpoint_notes` and its static policy.
+
+Both arms otherwise used the same scenario prompt, model, thinking level,
+compaction fingerprint, target commit, and local-only execution policy.
+
+## Benchmark design
+
+Eight scenarios targeted the Records Dashboard baseline
+`696115bf11aaa58dd5a69e417d968bf778815c16`. Four targeted the HostelHawk
+baseline `615448114a0a951b349bbdf1a3b89cae2fabb05c`.
+
+The harness created a detached worktree and fresh Pi session for each arm. It
+ran all arms sequentially, always Notes-absent before Notes-present, with a
+three-hour execution timeout. The recorder captured bounded, sanitized events
+for requests, tools, verification-like commands, checkpoint activity,
+compaction, and token usage. Raw session transcripts and private reasoning are
+not part of this report.
+
+The 12 scenarios exercised:
+
+- durable CRUD workflows and exports;
+- refresh, retry, lifecycle, and operator-control behavior;
+- recorded-document and provenance workflows;
+- architecture investigation and late requirement changes;
+- failure recovery; and
+- implementation followed by independent review and remediation.
+
+## Evidence tiers
+
+The report keeps four kinds of evidence separate:
+
+1. **Run validity:** matching configuration, clean or conditional start,
+   complete shutdown, timeout, and recorder integrity.
+2. **Efficiency telemetry:** wall time, requests, tools, errors, and tokens.
+3. **Mechanism telemetry:** checkpointing, compaction, repetition, and
+   rediscovery.
+4. **Implementation evidence:** retained diff, requirement coverage, tests,
+   documentation, and final verification claims.
+
+Only the first three are machine-derived. Implementation verdicts are
+read-only artifact judgments and are not independent acceptance results.
+
+## Run disposition
+
+| Cohort | Pairs | Disposition |
+| --- | ---: | --- |
+| Clean Records | 7 | Both arms completed with clean starts |
+| Records provenance monitor | 1 | Notes-present timed out at three hours |
+| HostelHawk | 4 | Both arms completed; automated reports excluded them for `dirty-start` |
+
+Both source checkouts were recorded as `sourceDirty=true`, which means their
+main working directories contained uncommitted files. Those files were not
+copied into the detached evaluation worktrees. The separate per-run
+`dirtyStart` field was false for every Records arm and true for every
+HostelHawk arm.
+
+The HostelHawk `dirtyStart` exclusion is likely a harness artifact. The harness
+created an untracked `node_modules` symlink before the recorder ran
+`git status`. HostelHawk ignores `node_modules/` directories but not the bare
+symlink, so all eight HostelHawk arms received `dirtyStart=true`. Both arms of
+every pair still used the same commit and task hash.
+
+For that reason, HostelHawk is manually comparable as conditional evidence,
+but it is reported separately from the clean Records cohort.
+
+No arm emitted the recorder's `goal_progress({ status: "done" })` completion
+signal. That signal is optional and observational; its absence is not evidence
+that all 24 implementations failed.
+
+## Efficiency results
+
+All deltas below are Notes-present minus Notes-absent. Negative time, request,
+tool, and token deltas indicate less work in the Notes-present trajectory.
+
+### Clean Records cohort
+
+| Scenario | Duration | Turns | Tools | Tool errors |
+| --- | ---: | ---: | ---: | ---: |
+| Saved due-diligence case file | -17.38 min | -3 | -15 | 0 |
+| County-record refresh | -25.31 min | -83 | -60 | -11 |
+| Recorded-document search | +3.21 min | -4 | -13 | -3 |
+| Owner-request lifecycle | -28.29 min | -1 | +12 | +8 |
+| Agent-task operations | -33.96 min | -89 | -65 | -2 |
+| Screening comparison and retry | -7.04 min | +48 | +40 | +3 |
+| Investigation bundle export | +24.18 min | +18 | +32 | +10 |
+| **Mean paired delta** | **-12.09 min** | **-16.29** | **-9.86** | **+0.71** |
+| **Median paired delta** | **-17.38 min** | **-3** | **-13** | **0** |
+
+Notes-present was faster in five of seven clean completed pairs. Mean input
+tokens fell by 1.752 million and mean output tokens by 13,224 per pair. These
+means are sensitive to individual scenarios: agent-task operations alone
+contributed an 11.407 million input-token reduction.
+
+### Timed-out Records pair
+
+The Notes-present provenance-monitor arm reached the fixed three-hour limit
+and was terminated by the harness with exit code 143. It ended without a
+normal recorder shutdown or final report.
+
+Before termination it had made 321 provider requests and 349 tool calls,
+compared with 183 requests and 212 tools in Notes-absent. It consumed 32.327
+million input tokens versus 18.313 million. This is a reliability outcome and
+must not be dropped or converted into a completed duration delta.
+
+### Conditional HostelHawk cohort
+
+| Scenario | Duration | Turns | Tools | Tool errors |
+| --- | ---: | ---: | ---: | ---: |
+| Multi-phase architecture investigation | +9.87 min | +11 | +12 | +3 |
+| Late requirement change | -89.38 min | -75 | -89 | +2 |
+| Failure-recovery hardening | -33.43 min | -90 | -85 | -2 |
+| Implementation review and remediation | +100.63 min | +119 | +130 | +31 |
+| **Mean paired delta** | **-3.08 min** | **-8.75** | **-8.00** | **+8.50** |
+| **Median paired delta** | **-11.78 min** | **-32** | **-36.50** | **+2.50** |
+
+This cohort is dominated by opposing outliers. Removing the implementation
+review changes the mean duration delta from -3.08 to -37.65 minutes. The late
+requirement run was much faster with Notes but appears to have traded some
+original-scope completeness for deeper compatibility work. The implementation
+review delivered broader remediation with Notes but at 3.36 times the runtime.
+
+### Overall usage
+
+| Metric | Notes-absent | Notes-present |
+| --- | ---: | ---: |
+| Provider requests | 2,402 | 2,391 |
+| Tool calls | 2,802 | 2,838 |
+| Tool errors | 88 | 140 |
+| Input tokens | 231,331,864 | 229,848,996 |
+| Output tokens | 1,895,579 | 1,839,948 |
+| Reported total tokens | 233,227,443 | 231,688,944 |
+
+The combined run reported 464,916,387 logical tokens. Output already includes
+reasoning tokens and must not be double-counted.
+
+Cache-read and cache-write fields were zero in every arm because the active
+local server did not expose prompt-token detail telemetry. Zero therefore
+means **unavailable telemetry**, not demonstrated absence of prefix caching.
+Exact cached-versus-uncached usage and equivalent billing cannot be recovered
+for this run.
+
+## Notes mechanism results
+
+### Checkpoint execution
+
+Notes-absent made no checkpoint calls. Every Notes-present scenario used the
+tool.
+
+| Scenario | Executions | Committed | Invalid |
+| --- | ---: | ---: | ---: |
+| Saved due-diligence case file | 5 | 3 | 2 |
+| County-record refresh | 4 | 4 | 0 |
+| Recorded-document search | 4 | 3 | 1 |
+| Owner-request lifecycle | 4 | 3 | 1 |
+| Agent-task operations | 4 | 3 | 1 |
+| Screening comparison and retry | 6 | 6 | 0 |
+| Provenance drift monitor | 13 | 8 | 5 |
+| Investigation bundle export | 2 | 1 | 1 |
+| Multi-phase architecture investigation | 3 | 3 | 0 |
+| Late requirement change | 2 | 2 | 0 |
+| Failure-recovery hardening | 6 | 1 | 5 |
+| Implementation review and remediation | 2 | 1 | 1 |
+| **Total** | **55** | **38** | **17** |
+
+The 17 rejected calls were malformed payloads, usually a list field supplied
+as a string or a required field omitted. This 30.9% invalid-call rate is a
+tool-usability defect worth addressing before measuring checkpoint benefit.
+
+The evaluator's `checkpoint_attempt` hook saw only the 38 schema-valid calls.
+Calls rejected by generic tool validation never reached that hook. Generated
+reports therefore showed zero checkpoint failures even though 17 executions
+failed. Future telemetry must count tool execution and validation rejection,
+not only accepted checkpoint attempts.
+
+Fifty Notes-present generations contained only checkpoint work, or 2.09 per
+100 provider requests. There were no ambient `notes_reminder` events in either
+arm, so the observed checkpoint overhead was not reminder-driven.
+
+### Compaction and continuity
+
+| Metric | Notes-absent | Notes-present |
+| --- | ---: | ---: |
+| Compaction attempts | 21 | 20 |
+| Successful compactions | 21 | 20 |
+| Failed compactions | 0 | 0 |
+| Mean recorded recovery | 69.49 s | 62.82 s |
+| Repeated tool calls | 44 (1.83/100 requests) | 69 (2.89/100) |
+| Post-compaction rediscoveries | 40 (1.67/100) | 49 (2.05/100) |
+
+The clean Records pairs averaged exactly the same number of compactions in
+both arms. Notes-present had slightly faster recorded recovery overall, but
+more repetition and rediscovery. Scenario trajectories and compaction timing
+differed, so these observations do not establish mediation or a Notes effect.
+
+The implementation-review Notes-present arm had one threshold compaction. It
+succeeded and recorded recovery after 43 seconds. A preliminary audit that
+listed this as a failure was checked against the event stream and corrected.
+
+## Implementation evidence
+
+The following verdicts come from a read-only comparison of prompts, retained
+worktree changes, and recorded verification evidence. They are useful case
+studies, but they were not blinded and no independent evaluator reran the
+acceptance criteria.
+
+| Scenario | Artifact verdict | Evidence summary |
+| --- | --- | --- |
+| Saved case file | Notes-absent stronger | Clearer deterministic export and richer case-file contract; Notes-present had broader UI but less visible findings/export coverage. |
+| County refresh | Notes-present stronger | More focused worker/UI coverage with durable lifecycle and compare-and-swap semantics; login redirect still dropped the original target. |
+| Recorded-document search | Mixed | Notes-present added useful revision and polling behavior, but lacked a final completion report and UI E2E remained auth-blocked. |
+| Owner-request lifecycle | Notes-present stronger | Added terminal send-failure handling, bounded budgets, cancellation, key scoping, UI, and broader integration coverage. |
+| Agent-task operations | Notes-present stronger | Stronger operator authorization, privacy projection, audit trail, proxy gating, pagination, and UI; live browser authorization remained unverified. |
+| Screening comparison/retry | Mixed, slightly favoring Notes-present | Added current-state policy checks and comparison UI, but both arms implemented different contracts and Notes-present E2E self-skipped. |
+| Provenance drift monitor | Notes-absent stronger | Notes-present produced a larger unfinished change and timed out without final verification. |
+| Investigation bundle export | Notes-present stronger | Coherent durable artifact/export path and explicit route-context remediation; complete integration evidence was limited by emulator contention. |
+| Multi-phase architecture | Notes-absent stronger | Both were partial; Notes-absent retained broader database, scraper, cache, and integration coverage. |
+| Late requirement change | Mixed | Notes-present targeted legacy compatibility well but had less visible evidence for the original full CRUD/UI scope. |
+| Failure recovery | Notes-present stronger | Smaller, explicit failed-alert-delivery contract with focused transition and SQL tests; no live database run was available. |
+| Implementation review | Notes-present stronger at disproportionate cost | Added quota enforcement, row locking, conflict behavior, concurrency evidence, and remediation, but incurred extensive retry and diagnostic churn. |
+
+Conservative summary:
+
+- Notes-present stronger: 6 scenarios
+- Notes-absent stronger: 2 scenarios
+- Mixed or inconclusive: 3 scenarios
+- Notes-present unfinished due to timeout: 1 scenario
+
+Speed and artifact quality did not move together. Of the six scenarios judged
+stronger with Notes, four were faster and two were slower. One of the two
+Notes-weaker outcomes was nevertheless faster. This reinforces that telemetry
+cannot substitute for an independent quality evaluator.
+
+## Implementation-review outlier
+
+The largest regression was implementation review and remediation:
+
+- duration: +100.63 minutes;
+- turns: +119;
+- tools: +130;
+- tool errors: +31;
+- input tokens: +12.875 million;
+- repeated tool calls: +22; and
+- post-compaction rediscoveries: +9.
+
+The extra work was not only empty churn. The Notes-present result added a
+per-account favorites quota, transactional row locking, conflict behavior,
+broader server/component tests, and an explicit audit that found and remediated
+two concrete gaps.
+
+The quality gain was not proportional to the cost. The trajectory included
+three attempted subagent launches—two blocked before launch and one started in
+the background—followed by three premature result checks that all found the
+agent still running. The telemetry does not isolate a defensible subagent wait
+cost. It also included 13 failed edit operations and repeated exact-pattern
+repair scripts. Nineteen of the 22 events labeled `test:error` were Python
+`pattern not found` assertions from edit/check scripts. The other three were
+inspection or chained shell commands that were also classified lexically;
+none establishes that a product test suite ran and failed. One invalid
+checkpoint call and one successful compaction added smaller overhead.
+
+The supported interpretation is that Notes-present followed a broader,
+more exhaustive remediation path and then executed it inefficiently. The
+single trajectory cannot determine whether Notes caused either the additional
+quality work or the churn.
+
+## Measurement defects and limitations
+
+### Confirmed harness defects
+
+1. **Cross-worktree repository identity broke pairing.** The experiment key
+   hashes the resolved worktree root. Each arm therefore received a different
+   experiment key despite matching commits. Scenario reporting selected the
+   first arm's key and silently omitted the second, producing zero generated
+   pairs. This report reconstructed pairs manually by scenario and arm.
+2. **Dirty-start detection included setup artifacts.** The recorder captured
+   Git status after the harness created dependency and environment symlinks.
+   This excluded all HostelHawk arms for an untracked dependency symlink.
+3. **Checkpoint failures were undercounted.** Schema-invalid calls failed
+   before `checkpoint_attempt`, so the report missed 17 failed executions.
+4. **Verification classification was lexical.** Commands containing words
+   such as `test`, `check`, or `build` were counted as verification even when
+   they were search or edit scripts. A non-error tool result was generally
+   treated as success without validating a prescribed gate.
+5. **General tool errors were omitted from pair reports.** Live activity had
+   them, but `ReportRow` and `PairDelta` did not.
+6. **Missing cache telemetry was coerced to zero.** The report had no
+   availability indicator and could not distinguish missing data from a real
+   zero.
+
+### Experimental limitations
+
+- Only one stochastic replicate was run per arm and scenario.
+- Notes-absent always ran first, confounding Notes with cache warming, server
+  state, filesystem state, and time.
+- The two arms could choose materially different implementations.
+- Shared dependency and environment symlinks weakened isolation.
+- One Notes-present arm timed out and must be treated as a failure or censored
+  observation, not silently excluded.
+- The agent wrote its own tests and reported its own completion. There was no
+  predeclared, independent correctness oracle.
+- Runtime identity omitted Pi version, full extension/tool inventory,
+  model-server build, sampling settings, dependency snapshot, and external
+  service state.
+- Harness wall time and evaluator session time began at different points.
+
+These limitations prevent significance tests, confidence intervals, pooled
+event-level regressions, or a causal estimate from this run. Turns and tool
+events are not independent samples. Trimming the timeout or the two large
+HostelHawk outliers would be post hoc and misleading.
+
+## Interpretation
+
+Publication-safe conclusion:
+
+> Across 12 heterogeneous long-horizon engineering scenarios, the harness
+> generated case evidence that agents actively used durable Notes and, in
+> several cases, produced stronger retained implementations with fewer turns
+> or less time. Other cases showed substantial checkpoint validation failures,
+> increased repetition, a timeout, and one severe efficiency regression.
+> Because assignment was sequential, replication was single-run,
+> implementations differed, and telemetry/pairing defects affected the
+> record, the run does not identify a causal productivity, reliability, or
+> quality effect of `pi-notes`.
+
+The practical value of this run is diagnostic. It identified scenarios worth
+repeating, exposed tool-schema and harness defects, and produced concrete
+hypotheses:
+
+- concise checkpoints may help some agents preserve task structure and reduce
+  search or implementation work;
+- checkpoint-only turns and malformed payloads can erase those gains;
+- Notes may encourage broader review behavior in some trajectories; and
+- continuity benefit should be evaluated with objective post-run quality, not
+  inferred from compaction or self-reported verification.
+
+## Required rerun
+
+Before making a product claim:
+
+1. Use a canonical repository identity and explicit
+   `scenario × replicate × pair` identifiers.
+2. Capture baseline cleanliness before injecting setup artifacts, and record
+   setup-generated changes separately.
+3. Randomize or counterbalance arm order within each scenario and replicate.
+4. Run at least five paired replicates per scenario; use more for timeout-prone
+   or high-variance tasks.
+5. Pin and hash Pi, extension revisions, tool surface, system prompt,
+   dependencies, model-server build, model checkpoint, sampling parameters,
+   context settings, and external-service state.
+6. Use isolated immutable dependencies, environment files, databases, and
+   server/cache state per arm.
+7. Predeclare timeouts, exclusions, primary outcomes, and scenario-specific
+   acceptance tests. Count timeouts as outcomes.
+8. Run an independent evaluator against each retained diff in a fresh
+   worktree. Use blinded reviewers and a fixed quality rubric.
+9. Record structured command identity, exit code, signal, timeout, and
+   framework result instead of lexical verification classification.
+10. Record cache-telemetry availability, cached and uncached input, reasoning
+    output, retry attempts, adaptive reasoning changes, and tool-error classes.
+11. Count malformed checkpoint executions and test a simpler schema or better
+    corrective guidance before measuring checkpoint reliability.
+12. Report paired per-scenario effects, ranges, missingness, and repository
+    strata. Do not pool events as independent observations.
+
+## Artifact policy
+
+The durable report records the harness ID, baseline commits, scenario names,
+and derived measurements. Generated manifests, event streams, worktree diffs,
+and session logs remain local evaluation artifacts. Machine-local absolute
+paths and raw transcripts are deliberately omitted because they are not
+portable and may contain prompts, tool arguments, local paths, or private
+reasoning.

package/README.md

@@ -63,6 +63,10 @@ It does not call `pi.setActiveTools()`. If another mode hides `checkpoint_notes`
 
 Lifecycle tests cover fresh identities, branch restoration, resume rematerialization, deferred read/research freshness, read-only threshold and streak reset, delayed checkpoint pressure, compaction pressure, inherited-resume integrity, and external-mutation gating.
 
+## Evaluation
+
+See [BENCHMARK.md](./BENCHMARK.md) for the exploratory long-horizon Notes A/B evaluation, its implementation-quality audit, measurement limitations, and controlled-rerun requirements. The current report is diagnostic case evidence, not a causal product benchmark.
+
 ## Optional integrations
 
 The core has no dependency on goal, plan mode, orchestrator, subagents, memory, or compaction extensions.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-notes",
-  "version": "0.1.0",
+  "version": "0.1.1",
   "description": "Session-local durable execution checkpoints for Pi",
   "type": "module",
   "repository": {
@@ -24,6 +24,7 @@
     "entry.ts",
     "index.ts",
     "README.md",
+    "BENCHMARK.md",
     "LICENSE"
   ],
   "pi": {