pi-memory-evolution 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +246 -0
- package/LICENSE +21 -0
- package/README.md +106 -0
- package/docs/conversation-recall.md +94 -0
- package/docs/core-quality.md +224 -0
- package/docs/design.md +328 -0
- package/docs/progress-pipeline.md +188 -0
- package/docs/quality-validation.md +85 -0
- package/docs/review-0.2.md +82 -0
- package/docs/testing.md +102 -0
- package/docs/usage.md +386 -0
- package/package.json +61 -0
- package/src/adapter/operations.ts +95 -0
- package/src/adapter/pi-api.ts +24 -0
- package/src/adapter/progress-observation.ts +83 -0
- package/src/adapter/session-context.ts +36 -0
- package/src/child-process.ts +8 -0
- package/src/index.ts +256 -0
- package/src/injector/digest.ts +29 -0
- package/src/memory/evolution.ts +64 -0
- package/src/memory/extractor.ts +63 -0
- package/src/memory/feedback.ts +11 -0
- package/src/memory/learning.ts +24 -0
- package/src/memory/legacy.ts +92 -0
- package/src/memory/memory-store.ts +502 -0
- package/src/memory/privacy.ts +51 -0
- package/src/memory/progress-targets.ts +52 -0
- package/src/memory/quality.ts +81 -0
- package/src/memory/query.ts +87 -0
- package/src/memory/recovery.ts +23 -0
- package/src/memory/retriever.ts +181 -0
- package/src/memory/search.ts +105 -0
- package/src/memory/sqlite.ts +7 -0
|
@@ -0,0 +1,188 @@
|
|
|
1
|
+
# Progress pipeline follow-up
|
|
2
|
+
|
|
3
|
+
This **unreleased 0.2.0** follow-up fixes reproduced capture/nomination failures after
|
|
4
|
+
`a3c5943`. It keeps schema 5, ordinary recall's relevance/identity gates, automatic model
|
|
5
|
+
selection/authentication, and the existing transactional write boundary.
|
|
6
|
+
|
|
7
|
+
## Reproduced failure chain
|
|
8
|
+
|
|
9
|
+
A read-only investigation found a stale project-state claim absent from all five saved
|
|
10
|
+
progress target lists in the diagnostic snapshot; all five corresponding model events
|
|
11
|
+
had zero changed records. The model could not update an ID it never received. `done`
|
|
12
|
+
was processing status, not proof of learning.
|
|
13
|
+
|
|
14
|
+
Replay of the tool-heavy development session showed:
|
|
15
|
+
|
|
16
|
+
1. A long implementation run ended with an assistant error, so no progress source was
|
|
17
|
+
captured despite completed tool operations.
|
|
18
|
+
2. In its successful continuation, a 64-entry context window retained only “continue”.
|
|
19
|
+
The last-eight-unique-paths policy also dropped the repository root. No targets meant
|
|
20
|
+
no source/job, even though a full test result existed.
|
|
21
|
+
3. In the later commit/push/restart turn, only the last eight tool results survived;
|
|
22
|
+
restart inspection displaced the real commit/push result.
|
|
23
|
+
4. Reusing answer retrieval for update nomination required an exact absolute repository
|
|
24
|
+
path. A project-state claim containing only its bare repository name could not qualify.
|
|
25
|
+
Per-path top-2 and answer redundancy gates further restricted update opportunities.
|
|
26
|
+
5. The unmodified state was less than one day old. Its freshness factor was about 0.914,
|
|
27
|
+
so time-based decay did not exclude it. Age is not evidence of task completion.
|
|
28
|
+
6. The user's explicit three-part requirements statement did not match the learning cue
|
|
29
|
+
regex and no compaction occurred, so it was not captured as a user statement.
|
|
30
|
+
|
|
31
|
+
This does not prove every old clause was false. Test completion and a commit do not imply
|
|
32
|
+
full product acceptance; updates must preserve unverified or still-pending clauses.
|
|
33
|
+
|
|
34
|
+
## User context and natural learning
|
|
35
|
+
|
|
36
|
+
The public active-branch context facade remains the only context source. A bounded scan
|
|
37
|
+
now considers up to 4096 entries and 4096 messages, retaining at most six sanitized user
|
|
38
|
+
texts of 2048 bytes each. Repeated consecutive topic-less continuations share a slot.
|
|
39
|
+
Reset and unknown-topic barriers are kept, not replaced by an older successful topic.
|
|
40
|
+
Assistant/tool text, raw compaction summaries, injected digests and slash commands do not
|
|
41
|
+
supply user topics. Retained user tails remain supported. No full-session archive replay,
|
|
42
|
+
cross-branch traversal or persistent topic cache was introduced.
|
|
43
|
+
|
|
44
|
+
`memory/learning.ts` distinguishes explicit cues and natural declarations such as:
|
|
45
|
+
|
|
46
|
+
- 我比较在意的三大功能……这些你觉得做得怎么样?
|
|
47
|
+
- 我们的核心需求是……
|
|
48
|
+
- 我希望系统每次都能自动……
|
|
49
|
+
- 我喜欢茶。
|
|
50
|
+
- Our priorities are …
|
|
51
|
+
- Our project must …
|
|
52
|
+
|
|
53
|
+
A statement may precede a question seeking feedback. A plain recall question, quoted
|
|
54
|
+
example, vague continuation or one-off execution request is not a new durable requirement.
|
|
55
|
+
This is conservative pattern-based intent recognition, not universal language understanding.
|
|
56
|
+
The active model still extracts provisional claims and may return no update.
|
|
57
|
+
|
|
58
|
+
A mixed statement/work turn can capture a separate user source and progress source.
|
|
59
|
+
Their model calls use the serial queue and retain separate write authority: tool output
|
|
60
|
+
cannot create preferences, and a user request is not proof an operation succeeded. This
|
|
61
|
+
can cost two background calls rather than the old single cue call that discarded progress.
|
|
62
|
+
Source idempotency and existing retry caps still apply.
|
|
63
|
+
|
|
64
|
+
## Evidence selection and interruptions
|
|
65
|
+
|
|
66
|
+
The progress inspector considers only the current user turn, with at most 4096 trailing
|
|
67
|
+
messages. It requires linked call/result IDs, a work request and at least one recognized
|
|
68
|
+
work operation. Allowed observation tool names are bash/write/edit/read/grep/find/ls;
|
|
69
|
+
unknown/custom tools are not automatically trusted as work observations.
|
|
70
|
+
|
|
71
|
+
Operation classes prioritize evidence instead of blindly taking the newest eight:
|
|
72
|
+
|
|
73
|
+
| Priority | Examples |
|
|
74
|
+
|---|---|
|
|
75
|
+
| 5 | git commit/push/merge/rebase/cherry-pick |
|
|
76
|
+
| 4 | test/build/process runners; git status/log/show/diff/ls-remote |
|
|
77
|
+
| 3 | file edits/writes and recognized mutation/deployment commands |
|
|
78
|
+
| 0 | supporting inspection |
|
|
79
|
+
|
|
80
|
+
The priority is a retention hint, not a success/verification classifier. Git arguments,
|
|
81
|
+
Python/Node scripts or exit code alone do not prove a particular task completed. The
|
|
82
|
+
model must inspect the actual operation/output, failures and qualifications. Ties favor
|
|
83
|
+
newer observations; the chosen set is restored to chronological order for the model.
|
|
84
|
+
The last eight most important observations fit within a 28,000-byte JSON budget; budget
|
|
85
|
+
pressure removes lower-priority observations first. Each stored operation is at most
|
|
86
|
+
1024 bytes, output at most 2048 bytes, with head/tail previews and redaction. Request/report
|
|
87
|
+
remain bounded. The payload exposes omitted counts and scan limits, not an illusion of
|
|
88
|
+
complete evidence. More than eight important distinct outcomes can still be omitted.
|
|
89
|
+
|
|
90
|
+
`completion` is either `completed` or `interrupted`. Error/aborted final assistant responses
|
|
91
|
+
retain already observed tool operations but omit the assistant report. This never certifies
|
|
92
|
+
the whole work item as finished. A request, unmatched result, unfinished tool-use response
|
|
93
|
+
or assistant-only report cannot substitute for observed work. Captures made while the
|
|
94
|
+
foreground signal is cancelled remain eligible for existing automatic recovery; they do
|
|
95
|
+
not require another compaction or a manual evolve command.
|
|
96
|
+
|
|
97
|
+
`memory_recall`/other internal memory tools and operation arguments referencing the owned
|
|
98
|
+
state directory are excluded. A mixed shell command mentioning that directory may be
|
|
99
|
+
conservatively omitted rather than feeding the extension's own records back as independent
|
|
100
|
+
corroboration. This is not a universal detector of indirect self-reference through renamed
|
|
101
|
+
files or custom tools.
|
|
102
|
+
|
|
103
|
+
There is **no new per-tool disk journal**. Hard kills before `agent_end`, work outside the
|
|
104
|
+
bounded scan, lost tool-result pairs, unsupported commands and missing retained context
|
|
105
|
+
remain limitations. Extending those guarantees requires a separate persistence design,
|
|
106
|
+
not calling an interrupted task complete.
|
|
107
|
+
|
|
108
|
+
## Operation resources and update nomination
|
|
109
|
+
|
|
110
|
+
A small non-executing shell lexer extracts hints from explicit `cd` and `git -C`, including
|
|
111
|
+
quoted paths and simple environment prefixes. It does not treat quoted semicolons or
|
|
112
|
+
heredoc source bodies as independent commands. Dynamic paths/substitutions/globs and
|
|
113
|
+
unsupported shell syntax lose hints rather than being executed or guessed.
|
|
114
|
+
|
|
115
|
+
File operations can discover a real checkout root by walking parent `.git` markers
|
|
116
|
+
without running git. Operation resources come from arguments, not output or the current
|
|
117
|
+
cwd alone. At most 16 resources are retained, in operation-priority order; up to eight
|
|
118
|
+
complete paths of at most 512 bytes each are included in the model payload as host hints.
|
|
119
|
+
Longer paths can still participate locally, but are not truncated into a fictional model
|
|
120
|
+
resource path.
|
|
121
|
+
|
|
122
|
+
`memory/progress-targets.ts` is separate from question-answer retrieval:
|
|
123
|
+
|
|
124
|
+
- Candidates are existing project states in the capture origin, excluding pinned,
|
|
125
|
+
forgotten, conflicted or explicitly incorrect records. Expired states remain eligible
|
|
126
|
+
for fresh evidence.
|
|
127
|
+
- Operation resource paths or explicit bare project names provide subject hints; a
|
|
128
|
+
generic cwd/origin is not authority. Distinct absolute resources with the same basename
|
|
129
|
+
cannot qualify by falling back to generic commit/push words.
|
|
130
|
+
- User-topic matches are a separate fallback, not tool-output-derived subject inference.
|
|
131
|
+
- Pending states receive nomination priority over historical completed notes.
|
|
132
|
+
- Up to eight IDs are nominated without per-path top-2, answer deduplication or an
|
|
133
|
+
answer-relative score cutoff. Ordinary recall's path and topic gates are unchanged.
|
|
134
|
+
|
|
135
|
+
Current nomination scores are resource path 60 / explicit project name 50, plus two per
|
|
136
|
+
matched user feature (up to eight), plus 20 for recognizable pending state wording. They
|
|
137
|
+
are deterministic candidate heuristics, not learned confidence. Oldest evidence/ID break
|
|
138
|
+
ties. Eight candidates are still a cap; large/multi-project tasks may need additional
|
|
139
|
+
observations and cannot be assumed fully updated in one pass.
|
|
140
|
+
|
|
141
|
+
The model still must identify the same subject/fact and use exact `replaces` IDs. Store
|
|
142
|
+
checks still enforce candidate authority, source time, generation, pinning and origin.
|
|
143
|
+
A nominated resource is not proof of completion. New preferences/facts from progress are
|
|
144
|
+
rejected, and unsupported compound clauses must remain unchanged.
|
|
145
|
+
|
|
146
|
+
## Diagnostics and activation
|
|
147
|
+
|
|
148
|
+
`/memory learning` shows a bounded, sanitized transient capture/nomination snapshot:
|
|
149
|
+
|
|
150
|
+
- natural/explicit learning intent and number of captured user statements;
|
|
151
|
+
- no-work-request/no-work-observation/unfinished-response versus observed work;
|
|
152
|
+
- scan limit, linked/ignored/retained/omitted observation counts and completion type;
|
|
153
|
+
- no-update-targets, already-captured or progress-captured;
|
|
154
|
+
- operation resources, selected IDs and candidate reasons, without memory bodies.
|
|
155
|
+
|
|
156
|
+
It also shows recent actual model transaction changed-record counts. `/memory status`
|
|
157
|
+
includes those persistent outcomes and explicitly says `done` means processed/retired,
|
|
158
|
+
not necessarily learned. Zero changes remain distinguishable from a provider failure.
|
|
159
|
+
Diagnostics do not persist raw provider responses or claim to know why the model chose
|
|
160
|
+
not to change a fact. User feedback and aliases are not inferred from lookup frequency.
|
|
161
|
+
|
|
162
|
+
No schema migration or old-record rewrite is needed. Reload/restart activates the new
|
|
163
|
+
code. Updating alone does not reopen completed jobs, replay old transcripts, backfill
|
|
164
|
+
previously missed requirements or invent completed states. New observations/statements
|
|
165
|
+
and compaction use the repaired pipeline. Back up state before version changes as usual.
|
|
166
|
+
|
|
167
|
+
## Validation
|
|
168
|
+
|
|
169
|
+
- **215 passing tests**, strict typecheck and package inspection via `npm run check`.
|
|
170
|
+
- Synthetic runtime tests cover long work/continuation, early commit/test preservation,
|
|
171
|
+
interruptions, cancelled foreground recovery, operation identity, ambiguous paths,
|
|
172
|
+
same-origin/pin/lifecycle boundaries, natural requirements, separate mixed-source
|
|
173
|
+
authority, replay idempotency, JSON expansion budgets and zero-change diagnostics.
|
|
174
|
+
- Real installed Pi/Bun with a loopback fake model performs an actual temporary Git
|
|
175
|
+
commit and failed push followed by **12** diagnostic tool results. The early result
|
|
176
|
+
reaches the model, the bare-project pending state is nominated and updated, and the
|
|
177
|
+
next provider payload contains the new partial state with full acceptance still open.
|
|
178
|
+
A natural priorities statement also learns without an explicit remember cue. Existing
|
|
179
|
+
recall/tool/feedback/recovery and model/auth tests remain enabled.
|
|
180
|
+
- Read-only replay of three relevant real-session turns (interrupted implementation,
|
|
181
|
+
successful continuation, commit/push/restart) now nominates both previously missed
|
|
182
|
+
stale states. The continuation retains test counts; the commit turn retains the actual
|
|
183
|
+
commit result. The connection reported `total_changes() = 0`.
|
|
184
|
+
|
|
185
|
+
The real-data replay exercises host selection only, not a paid model consolidation or
|
|
186
|
+
production update. It does not claim the historical records are already corrected. The
|
|
187
|
+
real-Pi update test uses synthetic data and a fake model, not a multi-day natural-language
|
|
188
|
+
accuracy benchmark. See [design.md](design.md) and [core-quality.md](core-quality.md).
|
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
# Recall and progress quality follow-up
|
|
2
|
+
|
|
3
|
+
This follows `f6599d0` (global cross-session recall), not the earlier directory-isolation
|
|
4
|
+
review. Version 0.2.0 remains unreleased. Validation was run on Node 26.8.1 and the
|
|
5
|
+
installed Pi 0.85 standalone Bun host.
|
|
6
|
+
|
|
7
|
+
## Three causes and changes
|
|
8
|
+
|
|
9
|
+
1. **Weak secondary recall:** partial CJK fragments and repository-name components could
|
|
10
|
+
qualify unrelated records. Raw overlap lacked enough query coverage and relative
|
|
11
|
+
relevance filtering. Retrieval now uses words/concepts, exact literals, document
|
|
12
|
+
frequency, coverage and redundant-facet filtering. Body/alias/origin-identifier field
|
|
13
|
+
factors are 1/0.8/0.2. Source IDs/current cwd have no authority bonus. Generic internal
|
|
14
|
+
`scope` fields are not treated as cross-context semantics: doing so distorted frequency
|
|
15
|
+
weights and could promote storage-directory facts over a cross-session preference.
|
|
16
|
+
2. **Stale progress:** ordinary work turns did not enter the cue/compaction-only evolution
|
|
17
|
+
loop. Completed work can now supply linked tool observations, including failures,
|
|
18
|
+
targeting at most 8 existing same-origin project states. Expired states can be updated
|
|
19
|
+
with fresh evidence, but forgotten/conflicted/pinned states cannot. Explicit operation
|
|
20
|
+
paths are not crowded out by generic task words. Store checks prevent even accidentally
|
|
21
|
+
widened candidates from relabeling facts as project progress.
|
|
22
|
+
3. **Cross-language misses:** literal matching could not connect an English question to
|
|
23
|
+
the Chinese preference. A bounded bilingual concept map supports existing records;
|
|
24
|
+
optional model-generated bilingual aliases extend matching without recall-time model
|
|
25
|
+
calls. Aliases are validated, persisted, copied defensively and undoable. Alias-only
|
|
26
|
+
changes do not refresh evidence dates; correction clears old aliases. Excerpts apply
|
|
27
|
+
the same literal/prose boundary, rather than locating a topic word inside an earlier
|
|
28
|
+
filename and omitting the actual relevant passage.
|
|
29
|
+
|
|
30
|
+
The detailed formula, caps, update restrictions and migration contract are in
|
|
31
|
+
[design.md](design.md). No project files/configuration are modified by memory evolution.
|
|
32
|
+
|
|
33
|
+
## Validation
|
|
34
|
+
|
|
35
|
+
- **122/122 tests**, strict TypeScript checking, and package dry-run inspection.
|
|
36
|
+
- Runtime coverage: **100% lines / 91.44% branches / 98.68% functions**. Coverage is not
|
|
37
|
+
a correctness proof, and host-specific behavior also needs the real-Pi check below.
|
|
38
|
+
- Real Pi RPC tests passed repeatedly using fresh processes, separate working directories,
|
|
39
|
+
a shared temporary memory database and a loopback fake model. They verify active-model/
|
|
40
|
+
auth reuse, cross-session recall, contextual continuation, topic changes, Chinese queries,
|
|
41
|
+
learned aliases, provenance, forget and absence of approval prompts.
|
|
42
|
+
- The real-host test also performs a real local Git commit in a temporary repository,
|
|
43
|
+
followed by a push that fails because no remote exists. Actual tool events reach the
|
|
44
|
+
progress model input. The fake model replaces “not committed or pushed” with “commit
|
|
45
|
+
created; push pending”; the next turn receives the updated state. Three memory-model
|
|
46
|
+
calls are observed: two explicit learning inputs and one work observation.
|
|
47
|
+
- Schema-2-to-3 upgrade tests preserve records/history/dates. Malformed aliases/targets,
|
|
48
|
+
invalid model updates, forgetting queued targets and manual correction are covered.
|
|
49
|
+
- One synthetic 600-summary/1,800-record sample: capture 662 ms, reopen 1 ms, cold/warm
|
|
50
|
+
recall 196/99 ms. Segmentation is memoized only within a query for repeated text/origins;
|
|
51
|
+
this is a local timing sample, not a large-corpus performance guarantee.
|
|
52
|
+
|
|
53
|
+
## Real-data replay, separately read-only
|
|
54
|
+
|
|
55
|
+
A direct SQLite read-only connection (`query_only`, no `MemoryStore` construction) read
|
|
56
|
+
146 existing records. Nine query/context cases passed; the connection reported zero
|
|
57
|
+
changes and the existing schema marker stayed at 2. No migration, correction, deletion,
|
|
58
|
+
model evolution or paid call was triggered by this replay. The active Pi session may
|
|
59
|
+
independently capture new memories; these counts are a snapshot, not a frozen fixture.
|
|
60
|
+
|
|
61
|
+
- The Chinese project-boundary question selected the actual cross-session preference.
|
|
62
|
+
- The equivalent English question ranked that preference first, with a relevant provenance
|
|
63
|
+
qualification second; the weak storage-directory fact was no longer selected.
|
|
64
|
+
- The reload/injection question selected two relevant rules, not old CI/setup filler.
|
|
65
|
+
- The model/auth question retrieved corresponding implementation/reuse context.
|
|
66
|
+
- Context-free continuation and unrelated networking topics were empty. A memory-topic
|
|
67
|
+
continuation recalled the preference; a later topic switch did not revive it.
|
|
68
|
+
|
|
69
|
+
## Limits and activation
|
|
70
|
+
|
|
71
|
+
These checks establish orchestration and bounded regression cases, not general live-model
|
|
72
|
+
accuracy. Tool results/reports and aliases remain untrusted evidence; generated states
|
|
73
|
+
are provisional. A quoted command is not execution, and a successful commit/test is not
|
|
74
|
+
proof of a successful push. The model can still make semantic mistakes.
|
|
75
|
+
|
|
76
|
+
The vocabulary is not universal translation; corpus-sensitive thresholds can miss useful
|
|
77
|
+
results. Cross-origin semantic identity is not automatically resolved. Upgrading does not
|
|
78
|
+
invent completion for existing stale entries or replay old tool transcripts. New eligible
|
|
79
|
+
observations/compactions can update tracked progress; exact-ID correction remains available.
|
|
80
|
+
|
|
81
|
+
The local installation references the checkout. Back up with Pi stopped before schema
|
|
82
|
+
upgrade, then reload/restart all instances sharing the database. Status should report
|
|
83
|
+
`SQLite ok (schema 3)` and global topic-based recall. Older builds require a matching
|
|
84
|
+
backup for rollback; do not manually downgrade the schema marker. No npm/tag release or
|
|
85
|
+
live paid-provider/multi-day TUI validation was performed.
|
|
@@ -0,0 +1,82 @@
|
|
|
1
|
+
# Follow-up review of memory evolution 0.2
|
|
2
|
+
|
|
3
|
+
**Historical implementation review at `038de65`.** Its directory-scoped recall assumption
|
|
4
|
+
was subsequently identified as a requirements error: memory must follow the conversation,
|
|
5
|
+
not the working directory. Current behavior is documented in [design.md](design.md):
|
|
6
|
+
all-origin recall, recent-user topic context, visible provenance and conservative writes.
|
|
7
|
+
The test counts below describe that earlier review, not the current test suite.
|
|
8
|
+
|
|
9
|
+
Baseline: `9c84010` on `main`. Scope: all implementation modules, tests, the real-Pi
|
|
10
|
+
smoke script, package metadata and documentation. This review keeps automatic memory
|
|
11
|
+
updates, Pi's active model/authentication and the no-tools/no-approval boundary.
|
|
12
|
+
|
|
13
|
+
## Reproduced issues and fixes
|
|
14
|
+
|
|
15
|
+
The first nine isolated regression cases all failed against the baseline before fixes.
|
|
16
|
+
Additional tests were added while inspecting lifecycle sequences and host contracts.
|
|
17
|
+
|
|
18
|
+
| Area | Defect | Fix |
|
|
19
|
+
|---|---|---|
|
|
20
|
+
| Privacy | Removing controls after matching could assemble an unchecked `password` label. Multiline quoted/JSON values and indented YAML secrets leaked past line-only matching. | Normalize controls first; suppress quoted values and indented continuation blocks before capture, model submission and display. |
|
|
21
|
+
| Markdown extraction | Inner triple backticks could close a four-backtick fence. Bold stripping changed recursive globs, including globs inside code spans. Recognized nested headings caused later progress siblings to disappear. | Match fence character/length, protect code spans, only remove simple bold labels, and maintain a heading stack. |
|
|
22
|
+
| Identity and scope | Colon-concatenated scope/kind/content could collide for valid colon-containing paths. An undocumented `*` scope was included in every scoped read. | Hash a JSON tuple for new IDs and use exact scope lookup. Existing IDs remain valid. |
|
|
23
|
+
| Forget/retry | Forget retired only the first parent; other pending sources repeating the same fact could still feed a model and relearn a paraphrase. | Retire known repeating pending/failed sources, including formatted claims beyond the normal ingestion quota. Keep the current valid replacement transaction's source intact. |
|
|
24
|
+
| Legacy migration | Parent correction erased unchanged children; earlier child edits could prevent later corrected parents from deriving new facts. Corrected legacy claims lost old-content suppression when adopted. | Preserve unchanged children, distinguish newer parent corrections from obsolete child content, and carry suppression hashes through adoption. |
|
|
25
|
+
| Undo/integrity | A structurally corrupted before/after pair could write an unrelated record during undo. Indexed identity/hash and source job errors were incompletely checked; unsupported schemas could receive DDL before rejection. | Validate paired/unique IDs, record metadata and indexed values; validate sources/history in status; reject unsupported versions before DDL. |
|
|
26
|
+
| Batch deduplication | The same content with different kinds could be inserted twice within one local/model batch. | Deduplicate staged claims by exact content as well as persisted records. |
|
|
27
|
+
| Model provenance | Reading the live `ctx.model` again after awaiting completion could label an old response with the newly selected model or throw after context invalidation. | Capture the model and its identity before the call. |
|
|
28
|
+
| Async lifecycle/UI | A throwing context getter outside the queue's `try` could poison later work and shutdown. Notification failure could masquerade as rollback after a successful commit. Commands could reopen storage after shutdown. | Bound the entire queued task, isolate notifications from commits, retain truthful skipped/failed outcomes, and stop commands after shutdown. |
|
|
29
|
+
| Inspection commands | Only 20 unpaginated records were accessible; older legacy imports could not be reached for adoption. `show` omitted useful provenance. | Paginated current/all/legacy lists with stable ordering; `show` includes source and timestamps. Truncated previews are marked. |
|
|
30
|
+
| Excerpts | English sentences were not separated, and a long matching sentence could be clipped before its actual match. Tiny byte budgets could be exceeded by an ellipsis. | Sentence-aware and match-centered excerpts, code-point-safe clipping, and explicit tiny-budget handling. |
|
|
31
|
+
| Evidence dates | Pin/unpin, adoption and undo could make old project-state evidence appear fresh for another seven days. | Preserve or restore the prior evidence date; retain operation time separately in event history. |
|
|
32
|
+
| Smoke harness | Spawn failure and early process exit were not handled cleanly; cleanup could finish before the child exited. | Handle process/stdin errors, await close with bounded termination, and clean temporary state on failure. |
|
|
33
|
+
|
|
34
|
+
The wildcard case requires such a row to exist; there is no public command creating
|
|
35
|
+
one. The unsafe undo case requires corrupted/edited local history, not ordinary model
|
|
36
|
+
JSON. Neither is evidence that production data was attacked or overwritten.
|
|
37
|
+
|
|
38
|
+
## Validation
|
|
39
|
+
|
|
40
|
+
- **88/88 tests passed**, including a deterministic 150-step mixed lifecycle sequence.
|
|
41
|
+
- Strict TypeScript check passed.
|
|
42
|
+
- Node coverage: **100% lines, 90.28% branches, 98.25% functions**. Coverage alone is not
|
|
43
|
+
a correctness guarantee; the baseline's high coverage did not catch these state-order bugs.
|
|
44
|
+
- Four real Node processes concurrently writing SQLite remain covered.
|
|
45
|
+
- Real Pi 0.85/Bun with a loopback fake OpenAI-compatible endpoint verified default-model
|
|
46
|
+
and authentication reuse, two automatic updates, recall injection, removal of the digest
|
|
47
|
+
after `/memory forget`, integrity checks and absence of approval dialogs.
|
|
48
|
+
- A nonexistent Pi executable fails without leaving new temporary state or a live server.
|
|
49
|
+
- Packaging is checked using `npm pack --dry-run`; tests and scripts are not runtime files.
|
|
50
|
+
- Synthetic performance sample: 600 summaries / 1,800 claims captured in about 1.05 s;
|
|
51
|
+
reopening about 2 ms; cold/warm recall about 169/69 ms. These are local observations,
|
|
52
|
+
not cross-machine benchmarks or a guaranteed performance budget.
|
|
53
|
+
|
|
54
|
+
All tests use temporary state and synthetic inputs. No paid model call, production-state
|
|
55
|
+
migration, cleanup or reset was performed as part of the validation.
|
|
56
|
+
|
|
57
|
+
Main regression files (links refer to a Git checkout; tests/scripts are not shipped in
|
|
58
|
+
an npm tarball):
|
|
59
|
+
Tests and maintenance scripts live in the Git checkout, not the runtime tarball:
|
|
60
|
+
|
|
61
|
+
- [State/privacy/parser regressions](https://github.com/btnalit/pi-memory-evolution/blob/main/src/memory/review-regressions.test.ts)
|
|
62
|
+
- [Pi lifecycle and command tests](https://github.com/btnalit/pi-memory-evolution/blob/main/src/index.test.ts)
|
|
63
|
+
- [Adapter tests](https://github.com/btnalit/pi-memory-evolution/blob/main/src/adapter/pi-api.test.ts)
|
|
64
|
+
- [Retriever tests](https://github.com/btnalit/pi-memory-evolution/blob/main/src/memory/retriever.test.ts)
|
|
65
|
+
- [Real-Pi smoke script](https://github.com/btnalit/pi-memory-evolution/blob/main/scripts/test-pi.mjs)
|
|
66
|
+
|
|
67
|
+
## Remaining boundaries
|
|
68
|
+
|
|
69
|
+
- Semantic model accuracy, arbitrary paraphrase equivalence and exhaustive secret
|
|
70
|
+
detection are not guaranteed. Inferred claims remain provisional.
|
|
71
|
+
- Suppression conservatively skips an entire known repeating source's pending model
|
|
72
|
+
pass. Unrelated local claims remain, but unlearned prose may need a new explicit source.
|
|
73
|
+
- Forget/undo is logical, not secure erasure. Previous files/history are preserved.
|
|
74
|
+
- Migration fixes apply to new imports; completed imports are not replayed and previously
|
|
75
|
+
discarded revision information is not automatically reconstructed.
|
|
76
|
+
- Structural checks are not authentication of all well-formed local database edits.
|
|
77
|
+
- No live-provider quality evaluation, multi-day TUI trial or minimum-Node-version matrix
|
|
78
|
+
was run. Local validation used Node 26.8.1 and the real Pi 0.85 standalone binary.
|
|
79
|
+
- History retention/export and periodic backlog draining remain outside this small design.
|
|
80
|
+
|
|
81
|
+
See [README.md](../README.md) for current commands and recovery procedures, and
|
|
82
|
+
[design.md](design.md) for runtime invariants.
|
package/docs/testing.md
ADDED
|
@@ -0,0 +1,102 @@
|
|
|
1
|
+
# Testing
|
|
2
|
+
|
|
3
|
+
Run these commands from a Git checkout after `npm ci --ignore-scripts`.
|
|
4
|
+
Tests use synthetic data and temporary directories, not production memories.
|
|
5
|
+
|
|
6
|
+
## Regression and package checks
|
|
7
|
+
|
|
8
|
+
```bash
|
|
9
|
+
npm run check
|
|
10
|
+
```
|
|
11
|
+
|
|
12
|
+
This runs strict TypeScript checking, the `src/**/*.test.ts` regression suite and
|
|
13
|
+
`check:package`. Package checking asserts that the Pi manifest points to the current
|
|
14
|
+
entry, host APIs remain peer dependencies, all production TypeScript sources are
|
|
15
|
+
packed, and local documentation links resolve to packaged files. Test sources,
|
|
16
|
+
helper scripts and memory state must not ship in the package.
|
|
17
|
+
|
|
18
|
+
`check:package` inspects `npm pack --dry-run --json`; it does not publish to npm.
|
|
19
|
+
|
|
20
|
+
## Installation smoke test
|
|
21
|
+
|
|
22
|
+
```bash
|
|
23
|
+
npm run test:install
|
|
24
|
+
# Or select another installed Pi executable:
|
|
25
|
+
PI_TEST_BINARY=/path/to/pi npm run test:install
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
Requires Pi 0.85+, Node.js 22.19+, npm, Git and `tar`. npm scripts can resolve the
|
|
29
|
+
bundled Pi from dev dependencies first; use `PI_TEST_BINARY` to select your installed
|
|
30
|
+
host explicitly. The script prints the tested host version. The test:
|
|
31
|
+
|
|
32
|
+
1. Builds and extracts the actual npm tarball, outside the development checkout.
|
|
33
|
+
2. Uses real `pi install`, `pi list` and default package discovery—no explicit `-e`
|
|
34
|
+
entry or memory-extension wrapper. Checks `/memory status`, `/memory learning`
|
|
35
|
+
and `/memory explain`, including schema 5, with no checkout `node_modules`.
|
|
36
|
+
3. Verifies repeated installation does not duplicate the package setting, then
|
|
37
|
+
removes it and confirms the command disappears while records/history remain.
|
|
38
|
+
4. Creates a local Git origin from the packed files and the real lockfile. A
|
|
39
|
+
fixture-only Git URL rewrite redirects an `.invalid` URL to this origin; only
|
|
40
|
+
`file` transport is permitted. No GitHub access is needed.
|
|
41
|
+
5. Exercises the native Git installer and its real npm dependency step, updates to
|
|
42
|
+
a new commit, switches from an old pinned tag back to the default branch, and
|
|
43
|
+
verifies source switching and removal preserve schema-5 state.
|
|
44
|
+
6. Serves the actual tarball through a loopback npm registry, with a fresh cache,
|
|
45
|
+
then checks native `pi install npm:pi-memory-evolution`, repeat installation,
|
|
46
|
+
normal loading and removal. No host peer packages are served or installed.
|
|
47
|
+
|
|
48
|
+
The test whitelists child environment variables, gives Pi a fresh agent directory
|
|
49
|
+
and HOME, disables startup network operations, and uses private npm caches/config with
|
|
50
|
+
lifecycle scripts, audit and update notifications disabled. npm is offline for Git
|
|
51
|
+
installation; only the npm fixture uses the loopback registry. Explicit Git updates
|
|
52
|
+
need `PI_OFFLINE=0` (otherwise Pi silently skips them), but file-only Git transport
|
|
53
|
+
and offline npm still prevent public network access. No real credentials are copied.
|
|
54
|
+
Only diagnostic slash commands are submitted; a model turn is a test failure.
|
|
55
|
+
Temporary files and child Pi processes are cleaned up.
|
|
56
|
+
|
|
57
|
+
This checks Pi's package-management and extension-loading path, **not** public
|
|
58
|
+
GitHub/public-registry availability, npm publication, every platform or the
|
|
59
|
+
behavior of dependency lifecycle scripts. There is no project-specific installer:
|
|
60
|
+
users install with Pi's native package manager. These tests are maintenance checks,
|
|
61
|
+
not an extra installation step for users.
|
|
62
|
+
|
|
63
|
+
## Real host with a simulated model
|
|
64
|
+
|
|
65
|
+
```bash
|
|
66
|
+
npm run test:pi
|
|
67
|
+
# PI_TEST_BINARY=/path/to/pi is also supported here.
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
This uses actual Pi processes with a loopback fake provider, testing active-model
|
|
71
|
+
and auth plumbing, automatic replacement, fresh cross-directory sessions,
|
|
72
|
+
contextual follow-ups, unknown-topic barriers, bilingual aliases, provenance,
|
|
73
|
+
feedback, forget and the read-only recall tool without approval dialogs.
|
|
74
|
+
|
|
75
|
+
It executes real Git commits in temporary repositories and intentionally failed
|
|
76
|
+
pushes with no remote. Long tool streams verify that early work evidence still
|
|
77
|
+
nominates old pending states. Persisted failures, malformed model output, startup
|
|
78
|
+
recovery and the real 15-second timer are also exercised; only synthetic retry due
|
|
79
|
+
times are accelerated.
|
|
80
|
+
|
|
81
|
+
These are real-host integration tests, **not live-provider semantic accuracy tests**.
|
|
82
|
+
|
|
83
|
+
## Live-provider validation
|
|
84
|
+
|
|
85
|
+
A live-provider run must use Pi's actual selected model/auth, synthetic fixtures and
|
|
86
|
+
an isolated memory state directory. Check actual records and transaction history,
|
|
87
|
+
not just an assistant acknowledgement or a job marked `done`. Inspect outgoing
|
|
88
|
+
provider context for recall, and retain unverified portions of composite states.
|
|
89
|
+
|
|
90
|
+
Use explicit time/call budgets and preserve failures. Do not replay production
|
|
91
|
+
transcripts, copy credentials, or hand-edit production claims to make a test pass.
|
|
92
|
+
`--offline` disables Pi startup probes, not model requests. Background learning can
|
|
93
|
+
consume provider quota and is not included in foreground session token totals.
|
|
94
|
+
|
|
95
|
+
There is no paid live-provider command in the repository's automatic checks.
|
|
96
|
+
A single successful live scenario does not establish multi-day reliability or full
|
|
97
|
+
product acceptance. Historical design/review documents describe their own tested
|
|
98
|
+
revisions; see [progress pipeline](progress-pipeline.md), [core quality](core-quality.md)
|
|
99
|
+
and [quality validation](quality-validation.md) for context.
|
|
100
|
+
|
|
101
|
+
No GitHub Actions workflow is configured. Run the local checks before submitting
|
|
102
|
+
changes; the Pi-dependent scripts are explicit commands, not part of `npm run check`.
|