pi-sleep-agent 0.8.17 → 0.8.19
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +30 -1
- package/docs/EVIDENCE.md +87 -0
- package/package.json +1 -1
- package/sleepkit/cli.py +14 -1
- package/sleepkit/consolidate.py +63 -3
- package/sleepkit/store.py +194 -5
package/README.md
CHANGED
|
@@ -27,9 +27,35 @@ Sleep adds a reviewable memory cycle to nine selectable coding-agent targets. It
|
|
|
27
27
|
|
|
28
28
|
- **Automatic capture.** Hooks and the Pi extension collect useful observations while you work.
|
|
29
29
|
- **Budgeted consolidation.** Sleep runs only after the record and time gates pass.
|
|
30
|
+
- **A named consolidation window.** One pass is shown the newest 120 T0 records. A buffer
|
|
31
|
+
past that is never silent: `distill --dry-run` and the pass itself report how many records
|
|
32
|
+
fell outside the window, `status` says it before the next pass runs, and `doctor` reports
|
|
33
|
+
it as its own finding. Records outside the window are shown to no pass, and the
|
|
34
|
+
per-session prune holds back the kinds that are expensive to re-derive (preferences,
|
|
35
|
+
procedures, environment warnings) instead of moving them out of reach with the file, so
|
|
36
|
+
what leaves T0 is what a pass could have read. The count is printed instead of being
|
|
37
|
+
inferred from the proposal count.
|
|
30
38
|
- **Review before permanence.** New memories stay provisional until you promote or refuse them.
|
|
31
39
|
- **Nightly dream cycle.** Mine repeated preferences, consolidate recent observations, and measure memory effects while you are away.
|
|
32
40
|
- **Measured replay.** Run the same task with memory off and on before deciding whether a memory helps. Each completed pass leaves a one-line quality summary (`helps / hurts / ties`) in `status` and in the nightly report, so the question stays answerable between passes; a pass whose two arms ran the same prompt says it measured nothing instead of reporting zeros.
|
|
41
|
+
- **Kind-aware lifetimes.** A record's `kind` decides how long a quiet stretch is normal.
|
|
42
|
+
`doctor` measures each provisional against its own row instead of one flat cutoff: a fact is
|
|
43
|
+
cheap to re-learn and cheap to drop (4 weeks unrediscovered / 8 weeks unserved), a preference
|
|
44
|
+
is a statement about you and people are slow (13 / 26), a procedure is durable but testable
|
|
45
|
+
(8 / 12), and an environment warning stays true long after nobody touched it (52 / 52). Every
|
|
46
|
+
finding names the kind and the threshold it used, records without a known kind keep the old
|
|
47
|
+
flat default, and nothing here drops or demotes anything.
|
|
48
|
+
- **Kind-aware pruning.** `run_sleep` clears one session's records at a time, and a record whose
|
|
49
|
+
kind is expensive to re-derive is never dropped before a pass could have read it: when a
|
|
50
|
+
session's file is longer than the consolidation window, the preferences, procedures and
|
|
51
|
+
warnings older than the window stay in T0 while everything a pass could have read is archived.
|
|
52
|
+
The prune report carries `kept_count`, `kept_outside_window` and `read_window`, so the debt is
|
|
53
|
+
a number instead of an inference. `prune` with no `--session` still clears the whole buffer
|
|
54
|
+
under the old file-level rule, which is what an explicit clear-out means.
|
|
55
|
+
- **Kind-aware promotion.** `promote` refuses to make an agent's reading of a *preference*
|
|
56
|
+
permanent: a statement about what you want needs your own words, while a fact, procedure or
|
|
57
|
+
warning is confirmed by the human running `promote` (you read it and chose it). The refusal
|
|
58
|
+
message names which rule stopped it, and the existing speculative-source check still runs first.
|
|
33
59
|
- **Repeated preference detection.** `mine` finds instructions you gave in multiple sessions without a model call.
|
|
34
60
|
- **Project isolation.** A project's provisional memory is injected only into that project.
|
|
35
61
|
- **Reviewable permanence.** Each approved memory in your index carries its review date and source, so you can tell when a saved instruction was pinned.
|
|
@@ -336,7 +362,9 @@ Run the two dream components separately when you want more control:
|
|
|
336
362
|
| `./sleep.py status -v` | ... and the last measured memory quality (`3 helps / 1 hurts / 4 ties from 8 samples`, or "no replay measurement has run yet") |
|
|
337
363
|
| `./sleep.py mine --dry-run` | Find preferences repeated across sessions |
|
|
338
364
|
| `./sleep.py nightly --dry-run` | Preview the unattended nightly workflow |
|
|
339
|
-
| `./sleep.py doctor` | Find duplicate or overlapping memory records, flag contradicting candidates (same subject, one side carries a correction marker) and print the `supersede <older> <newer>` command to run, flag provisional candidates
|
|
365
|
+
| `./sleep.py doctor` | Find duplicate or overlapping memory records, flag contradicting candidates (same subject, one side carries a correction marker) and print the `supersede <older> <newer>` command to run, flag provisional candidates unrediscovered past their kind's threshold (4 weeks for a fact, 13 for a preference, 8 for a procedure, 52 for a warning; kind and threshold appear in the finding), flag records no session was ever served past their kind's unused threshold (8 / 26 / 12 / 52), flag supersede pointers that name a record no longer in the store, report a T0 buffer larger
|
|
366
|
+
than the 120-record consolidation window (with the count left outside it), and report records
|
|
367
|
+
parked in quarantine awaiting a human (report only: it never supersedes, drops, merges or un-parks anything itself) |
|
|
340
368
|
| `./sleep.py doctor --stale <value>` | Show which approved records still hold a stale value (e.g. a changed URL) |
|
|
341
369
|
| `./sleep.py doctor --contradictions` | Ask a model to adjudicate shortlisted pairs of provisional records for conflicts (`--budget`, default $0.25); report-only, and each finding carries the `supersede <older> <newer>` command to run |
|
|
342
370
|
| `./sleep.py doctor --contradictions --tier t2` | Adjudicate pairs of **approved** permanent memory instead of candidates (`--tier both` does both); a T2 finding is labelled `contradicting-memory` and carries the `demote <older>` command, because approved memory is un-approved, never superseded |
|
|
@@ -348,6 +376,7 @@ the host agent's own `omo -p` (used when an `omo` binary is on PATH), then `clau
|
|
|
348
376
|
`SLEEP_LLM_TRANSPORT=omo|claude|http`. A transport that is not usable on this machine
|
|
349
377
|
raises instead of silently falling back, and omo is never handed the gateway's model
|
|
350
378
|
name: it routes through its own config, so only `SLEEP_MODEL` reaches it.
|
|
379
|
+
| `./sleep.py prune [--session <id>] [--older-than <hours>]` | Archive consumed T0 scratch, file-level: with `--session` only that session's records move, with no `--session` the whole buffer is cleared. This is the explicit clear-out, so it moves whole files and ignores kinds; the record-level rule that holds back the kinds no pass could reach belongs to the automatic pass (`run_sleep` prunes one session and reports `kept_count` / `kept_outside_window` / `read_window`) |
|
|
351
380
|
| `./sleep.py supersede <old> <new>` | Mark `<old>` invalid as of now and `<new>` valid as of now (`valid_until` / `valid_from`), so the window a memory was true is readable from the files |
|
|
352
381
|
| `./sleep.py inject --project "$PWD"` | Print provisional context for the current project, newest first, each line dated |
|
|
353
382
|
| `./sleep.py inject --project "$PWD" --query "..."` | Same, but rank records by overlap with the live request so the most relevant survive `--max` |
|
package/docs/EVIDENCE.md
CHANGED
|
@@ -496,3 +496,90 @@ crossing a process boundary with a prompt-recording fake model),
|
|
|
496
496
|
Not changed, and named here so the leftover is not mistaken for coverage: `candidate_pairs()`
|
|
497
497
|
still orders by `ts` and still makes one model call per pair, and `cmd_doctor` does not call it.
|
|
498
498
|
It is a duplicate of the ordering rule with a per-pair cost, left alone deliberately.
|
|
499
|
+
|
|
500
|
+
|
|
501
|
+
## 2026-11-03: the consolidation window is named, and the record left outside it is counted
|
|
502
|
+
|
|
503
|
+
`_fmt_buffer` showed a consolidation pass only the newest 120 T0 records. Everything older was
|
|
504
|
+
never read by any pass, and because pruning only clears what a pass consumed, never cleared either.
|
|
505
|
+
Nothing said so: the night reported `proposals: 2` whether the buffer held 12 records or 300, and
|
|
506
|
+
that number reads as a complete pass.
|
|
507
|
+
|
|
508
|
+
Measured before changing anything, and the honest reading is that the cap had never bitten here:
|
|
509
|
+
`~/.sleep/archive/buffer` holds 31 consumed buffers totalling 259 records, the largest a single
|
|
510
|
+
night ever held was 73, and the live buffer held 1. So this was a latent silent loss, not an
|
|
511
|
+
observed one, and the change is about the report rather than about the limit.
|
|
512
|
+
|
|
513
|
+
What the window now costs to see, all three surfaces checked against a real 127-record buffer in a
|
|
514
|
+
throwaway `SLEEP_HOME`:
|
|
515
|
+
|
|
516
|
+
| surface | without the window | with records outside it |
|
|
517
|
+
|---|---|---|
|
|
518
|
+
| `distill --dry-run` / the pass result | key absent | `outside_window: 7`, `warn: "7 of 127 buffer records fell outside the 120-record window and were not read or pruned"` |
|
|
519
|
+
| `sleep.py status` | T0 line unchanged | `T0 buffer 127 records ... [next pass reads the newest 120; 7 outside the window]` |
|
|
520
|
+
| `sleep.py doctor` | no finding | `{"issue": "buffer-exceeds-consolidation-window", "buffer_records": 127, "window": 120, "outside_window": 7}` |
|
|
521
|
+
|
|
522
|
+
`run_sleep` now persists `buffer_records` and `outside_window` in `state.json` next to `proposals`,
|
|
523
|
+
so a night that left records behind is readable after the fact. The window lives in one place,
|
|
524
|
+
`sleepkit.consolidate.BUFFER_WINDOW`, and the sentence is built by one helper (`_window_warning`,
|
|
525
|
+
wrapped by `_warn_field` so the key is absent rather than empty when there is nothing to report) so
|
|
526
|
+
the preview and the real pass cannot drift. Pinned by `tests/test_cli.py::BufferWindow` (5 tests,
|
|
527
|
+
crossing a process boundary): under the window nothing appears in any of the three surfaces, at
|
|
528
|
+
`WINDOW + 7` the count and the sentence say both numbers, and a pass the record/time gate *refuses*
|
|
529
|
+
still reports the buffer size it did not read.
|
|
530
|
+
|
|
531
|
+
Deliberately NOT changed, since the two are separable and only the reporting was asked for: the pass
|
|
532
|
+
still reads only the newest window and the rest still waits for the next night. Sending a large
|
|
533
|
+
buffer in several passes is a separate change with its own cost (one model call per slice).
|
|
534
|
+
|
|
535
|
+
## 2026-11-03: a record's kind decides how long a quiet stretch is normal
|
|
536
|
+
|
|
537
|
+
`kind` was carried on every record and read by nothing. All four kinds shared one pair of
|
|
538
|
+
thresholds (`STALE_CANDIDATE_WEEKS = 4.0`, `UNUSED_CANDIDATE_WEEKS = 8.0`), `prune_buffer` moved
|
|
539
|
+
whole files without looking, and `promote` never saw the field. The live T1 pool is 12 candidates -
|
|
540
|
+
6 preferences, 5 facts, 1 warning - so a statement about the person and a cheap observation were
|
|
541
|
+
being judged by the same clock, and a warning that stops an agent from repeating a failed attempt
|
|
542
|
+
had the shortest life of the four.
|
|
543
|
+
|
|
544
|
+
Three changes, each independently visible:
|
|
545
|
+
|
|
546
|
+
- `store.KIND_LIFETIME_WEEKS` gives each kind its own row - fact 4/8, preference 13/26, procedure
|
|
547
|
+
8/12, warning 52/52 (unrediscovered/unserved weeks) - and `doctor` now measures against it. Every
|
|
548
|
+
finding carries `kind` and the `threshold_weeks` it used, so a number in the report can be
|
|
549
|
+
checked against the row that produced it. A record whose kind is missing or unknown keeps the
|
|
550
|
+
historical flat default, which is what makes this safe to ship over an existing store.
|
|
551
|
+
- `store.prune_session` is the record-level prune: it holds back the kinds that are expensive to
|
|
552
|
+
re-derive (preference, procedure, warning) instead of moving them out of reach with the file.
|
|
553
|
+
Measured against a clean `SLEEP_HOME`: a 130-record session with 10 durable records leaves 10 on
|
|
554
|
+
disk and archives 120; a 125-record session of preferences alone keeps all 5 that the window
|
|
555
|
+
could not reach and reports `kept_outside_window: 5`, `read_window: 120`.
|
|
556
|
+
- `store.KIND_PROMOTE_NEEDS_USER` holds `preference`, so `promote` refuses to make an agent's
|
|
557
|
+
reading of a preference permanent (`a statement about what the user wants needs the user's own
|
|
558
|
+
words`). Facts, procedures and warnings stay promotable from any source, because the human
|
|
559
|
+
running `promote` is the confirmation - they read the body and chose it - and a warning captured
|
|
560
|
+
from an agent's own failed attempt is the normal way one arrives.
|
|
561
|
+
|
|
562
|
+
Where this stops, deliberately, so the edges are not mistaken for coverage:
|
|
563
|
+
|
|
564
|
+
- The prune hold-back is on `store.prune_session` and `run_sleep` calls it for the session it
|
|
565
|
+
consolidated. The `prune` COMMAND stays file-level: it is the explicit clear-out, `--older-than`
|
|
566
|
+
asks about file age and has no record-level meaning, and routing a typed command into the
|
|
567
|
+
record-level rule would quietly ignore that flag.
|
|
568
|
+
- `KIND_PROMOTE_NEEDS_USER` is one kind, not a general "every kind needs its own confirmation"
|
|
569
|
+
rule. A broader gate would fire on cases where an agent-recorded warning or procedure is
|
|
570
|
+
perfectly legitimate, and an always-on confirmation is a gate nobody reads.
|
|
571
|
+
- `candidate_pairs()` is untouched: it still orders by `ts` and still makes one model call per pair.
|
|
572
|
+
Named here because the file is back in this diff and the leftover should not be read as new.
|
|
573
|
+
- Nothing drops, demotes or expires on the basis of these tables. `doctor` reports; the thresholds
|
|
574
|
+
only decide when it speaks.
|
|
575
|
+
|
|
576
|
+
Pinned by `tests/test_core.py::KindLifetimes` (table matches the documented defaults; an unknown
|
|
577
|
+
kind falls back; a 30-day preference is not stale while a 30-day fact is; a 730-day warning still
|
|
578
|
+
sits below its own 52-week threshold) and `tests/test_core.py::RecordLevelPrune` (unread durable
|
|
579
|
+
records survive, a session with nothing to keep still clears, records inside the window survive
|
|
580
|
+
too, the reported rewrite is checked against the file, an idempotent second run, a missing session
|
|
581
|
+
is not an error), plus `tests/test_cli.py::KindPromotion` (agent preference refused, user
|
|
582
|
+
preference promotable, agent fact and warning promotable, speculative source still named) and the
|
|
583
|
+
per-session `buffer-exceeds-consolidation-window` finding
|
|
584
|
+
(`test_many_small_sessions_are_not_one_large_buffer`: six 20-record sessions are six complete
|
|
585
|
+
passes, not one 120-record buffer, and the doctor says nothing).
|
package/package.json
CHANGED
package/sleepkit/cli.py
CHANGED
|
@@ -70,7 +70,14 @@ def cmd_status(a) -> int:
|
|
|
70
70
|
_p(replay_mod.quality_line())
|
|
71
71
|
except Exception:
|
|
72
72
|
pass
|
|
73
|
-
|
|
73
|
+
line = "T0 buffer %d records / %d files / %d bytes (captured while you worked)" % (bc["records"], bc["files"], bc["bytes"])
|
|
74
|
+
# A window smaller than the buffer is the one case where the next pass cannot read
|
|
75
|
+
# everything it is standing on. Say it up front instead of leaving it to be inferred
|
|
76
|
+
# from the proposals count of a night that already happened.
|
|
77
|
+
if bc["records"] > consolidate.BUFFER_WINDOW:
|
|
78
|
+
line += " [next pass reads the newest %d; %d outside the window]" % (
|
|
79
|
+
consolidate.BUFFER_WINDOW, bc["records"] - consolidate.BUFFER_WINDOW)
|
|
80
|
+
_p(line)
|
|
74
81
|
_p("T1 candidates %d provisional, %d refused, %d total (distilled, shown as unverified context)" % (len(prov), len(ref), len(cands)))
|
|
75
82
|
_p(" quarantine %d parked (needs human eyes) (held out until you review)" % len(q))
|
|
76
83
|
retracted = []
|
|
@@ -202,6 +209,12 @@ def cmd_supersede(a) -> int:
|
|
|
202
209
|
|
|
203
210
|
|
|
204
211
|
def cmd_prune(a) -> int:
|
|
212
|
+
# `prune` is the explicit-act path, so it keeps the file-level rule it always had:
|
|
213
|
+
# both flags keep working, and "clear this session out" still clears the session.
|
|
214
|
+
# The record-level rule (hold back the kinds no pass could have read) belongs to an
|
|
215
|
+
# automatic pass, which has a read window to measure against and cannot ask; it is
|
|
216
|
+
# what `run_sleep` calls. Routing a typed command into it would also quietly ignore
|
|
217
|
+
# `--older-than`, which asks about file age and has no record-level meaning.
|
|
205
218
|
rep = store.prune_buffer(older_than_hours=a.older_than, session=a.session)
|
|
206
219
|
_p(json.dumps(rep, indent=2, sort_keys=True))
|
|
207
220
|
return 0
|
package/sleepkit/consolidate.py
CHANGED
|
@@ -24,7 +24,14 @@ def pending_dir() -> str:
|
|
|
24
24
|
return d
|
|
25
25
|
|
|
26
26
|
|
|
27
|
-
|
|
27
|
+
# The window one consolidation pass is shown. It is a display limit, not a policy:
|
|
28
|
+
# whatever falls below it is NOT read by the pass and NOT pruned, and until 0.8.18
|
|
29
|
+
# nothing said so. BUFFER_WINDOW is now named in one place so the count of records
|
|
30
|
+
# left outside the window can be reported instead of being inferred from len().
|
|
31
|
+
BUFFER_WINDOW = 120
|
|
32
|
+
|
|
33
|
+
|
|
34
|
+
def _fmt_buffer(records: list[dict], limit: int = BUFFER_WINDOW) -> str:
|
|
28
35
|
lines = []
|
|
29
36
|
for r in records[-limit:]:
|
|
30
37
|
lines.append("- [%s|%s] %s" % (r.get("kind", "fact"), r.get("source", "?"),
|
|
@@ -32,6 +39,41 @@ def _fmt_buffer(records: list[dict], limit: int = 120) -> str:
|
|
|
32
39
|
return "\n".join(lines)
|
|
33
40
|
|
|
34
41
|
|
|
42
|
+
def _window_split(records: list) -> tuple:
|
|
43
|
+
"""(sent, skipped): records inside the format window and those left outside it.
|
|
44
|
+
|
|
45
|
+
Only the newest BUFFER_WINDOW records are shown to the model, so a record with an
|
|
46
|
+
older ts is never read by any pass and never pruned (prune only runs on records the
|
|
47
|
+
pass consumed). One distinct session arriving with more than a window's worth of
|
|
48
|
+
captures would therefore be silently dropped from consolidation. The caller reports
|
|
49
|
+
the skipped count so that possibility is visible instead of invisible.
|
|
50
|
+
"""
|
|
51
|
+
if len(records) <= BUFFER_WINDOW:
|
|
52
|
+
return list(records), []
|
|
53
|
+
return list(records[-BUFFER_WINDOW:]), list(records[:-BUFFER_WINDOW])
|
|
54
|
+
|
|
55
|
+
|
|
56
|
+
def _warn_field(skipped: int, total: int) -> dict:
|
|
57
|
+
"""{"warn": sentence} when records were left out, {} when none were.
|
|
58
|
+
|
|
59
|
+
A key that is always present but sometimes empty makes every caller test the value;
|
|
60
|
+
absent-when-there-is-nothing-to-say keeps `"warn" not in result` meaningful.
|
|
61
|
+
"""
|
|
62
|
+
return {"warn": _window_warning(skipped, total)} if skipped > 0 else {}
|
|
63
|
+
|
|
64
|
+
|
|
65
|
+
def _window_warning(skipped: int, total: int) -> str:
|
|
66
|
+
"""One sentence for both paths that can leave records outside the window.
|
|
67
|
+
|
|
68
|
+
Shared so `--dry-run` and `--sleep` cannot drift: a preview that says nothing while
|
|
69
|
+
the real pass warns is worse than no preview.
|
|
70
|
+
"""
|
|
71
|
+
if skipped <= 0:
|
|
72
|
+
return ""
|
|
73
|
+
return ("%d of %d buffer records fell outside the %d-record window and were "
|
|
74
|
+
"not read or pruned" % (skipped, total, BUFFER_WINDOW))
|
|
75
|
+
|
|
76
|
+
|
|
35
77
|
def existing_pool() -> str:
|
|
36
78
|
lines = []
|
|
37
79
|
for c in store.list_candidates(with_body=True):
|
|
@@ -146,8 +188,11 @@ def distill(session: str | None = None, model: str | None = None,
|
|
|
146
188
|
return {"ok": True, "reason": "buffer empty", "proposals": 0, "accepted": 0,
|
|
147
189
|
"consumed": False}
|
|
148
190
|
projects, unaddressed = _project_groups(recs)
|
|
191
|
+
_sent, outside_window = _window_split(recs)
|
|
149
192
|
if dry_run:
|
|
150
193
|
return {"ok": True, "dry_run": True, "buffer_records": len(recs),
|
|
194
|
+
"outside_window": len(outside_window),
|
|
195
|
+
**_warn_field(len(outside_window), len(recs)),
|
|
151
196
|
"projects": [o for o, _ in projects], "unaddressed": len(unaddressed),
|
|
152
197
|
"would_send": _fmt_buffer(recs)[:2000]}
|
|
153
198
|
if not projects:
|
|
@@ -183,6 +228,11 @@ def distill(session: str | None = None, model: str | None = None,
|
|
|
183
228
|
ing["ok"] = False
|
|
184
229
|
ing["buffer_records"] = len(recs)
|
|
185
230
|
ing["unaddressed"] = len(unaddressed)
|
|
231
|
+
# Records the pass was never shown. Not an error and not a bug on its own; the point
|
|
232
|
+
# is that a night which leaves any behind says so, because "proposals: 2" on a
|
|
233
|
+
# 300-record buffer reads as a complete pass and is not one.
|
|
234
|
+
ing["outside_window"] = len(outside_window)
|
|
235
|
+
ing.update(_warn_field(len(outside_window), len(recs)))
|
|
186
236
|
# "consumed" is what entitles a sleep to prune: the pass produced proposals.
|
|
187
237
|
# Unaddressed records were never read by any pass, so they must not be pruned.
|
|
188
238
|
return ing
|
|
@@ -427,13 +477,23 @@ def run_sleep(session=None, model=None, budget_usd=1.0, min_records=3,
|
|
|
427
477
|
state["last_sleep"] = paths.now()
|
|
428
478
|
state["sleep_count"] = int(state.get("sleep_count", 0)) + 1
|
|
429
479
|
state["last_sleep_result"] = {k: result.get(k) for k in
|
|
430
|
-
("proposals", "accepted", "refused", "llm_exit"
|
|
480
|
+
("proposals", "accepted", "refused", "llm_exit",
|
|
481
|
+
"buffer_records", "outside_window")}
|
|
431
482
|
paths.write_state(state)
|
|
432
483
|
# Synaptic pruning only after a consolidation that actually produced
|
|
433
484
|
# proposals. Pruning a buffer the model never read would silently discard
|
|
434
485
|
# the session's observations.
|
|
486
|
+
#
|
|
487
|
+
# Per session, not the whole directory, because the interesting case is one
|
|
488
|
+
# session whose file outgrew the read window: a file-level move takes the
|
|
489
|
+
# oldest records with it, and those are exactly the records no pass read. The
|
|
490
|
+
# session is the one this pass consolidated; with no session given, the buffer
|
|
491
|
+
# is cleared as a whole under the old file-level rule.
|
|
435
492
|
if result.get("consumed"):
|
|
436
|
-
|
|
493
|
+
if session:
|
|
494
|
+
result["pruned"] = store.prune_session(session)
|
|
495
|
+
else:
|
|
496
|
+
result["pruned"] = store.prune_buffer(older_than_hours=0.0)
|
|
437
497
|
else:
|
|
438
498
|
result["pruned"] = {"skipped": True, "reason": "no proposals consumed"}
|
|
439
499
|
result["action"] = "ran"
|
package/sleepkit/store.py
CHANGED
|
@@ -33,6 +33,44 @@ STALE_CANDIDATE_WEEKS = 4.0
|
|
|
33
33
|
# with things that were never useful. Report only, like every other doctor line.
|
|
34
34
|
UNUSED_CANDIDATE_WEEKS = 8.0
|
|
35
35
|
|
|
36
|
+
# What the two constants above are a default FOR. A record's kind is not decoration:
|
|
37
|
+
# a warning that nobody re-learned in a month is not the same debt as a preference
|
|
38
|
+
# nobody re-learned in a month, and the flat constants made the four kinds behave
|
|
39
|
+
# identically. Each row is (stale_weeks, unused_weeks) and the rationale travels with
|
|
40
|
+
# the number, because a bare constant invites the next reader to "tidy" it back.
|
|
41
|
+
#
|
|
42
|
+
# fact 4 / 8 the old defaults; a fact is cheap to re-learn and cheap to drop
|
|
43
|
+
# preference 13 / 26 a preference is a statement about a person, and a person is
|
|
44
|
+
# slow. Expiring one after four quiet weeks deletes a true thing
|
|
45
|
+
# procedure 8 / 12 durable but testable: a procedure that stops being exercised is
|
|
46
|
+
# suspect sooner than a stated preference, later than a fact
|
|
47
|
+
# warning 52 / 52 an environment warning ("X does not fire on this engine") stays
|
|
48
|
+
# true long after nobody has touched it, and the cost of
|
|
49
|
+
# re-deriving it is paid in failed attempts
|
|
50
|
+
#
|
|
51
|
+
# Both remain REPORT-ONLY. Nothing here drops, demotes or expires a record; it decides
|
|
52
|
+
# when doctor names one. Use lifetime_for() rather than reading the table directly.
|
|
53
|
+
KIND_LIFETIME_WEEKS = {
|
|
54
|
+
"fact": (4.0, 8.0),
|
|
55
|
+
"preference": (13.0, 26.0),
|
|
56
|
+
"procedure": (8.0, 12.0),
|
|
57
|
+
"warning": (52.0, 52.0),
|
|
58
|
+
}
|
|
59
|
+
|
|
60
|
+
|
|
61
|
+
def lifetime_for(kind: str, default_weeks: float = 0.0) -> tuple[float, float, str]:
|
|
62
|
+
"""(stale_weeks, unused_weeks, kind) for one record's kind.
|
|
63
|
+
|
|
64
|
+
An unknown or missing kind gets the historical flat default, so a record written
|
|
65
|
+
by an older version, or by hand, is judged exactly as it was before this table
|
|
66
|
+
existed. The third element is the kind actually used, so a caller can print what
|
|
67
|
+
it judged against without re-deriving the fallback.
|
|
68
|
+
"""
|
|
69
|
+
row = KIND_LIFETIME_WEEKS.get(str(kind or "").strip().lower())
|
|
70
|
+
if row:
|
|
71
|
+
return row[0], row[1], str(kind)
|
|
72
|
+
return STALE_CANDIDATE_WEEKS, UNUSED_CANDIDATE_WEEKS, str(kind or "")
|
|
73
|
+
|
|
36
74
|
|
|
37
75
|
def _hours_since(ts: str) -> float | None:
|
|
38
76
|
"""Hours since an ISO timestamp written by paths.now(). Unparseable input
|
|
@@ -172,6 +210,22 @@ def buffer_counts() -> dict:
|
|
|
172
210
|
"bytes": sum(os.path.getsize(os.path.join(d, f)) for f in files)}
|
|
173
211
|
|
|
174
212
|
|
|
213
|
+
def buffer_session_counts() -> dict:
|
|
214
|
+
"""{session: record count} for the T0 buffer, one entry per file.
|
|
215
|
+
|
|
216
|
+
A pass consolidates ONE session, so this is the count a window means anything
|
|
217
|
+
against: `buffer_counts()` totals every file in the directory and answers a
|
|
218
|
+
different question ("how much scratch is on disk").
|
|
219
|
+
"""
|
|
220
|
+
d = paths.buffer_dir()
|
|
221
|
+
out = {}
|
|
222
|
+
for name in sorted(os.listdir(d)):
|
|
223
|
+
if not name.endswith(".jsonl"):
|
|
224
|
+
continue
|
|
225
|
+
out[name[:-6]] = len(read_buffer(name[:-6]))
|
|
226
|
+
return out
|
|
227
|
+
|
|
228
|
+
|
|
175
229
|
# ------------------------------------------------------- similarity/dedup ---
|
|
176
230
|
|
|
177
231
|
# A proposal that paraphrases an already-refused candidate must not reborn as a
|
|
@@ -449,6 +503,21 @@ def slugify(text: str, maxlen: int = 48) -> str:
|
|
|
449
503
|
return (s[:maxlen].rstrip("-")) or "note"
|
|
450
504
|
|
|
451
505
|
|
|
506
|
+
# What a kind is allowed to become permanent on. This is not the same question as
|
|
507
|
+
# "is this record true": a directive-shaped sentence about how the agent should
|
|
508
|
+
# behave is a statement about the PERSON, and only the person can confirm one, so a
|
|
509
|
+
# preference keeps the existing user-provenance requirement on top of the trust
|
|
510
|
+
# check. The other three kinds describe the world, and a human running `promote` is
|
|
511
|
+
# the confirmation: they read the body and chose it.
|
|
512
|
+
#
|
|
513
|
+
# A warning is the case worth naming, because it is the kind most likely to be
|
|
514
|
+
# captured from an agent's own failed attempt with no user in the loop. That is
|
|
515
|
+
# still legitimate - the human reading it is the check - but the refusal message
|
|
516
|
+
# below says which rule stopped it, so "why can I not promote this" never needs
|
|
517
|
+
# source reading.
|
|
518
|
+
KIND_PROMOTE_NEEDS_USER = ("preference",)
|
|
519
|
+
|
|
520
|
+
|
|
452
521
|
def promote(cid: str, memory_dir: str | None = None, dry_run: bool = False) -> tuple[bool, str]:
|
|
453
522
|
"""T1 -> T2. Explicit human act. Creates a NEW file, never edits one."""
|
|
454
523
|
got = get_candidate(cid)
|
|
@@ -461,6 +530,11 @@ def promote(cid: str, memory_dir: str | None = None, dry_run: bool = False) -> t
|
|
|
461
530
|
if src != "user" and meta.get("trust") == "speculative":
|
|
462
531
|
return False, ("refusing to persist speculative memory from source=%s; "
|
|
463
532
|
"re-record with the user's own words or run a replay first" % src)
|
|
533
|
+
kind = str(meta.get("kind", "") or "")
|
|
534
|
+
if kind in KIND_PROMOTE_NEEDS_USER and src != "user":
|
|
535
|
+
return False, ("refusing to persist a %s from source=%s: a statement about what "
|
|
536
|
+
"the user wants needs the user's own words, not an agent's reading "
|
|
537
|
+
"of them" % (kind, src))
|
|
464
538
|
mdir = memory_dir or paths.memory_dir()
|
|
465
539
|
name = "%s.md" % slugify(body.split(".")[0][:60] or cid)
|
|
466
540
|
dest = os.path.join(mdir, name)
|
|
@@ -810,7 +884,14 @@ def supersede(old_id: str, new_id: str) -> bool:
|
|
|
810
884
|
|
|
811
885
|
# ------------------------------------------------------------ pruning -------
|
|
812
886
|
def prune_buffer(older_than_hours: float = 0.0, session: str | None = None) -> dict:
|
|
813
|
-
"""T0 reset: consumed scratch moves to archive. Nothing is deleted.
|
|
887
|
+
"""T0 reset: consumed scratch moves to archive. Nothing is deleted.
|
|
888
|
+
|
|
889
|
+
File-level and kind-blind by design. It answers "has this session's scratch been
|
|
890
|
+
dealt with?", which is the right question when the caller is an explicit human
|
|
891
|
+
command. It is the WRONG tool for a record-level decision, because a file holds
|
|
892
|
+
records of every kind and moving it takes them all: use prune_session() when a
|
|
893
|
+
record that no pass read must survive.
|
|
894
|
+
"""
|
|
814
895
|
moved = []
|
|
815
896
|
d = paths.buffer_dir()
|
|
816
897
|
cutoff = time.time() - older_than_hours * 3600
|
|
@@ -829,6 +910,92 @@ def prune_buffer(older_than_hours: float = 0.0, session: str | None = None) -> d
|
|
|
829
910
|
return {"moved": moved, "remaining": buffer_counts()}
|
|
830
911
|
|
|
831
912
|
|
|
913
|
+
# A kind whose records are dropped before consolidation has lost work that cannot be
|
|
914
|
+
# recovered by reading the archive later: the raw text survives, but no pass and no
|
|
915
|
+
# session will look at it again. These are the kinds where that loss is expensive -
|
|
916
|
+
# re-deriving a preference means asking the person again, and re-deriving a warning
|
|
917
|
+
# means paying for the failed attempt a second time. A fact is the one kind whose
|
|
918
|
+
# loss is cheap (it can be re-learned by looking), which is why it is deliberately
|
|
919
|
+
# NOT in this set: the default path must not be the expensive one.
|
|
920
|
+
KEEP_WHEN_UNREAD = ("preference", "procedure", "warning")
|
|
921
|
+
|
|
922
|
+
|
|
923
|
+
def prune_session(session: str, keep_kinds=KEEP_WHEN_UNREAD) -> dict:
|
|
924
|
+
"""Prune ONE session's file, holding back records no pass will read.
|
|
925
|
+
|
|
926
|
+
The pass is shown the newest `consolidate.BUFFER_WINDOW` records, so anything older
|
|
927
|
+
than that is read by no pass - and a record no pass reads is also never pruned
|
|
928
|
+
(pruning follows consumption). What a file-level prune does anyway is move the whole
|
|
929
|
+
file, oldest records included, which turns "fell outside the window" from a visible
|
|
930
|
+
backlog into a silent one. This function moves the records the pass could have read
|
|
931
|
+
and keeps the rest, so a withheld record stays where the next pass can still reach it.
|
|
932
|
+
|
|
933
|
+
Imports `consolidate` lazily on purpose: consolidate imports this module at its top,
|
|
934
|
+
so a module-level import here would be circular. The one constant borrowed back is
|
|
935
|
+
the window, kept single-sourced rather than copied.
|
|
936
|
+
"""
|
|
937
|
+
from . import consolidate as _consolidate
|
|
938
|
+
d = paths.buffer_dir()
|
|
939
|
+
src = os.path.join(d, session + ".jsonl")
|
|
940
|
+
if not os.path.isfile(src):
|
|
941
|
+
return {"skipped": True, "reason": "no such session", "kept": [], "moved": ""}
|
|
942
|
+
records = []
|
|
943
|
+
with open(src, encoding="utf-8") as fh:
|
|
944
|
+
for line in fh:
|
|
945
|
+
line = line.strip()
|
|
946
|
+
if not line:
|
|
947
|
+
continue
|
|
948
|
+
try:
|
|
949
|
+
records.append(json.loads(line))
|
|
950
|
+
except json.JSONDecodeError:
|
|
951
|
+
continue
|
|
952
|
+
# The window decides what a pass READS, and a pass having read a record is what
|
|
953
|
+
# licenses pruning it. It says nothing about whether the record was worth keeping:
|
|
954
|
+
# a preference inside the window is still a preference, and a prune has no business
|
|
955
|
+
# dropping it because the pass happened to look at it. A record outside the window
|
|
956
|
+
# is the one case where nothing will ever read it again. So the kind filter runs
|
|
957
|
+
# over the WHOLE file rather than over the tail alone, and what survives stays.
|
|
958
|
+
_window = _consolidate.BUFFER_WINDOW
|
|
959
|
+
in_window = records[max(0, len(records) - _window):]
|
|
960
|
+
held = [r for r in records if str(r.get("kind", "")) in keep_kinds]
|
|
961
|
+
held_outside = [r for r in held if r not in in_window]
|
|
962
|
+
if not held:
|
|
963
|
+
return prune_buffer(older_than_hours=0.0, session=session)
|
|
964
|
+
dropped = [r for r in records if str(r.get("kind", "")) not in keep_kinds]
|
|
965
|
+
# The keep file holds ONLY the records whose kind earned survival; the rest go to
|
|
966
|
+
# the archive. An earlier draft appended the in-window records here as well, which
|
|
967
|
+
# silently rewrote the file with its own contents and reported a "kept" count while
|
|
968
|
+
# nothing changed - the exact shape of bug this function exists to prevent, so the
|
|
969
|
+
# rewrite is asserted below rather than assumed.
|
|
970
|
+
keep_sorted = sorted(held, key=lambda r: str(r.get("ts", "")))
|
|
971
|
+
tmp = src + ".tmp"
|
|
972
|
+
with open(tmp, "w", encoding="utf-8") as fh:
|
|
973
|
+
for r in keep_sorted:
|
|
974
|
+
fh.write(json.dumps(r, sort_keys=True) + "\n")
|
|
975
|
+
os.replace(tmp, src)
|
|
976
|
+
after = read_buffer(session)
|
|
977
|
+
if len(after) != len(keep_sorted):
|
|
978
|
+
raise RuntimeError("prune_session rewrite failed: %d kept, %d on disk"
|
|
979
|
+
% (len(keep_sorted), len(after)))
|
|
980
|
+
# The records the pass COULD read, plus the withheld ones whose kind is safe to
|
|
981
|
+
# drop, go to the archive rather than the keep file; nothing is deleted and the
|
|
982
|
+
# archive path is the one prune_buffer already uses.
|
|
983
|
+
dest = os.path.join(paths.archive_dir("buffer"),
|
|
984
|
+
time.strftime("%Y%m%d-") + session + ".jsonl")
|
|
985
|
+
n = 1
|
|
986
|
+
while os.path.exists(dest):
|
|
987
|
+
dest = os.path.join(paths.archive_dir("buffer"),
|
|
988
|
+
time.strftime("%Y%m%d-") + "%s-%d.jsonl" % (session, n))
|
|
989
|
+
n += 1
|
|
990
|
+
with open(dest, "w", encoding="utf-8") as fh:
|
|
991
|
+
for r in dropped:
|
|
992
|
+
fh.write(json.dumps(r, sort_keys=True) + "\n")
|
|
993
|
+
return {"moved": dest, "archived_records": len(dropped),
|
|
994
|
+
"kept": [r["id"] for r in held], "kept_count": len(held),
|
|
995
|
+
"kept_outside_window": len(held_outside), "read_window": len(in_window),
|
|
996
|
+
"remaining": buffer_counts()}
|
|
997
|
+
|
|
998
|
+
|
|
832
999
|
# Tags are the only index a record carries besides its project, so they are kept
|
|
833
1000
|
# machine-shaped: lowercase, no spaces, bounded in number and length. A model may
|
|
834
1001
|
# propose them, which is exactly why they are normalized here rather than trusted.
|
|
@@ -1041,7 +1208,6 @@ def doctor(memory_dir: str | None = None, targets: list[str] | None = None,
|
|
|
1041
1208
|
"pair": [pool[i][0], pool[j][0]], "similarity": round(sim, 2),
|
|
1042
1209
|
"older": older, "newer": newer,
|
|
1043
1210
|
"suggest": "sleep.py supersede %s %s" % (older, newer)})
|
|
1044
|
-
cutoff_hours = STALE_CANDIDATE_WEEKS * 7 * 24
|
|
1045
1211
|
for c in list_candidates():
|
|
1046
1212
|
if c.get("status") != "provisional":
|
|
1047
1213
|
continue
|
|
@@ -1049,10 +1215,15 @@ def doctor(memory_dir: str | None = None, targets: list[str] | None = None,
|
|
|
1049
1215
|
# record's own creation ts. A provisional that is neither being promoted
|
|
1050
1216
|
# nor re-learned for weeks is noise the queue keeps injecting into every
|
|
1051
1217
|
# session's context, so the doctor surfaces it (report only, nothing drops).
|
|
1218
|
+
# "Weeks" is per kind: the same quiet month means different things for a
|
|
1219
|
+
# preference and for an environment warning (see KIND_LIFETIME_WEEKS).
|
|
1220
|
+
stale_weeks, _unused_weeks, kind = lifetime_for(c.get("kind", ""))
|
|
1052
1221
|
last = c.get("rediscovered_at") or c.get("ts")
|
|
1053
1222
|
age_h = _hours_since(str(last)) if last else None
|
|
1054
|
-
if age_h is not None and age_h >=
|
|
1223
|
+
if age_h is not None and age_h >= stale_weeks * 7 * 24:
|
|
1055
1224
|
out.append({"issue": "stale-candidate-unrediscovered", "id": c["id"],
|
|
1225
|
+
"kind": kind, "threshold_weeks": stale_weeks,
|
|
1226
|
+
"age_days": int(age_h // 24),
|
|
1056
1227
|
"last_seen": str(last)[:10], "rediscovered": int(c.get("rediscovered", 0) or 0)})
|
|
1057
1228
|
# The sibling of the check above, measuring a different emptiness. That one says
|
|
1058
1229
|
# "nobody re-learned this"; this one says "nobody ever asked for this": no inject
|
|
@@ -1060,18 +1231,36 @@ def doctor(memory_dir: str | None = None, targets: list[str] | None = None,
|
|
|
1060
1231
|
# perfectly true and still be worth naming here, because the cost it carries is
|
|
1061
1232
|
# paid by every session that shares the project and the evidence it is worth
|
|
1062
1233
|
# paying is published in the record itself.
|
|
1063
|
-
unused_cutoff = UNUSED_CANDIDATE_WEEKS * 7 * 24
|
|
1064
1234
|
for c in list_candidates():
|
|
1065
1235
|
if c.get("status") != "provisional":
|
|
1066
1236
|
continue
|
|
1067
1237
|
served = int(c.get("injected", 0) or 0) + int(c.get("rediscovered", 0) or 0)
|
|
1068
1238
|
if served:
|
|
1069
1239
|
continue
|
|
1240
|
+
_stale_weeks, unused_weeks, kind = lifetime_for(c.get("kind", ""))
|
|
1070
1241
|
age_h = _hours_since(str(c.get("ts", ""))) if c.get("ts") else None
|
|
1071
|
-
if age_h is not None and age_h >=
|
|
1242
|
+
if age_h is not None and age_h >= unused_weeks * 7 * 24:
|
|
1072
1243
|
out.append({"issue": "unused-candidate-never-served", "id": c["id"],
|
|
1244
|
+
"kind": kind, "threshold_weeks": unused_weeks,
|
|
1073
1245
|
"age_days": int(age_h // 24),
|
|
1074
1246
|
"injected": 0, "rediscovered": 0})
|
|
1247
|
+
# The one input limit that is not a content judgement: a pass is shown only the
|
|
1248
|
+
# newest WINDOW records OF THE SESSION IT CONSOLIDATES, so a session larger than
|
|
1249
|
+
# that carries records no pass will read. Counted per session, on the honest
|
|
1250
|
+
# definition of a pass - `run_sleep`/`distill` hand over one session file - because
|
|
1251
|
+
# the total across sessions compares a number to a limit it was never measured
|
|
1252
|
+
# against: fifty sessions of ten records each are fifty complete passes, not one
|
|
1253
|
+
# buffer of five hundred with four hundred and eighty unread. Reported rather than
|
|
1254
|
+
# raised to a failure, because a large session may be a real backlog.
|
|
1255
|
+
from . import consolidate as _consolidate
|
|
1256
|
+
_window = _consolidate.BUFFER_WINDOW
|
|
1257
|
+
for _session, _count in sorted(buffer_session_counts().items()):
|
|
1258
|
+
if _count > _window:
|
|
1259
|
+
out.append({"issue": "buffer-exceeds-consolidation-window",
|
|
1260
|
+
"session": _session,
|
|
1261
|
+
"buffer_records": _count,
|
|
1262
|
+
"window": _window,
|
|
1263
|
+
"outside_window": _count - _window})
|
|
1075
1264
|
for cid, body in pool:
|
|
1076
1265
|
for path, text in bodies.items():
|
|
1077
1266
|
# MEMORY.md is the index: it restates every entry by design, so scoring
|