pi-sleep-agent 0.8.17 → 0.8.19

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -27,9 +27,35 @@ Sleep adds a reviewable memory cycle to nine selectable coding-agent targets. It
27
27
 
28
28
  - **Automatic capture.** Hooks and the Pi extension collect useful observations while you work.
29
29
  - **Budgeted consolidation.** Sleep runs only after the record and time gates pass.
30
+ - **A named consolidation window.** One pass is shown the newest 120 T0 records. A buffer
31
+ past that is never silent: `distill --dry-run` and the pass itself report how many records
32
+ fell outside the window, `status` says it before the next pass runs, and `doctor` reports
33
+ it as its own finding. Records outside the window are shown to no pass, and the
34
+ per-session prune holds back the kinds that are expensive to re-derive (preferences,
35
+ procedures, environment warnings) instead of moving them out of reach with the file, so
36
+ what leaves T0 is what a pass could have read. The count is printed instead of being
37
+ inferred from the proposal count.
30
38
  - **Review before permanence.** New memories stay provisional until you promote or refuse them.
31
39
  - **Nightly dream cycle.** Mine repeated preferences, consolidate recent observations, and measure memory effects while you are away.
32
40
  - **Measured replay.** Run the same task with memory off and on before deciding whether a memory helps. Each completed pass leaves a one-line quality summary (`helps / hurts / ties`) in `status` and in the nightly report, so the question stays answerable between passes; a pass whose two arms ran the same prompt says it measured nothing instead of reporting zeros.
41
+ - **Kind-aware lifetimes.** A record's `kind` decides how long a quiet stretch is normal.
42
+ `doctor` measures each provisional against its own row instead of one flat cutoff: a fact is
43
+ cheap to re-learn and cheap to drop (4 weeks unrediscovered / 8 weeks unserved), a preference
44
+ is a statement about you and people are slow (13 / 26), a procedure is durable but testable
45
+ (8 / 12), and an environment warning stays true long after nobody touched it (52 / 52). Every
46
+ finding names the kind and the threshold it used, records without a known kind keep the old
47
+ flat default, and nothing here drops or demotes anything.
48
+ - **Kind-aware pruning.** `run_sleep` clears one session's records at a time, and a record whose
49
+ kind is expensive to re-derive is never dropped before a pass could have read it: when a
50
+ session's file is longer than the consolidation window, the preferences, procedures and
51
+ warnings older than the window stay in T0 while everything a pass could have read is archived.
52
+ The prune report carries `kept_count`, `kept_outside_window` and `read_window`, so the debt is
53
+ a number instead of an inference. `prune` with no `--session` still clears the whole buffer
54
+ under the old file-level rule, which is what an explicit clear-out means.
55
+ - **Kind-aware promotion.** `promote` refuses to make an agent's reading of a *preference*
56
+ permanent: a statement about what you want needs your own words, while a fact, procedure or
57
+ warning is confirmed by the human running `promote` (you read it and chose it). The refusal
58
+ message names which rule stopped it, and the existing speculative-source check still runs first.
33
59
  - **Repeated preference detection.** `mine` finds instructions you gave in multiple sessions without a model call.
34
60
  - **Project isolation.** A project's provisional memory is injected only into that project.
35
61
  - **Reviewable permanence.** Each approved memory in your index carries its review date and source, so you can tell when a saved instruction was pinned.
@@ -336,7 +362,9 @@ Run the two dream components separately when you want more control:
336
362
  | `./sleep.py status -v` | ... and the last measured memory quality (`3 helps / 1 hurts / 4 ties from 8 samples`, or "no replay measurement has run yet") |
337
363
  | `./sleep.py mine --dry-run` | Find preferences repeated across sessions |
338
364
  | `./sleep.py nightly --dry-run` | Preview the unattended nightly workflow |
339
- | `./sleep.py doctor` | Find duplicate or overlapping memory records, flag contradicting candidates (same subject, one side carries a correction marker) and print the `supersede <older> <newer>` command to run, flag provisional candidates not rediscovered for 4+ weeks, flag records no session was ever served for 8+ weeks, flag supersede pointers that name a record no longer in the store, and report records parked in quarantine awaiting a human (report only: it never supersedes, drops, merges or un-parks anything itself) |
365
+ | `./sleep.py doctor` | Find duplicate or overlapping memory records, flag contradicting candidates (same subject, one side carries a correction marker) and print the `supersede <older> <newer>` command to run, flag provisional candidates unrediscovered past their kind's threshold (4 weeks for a fact, 13 for a preference, 8 for a procedure, 52 for a warning; kind and threshold appear in the finding), flag records no session was ever served past their kind's unused threshold (8 / 26 / 12 / 52), flag supersede pointers that name a record no longer in the store, report a T0 buffer larger
366
+ than the 120-record consolidation window (with the count left outside it), and report records
367
+ parked in quarantine awaiting a human (report only: it never supersedes, drops, merges or un-parks anything itself) |
340
368
  | `./sleep.py doctor --stale <value>` | Show which approved records still hold a stale value (e.g. a changed URL) |
341
369
  | `./sleep.py doctor --contradictions` | Ask a model to adjudicate shortlisted pairs of provisional records for conflicts (`--budget`, default $0.25); report-only, and each finding carries the `supersede <older> <newer>` command to run |
342
370
  | `./sleep.py doctor --contradictions --tier t2` | Adjudicate pairs of **approved** permanent memory instead of candidates (`--tier both` does both); a T2 finding is labelled `contradicting-memory` and carries the `demote <older>` command, because approved memory is un-approved, never superseded |
@@ -348,6 +376,7 @@ the host agent's own `omo -p` (used when an `omo` binary is on PATH), then `clau
348
376
  `SLEEP_LLM_TRANSPORT=omo|claude|http`. A transport that is not usable on this machine
349
377
  raises instead of silently falling back, and omo is never handed the gateway's model
350
378
  name: it routes through its own config, so only `SLEEP_MODEL` reaches it.
379
+ | `./sleep.py prune [--session <id>] [--older-than <hours>]` | Archive consumed T0 scratch, file-level: with `--session` only that session's records move, with no `--session` the whole buffer is cleared. This is the explicit clear-out, so it moves whole files and ignores kinds; the record-level rule that holds back the kinds no pass could reach belongs to the automatic pass (`run_sleep` prunes one session and reports `kept_count` / `kept_outside_window` / `read_window`) |
351
380
  | `./sleep.py supersede <old> <new>` | Mark `<old>` invalid as of now and `<new>` valid as of now (`valid_until` / `valid_from`), so the window a memory was true is readable from the files |
352
381
  | `./sleep.py inject --project "$PWD"` | Print provisional context for the current project, newest first, each line dated |
353
382
  | `./sleep.py inject --project "$PWD" --query "..."` | Same, but rank records by overlap with the live request so the most relevant survive `--max` |
package/docs/EVIDENCE.md CHANGED
@@ -496,3 +496,90 @@ crossing a process boundary with a prompt-recording fake model),
496
496
  Not changed, and named here so the leftover is not mistaken for coverage: `candidate_pairs()`
497
497
  still orders by `ts` and still makes one model call per pair, and `cmd_doctor` does not call it.
498
498
  It is a duplicate of the ordering rule with a per-pair cost, left alone deliberately.
499
+
500
+
501
+ ## 2026-11-03: the consolidation window is named, and the record left outside it is counted
502
+
503
+ `_fmt_buffer` showed a consolidation pass only the newest 120 T0 records. Everything older was
504
+ never read by any pass, and because pruning only clears what a pass consumed, never cleared either.
505
+ Nothing said so: the night reported `proposals: 2` whether the buffer held 12 records or 300, and
506
+ that number reads as a complete pass.
507
+
508
+ Measured before changing anything, and the honest reading is that the cap had never bitten here:
509
+ `~/.sleep/archive/buffer` holds 31 consumed buffers totalling 259 records, the largest a single
510
+ night ever held was 73, and the live buffer held 1. So this was a latent silent loss, not an
511
+ observed one, and the change is about the report rather than about the limit.
512
+
513
+ What the window now costs to see, all three surfaces checked against a real 127-record buffer in a
514
+ throwaway `SLEEP_HOME`:
515
+
516
+ | surface | without the window | with records outside it |
517
+ |---|---|---|
518
+ | `distill --dry-run` / the pass result | key absent | `outside_window: 7`, `warn: "7 of 127 buffer records fell outside the 120-record window and were not read or pruned"` |
519
+ | `sleep.py status` | T0 line unchanged | `T0 buffer 127 records ... [next pass reads the newest 120; 7 outside the window]` |
520
+ | `sleep.py doctor` | no finding | `{"issue": "buffer-exceeds-consolidation-window", "buffer_records": 127, "window": 120, "outside_window": 7}` |
521
+
522
+ `run_sleep` now persists `buffer_records` and `outside_window` in `state.json` next to `proposals`,
523
+ so a night that left records behind is readable after the fact. The window lives in one place,
524
+ `sleepkit.consolidate.BUFFER_WINDOW`, and the sentence is built by one helper (`_window_warning`,
525
+ wrapped by `_warn_field` so the key is absent rather than empty when there is nothing to report) so
526
+ the preview and the real pass cannot drift. Pinned by `tests/test_cli.py::BufferWindow` (5 tests,
527
+ crossing a process boundary): under the window nothing appears in any of the three surfaces, at
528
+ `WINDOW + 7` the count and the sentence say both numbers, and a pass the record/time gate *refuses*
529
+ still reports the buffer size it did not read.
530
+
531
+ Deliberately NOT changed, since the two are separable and only the reporting was asked for: the pass
532
+ still reads only the newest window and the rest still waits for the next night. Sending a large
533
+ buffer in several passes is a separate change with its own cost (one model call per slice).
534
+
535
+ ## 2026-11-03: a record's kind decides how long a quiet stretch is normal
536
+
537
+ `kind` was carried on every record and read by nothing. All four kinds shared one pair of
538
+ thresholds (`STALE_CANDIDATE_WEEKS = 4.0`, `UNUSED_CANDIDATE_WEEKS = 8.0`), `prune_buffer` moved
539
+ whole files without looking, and `promote` never saw the field. The live T1 pool is 12 candidates -
540
+ 6 preferences, 5 facts, 1 warning - so a statement about the person and a cheap observation were
541
+ being judged by the same clock, and a warning that stops an agent from repeating a failed attempt
542
+ had the shortest life of the four.
543
+
544
+ Three changes, each independently visible:
545
+
546
+ - `store.KIND_LIFETIME_WEEKS` gives each kind its own row - fact 4/8, preference 13/26, procedure
547
+ 8/12, warning 52/52 (unrediscovered/unserved weeks) - and `doctor` now measures against it. Every
548
+ finding carries `kind` and the `threshold_weeks` it used, so a number in the report can be
549
+ checked against the row that produced it. A record whose kind is missing or unknown keeps the
550
+ historical flat default, which is what makes this safe to ship over an existing store.
551
+ - `store.prune_session` is the record-level prune: it holds back the kinds that are expensive to
552
+ re-derive (preference, procedure, warning) instead of moving them out of reach with the file.
553
+ Measured against a clean `SLEEP_HOME`: a 130-record session with 10 durable records leaves 10 on
554
+ disk and archives 120; a 125-record session of preferences alone keeps all 5 that the window
555
+ could not reach and reports `kept_outside_window: 5`, `read_window: 120`.
556
+ - `store.KIND_PROMOTE_NEEDS_USER` holds `preference`, so `promote` refuses to make an agent's
557
+ reading of a preference permanent (`a statement about what the user wants needs the user's own
558
+ words`). Facts, procedures and warnings stay promotable from any source, because the human
559
+ running `promote` is the confirmation - they read the body and chose it - and a warning captured
560
+ from an agent's own failed attempt is the normal way one arrives.
561
+
562
+ Where this stops, deliberately, so the edges are not mistaken for coverage:
563
+
564
+ - The prune hold-back is on `store.prune_session` and `run_sleep` calls it for the session it
565
+ consolidated. The `prune` COMMAND stays file-level: it is the explicit clear-out, `--older-than`
566
+ asks about file age and has no record-level meaning, and routing a typed command into the
567
+ record-level rule would quietly ignore that flag.
568
+ - `KIND_PROMOTE_NEEDS_USER` is one kind, not a general "every kind needs its own confirmation"
569
+ rule. A broader gate would fire on cases where an agent-recorded warning or procedure is
570
+ perfectly legitimate, and an always-on confirmation is a gate nobody reads.
571
+ - `candidate_pairs()` is untouched: it still orders by `ts` and still makes one model call per pair.
572
+ Named here because the file is back in this diff and the leftover should not be read as new.
573
+ - Nothing drops, demotes or expires on the basis of these tables. `doctor` reports; the thresholds
574
+ only decide when it speaks.
575
+
576
+ Pinned by `tests/test_core.py::KindLifetimes` (table matches the documented defaults; an unknown
577
+ kind falls back; a 30-day preference is not stale while a 30-day fact is; a 730-day warning still
578
+ sits below its own 52-week threshold) and `tests/test_core.py::RecordLevelPrune` (unread durable
579
+ records survive, a session with nothing to keep still clears, records inside the window survive
580
+ too, the reported rewrite is checked against the file, an idempotent second run, a missing session
581
+ is not an error), plus `tests/test_cli.py::KindPromotion` (agent preference refused, user
582
+ preference promotable, agent fact and warning promotable, speculative source still named) and the
583
+ per-session `buffer-exceeds-consolidation-window` finding
584
+ (`test_many_small_sessions_are_not_one_large_buffer`: six 20-record sessions are six complete
585
+ passes, not one 120-record buffer, and the doctor says nothing).
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "pi-sleep-agent",
3
- "version": "0.8.17",
3
+ "version": "0.8.19",
4
4
  "description": "A reviewable sleep and dream cycle for Pi agent memory.",
5
5
  "type": "module",
6
6
  "license": "MIT",
package/sleepkit/cli.py CHANGED
@@ -70,7 +70,14 @@ def cmd_status(a) -> int:
70
70
  _p(replay_mod.quality_line())
71
71
  except Exception:
72
72
  pass
73
- _p("T0 buffer %d records / %d files / %d bytes (captured while you worked)" % (bc["records"], bc["files"], bc["bytes"]))
73
+ line = "T0 buffer %d records / %d files / %d bytes (captured while you worked)" % (bc["records"], bc["files"], bc["bytes"])
74
+ # A window smaller than the buffer is the one case where the next pass cannot read
75
+ # everything it is standing on. Say it up front instead of leaving it to be inferred
76
+ # from the proposals count of a night that already happened.
77
+ if bc["records"] > consolidate.BUFFER_WINDOW:
78
+ line += " [next pass reads the newest %d; %d outside the window]" % (
79
+ consolidate.BUFFER_WINDOW, bc["records"] - consolidate.BUFFER_WINDOW)
80
+ _p(line)
74
81
  _p("T1 candidates %d provisional, %d refused, %d total (distilled, shown as unverified context)" % (len(prov), len(ref), len(cands)))
75
82
  _p(" quarantine %d parked (needs human eyes) (held out until you review)" % len(q))
76
83
  retracted = []
@@ -202,6 +209,12 @@ def cmd_supersede(a) -> int:
202
209
 
203
210
 
204
211
  def cmd_prune(a) -> int:
212
+ # `prune` is the explicit-act path, so it keeps the file-level rule it always had:
213
+ # both flags keep working, and "clear this session out" still clears the session.
214
+ # The record-level rule (hold back the kinds no pass could have read) belongs to an
215
+ # automatic pass, which has a read window to measure against and cannot ask; it is
216
+ # what `run_sleep` calls. Routing a typed command into it would also quietly ignore
217
+ # `--older-than`, which asks about file age and has no record-level meaning.
205
218
  rep = store.prune_buffer(older_than_hours=a.older_than, session=a.session)
206
219
  _p(json.dumps(rep, indent=2, sort_keys=True))
207
220
  return 0
@@ -24,7 +24,14 @@ def pending_dir() -> str:
24
24
  return d
25
25
 
26
26
 
27
- def _fmt_buffer(records: list[dict], limit: int = 120) -> str:
27
+ # The window one consolidation pass is shown. It is a display limit, not a policy:
28
+ # whatever falls below it is NOT read by the pass and NOT pruned, and until 0.8.18
29
+ # nothing said so. BUFFER_WINDOW is now named in one place so the count of records
30
+ # left outside the window can be reported instead of being inferred from len().
31
+ BUFFER_WINDOW = 120
32
+
33
+
34
+ def _fmt_buffer(records: list[dict], limit: int = BUFFER_WINDOW) -> str:
28
35
  lines = []
29
36
  for r in records[-limit:]:
30
37
  lines.append("- [%s|%s] %s" % (r.get("kind", "fact"), r.get("source", "?"),
@@ -32,6 +39,41 @@ def _fmt_buffer(records: list[dict], limit: int = 120) -> str:
32
39
  return "\n".join(lines)
33
40
 
34
41
 
42
+ def _window_split(records: list) -> tuple:
43
+ """(sent, skipped): records inside the format window and those left outside it.
44
+
45
+ Only the newest BUFFER_WINDOW records are shown to the model, so a record with an
46
+ older ts is never read by any pass and never pruned (prune only runs on records the
47
+ pass consumed). One distinct session arriving with more than a window's worth of
48
+ captures would therefore be silently dropped from consolidation. The caller reports
49
+ the skipped count so that possibility is visible instead of invisible.
50
+ """
51
+ if len(records) <= BUFFER_WINDOW:
52
+ return list(records), []
53
+ return list(records[-BUFFER_WINDOW:]), list(records[:-BUFFER_WINDOW])
54
+
55
+
56
+ def _warn_field(skipped: int, total: int) -> dict:
57
+ """{"warn": sentence} when records were left out, {} when none were.
58
+
59
+ A key that is always present but sometimes empty makes every caller test the value;
60
+ absent-when-there-is-nothing-to-say keeps `"warn" not in result` meaningful.
61
+ """
62
+ return {"warn": _window_warning(skipped, total)} if skipped > 0 else {}
63
+
64
+
65
+ def _window_warning(skipped: int, total: int) -> str:
66
+ """One sentence for both paths that can leave records outside the window.
67
+
68
+ Shared so `--dry-run` and `--sleep` cannot drift: a preview that says nothing while
69
+ the real pass warns is worse than no preview.
70
+ """
71
+ if skipped <= 0:
72
+ return ""
73
+ return ("%d of %d buffer records fell outside the %d-record window and were "
74
+ "not read or pruned" % (skipped, total, BUFFER_WINDOW))
75
+
76
+
35
77
  def existing_pool() -> str:
36
78
  lines = []
37
79
  for c in store.list_candidates(with_body=True):
@@ -146,8 +188,11 @@ def distill(session: str | None = None, model: str | None = None,
146
188
  return {"ok": True, "reason": "buffer empty", "proposals": 0, "accepted": 0,
147
189
  "consumed": False}
148
190
  projects, unaddressed = _project_groups(recs)
191
+ _sent, outside_window = _window_split(recs)
149
192
  if dry_run:
150
193
  return {"ok": True, "dry_run": True, "buffer_records": len(recs),
194
+ "outside_window": len(outside_window),
195
+ **_warn_field(len(outside_window), len(recs)),
151
196
  "projects": [o for o, _ in projects], "unaddressed": len(unaddressed),
152
197
  "would_send": _fmt_buffer(recs)[:2000]}
153
198
  if not projects:
@@ -183,6 +228,11 @@ def distill(session: str | None = None, model: str | None = None,
183
228
  ing["ok"] = False
184
229
  ing["buffer_records"] = len(recs)
185
230
  ing["unaddressed"] = len(unaddressed)
231
+ # Records the pass was never shown. Not an error and not a bug on its own; the point
232
+ # is that a night which leaves any behind says so, because "proposals: 2" on a
233
+ # 300-record buffer reads as a complete pass and is not one.
234
+ ing["outside_window"] = len(outside_window)
235
+ ing.update(_warn_field(len(outside_window), len(recs)))
186
236
  # "consumed" is what entitles a sleep to prune: the pass produced proposals.
187
237
  # Unaddressed records were never read by any pass, so they must not be pruned.
188
238
  return ing
@@ -427,13 +477,23 @@ def run_sleep(session=None, model=None, budget_usd=1.0, min_records=3,
427
477
  state["last_sleep"] = paths.now()
428
478
  state["sleep_count"] = int(state.get("sleep_count", 0)) + 1
429
479
  state["last_sleep_result"] = {k: result.get(k) for k in
430
- ("proposals", "accepted", "refused", "llm_exit")}
480
+ ("proposals", "accepted", "refused", "llm_exit",
481
+ "buffer_records", "outside_window")}
431
482
  paths.write_state(state)
432
483
  # Synaptic pruning only after a consolidation that actually produced
433
484
  # proposals. Pruning a buffer the model never read would silently discard
434
485
  # the session's observations.
486
+ #
487
+ # Per session, not the whole directory, because the interesting case is one
488
+ # session whose file outgrew the read window: a file-level move takes the
489
+ # oldest records with it, and those are exactly the records no pass read. The
490
+ # session is the one this pass consolidated; with no session given, the buffer
491
+ # is cleared as a whole under the old file-level rule.
435
492
  if result.get("consumed"):
436
- result["pruned"] = store.prune_buffer(older_than_hours=0.0)
493
+ if session:
494
+ result["pruned"] = store.prune_session(session)
495
+ else:
496
+ result["pruned"] = store.prune_buffer(older_than_hours=0.0)
437
497
  else:
438
498
  result["pruned"] = {"skipped": True, "reason": "no proposals consumed"}
439
499
  result["action"] = "ran"
package/sleepkit/store.py CHANGED
@@ -33,6 +33,44 @@ STALE_CANDIDATE_WEEKS = 4.0
33
33
  # with things that were never useful. Report only, like every other doctor line.
34
34
  UNUSED_CANDIDATE_WEEKS = 8.0
35
35
 
36
+ # What the two constants above are a default FOR. A record's kind is not decoration:
37
+ # a warning that nobody re-learned in a month is not the same debt as a preference
38
+ # nobody re-learned in a month, and the flat constants made the four kinds behave
39
+ # identically. Each row is (stale_weeks, unused_weeks) and the rationale travels with
40
+ # the number, because a bare constant invites the next reader to "tidy" it back.
41
+ #
42
+ # fact 4 / 8 the old defaults; a fact is cheap to re-learn and cheap to drop
43
+ # preference 13 / 26 a preference is a statement about a person, and a person is
44
+ # slow. Expiring one after four quiet weeks deletes a true thing
45
+ # procedure 8 / 12 durable but testable: a procedure that stops being exercised is
46
+ # suspect sooner than a stated preference, later than a fact
47
+ # warning 52 / 52 an environment warning ("X does not fire on this engine") stays
48
+ # true long after nobody has touched it, and the cost of
49
+ # re-deriving it is paid in failed attempts
50
+ #
51
+ # Both remain REPORT-ONLY. Nothing here drops, demotes or expires a record; it decides
52
+ # when doctor names one. Use lifetime_for() rather than reading the table directly.
53
+ KIND_LIFETIME_WEEKS = {
54
+ "fact": (4.0, 8.0),
55
+ "preference": (13.0, 26.0),
56
+ "procedure": (8.0, 12.0),
57
+ "warning": (52.0, 52.0),
58
+ }
59
+
60
+
61
+ def lifetime_for(kind: str, default_weeks: float = 0.0) -> tuple[float, float, str]:
62
+ """(stale_weeks, unused_weeks, kind) for one record's kind.
63
+
64
+ An unknown or missing kind gets the historical flat default, so a record written
65
+ by an older version, or by hand, is judged exactly as it was before this table
66
+ existed. The third element is the kind actually used, so a caller can print what
67
+ it judged against without re-deriving the fallback.
68
+ """
69
+ row = KIND_LIFETIME_WEEKS.get(str(kind or "").strip().lower())
70
+ if row:
71
+ return row[0], row[1], str(kind)
72
+ return STALE_CANDIDATE_WEEKS, UNUSED_CANDIDATE_WEEKS, str(kind or "")
73
+
36
74
 
37
75
  def _hours_since(ts: str) -> float | None:
38
76
  """Hours since an ISO timestamp written by paths.now(). Unparseable input
@@ -172,6 +210,22 @@ def buffer_counts() -> dict:
172
210
  "bytes": sum(os.path.getsize(os.path.join(d, f)) for f in files)}
173
211
 
174
212
 
213
+ def buffer_session_counts() -> dict:
214
+ """{session: record count} for the T0 buffer, one entry per file.
215
+
216
+ A pass consolidates ONE session, so this is the count a window means anything
217
+ against: `buffer_counts()` totals every file in the directory and answers a
218
+ different question ("how much scratch is on disk").
219
+ """
220
+ d = paths.buffer_dir()
221
+ out = {}
222
+ for name in sorted(os.listdir(d)):
223
+ if not name.endswith(".jsonl"):
224
+ continue
225
+ out[name[:-6]] = len(read_buffer(name[:-6]))
226
+ return out
227
+
228
+
175
229
  # ------------------------------------------------------- similarity/dedup ---
176
230
 
177
231
  # A proposal that paraphrases an already-refused candidate must not reborn as a
@@ -449,6 +503,21 @@ def slugify(text: str, maxlen: int = 48) -> str:
449
503
  return (s[:maxlen].rstrip("-")) or "note"
450
504
 
451
505
 
506
+ # What a kind is allowed to become permanent on. This is not the same question as
507
+ # "is this record true": a directive-shaped sentence about how the agent should
508
+ # behave is a statement about the PERSON, and only the person can confirm one, so a
509
+ # preference keeps the existing user-provenance requirement on top of the trust
510
+ # check. The other three kinds describe the world, and a human running `promote` is
511
+ # the confirmation: they read the body and chose it.
512
+ #
513
+ # A warning is the case worth naming, because it is the kind most likely to be
514
+ # captured from an agent's own failed attempt with no user in the loop. That is
515
+ # still legitimate - the human reading it is the check - but the refusal message
516
+ # below says which rule stopped it, so "why can I not promote this" never needs
517
+ # source reading.
518
+ KIND_PROMOTE_NEEDS_USER = ("preference",)
519
+
520
+
452
521
  def promote(cid: str, memory_dir: str | None = None, dry_run: bool = False) -> tuple[bool, str]:
453
522
  """T1 -> T2. Explicit human act. Creates a NEW file, never edits one."""
454
523
  got = get_candidate(cid)
@@ -461,6 +530,11 @@ def promote(cid: str, memory_dir: str | None = None, dry_run: bool = False) -> t
461
530
  if src != "user" and meta.get("trust") == "speculative":
462
531
  return False, ("refusing to persist speculative memory from source=%s; "
463
532
  "re-record with the user's own words or run a replay first" % src)
533
+ kind = str(meta.get("kind", "") or "")
534
+ if kind in KIND_PROMOTE_NEEDS_USER and src != "user":
535
+ return False, ("refusing to persist a %s from source=%s: a statement about what "
536
+ "the user wants needs the user's own words, not an agent's reading "
537
+ "of them" % (kind, src))
464
538
  mdir = memory_dir or paths.memory_dir()
465
539
  name = "%s.md" % slugify(body.split(".")[0][:60] or cid)
466
540
  dest = os.path.join(mdir, name)
@@ -810,7 +884,14 @@ def supersede(old_id: str, new_id: str) -> bool:
810
884
 
811
885
  # ------------------------------------------------------------ pruning -------
812
886
  def prune_buffer(older_than_hours: float = 0.0, session: str | None = None) -> dict:
813
- """T0 reset: consumed scratch moves to archive. Nothing is deleted."""
887
+ """T0 reset: consumed scratch moves to archive. Nothing is deleted.
888
+
889
+ File-level and kind-blind by design. It answers "has this session's scratch been
890
+ dealt with?", which is the right question when the caller is an explicit human
891
+ command. It is the WRONG tool for a record-level decision, because a file holds
892
+ records of every kind and moving it takes them all: use prune_session() when a
893
+ record that no pass read must survive.
894
+ """
814
895
  moved = []
815
896
  d = paths.buffer_dir()
816
897
  cutoff = time.time() - older_than_hours * 3600
@@ -829,6 +910,92 @@ def prune_buffer(older_than_hours: float = 0.0, session: str | None = None) -> d
829
910
  return {"moved": moved, "remaining": buffer_counts()}
830
911
 
831
912
 
913
+ # A kind whose records are dropped before consolidation has lost work that cannot be
914
+ # recovered by reading the archive later: the raw text survives, but no pass and no
915
+ # session will look at it again. These are the kinds where that loss is expensive -
916
+ # re-deriving a preference means asking the person again, and re-deriving a warning
917
+ # means paying for the failed attempt a second time. A fact is the one kind whose
918
+ # loss is cheap (it can be re-learned by looking), which is why it is deliberately
919
+ # NOT in this set: the default path must not be the expensive one.
920
+ KEEP_WHEN_UNREAD = ("preference", "procedure", "warning")
921
+
922
+
923
+ def prune_session(session: str, keep_kinds=KEEP_WHEN_UNREAD) -> dict:
924
+ """Prune ONE session's file, holding back records no pass will read.
925
+
926
+ The pass is shown the newest `consolidate.BUFFER_WINDOW` records, so anything older
927
+ than that is read by no pass - and a record no pass reads is also never pruned
928
+ (pruning follows consumption). What a file-level prune does anyway is move the whole
929
+ file, oldest records included, which turns "fell outside the window" from a visible
930
+ backlog into a silent one. This function moves the records the pass could have read
931
+ and keeps the rest, so a withheld record stays where the next pass can still reach it.
932
+
933
+ Imports `consolidate` lazily on purpose: consolidate imports this module at its top,
934
+ so a module-level import here would be circular. The one constant borrowed back is
935
+ the window, kept single-sourced rather than copied.
936
+ """
937
+ from . import consolidate as _consolidate
938
+ d = paths.buffer_dir()
939
+ src = os.path.join(d, session + ".jsonl")
940
+ if not os.path.isfile(src):
941
+ return {"skipped": True, "reason": "no such session", "kept": [], "moved": ""}
942
+ records = []
943
+ with open(src, encoding="utf-8") as fh:
944
+ for line in fh:
945
+ line = line.strip()
946
+ if not line:
947
+ continue
948
+ try:
949
+ records.append(json.loads(line))
950
+ except json.JSONDecodeError:
951
+ continue
952
+ # The window decides what a pass READS, and a pass having read a record is what
953
+ # licenses pruning it. It says nothing about whether the record was worth keeping:
954
+ # a preference inside the window is still a preference, and a prune has no business
955
+ # dropping it because the pass happened to look at it. A record outside the window
956
+ # is the one case where nothing will ever read it again. So the kind filter runs
957
+ # over the WHOLE file rather than over the tail alone, and what survives stays.
958
+ _window = _consolidate.BUFFER_WINDOW
959
+ in_window = records[max(0, len(records) - _window):]
960
+ held = [r for r in records if str(r.get("kind", "")) in keep_kinds]
961
+ held_outside = [r for r in held if r not in in_window]
962
+ if not held:
963
+ return prune_buffer(older_than_hours=0.0, session=session)
964
+ dropped = [r for r in records if str(r.get("kind", "")) not in keep_kinds]
965
+ # The keep file holds ONLY the records whose kind earned survival; the rest go to
966
+ # the archive. An earlier draft appended the in-window records here as well, which
967
+ # silently rewrote the file with its own contents and reported a "kept" count while
968
+ # nothing changed - the exact shape of bug this function exists to prevent, so the
969
+ # rewrite is asserted below rather than assumed.
970
+ keep_sorted = sorted(held, key=lambda r: str(r.get("ts", "")))
971
+ tmp = src + ".tmp"
972
+ with open(tmp, "w", encoding="utf-8") as fh:
973
+ for r in keep_sorted:
974
+ fh.write(json.dumps(r, sort_keys=True) + "\n")
975
+ os.replace(tmp, src)
976
+ after = read_buffer(session)
977
+ if len(after) != len(keep_sorted):
978
+ raise RuntimeError("prune_session rewrite failed: %d kept, %d on disk"
979
+ % (len(keep_sorted), len(after)))
980
+ # The records the pass COULD read, plus the withheld ones whose kind is safe to
981
+ # drop, go to the archive rather than the keep file; nothing is deleted and the
982
+ # archive path is the one prune_buffer already uses.
983
+ dest = os.path.join(paths.archive_dir("buffer"),
984
+ time.strftime("%Y%m%d-") + session + ".jsonl")
985
+ n = 1
986
+ while os.path.exists(dest):
987
+ dest = os.path.join(paths.archive_dir("buffer"),
988
+ time.strftime("%Y%m%d-") + "%s-%d.jsonl" % (session, n))
989
+ n += 1
990
+ with open(dest, "w", encoding="utf-8") as fh:
991
+ for r in dropped:
992
+ fh.write(json.dumps(r, sort_keys=True) + "\n")
993
+ return {"moved": dest, "archived_records": len(dropped),
994
+ "kept": [r["id"] for r in held], "kept_count": len(held),
995
+ "kept_outside_window": len(held_outside), "read_window": len(in_window),
996
+ "remaining": buffer_counts()}
997
+
998
+
832
999
  # Tags are the only index a record carries besides its project, so they are kept
833
1000
  # machine-shaped: lowercase, no spaces, bounded in number and length. A model may
834
1001
  # propose them, which is exactly why they are normalized here rather than trusted.
@@ -1041,7 +1208,6 @@ def doctor(memory_dir: str | None = None, targets: list[str] | None = None,
1041
1208
  "pair": [pool[i][0], pool[j][0]], "similarity": round(sim, 2),
1042
1209
  "older": older, "newer": newer,
1043
1210
  "suggest": "sleep.py supersede %s %s" % (older, newer)})
1044
- cutoff_hours = STALE_CANDIDATE_WEEKS * 7 * 24
1045
1211
  for c in list_candidates():
1046
1212
  if c.get("status") != "provisional":
1047
1213
  continue
@@ -1049,10 +1215,15 @@ def doctor(memory_dir: str | None = None, targets: list[str] | None = None,
1049
1215
  # record's own creation ts. A provisional that is neither being promoted
1050
1216
  # nor re-learned for weeks is noise the queue keeps injecting into every
1051
1217
  # session's context, so the doctor surfaces it (report only, nothing drops).
1218
+ # "Weeks" is per kind: the same quiet month means different things for a
1219
+ # preference and for an environment warning (see KIND_LIFETIME_WEEKS).
1220
+ stale_weeks, _unused_weeks, kind = lifetime_for(c.get("kind", ""))
1052
1221
  last = c.get("rediscovered_at") or c.get("ts")
1053
1222
  age_h = _hours_since(str(last)) if last else None
1054
- if age_h is not None and age_h >= cutoff_hours:
1223
+ if age_h is not None and age_h >= stale_weeks * 7 * 24:
1055
1224
  out.append({"issue": "stale-candidate-unrediscovered", "id": c["id"],
1225
+ "kind": kind, "threshold_weeks": stale_weeks,
1226
+ "age_days": int(age_h // 24),
1056
1227
  "last_seen": str(last)[:10], "rediscovered": int(c.get("rediscovered", 0) or 0)})
1057
1228
  # The sibling of the check above, measuring a different emptiness. That one says
1058
1229
  # "nobody re-learned this"; this one says "nobody ever asked for this": no inject
@@ -1060,18 +1231,36 @@ def doctor(memory_dir: str | None = None, targets: list[str] | None = None,
1060
1231
  # perfectly true and still be worth naming here, because the cost it carries is
1061
1232
  # paid by every session that shares the project and the evidence it is worth
1062
1233
  # paying is published in the record itself.
1063
- unused_cutoff = UNUSED_CANDIDATE_WEEKS * 7 * 24
1064
1234
  for c in list_candidates():
1065
1235
  if c.get("status") != "provisional":
1066
1236
  continue
1067
1237
  served = int(c.get("injected", 0) or 0) + int(c.get("rediscovered", 0) or 0)
1068
1238
  if served:
1069
1239
  continue
1240
+ _stale_weeks, unused_weeks, kind = lifetime_for(c.get("kind", ""))
1070
1241
  age_h = _hours_since(str(c.get("ts", ""))) if c.get("ts") else None
1071
- if age_h is not None and age_h >= unused_cutoff:
1242
+ if age_h is not None and age_h >= unused_weeks * 7 * 24:
1072
1243
  out.append({"issue": "unused-candidate-never-served", "id": c["id"],
1244
+ "kind": kind, "threshold_weeks": unused_weeks,
1073
1245
  "age_days": int(age_h // 24),
1074
1246
  "injected": 0, "rediscovered": 0})
1247
+ # The one input limit that is not a content judgement: a pass is shown only the
1248
+ # newest WINDOW records OF THE SESSION IT CONSOLIDATES, so a session larger than
1249
+ # that carries records no pass will read. Counted per session, on the honest
1250
+ # definition of a pass - `run_sleep`/`distill` hand over one session file - because
1251
+ # the total across sessions compares a number to a limit it was never measured
1252
+ # against: fifty sessions of ten records each are fifty complete passes, not one
1253
+ # buffer of five hundred with four hundred and eighty unread. Reported rather than
1254
+ # raised to a failure, because a large session may be a real backlog.
1255
+ from . import consolidate as _consolidate
1256
+ _window = _consolidate.BUFFER_WINDOW
1257
+ for _session, _count in sorted(buffer_session_counts().items()):
1258
+ if _count > _window:
1259
+ out.append({"issue": "buffer-exceeds-consolidation-window",
1260
+ "session": _session,
1261
+ "buffer_records": _count,
1262
+ "window": _window,
1263
+ "outside_window": _count - _window})
1075
1264
  for cid, body in pool:
1076
1265
  for path, text in bodies.items():
1077
1266
  # MEMORY.md is the index: it restates every entry by design, so scoring