pi-sleep-agent 0.8.18 → 0.8.19
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +25 -3
- package/docs/EVIDENCE.md +53 -1
- package/package.json +1 -1
- package/sleepkit/cli.py +6 -0
- package/sleepkit/consolidate.py +10 -1
- package/sleepkit/store.py +193 -16
package/README.md
CHANGED
|
@@ -30,11 +30,32 @@ Sleep adds a reviewable memory cycle to nine selectable coding-agent targets. It
|
|
|
30
30
|
- **A named consolidation window.** One pass is shown the newest 120 T0 records. A buffer
|
|
31
31
|
past that is never silent: `distill --dry-run` and the pass itself report how many records
|
|
32
32
|
fell outside the window, `status` says it before the next pass runs, and `doctor` reports
|
|
33
|
-
it as its own finding. Records outside the window are shown to no pass and
|
|
34
|
-
prune
|
|
33
|
+
it as its own finding. Records outside the window are shown to no pass, and the
|
|
34
|
+
per-session prune holds back the kinds that are expensive to re-derive (preferences,
|
|
35
|
+
procedures, environment warnings) instead of moving them out of reach with the file, so
|
|
36
|
+
what leaves T0 is what a pass could have read. The count is printed instead of being
|
|
37
|
+
inferred from the proposal count.
|
|
35
38
|
- **Review before permanence.** New memories stay provisional until you promote or refuse them.
|
|
36
39
|
- **Nightly dream cycle.** Mine repeated preferences, consolidate recent observations, and measure memory effects while you are away.
|
|
37
40
|
- **Measured replay.** Run the same task with memory off and on before deciding whether a memory helps. Each completed pass leaves a one-line quality summary (`helps / hurts / ties`) in `status` and in the nightly report, so the question stays answerable between passes; a pass whose two arms ran the same prompt says it measured nothing instead of reporting zeros.
|
|
41
|
+
- **Kind-aware lifetimes.** A record's `kind` decides how long a quiet stretch is normal.
|
|
42
|
+
`doctor` measures each provisional against its own row instead of one flat cutoff: a fact is
|
|
43
|
+
cheap to re-learn and cheap to drop (4 weeks unrediscovered / 8 weeks unserved), a preference
|
|
44
|
+
is a statement about you and people are slow (13 / 26), a procedure is durable but testable
|
|
45
|
+
(8 / 12), and an environment warning stays true long after nobody touched it (52 / 52). Every
|
|
46
|
+
finding names the kind and the threshold it used, records without a known kind keep the old
|
|
47
|
+
flat default, and nothing here drops or demotes anything.
|
|
48
|
+
- **Kind-aware pruning.** `run_sleep` clears one session's records at a time, and a record whose
|
|
49
|
+
kind is expensive to re-derive is never dropped before a pass could have read it: when a
|
|
50
|
+
session's file is longer than the consolidation window, the preferences, procedures and
|
|
51
|
+
warnings older than the window stay in T0 while everything a pass could have read is archived.
|
|
52
|
+
The prune report carries `kept_count`, `kept_outside_window` and `read_window`, so the debt is
|
|
53
|
+
a number instead of an inference. `prune` with no `--session` still clears the whole buffer
|
|
54
|
+
under the old file-level rule, which is what an explicit clear-out means.
|
|
55
|
+
- **Kind-aware promotion.** `promote` refuses to make an agent's reading of a *preference*
|
|
56
|
+
permanent: a statement about what you want needs your own words, while a fact, procedure or
|
|
57
|
+
warning is confirmed by the human running `promote` (you read it and chose it). The refusal
|
|
58
|
+
message names which rule stopped it, and the existing speculative-source check still runs first.
|
|
38
59
|
- **Repeated preference detection.** `mine` finds instructions you gave in multiple sessions without a model call.
|
|
39
60
|
- **Project isolation.** A project's provisional memory is injected only into that project.
|
|
40
61
|
- **Reviewable permanence.** Each approved memory in your index carries its review date and source, so you can tell when a saved instruction was pinned.
|
|
@@ -341,7 +362,7 @@ Run the two dream components separately when you want more control:
|
|
|
341
362
|
| `./sleep.py status -v` | ... and the last measured memory quality (`3 helps / 1 hurts / 4 ties from 8 samples`, or "no replay measurement has run yet") |
|
|
342
363
|
| `./sleep.py mine --dry-run` | Find preferences repeated across sessions |
|
|
343
364
|
| `./sleep.py nightly --dry-run` | Preview the unattended nightly workflow |
|
|
344
|
-
| `./sleep.py doctor` | Find duplicate or overlapping memory records, flag contradicting candidates (same subject, one side carries a correction marker) and print the `supersede <older> <newer>` command to run, flag provisional candidates
|
|
365
|
+
| `./sleep.py doctor` | Find duplicate or overlapping memory records, flag contradicting candidates (same subject, one side carries a correction marker) and print the `supersede <older> <newer>` command to run, flag provisional candidates unrediscovered past their kind's threshold (4 weeks for a fact, 13 for a preference, 8 for a procedure, 52 for a warning; kind and threshold appear in the finding), flag records no session was ever served past their kind's unused threshold (8 / 26 / 12 / 52), flag supersede pointers that name a record no longer in the store, report a T0 buffer larger
|
|
345
366
|
than the 120-record consolidation window (with the count left outside it), and report records
|
|
346
367
|
parked in quarantine awaiting a human (report only: it never supersedes, drops, merges or un-parks anything itself) |
|
|
347
368
|
| `./sleep.py doctor --stale <value>` | Show which approved records still hold a stale value (e.g. a changed URL) |
|
|
@@ -355,6 +376,7 @@ the host agent's own `omo -p` (used when an `omo` binary is on PATH), then `clau
|
|
|
355
376
|
`SLEEP_LLM_TRANSPORT=omo|claude|http`. A transport that is not usable on this machine
|
|
356
377
|
raises instead of silently falling back, and omo is never handed the gateway's model
|
|
357
378
|
name: it routes through its own config, so only `SLEEP_MODEL` reaches it.
|
|
379
|
+
| `./sleep.py prune [--session <id>] [--older-than <hours>]` | Archive consumed T0 scratch, file-level: with `--session` only that session's records move, with no `--session` the whole buffer is cleared. This is the explicit clear-out, so it moves whole files and ignores kinds; the record-level rule that holds back the kinds no pass could reach belongs to the automatic pass (`run_sleep` prunes one session and reports `kept_count` / `kept_outside_window` / `read_window`) |
|
|
358
380
|
| `./sleep.py supersede <old> <new>` | Mark `<old>` invalid as of now and `<new>` valid as of now (`valid_until` / `valid_from`), so the window a memory was true is readable from the files |
|
|
359
381
|
| `./sleep.py inject --project "$PWD"` | Print provisional context for the current project, newest first, each line dated |
|
|
360
382
|
| `./sleep.py inject --project "$PWD" --query "..."` | Same, but rank records by overlap with the live request so the most relevant survive `--max` |
|
package/docs/EVIDENCE.md
CHANGED
|
@@ -530,4 +530,56 @@ still reports the buffer size it did not read.
|
|
|
530
530
|
|
|
531
531
|
Deliberately NOT changed, since the two are separable and only the reporting was asked for: the pass
|
|
532
532
|
still reads only the newest window and the rest still waits for the next night. Sending a large
|
|
533
|
-
buffer in several passes is a separate change with its own cost (one model call per slice).
|
|
533
|
+
buffer in several passes is a separate change with its own cost (one model call per slice).
|
|
534
|
+
|
|
535
|
+
## 2026-11-03: a record's kind decides how long a quiet stretch is normal
|
|
536
|
+
|
|
537
|
+
`kind` was carried on every record and read by nothing. All four kinds shared one pair of
|
|
538
|
+
thresholds (`STALE_CANDIDATE_WEEKS = 4.0`, `UNUSED_CANDIDATE_WEEKS = 8.0`), `prune_buffer` moved
|
|
539
|
+
whole files without looking, and `promote` never saw the field. The live T1 pool is 12 candidates -
|
|
540
|
+
6 preferences, 5 facts, 1 warning - so a statement about the person and a cheap observation were
|
|
541
|
+
being judged by the same clock, and a warning that stops an agent from repeating a failed attempt
|
|
542
|
+
had the shortest life of the four.
|
|
543
|
+
|
|
544
|
+
Three changes, each independently visible:
|
|
545
|
+
|
|
546
|
+
- `store.KIND_LIFETIME_WEEKS` gives each kind its own row - fact 4/8, preference 13/26, procedure
|
|
547
|
+
8/12, warning 52/52 (unrediscovered/unserved weeks) - and `doctor` now measures against it. Every
|
|
548
|
+
finding carries `kind` and the `threshold_weeks` it used, so a number in the report can be
|
|
549
|
+
checked against the row that produced it. A record whose kind is missing or unknown keeps the
|
|
550
|
+
historical flat default, which is what makes this safe to ship over an existing store.
|
|
551
|
+
- `store.prune_session` is the record-level prune: it holds back the kinds that are expensive to
|
|
552
|
+
re-derive (preference, procedure, warning) instead of moving them out of reach with the file.
|
|
553
|
+
Measured against a clean `SLEEP_HOME`: a 130-record session with 10 durable records leaves 10 on
|
|
554
|
+
disk and archives 120; a 125-record session of preferences alone keeps all 5 that the window
|
|
555
|
+
could not reach and reports `kept_outside_window: 5`, `read_window: 120`.
|
|
556
|
+
- `store.KIND_PROMOTE_NEEDS_USER` holds `preference`, so `promote` refuses to make an agent's
|
|
557
|
+
reading of a preference permanent (`a statement about what the user wants needs the user's own
|
|
558
|
+
words`). Facts, procedures and warnings stay promotable from any source, because the human
|
|
559
|
+
running `promote` is the confirmation - they read the body and chose it - and a warning captured
|
|
560
|
+
from an agent's own failed attempt is the normal way one arrives.
|
|
561
|
+
|
|
562
|
+
Where this stops, deliberately, so the edges are not mistaken for coverage:
|
|
563
|
+
|
|
564
|
+
- The prune hold-back is on `store.prune_session` and `run_sleep` calls it for the session it
|
|
565
|
+
consolidated. The `prune` COMMAND stays file-level: it is the explicit clear-out, `--older-than`
|
|
566
|
+
asks about file age and has no record-level meaning, and routing a typed command into the
|
|
567
|
+
record-level rule would quietly ignore that flag.
|
|
568
|
+
- `KIND_PROMOTE_NEEDS_USER` is one kind, not a general "every kind needs its own confirmation"
|
|
569
|
+
rule. A broader gate would fire on cases where an agent-recorded warning or procedure is
|
|
570
|
+
perfectly legitimate, and an always-on confirmation is a gate nobody reads.
|
|
571
|
+
- `candidate_pairs()` is untouched: it still orders by `ts` and still makes one model call per pair.
|
|
572
|
+
Named here because the file is back in this diff and the leftover should not be read as new.
|
|
573
|
+
- Nothing drops, demotes or expires on the basis of these tables. `doctor` reports; the thresholds
|
|
574
|
+
only decide when it speaks.
|
|
575
|
+
|
|
576
|
+
Pinned by `tests/test_core.py::KindLifetimes` (table matches the documented defaults; an unknown
|
|
577
|
+
kind falls back; a 30-day preference is not stale while a 30-day fact is; a 730-day warning still
|
|
578
|
+
sits below its own 52-week threshold) and `tests/test_core.py::RecordLevelPrune` (unread durable
|
|
579
|
+
records survive, a session with nothing to keep still clears, records inside the window survive
|
|
580
|
+
too, the reported rewrite is checked against the file, an idempotent second run, a missing session
|
|
581
|
+
is not an error), plus `tests/test_cli.py::KindPromotion` (agent preference refused, user
|
|
582
|
+
preference promotable, agent fact and warning promotable, speculative source still named) and the
|
|
583
|
+
per-session `buffer-exceeds-consolidation-window` finding
|
|
584
|
+
(`test_many_small_sessions_are_not_one_large_buffer`: six 20-record sessions are six complete
|
|
585
|
+
passes, not one 120-record buffer, and the doctor says nothing).
|
package/package.json
CHANGED
package/sleepkit/cli.py
CHANGED
|
@@ -209,6 +209,12 @@ def cmd_supersede(a) -> int:
|
|
|
209
209
|
|
|
210
210
|
|
|
211
211
|
def cmd_prune(a) -> int:
|
|
212
|
+
# `prune` is the explicit-act path, so it keeps the file-level rule it always had:
|
|
213
|
+
# both flags keep working, and "clear this session out" still clears the session.
|
|
214
|
+
# The record-level rule (hold back the kinds no pass could have read) belongs to an
|
|
215
|
+
# automatic pass, which has a read window to measure against and cannot ask; it is
|
|
216
|
+
# what `run_sleep` calls. Routing a typed command into it would also quietly ignore
|
|
217
|
+
# `--older-than`, which asks about file age and has no record-level meaning.
|
|
212
218
|
rep = store.prune_buffer(older_than_hours=a.older_than, session=a.session)
|
|
213
219
|
_p(json.dumps(rep, indent=2, sort_keys=True))
|
|
214
220
|
return 0
|
package/sleepkit/consolidate.py
CHANGED
|
@@ -483,8 +483,17 @@ def run_sleep(session=None, model=None, budget_usd=1.0, min_records=3,
|
|
|
483
483
|
# Synaptic pruning only after a consolidation that actually produced
|
|
484
484
|
# proposals. Pruning a buffer the model never read would silently discard
|
|
485
485
|
# the session's observations.
|
|
486
|
+
#
|
|
487
|
+
# Per session, not the whole directory, because the interesting case is one
|
|
488
|
+
# session whose file outgrew the read window: a file-level move takes the
|
|
489
|
+
# oldest records with it, and those are exactly the records no pass read. The
|
|
490
|
+
# session is the one this pass consolidated; with no session given, the buffer
|
|
491
|
+
# is cleared as a whole under the old file-level rule.
|
|
486
492
|
if result.get("consumed"):
|
|
487
|
-
|
|
493
|
+
if session:
|
|
494
|
+
result["pruned"] = store.prune_session(session)
|
|
495
|
+
else:
|
|
496
|
+
result["pruned"] = store.prune_buffer(older_than_hours=0.0)
|
|
488
497
|
else:
|
|
489
498
|
result["pruned"] = {"skipped": True, "reason": "no proposals consumed"}
|
|
490
499
|
result["action"] = "ran"
|
package/sleepkit/store.py
CHANGED
|
@@ -33,6 +33,44 @@ STALE_CANDIDATE_WEEKS = 4.0
|
|
|
33
33
|
# with things that were never useful. Report only, like every other doctor line.
|
|
34
34
|
UNUSED_CANDIDATE_WEEKS = 8.0
|
|
35
35
|
|
|
36
|
+
# What the two constants above are a default FOR. A record's kind is not decoration:
|
|
37
|
+
# a warning that nobody re-learned in a month is not the same debt as a preference
|
|
38
|
+
# nobody re-learned in a month, and the flat constants made the four kinds behave
|
|
39
|
+
# identically. Each row is (stale_weeks, unused_weeks) and the rationale travels with
|
|
40
|
+
# the number, because a bare constant invites the next reader to "tidy" it back.
|
|
41
|
+
#
|
|
42
|
+
# fact 4 / 8 the old defaults; a fact is cheap to re-learn and cheap to drop
|
|
43
|
+
# preference 13 / 26 a preference is a statement about a person, and a person is
|
|
44
|
+
# slow. Expiring one after four quiet weeks deletes a true thing
|
|
45
|
+
# procedure 8 / 12 durable but testable: a procedure that stops being exercised is
|
|
46
|
+
# suspect sooner than a stated preference, later than a fact
|
|
47
|
+
# warning 52 / 52 an environment warning ("X does not fire on this engine") stays
|
|
48
|
+
# true long after nobody has touched it, and the cost of
|
|
49
|
+
# re-deriving it is paid in failed attempts
|
|
50
|
+
#
|
|
51
|
+
# Both remain REPORT-ONLY. Nothing here drops, demotes or expires a record; it decides
|
|
52
|
+
# when doctor names one. Use lifetime_for() rather than reading the table directly.
|
|
53
|
+
KIND_LIFETIME_WEEKS = {
|
|
54
|
+
"fact": (4.0, 8.0),
|
|
55
|
+
"preference": (13.0, 26.0),
|
|
56
|
+
"procedure": (8.0, 12.0),
|
|
57
|
+
"warning": (52.0, 52.0),
|
|
58
|
+
}
|
|
59
|
+
|
|
60
|
+
|
|
61
|
+
def lifetime_for(kind: str, default_weeks: float = 0.0) -> tuple[float, float, str]:
|
|
62
|
+
"""(stale_weeks, unused_weeks, kind) for one record's kind.
|
|
63
|
+
|
|
64
|
+
An unknown or missing kind gets the historical flat default, so a record written
|
|
65
|
+
by an older version, or by hand, is judged exactly as it was before this table
|
|
66
|
+
existed. The third element is the kind actually used, so a caller can print what
|
|
67
|
+
it judged against without re-deriving the fallback.
|
|
68
|
+
"""
|
|
69
|
+
row = KIND_LIFETIME_WEEKS.get(str(kind or "").strip().lower())
|
|
70
|
+
if row:
|
|
71
|
+
return row[0], row[1], str(kind)
|
|
72
|
+
return STALE_CANDIDATE_WEEKS, UNUSED_CANDIDATE_WEEKS, str(kind or "")
|
|
73
|
+
|
|
36
74
|
|
|
37
75
|
def _hours_since(ts: str) -> float | None:
|
|
38
76
|
"""Hours since an ISO timestamp written by paths.now(). Unparseable input
|
|
@@ -172,6 +210,22 @@ def buffer_counts() -> dict:
|
|
|
172
210
|
"bytes": sum(os.path.getsize(os.path.join(d, f)) for f in files)}
|
|
173
211
|
|
|
174
212
|
|
|
213
|
+
def buffer_session_counts() -> dict:
|
|
214
|
+
"""{session: record count} for the T0 buffer, one entry per file.
|
|
215
|
+
|
|
216
|
+
A pass consolidates ONE session, so this is the count a window means anything
|
|
217
|
+
against: `buffer_counts()` totals every file in the directory and answers a
|
|
218
|
+
different question ("how much scratch is on disk").
|
|
219
|
+
"""
|
|
220
|
+
d = paths.buffer_dir()
|
|
221
|
+
out = {}
|
|
222
|
+
for name in sorted(os.listdir(d)):
|
|
223
|
+
if not name.endswith(".jsonl"):
|
|
224
|
+
continue
|
|
225
|
+
out[name[:-6]] = len(read_buffer(name[:-6]))
|
|
226
|
+
return out
|
|
227
|
+
|
|
228
|
+
|
|
175
229
|
# ------------------------------------------------------- similarity/dedup ---
|
|
176
230
|
|
|
177
231
|
# A proposal that paraphrases an already-refused candidate must not reborn as a
|
|
@@ -449,6 +503,21 @@ def slugify(text: str, maxlen: int = 48) -> str:
|
|
|
449
503
|
return (s[:maxlen].rstrip("-")) or "note"
|
|
450
504
|
|
|
451
505
|
|
|
506
|
+
# What a kind is allowed to become permanent on. This is not the same question as
|
|
507
|
+
# "is this record true": a directive-shaped sentence about how the agent should
|
|
508
|
+
# behave is a statement about the PERSON, and only the person can confirm one, so a
|
|
509
|
+
# preference keeps the existing user-provenance requirement on top of the trust
|
|
510
|
+
# check. The other three kinds describe the world, and a human running `promote` is
|
|
511
|
+
# the confirmation: they read the body and chose it.
|
|
512
|
+
#
|
|
513
|
+
# A warning is the case worth naming, because it is the kind most likely to be
|
|
514
|
+
# captured from an agent's own failed attempt with no user in the loop. That is
|
|
515
|
+
# still legitimate - the human reading it is the check - but the refusal message
|
|
516
|
+
# below says which rule stopped it, so "why can I not promote this" never needs
|
|
517
|
+
# source reading.
|
|
518
|
+
KIND_PROMOTE_NEEDS_USER = ("preference",)
|
|
519
|
+
|
|
520
|
+
|
|
452
521
|
def promote(cid: str, memory_dir: str | None = None, dry_run: bool = False) -> tuple[bool, str]:
|
|
453
522
|
"""T1 -> T2. Explicit human act. Creates a NEW file, never edits one."""
|
|
454
523
|
got = get_candidate(cid)
|
|
@@ -461,6 +530,11 @@ def promote(cid: str, memory_dir: str | None = None, dry_run: bool = False) -> t
|
|
|
461
530
|
if src != "user" and meta.get("trust") == "speculative":
|
|
462
531
|
return False, ("refusing to persist speculative memory from source=%s; "
|
|
463
532
|
"re-record with the user's own words or run a replay first" % src)
|
|
533
|
+
kind = str(meta.get("kind", "") or "")
|
|
534
|
+
if kind in KIND_PROMOTE_NEEDS_USER and src != "user":
|
|
535
|
+
return False, ("refusing to persist a %s from source=%s: a statement about what "
|
|
536
|
+
"the user wants needs the user's own words, not an agent's reading "
|
|
537
|
+
"of them" % (kind, src))
|
|
464
538
|
mdir = memory_dir or paths.memory_dir()
|
|
465
539
|
name = "%s.md" % slugify(body.split(".")[0][:60] or cid)
|
|
466
540
|
dest = os.path.join(mdir, name)
|
|
@@ -810,7 +884,14 @@ def supersede(old_id: str, new_id: str) -> bool:
|
|
|
810
884
|
|
|
811
885
|
# ------------------------------------------------------------ pruning -------
|
|
812
886
|
def prune_buffer(older_than_hours: float = 0.0, session: str | None = None) -> dict:
|
|
813
|
-
"""T0 reset: consumed scratch moves to archive. Nothing is deleted.
|
|
887
|
+
"""T0 reset: consumed scratch moves to archive. Nothing is deleted.
|
|
888
|
+
|
|
889
|
+
File-level and kind-blind by design. It answers "has this session's scratch been
|
|
890
|
+
dealt with?", which is the right question when the caller is an explicit human
|
|
891
|
+
command. It is the WRONG tool for a record-level decision, because a file holds
|
|
892
|
+
records of every kind and moving it takes them all: use prune_session() when a
|
|
893
|
+
record that no pass read must survive.
|
|
894
|
+
"""
|
|
814
895
|
moved = []
|
|
815
896
|
d = paths.buffer_dir()
|
|
816
897
|
cutoff = time.time() - older_than_hours * 3600
|
|
@@ -829,6 +910,92 @@ def prune_buffer(older_than_hours: float = 0.0, session: str | None = None) -> d
|
|
|
829
910
|
return {"moved": moved, "remaining": buffer_counts()}
|
|
830
911
|
|
|
831
912
|
|
|
913
|
+
# A kind whose records are dropped before consolidation has lost work that cannot be
|
|
914
|
+
# recovered by reading the archive later: the raw text survives, but no pass and no
|
|
915
|
+
# session will look at it again. These are the kinds where that loss is expensive -
|
|
916
|
+
# re-deriving a preference means asking the person again, and re-deriving a warning
|
|
917
|
+
# means paying for the failed attempt a second time. A fact is the one kind whose
|
|
918
|
+
# loss is cheap (it can be re-learned by looking), which is why it is deliberately
|
|
919
|
+
# NOT in this set: the default path must not be the expensive one.
|
|
920
|
+
KEEP_WHEN_UNREAD = ("preference", "procedure", "warning")
|
|
921
|
+
|
|
922
|
+
|
|
923
|
+
def prune_session(session: str, keep_kinds=KEEP_WHEN_UNREAD) -> dict:
|
|
924
|
+
"""Prune ONE session's file, holding back records no pass will read.
|
|
925
|
+
|
|
926
|
+
The pass is shown the newest `consolidate.BUFFER_WINDOW` records, so anything older
|
|
927
|
+
than that is read by no pass - and a record no pass reads is also never pruned
|
|
928
|
+
(pruning follows consumption). What a file-level prune does anyway is move the whole
|
|
929
|
+
file, oldest records included, which turns "fell outside the window" from a visible
|
|
930
|
+
backlog into a silent one. This function moves the records the pass could have read
|
|
931
|
+
and keeps the rest, so a withheld record stays where the next pass can still reach it.
|
|
932
|
+
|
|
933
|
+
Imports `consolidate` lazily on purpose: consolidate imports this module at its top,
|
|
934
|
+
so a module-level import here would be circular. The one constant borrowed back is
|
|
935
|
+
the window, kept single-sourced rather than copied.
|
|
936
|
+
"""
|
|
937
|
+
from . import consolidate as _consolidate
|
|
938
|
+
d = paths.buffer_dir()
|
|
939
|
+
src = os.path.join(d, session + ".jsonl")
|
|
940
|
+
if not os.path.isfile(src):
|
|
941
|
+
return {"skipped": True, "reason": "no such session", "kept": [], "moved": ""}
|
|
942
|
+
records = []
|
|
943
|
+
with open(src, encoding="utf-8") as fh:
|
|
944
|
+
for line in fh:
|
|
945
|
+
line = line.strip()
|
|
946
|
+
if not line:
|
|
947
|
+
continue
|
|
948
|
+
try:
|
|
949
|
+
records.append(json.loads(line))
|
|
950
|
+
except json.JSONDecodeError:
|
|
951
|
+
continue
|
|
952
|
+
# The window decides what a pass READS, and a pass having read a record is what
|
|
953
|
+
# licenses pruning it. It says nothing about whether the record was worth keeping:
|
|
954
|
+
# a preference inside the window is still a preference, and a prune has no business
|
|
955
|
+
# dropping it because the pass happened to look at it. A record outside the window
|
|
956
|
+
# is the one case where nothing will ever read it again. So the kind filter runs
|
|
957
|
+
# over the WHOLE file rather than over the tail alone, and what survives stays.
|
|
958
|
+
_window = _consolidate.BUFFER_WINDOW
|
|
959
|
+
in_window = records[max(0, len(records) - _window):]
|
|
960
|
+
held = [r for r in records if str(r.get("kind", "")) in keep_kinds]
|
|
961
|
+
held_outside = [r for r in held if r not in in_window]
|
|
962
|
+
if not held:
|
|
963
|
+
return prune_buffer(older_than_hours=0.0, session=session)
|
|
964
|
+
dropped = [r for r in records if str(r.get("kind", "")) not in keep_kinds]
|
|
965
|
+
# The keep file holds ONLY the records whose kind earned survival; the rest go to
|
|
966
|
+
# the archive. An earlier draft appended the in-window records here as well, which
|
|
967
|
+
# silently rewrote the file with its own contents and reported a "kept" count while
|
|
968
|
+
# nothing changed - the exact shape of bug this function exists to prevent, so the
|
|
969
|
+
# rewrite is asserted below rather than assumed.
|
|
970
|
+
keep_sorted = sorted(held, key=lambda r: str(r.get("ts", "")))
|
|
971
|
+
tmp = src + ".tmp"
|
|
972
|
+
with open(tmp, "w", encoding="utf-8") as fh:
|
|
973
|
+
for r in keep_sorted:
|
|
974
|
+
fh.write(json.dumps(r, sort_keys=True) + "\n")
|
|
975
|
+
os.replace(tmp, src)
|
|
976
|
+
after = read_buffer(session)
|
|
977
|
+
if len(after) != len(keep_sorted):
|
|
978
|
+
raise RuntimeError("prune_session rewrite failed: %d kept, %d on disk"
|
|
979
|
+
% (len(keep_sorted), len(after)))
|
|
980
|
+
# The records the pass COULD read, plus the withheld ones whose kind is safe to
|
|
981
|
+
# drop, go to the archive rather than the keep file; nothing is deleted and the
|
|
982
|
+
# archive path is the one prune_buffer already uses.
|
|
983
|
+
dest = os.path.join(paths.archive_dir("buffer"),
|
|
984
|
+
time.strftime("%Y%m%d-") + session + ".jsonl")
|
|
985
|
+
n = 1
|
|
986
|
+
while os.path.exists(dest):
|
|
987
|
+
dest = os.path.join(paths.archive_dir("buffer"),
|
|
988
|
+
time.strftime("%Y%m%d-") + "%s-%d.jsonl" % (session, n))
|
|
989
|
+
n += 1
|
|
990
|
+
with open(dest, "w", encoding="utf-8") as fh:
|
|
991
|
+
for r in dropped:
|
|
992
|
+
fh.write(json.dumps(r, sort_keys=True) + "\n")
|
|
993
|
+
return {"moved": dest, "archived_records": len(dropped),
|
|
994
|
+
"kept": [r["id"] for r in held], "kept_count": len(held),
|
|
995
|
+
"kept_outside_window": len(held_outside), "read_window": len(in_window),
|
|
996
|
+
"remaining": buffer_counts()}
|
|
997
|
+
|
|
998
|
+
|
|
832
999
|
# Tags are the only index a record carries besides its project, so they are kept
|
|
833
1000
|
# machine-shaped: lowercase, no spaces, bounded in number and length. A model may
|
|
834
1001
|
# propose them, which is exactly why they are normalized here rather than trusted.
|
|
@@ -1041,7 +1208,6 @@ def doctor(memory_dir: str | None = None, targets: list[str] | None = None,
|
|
|
1041
1208
|
"pair": [pool[i][0], pool[j][0]], "similarity": round(sim, 2),
|
|
1042
1209
|
"older": older, "newer": newer,
|
|
1043
1210
|
"suggest": "sleep.py supersede %s %s" % (older, newer)})
|
|
1044
|
-
cutoff_hours = STALE_CANDIDATE_WEEKS * 7 * 24
|
|
1045
1211
|
for c in list_candidates():
|
|
1046
1212
|
if c.get("status") != "provisional":
|
|
1047
1213
|
continue
|
|
@@ -1049,10 +1215,15 @@ def doctor(memory_dir: str | None = None, targets: list[str] | None = None,
|
|
|
1049
1215
|
# record's own creation ts. A provisional that is neither being promoted
|
|
1050
1216
|
# nor re-learned for weeks is noise the queue keeps injecting into every
|
|
1051
1217
|
# session's context, so the doctor surfaces it (report only, nothing drops).
|
|
1218
|
+
# "Weeks" is per kind: the same quiet month means different things for a
|
|
1219
|
+
# preference and for an environment warning (see KIND_LIFETIME_WEEKS).
|
|
1220
|
+
stale_weeks, _unused_weeks, kind = lifetime_for(c.get("kind", ""))
|
|
1052
1221
|
last = c.get("rediscovered_at") or c.get("ts")
|
|
1053
1222
|
age_h = _hours_since(str(last)) if last else None
|
|
1054
|
-
if age_h is not None and age_h >=
|
|
1223
|
+
if age_h is not None and age_h >= stale_weeks * 7 * 24:
|
|
1055
1224
|
out.append({"issue": "stale-candidate-unrediscovered", "id": c["id"],
|
|
1225
|
+
"kind": kind, "threshold_weeks": stale_weeks,
|
|
1226
|
+
"age_days": int(age_h // 24),
|
|
1056
1227
|
"last_seen": str(last)[:10], "rediscovered": int(c.get("rediscovered", 0) or 0)})
|
|
1057
1228
|
# The sibling of the check above, measuring a different emptiness. That one says
|
|
1058
1229
|
# "nobody re-learned this"; this one says "nobody ever asked for this": no inject
|
|
@@ -1060,30 +1231,36 @@ def doctor(memory_dir: str | None = None, targets: list[str] | None = None,
|
|
|
1060
1231
|
# perfectly true and still be worth naming here, because the cost it carries is
|
|
1061
1232
|
# paid by every session that shares the project and the evidence it is worth
|
|
1062
1233
|
# paying is published in the record itself.
|
|
1063
|
-
unused_cutoff = UNUSED_CANDIDATE_WEEKS * 7 * 24
|
|
1064
1234
|
for c in list_candidates():
|
|
1065
1235
|
if c.get("status") != "provisional":
|
|
1066
1236
|
continue
|
|
1067
1237
|
served = int(c.get("injected", 0) or 0) + int(c.get("rediscovered", 0) or 0)
|
|
1068
1238
|
if served:
|
|
1069
1239
|
continue
|
|
1240
|
+
_stale_weeks, unused_weeks, kind = lifetime_for(c.get("kind", ""))
|
|
1070
1241
|
age_h = _hours_since(str(c.get("ts", ""))) if c.get("ts") else None
|
|
1071
|
-
if age_h is not None and age_h >=
|
|
1242
|
+
if age_h is not None and age_h >= unused_weeks * 7 * 24:
|
|
1072
1243
|
out.append({"issue": "unused-candidate-never-served", "id": c["id"],
|
|
1244
|
+
"kind": kind, "threshold_weeks": unused_weeks,
|
|
1073
1245
|
"age_days": int(age_h // 24),
|
|
1074
1246
|
"injected": 0, "rediscovered": 0})
|
|
1075
|
-
# The one input limit that is not a content judgement:
|
|
1076
|
-
#
|
|
1077
|
-
#
|
|
1078
|
-
#
|
|
1079
|
-
#
|
|
1247
|
+
# The one input limit that is not a content judgement: a pass is shown only the
|
|
1248
|
+
# newest WINDOW records OF THE SESSION IT CONSOLIDATES, so a session larger than
|
|
1249
|
+
# that carries records no pass will read. Counted per session, on the honest
|
|
1250
|
+
# definition of a pass - `run_sleep`/`distill` hand over one session file - because
|
|
1251
|
+
# the total across sessions compares a number to a limit it was never measured
|
|
1252
|
+
# against: fifty sessions of ten records each are fifty complete passes, not one
|
|
1253
|
+
# buffer of five hundred with four hundred and eighty unread. Reported rather than
|
|
1254
|
+
# raised to a failure, because a large session may be a real backlog.
|
|
1080
1255
|
from . import consolidate as _consolidate
|
|
1081
|
-
|
|
1082
|
-
|
|
1083
|
-
|
|
1084
|
-
|
|
1085
|
-
|
|
1086
|
-
|
|
1256
|
+
_window = _consolidate.BUFFER_WINDOW
|
|
1257
|
+
for _session, _count in sorted(buffer_session_counts().items()):
|
|
1258
|
+
if _count > _window:
|
|
1259
|
+
out.append({"issue": "buffer-exceeds-consolidation-window",
|
|
1260
|
+
"session": _session,
|
|
1261
|
+
"buffer_records": _count,
|
|
1262
|
+
"window": _window,
|
|
1263
|
+
"outside_window": _count - _window})
|
|
1087
1264
|
for cid, body in pool:
|
|
1088
1265
|
for path, text in bodies.items():
|
|
1089
1266
|
# MEMORY.md is the index: it restates every entry by design, so scoring
|