pi-sleep-agent 0.8.18 → 0.8.20

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -30,11 +30,39 @@ Sleep adds a reviewable memory cycle to nine selectable coding-agent targets. It
30
30
  - **A named consolidation window.** One pass is shown the newest 120 T0 records. A buffer
31
31
  past that is never silent: `distill --dry-run` and the pass itself report how many records
32
32
  fell outside the window, `status` says it before the next pass runs, and `doctor` reports
33
- it as its own finding. Records outside the window are shown to no pass and cleared by no
34
- prune, which is why the number is printed instead of being inferred from the proposal count.
33
+ it as its own finding. Records outside the window are shown to no pass, and the
34
+ per-session prune holds back the kinds that are expensive to re-derive (preferences,
35
+ procedures, environment warnings) instead of moving them out of reach with the file, so
36
+ what leaves T0 is what a pass could have read. The count is printed instead of being
37
+ inferred from the proposal count.
38
+ - **A pass that reads its own report.** `doctor` findings are not decoration: `sleep` and
39
+ `nightly` score them, and a problem that gets worse while it waits (a backlog past the read
40
+ window, an unconfirmed memory still being served as context) waives the `--min-interval` check
41
+ so the pass runs now instead of on the clock. The record floor always holds, the reason in the
42
+ report names the interval as waived and the score that waived it, and `--min-urgency 0` (or
43
+ `--no-doctor-gate`) restores the flat policy exactly. Scoring reads the hygiene report only;
44
+ it makes no model call, so the unattended path spends nothing new.
35
45
  - **Review before permanence.** New memories stay provisional until you promote or refuse them.
36
46
  - **Nightly dream cycle.** Mine repeated preferences, consolidate recent observations, and measure memory effects while you are away.
37
47
  - **Measured replay.** Run the same task with memory off and on before deciding whether a memory helps. Each completed pass leaves a one-line quality summary (`helps / hurts / ties`) in `status` and in the nightly report, so the question stays answerable between passes; a pass whose two arms ran the same prompt says it measured nothing instead of reporting zeros.
48
+ - **Kind-aware lifetimes.** A record's `kind` decides how long a quiet stretch is normal.
49
+ `doctor` measures each provisional against its own row instead of one flat cutoff: a fact is
50
+ cheap to re-learn and cheap to drop (4 weeks unrediscovered / 8 weeks unserved), a preference
51
+ is a statement about you and people are slow (13 / 26), a procedure is durable but testable
52
+ (8 / 12), and an environment warning stays true long after nobody touched it (52 / 52). Every
53
+ finding names the kind and the threshold it used, records without a known kind keep the old
54
+ flat default, and nothing here drops or demotes anything.
55
+ - **Kind-aware pruning.** `run_sleep` clears one session's records at a time, and a record whose
56
+ kind is expensive to re-derive is never dropped before a pass could have read it: when a
57
+ session's file is longer than the consolidation window, the preferences, procedures and
58
+ warnings older than the window stay in T0 while everything a pass could have read is archived.
59
+ The prune report carries `kept_count`, `kept_outside_window` and `read_window`, so the debt is
60
+ a number instead of an inference. `prune` with no `--session` still clears the whole buffer
61
+ under the old file-level rule, which is what an explicit clear-out means.
62
+ - **Kind-aware promotion.** `promote` refuses to make an agent's reading of a *preference*
63
+ permanent: a statement about what you want needs your own words, while a fact, procedure or
64
+ warning is confirmed by the human running `promote` (you read it and chose it). The refusal
65
+ message names which rule stopped it, and the existing speculative-source check still runs first.
38
66
  - **Repeated preference detection.** `mine` finds instructions you gave in multiple sessions without a model call.
39
67
  - **Project isolation.** A project's provisional memory is injected only into that project.
40
68
  - **Reviewable permanence.** Each approved memory in your index carries its review date and source, so you can tell when a saved instruction was pinned.
@@ -341,7 +369,8 @@ Run the two dream components separately when you want more control:
341
369
  | `./sleep.py status -v` | ... and the last measured memory quality (`3 helps / 1 hurts / 4 ties from 8 samples`, or "no replay measurement has run yet") |
342
370
  | `./sleep.py mine --dry-run` | Find preferences repeated across sessions |
343
371
  | `./sleep.py nightly --dry-run` | Preview the unattended nightly workflow |
344
- | `./sleep.py doctor` | Find duplicate or overlapping memory records, flag contradicting candidates (same subject, one side carries a correction marker) and print the `supersede <older> <newer>` command to run, flag provisional candidates not rediscovered for 4+ weeks, flag records no session was ever served for 8+ weeks, flag supersede pointers that name a record no longer in the store, report a T0 buffer larger
372
+ | `./sleep.py sleep --min-urgency <points>` | Score the hygiene findings and let them waive `--min-interval` (default 2: a growing backlog or a stale candidate clears it, a dangling link does not). `--min-urgency 0` or `--no-doctor-gate` keeps the flat policy, and the record floor (`--min-records`) is never waived |
373
+ | `./sleep.py doctor` | Find duplicate or overlapping memory records, flag contradicting candidates (same subject, one side carries a correction marker) and print the `supersede <older> <newer>` command to run, flag provisional candidates unrediscovered past their kind's threshold (4 weeks for a fact, 13 for a preference, 8 for a procedure, 52 for a warning; kind and threshold appear in the finding), flag records no session was ever served past their kind's unused threshold (8 / 26 / 12 / 52), flag supersede pointers that name a record no longer in the store, report a T0 buffer larger
345
374
  than the 120-record consolidation window (with the count left outside it), and report records
346
375
  parked in quarantine awaiting a human (report only: it never supersedes, drops, merges or un-parks anything itself) |
347
376
  | `./sleep.py doctor --stale <value>` | Show which approved records still hold a stale value (e.g. a changed URL) |
@@ -355,6 +384,7 @@ the host agent's own `omo -p` (used when an `omo` binary is on PATH), then `clau
355
384
  `SLEEP_LLM_TRANSPORT=omo|claude|http`. A transport that is not usable on this machine
356
385
  raises instead of silently falling back, and omo is never handed the gateway's model
357
386
  name: it routes through its own config, so only `SLEEP_MODEL` reaches it.
387
+ | `./sleep.py prune [--session <id>] [--older-than <hours>]` | Archive consumed T0 scratch, file-level: with `--session` only that session's records move, with no `--session` the whole buffer is cleared. This is the explicit clear-out, so it moves whole files and ignores kinds; the record-level rule that holds back the kinds no pass could reach belongs to the automatic pass (`run_sleep` prunes one session and reports `kept_count` / `kept_outside_window` / `read_window`) |
358
388
  | `./sleep.py supersede <old> <new>` | Mark `<old>` invalid as of now and `<new>` valid as of now (`valid_until` / `valid_from`), so the window a memory was true is readable from the files |
359
389
  | `./sleep.py inject --project "$PWD"` | Print provisional context for the current project, newest first, each line dated |
360
390
  | `./sleep.py inject --project "$PWD" --query "..."` | Same, but rank records by overlap with the live request so the most relevant survive `--max` |
package/docs/EVIDENCE.md CHANGED
@@ -530,4 +530,109 @@ still reports the buffer size it did not read.
530
530
 
531
531
  Deliberately NOT changed, since the two are separable and only the reporting was asked for: the pass
532
532
  still reads only the newest window and the rest still waits for the next night. Sending a large
533
- buffer in several passes is a separate change with its own cost (one model call per slice).
533
+ buffer in several passes is a separate change with its own cost (one model call per slice).
534
+
535
+ ## 2026-11-03: a record's kind decides how long a quiet stretch is normal
536
+
537
+ `kind` was carried on every record and read by nothing. All four kinds shared one pair of
538
+ thresholds (`STALE_CANDIDATE_WEEKS = 4.0`, `UNUSED_CANDIDATE_WEEKS = 8.0`), `prune_buffer` moved
539
+ whole files without looking, and `promote` never saw the field. The live T1 pool is 12 candidates -
540
+ 6 preferences, 5 facts, 1 warning - so a statement about the person and a cheap observation were
541
+ being judged by the same clock, and a warning that stops an agent from repeating a failed attempt
542
+ had the shortest life of the four.
543
+
544
+ Three changes, each independently visible:
545
+
546
+ - `store.KIND_LIFETIME_WEEKS` gives each kind its own row - fact 4/8, preference 13/26, procedure
547
+ 8/12, warning 52/52 (unrediscovered/unserved weeks) - and `doctor` now measures against it. Every
548
+ finding carries `kind` and the `threshold_weeks` it used, so a number in the report can be
549
+ checked against the row that produced it. A record whose kind is missing or unknown keeps the
550
+ historical flat default, which is what makes this safe to ship over an existing store.
551
+ - `store.prune_session` is the record-level prune: it holds back the kinds that are expensive to
552
+ re-derive (preference, procedure, warning) instead of moving them out of reach with the file.
553
+ Measured against a clean `SLEEP_HOME`: a 130-record session with 10 durable records leaves 10 on
554
+ disk and archives 120; a 125-record session of preferences alone keeps all 5 that the window
555
+ could not reach and reports `kept_outside_window: 5`, `read_window: 120`.
556
+ - `store.KIND_PROMOTE_NEEDS_USER` holds `preference`, so `promote` refuses to make an agent's
557
+ reading of a preference permanent (`a statement about what the user wants needs the user's own
558
+ words`). Facts, procedures and warnings stay promotable from any source, because the human
559
+ running `promote` is the confirmation - they read the body and chose it - and a warning captured
560
+ from an agent's own failed attempt is the normal way one arrives.
561
+
562
+ Where this stops, deliberately, so the edges are not mistaken for coverage:
563
+
564
+ - The prune hold-back is on `store.prune_session` and `run_sleep` calls it for the session it
565
+ consolidated. The `prune` COMMAND stays file-level: it is the explicit clear-out, `--older-than`
566
+ asks about file age and has no record-level meaning, and routing a typed command into the
567
+ record-level rule would quietly ignore that flag.
568
+ - `KIND_PROMOTE_NEEDS_USER` is one kind, not a general "every kind needs its own confirmation"
569
+ rule. A broader gate would fire on cases where an agent-recorded warning or procedure is
570
+ perfectly legitimate, and an always-on confirmation is a gate nobody reads.
571
+ - `candidate_pairs()` is untouched: it still orders by `ts` and still makes one model call per pair.
572
+ Named here because the file is back in this diff and the leftover should not be read as new.
573
+ - Nothing drops, demotes or expires on the basis of these tables. `doctor` reports; the thresholds
574
+ only decide when it speaks.
575
+
576
+ Pinned by `tests/test_core.py::KindLifetimes` (table matches the documented defaults; an unknown
577
+ kind falls back; a 30-day preference is not stale while a 30-day fact is; a 730-day warning still
578
+ sits below its own 52-week threshold) and `tests/test_core.py::RecordLevelPrune` (unread durable
579
+ records survive, a session with nothing to keep still clears, records inside the window survive
580
+ too, the reported rewrite is checked against the file, an idempotent second run, a missing session
581
+ is not an error), plus `tests/test_cli.py::KindPromotion` (agent preference refused, user
582
+ preference promotable, agent fact and warning promotable, speculative source still named) and the
583
+ per-session `buffer-exceeds-consolidation-window` finding
584
+ (`test_many_small_sessions_are_not_one_large_buffer`: six 20-record sessions are six complete
585
+ passes, not one 120-record buffer, and the doctor says nothing).
586
+
587
+
588
+ ## 2026-11-03: the hygiene report feeds the gate instead of sitting beside it
589
+
590
+ Section 6 of `docs/TMP-sleep-framework-gaps.md` named the gap: "zamanlama sabit eşikler ...
591
+ `doctor`'ın bulguları rapor, tetikleyici değil". That was exact. `doctor` computed a backlog past
592
+ the read window, stale candidates that no longer earn their place, and an armed-and-never-fired
593
+ schedule; `sleep_gate` then asked only "3 records and 6 hours since the last pass". The report and
594
+ the decision lived in the same process and never spoke.
595
+
596
+ `consolidate.URGENCY_WEIGHTS` puts a value on each finding a doctor can emit - a growing backlog 3,
597
+ a stale candidate or an unserved record 2, a dangling link or a quarantine 1 - and
598
+ `consolidate.urgency(findings)` sums them into a score with a reason that names the largest
599
+ contributor and the counts. Everything not in the table weighs 0, so a finding added later cannot
600
+ open a gate by accident; that is deliberate, not an oversight.
601
+
602
+ `sleep_gate` takes `urgency_score` and `min_urgency`. A score at or above the bar waives the
603
+ interval check ONLY: the record floor is never waived, because "enough happened to be worth
604
+ reading" is a different question from "a problem is waiting". Both defaults reproduce the old
605
+ behaviour byte for byte (`urgency_score=0` and `min_urgency=0` each do it independently), which is
606
+ what makes this safe to land under a hook or a cron entry that passes neither.
607
+
608
+ Measured on a clean `SLEEP_HOME`, 130 buffered records and one 200-day-old preference, with
609
+ `last_sleep` set to now so the interval blocks a second pass:
610
+
611
+ - `sleep --dry-run` -> `would-run`, `urgency 7`, reason `ran early: urgency 7 >= 2 | last sleep 0.0h
612
+ ago, minimum interval 6h waived`
613
+ - `sleep --dry-run --no-doctor-gate` -> `skipped`, `urgency 0`, reason `last sleep 0.0h ago, minimum
614
+ interval 6h`
615
+ - `nightly --dry-run --no-mine` -> the doctor report and the sleep gate carry the SAME three issues
616
+ (`stale-candidate-unrediscovered`, `unused-candidate-never-served`,
617
+ `buffer-exceeds-consolidation-window`); the gate is handed the report the night already computed
618
+ rather than reading candidate frontmatter a second time.
619
+
620
+ Cost: `burst` is unchanged. Scoring calls `store.doctor()` with no needle, which reads candidate
621
+ frontmatter and the T0 counts and makes no model call; only `doctor --contradictions` spends, and
622
+ that is not on this path. `nightly` reuses its own report, so the unattended night reads once.
623
+
624
+ Deliberately not done, so the edges are not mistaken for coverage:
625
+
626
+ - No decay, no deletion, no demotion. The score decides WHEN a pass runs, never what a record is
627
+ worth. Dropping records is still a human act, and `prune` is still the explicit clear-out.
628
+ - The record floor is not waivable. A buffer with two records and a month of backlog waits.
629
+ - `replay` keeps its own clock (`--replay-interval`): what replay measures is whether a memory
630
+ helps, and a hygiene finding says nothing about that.
631
+ - One finding still weighs more than a person's time than it should: `quarantine-awaiting-review`
632
+ is scored 1 because a night cannot resolve a quarantine, so a quarantine alone will not wake a
633
+ pass. A person who wants it to can pass `--min-urgency 1`.
634
+
635
+ Pinned by `tests/test_core.py::T` (`test_urgency_weights_come_from_the_table_and_name_it`,
636
+ `test_every_weight_the_doctor_can_emit_is_a_known_row`,
637
+ `test_urgency_waives_the_interval_but_never_the_record_floor`,
638
+ `test_run_sleep_reports_why_it_ran_early`).
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "pi-sleep-agent",
3
- "version": "0.8.18",
3
+ "version": "0.8.20",
4
4
  "description": "A reviewable sleep and dream cycle for Pi agent memory.",
5
5
  "type": "module",
6
6
  "license": "MIT",
package/sleepkit/cli.py CHANGED
@@ -46,7 +46,8 @@ def cmd_sleep(a) -> int:
46
46
  rep = consolidate.run_sleep(session=a.session, model=a.model, budget_usd=a.budget,
47
47
  min_records=a.min_records,
48
48
  min_interval_hours=a.min_interval,
49
- force=a.force, dry_run=a.dry_run)
49
+ force=a.force, dry_run=a.dry_run,
50
+ min_urgency=0 if a.no_doctor_gate else a.min_urgency)
50
51
  _p(json.dumps(rep, indent=2, sort_keys=True, default=str))
51
52
  if rep.get("action") == "ran":
52
53
  return 0 if rep.get("ok") is not False else 3
@@ -209,6 +210,12 @@ def cmd_supersede(a) -> int:
209
210
 
210
211
 
211
212
  def cmd_prune(a) -> int:
213
+ # `prune` is the explicit-act path, so it keeps the file-level rule it always had:
214
+ # both flags keep working, and "clear this session out" still clears the session.
215
+ # The record-level rule (hold back the kinds no pass could have read) belongs to an
216
+ # automatic pass, which has a read window to measure against and cannot ask; it is
217
+ # what `run_sleep` calls. Routing a typed command into it would also quietly ignore
218
+ # `--older-than`, which asks about file age and has no record-level meaning.
212
219
  rep = store.prune_buffer(older_than_hours=a.older_than, session=a.session)
213
220
  _p(json.dumps(rep, indent=2, sort_keys=True))
214
221
  return 0
@@ -451,12 +458,18 @@ def cmd_nightly(a) -> int:
451
458
  # The hygiene report lands in the night regardless of the gates: it is
452
459
  # read-only, so the worst an unattended pass can do is surface a stale
453
460
  # value (e.g. a changed deployment URL) for a person to act on later.
454
- rep["doctor"] = {"issues": store.doctor(needle=a.stale)}
461
+ issues = store.doctor(needle=a.stale)
462
+ rep["doctor"] = {"issues": issues}
463
+ # The hygiene report is handed to the gate rather than recomputed: same read, one
464
+ # night, and it is what lets a finding move the pass off the flat interval. With
465
+ # --no-doctor-gate the gate gets the old policy and the report is unaffected.
455
466
  rep["sleep"] = consolidate.run_sleep(session=a.session, model=a.model,
456
467
  budget_usd=a.budget,
457
468
  min_records=a.min_records,
458
469
  min_interval_hours=a.min_interval,
459
- force=a.force, dry_run=a.dry_run)
470
+ force=a.force, dry_run=a.dry_run,
471
+ findings=issues,
472
+ min_urgency=0 if a.no_doctor_gate else a.min_urgency)
460
473
  rep["replay"] = replay_mod.nightly(tasks_path=a.replay_tasks, model=a.model,
461
474
  budget=a.replay_budget,
462
475
  limit=a.replay_limit,
@@ -666,6 +679,12 @@ def main(argv=None) -> int:
666
679
  sl.add_argument("--min-records", type=int, default=3, dest="min_records")
667
680
  sl.add_argument("--min-interval", type=float, default=6.0, dest="min_interval",
668
681
  help="hours between sleeps")
682
+ sl.add_argument("--min-urgency", type=float, default=consolidate.URGENCY_DEFAULT,
683
+ dest="min_urgency",
684
+ help="doctor findings worth at least this many points skip the "
685
+ "interval check (0 = flat policy, no hygiene scan)")
686
+ sl.add_argument("--no-doctor-gate", action="store_true", dest="no_doctor_gate",
687
+ help="do not score the hygiene report; the interval is never waived")
669
688
  sl.add_argument("--force", action="store_true")
670
689
  sl.add_argument("--dry-run", action="store_true")
671
690
  sl.set_defaults(fn=cmd_sleep)
@@ -742,6 +761,12 @@ def main(argv=None) -> int:
742
761
  nt.add_argument("--min-records", type=int, default=3, dest="min_records")
743
762
  nt.add_argument("--min-interval", type=float, default=6.0, dest="min_interval",
744
763
  help="hours between consolidations")
764
+ nt.add_argument("--min-urgency", type=float, default=consolidate.URGENCY_DEFAULT,
765
+ dest="min_urgency",
766
+ help="hygiene findings worth at least this many points skip the "
767
+ "interval check (0 = flat policy)")
768
+ nt.add_argument("--no-doctor-gate", action="store_true", dest="no_doctor_gate",
769
+ help="score the hygiene report but never waive the interval")
745
770
  nt.add_argument("--force", action="store_true")
746
771
  nt.add_argument("--dry-run", action="store_true")
747
772
  nt.add_argument("--replay-interval", type=float, default=168.0, dest="replay_interval",
@@ -446,28 +446,137 @@ def hours_since(ts):
446
446
  return (datetime.now(dt.tzinfo) - dt).total_seconds() / 3600.0
447
447
 
448
448
 
449
- def sleep_gate(records: int, min_records: int, min_interval_hours: float) -> tuple[bool, str]:
449
+ # What a doctor finding is worth as a reason to run NOW. The thresholds above answer
450
+ # "is it time"; these answer "is it needed", and until now nothing consumed them: the
451
+ # pass ran on a flat 6h/3-record policy while `doctor` printed a backlog, stale
452
+ # candidates and an armed-but-never-fired schedule into a report nobody wired to a
453
+ # decision. Each row is (weight, label) and the label travels with the number so a
454
+ # report can never name a weight that did not produce it.
455
+ #
456
+ # buffer-exceeds-consolidation-window 3 the problem gets worse every hour it waits
457
+ # stale-candidate-unrediscovered 2 an unconfirmed memory is still being
458
+ # injected as context; leaving it is not
459
+ # neutral, it is a wrong thing kept alive
460
+ # unused-candidate-never-served 2 a record no session ever saw is a memory
461
+ # that has not paid for itself either way
462
+ # broken-supersede-link 1 a real defect, but it does not grow
463
+ # quarantine-awaiting-review 1 needs a person, not a pass; scoring it low
464
+ # keeps a night from running for something a
465
+ # night cannot fix
466
+ # contradicting-candidates 1 adjudication is a human act; the pass may
467
+ # surface it, but it cannot settle it
468
+ # near-duplicate-candidates 1
469
+ # candidate-duplicates-memory 1
470
+ # stale-value-in-memory 1
471
+ #
472
+ # Everything else (duplicate-heading-in-file and any future issue) weighs 0 on
473
+ # purpose: a new finding must be given a weight deliberately, and an unknown one must
474
+ # never be able to open a gate by accident.
475
+ URGENCY_WEIGHTS = {
476
+ "buffer-exceeds-consolidation-window": (3, "the backlog grows until a pass runs"),
477
+ "stale-candidate-unrediscovered": (2, "unconfirmed memory is still served as context"),
478
+ "unused-candidate-never-served": (2, "a record no session was ever served"),
479
+ "broken-supersede-link": (1, "a link dangles but does not grow"),
480
+ "quarantine-awaiting-review": (1, "needs a person, not a pass"),
481
+ "contradicting-candidates": (1, "the pass can surface it, only a person can settle it"),
482
+ "near-duplicate-candidates": (1, "hygiene, cheap to leave for the next pass"),
483
+ "candidate-duplicates-memory": (1, "hygiene, cheap to leave for the next pass"),
484
+ "stale-value-in-memory": (1, "a stale fact, but the record still holds its subject"),
485
+ }
486
+
487
+ # The default bar a finding must clear to waive the interval. 2 means "a problem that
488
+ # grows, or a memory that is being served while unconfirmed"; 0 turns the whole
489
+ # mechanism off and restores the flat policy exactly.
490
+ URGENCY_DEFAULT = 2.0
491
+
492
+
493
+ def urgency(findings: list) -> tuple[int, str]:
494
+ """(score, reason) for a doctor report. Read-only, no model call.
495
+
496
+ Deliberately a SUM of at most a few points rather than a max: three stale
497
+ candidates and a growing backlog is a different night from one dangling link, and
498
+ the number is reported so the difference is inspectable instead of implied. The
499
+ reason names the largest single contributor, because that is the one a person
500
+ reading the log would act on first.
501
+ """
502
+ total = 0
503
+ best_w = 0
504
+ best_issue = ""
505
+ counts: dict = {}
506
+ for f in findings or []:
507
+ issue = str((f or {}).get("issue", ""))
508
+ w, _label = URGENCY_WEIGHTS.get(issue, (0, ""))
509
+ if not w:
510
+ continue
511
+ total += w
512
+ counts[issue] = counts.get(issue, 0) + 1
513
+ # Ties go to the first row in table order, which is also weight order, so the
514
+ # reason is stable across runs and does not depend on dict iteration order.
515
+ if w > best_w:
516
+ best_w, best_issue = w, issue
517
+ if not total:
518
+ return 0, ""
519
+ parts = ", ".join("%s x%d" % (k, v) for k, v in sorted(counts.items()))
520
+ return total, "%s (+%d; %s)" % (best_issue, total, parts)
521
+
522
+
523
+ def sleep_gate(records: int, min_records: int, min_interval_hours: float,
524
+ urgency_score: int = 0, min_urgency: float = URGENCY_DEFAULT) -> tuple[bool, str]:
525
+ """Is it time? The record floor always holds; urgency can waive only the interval.
526
+
527
+ `urgency_score <= 0` and `min_urgency <= 0` both reproduce the historical behaviour
528
+ byte for byte, which is what makes this safe to add under callers that do not pass
529
+ either (a hook, a hand-typed `sleep`, an older script). A waive is reported as
530
+ such, because "policy satisfied" and "ran early because 4 stale candidates were
531
+ still being injected" are different facts about the night and a log has to tell
532
+ them apart.
533
+ """
450
534
  state = paths.read_state()
451
535
  last = state.get("last_sleep")
452
536
  if records < min_records:
453
537
  return False, f"buffer has {records} record(s), needs {min_records}"
454
538
  elapsed = hours_since(last)
455
539
  if elapsed is not None and elapsed < min_interval_hours:
456
- return False, f"last sleep {elapsed:.1f}h ago, minimum interval {min_interval_hours:g}h"
540
+ if urgency_score > 0 and min_urgency > 0 and urgency_score >= min_urgency:
541
+ return True, ("ran early: urgency %d >= %g | last sleep %.1fh ago, "
542
+ "minimum interval %gh waived"
543
+ % (urgency_score, min_urgency, elapsed, min_interval_hours))
544
+ return False, f"last sleep {elapsed:.1f}h ago, minimum interval {min_interval_hours:g}h"
457
545
  return True, "policy satisfied"
458
546
 
459
547
 
460
548
  def run_sleep(session=None, model=None, budget_usd=1.0, min_records=3,
461
- min_interval_hours=6.0, force=False, dry_run=False):
549
+ min_interval_hours=6.0, force=False, dry_run=False,
550
+ findings=None, min_urgency=URGENCY_DEFAULT):
462
551
  """The entry point a hook or a cron job calls. Cheap when there is nothing
463
- to consolidate: one stat and one JSON read, no model call."""
552
+ to consolidate: one stat, one JSON read and a read-only hygiene scan, no model call.
553
+
554
+ `findings` is a caller-supplied `store.doctor()` report, and passing one is the
555
+ point of the argument: `nightly` already computes it for its own report, and
556
+ reading candidate frontmatter twice in one night would be waste dressed as
557
+ thoroughness. When it is None and `min_urgency` is on, the scan runs here, which
558
+ costs file reads and no API call (only `doctor --contradictions` spends, and that
559
+ is not on this path).
560
+ """
464
561
  records = store.buffer_counts()["records"]
465
- ok, reason = sleep_gate(records, min_records, min_interval_hours)
562
+ score = 0
563
+ detail = ""
564
+ if min_urgency and min_urgency > 0:
565
+ if findings is None:
566
+ findings = store.doctor()
567
+ score, detail = urgency(findings)
568
+ ok, reason = sleep_gate(records, min_records, min_interval_hours,
569
+ urgency_score=score, min_urgency=min_urgency)
570
+ gate = {"min_records": min_records, "min_interval_hours": min_interval_hours,
571
+ "min_urgency": min_urgency, "urgency": score}
572
+ if detail:
573
+ gate["urgency_detail"] = detail
466
574
  if not ok and not force:
467
- return {"action": "skipped", "reason": reason, "buffer_records": records}
575
+ return {"action": "skipped", "reason": reason, "buffer_records": records,
576
+ "gate": gate}
468
577
  if dry_run:
469
578
  return {"action": "would-run", "reason": reason if ok else reason + " (forced)",
470
- "buffer_records": records}
579
+ "buffer_records": records, "gate": gate}
471
580
  if not acquire_lock():
472
581
  return {"action": "locked", "reason": "another consolidation is running",
473
582
  "buffer_records": records}
@@ -478,13 +587,22 @@ def run_sleep(session=None, model=None, budget_usd=1.0, min_records=3,
478
587
  state["sleep_count"] = int(state.get("sleep_count", 0)) + 1
479
588
  state["last_sleep_result"] = {k: result.get(k) for k in
480
589
  ("proposals", "accepted", "refused", "llm_exit",
481
- "buffer_records", "outside_window")}
590
+ "buffer_records", "outside_window", "gate")}
482
591
  paths.write_state(state)
483
592
  # Synaptic pruning only after a consolidation that actually produced
484
593
  # proposals. Pruning a buffer the model never read would silently discard
485
594
  # the session's observations.
595
+ #
596
+ # Per session, not the whole directory, because the interesting case is one
597
+ # session whose file outgrew the read window: a file-level move takes the
598
+ # oldest records with it, and those are exactly the records no pass read. The
599
+ # session is the one this pass consolidated; with no session given, the buffer
600
+ # is cleared as a whole under the old file-level rule.
486
601
  if result.get("consumed"):
487
- result["pruned"] = store.prune_buffer(older_than_hours=0.0)
602
+ if session:
603
+ result["pruned"] = store.prune_session(session)
604
+ else:
605
+ result["pruned"] = store.prune_buffer(older_than_hours=0.0)
488
606
  else:
489
607
  result["pruned"] = {"skipped": True, "reason": "no proposals consumed"}
490
608
  result["action"] = "ran"
package/sleepkit/store.py CHANGED
@@ -33,6 +33,44 @@ STALE_CANDIDATE_WEEKS = 4.0
33
33
  # with things that were never useful. Report only, like every other doctor line.
34
34
  UNUSED_CANDIDATE_WEEKS = 8.0
35
35
 
36
+ # What the two constants above are a default FOR. A record's kind is not decoration:
37
+ # a warning that nobody re-learned in a month is not the same debt as a preference
38
+ # nobody re-learned in a month, and the flat constants made the four kinds behave
39
+ # identically. Each row is (stale_weeks, unused_weeks) and the rationale travels with
40
+ # the number, because a bare constant invites the next reader to "tidy" it back.
41
+ #
42
+ # fact 4 / 8 the old defaults; a fact is cheap to re-learn and cheap to drop
43
+ # preference 13 / 26 a preference is a statement about a person, and a person is
44
+ # slow. Expiring one after four quiet weeks deletes a true thing
45
+ # procedure 8 / 12 durable but testable: a procedure that stops being exercised is
46
+ # suspect sooner than a stated preference, later than a fact
47
+ # warning 52 / 52 an environment warning ("X does not fire on this engine") stays
48
+ # true long after nobody has touched it, and the cost of
49
+ # re-deriving it is paid in failed attempts
50
+ #
51
+ # Both remain REPORT-ONLY. Nothing here drops, demotes or expires a record; it decides
52
+ # when doctor names one. Use lifetime_for() rather than reading the table directly.
53
+ KIND_LIFETIME_WEEKS = {
54
+ "fact": (4.0, 8.0),
55
+ "preference": (13.0, 26.0),
56
+ "procedure": (8.0, 12.0),
57
+ "warning": (52.0, 52.0),
58
+ }
59
+
60
+
61
+ def lifetime_for(kind: str, default_weeks: float = 0.0) -> tuple[float, float, str]:
62
+ """(stale_weeks, unused_weeks, kind) for one record's kind.
63
+
64
+ An unknown or missing kind gets the historical flat default, so a record written
65
+ by an older version, or by hand, is judged exactly as it was before this table
66
+ existed. The third element is the kind actually used, so a caller can print what
67
+ it judged against without re-deriving the fallback.
68
+ """
69
+ row = KIND_LIFETIME_WEEKS.get(str(kind or "").strip().lower())
70
+ if row:
71
+ return row[0], row[1], str(kind)
72
+ return STALE_CANDIDATE_WEEKS, UNUSED_CANDIDATE_WEEKS, str(kind or "")
73
+
36
74
 
37
75
  def _hours_since(ts: str) -> float | None:
38
76
  """Hours since an ISO timestamp written by paths.now(). Unparseable input
@@ -172,6 +210,22 @@ def buffer_counts() -> dict:
172
210
  "bytes": sum(os.path.getsize(os.path.join(d, f)) for f in files)}
173
211
 
174
212
 
213
+ def buffer_session_counts() -> dict:
214
+ """{session: record count} for the T0 buffer, one entry per file.
215
+
216
+ A pass consolidates ONE session, so this is the count a window means anything
217
+ against: `buffer_counts()` totals every file in the directory and answers a
218
+ different question ("how much scratch is on disk").
219
+ """
220
+ d = paths.buffer_dir()
221
+ out = {}
222
+ for name in sorted(os.listdir(d)):
223
+ if not name.endswith(".jsonl"):
224
+ continue
225
+ out[name[:-6]] = len(read_buffer(name[:-6]))
226
+ return out
227
+
228
+
175
229
  # ------------------------------------------------------- similarity/dedup ---
176
230
 
177
231
  # A proposal that paraphrases an already-refused candidate must not reborn as a
@@ -449,6 +503,21 @@ def slugify(text: str, maxlen: int = 48) -> str:
449
503
  return (s[:maxlen].rstrip("-")) or "note"
450
504
 
451
505
 
506
+ # What a kind is allowed to become permanent on. This is not the same question as
507
+ # "is this record true": a directive-shaped sentence about how the agent should
508
+ # behave is a statement about the PERSON, and only the person can confirm one, so a
509
+ # preference keeps the existing user-provenance requirement on top of the trust
510
+ # check. The other three kinds describe the world, and a human running `promote` is
511
+ # the confirmation: they read the body and chose it.
512
+ #
513
+ # A warning is the case worth naming, because it is the kind most likely to be
514
+ # captured from an agent's own failed attempt with no user in the loop. That is
515
+ # still legitimate - the human reading it is the check - but the refusal message
516
+ # below says which rule stopped it, so "why can I not promote this" never needs
517
+ # source reading.
518
+ KIND_PROMOTE_NEEDS_USER = ("preference",)
519
+
520
+
452
521
  def promote(cid: str, memory_dir: str | None = None, dry_run: bool = False) -> tuple[bool, str]:
453
522
  """T1 -> T2. Explicit human act. Creates a NEW file, never edits one."""
454
523
  got = get_candidate(cid)
@@ -461,6 +530,11 @@ def promote(cid: str, memory_dir: str | None = None, dry_run: bool = False) -> t
461
530
  if src != "user" and meta.get("trust") == "speculative":
462
531
  return False, ("refusing to persist speculative memory from source=%s; "
463
532
  "re-record with the user's own words or run a replay first" % src)
533
+ kind = str(meta.get("kind", "") or "")
534
+ if kind in KIND_PROMOTE_NEEDS_USER and src != "user":
535
+ return False, ("refusing to persist a %s from source=%s: a statement about what "
536
+ "the user wants needs the user's own words, not an agent's reading "
537
+ "of them" % (kind, src))
464
538
  mdir = memory_dir or paths.memory_dir()
465
539
  name = "%s.md" % slugify(body.split(".")[0][:60] or cid)
466
540
  dest = os.path.join(mdir, name)
@@ -810,7 +884,14 @@ def supersede(old_id: str, new_id: str) -> bool:
810
884
 
811
885
  # ------------------------------------------------------------ pruning -------
812
886
  def prune_buffer(older_than_hours: float = 0.0, session: str | None = None) -> dict:
813
- """T0 reset: consumed scratch moves to archive. Nothing is deleted."""
887
+ """T0 reset: consumed scratch moves to archive. Nothing is deleted.
888
+
889
+ File-level and kind-blind by design. It answers "has this session's scratch been
890
+ dealt with?", which is the right question when the caller is an explicit human
891
+ command. It is the WRONG tool for a record-level decision, because a file holds
892
+ records of every kind and moving it takes them all: use prune_session() when a
893
+ record that no pass read must survive.
894
+ """
814
895
  moved = []
815
896
  d = paths.buffer_dir()
816
897
  cutoff = time.time() - older_than_hours * 3600
@@ -829,6 +910,92 @@ def prune_buffer(older_than_hours: float = 0.0, session: str | None = None) -> d
829
910
  return {"moved": moved, "remaining": buffer_counts()}
830
911
 
831
912
 
913
+ # A kind whose records are dropped before consolidation has lost work that cannot be
914
+ # recovered by reading the archive later: the raw text survives, but no pass and no
915
+ # session will look at it again. These are the kinds where that loss is expensive -
916
+ # re-deriving a preference means asking the person again, and re-deriving a warning
917
+ # means paying for the failed attempt a second time. A fact is the one kind whose
918
+ # loss is cheap (it can be re-learned by looking), which is why it is deliberately
919
+ # NOT in this set: the default path must not be the expensive one.
920
+ KEEP_WHEN_UNREAD = ("preference", "procedure", "warning")
921
+
922
+
923
+ def prune_session(session: str, keep_kinds=KEEP_WHEN_UNREAD) -> dict:
924
+ """Prune ONE session's file, holding back records no pass will read.
925
+
926
+ The pass is shown the newest `consolidate.BUFFER_WINDOW` records, so anything older
927
+ than that is read by no pass - and a record no pass reads is also never pruned
928
+ (pruning follows consumption). What a file-level prune does anyway is move the whole
929
+ file, oldest records included, which turns "fell outside the window" from a visible
930
+ backlog into a silent one. This function moves the records the pass could have read
931
+ and keeps the rest, so a withheld record stays where the next pass can still reach it.
932
+
933
+ Imports `consolidate` lazily on purpose: consolidate imports this module at its top,
934
+ so a module-level import here would be circular. The one constant borrowed back is
935
+ the window, kept single-sourced rather than copied.
936
+ """
937
+ from . import consolidate as _consolidate
938
+ d = paths.buffer_dir()
939
+ src = os.path.join(d, session + ".jsonl")
940
+ if not os.path.isfile(src):
941
+ return {"skipped": True, "reason": "no such session", "kept": [], "moved": ""}
942
+ records = []
943
+ with open(src, encoding="utf-8") as fh:
944
+ for line in fh:
945
+ line = line.strip()
946
+ if not line:
947
+ continue
948
+ try:
949
+ records.append(json.loads(line))
950
+ except json.JSONDecodeError:
951
+ continue
952
+ # The window decides what a pass READS, and a pass having read a record is what
953
+ # licenses pruning it. It says nothing about whether the record was worth keeping:
954
+ # a preference inside the window is still a preference, and a prune has no business
955
+ # dropping it because the pass happened to look at it. A record outside the window
956
+ # is the one case where nothing will ever read it again. So the kind filter runs
957
+ # over the WHOLE file rather than over the tail alone, and what survives stays.
958
+ _window = _consolidate.BUFFER_WINDOW
959
+ in_window = records[max(0, len(records) - _window):]
960
+ held = [r for r in records if str(r.get("kind", "")) in keep_kinds]
961
+ held_outside = [r for r in held if r not in in_window]
962
+ if not held:
963
+ return prune_buffer(older_than_hours=0.0, session=session)
964
+ dropped = [r for r in records if str(r.get("kind", "")) not in keep_kinds]
965
+ # The keep file holds ONLY the records whose kind earned survival; the rest go to
966
+ # the archive. An earlier draft appended the in-window records here as well, which
967
+ # silently rewrote the file with its own contents and reported a "kept" count while
968
+ # nothing changed - the exact shape of bug this function exists to prevent, so the
969
+ # rewrite is asserted below rather than assumed.
970
+ keep_sorted = sorted(held, key=lambda r: str(r.get("ts", "")))
971
+ tmp = src + ".tmp"
972
+ with open(tmp, "w", encoding="utf-8") as fh:
973
+ for r in keep_sorted:
974
+ fh.write(json.dumps(r, sort_keys=True) + "\n")
975
+ os.replace(tmp, src)
976
+ after = read_buffer(session)
977
+ if len(after) != len(keep_sorted):
978
+ raise RuntimeError("prune_session rewrite failed: %d kept, %d on disk"
979
+ % (len(keep_sorted), len(after)))
980
+ # The records the pass COULD read, plus the withheld ones whose kind is safe to
981
+ # drop, go to the archive rather than the keep file; nothing is deleted and the
982
+ # archive path is the one prune_buffer already uses.
983
+ dest = os.path.join(paths.archive_dir("buffer"),
984
+ time.strftime("%Y%m%d-") + session + ".jsonl")
985
+ n = 1
986
+ while os.path.exists(dest):
987
+ dest = os.path.join(paths.archive_dir("buffer"),
988
+ time.strftime("%Y%m%d-") + "%s-%d.jsonl" % (session, n))
989
+ n += 1
990
+ with open(dest, "w", encoding="utf-8") as fh:
991
+ for r in dropped:
992
+ fh.write(json.dumps(r, sort_keys=True) + "\n")
993
+ return {"moved": dest, "archived_records": len(dropped),
994
+ "kept": [r["id"] for r in held], "kept_count": len(held),
995
+ "kept_outside_window": len(held_outside), "read_window": len(in_window),
996
+ "remaining": buffer_counts()}
997
+
998
+
832
999
  # Tags are the only index a record carries besides its project, so they are kept
833
1000
  # machine-shaped: lowercase, no spaces, bounded in number and length. A model may
834
1001
  # propose them, which is exactly why they are normalized here rather than trusted.
@@ -1041,7 +1208,6 @@ def doctor(memory_dir: str | None = None, targets: list[str] | None = None,
1041
1208
  "pair": [pool[i][0], pool[j][0]], "similarity": round(sim, 2),
1042
1209
  "older": older, "newer": newer,
1043
1210
  "suggest": "sleep.py supersede %s %s" % (older, newer)})
1044
- cutoff_hours = STALE_CANDIDATE_WEEKS * 7 * 24
1045
1211
  for c in list_candidates():
1046
1212
  if c.get("status") != "provisional":
1047
1213
  continue
@@ -1049,10 +1215,15 @@ def doctor(memory_dir: str | None = None, targets: list[str] | None = None,
1049
1215
  # record's own creation ts. A provisional that is neither being promoted
1050
1216
  # nor re-learned for weeks is noise the queue keeps injecting into every
1051
1217
  # session's context, so the doctor surfaces it (report only, nothing drops).
1218
+ # "Weeks" is per kind: the same quiet month means different things for a
1219
+ # preference and for an environment warning (see KIND_LIFETIME_WEEKS).
1220
+ stale_weeks, _unused_weeks, kind = lifetime_for(c.get("kind", ""))
1052
1221
  last = c.get("rediscovered_at") or c.get("ts")
1053
1222
  age_h = _hours_since(str(last)) if last else None
1054
- if age_h is not None and age_h >= cutoff_hours:
1223
+ if age_h is not None and age_h >= stale_weeks * 7 * 24:
1055
1224
  out.append({"issue": "stale-candidate-unrediscovered", "id": c["id"],
1225
+ "kind": kind, "threshold_weeks": stale_weeks,
1226
+ "age_days": int(age_h // 24),
1056
1227
  "last_seen": str(last)[:10], "rediscovered": int(c.get("rediscovered", 0) or 0)})
1057
1228
  # The sibling of the check above, measuring a different emptiness. That one says
1058
1229
  # "nobody re-learned this"; this one says "nobody ever asked for this": no inject
@@ -1060,30 +1231,36 @@ def doctor(memory_dir: str | None = None, targets: list[str] | None = None,
1060
1231
  # perfectly true and still be worth naming here, because the cost it carries is
1061
1232
  # paid by every session that shares the project and the evidence it is worth
1062
1233
  # paying is published in the record itself.
1063
- unused_cutoff = UNUSED_CANDIDATE_WEEKS * 7 * 24
1064
1234
  for c in list_candidates():
1065
1235
  if c.get("status") != "provisional":
1066
1236
  continue
1067
1237
  served = int(c.get("injected", 0) or 0) + int(c.get("rediscovered", 0) or 0)
1068
1238
  if served:
1069
1239
  continue
1240
+ _stale_weeks, unused_weeks, kind = lifetime_for(c.get("kind", ""))
1070
1241
  age_h = _hours_since(str(c.get("ts", ""))) if c.get("ts") else None
1071
- if age_h is not None and age_h >= unused_cutoff:
1242
+ if age_h is not None and age_h >= unused_weeks * 7 * 24:
1072
1243
  out.append({"issue": "unused-candidate-never-served", "id": c["id"],
1244
+ "kind": kind, "threshold_weeks": unused_weeks,
1073
1245
  "age_days": int(age_h // 24),
1074
1246
  "injected": 0, "rediscovered": 0})
1075
- # The one input limit that is not a content judgement: consolidation is shown only
1076
- # the newest WINDOW records, so a buffer larger than that carries records no pass
1077
- # will read and no prune will clear. Measured on the honest definition of a pass -
1078
- # what the CLI actually hands over - and reported here rather than raised to a
1079
- # failure, because a large buffer may be a real backlog rather than an error.
1247
+ # The one input limit that is not a content judgement: a pass is shown only the
1248
+ # newest WINDOW records OF THE SESSION IT CONSOLIDATES, so a session larger than
1249
+ # that carries records no pass will read. Counted per session, on the honest
1250
+ # definition of a pass - `run_sleep`/`distill` hand over one session file - because
1251
+ # the total across sessions compares a number to a limit it was never measured
1252
+ # against: fifty sessions of ten records each are fifty complete passes, not one
1253
+ # buffer of five hundred with four hundred and eighty unread. Reported rather than
1254
+ # raised to a failure, because a large session may be a real backlog.
1080
1255
  from . import consolidate as _consolidate
1081
- t0_records = buffer_counts()["records"]
1082
- if t0_records > _consolidate.BUFFER_WINDOW:
1083
- out.append({"issue": "buffer-exceeds-consolidation-window",
1084
- "buffer_records": t0_records,
1085
- "window": _consolidate.BUFFER_WINDOW,
1086
- "outside_window": t0_records - _consolidate.BUFFER_WINDOW})
1256
+ _window = _consolidate.BUFFER_WINDOW
1257
+ for _session, _count in sorted(buffer_session_counts().items()):
1258
+ if _count > _window:
1259
+ out.append({"issue": "buffer-exceeds-consolidation-window",
1260
+ "session": _session,
1261
+ "buffer_records": _count,
1262
+ "window": _window,
1263
+ "outside_window": _count - _window})
1087
1264
  for cid, body in pool:
1088
1265
  for path, text in bodies.items():
1089
1266
  # MEMORY.md is the index: it restates every entry by design, so scoring