ceiling-guard 0.5.3__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,810 @@
1
+ Metadata-Version: 2.4
2
+ Name: ceiling-guard
3
+ Version: 0.5.3
4
+ Summary: See how much of your Claude Code, Codex or Grok subscription window is left, and get a handoff before a plan limit ends the run
5
+ Author: Ak2tx LLC
6
+ License: MIT License
7
+
8
+ Copyright (c) 2026 Ak2tx LLC
9
+
10
+ Permission is hereby granted, free of charge, to any person obtaining a copy
11
+ of this software and associated documentation files (the "Software"), to deal
12
+ in the Software without restriction, including without limitation the rights
13
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
14
+ copies of the Software, and to permit persons to whom the Software is
15
+ furnished to do so, subject to the following conditions:
16
+
17
+ The above copyright notice and this permission notice shall be included in all
18
+ copies or substantial portions of the Software.
19
+
20
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
21
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
22
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
23
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
24
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
25
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
26
+ SOFTWARE.
27
+
28
+ Project-URL: Homepage, https://github.com/ak2tx/ceiling-guard
29
+ Project-URL: Source, https://github.com/ak2tx/ceiling-guard
30
+ Project-URL: Issues, https://github.com/ak2tx/ceiling-guard/issues
31
+ Project-URL: Changelog, https://github.com/ak2tx/ceiling-guard/blob/main/CHANGELOG.md
32
+ Keywords: ai-agents,llm,claude,chatgpt,codex,grok,usage-limits,rate-limits,quota,plan-window,token-budget,watchdog,monitoring,observability,agent-loop,cost-control
33
+ Classifier: Development Status :: 4 - Beta
34
+ Classifier: Intended Audience :: Developers
35
+ Classifier: License :: OSI Approved :: MIT License
36
+ Classifier: Operating System :: POSIX :: Linux
37
+ Classifier: Operating System :: MacOS :: MacOS X
38
+ Classifier: Operating System :: Microsoft :: Windows
39
+ Classifier: Programming Language :: Python :: 3 :: Only
40
+ Classifier: Topic :: System :: Monitoring
41
+ Classifier: Topic :: Software Development :: Quality Assurance
42
+ Requires-Python: >=3.11
43
+ Description-Content-Type: text/markdown
44
+ License-File: LICENSE
45
+ Dynamic: license-file
46
+
47
+ # Ceiling Guard
48
+
49
+ Don't lose another agent run to a plan limit.
50
+
51
+ Ceiling Guard shows how much of your Claude Code, Codex or Grok
52
+ subscription window is left, warns your agent before it runs out, and writes a
53
+ handoff for the next run. It reads usage the official CLIs already record on
54
+ your machine — no provider credentials, no proxy, no account access.
55
+
56
+ Codex and Grok usage appears immediately. Claude publishes its percentage only
57
+ during a non-interactive run, so wrap one Claude command to start tracking it.
58
+
59
+ Zero dependencies. Python 3.11+. Linux, macOS, Windows, or a container.
60
+
61
+ ```bash
62
+ pipx install ceiling-guard
63
+
64
+ cg quota # read-only, nothing to configure
65
+ cg wrap -- claude -p "continue the migration"
66
+ ```
67
+
68
+ `cg wrap` runs your command unchanged and observes it; it terminates nothing
69
+ unless you separately arm the daemon. For `claude -p` it adds
70
+ `--output-format stream-json --verbose` and renders the answer back to plain
71
+ text, because that event stream is the only place Claude's percentage exists —
72
+ it is never written to disk. It says so on stderr when it does this rather
73
+ than silently rewriting your command line.
74
+
75
+ Put it where you already look, instead of remembering to ask:
76
+
77
+ ```bash
78
+ cg statusline # -> cg 48% (or "cg -" when it cannot see)
79
+
80
+ # bash/zsh prompt
81
+ PS1='$(cg statusline) '"$PS1"
82
+
83
+ # tmux
84
+ set -g status-right '#(cg statusline)'
85
+ ```
86
+
87
+ `statusline` reads no config, contacts no daemon and always exits 0, so it
88
+ cannot break the prompt it is added to. It prints `-` rather than a stale
89
+ percentage: a number that was true an hour ago is the failure this tool
90
+ exists to prevent.
91
+
92
+ ```
93
+ PROVIDER WINDOW USED RESETS AGE STATUS
94
+ openai primary 97.0% Sat 11:19 2m ok
95
+ grok weekly 63.0% Sat 23:21 23m ok
96
+ anthropic five_hour 94.0% Sat 12:30 0m ok <- after one wrapped Claude job
97
+ seven_day 73.0% Mon 02:00 0m ok
98
+ ```
99
+
100
+ A browser-only subscription is not a user of this tool: the reading comes from
101
+ what a CLI writes locally, and a browser writes nothing here.
102
+
103
+ Those numbers come from files the official clients already write, and from
104
+ stream events passing through `cg wrap`. Each adapter was checked against the
105
+ provider's own usage screen; the working is in
106
+ [docs/quota-accuracy.md](docs/quota-accuracy.md).
107
+
108
+ ## What you will actually see
109
+
110
+ Those three sentences are the claim. This table is the evidence for them, and
111
+ it belongs here rather than in a footnote.
112
+
113
+ | you | `cg quota` shows |
114
+ |---|---|
115
+ | use the `codex` or `grok` CLI | their windows immediately. Those clients write usage to disk themselves. |
116
+ | use the `claude` CLI | nothing, until you wrap one session |
117
+ | use claude.ai or ChatGPT in a browser | nothing, ever |
118
+
119
+ Runs hosted inside another vendor's client spend that vendor's pool, not your
120
+ provider's. Cursor is the common case: CG does not see that ceiling. The
121
+ measurement is in [docs/quota-accuracy.md](docs/quota-accuracy.md).
122
+
123
+ Claude is the awkward case. It does publish a percentage, but only as a
124
+ `rate_limit_event` inside its `stream-json` output, and it never writes that to
125
+ disk. Polling a file will not find it. The number has to be caught in flight:
126
+
127
+ ```bash
128
+ cg wrap -- claude -p "whatever you were going to run anyway"
129
+ cg quota # the anthropic rows are there now
130
+ ```
131
+
132
+ The substitution is only applied when the child is `claude`, run with `-p`,
133
+ and you have not chosen a format yourself. A wrapper script under another name
134
+ will not be recognised, and that is deliberate: guessing at what a binary is
135
+ would be worse than missing one.
136
+
137
+ `cg wrap` terminates nothing, and says so on its first line every time.
138
+
139
+ It is not a transparent sleeve like `timeout` or `nice`, and it should not
140
+ claim to be. For `claude -p` specifically it adds `--output-format stream-json
141
+ --verbose`, because that is the only form in which the percentage exists, then
142
+ prints the answer back as plain text so the run looks the same to you. It tells
143
+ you it did that. An explicit `--output-format` of your own wins, an interactive
144
+ `claude` with no `-p` is left alone, and no other program is ever rewritten.
145
+
146
+ Browser sessions leave no local record at all. If that is how you use your
147
+ subscription, this tool has nothing to read and you should stop here.
148
+
149
+ ## Warn the job before the window closes
150
+
151
+ Reading the number is useful on its own. The reason it exists is what comes
152
+ next: telling a running job that it is nearly out of room, while there is still
153
+ room to do something about it.
154
+
155
+ ```python
156
+ from continuity_guard import guard
157
+
158
+ with guard("nightly-refactor", profile="overnight") as s:
159
+ for step in agent.run():
160
+ s.progress(step=step.name, tool=step.tool, args=step.args,
161
+ tokens=step.tokens, depth=step.depth)
162
+ if s.should_wrap_up(): # cheap, non-blocking, False by default
163
+ save_state(s.headroom()) # what to save is yours to decide
164
+ break
165
+ ```
166
+
167
+ Every signalled session also gets a handoff written for it, whether or not it
168
+ answers. That matters because the sessions most worth preserving are usually
169
+ the ones too wedged to preserve themselves. What that document does and does
170
+ not preserve is set out in
171
+ [what the handoff is and is not](#what-the-handoff-is-and-is-not).
172
+
173
+ ## The watchdog, when you want it
174
+
175
+ Twelve detectors cover the agent failures that return HTTP 200 while nothing
176
+ useful happens: loops, oscillation, pace collapse, token burn, runaway
177
+ recursion, crash loops.
178
+
179
+ It ships observe-only. Nothing is terminated until you have watched it against
180
+ your own traffic and decided to arm it.
181
+
182
+ ```bash
183
+ cg init && cg install # scaffold config, run as a service
184
+ cg status # how close each session is to firing, now
185
+ ```
186
+
187
+ The order of this page is the order we suggest you adopt it. Reading the quota
188
+ costs you nothing and needs no permission. The wrap-up signal costs one line in
189
+ your loop. Kill authority comes last, because it is the only part that can take
190
+ something away from you.
191
+
192
+ ---
193
+
194
+ ## Why a heartbeat is not enough
195
+
196
+ A heartbeat proves a thread is running. It says nothing about whether work is
197
+ happening, and the expensive failures are the ones where every request returns
198
+ 200, latency sits inside the SLO, and the bill triples overnight.
199
+
200
+ So there are twelve detectors:
201
+
202
+ | Detector | Catches | Signal |
203
+ |---|---|---|
204
+ | `death` | process gone | socket EOF, free from the kernel, no timeout |
205
+ | `silence` | wedged, blocked forever | no frames at all |
206
+ | `no_progress` | the lying heartbeat: alive, not working | frames arrive, `seq` frozen |
207
+ | `decay` | pace collapse | consecutive slow gaps vs the session's **own** learned baseline |
208
+ | `repeat` | the agent loop | identical tool + identical args, repeating |
209
+ | `cycle` | oscillation / thrash | edit, test, revert, repeating |
210
+ | `burn` | the failure that never errors | tokens/min high **and** no progress |
211
+ | `budget` | all-night spend | cumulative token ceiling, progress or not |
212
+ | `deadline` | runs that never end | wall-clock cap on one session |
213
+ | `errors` | retry storms | failed tool calls over a window |
214
+ | `depth` | runaway sub-agent spawning | recursion depth exceeded |
215
+ | `flap` | crash loops | repeated death + restart of the same name |
216
+
217
+ Four of those are invisible to ordinary monitoring: `repeat`, `cycle`, `burn`
218
+ and `errors`.
219
+
220
+ Three of them are easy to confuse with each other, so:
221
+
222
+ `cycle` exists because `repeat` cannot catch thrash. The classic agent failure
223
+ is edit, run tests, revert, edit, run tests, revert. Every call differs from the
224
+ one before it, so exact-duplicate detection sees healthy variety while the agent
225
+ goes nowhere.
226
+
227
+ `budget` is not `burn`. `burn` is a rate, and an agent can stay under every rate
228
+ limit and still spend all night. An absolute ceiling is the "one stuck agent
229
+ spends $6,000 tonight" guard, and it fires whether or not work is progressing.
230
+ Enforcement happens between steps, so a real total can overshoot by roughly one
231
+ model call. Treat it as a ceiling, not a guarantee.
232
+
233
+ `decay` is the only adaptive detector. It learns a session's own pace from its
234
+ first steps, freezes that baseline, then fires if the pace collapses relative to
235
+ it. A 2s-per-step interactive agent and a 90s-per-step build agent both work
236
+ with no tuning.
237
+
238
+ Four constraints keep `decay` safe. Each was added after a measurement showed
239
+ the previous version firing on ordinary build traffic:
240
+
241
+ - The effective threshold can never fall below the `floor` you set. Adaptation
242
+ only ever *loosens*. A detector that can tighten itself will invent false
243
+ positives on a workload it mis-learned.
244
+ - The `warn` level is relative and sensitive (`base * warn_k`, floor `45s`).
245
+ The `hard` level is an absolute backstop (`hard_floor`, default `200s`) set
246
+ above the slowest gap a healthy heavy-tailed workload produces, so build and
247
+ research sessions cannot physically reach the terminate level.
248
+ - The baseline is a central quantile of the early gaps (`baseline_q`, default
249
+ the median). Decay asks whether the *typical* pace collapsed. The upper tail
250
+ belongs to `silence`, which is asking a different question.
251
+ - It requires **consecutive** slow gaps. "3 of the last 7 steps were slow" just
252
+ describes a build-heavy workload. An unbroken run is what a real collapse
253
+ looks like.
254
+
255
+ Those defaults come from measured sweeps on build-heavy traffic, where 22% of
256
+ steps take 30 to 150 seconds. Moving `min_gaps` from 3 to 4 to 5 took
257
+ warn-level false positives from 12.5% to 1.2% to 0.4%, with detection at 100%
258
+ throughout, so `min_gaps = 5` ships. Adding `hard_floor` took the hard-level
259
+ false positive from 1.2% to 0.0% at high sample counts with recall unchanged.
260
+
261
+ Death detection costs nothing at all. The session holds one socket open; when
262
+ the process dies the kernel closes it and the daemon knows in milliseconds.
263
+ That is an OS fact rather than an inference, so there is no polling, no timeout,
264
+ and no way for it to be wrong.
265
+
266
+ `RETIRE` is not optional. Without a clean-exit signal every successful run ends
267
+ in a socket close indistinguishable from a crash, and the tool alarms on its own
268
+ users' happy path. The context manager sends it on normal exit and on exception,
269
+ because an exception is still an observed, intentional end.
270
+
271
+ ## Tuning
272
+
273
+ Everything is adjustable per session, and thresholds reload live.
274
+
275
+ ```
276
+ session inline overrides > named profile > defaults
277
+ ```
278
+
279
+ ```python
280
+ guard("agent") # defaults
281
+ guard("agent", profile="tight") # built-in profile
282
+ guard("agent", profile="loose",
283
+ silence={"warn": "10m", "hard": "2h"}) # inline override
284
+ ```
285
+
286
+ Four profiles ship. `tight` is for interactive work where you want fast
287
+ feedback. `loose` is the false-positive-safe end, for long tool calls and
288
+ research. `overnight` is for unattended runs: it catches loops and burn early
289
+ and will kill a hard stall. `wallet` sets a low per-session token ceiling, for
290
+ when a runaway bill would hurt more than a truncated run. Define your own in
291
+ the config.
292
+
293
+ The numbers that differ from the defaults:
294
+
295
+ | | `silence` warn/hard | `no_progress` warn/hard | `budget` hard | action |
296
+ |---|---|---|---|---|
297
+ | **defaults** | 3m / 30m | 3m / 45m | 1,000,000 | `notify` |
298
+ | **tight** | 20s / 2m | 60s / 5m | 1,000,000 | `notify` |
299
+ | **loose** | 5m / 45m | 15m / 2h | 4,000,000 | `notify` |
300
+ | **overnight** | 2m / 15m | 10m / 45m | 1,500,000 | **`terminate`** |
301
+ | **wallet** | 3m / 30m | 3m / 45m | **25,000** | `notify` |
302
+
303
+ `overnight` is the only built-in profile that terminates, and even then only
304
+ once you have taken the daemon out of shadow mode. `wallet` still just
305
+ notifies: it stops the spend by telling you, not by killing. For the full
306
+ effective set including your own overrides, run `cg profiles`. It prints the
307
+ config file it read on the first line.
308
+
309
+ Thresholds reload live. Edit the config, then, on POSIX:
310
+
311
+ ```bash
312
+ cg reload # SIGHUP: live sessions re-resolve thresholds, state preserved
313
+ cg status # how close each session is to firing, right now
314
+ cg history # what fired, when, and whether it acted
315
+ cg profiles # effective thresholds per profile
316
+ ```
317
+
318
+ ### Seeing that it is actually working
319
+
320
+ The commonest reason a watchdog gets uninstalled is that nobody can tell
321
+ whether it is awake. Here are the two commands that answer that, with real
322
+ output.
323
+
324
+ `cg status` shows what is running and how close it is to each threshold:
325
+
326
+ ```
327
+ config: /home/you/.continuity-guard/config.toml
328
+ socket /run/continuity-guard/cg.sock 2 session(s) 12:29:53
329
+
330
+ SESSION PID SEQ QUIET STUCK TOK/MIN REP PROFILE MODE
331
+ nightly-refactor 130847 26 1s 1s 19200 8 default shadow
332
+ nightly-refactor 130889 8 2s 2s 19200 8 default shadow
333
+ ```
334
+
335
+ `cg history` shows what fired, when, and whether it acted:
336
+
337
+ ```
338
+ config: /home/you/.continuity-guard/config.toml
339
+ WHEN SESSION DETECTOR LEVEL ACTION DETAIL
340
+ 2026-08-29 12:30:02 nightly-refactor repeat warn notify repeated apply_patch(33d034ea) x13 (unconfirmed: 21s of 60s) [shadow]
341
+ 2026-08-29 12:29:41 nightly-refactor repeat warn notify repeated apply_patch(33d034ea) x3 (unconfirmed: 0s of 60s) [shadow]
342
+ ```
343
+
344
+ That second line contains the whole safety design. `repeat` saw the same call
345
+ thirteen times. It said so at `warn`, marked itself `unconfirmed: 21s of 60s`,
346
+ took the action `notify`, and tagged the record `[shadow]`. Nothing was killed.
347
+ A hard verdict has to hold unbroken for its confirmation window before it
348
+ escalates, and in shadow mode it never acts at all.
349
+
350
+ Both commands print the config file they read on the first line. That is not
351
+ decoration. A stale `~/.continuity-guard/config.toml` will quietly disagree
352
+ with this page, and if you cannot tell what is configured you cannot trust
353
+ what it will do.
354
+
355
+ The columns in `cg status` are the values the thresholds are compared against:
356
+ quiet time, stuck time, tokens per minute, current repeat run. Tune against
357
+ those rather than against guesses.
358
+
359
+ ---
360
+
361
+ ## Plan windows, and not losing a run to one
362
+
363
+ The reading is the easy half. This section is the mechanics: how the number is
364
+ kept honest, and what happens to a live session when the window runs low.
365
+
366
+ A stale reading is refused rather than trusted. Once, a Grok cache read 13.0%
367
+ while the account was actually at 63%, from a file five days old. That is the
368
+ most dangerous state a meter can be in, because it looks like knowledge. Any
369
+ snapshot older than an hour is marked `IGNORED` and cannot trigger anything,
370
+ and an unknown age counts as stale.
371
+
372
+ At `wrapup_at`, which defaults to 95% of the binding window, every live session
373
+ is told to save its state. A handoff is written for each of them whether or not
374
+ they respond. A wedged agent cannot act on the signal, and that is when the
375
+ record is worth the most.
376
+
377
+ ```python
378
+ with guard("nightly-refactor", profile="overnight") as s:
379
+ for step in agent.run():
380
+ s.progress(...)
381
+ if s.should_wrap_up(): # cheap, non-blocking
382
+ write_handoff(s.headroom()) # what to save is yours to decide
383
+ break
384
+ ```
385
+
386
+ The handoff Ceiling Guard writes itself is a record of what it observed:
387
+ steps, tool calls, timings, tokens, the plan window at the time. Not a summary
388
+ of intent; the agent did not write it. If your agent also acts on the signal
389
+ you get a better one on top.
390
+
391
+ Ceiling signals go to the three ceiling conditions (a plan window, a `budget`,
392
+ a `deadline`) and never to faults. Nothing is wrong with a session that is
393
+ merely running out; a wedged one cannot answer anyway.
394
+
395
+ ### What the handoff is and is not
396
+
397
+ You always know what a run reached, including when the agent died without
398
+ cooperating. Enumerable progress survives: which items are done, which remain,
399
+ and the command that resumes past them. When the work is a list, and a great
400
+ deal of agent work is a list, that is most of what you needed. The next run
401
+ picks up without repeating itself.
402
+
403
+ What does not survive is the agent's understanding. The half-formed
404
+ hypothesis, the reason approach A was abandoned, the thing it had noticed but
405
+ not yet put into words: that lives in the model's context and goes when the
406
+ context goes. Writing it down is lossy, and the loss is invisible in the
407
+ result, because you cannot tell from a summary what is missing from it. This
408
+ is the context-compaction problem and nothing here solves it.
409
+
410
+ Call it a warm start rather than a continuation. A successor reads the handoff
411
+ and begins informed instead of blind. It does not resume mid-thought.
412
+
413
+ Two consequences are worth planning around.
414
+
415
+ Task shape decides how much this is worth to you. Enumerable, independent units
416
+ hand off well. Exploratory debugging hands off badly, because its value was
417
+ never in the enumeration; it was in the hypothesis space, which is the part
418
+ that does not survive.
419
+
420
+ Findings also have a shelf life. In one measured case, two of four findings in
421
+ a handoff had already been fixed a few hours later, and a third was wrong on
422
+ its own terms. A document describing a moving codebase decays. Handoffs carry
423
+ an observed-at stamp so a reader can judge for themselves, which is mitigation
424
+ rather than a fix.
425
+
426
+ ---
427
+
428
+ ## The socket is kill authority
429
+
430
+ An armed daemon terminates processes it is told about, and everything in a
431
+ session's announcement is that peer's claim about itself. The socket is
432
+ therefore a privileged surface and is treated as one. Four checks, each
433
+ independent, because an operator will eventually widen one of them for a good
434
+ reason:
435
+
436
+ - The Unix socket is `0600`. It used to be `0666`. Set `CG_SOCKET_MODE=660`
437
+ with a shared group to widen it deliberately, when several accounts must
438
+ genuinely share one daemon.
439
+ - A session may tune its own thresholds but never its own `action`. Overrides
440
+ arriving over the socket have operator-only keys stripped, so `action` comes
441
+ from your config file or not at all. Whatever was dropped is logged.
442
+ - Identity is verified rather than assumed. Terminating requires a
443
+ `start_time` matching the live process. A missing one used to fall back to
444
+ "is anything alive with this PID", which accepts every process on the box.
445
+ - The announced PID must be the connecting peer or a descendant of it, since a
446
+ supervisor announcing its child is the normal case. Anything else disarms
447
+ termination for that session and says so in the log. Observation continues.
448
+
449
+ TCP is not authenticated. `socket = "tcp://..."` binds `127.0.0.1` by default
450
+ and should stay there. There is no token and no TLS, and a reachable armed
451
+ daemon is a kill primitive for anything its user can signal. Peer credentials
452
+ do not exist over TCP, so termination is disarmed there deliberately.
453
+
454
+ The full threat model, and the disclosure of the local privilege issue fixed
455
+ in 0.5.0, are in [SECURITY.md](SECURITY.md).
456
+
457
+ ---
458
+
459
+ ## Shadow mode
460
+
461
+ The daemon ships observe-only. With `shadow = true` it evaluates every
462
+ contract and logs every action it would have taken, and kills nothing.
463
+
464
+ Run it that way against real workloads first, and measure your own
465
+ false-positive rate. Nobody should grant kill authority to a new watchdog
466
+ before they have seen that number, and shadow mode produces the only metric
467
+ worth having: N sessions observed, zero healthy terminations.
468
+
469
+ ```bash
470
+ python3 -m continuity_guard.daemon -c config.toml # config decides
471
+ python3 -m continuity_guard.daemon -c config.toml --shadow # force observe-only
472
+ python3 -m continuity_guard.daemon -c config.toml --armed # force enforcement
473
+ ```
474
+
475
+ ---
476
+
477
+ ## Notifications
478
+
479
+ Every sink declares which levels and detectors it wants.
480
+
481
+ ```toml
482
+ [[notifications]]
483
+ kind = "file"
484
+ path = "~/continuity-guard-alerts.log"
485
+ on = ["warn", "hard", "dead"]
486
+
487
+ [[notifications]]
488
+ kind = "exec" # event passed as CG_* env vars
489
+ command = "~/bin/cg-notify.sh"
490
+ on = ["hard", "dead"]
491
+
492
+ [[notifications]]
493
+ kind = "webhook"
494
+ url = "http://127.0.0.1:9000/cg"
495
+ on = ["hard", "burn", "repeat"]
496
+ ```
497
+
498
+ A broken sink is logged and swallowed. Notification failure must never stop the
499
+ daemon doing its actual job.
500
+
501
+ ---
502
+
503
+ ## Safety properties
504
+
505
+ **PID reuse is guarded.** A PID recorded an hour ago may belong to something
506
+ else by the time you act on it, so every kill verifies PID and process start
507
+ time as a matched pair and refuses if they disagree.
508
+
509
+ **Children are not orphaned.** Agent sessions spawn shells, tool calls and
510
+ model servers. Termination signals the process group, so nothing is left
511
+ holding ports, files or money.
512
+
513
+ **Grace before force.** `SIGTERM`, wait `grace`, then `SIGKILL`.
514
+
515
+ **Sleep is not death.** `CLOCK_BOOTTIME` and `CLOCK_MONOTONIC` are compared
516
+ every tick. A machine that suspended re-arms its deadlines with a grace window
517
+ rather than firing. After a gap you have no trustworthy information, and the
518
+ right answer to "I don't know" is to re-observe.
519
+
520
+ **Refractory is durable.** Fires are suppressed per `session::detector::level`
521
+ in SQLite, with exponential backoff, keyed per level so an escalation is never
522
+ swallowed by the warning before it. It survives daemon restart, which matters
523
+ because the restart is exactly the moment when every session looks freshly
524
+ overdue at once.
525
+
526
+ **Heartbeats are never fsynced.** Only state transitions are persisted.
527
+ Writing every progress frame to disk would dominate I/O, and wear out a Pi's
528
+ SD card, for no benefit.
529
+
530
+ **Kill authority is local, and only local.** A remote observer cannot tell "the
531
+ session died" from "I cannot currently reach the session". Silence over a
532
+ network is evidence of silence, not of death. If a hosted tier ever exists it
533
+ will send a deadman notification, which is a claim about what the observer
534
+ knows, and it will never issue a kill. That makes split-brain structurally
535
+ impossible rather than merely unlikely.
536
+
537
+ **What this does not promise.** Killing a process is not the same as making the
538
+ outcome safe. If a session already fired an API call, sent a message or charged
539
+ a card before it stalled, termination does not undo any of it. Whether a hard
540
+ kill leaves clean resumable state is a property of your application, not of
541
+ this watchdog.
542
+
543
+ ---
544
+
545
+ ## Works with any agent
546
+
547
+ Four ways in, in order of how much you get for the effort. All feed the same
548
+ twelve detectors.
549
+
550
+ **1. The proxy. Zero code changes, any language, richest signals.**
551
+ Nearly every agent and local runtime speaks OpenAI chat-completions: Ollama,
552
+ vLLM, llama.cpp, LM Studio, LiteLLM, OpenRouter, the xAI (Grok) API, grok-cli,
553
+ Aider, OpenHands, CrewAI, AutoGen, LangGraph. Point the base URL at the proxy:
554
+
555
+ ```bash
556
+ cg-proxy --upstream http://localhost:11434 --port 8111 # Ollama
557
+ cg-proxy --upstream https://api.x.ai --port 8111 # Grok
558
+ export OPENAI_BASE_URL=http://127.0.0.1:8111/v1
559
+ ```
560
+
561
+ The wire format carries exactly what the detectors want, with no guessing:
562
+ `usage.total_tokens` → burn/budget, `tool_calls[].function` → repeat/cycle,
563
+ HTTP status → errors, request cadence → silence/decay. Streaming works, and
564
+ gives token-level liveness for free. Name sessions with an `X-CG-Session`
565
+ header, or run one proxy per agent with `--name`.
566
+
567
+ *Limitation, and it is real:* the proxy sees model traffic, not the process.
568
+ An agent that exited cleanly and one that hung look identical from there. Pair
569
+ it with the supervisor or library for process lifecycle.
570
+
571
+ **2. `instrument_openai()`. One line, for any framework using the openai SDK.**
572
+
573
+ ```python
574
+ from continuity_guard.integrations import instrument_openai
575
+ client = instrument_openai(OpenAI(base_url=...), "my-agent", profile="overnight")
576
+ ```
577
+
578
+ **3. `CallbackHandler`. LangChain, LangGraph, and anything that copied that
579
+ interface.** Duck-typed: this package never imports LangChain.
580
+
581
+ ```python
582
+ from continuity_guard.integrations import CallbackHandler
583
+ with CallbackHandler("research-agent", profile="loose") as cb:
584
+ graph.invoke(state, config={"callbacks": [cb]})
585
+ ```
586
+
587
+ **4. `Tracker` / `@guarded`. Your own loop.**
588
+
589
+ ```python
590
+ from continuity_guard.integrations import guarded
591
+
592
+ @guarded("nightly-refactor", profile="overnight")
593
+ def run(task, cg=None):
594
+ for step in agent(task):
595
+ cg.step(tool=step.tool, args=step.args, tokens=step.tokens, ok=step.ok)
596
+ ```
597
+
598
+ And for CLI agents that offer no hooks at all, `cg wrap` wraps the
599
+ process and derives progress from output.
600
+
601
+ ## Accuracy
602
+
603
+ `tests/bench_accuracy.py` drives the real detector code against a simulated
604
+ clock, which gets thousands of sessions in seconds where the realtime soak
605
+ manages about a dozen in ten minutes. The healthy workloads are adversarial by
606
+ design. Each one is a legitimate pattern chosen because it looks like a
607
+ specific fault.
608
+
609
+ | Healthy archetype | Looks like | Why it is legitimate |
610
+ |---|---|---|
611
+ | `polling` | `repeat` | re-reads the same status file forever, while progressing |
612
+ | `pipeline` | `cycle` | read → edit → test, a new file each pass |
613
+ | `flaky` | `errors` | 30% of tool calls genuinely fail (empty greps, 404s) |
614
+ | `bursty` | `decay` | fast bursts split by 4-minute thinking pauses |
615
+ | `rate_limited` | `decay` | periodic provider backoffs |
616
+ | `build` | `decay` | 22% of steps take 30–150s |
617
+ | `long_tool` | `silence` | single 20-minute test-suite runs |
618
+
619
+ Current numbers on the archetypes above (default profile):
620
+
621
+ ```
622
+ overall any-level FP 0.0% hard-level FP 0.0%
623
+ loop 100% cycle 100% stall 100% burn 100% errors 100%
624
+ depth 100% decay 100% budget 100% cold_freeze 100% cold_stall 100%
625
+ ```
626
+
627
+ **That figure is not universal, and the qualifier is the honest part.** Three
628
+ workload shapes are known to produce false positives and are excluded from the
629
+ gate, measured and published rather than deleted
630
+ (`tests/bench_accuracy.py --known-fail`):
631
+
632
+ | shape | any-level FP | hard-level FP | detector |
633
+ |---|---|---|---|
634
+ | polls one endpoint forever | 100% | 0% | `repeat` |
635
+ | alternates between two queues | 100% | **100%** | `cycle` |
636
+ | fast setup, then long work | 100% | 0% | `decay` |
637
+
638
+ Together those three measure **any-level FP 25.0%, hard-level FP 8.33%, min
639
+ recall 92%** (`tests/bench_accuracy.py --known-fail`, which reports and exits
640
+ 0 rather than gating).
641
+
642
+ The second is a real hard-level false positive on healthy work: an agent
643
+ doing `poll(queue-a)`, `poll(queue-b)` with `seq` advancing every frame — real,
644
+ reported progress — reaches a hard `cycle` verdict on the sixth frame. That is
645
+ what a queue worker, a CI watcher and a deploy monitor look like.
646
+
647
+ The default action for `repeat` and `cycle` is `notify`, so nothing is
648
+ terminated out of the box. **Under a terminating profile it would be.** If your
649
+ agent polls, set `repeat.action` and `cycle.action` to `notify` explicitly, or
650
+ raise their thresholds.
651
+
652
+ No fix is offered, because a healthy periodic poller and a pathological loop
653
+ emit identical frames indefinitely and no finite confirmation window separates
654
+ them. See `docs/quota-accuracy.md`. The corpus missed this for a long time
655
+ because the archetype named `polling` did not actually poll — its docstring
656
+ claimed it did.
657
+
658
+ ### Against real traffic, not just the corpus
659
+
660
+ The numbers above come from an adversarial corpus: synthetic sessions with
661
+ known ground truth, which is what makes false-positive rates measurable at all.
662
+ That is the right instrument for tuning and the wrong one for the question
663
+ anyone actually asks, which is whether it will kill their work.
664
+
665
+ So it also runs in shadow mode against real agent traffic, on two hosts (a
666
+ Raspberry Pi 5 and an x86-64 NUC) across four provider paths: the `claude` CLI
667
+ under `cg wrap`, and Grok, OpenRouter and local models through `cg-proxy`.
668
+ Shadow mode records the verdict and takes no action, so every session is a free
669
+ observation of what an armed daemon would have done.
670
+
671
+ As of 2026-08-29, since the harness fix in `63d4bc8`:
672
+
673
+ | | observed | healthy | would have been terminated |
674
+ |---|---|---|---|
675
+ | real model sessions | 629 | 475 | **0** |
676
+ | including synthetic soak | — | 3,470 | **0** |
677
+
678
+ Seven distinct real models. The one healthy session ever flagged in this
679
+ pipeline (`real-1787852493-clean`, a Claude-Haiku repo-QA run) was a **harness**
680
+ defect, not a detector one: the test loop's regex did not capture tool *input*,
681
+ so every call hashed identically and `repeat` fired correctly on what it was
682
+ shown. It is annotated in the ledger rather than deleted, and `63d4bc8` fixed
683
+ the loop.
684
+
685
+ That table is a false-positive measurement, not a recall one. Faults in the real loop are injected against short sessions
686
+ that often end before a detector's window elapses, so the catch rate there says
687
+ nothing useful and is not quoted; recall comes from the corpus and the matrix,
688
+ where the ground truth is exact.
689
+
690
+ The honest summary is that in roughly 630 real sessions, across seven models and
691
+ four provider paths, an armed daemon would have terminated nothing healthy. The
692
+ run is still accumulating, so treat that as a floor rather than a final number.
693
+ It is also one operator's machines. It is not a substitute for measuring your
694
+ own traffic before you arm anything.
695
+
696
+ Three things that only showed up at these sample sizes, all now fixed:
697
+
698
+ **The decay hard level needed its own floor.** When a session's learned
699
+ baseline is small, `floor` dominates both decay levels and `warn_k`/`hard_k`
700
+ collapse to the same threshold, so any qualifying slow run fires straight to
701
+ `hard`. On build traffic that was a 1.2% hard-level false positive at
702
+ `--trials 200`+ (invisible at 60). `decay.hard_floor` (default 200s) is now an
703
+ absolute backstop above the slowest gap healthy heavy-tailed traffic
704
+ produces; warn keeps the relative `floor`. Recall unchanged.
705
+
706
+ **`silence` and `decay` need opposite statistics from the same data.** Decay asks
707
+ "has the typical pace collapsed?", which is a central quantile. Silence asks
708
+ "how long may this agent legitimately go quiet?", which is the upper tail. Sharing one baseline
709
+ made bursty workloads false-positive at 94%, because a median baseline calls the
710
+ pauses anomalies when the pauses *are* the workload.
711
+
712
+ **Before the first completed step there is no information at all.** Firing a
713
+ warning there is guessing, and it is how a 20-minute-per-call agent gets flagged
714
+ in its first three minutes. Warnings are now suppressed until a session has
715
+ produced one step; the hard ceiling still applies, widened.
716
+
717
+ ## Testing with agents
718
+
719
+ See **[docs/integrating.md](docs/integrating.md)** for the safe wiring recipe per
720
+ agent (and the instrumentation footguns to avoid), and **[TESTING.md](TESTING.md)**
721
+ for the full Pi runbook. In short:
722
+
723
+ ```bash
724
+ ./tests/run_matrix.sh # 25 rows, 26 checks
725
+ python3 tools/soak.py --sessions 8 --duration 8h # calibrate thresholds, free
726
+ cg wrap --name nightly -- claude -p "..." # real CLI agent
727
+ ```
728
+
729
+ `tools/soak.py` runs concurrent sessions with known ground truth and scores the
730
+ daemon against reality, separating spurious notifications (noise) from
731
+ spurious *terminations* (destroyed work). Ship with the second at zero.
732
+
733
+ `cg wrap` (also `cg-supervise`) wraps any CLI agent with no changes to it, registering
734
+ the child's PID so termination reaches the agent rather than the wrapper.
735
+
736
+ ## Test matrix
737
+
738
+ `./tests/run_matrix.sh` — 25 rows / 26 checks (18 realtime, a protocol fuzz, a ceiling-signal check, an authority check, a packaging check, an environment-knob check, a wrapper output/signal check, and a statusline check). Rows 1 and 5 are the ones that decide whether
739
+ anyone trusts this: a watchdog that catches every stall but occasionally kills
740
+ healthy work gets uninstalled after the first false positive, and deserves to be.
741
+
742
+ ```
743
+ 1 clean completion MUST NOT fire
744
+ 2 hard death (no retire) socket EOF
745
+ 3 blackhole: alive but silent SIGSTOP
746
+ 4 lying heartbeat frames arrive, seq frozen
747
+ 5 legitimately slow work MUST NOT fire
748
+ 6 agent loop identical tool+args
749
+ 7 token burn, no progress
750
+ 8 runaway recursion depth
751
+ 9 oscillation edit/test/revert, every call differs
752
+ 10 retry storm tool calls failing
753
+ 11 budget ceiling spend WITH progress; burn must stay silent
754
+ 12 wall-clock deadline
755
+ 13 decay fast baseline, then a crawl
756
+ 14 decay MUST NOT fire uniformly slow but steady work
757
+ 15 flap crash loop across sessions
758
+ 16 refractory survives restart no duplicate-fire storm
759
+ 17 ARMED: group termination children not orphaned
760
+ 18 PID reuse guard stale identity refused
761
+ 19 protocol contract + fuzz malformed/binary/partial/giant/seq/churn
762
+ 20 ceiling signal reaches agent client drain + budget/deadline delivery
763
+ 21 authority + ceiling guarantees peer cannot make the daemon kill
764
+ 22 installed wheel can do what the docs say
765
+ ```
766
+
767
+ `demo_agent.py` is the fault injector, one process that reproduces every mode on
768
+ demand:
769
+
770
+ ```bash
771
+ python3 demo_agent.py healthy|slow|loop|cycle|stall|burn|budget|decay|errors|deep|freeze|die|children [profile]
772
+ ```
773
+
774
+ ---
775
+
776
+ ## Install as a service
777
+
778
+ ```bash
779
+ sudo mkdir -p /opt/continuity-guard /etc/continuity-guard
780
+ sudo cp -r continuity_guard /opt/continuity-guard/
781
+ sudo cp config.example.toml /etc/continuity-guard/config.toml
782
+ sudo cp systemd/continuity-guard.service /etc/systemd/system/
783
+ sudo systemctl enable --now continuity-guard
784
+ systemctl status continuity-guard
785
+ ```
786
+
787
+ The unit sets `Restart=always` and `OOMScoreAdjust=-500`: the init system is what
788
+ watches the watchdog, and the daemon must outlive the memory pressure that kills
789
+ what it watches.
790
+
791
+ ---
792
+
793
+ ## Status
794
+
795
+ v0.5.3, reference implementation. **26/26 matrix checks across 25 rows** — 18 realtime
796
+ checks, a protocol fuzz, a ceiling-signal check, an authority check, a
797
+ packaging check, an environment-knob check, and a wrapper output/signal
798
+ check — passing on both architectures on this commit: Raspberry Pi 5
799
+ (aarch64, Python 3.13.5) and an x86-64 host (Python 3.12.3). See
800
+ docs/platform-support.md.
801
+
802
+ Pure Python so the contract semantics and thresholds can be validated fast on
803
+ real workloads. The wire protocol (`continuity_guard/protocol.py`, plus
804
+ `sanitize()` as its acceptance rule) is the durable artifact; the native daemon
805
+ reimplements against it once the semantics are proven here.
806
+
807
+ See `docs/` for the engineering history and the measurement behind each
808
+ decision.
809
+
810
+ MIT. Ak2tx LLC.