@bridge4dev/runner 0.53.0 → 0.55.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,223 @@
1
+ import fs from 'node:fs';
2
+ import os from 'node:os';
3
+ import path from 'node:path';
4
+ /**
5
+ * What this machine can say about its own load, read straight from `/proc`.
6
+ *
7
+ * Until this existed, an overloaded dev server was visible in exactly one
8
+ * place: `sar` on the machine itself. The product could not say «this machine
9
+ * is on its knees», so it said nothing, and the first symptom a user saw was a
10
+ * session that had gone quiet (plan `agent-sessions-host-resources.md` §5.3).
11
+ *
12
+ * Two files and nothing else. That is a security property, not an
13
+ * implementation detail: this runner reports to a server, so what it may read
14
+ * for telemetry is listed here in full — `/proc/loadavg` and `/proc/meminfo`,
15
+ * both world-readable, neither containing a path, a command line, an
16
+ * environment variable or a process name. No `/proc/<pid>` walk, no `ps`.
17
+ *
18
+ * Mirror of `packages/shared/src/schemas/host-load.ts` and the `HOST_LOAD_*`
19
+ * block of `packages/shared/src/constants/runner.ts` — the DevBridge side is
20
+ * the source of truth, exactly like `levels.ts` mirrors the level thresholds
21
+ * and `protocol.ts` mirrors the API's wire types. Copied rather than imported
22
+ * on purpose: this package is published to npm on its own and installed by
23
+ * users who have no DevBridge workspace, so a `@devbridge/shared` import would
24
+ * make the published tarball unresolvable (see `recipe-schema.ts`).
25
+ *
26
+ * CHECKED: `host-load.test.ts` reads the shared file and compares every number
27
+ * below against it. A threshold edited on one side only would make the runner
28
+ * send frames the API throws away — or go silent while a card waits for one.
29
+ */
30
+ /**
31
+ * How often the runner LOOKS at `/proc`. Not how often it sends.
32
+ *
33
+ * Fine enough that a card is never more than half a minute stale, coarse
34
+ * enough that reading two small files costs nothing measurable.
35
+ */
36
+ export const HOST_LOAD_SAMPLE_INTERVAL_MS = 30_000;
37
+ /**
38
+ * The heartbeat: an unchanged machine still reports this often.
39
+ *
40
+ * Without it a quiet runner would go silent and the API's key would expire,
41
+ * turning «nothing is happening» into «we have no idea» on the card. With it,
42
+ * silence really does mean the runner stopped talking.
43
+ *
44
+ * Bound to the API's `HOST_LOAD_TTL_MS` (120 s), and the order is the design:
45
+ * a heartbeat LONGER than the TTL means a healthy quiet machine shows «no
46
+ * data» for the gap between them. The first draft had 5 minutes against a
47
+ * 2-minute TTL — three silent minutes out of every five.
48
+ */
49
+ export const HOST_LOAD_HEARTBEAT_MS = 60_000;
50
+ /**
51
+ * How much load1 must move before an otherwise unchanged sample earns a frame.
52
+ *
53
+ * Same principle as `publishSlots`: a tick that would repeat what the server
54
+ * already knows is not sent at all. 0.5 is below the resolution at which a
55
+ * person reads the number and well above the noise of an idle machine.
56
+ */
57
+ export const HOST_LOAD_MIN_DELTA_LOAD1 = 0.5;
58
+ /** The same idea for memory, as a fraction — see `hostLoadChangedEnough`. */
59
+ export const HOST_LOAD_MIN_DELTA_MEM_RATIO = 0.05;
60
+ const PROC_DIR = '/proc';
61
+ /** kB in `/proc/meminfo` means kibibytes, and has since the field was added. */
62
+ const MEMINFO_UNIT_BYTES = 1024;
63
+ /**
64
+ * `MemTotal: 16307180 kB` → `['MemTotal', 16307180 * 1024]`.
65
+ *
66
+ * The `kB` suffix is required rather than assumed. A handful of `/proc/meminfo`
67
+ * rows carry no unit at all (`HugePages_Total`), and multiplying one of those
68
+ * by 1024 because it happened to be named like the row we wanted would produce
69
+ * a number that is wrong by three orders of magnitude and looks perfectly
70
+ * plausible on a card.
71
+ */
72
+ const MEMINFO_LINE = /^([A-Za-z0-9_()]+):\s+(\d+)\s+kB$/;
73
+ function parseMeminfo(text) {
74
+ const values = new Map();
75
+ for (const line of text.split('\n')) {
76
+ const match = MEMINFO_LINE.exec(line.trim());
77
+ if (!match)
78
+ continue;
79
+ const [, key, amount] = match;
80
+ if (!key || amount === undefined)
81
+ continue;
82
+ const parsed = Number(amount);
83
+ if (!Number.isFinite(parsed))
84
+ continue;
85
+ values.set(key, parsed * MEMINFO_UNIT_BYTES);
86
+ }
87
+ return values;
88
+ }
89
+ /** `0.42 0.53 0.60 2/1234 56789` → the three windows, or `null` if it is not that. */
90
+ function parseLoadavg(text) {
91
+ const parts = text.trim().split(/\s+/);
92
+ const windows = parts.slice(0, 3).map(Number);
93
+ if (windows.length < 3)
94
+ return null;
95
+ for (const value of windows) {
96
+ if (!Number.isFinite(value) || value < 0)
97
+ return null;
98
+ }
99
+ const [load1, load5, load15] = windows;
100
+ return { load1, load5, load15 };
101
+ }
102
+ /**
103
+ * Measure this machine, or say honestly that we cannot.
104
+ *
105
+ * `null` is a first-class answer and every caller must treat it as «send
106
+ * nothing». There is deliberately no `process.platform` check in front of it:
107
+ * the absence of the files IS the check, and it is the wider one — it also
108
+ * covers a Linux container with `/proc` masked or mounted `hidepid`, which a
109
+ * platform test would sail straight past into an exception. The runner is
110
+ * Linux-only by its installer, but «only runs on Linux» must never mean
111
+ * «crashes anywhere else»: this is called from a timer, and an exception here
112
+ * would take the whole session supervisor with it.
113
+ */
114
+ export function readHostLoad(sources = {}) {
115
+ const procDir = sources.procDir ?? PROC_DIR;
116
+ try {
117
+ const load = parseLoadavg(fs.readFileSync(path.join(procDir, 'loadavg'), 'utf8'));
118
+ if (!load)
119
+ return null;
120
+ const meminfo = parseMeminfo(fs.readFileSync(path.join(procDir, 'meminfo'), 'utf8'));
121
+ const memAvailableBytes = meminfo.get('MemAvailable');
122
+ // No `MemAvailable` means a kernel older than 3.14 (2014). Reporting
123
+ // `MemFree` in its place would be worse than reporting nothing: on a
124
+ // healthy machine it is near zero by design, so every such card would
125
+ // permanently read «out of memory».
126
+ if (memAvailableBytes === undefined)
127
+ return null;
128
+ // Swap is reported only when BOTH halves are there. A kernel built without
129
+ // swap support has neither, and 0/0 is the truthful answer for it — but
130
+ // one half without the other is a shape we do not understand, and guessing
131
+ // `free = 0` for a real swap area would paint that machine as permanently
132
+ // paging, which is the one verdict this frame exists to make trustworthy.
133
+ const swapTotalRaw = meminfo.get('SwapTotal');
134
+ const swapFreeRaw = meminfo.get('SwapFree');
135
+ const swapKnown = swapTotalRaw !== undefined && swapFreeRaw !== undefined;
136
+ const swapTotalBytes = swapKnown ? swapTotalRaw : 0;
137
+ const swapFreeBytes = swapKnown ? swapFreeRaw : 0;
138
+ const cpuCount = sources.cpuCount ?? os.cpus().length;
139
+ // A frame the API's schema would reject is worse than no frame, and
140
+ // `os.cpus()` has been seen returning an empty array in containers. Nor may
141
+ // it be quietly replaced by 1: load1 is read as a ratio of this number, so
142
+ // a made-up core count would paint a working eight-core machine red.
143
+ if (!Number.isInteger(cpuCount) || cpuCount < 1)
144
+ return null;
145
+ const at = (sources.now?.() ?? new Date()).toISOString();
146
+ return {
147
+ ...load,
148
+ cpuCount,
149
+ memAvailableBytes,
150
+ swapTotalBytes,
151
+ swapFreeBytes,
152
+ at,
153
+ };
154
+ }
155
+ catch {
156
+ // ENOENT on a Mac, EACCES in a locked-down container, EIO on a machine
157
+ // that is already having a very bad day. All of them mean the same thing
158
+ // to the caller, and none of them is worth a log line every 30 seconds.
159
+ return null;
160
+ }
161
+ }
162
+ /**
163
+ * Is the heartbeat due, given how long ago the last frame actually went out?
164
+ *
165
+ * The second half of the answer is the whole reason this is a function: a
166
+ * backwards clock step makes «how long ago» NEGATIVE, and a machine whose load
167
+ * never moves would then say nothing for the entire offset — an hour of silence
168
+ * on a machine that may be on fire. A clock correction is due, not early.
169
+ *
170
+ * The API makes the same allowance for the same step, and has to: it drops
171
+ * frames older than the one it holds unless the step is larger than a
172
+ * measurement's lifetime (`HOST_LOAD_TTL_MS`, `runner-gateway.ts`). Half of this
173
+ * fix on one side of the socket is no fix at all.
174
+ */
175
+ export function hostLoadHeartbeatDue(sinceLastSentMs, heartbeatMs) {
176
+ return sinceLastSentMs >= heartbeatMs || sinceLastSentMs < 0;
177
+ }
178
+ /**
179
+ * Is this sample worth a frame at all? The «a quiet runner says nothing» rule.
180
+ *
181
+ * Same shape as `publishSlots`: what the server already knows is not repeated.
182
+ * The alternative is 2 880 identical frames a day per machine, each one waking
183
+ * the gateway, a Redis write and every open dashboard socket.
184
+ *
185
+ * The memory threshold is 5 % of the machine's TOTAL memory, not 5 % of what
186
+ * is currently available. Two reasons, and both were the deciding one:
187
+ *
188
+ * - a fraction of `memAvailable` shrinks as the machine fills, so the frames
189
+ * would get chattiest exactly when the machine is least able to afford it —
190
+ * 5 % of the last 200 MB is a 10 MB trigger;
191
+ * - `memTotal` is fixed per machine, so «the card moves when a GB moves» reads
192
+ * the same on every server instead of meaning something different on each.
193
+ *
194
+ * Swap counts too, though the plan names only load and memory. It is one of the
195
+ * three verdicts behind «overloaded», and it is the only one with no other
196
+ * trigger: a machine that starts paging with its load and its `MemAvailable`
197
+ * both flat would otherwise wait up to five minutes for the heartbeat to say
198
+ * so, and paging is the state whose wall-clock cost is unbounded.
199
+ */
200
+ export function hostLoadChangedEnough(previous, next, memTotalBytes = os.totalmem()) {
201
+ // Nothing said yet — the first measurement after a (re)connect is always news.
202
+ if (!previous)
203
+ return true;
204
+ if (Math.abs(next.load1 - previous.load1) >= HOST_LOAD_MIN_DELTA_LOAD1)
205
+ return true;
206
+ // A machine that cannot say how much memory it has still gets a working
207
+ // threshold: the sample's own availability is a poor denominator but a
208
+ // finite one, and the alternative is dividing by zero forever.
209
+ const memBase = memTotalBytes > 0 ? memTotalBytes : next.memAvailableBytes;
210
+ const memDelta = Math.abs(next.memAvailableBytes - previous.memAvailableBytes);
211
+ if (memBase > 0 && memDelta >= memBase * HOST_LOAD_MIN_DELTA_MEM_RATIO)
212
+ return true;
213
+ // Swap size itself can change (`swapon`), and that is news by the same rule.
214
+ if (next.swapTotalBytes !== previous.swapTotalBytes)
215
+ return true;
216
+ if (next.swapTotalBytes > 0) {
217
+ const swapDelta = Math.abs(next.swapFreeBytes - previous.swapFreeBytes);
218
+ if (swapDelta >= next.swapTotalBytes * HOST_LOAD_MIN_DELTA_MEM_RATIO)
219
+ return true;
220
+ }
221
+ return false;
222
+ }
223
+ //# sourceMappingURL=host-load.js.map
package/dist/index.js CHANGED
@@ -17,12 +17,14 @@ import { Supervisor } from './supervisor.js';
17
17
  import { readStatusFile, isPidAlive, writeStatusFile, STATUS_FRESH_MS } from './status-file.js';
18
18
  import { RunnerWsClient } from './ws-client.js';
19
19
  import { RUNNER_VERSION } from './version.js';
20
- import { buildUnit, cpuQuotaPercent, limitsOverrideIsOutdated, limitsOverridePath, memoryPolicy, readMemoryFacts, unitExecTarget, unitPath, writeLimitsOverride, LIMITS_VERSION, } from './service-unit.js';
20
+ import { buildUnit, cpuQuotaPercent, devbridgeSliceOverridePath, limitsOverrideIsOutdated, limitsOverridePath, memoryPolicy, readMemoryFacts, sessionsSliceOverridePath, unitExecTarget, unitPath, writeLimitsOverride, DEVBRIDGE_SLICE, LIMITS_VERSION, SESSION_CPU_WEIGHT, SESSIONS_SLICE, } from './service-unit.js';
21
+ import { initSessionCage, listSessionScopes, sessionCage, sweepOrphanSessionScopes, SESSION_TASKS_MAX, } from './session-cage.js';
21
22
  import { SEARCH_GUARD_ENABLED, claudeSettingsPath, installSearchGuard, removeSearchGuard, searchGuardCommand, searchGuardHome, searchGuardHookPath, searchGuardStatus, } from './claude-settings.js';
22
23
  import { readOomKills, recordCrash, takeLastExit } from './crash-note.js';
23
24
  import { agentAuthStatuses } from './auth-relay.js';
24
25
  import { addSafeDirectory, agentConfigContour, dockerCheck, ensureAgentPath, firstUnreachableAncestor, hasSafeDirectory, inspectPath, knownWorkspacePaths, lingerEnabled, nodeCheck, otherHomeWithAgents, runnerIdentity, safeDirectoryCommand, systemctlHint, systemdUserBusReachable, systemdUserEnv, } from './environment.js';
25
26
  import { mcpConfigDir } from './paths.js';
27
+ import { readMemoryFactsFromSystemd } from './systemd-memory.js';
26
28
  const execFileAsync = promisify(execFile);
27
29
  /**
28
30
  * Identity of this process, and the note the previous one left.
@@ -405,6 +407,55 @@ function runnerCapabilities(apiUrlOverride) {
405
407
  // Facts passed on purpose: since 0.39.0 «current» also means the measured
406
408
  // ceiling still fits this machine, not just that the version matches.
407
409
  limitsCurrent: !limitsOverrideIsOutdated(undefined, undefined, readMemoryFacts()),
410
+ /**
411
+ * How the sessions on this machine are actually contained (plan §5.4.3).
412
+ *
413
+ * `scope` — a cgroup and a memory ceiling per session; `nice-only` — the
414
+ * machine cannot do that (cgroup v1, no user bus, a foreign supervisor)
415
+ * and only the CPU priority applies; `none` — not even that.
416
+ *
417
+ * Reported rather than assumed because the containment fails QUIETLY:
418
+ * `systemd-run` accepts every limit on a cgroup v1 machine and applies
419
+ * none of them. Without this field «it did not work at the client's» is
420
+ * something to be guessed at; with it the card can say so.
421
+ */
422
+ sessionCage: sessionCage().mode,
423
+ // Why not `scope`, in one sentence, for the card to show under it.
424
+ // Empty string when the cage is on.
425
+ sessionCageReason: sessionCage().reason,
426
+ /**
427
+ * The ladder one session lives under (#387), in bytes. Null when there is
428
+ * no cage — then the only ceiling is the service's, over all of them.
429
+ *
430
+ * `High` is the brake: past it the kernel slows the session down instead
431
+ * of killing. `Max` is the wall, far above — the machine's measured
432
+ * headroom, where only a runaway ever arrives. Runners up to 0.54.0 sent
433
+ * `Max` alone, and it was the kill line at 2.5 GiB; a card that sees no
434
+ * `High` is looking at one of those.
435
+ */
436
+ sessionMemoryHighBytes: sessionCage().memoryHighBytes,
437
+ sessionMemoryMaxBytes: sessionCage().memoryMaxBytes,
438
+ /** Swap one session may push into while braked. 0 on a machine without swap. */
439
+ sessionSwapMaxBytes: sessionCage().swapMaxBytes,
440
+ /**
441
+ * Whether reaching the wall costs the session ONE process (`OOMPolicy=
442
+ * continue` on the scope) or is left to that systemd's own default —
443
+ * older systemds do not take the setting on a scope at all. The card
444
+ * words the wall by it.
445
+ */
446
+ sessionOomContinue: sessionCage().oomContinue,
447
+ /**
448
+ * The ceiling over ALL sessions, as systemd has it in force — not as the
449
+ * drop-in on disk says.
450
+ *
451
+ * The live probe proves the personal ceiling and nothing else: a
452
+ * `MemoryMax` on a scope applies whatever the parent slice holds. So a
453
+ * machine whose `daemon-reload` never landed reported `scope` and
454
+ * `limitsCurrent: true` while three sessions of 2.5 GB ran over a slice
455
+ * at `infinity`, and the card said «Sessions capped»
456
+ * (QA-2026-09-07 MAJOR-3). Null is that machine, and the card says so.
457
+ */
458
+ sessionsMemoryMaxBytes: sessionCage().sessionsSliceMemoryMaxBytes,
408
459
  },
409
460
  /**
410
461
  * Identity of THIS process, so the API can tell a network blink from a
@@ -482,6 +533,11 @@ async function cmdPair(args) {
482
533
  }
483
534
  const apiUrl = (argValue(args, '--api') ?? 'https://api.bridge4.dev').replace(/\/$/, '');
484
535
  const name = argValue(args, '--name') ?? os.hostname();
536
+ // Probed here as well as at daemon start, so the very first record of this
537
+ // server already carries the truth about its containment instead of the safe
538
+ // default. Costs one throwaway process, once.
539
+ await initSessionCage();
540
+ const capabilities = runnerCapabilities(apiUrl);
485
541
  const response = await fetch(`${apiUrl}/api/v1/dev/servers/claim`, {
486
542
  method: 'POST',
487
543
  headers: { 'content-type': 'application/json' },
@@ -490,7 +546,7 @@ async function cmdPair(args) {
490
546
  name,
491
547
  runnerVersion: RUNNER_VERSION,
492
548
  osInfo: `${os.type()} ${os.release()} ${os.arch()}`.slice(0, 200),
493
- capabilities: runnerCapabilities(apiUrl),
549
+ capabilities,
494
550
  }),
495
551
  });
496
552
  const body = (await response.json().catch(() => null));
@@ -553,8 +609,17 @@ const RESTART_DELAY_MS = 1_500;
553
609
  */
554
610
  async function repairResourceLimits() {
555
611
  try {
556
- const facts = readMemoryFacts();
557
- if (facts && memoryPolicy(facts).starved) {
612
+ const { facts, sessionsUsageBytes } = await readMemoryFactsFromSystemd();
613
+ if (facts === null) {
614
+ // Neither systemd nor the cgroup files could answer, which on the daemon's
615
+ // own path means the machine is in a shape this policy cannot reason about
616
+ // at all. Leaving what is in force in force is the safe half of the
617
+ // decision; the loud line is the other half, because the alternative is a
618
+ // machine that silently keeps an old ceiling forever.
619
+ log.warn('daemon: resource limits unchanged — memory usage could not be measured; retrying in an hour', { recheckMs: LIMITS_RECHECK_MS });
620
+ return;
621
+ }
622
+ if (memoryPolicy(facts).starved) {
558
623
  // The only signal the machine's owner will ever get that the neighbours
559
624
  // have taken the box: the honest headroom was below the 2 GB floor, so the
560
625
  // ceiling we are about to write sits above what is actually free.
@@ -563,12 +628,12 @@ async function repairResourceLimits() {
563
628
  totalMB: Math.round(facts.totalBytes / 1048576),
564
629
  });
565
630
  }
566
- if (!writeLimitsOverride(false, undefined, facts))
631
+ if (!writeLimitsOverride(false, undefined, facts, sessionsUsageBytes))
567
632
  return;
568
633
  log.warn('daemon: resource limits drop-in written — reloading systemd', {
569
634
  path: limitsOverridePath(),
570
635
  version: LIMITS_VERSION,
571
- ...(facts ? memoryPolicy(facts) : {}),
636
+ ...memoryPolicy(facts),
572
637
  });
573
638
  await execFileAsync('systemctl', ['--user', 'daemon-reload'], {
574
639
  timeout: 15_000,
@@ -657,33 +722,6 @@ function repairSearchGuard() {
657
722
  * check writes nothing when the version matches and the ceiling still fits.
658
723
  */
659
724
  const LIMITS_RECHECK_MS = 3_600_000;
660
- /**
661
- * What the SERVICE's cgroup holds, for the paths that are not the service.
662
- *
663
- * `readMemoryFacts()` defaults to reading `/proc/self`, which is right for the
664
- * daemon and wrong for everything a person types: `doctor --fix` and
665
- * `install-service` run in the operator's own `session-N.scope`, worth a few MB.
666
- * Measuring that and calling it «what the runner holds» made the CLI compute a
667
- * ceiling 34 % away from the daemon's answer on a real host — each path then saw
668
- * the other as drift and rewrote the file, forever.
669
- *
670
- * `MemoryCurrent` is `memory.current`, so it still counts page cache the way
671
- * `MemAvailable` does. Subtracting it is not worth a second systemd call here:
672
- * the CLI paths run once, by hand, and erring toward a LOWER ceiling is the safe
673
- * direction. Returns null when the service is not running or systemd cannot be
674
- * reached — the caller then has no facts, and writes nothing rather than writing
675
- * a ceiling with no floor under it.
676
- */
677
- async function serviceMemoryCurrent() {
678
- try {
679
- const { stdout } = await execFileAsync('systemctl', ['--user', 'show', 'devbridge-runner', '-p', 'MemoryCurrent', '--value'], { timeout: 10_000, env: systemdUserEnv() });
680
- const value = Number(stdout.trim());
681
- return Number.isFinite(value) && value > 0 ? value : null;
682
- }
683
- catch {
684
- return null;
685
- }
686
- }
687
725
  /**
688
726
  * Ticket #119. `ClaudeAdapter.stop()` removes each session's MCP config file,
689
727
  * but a kill -9, an OOM or a machine reboot leaves it behind — a live project
@@ -719,7 +757,17 @@ async function cmdDaemon() {
719
757
  const config = requireConfig();
720
758
  log.info('daemon starting', { version: RUNNER_VERSION, server: config.server.name });
721
759
  sweepOrphanedMcpConfigs();
760
+ // Nothing of ours is running yet, so every `devbridge-session-*.scope` on this
761
+ // machine belongs to a process that is gone. Stopping the scope takes the
762
+ // whole tree under it — which is the answer to the `ugrep` that outlived its
763
+ // session by 10 h 51 min on 16.08, and to every scope the OOM killer left in
764
+ // `failed` (a name systemd will otherwise refuse to reuse).
765
+ await sweepOrphanSessionScopes();
722
766
  await repairResourceLimits();
767
+ // After the drop-ins are on disk and reloaded, never before: the probe below
768
+ // creates `devbridge-sessions.slice`, and a slice first loaded without its
769
+ // policy would hold no ceiling until the next daemon-reload.
770
+ await initSessionCage();
723
771
  repairSearchGuard();
724
772
  // A Claude token this runner captured through the sign-in relay. Applied
725
773
  // BEFORE any adapter exists, because `scrubbedEnv()` copies it out of this
@@ -892,8 +940,29 @@ async function cmdInstallService() {
892
940
  // `buildLimitsOverride`. Forced here: a fresh install must have it even if a
893
941
  // file from an older runner is already sitting there. Facts come from the
894
942
  // service rather than from `/proc/self`, which here is the installing shell.
895
- writeLimitsOverride(true, undefined, readMemoryFacts(await serviceMemoryCurrent()));
896
- print(`Wrote ${limitsOverridePath()}`);
943
+ // The slice reading goes with them: re-installing over a machine that is
944
+ // running sessions right now must not write a ceiling under them.
945
+ //
946
+ // Unmeasurable is a WARNING here, not a refusal, and the order above is why:
947
+ // the unit is already on disk, so the machine gets a runner either way. The
948
+ // states that produce «unmeasurable» — a failed service, a service mid
949
+ // auto-restart, an unhappy user bus — are exactly the states someone runs
950
+ // `install.sh --repair` from, and a repair that installs nothing because it
951
+ // could not size a ceiling is a repair that does not repair.
952
+ const installMem = await readMemoryFactsFromSystemd();
953
+ if (installMem.facts === null) {
954
+ print('warning: could not measure this machine — resource limits were left as they are. ' +
955
+ 'Nothing was removed; the daemon re-measures hourly, or run `devbridge-runner doctor --fix`.');
956
+ }
957
+ else {
958
+ writeLimitsOverride(true, undefined, installMem.facts, installMem.sessionsUsageBytes);
959
+ print(`Wrote ${limitsOverridePath()}`);
960
+ // Two more files since 0.54.0: the sessions no longer live inside the
961
+ // service's cgroup, so the ceiling over all of them and the CPU share that
962
+ // keeps the daemon ahead of them belong to the slices (`session-cage.ts`).
963
+ print(`Wrote ${sessionsSliceOverridePath()}`);
964
+ print(`Wrote ${devbridgeSliceOverridePath()}`);
965
+ }
897
966
  // Привратник шаблонов поиска — та же логика, что и у политики ресурсов:
898
967
  // свежая установка должна получить его сразу. Пишем в ЧУЖОЙ файл настроек,
899
968
  // поэтому молча пропускаем, если тронуть его нельзя (`claude-settings.ts`).
@@ -1330,7 +1399,7 @@ async function cmdDoctor(args) {
1330
1399
  print(` fits sessions ~${advised} (at ~1.5 GB per session under load)`);
1331
1400
  print('');
1332
1401
  print('Service limits');
1333
- const memFacts = readMemoryFacts(await serviceMemoryCurrent());
1402
+ const { facts: memFacts, sessionsUsageBytes: memSessionsUsage } = await readMemoryFactsFromSystemd();
1334
1403
  const outdated = limitsOverrideIsOutdated(undefined, undefined, memFacts);
1335
1404
  print(` drop-in ${limitsOverridePath()}`);
1336
1405
  print(` version ${outdated ? `MISSING or OLD (want ${LIMITS_VERSION})` : LIMITS_VERSION}`);
@@ -1346,6 +1415,65 @@ async function cmdDoctor(args) {
1346
1415
  ? `${mb(memFacts.availableBytes)} available + ${mb(memFacts.ownUsageBytes)} ours`
1347
1416
  : `55% of ${mb(memFacts.totalBytes)} — still booting, remeasured on the next start`}`);
1348
1417
  }
1418
+ else {
1419
+ // Не молча: без фактов о машине политика не считается вовсе, и `--fix`
1420
+ // тогда НИЧЕГО не пишет — ни на сервис, ни на слайс. Строка про слайс
1421
+ // остаётся полезной сама по себе: она говорит, что стоит на кону, если
1422
+ // измерение так и не появится.
1423
+ print(" memory ceiling not computed — this machine's usage is unreadable");
1424
+ print(' --fix leaves every limits file exactly as it is');
1425
+ print(` sessions slice ${memSessionsUsage === null
1426
+ ? 'unreadable too'
1427
+ : memSessionsUsage === 0
1428
+ ? 'holds nothing right now'
1429
+ : `holds ${Math.round(memSessionsUsage / 1024 / 1024)} MB — a ceiling under that would kill it`}`);
1430
+ }
1431
+ // The cage, and the reason it matters more than the numbers above it: those
1432
+ // are the ceiling over the WHOLE service, and a session that has left the
1433
+ // service's cgroup is not under them any more. If this section says anything
1434
+ // but `scope`, the only containment on this machine is the CPU priority.
1435
+ const cage = await initSessionCage();
1436
+ print('');
1437
+ print('Session cage (a cgroup per session)');
1438
+ print(` mode ${cage.mode}`);
1439
+ if (cage.reason)
1440
+ print(` why ${cage.reason}`);
1441
+ print(` slice ${SESSIONS_SLICE} (under ${DEVBRIDGE_SLICE}, CPUWeight=${SESSION_CPU_WEIGHT})`);
1442
+ print(` drop-ins ${sessionsSliceOverridePath()}`);
1443
+ print(` ${devbridgeSliceOverridePath()}`);
1444
+ if (cage.mode === 'scope') {
1445
+ const mb = (bytes) => bytes === null ? 'infinity' : `${Math.round(bytes / 1024 / 1024)} MB`;
1446
+ print(` per session MemoryHigh ${mb(cage.memoryHighBytes)} (brake: slows, never kills), MemoryMax ${mb(cage.memoryMaxBytes)} (wall), MemorySwapMax ${mb(cage.swapMaxBytes)}, TasksMax ${SESSION_TASKS_MAX}`);
1447
+ print(` at the wall ${cage.oomContinue
1448
+ ? 'OOMPolicy=continue — the kernel kills the hungriest process, the session stays up'
1449
+ : 'this systemd does not take OOMPolicy on a scope (it is newer than the rest of the cage), so what happens at the wall is its own default — on systemd 253+ that is «stop the whole session»'}`);
1450
+ // What systemd has IN FORCE on the slice, beside the paths of the files
1451
+ // that were supposed to put it there. «В файле 7680M, действует 6.0G» is the
1452
+ // lesson of §5.5, and the drop-in of a slice can fail to land in exactly the
1453
+ // same way: a reload that never happened leaves every session personally
1454
+ // capped and collectively unlimited (QA-2026-09-07 MAJOR-3).
1455
+ print(` all sessions MemoryMax ${mb(cage.sessionsSliceMemoryMaxBytes)} in force on ${SESSIONS_SLICE}`);
1456
+ if (cage.sessionsSliceMemoryMaxBytes === null) {
1457
+ print(' NOT CAPPED — the drop-in above has not been applied.');
1458
+ print(` Apply with: ${systemctlHint('daemon-reload')}`);
1459
+ }
1460
+ print(` service ceiling ${mb(cage.serviceMemoryMaxBytes)} (the per-session brake is half of the containing ceiling, capped at 2.5 GiB; the wall IS the containing ceiling)`);
1461
+ print(` expand-env flag ${cage.expandEnvironmentFlag ? 'passed (systemd ≥ 254)' : 'not passed — this systemd does not know it, and --scope does not expand anyway'}`);
1462
+ }
1463
+ else {
1464
+ print(' per session none — only the CPU priority from `process-priority.ts` applies');
1465
+ }
1466
+ const scopes = await listSessionScopes();
1467
+ if (scopes.length === 0) {
1468
+ print(' live scopes none');
1469
+ }
1470
+ else {
1471
+ print(` live scopes ${scopes.length}`);
1472
+ for (const scope of scopes) {
1473
+ const mb = (bytes) => bytes === null ? '?' : `${Math.round(bytes / 1024 / 1024)} MB`;
1474
+ print(` ${scope.unit} ${scope.activeState ?? '?'} ${mb(scope.memoryCurrentBytes)} of ${mb(scope.memoryMaxBytes)} tasks ${scope.tasksCurrent ?? '?'}${scope.result && scope.result !== 'success' ? ` ${scope.result}` : ''}`);
1475
+ }
1476
+ }
1349
1477
  let effective;
1350
1478
  try {
1351
1479
  const { stdout } = await execFileAsync('systemctl', [
@@ -1372,6 +1500,19 @@ async function cmdDoctor(args) {
1372
1500
  print('Effective (systemd)');
1373
1501
  for (const line of effective)
1374
1502
  print(` ${line}`);
1503
+ // «В файле 7680M, действует 6.0G». Ровно это и получалось, пока самообновление
1504
+ // звало `systemctl --user daemon-reload` без `XDG_RUNTIME_DIR`: systemd отвечал
1505
+ // «Failed to connect to bus» и выходил с кодом 0, drop-in переписывался, а
1506
+ // systemd продолжал жить по старому юниту (план §5.5). Причину починили, но
1507
+ // расхождение может прийти и иначе — правкой юнита руками, оборванным
1508
+ // обновлением, — и без этой строки строки выше выглядят как то, что
1509
+ // действует, хотя это уже не так.
1510
+ if ((await systemctlProperty('NeedDaemonReload')) === 'yes') {
1511
+ print('');
1512
+ print('STALE — systemd has NOT re-read the unit since it changed on disk,');
1513
+ print(' so the numbers above are the OLD ones, not what the drop-in says.');
1514
+ print(` Apply with: ${systemctlHint('daemon-reload')}`);
1515
+ }
1375
1516
  // Привратник шаблонов поиска. Печатается всегда, в том числе когда он ВЫКЛЮЧЕН
1376
1517
  // в коде: «стоит, хотя выключен» — это и есть та машина, которую пропустило
1377
1518
  // обновление, и увидеть её больше неоткуда.
@@ -1438,10 +1579,24 @@ async function cmdDoctor(args) {
1438
1579
  process.exit(1);
1439
1580
  return;
1440
1581
  }
1441
- const fixFacts = readMemoryFacts(await serviceMemoryCurrent());
1442
- writeLimitsOverride(true, undefined, fixFacts);
1582
+ const { facts: fixFacts, sessionsUsageBytes: fixSessionsUsage } = await readMemoryFactsFromSystemd();
1443
1583
  print('');
1444
- print(`Wrote ${limitsOverridePath()}`);
1584
+ // Unmeasurable leaves the files exactly as they are and says so — the one
1585
+ // thing this must never do is replace a ceiling in force with a guess, or
1586
+ // rewrite the file without its `MemoryMax` line, which RESETS the ceiling.
1587
+ // The rest of the repair still runs: `doctor --fix` is also the search guard,
1588
+ // the unit file and the daemon-reload, and a machine that cannot be measured
1589
+ // this second still needs those. The non-zero exit at the end is what keeps
1590
+ // «could not do the main thing» from reading as success to automation.
1591
+ if (fixFacts === null) {
1592
+ print('Resource limits: LEFT UNCHANGED — this machine could not be measured.');
1593
+ print(' Nothing was removed. Re-run once `systemctl --user` answers again.');
1594
+ }
1595
+ else {
1596
+ // Keep the service and the sessions under the same measured policy.
1597
+ writeLimitsOverride(true, undefined, fixFacts, fixSessionsUsage);
1598
+ print(`Wrote ${limitsOverridePath()}`);
1599
+ }
1445
1600
  const guardOutcome = applySearchGuard(print);
1446
1601
  if (guardOutcome === 'installed') {
1447
1602
  print(`Installed search-pattern guard into ${claudeSettingsPath(searchGuardHome())}`);
@@ -1458,7 +1613,19 @@ async function cmdDoctor(args) {
1458
1613
  const applied = memoryPolicy(fixFacts);
1459
1614
  if (applied.maxBytes > honest.maxBytes) {
1460
1615
  const mb = (b) => Math.round(b / 1024 / 1024);
1461
- print(`note: ceiling raised to ${mb(applied.maxBytes)} MB — the service already holds ${mb(fixFacts.ownUsageBytes)} MB, and ${mb(honest.maxBytes)} MB would kill it on reload. Re-run when idle.`);
1616
+ // Which cgroup forced the ceiling up, not just «the service»: on a loaded
1617
+ // machine it is the sessions slice, and naming the service there sent the
1618
+ // reader looking at a daemon holding 400 MB (QA-2026-09-07 BLOCKER-1).
1619
+ // The SAME numbers the floor is computed from (`managedUsageFloorBytes`),
1620
+ // not the headroom ones: where the split is unknown those two differ, and
1621
+ // a note sourced from the wrong pair names the wrong cgroup and a smaller
1622
+ // number than the one that actually forced the ceiling up.
1623
+ const ownFloor = fixFacts.ownFloorBytes ?? fixFacts.ownUsageBytes;
1624
+ const sessionsFloor = fixFacts.sessionsFloorBytes ?? fixFacts.sessionsUsageBytes;
1625
+ const holder = sessionsFloor > ownFloor
1626
+ ? `the sessions on this machine already hold ${mb(sessionsFloor)} MB`
1627
+ : `the service already holds ${mb(ownFloor)} MB`;
1628
+ print(`note: ceiling raised to ${mb(applied.maxBytes)} MB — ${holder}, and ${mb(honest.maxBytes)} MB would kill it on reload. Re-run when idle.`);
1462
1629
  }
1463
1630
  }
1464
1631
  try {
@@ -1469,6 +1636,10 @@ async function cmdDoctor(args) {
1469
1636
  catch (error) {
1470
1637
  fail(`daemon-reload failed: ${String(error instanceof Error ? error.message : error)}`);
1471
1638
  }
1639
+ // Everything else was repaired; the limits were not. Said with the exit code
1640
+ // as well as with the line above, because `doctor --fix` is run by scripts.
1641
+ if (fixFacts === null)
1642
+ process.exit(1);
1472
1643
  }
1473
1644
  // ─── set-token ───────────────────────────────────────────────────────
1474
1645
  function cmdSetToken(args) {
@@ -0,0 +1,55 @@
1
+ /**
2
+ * Everything an agent starts runs at a lower CPU priority than the daemon that
3
+ * supervises it.
4
+ *
5
+ * The failure this exists for is the runner starving itself with its own
6
+ * children (16.08): a session's build saturated the box, the daemon lost four
7
+ * heartbeats in a row, the server went Offline and every session on that
8
+ * machine answered 504 — while the work it was doing was fine. The daemon's job
9
+ * during a heavy turn is a few milliseconds of socket traffic; it should not
10
+ * have to queue behind a `tsc` it launched itself.
11
+ *
12
+ * **What this gives.** Inside the service's own cgroup the daemon (nice 0) gets
13
+ * the processor before the agents (nice 10); neighbouring sessions, all at 10,
14
+ * still share it evenly between themselves. Children inherit the nice value at
15
+ * fork, so a `vitest` the agent starts through Bash — several levels down from
16
+ * the process we renice — is covered without us knowing about it.
17
+ *
18
+ * Inheritance is why every call site is the line straight after its `spawn`:
19
+ * measured here, a grandchild forked AFTER the call comes up at 10 and one
20
+ * forked in the microseconds before it stays at 0. An agent needs hundreds of
21
+ * milliseconds to boot before it forks anything, so that window is empty in
22
+ * practice — but it is a window, and it only grows if the call drifts down the
23
+ * function.
24
+ *
25
+ * **What it does NOT give.** Against processes OUTSIDE the cgroup (production
26
+ * in `system.slice`) it does nothing at all: across cgroups the split is
27
+ * decided by `cpu.weight`, and nice only orders tasks within one. It does not
28
+ * touch memory, which is the mechanism behind three of the four incidents on
29
+ * other people's machines. This is the approach to stage 2 (a scope per session
30
+ * with a memory ceiling) and the fallback for it on cgroup v1, where stage 2
31
+ * cannot work — not a replacement for it.
32
+ *
33
+ * Priority is a convenience, not correctness: nothing here throws. A session
34
+ * that runs at the wrong priority is a slower machine; a session that fails to
35
+ * start because renicing failed is a broken product.
36
+ */
37
+ /**
38
+ * The nice value every process the runner spawns for an agent gets.
39
+ *
40
+ * 10 rather than 19: the point is to lose to the daemon and to anything the
41
+ * owner is doing by hand, not to be scheduled last behind every background cron
42
+ * on the box. The scheduler's weight table gives nice 10 about a ninth of the
43
+ * share of nice 0 under contention (1024 → 110), which is all the room the
44
+ * heartbeat needs — 19 would buy an order of magnitude more and cost a session
45
+ * its throughput whenever anything else on the machine woke up.
46
+ */
47
+ export declare const NICE = 10;
48
+ /**
49
+ * Push one spawned process down to {@link NICE}. Never throws.
50
+ *
51
+ * Takes `number | undefined` because that is exactly what `child.pid` is: a
52
+ * spawn that failed has none, and the caller should not have to ask.
53
+ */
54
+ export declare function lowerPriority(pid: number | undefined): void;
55
+ //# sourceMappingURL=process-priority.d.ts.map