@jonmilner/local-ci 0.18.1 → 0.19.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +8 -3
- package/dist/check-lock.d.ts +11 -4
- package/dist/check-lock.js +18 -8
- package/dist/cli.js +21 -7
- package/dist/lock.d.ts +21 -4
- package/dist/lock.js +30 -16
- package/dist/memory.d.ts +33 -0
- package/dist/memory.js +84 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -28,6 +28,7 @@ That writes `pre-push` and `commit-msg` shims, each one line that calls local-ci
|
|
|
28
28
|
- `local-ci status [<sha>]`: posts `check` for a commit (default `HEAD`) that already passed the full check, or runs the full check first; a fast pass alone never counts, so it always runs the full set before posting `success`. When the check command fails, it posts `check` = `failure` (`<command> failed on <hostname>`) and exits non-zero; a later passing run posts `success` over it. A dirty tree, a commit that is not `HEAD`, or a `gh` error posts nothing. Use it after a push that skipped hooks (`--no-verify`) or a lost background poster.
|
|
29
29
|
- `local-ci run -- <command>`: runs `<command>` under the machine-wide check lock (see [State](#state)), for a full check started by hand or from a script: `local-ci run -- pnpm check`. It waits its turn with every other check, and they wait for it; like any check, it runs beside a background job rather than waiting for one (see [State](#state)). The arguments after `--` go to the program as they are, with no shell (`local-ci run -- sh -c 'a && b'` for a shell line). Stdin is `/dev/null` and the command has no controlling terminal, so anything that prompts on `/dev/tty` fails; output goes straight through. It exits with the command's status, 128 plus the signal number if a signal killed the command, or 1 if the command can't start. It works outside a git repo, needs no `gh` login, posts no status and records no pass. Inside a check that already holds the lock it runs at once, so a repo's `check` script can wrap itself in `local-ci run` without deadlocking its own pre-push.
|
|
30
30
|
- `local-ci run --fast`: runs the repo's [fast set](#fast-check-set), or its full check command when it declares none or `LOCAL_CI_FULL=1` is set, through `sh -c` in the repo root, on the working tree as it is (uncommitted files included), under the check lock like `run --`. It posts no status and records no pass, and exits with the command's status. Use it to check a branch with the default branch merged in, before merging, without a push.
|
|
31
|
+
- `local-ci run --peak-mb <n> -- <command>` (or `run --peak-mb <n> --fast`): either run, waiting for `<n>` MB of free memory instead of the default 3072 before it starts (see [the memory gate](#the-memory-gate)). `<n>` is a whole number; anything else exits 1.
|
|
31
32
|
- `local-ci release`: publishes the repo's packages from this machine. Off unless `package.json` has `"local-ci": { "release": true }`; otherwise it prints `release not enabled` and exits 0. It needs a clean working tree, then fetches `origin` and takes a lock, so a second run exits with `release in progress`. In a temporary worktree at `origin`'s default branch:
|
|
32
33
|
1. With pending changesets, it creates `release/<short-sha>`, runs the root `version-packages` script if `package.json` has one (for steps such as a size-history entry that belong in the version PR) or `pnpm changeset version` otherwise, commits `chore: version packages`, pushes with `LOCAL_CI_FULL=1` (the hook runs the full check and posts `check` = `success`, which the merge waits for; the push adds `-o ServerAliveInterval=15` to the ssh command git would run (an OpenSSH one), so a long check can't drop it, see [SSH keepalive](#ssh-keepalive)), opens the PR, posts `pr-title`, waits for both statuses, and squash-merges it.
|
|
33
34
|
2. At the merge commit (or the default branch, with no changesets pending), it runs `local-ci status`, then `pnpm release` with the npm token, then pushes the tags `changeset publish` created.
|
|
@@ -92,7 +93,7 @@ Logs, the record of passed commits and the release lock live in `~/.local/state/
|
|
|
92
93
|
|
|
93
94
|
A pass is recorded by commit and by tree: `<repo>.passed` gets a `<sha>\t<command>` line and a `tree:<tree>\t<command as JSON>` line (`git rev-parse <sha>^{tree}`; JSON keeps a command over several lines on one line). A commit that has not passed but whose tree passed under the same check command (a squash or merge commit with the PR head's files, or a reworded commit) counts as passed without running the check: `pre-push` and `status` skip it, record a pass for its sha, and print and log `<short-sha>: <command> already passed on tree <short-tree>; reused without running`. A different tree or command runs the check. Unlike a commit record, a tree record counts only for a clean `HEAD`, so the command it matches is the one that commit's `package.json` names. It assumes the check reads only the commit's files: a check that reads the commit message or history (a commitlint of `HEAD`, a diff from a merge base) can pass on one commit and fail on another with the same tree, and its result is reused anyway. Records written before tree lines existed still match their sha, but no tree. A fast pass is recorded apart, as `fast:<sha>\t<command as JSON>` with no tree line, so it never counts as a full pass for `pre-push`, `status`, `post` or `renovate`, and versions before fast sets skip it.
|
|
94
95
|
|
|
95
|
-
One check command runs at a time on the machine, across every repo, so overlapping checks can't starve each other's tests: `pre-push`, `status`, the `status` runs `renovate` starts, and `local-ci run` each take `check.lock` in the same directory around the command they run (not the status posts). Nothing else takes it: a bare `pnpm check` or `pnpm test` runs beside every other check, so start a full check by hand with `local-ci run -- pnpm check`. The lock
|
|
96
|
+
One check command runs at a time on the machine, across every repo, so overlapping checks can't starve each other's tests: `pre-push`, `status`, the `status` runs `renovate` starts, and `local-ci run` each take `check.lock` in the same directory around the command they run (not the status posts). Nothing else takes it: a bare `pnpm check` or `pnpm test` runs beside every other check, so start a full check by hand with `local-ci run -- pnpm check`. The lock also waits for memory before a run starts (see [below](#the-memory-gate)). A run that finds it held prints `local-ci: waiting for the check lock, held by pid <pid> (<repo> <short-sha>)` (`<repo> <command>` for `local-ci run`, with the command cut to 60 characters), again whenever another run takes it first, and polls every second until the holder exits or dies; Ctrl-C stops the wait. Runs wait in arrival order, checks included, except that a check never waits behind a background job (see [below](#taking-the-check-lock-from-your-own-scripts)). After the wait it skips a commit that passed meanwhile and re-applies the clean-tree rule. The lock goes when the check passes, fails or is killed by a signal. Ctrl-C, SIGTERM, SIGHUP or SIGQUIT (Ctrl-\\) to local-ci passes the signal to the check's whole process group (so a check script's children get it too), waits for the check to exit, then releases the lock and dies of the signal, so a check never outlives its lock. A lock whose holder died (SIGKILL, a crash) or that predates the last boot is taken over by the next run. A local-ci check nested inside a check (one that pushes, say) sees its ancestor holds the lock and runs without waiting. A check that finds a background job's lock (see below) runs beside the job instead of waiting and leaves that lock alone, printing `local-ci: running beside background job pid <pid> (<holder>), without the check lock`; it holds `check.lock.beside` meanwhile, so another check waits for it rather than overlapping, and signals still stop it and release that lock as above. Each check run appends its duration and outcome to the log when it ends, e.g. `[<end time>] abc1234: pnpm check passed in 71.2s` (or `failed in`, or `did not finish after` when the check is killed by a signal or can't start); a commit that already passed is skipped and logs nothing (one that reuses a pass on its tree logs that instead, see above). Ctrl-C stops the check, which logs `did not finish after`.
|
|
96
97
|
|
|
97
98
|
### Taking the check lock from your own scripts
|
|
98
99
|
|
|
@@ -106,7 +107,11 @@ await withCheckLock("my-repo browser", async () => {
|
|
|
106
107
|
});
|
|
107
108
|
```
|
|
108
109
|
|
|
109
|
-
`withCheckLock(holder, fn, options?)` waits while another holder has the lock, logging `local-ci: waiting for the check lock, held by pid <pid> (<holder>)` to stderr as the CLI does, runs `fn`, and releases the lock when `fn` returns or throws. `holder` is the label other waiters see. `lockCheck(holder, options?)` does the take and returns the release function, for callers that can't wrap their work in a function. `options.file` overrides the lock path
|
|
110
|
+
`withCheckLock(holder, fn, options?)` waits while another holder has the lock, logging `local-ci: waiting for the check lock, held by pid <pid> (<holder>)` to stderr as the CLI does, runs `fn`, and releases the lock when `fn` returns or throws. `holder` is the label other waiters see. `lockCheck(holder, options?)` does the take and returns the release function, for callers that can't wrap their work in a function. `options.file` overrides the lock path, `options.onWait` replaces the wait message, and `options.peakMb` sets the free memory, in MB, the job waits for (default 3072; see [the memory gate](#the-memory-gate)). The path is `check.lock` in the state directory above, so no repo config is needed. A call nested under an ancestor process that holds the lock runs without waiting and leaves the lock alone. Pass `options.background: true` for a long job run after a push, such as a browser suite: it still waits for a running check, but `local-ci` checks (and other non-background callers) run beside it instead of waiting, so a push never sits in its hook behind its run. Every caller queues in arrival order: each waiter writes a ticket file to `check.lock.queue/` with its pid, arrival time and head start, plus a fourth line `check` for a check (not `background`), and takes the lock only when no live ticket sorts before its own. A check counts only other checks' tickets, so it never waits behind a background job, queued or running; a background job counts every ticket, so it waits for checks that arrived before it. `options.headStartMs` (milliseconds, default 0) moves a background job up by that much: it sorts by arrival minus head start, so it passes jobs that arrived less than that long before it, and no others. A ticket whose process died, that predates the last boot, or that can't be parsed is removed by the next waiter that finds it, so a killed waiter can't wedge the queue. A job held back only by a ticket, with the lock free, says so: `local-ci: waiting for the check lock, held by pid <pid> (queued ahead)`. Checks waiting to run beside a background job queue the same way, in `check.lock.beside.queue/`, keeping their arrival time. Versions before the queue ignore tickets, and versions before check tickets let checks take the lock without one, so a waiter from such a version (another repo pinned to an older local-ci) can still take the lock out of turn, without waiting on memory. A check run beside a background job takes `check.lock.beside` instead, so checks still run one at a time: a second check started during the job waits for the first, and whoever takes `check.lock` after the background job ends (another background job or a check) waits for a check still running beside it. A background job waiting that way holds `check.lock` as a check would, without the `background` line, until the beside check ends, so checks arriving meanwhile wait in turn rather than run beside a job that has not started. A check nested in a beside check runs under its lock without waiting. A beside lock whose holder died is taken over, as `check.lock` is. Versions before the beside lock ignore it, so a check from such a version (another repo pinned to an older local-ci) can still run beside a newer beside check. The lock file marks such a lock with a third line, `background`; a file without it (as older versions wrote) is waited on as before. SIGINT, SIGTERM or SIGHUP releases the lock and ends the process, so a job that needs its own signal cleanup should run it before calling the release itself. The package has an `exports` map, so deep imports other than `./lock` and `./vitest` no longer resolve.
|
|
111
|
+
|
|
112
|
+
### The memory gate
|
|
113
|
+
|
|
114
|
+
A run first in line, with the lock free, starts only while macOS reports normal memory pressure (`sysctl -n kern.memorystatus_vm_pressure_level` is 1) and free memory (`kern.memorystatus_level` percent of `hw.memsize`) covers its usual peak: 3072 MB (3 GB) unless the caller passes `peakMb` or `local-ci run --peak-mb <n>`. Until then it keeps its place in line, prints `local-ci: waiting for memory (<free> GB free, <peak> GB expected, pressure <level>)` (again after any wait on a holder), and reads memory again every 2 seconds; the runs behind it wait too. A check that runs beside a background job doesn't wait on memory, so a push never sits in its hook behind a background job's run. It never waits on memory forever: after 10 minutes waiting on memory alone (time spent waiting on a holder doesn't count), it prints `local-ci: starting without enough free memory after 10 min (<free> GB free, <peak> GB expected)` and starts, as every run did before the gate; a later retry in the same take doesn't wait on memory again or print the line again. Where memory can't be read (off macOS, a `sysctl` error), it prints `local-ci: can't read free memory (<reason>); starting without waiting for it` once per process and runs as if there were no gate, trying the read again on the next take. A run nested under an ancestor that holds the lock never reads memory, as it never waits for the lock. `LOCAL_CI_MEMORY_GATE=0` turns the gate off, for a machine or test run that should never wait on memory. Ctrl-C stops the wait like any other.
|
|
110
115
|
|
|
111
116
|
### Browser test runs under the check lock
|
|
112
117
|
|
|
@@ -119,6 +124,6 @@ import { defineConfig } from "vitest/config";
|
|
|
119
124
|
export default withBrowserLock(defineConfig({ test: { projects: [/* unit, storybook, ... */] } }));
|
|
120
125
|
```
|
|
121
126
|
|
|
122
|
-
Pass it the config `defineConfig` returns, as shown: its type rejects a `test` object literal with keys other than `globalSetup` (such as `projects`), so `withBrowserLock({ test: { projects } })` and `defineConfig(withBrowserLock({...}))` fail to typecheck, and it takes no function config (`defineConfig(() => ...)`). `withBrowserLock(config)` returns the config's own type and appends the setup to the root `test.globalSetup`, which vitest runs once per run in its main process, before any worker or browser starts; `browserLockSetup` is the setup file's path, for configs that list `globalSetup` themselves. The setup looks at the projects the run selected (after `--project` filters): if any has `browser.enabled`, it takes the lock, waiting as long as it takes and printing `local-ci: waiting for the check lock, held by pid <pid> (<holder>)`, and its teardown releases it when the run ends, pass or fail. A run with only node or jsdom projects never takes it. The holder is `<name> vitest browser`, `<name>` from the nearest `package.json`. A run nested under a holder (`local-ci run -- pnpm test`, a `pnpm check` that local-ci runs) goes ahead without waiting. Projects that each list the setup share one lock, even through separate copies of the module, so a run takes it once. Ctrl-C, SIGTERM or SIGHUP releases it and ends the run; a run killed outright is taken over as a dead holder. Watch mode takes no lock, so a watcher left open never blocks the machine's checks; agents and scripts, with no TTY, get run mode. It needs vitest 3 or later, whose setup sees the run's projects.
|
|
127
|
+
Pass it the config `defineConfig` returns, as shown: its type rejects a `test` object literal with keys other than `globalSetup` (such as `projects`), so `withBrowserLock({ test: { projects } })` and `defineConfig(withBrowserLock({...}))` fail to typecheck, and it takes no function config (`defineConfig(() => ...)`). `withBrowserLock(config)` returns the config's own type and appends the setup to the root `test.globalSetup`, which vitest runs once per run in its main process, before any worker or browser starts; `browserLockSetup` is the setup file's path, for configs that list `globalSetup` themselves. The setup looks at the projects the run selected (after `--project` filters): if any has `browser.enabled`, it takes the lock (memory gate included), waiting as long as it takes and printing `local-ci: waiting for the check lock, held by pid <pid> (<holder>)`, and its teardown releases it when the run ends, pass or fail. A run with only node or jsdom projects never takes it. The holder is `<name> vitest browser`, `<name>` from the nearest `package.json`. A run nested under a holder (`local-ci run -- pnpm test`, a `pnpm check` that local-ci runs) goes ahead without waiting. Projects that each list the setup share one lock, even through separate copies of the module, so a run takes it once. Ctrl-C, SIGTERM or SIGHUP releases it and ends the run; a run killed outright is taken over as a dead holder. Watch mode takes no lock, so a watcher left open never blocks the machine's checks; agents and scripts, with no TTY, get run mode. It needs vitest 3 or later, whose setup sees the run's projects.
|
|
123
128
|
|
|
124
129
|
Part of [jonmilner/config](https://github.com/jonmilner/config#readme).
|
package/dist/check-lock.d.ts
CHANGED
|
@@ -21,6 +21,11 @@ export interface LockCheckOptions {
|
|
|
21
21
|
* arrived when queueing behind other background jobs. Default 0. Ignored for a check.
|
|
22
22
|
*/
|
|
23
23
|
headStartMs?: number;
|
|
24
|
+
/**
|
|
25
|
+
* The free memory, in MB, this run usually needs at its peak; default 3072 (3 GB). It starts only
|
|
26
|
+
* at normal memory pressure with this much free, or after waiting 10 minutes on memory alone.
|
|
27
|
+
*/
|
|
28
|
+
peakMb?: number;
|
|
24
29
|
}
|
|
25
30
|
/** The machine-wide check lock's path, in `$XDG_STATE_HOME/local-ci` (default `~/.local/state`). */
|
|
26
31
|
export declare function checkLockPath(env?: NodeJS.ProcessEnv): string;
|
|
@@ -30,10 +35,12 @@ export declare function checkLockPath(env?: NodeJS.ProcessEnv): string;
|
|
|
30
35
|
* that holds the lock runs without waiting, and its release leaves the lock alone. A check (not
|
|
31
36
|
* `background`) that finds a background job holding it runs beside the job under a second lock,
|
|
32
37
|
* `<file>.beside`, instead: a second such check waits for it, and whoever takes the check lock
|
|
33
|
-
* next waits for it to clear.
|
|
34
|
-
* `headStartMs
|
|
35
|
-
*
|
|
38
|
+
* next waits for it to clear. Callers take the lock in arrival order, a background job's less any
|
|
39
|
+
* `headStartMs`, except that a check never waits behind a background job's ticket. Once first in
|
|
40
|
+
* line with the lock free, a caller starts only at normal memory pressure with `peakMb` free, or
|
|
41
|
+
* after waiting 10 minutes on memory alone; `LOCAL_CI_MEMORY_GATE=0` turns that wait off. SIGINT,
|
|
42
|
+
* SIGTERM, SIGHUP and SIGQUIT release the lock and end the process.
|
|
36
43
|
*/
|
|
37
|
-
export declare function lockCheck(holder: string, { file, onWait, background, headStartMs }?: LockCheckOptions): Promise<() => void>;
|
|
44
|
+
export declare function lockCheck(holder: string, { file, onWait, background, headStartMs, peakMb, }?: LockCheckOptions): Promise<() => void>;
|
|
38
45
|
/** Runs `fn` under the machine-wide check lock, so it queues behind `local-ci` checks and vice versa. */
|
|
39
46
|
export declare function withCheckLock<T>(holder: string, fn: () => Promise<T>, options?: LockCheckOptions): Promise<T>;
|
package/dist/check-lock.js
CHANGED
|
@@ -2,6 +2,7 @@ import { mkdirSync } from "node:fs";
|
|
|
2
2
|
import { join } from "node:path";
|
|
3
3
|
import { stateDir } from "./config.js";
|
|
4
4
|
import { LockHeld, lockHolder, markLock, takeLock, waitForLock } from "./lock.js";
|
|
5
|
+
import { DEFAULT_PEAK_MB, MEMORY_GATE_ENV, memoryGate } from "./memory.js";
|
|
5
6
|
import { processTable } from "./processes.js";
|
|
6
7
|
import { ancestorsFrom } from "./push.js";
|
|
7
8
|
/** This process's ancestors' pids, or just its parent when `ps` can't list them. */
|
|
@@ -75,11 +76,17 @@ function besideLockPath(file) {
|
|
|
75
76
|
* that holds the lock runs without waiting, and its release leaves the lock alone. A check (not
|
|
76
77
|
* `background`) that finds a background job holding it runs beside the job under a second lock,
|
|
77
78
|
* `<file>.beside`, instead: a second such check waits for it, and whoever takes the check lock
|
|
78
|
-
* next waits for it to clear.
|
|
79
|
-
* `headStartMs
|
|
80
|
-
*
|
|
79
|
+
* next waits for it to clear. Callers take the lock in arrival order, a background job's less any
|
|
80
|
+
* `headStartMs`, except that a check never waits behind a background job's ticket. Once first in
|
|
81
|
+
* line with the lock free, a caller starts only at normal memory pressure with `peakMb` free, or
|
|
82
|
+
* after waiting 10 minutes on memory alone; `LOCAL_CI_MEMORY_GATE=0` turns that wait off. SIGINT,
|
|
83
|
+
* SIGTERM, SIGHUP and SIGQUIT release the lock and end the process.
|
|
81
84
|
*/
|
|
82
|
-
export async function lockCheck(holder, { file = checkLockPath(), onWait, background = false, headStartMs = 0 } = {}) {
|
|
85
|
+
export async function lockCheck(holder, { file = checkLockPath(), onWait, background = false, headStartMs = 0, peakMb = DEFAULT_PEAK_MB, } = {}) {
|
|
86
|
+
// One gate for the whole take, so its 10 minutes count every wait on memory across retries.
|
|
87
|
+
const memory = memoryGate({ peakMb });
|
|
88
|
+
const gate = process.env[MEMORY_GATE_ENV] === "0" ? undefined : memory;
|
|
89
|
+
const arrival = Date.now();
|
|
83
90
|
const sidecar = besideLockPath(file);
|
|
84
91
|
let mine;
|
|
85
92
|
// A check nested in a check (one that pushes, say) runs under its ancestor's lock rather than
|
|
@@ -96,13 +103,15 @@ export async function lockCheck(holder, { file = checkLockPath(), onWait, backgr
|
|
|
96
103
|
onWait: onWait ??
|
|
97
104
|
(({ pid, holder }) => process.stderr.write(`local-ci: waiting for the check lock, held by pid ${pid} (${holder || "unknown"})\n`)),
|
|
98
105
|
};
|
|
106
|
+
// One arrival too, so a check that waits on more than one lock keeps its place in each line.
|
|
107
|
+
const checkQueue = { headStartMs: 0, check: true, arrival };
|
|
99
108
|
for (;;) {
|
|
100
109
|
let beside = false;
|
|
101
110
|
const skip = (held) => nested(held) || (beside = besides(held));
|
|
102
|
-
const queue = background ? { headStartMs } :
|
|
111
|
+
const queue = background ? { headStartMs, arrival } : checkQueue;
|
|
103
112
|
// Taken as a check's lock even for a background job, so checks wait for it rather than run
|
|
104
113
|
// beside it until it starts its run.
|
|
105
|
-
const unlock = await waitForLock(file, { ...options, skip, queue });
|
|
114
|
+
const unlock = await waitForLock(file, { ...options, skip, queue, gate });
|
|
106
115
|
if (unlock) {
|
|
107
116
|
const release = releaseOnSignal(unlock);
|
|
108
117
|
try {
|
|
@@ -119,8 +128,9 @@ export async function lockCheck(holder, { file = checkLockPath(), onWait, backgr
|
|
|
119
128
|
}
|
|
120
129
|
if (!beside)
|
|
121
130
|
return () => { };
|
|
122
|
-
// One check at a time beside a background job.
|
|
123
|
-
|
|
131
|
+
// One check at a time beside a background job, in arrival order. It doesn't wait on memory,
|
|
132
|
+
// so a push never sits in its hook behind a background job's run.
|
|
133
|
+
const unside = await waitForLock(sidecar, { ...options, skip: nested, queue: checkQueue });
|
|
124
134
|
if (!unside)
|
|
125
135
|
return () => { };
|
|
126
136
|
// The background job may have ended, or another taken its place, while this waited.
|
package/dist/cli.js
CHANGED
|
@@ -32,6 +32,8 @@ const USAGE = `Usage:
|
|
|
32
32
|
run a command under the machine-wide check lock
|
|
33
33
|
local-ci run --fast run the fast check (the full one without a fast set) on the working
|
|
34
34
|
tree under the check lock, posting nothing
|
|
35
|
+
local-ci run --peak-mb <n> ...
|
|
36
|
+
either run, starting once <n> MB is free (default 3072)
|
|
35
37
|
local-ci release version pending changesets through a PR, publish to npm and push tags
|
|
36
38
|
local-ci release --check say whether release has anything to do, changing nothing
|
|
37
39
|
local-ci renovate check each open Renovate PR in your repos and post check and pr-title
|
|
@@ -167,36 +169,48 @@ async function runLocked({ root, passed, log: logFile }, sha, tree, fast) {
|
|
|
167
169
|
/**
|
|
168
170
|
* Runs `-- <command>` under the check lock, so a full check started by hand waits its turn with
|
|
169
171
|
* the checks `pre-push` runs. Exits with the command's status, or 128 plus the signal that ended it.
|
|
172
|
+
* A leading `--peak-mb <n>` sets the free memory, in MB, the run waits for.
|
|
170
173
|
*/
|
|
171
|
-
async function runUnderLock(
|
|
174
|
+
async function runUnderLock(all) {
|
|
175
|
+
const peaked = all[0] === "--peak-mb";
|
|
176
|
+
const args = peaked ? all.slice(2) : all;
|
|
177
|
+
let peakMb;
|
|
178
|
+
if (peaked) {
|
|
179
|
+
const value = all[1] ?? "";
|
|
180
|
+
if (!/^\d+$/.test(value))
|
|
181
|
+
throw new Error(`run: --peak-mb expects a whole number of MB, got "${value}"\n${USAGE}`);
|
|
182
|
+
peakMb = Number(value);
|
|
183
|
+
}
|
|
172
184
|
if (args[0] === "--fast")
|
|
173
|
-
return runFast(args.slice(1));
|
|
185
|
+
return runFast(args.slice(1), peakMb);
|
|
174
186
|
const [separator, file, ...rest] = args;
|
|
175
187
|
if (separator !== "--" || !file)
|
|
176
188
|
throw new Error(`run: expected -- <command>\n${USAGE}`);
|
|
177
189
|
// Outside a git repo there is no repo name, and nothing else here needs one.
|
|
178
190
|
const name = (await repo().catch(() => undefined))?.name ?? basename(process.cwd());
|
|
179
|
-
await runLabelled(name, [file, ...rest].join(" "), file, rest);
|
|
191
|
+
await runLabelled(name, [file, ...rest].join(" "), file, rest, { peakMb });
|
|
180
192
|
}
|
|
181
193
|
/**
|
|
182
194
|
* `run --fast`: the repo's fast command, else its full one, on the working tree as it is, under
|
|
183
195
|
* the check lock. It posts no status and records no pass, since the tree need not be a commit.
|
|
184
196
|
*/
|
|
185
|
-
async function runFast(extra) {
|
|
197
|
+
async function runFast(extra, peakMb) {
|
|
186
198
|
if (extra.length)
|
|
187
199
|
throw new Error(`run --fast: unexpected ${extra.join(" ")}\n${USAGE}`);
|
|
188
200
|
const { root, name } = await repo();
|
|
189
201
|
const full = process.env[FULL_CHECK] === "1";
|
|
190
202
|
const command = (full ? undefined : resolveFastCommand(root)) ?? resolveCommand(root);
|
|
191
203
|
err(`local-ci: running ${command} on the working tree`);
|
|
192
|
-
await runLabelled(name, command, "sh", ["-c", command], root);
|
|
204
|
+
await runLabelled(name, command, "sh", ["-c", command], { cwd: root, peakMb });
|
|
193
205
|
}
|
|
194
206
|
/** Runs `file` under the check lock, labelled `<name> <command>`, and exits with its status. */
|
|
195
|
-
async function runLabelled(name, command, file, args, cwd) {
|
|
207
|
+
async function runLabelled(name, command, file, args, { cwd, peakMb } = {}) {
|
|
196
208
|
// Other sessions print this while they wait, so a long script mustn't fill their output.
|
|
197
209
|
const label = command.length > 60 ? `${command.slice(0, 57)}...` : command;
|
|
198
210
|
// After a signal the release dies of it, so a command told to stop never reports a status.
|
|
199
|
-
const result = await withCheckLock(`${name} ${label}`, () => runHeld(file, args, cwd)
|
|
211
|
+
const result = await withCheckLock(`${name} ${label}`, () => runHeld(file, args, cwd), {
|
|
212
|
+
peakMb,
|
|
213
|
+
});
|
|
200
214
|
if (result.error)
|
|
201
215
|
throw new Error(`could not run ${file}: ${result.error}`);
|
|
202
216
|
process.exitCode = result.status ?? 128 + constants.signals[result.signal];
|
package/dist/lock.d.ts
CHANGED
|
@@ -33,6 +33,22 @@ export declare function markLock(file: string, holder?: string, options?: TakeLo
|
|
|
33
33
|
* for a background job's lock. A file with fewer lines is a foreground lock, as older ones are.
|
|
34
34
|
*/
|
|
35
35
|
export declare function takeLock(file: string, what?: string, holder?: string, options?: TakeLockOptions): () => void;
|
|
36
|
+
/** How a waiter queues: its head start, whether it is a check, and its arrival (default now). */
|
|
37
|
+
export interface Queue {
|
|
38
|
+
headStartMs: number;
|
|
39
|
+
check?: boolean;
|
|
40
|
+
arrival?: number;
|
|
41
|
+
}
|
|
42
|
+
/** A condition, besides a free lock and a first place in line, that a waiter needs before it starts. */
|
|
43
|
+
export interface StartGate {
|
|
44
|
+
/** How long a waiter sleeps after the gate keeps it shut, before it asks again. */
|
|
45
|
+
intervalMs: number;
|
|
46
|
+
/**
|
|
47
|
+
* Whether to take the lock now, asked on each poll that finds the lock free and the waiter first
|
|
48
|
+
* in line. `fresh` is true on the first such poll after waiting on something else, or none.
|
|
49
|
+
*/
|
|
50
|
+
open(fresh: boolean): boolean;
|
|
51
|
+
}
|
|
36
52
|
export interface WaitForLockOptions extends TakeLockOptions {
|
|
37
53
|
what: string;
|
|
38
54
|
holder?: string;
|
|
@@ -45,11 +61,12 @@ export interface WaitForLockOptions extends TakeLockOptions {
|
|
|
45
61
|
onWait?: (held: LockHeld) => void;
|
|
46
62
|
/**
|
|
47
63
|
* Waits in line for the lock: this caller takes it only when no live waiter's arrival minus head
|
|
48
|
-
* start sorts before its own. Waiters without this option don't queue and aren't held back.
|
|
64
|
+
* start sorts before its own. Waiters without this option don't queue and aren't held back. A
|
|
65
|
+
* `check` waiter is held back only by other checks' tickets.
|
|
49
66
|
*/
|
|
50
|
-
queue?:
|
|
51
|
-
|
|
52
|
-
|
|
67
|
+
queue?: Queue;
|
|
68
|
+
/** Asked before taking a free lock; while it stays shut the waiter keeps its place and polls. */
|
|
69
|
+
gate?: StartGate;
|
|
53
70
|
intervalMs?: number;
|
|
54
71
|
sleep?: (ms: number) => Promise<void>;
|
|
55
72
|
}
|
package/dist/lock.js
CHANGED
|
@@ -133,6 +133,8 @@ export function takeLock(file, what = "release", holder, options = {}) {
|
|
|
133
133
|
rmSync(mine, { force: true });
|
|
134
134
|
}
|
|
135
135
|
}
|
|
136
|
+
/** The fourth line of a check's ticket; a ticket without it (as older versions wrote) is a background job's. */
|
|
137
|
+
const CHECK = "check";
|
|
136
138
|
const queueDir = (file) => `${file}.queue`;
|
|
137
139
|
/** Earliest arrival minus head start first; arrival, then name, settle a tie the same for everyone. */
|
|
138
140
|
function ticketOrder(a, b) {
|
|
@@ -140,8 +142,8 @@ function ticketOrder(a, b) {
|
|
|
140
142
|
a.arrival - b.arrival ||
|
|
141
143
|
(a.name < b.name ? -1 : 1));
|
|
142
144
|
}
|
|
143
|
-
/** Joins the queue for `file`, arriving now. */
|
|
144
|
-
function takeTicket(file, headStartMs) {
|
|
145
|
+
/** Joins the queue for `file`, arriving at `arrival` (default now). */
|
|
146
|
+
function takeTicket(file, { headStartMs, check = false, arrival = Date.now() }) {
|
|
145
147
|
if (!Number.isFinite(headStartMs) || headStartMs < 0) {
|
|
146
148
|
throw new RangeError(`headStartMs must be a finite number of 0 or more, got ${headStartMs}`);
|
|
147
149
|
}
|
|
@@ -150,12 +152,14 @@ function takeTicket(file, headStartMs) {
|
|
|
150
152
|
const ticket = {
|
|
151
153
|
name: `${process.pid}-${randomUUID()}`,
|
|
152
154
|
pid: process.pid,
|
|
153
|
-
arrival
|
|
155
|
+
arrival,
|
|
154
156
|
headStartMs,
|
|
157
|
+
check,
|
|
155
158
|
};
|
|
156
159
|
// Renaming publishes it whole; the temporary name is beside the queue so nobody lists it.
|
|
157
160
|
const unpublished = `${dir}.${ticket.name}`;
|
|
158
|
-
|
|
161
|
+
const fields = [ticket.pid, ticket.arrival, ticket.headStartMs, ...(check ? [CHECK] : [])];
|
|
162
|
+
writeFileSync(unpublished, fields.join("\n"));
|
|
159
163
|
renameSync(unpublished, join(dir, ticket.name));
|
|
160
164
|
return ticket;
|
|
161
165
|
}
|
|
@@ -168,12 +172,16 @@ function readTicket(path, name) {
|
|
|
168
172
|
catch {
|
|
169
173
|
return undefined;
|
|
170
174
|
}
|
|
171
|
-
const
|
|
172
|
-
|
|
175
|
+
const lines = content.split("\n");
|
|
176
|
+
const [pid = NaN, arrival = NaN, headStartMs = NaN] = lines
|
|
177
|
+
.slice(0, 3)
|
|
173
178
|
.map((field) => (field === "" ? NaN : Number(field)));
|
|
174
|
-
return { name, pid, arrival, headStartMs };
|
|
179
|
+
return { name, pid, arrival, headStartMs, check: lines[3] === CHECK };
|
|
175
180
|
}
|
|
176
|
-
/**
|
|
181
|
+
/**
|
|
182
|
+
* The live ticket that goes next before `mine`, if any, removing dead and pre-boot ones it finds. A
|
|
183
|
+
* check's ticket goes only behind other checks', so a check never waits behind a background job.
|
|
184
|
+
*/
|
|
177
185
|
function ticketAhead(file, mine) {
|
|
178
186
|
const dir = queueDir(file);
|
|
179
187
|
let names;
|
|
@@ -199,28 +207,34 @@ function ticketAhead(file, mine) {
|
|
|
199
207
|
!Number.isFinite(ticket.headStartMs)) {
|
|
200
208
|
rmSync(path, { force: true });
|
|
201
209
|
}
|
|
202
|
-
else if (ticketOrder(ticket, ahead ?? mine) < 0) {
|
|
210
|
+
else if ((ticket.check || !mine.check) && ticketOrder(ticket, ahead ?? mine) < 0) {
|
|
203
211
|
ahead = ticket;
|
|
204
212
|
}
|
|
205
213
|
}
|
|
206
214
|
return ahead;
|
|
207
215
|
}
|
|
216
|
+
/** The `last` wait a gate leaves, which no holder's `<pid>\n<holder>` can equal. */
|
|
217
|
+
const GATED = "\0gate";
|
|
208
218
|
/**
|
|
209
219
|
* Takes the lock, polling while a live process holds it, and returns its release, or `undefined`
|
|
210
220
|
* once it finds a holder `skip` accepts.
|
|
211
221
|
*/
|
|
212
222
|
export async function waitForLock(file, options) {
|
|
213
|
-
const { what, holder, skip, onWait, queue,
|
|
214
|
-
const
|
|
223
|
+
const { what, holder, skip, onWait, queue, gate } = options;
|
|
224
|
+
const { intervalMs = 1_000, sleep = defaultSleep } = options;
|
|
225
|
+
const ticket = queue && takeTicket(file, queue);
|
|
215
226
|
let last;
|
|
216
227
|
try {
|
|
217
228
|
for (;;) {
|
|
218
229
|
const ahead = ticket && ticketAhead(file, ticket);
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
230
|
+
// A gate is asked only about a free lock, so look before taking it.
|
|
231
|
+
let held = ahead || gate ? lockHolder(file, what) : undefined;
|
|
232
|
+
if (!ahead && !held) {
|
|
233
|
+
if (gate && !gate.open(last !== GATED)) {
|
|
234
|
+
last = GATED;
|
|
235
|
+
await sleep(gate.intervalMs);
|
|
236
|
+
continue;
|
|
237
|
+
}
|
|
224
238
|
try {
|
|
225
239
|
return takeLock(file, what, holder, { background: options.background });
|
|
226
240
|
}
|
package/dist/memory.d.ts
ADDED
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
import type { StartGate } from "./lock.ts";
|
|
2
|
+
/** What the gate reads: macOS's memory pressure level (1 normal, 2 warn, 4 critical) and free MB. */
|
|
3
|
+
export interface MemoryReading {
|
|
4
|
+
pressure: number;
|
|
5
|
+
freeMb: number;
|
|
6
|
+
}
|
|
7
|
+
/** The peak a run is expected to need when its caller names none, in MB. */
|
|
8
|
+
export declare const DEFAULT_PEAK_MB = 3072;
|
|
9
|
+
/** How long a run waits on memory alone before it starts anyway. */
|
|
10
|
+
export declare const MEMORY_TIMEOUT_MS = 600000;
|
|
11
|
+
/** How often a run waiting on memory reads it again. */
|
|
12
|
+
export declare const MEMORY_POLL_MS = 2000;
|
|
13
|
+
/** Turns the gate off when set to `0`, for machines or test runs that should never wait on memory. */
|
|
14
|
+
export declare const MEMORY_GATE_ENV = "LOCAL_CI_MEMORY_GATE";
|
|
15
|
+
/**
|
|
16
|
+
* Reads memory pressure, and free memory as `kern.memorystatus_level` percent of `hw.memsize`, the
|
|
17
|
+
* same reads claude-inbox's memory guard uses. Throws where `sysctl` or one of its keys is missing.
|
|
18
|
+
*/
|
|
19
|
+
export declare function readMemory(sysctl?: (keys: string[]) => string): MemoryReading;
|
|
20
|
+
export interface MemoryGateOptions {
|
|
21
|
+
/** The free memory the run needs, in MB; default 3072. */
|
|
22
|
+
peakMb?: number;
|
|
23
|
+
read?: () => MemoryReading;
|
|
24
|
+
timeoutMs?: number;
|
|
25
|
+
intervalMs?: number;
|
|
26
|
+
log?: (line: string) => void;
|
|
27
|
+
}
|
|
28
|
+
/**
|
|
29
|
+
* A gate that opens at pressure 1 with at least `peakMb` free, printing a wait line while shut,
|
|
30
|
+
* and opens anyway once it has waited `timeoutMs` (counted in polls) or when memory can't be read.
|
|
31
|
+
* A failed read is retried on the next ask, so one slow `sysctl` doesn't turn the gate off.
|
|
32
|
+
*/
|
|
33
|
+
export declare function memoryGate(options?: MemoryGateOptions): StartGate;
|
package/dist/memory.js
ADDED
|
@@ -0,0 +1,84 @@
|
|
|
1
|
+
// The memory gate on the check lock: a run starts only while macOS reports normal memory pressure
|
|
2
|
+
// and enough free memory for its usual peak, so heavy checks don't start into a memory shortage.
|
|
3
|
+
import { execFileSync } from "node:child_process";
|
|
4
|
+
/** The peak a run is expected to need when its caller names none, in MB. */
|
|
5
|
+
export const DEFAULT_PEAK_MB = 3_072;
|
|
6
|
+
/** How long a run waits on memory alone before it starts anyway. */
|
|
7
|
+
export const MEMORY_TIMEOUT_MS = 600_000;
|
|
8
|
+
/** How often a run waiting on memory reads it again. */
|
|
9
|
+
export const MEMORY_POLL_MS = 2_000;
|
|
10
|
+
/** Turns the gate off when set to `0`, for machines or test runs that should never wait on memory. */
|
|
11
|
+
export const MEMORY_GATE_ENV = "LOCAL_CI_MEMORY_GATE";
|
|
12
|
+
const KEYS = ["kern.memorystatus_vm_pressure_level", "kern.memorystatus_level", "hw.memsize"];
|
|
13
|
+
/**
|
|
14
|
+
* Reads memory pressure, and free memory as `kern.memorystatus_level` percent of `hw.memsize`, the
|
|
15
|
+
* same reads claude-inbox's memory guard uses. Throws where `sysctl` or one of its keys is missing.
|
|
16
|
+
*/
|
|
17
|
+
export function readMemory(sysctl = (keys) =>
|
|
18
|
+
// sysctl's own errors would print beside the warning; its exit status says enough.
|
|
19
|
+
execFileSync("sysctl", ["-n", ...keys], {
|
|
20
|
+
encoding: "utf8",
|
|
21
|
+
stdio: ["ignore", "pipe", "ignore"],
|
|
22
|
+
timeout: 5_000,
|
|
23
|
+
})) {
|
|
24
|
+
const values = sysctl(KEYS).trim().split("\n").map(Number);
|
|
25
|
+
const [pressure = NaN, level = NaN, bytes = NaN] = values;
|
|
26
|
+
if (values.length !== KEYS.length || ![pressure, level, bytes].every(Number.isFinite)) {
|
|
27
|
+
throw new Error(`unexpected sysctl output for ${KEYS.join(", ")}`);
|
|
28
|
+
}
|
|
29
|
+
return { pressure, freeMb: ((level / 100) * bytes) / 1024 ** 2 };
|
|
30
|
+
}
|
|
31
|
+
/** A reader whose failure has been reported, so the warning prints once per process. */
|
|
32
|
+
const reported = new WeakSet();
|
|
33
|
+
/** MB as GB, to at most one decimal. */
|
|
34
|
+
const gb = (mb) => Number((mb / 1024).toFixed(1));
|
|
35
|
+
/**
|
|
36
|
+
* A gate that opens at pressure 1 with at least `peakMb` free, printing a wait line while shut,
|
|
37
|
+
* and opens anyway once it has waited `timeoutMs` (counted in polls) or when memory can't be read.
|
|
38
|
+
* A failed read is retried on the next ask, so one slow `sysctl` doesn't turn the gate off.
|
|
39
|
+
*/
|
|
40
|
+
export function memoryGate(options = {}) {
|
|
41
|
+
const { peakMb = DEFAULT_PEAK_MB, read = readMemory, timeoutMs = MEMORY_TIMEOUT_MS } = options;
|
|
42
|
+
const { intervalMs = MEMORY_POLL_MS } = options;
|
|
43
|
+
const { log = (line) => process.stderr.write(`${line}\n`) } = options;
|
|
44
|
+
if (!Number.isFinite(peakMb) || peakMb < 0) {
|
|
45
|
+
throw new RangeError(`peakMb must be a finite number of 0 or more, got ${peakMb}`);
|
|
46
|
+
}
|
|
47
|
+
let waited = 0;
|
|
48
|
+
let timedOut = false;
|
|
49
|
+
return {
|
|
50
|
+
intervalMs,
|
|
51
|
+
open(fresh) {
|
|
52
|
+
// Past the timeout, a retry in the same take starts at once, quietly.
|
|
53
|
+
if (timedOut)
|
|
54
|
+
return true;
|
|
55
|
+
let reading;
|
|
56
|
+
try {
|
|
57
|
+
reading = read();
|
|
58
|
+
}
|
|
59
|
+
catch (error) {
|
|
60
|
+
if (!reported.has(read)) {
|
|
61
|
+
reported.add(read);
|
|
62
|
+
// execFileSync's message runs over several lines; the first names the failure.
|
|
63
|
+
const reason = (error instanceof Error ? error.message : String(error)).split("\n")[0];
|
|
64
|
+
log(`local-ci: can't read free memory (${reason}); starting without waiting for it`);
|
|
65
|
+
}
|
|
66
|
+
return true;
|
|
67
|
+
}
|
|
68
|
+
const { pressure, freeMb } = reading;
|
|
69
|
+
if (pressure === 1 && freeMb >= peakMb)
|
|
70
|
+
return true;
|
|
71
|
+
const amounts = `${gb(freeMb)} GB free, ${gb(peakMb)} GB expected`;
|
|
72
|
+
if (waited >= timeoutMs) {
|
|
73
|
+
timedOut = true;
|
|
74
|
+
const minutes = Number((timeoutMs / 60_000).toFixed(1));
|
|
75
|
+
log(`local-ci: starting without enough free memory after ${minutes} min (${amounts})`);
|
|
76
|
+
return true;
|
|
77
|
+
}
|
|
78
|
+
if (fresh)
|
|
79
|
+
log(`local-ci: waiting for memory (${amounts}, pressure ${pressure})`);
|
|
80
|
+
waited += intervalMs;
|
|
81
|
+
return false;
|
|
82
|
+
},
|
|
83
|
+
};
|
|
84
|
+
}
|