@tobi-or-not/quorum 0.4.0 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +1 -1
- package/dist/bench/fingerprint.d.ts +7 -5
- package/dist/bench/fingerprint.js +22 -9
- package/dist/bench/records.d.ts +70 -0
- package/dist/bench/records.js +30 -0
- package/dist/bench/runner.d.ts +18 -0
- package/dist/bench/runner.js +29 -3
- package/dist/cli/bench.d.ts +11 -4
- package/dist/cli/bench.js +34 -17
- package/dist/cli/bin.js +2 -2
- package/dist/cli/main.d.ts +34 -14
- package/dist/cli/main.js +99 -20
- package/dist/cli/print-config.d.ts +8 -6
- package/dist/cli/print-config.js +21 -8
- package/dist/cli/render.d.ts +2 -2
- package/dist/cli/render.js +21 -2
- package/dist/config/env.d.ts +32 -19
- package/dist/config/env.js +95 -32
- package/dist/config/from-config.js +15 -3
- package/dist/config/resolve.d.ts +9 -1
- package/dist/config/resolve.js +17 -1
- package/dist/config/schema.d.ts +141 -5
- package/dist/config/schema.js +125 -12
- package/dist/core/clock.d.ts +17 -0
- package/dist/core/clock.js +14 -0
- package/dist/core/coverage.d.ts +49 -0
- package/dist/core/coverage.js +144 -0
- package/dist/core/diff.d.ts +24 -0
- package/dist/core/diff.js +63 -18
- package/dist/core/errors.d.ts +12 -3
- package/dist/core/errors.js +21 -2
- package/dist/core/exclude.d.ts +42 -0
- package/dist/core/exclude.js +189 -0
- package/dist/core/limits.d.ts +82 -0
- package/dist/core/limits.js +163 -0
- package/dist/core/output.d.ts +3 -2
- package/dist/core/output.js +3 -2
- package/dist/core/ports.d.ts +57 -2
- package/dist/core/ports.js +36 -1
- package/dist/core/quorum.d.ts +24 -8
- package/dist/core/quorum.js +187 -56
- package/dist/core/retry.d.ts +36 -0
- package/dist/core/retry.js +46 -0
- package/dist/core/stages/context.d.ts +118 -23
- package/dist/core/stages/context.js +304 -46
- package/dist/core/stages/panel.d.ts +5 -4
- package/dist/core/stages/panel.js +30 -11
- package/dist/core/stages/research.d.ts +15 -6
- package/dist/core/stages/research.js +75 -18
- package/dist/core/tokens.d.ts +31 -0
- package/dist/core/tokens.js +81 -0
- package/dist/core/types.d.ts +77 -10
- package/dist/index.d.ts +1 -1
- package/dist/providers/anthropic.d.ts +2 -1
- package/dist/providers/anthropic.js +8 -3
- package/dist/providers/claude-code.d.ts +51 -11
- package/dist/providers/claude-code.js +60 -25
- package/dist/providers/gemini.d.ts +4 -1
- package/dist/providers/gemini.js +63 -11
- package/dist/providers/http.d.ts +49 -4
- package/dist/providers/http.js +155 -14
- package/dist/providers/ollama.d.ts +3 -1
- package/dist/providers/ollama.js +11 -5
- package/dist/providers/openai-compatible.d.ts +3 -1
- package/dist/providers/openai-compatible.js +11 -5
- package/dist/providers/options.d.ts +12 -3
- package/dist/providers/options.js +30 -10
- package/dist/providers/size.d.ts +10 -14
- package/dist/providers/size.js +15 -69
- package/package.json +1 -1
- package/schema/quorum.schema.json +34 -1
package/README.md
CHANGED
|
@@ -58,7 +58,7 @@ quorum ./my.diff > result.json
|
|
|
58
58
|
|
|
59
59
|
Requires Claude Code 2.1.311 or newer. [`claudeCode`](./docs/20-using/30-providers.md#claudecode) has the command line, the isolation and the auth modes, and [Claude Code](./docs/20-using/20-configuration.md#claude-code) the setup, in CI too.
|
|
60
60
|
|
|
61
|
-
The JSON result goes to stdout, and the exit code says what to do with it: `0` for a complete review in which no gating profile kept a `high` finding, `1` when one did, complete or not, `2` for a usage or configuration error, `3` when
|
|
61
|
+
The JSON result goes to stdout, and the exit code says what to do with it: `0` for a complete review in which no gating profile kept a `high` finding, `1` when one did, complete or not, `2` for a usage or configuration error, `3` when a gating profile's research call failed, `4` when a gating profile's review is incomplete otherwise, and `130` when the run was interrupted. [Quick start](./docs/10-getting-started/20-quickstart.md) walks one review through, and [Start here](./docs/01-readme.md) lays out four tracks: review a diff, use it from code or review something else, know what it protects against, and measure configurations.
|
|
62
62
|
|
|
63
63
|
## Documentation
|
|
64
64
|
|
|
@@ -6,11 +6,13 @@
|
|
|
6
6
|
* The form holds every setting a review uses but the key: the profiles that run, in run order, each with
|
|
7
7
|
* its id, prompts, lenses, policy and gating, and its research provider and each lens's provider written
|
|
8
8
|
* out in place of their names; every field of a provider defined as data but its key, the values of
|
|
9
|
-
* `headers`, `query` and `extraBody` and the whole base URL included
|
|
10
|
-
*
|
|
11
|
-
*
|
|
12
|
-
*
|
|
13
|
-
*
|
|
9
|
+
* `headers`, `query` and `extraBody` and the whole base URL included, `timeoutMs` and `retries` among them;
|
|
10
|
+
* a provider object as its config key, its model and the `timeoutMs` and `retries` it states, so one that
|
|
11
|
+
* states none keeps the form it had, and never the `worstCase` it states, which is code like its `complete`;
|
|
12
|
+
* `concurrency`, `maxFindings`, `maxReviewTokens` and `reviewTimeoutMs` as the review applies them, defaults
|
|
13
|
+
* filled in, and `exclude` when the config lists a pattern. It holds no config-file path, no warning and no
|
|
14
|
+
* provider name, so the same settings under other names are one configuration, and a rotated key keeps a
|
|
15
|
+
* run. A `claude-code` provider's resolved command path is one of its settings, so it is in the form.
|
|
14
16
|
*
|
|
15
17
|
* The form can hold a secret (a credential kept in a header, a query parameter or the base path), so it
|
|
16
18
|
* never leaves this module: not written, not logged, not put in an error message. Only the HMAC does.
|
|
@@ -6,11 +6,13 @@
|
|
|
6
6
|
* The form holds every setting a review uses but the key: the profiles that run, in run order, each with
|
|
7
7
|
* its id, prompts, lenses, policy and gating, and its research provider and each lens's provider written
|
|
8
8
|
* out in place of their names; every field of a provider defined as data but its key, the values of
|
|
9
|
-
* `headers`, `query` and `extraBody` and the whole base URL included
|
|
10
|
-
*
|
|
11
|
-
*
|
|
12
|
-
*
|
|
13
|
-
*
|
|
9
|
+
* `headers`, `query` and `extraBody` and the whole base URL included, `timeoutMs` and `retries` among them;
|
|
10
|
+
* a provider object as its config key, its model and the `timeoutMs` and `retries` it states, so one that
|
|
11
|
+
* states none keeps the form it had, and never the `worstCase` it states, which is code like its `complete`;
|
|
12
|
+
* `concurrency`, `maxFindings`, `maxReviewTokens` and `reviewTimeoutMs` as the review applies them, defaults
|
|
13
|
+
* filled in, and `exclude` when the config lists a pattern. It holds no config-file path, no warning and no
|
|
14
|
+
* provider name, so the same settings under other names are one configuration, and a rotated key keeps a
|
|
15
|
+
* run. A `claude-code` provider's resolved command path is one of its settings, so it is in the form.
|
|
14
16
|
*
|
|
15
17
|
* The form can hold a secret (a credential kept in a header, a query parameter or the base path), so it
|
|
16
18
|
* never leaves this module: not written, not logged, not put in an error message. Only the HMAC does.
|
|
@@ -20,18 +22,24 @@
|
|
|
20
22
|
*/
|
|
21
23
|
import { createHmac, randomBytes } from 'node:crypto';
|
|
22
24
|
import { isPlainObject } from "../config/load.js";
|
|
23
|
-
import { isProvider } from "../config/schema.js";
|
|
25
|
+
import { isProvider, statedFacts } from "../config/schema.js";
|
|
24
26
|
import { DEFAULT_CONCURRENCY } from "../core/concurrency.js";
|
|
27
|
+
import { DEFAULT_MAX_REVIEW_TOKENS, DEFAULT_REVIEW_TIMEOUT_MS } from "../core/limits.js";
|
|
25
28
|
import { DEFAULT_MAX_FINDINGS } from "../core/stages/research.js";
|
|
26
29
|
const SALT = /^[0-9a-f]{64}$/;
|
|
27
30
|
/** A fresh salt: 32 random bytes as 64 lowercase hex digits. */
|
|
28
31
|
export function newSalt() {
|
|
29
32
|
return randomBytes(32).toString('hex');
|
|
30
33
|
}
|
|
31
|
-
/** A provider as the form holds it: a provider object by its config key and
|
|
34
|
+
/** A provider as the form holds it: a provider object by its config key, its model and the facts it states, a spec by its fields less the key. */
|
|
32
35
|
function providerForm(provider) {
|
|
33
|
-
if (isProvider(provider))
|
|
34
|
-
return {
|
|
36
|
+
if (isProvider(provider)) {
|
|
37
|
+
return {
|
|
38
|
+
object: provider.id,
|
|
39
|
+
model: provider.model,
|
|
40
|
+
...statedFacts(provider).stated,
|
|
41
|
+
};
|
|
42
|
+
}
|
|
35
43
|
return Object.fromEntries(Object.entries(provider).filter(([field]) => field !== 'apiKey'));
|
|
36
44
|
}
|
|
37
45
|
/** The form of the provider a profile binds by `name`; a name the config holds no provider under is a RangeError, never a silent gap. */
|
|
@@ -64,6 +72,11 @@ export function canonicalSettings(config) {
|
|
|
64
72
|
return JSON.stringify({
|
|
65
73
|
concurrency: config.concurrency ?? DEFAULT_CONCURRENCY,
|
|
66
74
|
maxFindings: config.maxFindings ?? DEFAULT_MAX_FINDINGS,
|
|
75
|
+
maxReviewTokens: config.maxReviewTokens ?? DEFAULT_MAX_REVIEW_TOKENS,
|
|
76
|
+
reviewTimeoutMs: config.reviewTimeoutMs ?? DEFAULT_REVIEW_TIMEOUT_MS,
|
|
77
|
+
...config.exclude === undefined
|
|
78
|
+
? {}
|
|
79
|
+
: { exclude: config.exclude },
|
|
67
80
|
profiles,
|
|
68
81
|
}, sortKeys);
|
|
69
82
|
}
|
package/dist/bench/records.d.ts
CHANGED
|
@@ -106,6 +106,25 @@ export declare const resultSchema: z.ZodObject<{
|
|
|
106
106
|
reasoning: z.ZodString;
|
|
107
107
|
}, z.core.$strict>>;
|
|
108
108
|
error: z.ZodNullable<z.ZodString>;
|
|
109
|
+
failure: z.ZodNullable<z.ZodObject<{
|
|
110
|
+
kind: z.ZodEnum<{
|
|
111
|
+
http: "http";
|
|
112
|
+
network: "network";
|
|
113
|
+
process: "process";
|
|
114
|
+
timeout: "timeout";
|
|
115
|
+
aborted: "aborted";
|
|
116
|
+
refusal: "refusal";
|
|
117
|
+
truncated: "truncated";
|
|
118
|
+
"invalid-response": "invalid-response";
|
|
119
|
+
unsupported: "unsupported";
|
|
120
|
+
"too-large": "too-large";
|
|
121
|
+
"token-cap": "token-cap";
|
|
122
|
+
"review-deadline": "review-deadline";
|
|
123
|
+
"unusable-reply": "unusable-reply";
|
|
124
|
+
unknown: "unknown";
|
|
125
|
+
}>;
|
|
126
|
+
status: z.ZodOptional<z.ZodInt>;
|
|
127
|
+
}, z.core.$strict>>;
|
|
109
128
|
}, z.core.$strict>>;
|
|
110
129
|
}, z.core.$strict>>;
|
|
111
130
|
discarded: z.ZodArray<z.ZodObject<{
|
|
@@ -131,6 +150,25 @@ export declare const resultSchema: z.ZodObject<{
|
|
|
131
150
|
reasoning: z.ZodString;
|
|
132
151
|
}, z.core.$strict>>;
|
|
133
152
|
error: z.ZodNullable<z.ZodString>;
|
|
153
|
+
failure: z.ZodNullable<z.ZodObject<{
|
|
154
|
+
kind: z.ZodEnum<{
|
|
155
|
+
http: "http";
|
|
156
|
+
network: "network";
|
|
157
|
+
process: "process";
|
|
158
|
+
timeout: "timeout";
|
|
159
|
+
aborted: "aborted";
|
|
160
|
+
refusal: "refusal";
|
|
161
|
+
truncated: "truncated";
|
|
162
|
+
"invalid-response": "invalid-response";
|
|
163
|
+
unsupported: "unsupported";
|
|
164
|
+
"too-large": "too-large";
|
|
165
|
+
"token-cap": "token-cap";
|
|
166
|
+
"review-deadline": "review-deadline";
|
|
167
|
+
"unusable-reply": "unusable-reply";
|
|
168
|
+
unknown: "unknown";
|
|
169
|
+
}>;
|
|
170
|
+
status: z.ZodOptional<z.ZodInt>;
|
|
171
|
+
}, z.core.$strict>>;
|
|
134
172
|
}, z.core.$strict>>;
|
|
135
173
|
}, z.core.$strict>>;
|
|
136
174
|
unreviewed: z.ZodArray<z.ZodObject<{
|
|
@@ -156,6 +194,25 @@ export declare const resultSchema: z.ZodObject<{
|
|
|
156
194
|
reasoning: z.ZodString;
|
|
157
195
|
}, z.core.$strict>>;
|
|
158
196
|
error: z.ZodNullable<z.ZodString>;
|
|
197
|
+
failure: z.ZodNullable<z.ZodObject<{
|
|
198
|
+
kind: z.ZodEnum<{
|
|
199
|
+
http: "http";
|
|
200
|
+
network: "network";
|
|
201
|
+
process: "process";
|
|
202
|
+
timeout: "timeout";
|
|
203
|
+
aborted: "aborted";
|
|
204
|
+
refusal: "refusal";
|
|
205
|
+
truncated: "truncated";
|
|
206
|
+
"invalid-response": "invalid-response";
|
|
207
|
+
unsupported: "unsupported";
|
|
208
|
+
"too-large": "too-large";
|
|
209
|
+
"token-cap": "token-cap";
|
|
210
|
+
"review-deadline": "review-deadline";
|
|
211
|
+
"unusable-reply": "unusable-reply";
|
|
212
|
+
unknown: "unknown";
|
|
213
|
+
}>;
|
|
214
|
+
status: z.ZodOptional<z.ZodInt>;
|
|
215
|
+
}, z.core.$strict>>;
|
|
159
216
|
}, z.core.$strict>>;
|
|
160
217
|
}, z.core.$strict>>;
|
|
161
218
|
meta: z.ZodObject<{
|
|
@@ -189,15 +246,28 @@ export declare const resultSchema: z.ZodObject<{
|
|
|
189
246
|
incomplete: z.ZodArray<z.ZodObject<{
|
|
190
247
|
profile: z.ZodString;
|
|
191
248
|
reason: z.ZodEnum<{
|
|
249
|
+
"token-cap": "token-cap";
|
|
250
|
+
"review-deadline": "review-deadline";
|
|
192
251
|
"research-unusable": "research-unusable";
|
|
252
|
+
"research-failed": "research-failed";
|
|
193
253
|
"findings-capped": "findings-capped";
|
|
194
254
|
"finding-unreadable": "finding-unreadable";
|
|
195
255
|
"vote-failed": "vote-failed";
|
|
196
256
|
"request-too-large": "request-too-large";
|
|
257
|
+
binary: "binary";
|
|
258
|
+
submodule: "submodule";
|
|
197
259
|
}>;
|
|
198
260
|
file: z.ZodOptional<z.ZodString>;
|
|
199
261
|
detail: z.ZodString;
|
|
200
262
|
}, z.core.$strict>>;
|
|
263
|
+
leftOut: z.ZodArray<z.ZodObject<{
|
|
264
|
+
file: z.ZodString;
|
|
265
|
+
reason: z.ZodEnum<{
|
|
266
|
+
binary: "binary";
|
|
267
|
+
submodule: "submodule";
|
|
268
|
+
excluded: "excluded";
|
|
269
|
+
}>;
|
|
270
|
+
}, z.core.$strict>>;
|
|
201
271
|
flags: z.ZodArray<z.ZodObject<{
|
|
202
272
|
code: z.ZodEnum<{
|
|
203
273
|
"hidden-characters": "hidden-characters";
|
package/dist/bench/records.js
CHANGED
|
@@ -41,10 +41,30 @@ const vote = z.strictObject({
|
|
|
41
41
|
keep: z.boolean(),
|
|
42
42
|
reasoning: z.string(),
|
|
43
43
|
});
|
|
44
|
+
const voteFailure = z.strictObject({
|
|
45
|
+
kind: z.enum([
|
|
46
|
+
'http',
|
|
47
|
+
'network',
|
|
48
|
+
'process',
|
|
49
|
+
'timeout',
|
|
50
|
+
'aborted',
|
|
51
|
+
'refusal',
|
|
52
|
+
'truncated',
|
|
53
|
+
'invalid-response',
|
|
54
|
+
'unsupported',
|
|
55
|
+
'too-large',
|
|
56
|
+
'token-cap',
|
|
57
|
+
'review-deadline',
|
|
58
|
+
'unusable-reply',
|
|
59
|
+
'unknown',
|
|
60
|
+
]),
|
|
61
|
+
status: z.int().min(100).max(599).optional(),
|
|
62
|
+
});
|
|
44
63
|
const lensVote = z.strictObject({
|
|
45
64
|
angle: z.string(),
|
|
46
65
|
vote: vote.nullable(),
|
|
47
66
|
error: z.string().nullable(),
|
|
67
|
+
failure: voteFailure.nullable(),
|
|
48
68
|
});
|
|
49
69
|
const verdict = z.strictObject({
|
|
50
70
|
id: z.string(),
|
|
@@ -87,14 +107,23 @@ const incompleteEntry = z.strictObject({
|
|
|
87
107
|
profile: z.string(),
|
|
88
108
|
reason: z.enum([
|
|
89
109
|
'research-unusable',
|
|
110
|
+
'research-failed',
|
|
90
111
|
'findings-capped',
|
|
91
112
|
'finding-unreadable',
|
|
92
113
|
'vote-failed',
|
|
93
114
|
'request-too-large',
|
|
115
|
+
'token-cap',
|
|
116
|
+
'review-deadline',
|
|
117
|
+
'binary',
|
|
118
|
+
'submodule',
|
|
94
119
|
]),
|
|
95
120
|
file: z.string().optional(),
|
|
96
121
|
detail: z.string(),
|
|
97
122
|
});
|
|
123
|
+
const leftOutFile = z.strictObject({
|
|
124
|
+
file: z.string(),
|
|
125
|
+
reason: z.enum(['excluded', 'binary', 'submodule']),
|
|
126
|
+
});
|
|
98
127
|
const flag = z.strictObject({
|
|
99
128
|
code: z.enum(['hidden-characters', 'ai-addressed-text']),
|
|
100
129
|
file: z.string().optional(),
|
|
@@ -117,6 +146,7 @@ export const resultSchema = z.strictObject({
|
|
|
117
146
|
profiles: z.record(z.string(), profileMeta),
|
|
118
147
|
usage: usage.optional(),
|
|
119
148
|
incomplete: z.array(incompleteEntry),
|
|
149
|
+
leftOut: z.array(leftOutFile),
|
|
120
150
|
flags: z.array(flag),
|
|
121
151
|
}),
|
|
122
152
|
});
|
package/dist/bench/runner.d.ts
CHANGED
|
@@ -44,12 +44,21 @@ export interface RunReviewsOptions {
|
|
|
44
44
|
readonly events: (event: RunnerEvent) => void;
|
|
45
45
|
/** The most a line may hold, its line break not counted, in bytes; `MAX_RECORD_BYTES` when left out. */
|
|
46
46
|
readonly maxRecordBytes?: number;
|
|
47
|
+
/**
|
|
48
|
+
* Once the run's reviews have spent this many tokens, as each review's cap counts them, no review starts.
|
|
49
|
+
* No cap when left out.
|
|
50
|
+
*/
|
|
51
|
+
readonly maxRunTokens?: number;
|
|
47
52
|
}
|
|
48
53
|
export interface RunReviewsOutcome {
|
|
49
54
|
/** The reviews written to the record. */
|
|
50
55
|
readonly recorded: number;
|
|
51
56
|
/** The reviews that failed, a rejection that is no denial, or whose record was too large to write. */
|
|
52
57
|
readonly failed: number;
|
|
58
|
+
/** The reviews that did not start because the run had spent `maxRunTokens`. */
|
|
59
|
+
readonly notRun: number;
|
|
60
|
+
/** The tokens the run's `call` events counted against their reviews' caps, as `maxRunTokens` reads them. */
|
|
61
|
+
readonly spent: number;
|
|
53
62
|
/** Whether the caller's signal was aborted by the time the run ended. */
|
|
54
63
|
readonly interrupted: boolean;
|
|
55
64
|
}
|
|
@@ -68,6 +77,8 @@ export declare function planReviews(options: {
|
|
|
68
77
|
* other rejection (an HTTP, network, timeout or process failure, a reply the server shaped wrong) says
|
|
69
78
|
* nothing of the model's answer, so the review fails and runs again. The classes are matched with
|
|
70
79
|
* `instanceof`: an error from a second copy of quorum is no denial, so it fails, which fails closed.
|
|
80
|
+
* Only a gating profile's failed research rejects a review: an advisory profile's, the call or its
|
|
81
|
+
* reply, resolves it with a `research-failed` gap, recorded with its result, neither failed nor denied.
|
|
71
82
|
*/
|
|
72
83
|
export declare function denialOf(error: unknown): Denial | undefined;
|
|
73
84
|
/**
|
|
@@ -87,6 +98,13 @@ export declare function denialOf(error: unknown): Denial | undefined;
|
|
|
87
98
|
* nothing for that review, cuts the others, and the run rejects with the error of the write once every
|
|
88
99
|
* review has settled. A callback of `events` that throws does the same.
|
|
89
100
|
*
|
|
101
|
+
* The run counts what each review's cap counts: the `spent` of every `call` event of every review, one that
|
|
102
|
+
* failed or was denied included. Before a review starts, and after the abort's check, the run checks that
|
|
103
|
+
* the total is below `maxRunTokens`: once it is not, the review does not start and counts as `notRun`, the
|
|
104
|
+
* reviews in flight finish and are recorded, and the outcome says what was spent. The check is made before
|
|
105
|
+
* each review starts, so the reviews in flight can pass the cap by up to `jobs` reviews' own caps. The calls
|
|
106
|
+
* a failing review's own abort cut send no event, so the count misses them.
|
|
107
|
+
*
|
|
90
108
|
* @throws the error of the first `append` (or `events`) that threw
|
|
91
109
|
*/
|
|
92
110
|
export declare function runReviews(options: RunReviewsOptions): Promise<RunReviewsOutcome>;
|
package/dist/bench/runner.js
CHANGED
|
@@ -66,6 +66,8 @@ function tallyCalls() {
|
|
|
66
66
|
* other rejection (an HTTP, network, timeout or process failure, a reply the server shaped wrong) says
|
|
67
67
|
* nothing of the model's answer, so the review fails and runs again. The classes are matched with
|
|
68
68
|
* `instanceof`: an error from a second copy of quorum is no denial, so it fails, which fails closed.
|
|
69
|
+
* Only a gating profile's failed research rejects a review: an advisory profile's, the call or its
|
|
70
|
+
* reply, resolves it with a `research-failed` gap, recorded with its result, neither failed nor denied.
|
|
69
71
|
*/
|
|
70
72
|
export function denialOf(error) {
|
|
71
73
|
if (error instanceof OutputError)
|
|
@@ -91,10 +93,17 @@ export function denialOf(error) {
|
|
|
91
93
|
* nothing for that review, cuts the others, and the run rejects with the error of the write once every
|
|
92
94
|
* review has settled. A callback of `events` that throws does the same.
|
|
93
95
|
*
|
|
96
|
+
* The run counts what each review's cap counts: the `spent` of every `call` event of every review, one that
|
|
97
|
+
* failed or was denied included. Before a review starts, and after the abort's check, the run checks that
|
|
98
|
+
* the total is below `maxRunTokens`: once it is not, the review does not start and counts as `notRun`, the
|
|
99
|
+
* reviews in flight finish and are recorded, and the outcome says what was spent. The check is made before
|
|
100
|
+
* each review starts, so the reviews in flight can pass the cap by up to `jobs` reviews' own caps. The calls
|
|
101
|
+
* a failing review's own abort cut send no event, so the count misses them.
|
|
102
|
+
*
|
|
94
103
|
* @throws the error of the first `append` (or `events`) that threw
|
|
95
104
|
*/
|
|
96
105
|
export async function runReviews(options) {
|
|
97
|
-
const { suite, configurations, reviews, jobs, env, append, events, maxRecordBytes = MAX_RECORD_BYTES } = options;
|
|
106
|
+
const { suite, configurations, reviews, jobs, env, append, events, maxRecordBytes = MAX_RECORD_BYTES, maxRunTokens = Infinity } = options;
|
|
98
107
|
const own = new AbortController();
|
|
99
108
|
const signal = options.signal === undefined
|
|
100
109
|
? own.signal
|
|
@@ -105,6 +114,8 @@ export async function runReviews(options) {
|
|
|
105
114
|
]));
|
|
106
115
|
let recorded = 0;
|
|
107
116
|
let failed = 0;
|
|
117
|
+
let notRun = 0;
|
|
118
|
+
let spent = 0;
|
|
108
119
|
let stopped;
|
|
109
120
|
const stop = (error) => {
|
|
110
121
|
stopped ??= { error };
|
|
@@ -128,7 +139,12 @@ export async function runReviews(options) {
|
|
|
128
139
|
if (configuration === undefined || diff === undefined)
|
|
129
140
|
throw new Error(UNKNOWN_REVIEW);
|
|
130
141
|
const tally = tallyCalls();
|
|
131
|
-
const
|
|
142
|
+
const sink = (event) => {
|
|
143
|
+
if (event.type === 'call')
|
|
144
|
+
spent += event.spent;
|
|
145
|
+
tally.sink(event);
|
|
146
|
+
};
|
|
147
|
+
const quorum = await fromConfig(configuration.config, { env, events: sink });
|
|
132
148
|
const started = performance.now();
|
|
133
149
|
const facts = () => ({
|
|
134
150
|
fingerprint: configuration.fingerprint,
|
|
@@ -160,6 +176,10 @@ export async function runReviews(options) {
|
|
|
160
176
|
async function runOne(review) {
|
|
161
177
|
if (signal.aborted)
|
|
162
178
|
return;
|
|
179
|
+
if (spent >= maxRunTokens) {
|
|
180
|
+
notRun += 1;
|
|
181
|
+
return;
|
|
182
|
+
}
|
|
163
183
|
let record;
|
|
164
184
|
let line;
|
|
165
185
|
try {
|
|
@@ -191,5 +211,11 @@ export async function runReviews(options) {
|
|
|
191
211
|
await Promise.all(reviews.map(review => limit(() => runOne(review))));
|
|
192
212
|
if (stopped !== undefined)
|
|
193
213
|
throw stopped.error;
|
|
194
|
-
return {
|
|
214
|
+
return {
|
|
215
|
+
recorded,
|
|
216
|
+
failed,
|
|
217
|
+
notRun,
|
|
218
|
+
spent,
|
|
219
|
+
interrupted: options.signal?.aborted === true,
|
|
220
|
+
};
|
|
195
221
|
}
|
package/dist/cli/bench.d.ts
CHANGED
|
@@ -31,12 +31,19 @@ export declare const BENCH_USAGE: string;
|
|
|
31
31
|
* `results.jsonl` to append, so a cut last line stays, and warns of it. It refuses a run directory that
|
|
32
32
|
* holds no run, and `--retry-incomplete`, which runs reviews.
|
|
33
33
|
*
|
|
34
|
+
* `--max-run-tokens <n>` stops the run from starting reviews once its reviews have spent `n` tokens, counted
|
|
35
|
+
* from each call's `spent`, the tokens its review's own cap counted (`maxReviewTokens`): the reviews in flight
|
|
36
|
+
* finish and are recorded, the reports are written, one stderr line says how many reviews were not run, and
|
|
37
|
+
* the next run runs them. The check is made before each review starts, so the reviews in flight can pass
|
|
38
|
+
* the cap by up to `--jobs` reviews' own caps, and the calls a failing review's abort cut are not counted.
|
|
39
|
+
*
|
|
34
40
|
* Exit codes: 0 when every planned review is recorded, a denied one included, or `--report-only` wrote the
|
|
35
41
|
* reports, 2 for a usage, suite, configuration or run directory error, 3 when the run stops (a file cannot
|
|
36
|
-
* be written or read), 4 when a review failed (its research call failed, or its record
|
|
37
|
-
* write: the others are recorded, and the next run runs the failed ones)
|
|
38
|
-
* reports are written on 0, 4 and 130,
|
|
39
|
-
*
|
|
42
|
+
* be written or read), 4 when a review failed (its gating profile's research call failed, or its record
|
|
43
|
+
* was too large to write: the others are recorded, and the next run runs the failed ones) or
|
|
44
|
+
* `--max-run-tokens` left reviews unrun, 130 when interrupted. The reports are written on 0, 4 and 130,
|
|
45
|
+
* so a stopped run shows what it recorded; an interrupt during the checks and the plan leaves the run
|
|
46
|
+
* directory as it was, even one that does not exist yet.
|
|
40
47
|
*
|
|
41
48
|
* Every line it writes is inert (invariant I3).
|
|
42
49
|
*/
|
package/dist/cli/bench.js
CHANGED
|
@@ -26,22 +26,23 @@ export const BENCH_USAGE = [
|
|
|
26
26
|
'Review every case of a suite with every config, --trials times each, and report what each caught.',
|
|
27
27
|
'',
|
|
28
28
|
'Options:',
|
|
29
|
-
' -c, --config <path>
|
|
30
|
-
' -p, --profile <id>
|
|
31
|
-
' --trials <n>
|
|
32
|
-
' --case <id>
|
|
33
|
-
' -j, --jobs <n>
|
|
34
|
-
'
|
|
35
|
-
'
|
|
36
|
-
' --
|
|
37
|
-
' --
|
|
38
|
-
'
|
|
29
|
+
' -c, --config <path> A config file to compare; repeat it, and the first is the reference',
|
|
30
|
+
' -p, --profile <id> Run only this profile; repeat it to run several',
|
|
31
|
+
' --trials <n> Reviews of each case per config: the suite\'s trials, else 1',
|
|
32
|
+
' --case <id> Run only this case; repeat it to run several',
|
|
33
|
+
' -j, --jobs <n> The reviews run at once (default 1)',
|
|
34
|
+
' --max-run-tokens <n> Start no review once the run has spent <n> tokens; the next run runs the rest',
|
|
35
|
+
' -o, --out <dir> The run directory (default quorum-bench/<suite name>)',
|
|
36
|
+
' --retry-incomplete Run again the recorded reviews that came back incomplete',
|
|
37
|
+
' --report-only Write the reports of what is recorded, and run no review',
|
|
38
|
+
' --dry-run Check the suite and the configs, print the plan, and exit',
|
|
39
|
+
' -h, --help Print this help and exit',
|
|
39
40
|
'',
|
|
40
41
|
].join('\n');
|
|
41
42
|
/**
|
|
42
43
|
* The bench's own exit codes (Decision 18), not the review's `EXIT`: `failed` means reviews failed and were
|
|
43
|
-
* not recorded, not that a review left a gap or that a
|
|
44
|
-
* run could not go on.
|
|
44
|
+
* not recorded, or that `--max-run-tokens` left reviews unrun, not that a review left a gap or that a
|
|
45
|
+
* model's reply denied it, and `stopped` means the run could not go on.
|
|
45
46
|
*/
|
|
46
47
|
const BENCH_EXIT = {
|
|
47
48
|
ok: 0,
|
|
@@ -61,6 +62,7 @@ const OPTIONS = {
|
|
|
61
62
|
trials: { type: 'string' },
|
|
62
63
|
case: { type: 'string', multiple: true },
|
|
63
64
|
jobs: { type: 'string', short: 'j' },
|
|
65
|
+
'max-run-tokens': { type: 'string' },
|
|
64
66
|
out: { type: 'string', short: 'o' },
|
|
65
67
|
'retry-incomplete': { type: 'boolean' },
|
|
66
68
|
'report-only': { type: 'boolean' },
|
|
@@ -97,6 +99,7 @@ function checkOptions(values, positionals, cwd) {
|
|
|
97
99
|
throw new Error(`unexpected arguments: ${extra.join(' ')}`);
|
|
98
100
|
const trials = wholeNumber('--trials', values.trials, MAX_TRIALS);
|
|
99
101
|
const jobs = wholeNumber('--jobs', values.jobs, MAX_JOBS);
|
|
102
|
+
const maxRunTokens = wholeNumber('--max-run-tokens', values['max-run-tokens'], Number.MAX_SAFE_INTEGER);
|
|
100
103
|
const configs = values.config ?? [];
|
|
101
104
|
if (configs.includes(''))
|
|
102
105
|
throw usageIssue('--config', 'must name a file');
|
|
@@ -121,6 +124,7 @@ function checkOptions(values, positionals, cwd) {
|
|
|
121
124
|
trials,
|
|
122
125
|
cases,
|
|
123
126
|
jobs: jobs ?? 1,
|
|
127
|
+
maxRunTokens,
|
|
124
128
|
out: values.out,
|
|
125
129
|
retryIncomplete: values['retry-incomplete'] === true,
|
|
126
130
|
reportOnly: values['report-only'] === true,
|
|
@@ -355,7 +359,7 @@ async function writeRun({ out, state, runFile }) {
|
|
|
355
359
|
* writer is closed whatever happens; when the run rejects, with the error of a failed write, that error
|
|
356
360
|
* is the one thrown.
|
|
357
361
|
*/
|
|
358
|
-
async function execute(io, { checked, scores, records, toRun, jobs }) {
|
|
362
|
+
async function execute(io, { checked, scores, records, toRun, jobs, maxRunTokens }) {
|
|
359
363
|
const { suite, loaded, byId, out, runFile } = checked;
|
|
360
364
|
await writeRun(checked);
|
|
361
365
|
const writer = await openRecords(out, records);
|
|
@@ -367,6 +371,7 @@ async function execute(io, { checked, scores, records, toRun, jobs }) {
|
|
|
367
371
|
configurations: loaded.map(({ config }, index) => ({ config, fingerprint: runFile.configurations[index].fingerprint })),
|
|
368
372
|
reviews: toRun,
|
|
369
373
|
jobs,
|
|
374
|
+
maxRunTokens,
|
|
370
375
|
env: io.env,
|
|
371
376
|
signal: io.signal,
|
|
372
377
|
append: writer.append,
|
|
@@ -458,12 +463,17 @@ async function bench(io, options) {
|
|
|
458
463
|
records,
|
|
459
464
|
toRun,
|
|
460
465
|
jobs: options.jobs,
|
|
466
|
+
maxRunTokens: options.maxRunTokens,
|
|
461
467
|
});
|
|
462
468
|
await writeReports(io, checked, trials, scores);
|
|
463
469
|
if (outcome.interrupted) {
|
|
464
470
|
io.stderr.write('interrupted\n');
|
|
465
471
|
return BENCH_EXIT.interrupted;
|
|
466
472
|
}
|
|
473
|
+
if (outcome.notRun > 0 && options.maxRunTokens !== undefined) {
|
|
474
|
+
writeLines(io.stderr, [`--max-run-tokens: the run reached its cap of ${options.maxRunTokens} tokens (it spent ${outcome.spent}), so ${outcome.notRun} review(s) were not run; the next run runs them`]);
|
|
475
|
+
return BENCH_EXIT.failed;
|
|
476
|
+
}
|
|
467
477
|
return outcome.failed > 0
|
|
468
478
|
? BENCH_EXIT.failed
|
|
469
479
|
: BENCH_EXIT.ok;
|
|
@@ -499,12 +509,19 @@ function usageError(io, error) {
|
|
|
499
509
|
* `results.jsonl` to append, so a cut last line stays, and warns of it. It refuses a run directory that
|
|
500
510
|
* holds no run, and `--retry-incomplete`, which runs reviews.
|
|
501
511
|
*
|
|
512
|
+
* `--max-run-tokens <n>` stops the run from starting reviews once its reviews have spent `n` tokens, counted
|
|
513
|
+
* from each call's `spent`, the tokens its review's own cap counted (`maxReviewTokens`): the reviews in flight
|
|
514
|
+
* finish and are recorded, the reports are written, one stderr line says how many reviews were not run, and
|
|
515
|
+
* the next run runs them. The check is made before each review starts, so the reviews in flight can pass
|
|
516
|
+
* the cap by up to `--jobs` reviews' own caps, and the calls a failing review's abort cut are not counted.
|
|
517
|
+
*
|
|
502
518
|
* Exit codes: 0 when every planned review is recorded, a denied one included, or `--report-only` wrote the
|
|
503
519
|
* reports, 2 for a usage, suite, configuration or run directory error, 3 when the run stops (a file cannot
|
|
504
|
-
* be written or read), 4 when a review failed (its research call failed, or its record
|
|
505
|
-
* write: the others are recorded, and the next run runs the failed ones)
|
|
506
|
-
* reports are written on 0, 4 and 130,
|
|
507
|
-
*
|
|
520
|
+
* be written or read), 4 when a review failed (its gating profile's research call failed, or its record
|
|
521
|
+
* was too large to write: the others are recorded, and the next run runs the failed ones) or
|
|
522
|
+
* `--max-run-tokens` left reviews unrun, 130 when interrupted. The reports are written on 0, 4 and 130,
|
|
523
|
+
* so a stopped run shows what it recorded; an interrupt during the checks and the plan leaves the run
|
|
524
|
+
* directory as it was, even one that does not exist yet.
|
|
508
525
|
*
|
|
509
526
|
* Every line it writes is inert (invariant I3).
|
|
510
527
|
*/
|
package/dist/cli/bin.js
CHANGED
|
@@ -13,8 +13,8 @@ function fail(error) {
|
|
|
13
13
|
}
|
|
14
14
|
process.on('uncaughtException', fail);
|
|
15
15
|
process.on('unhandledRejection', fail);
|
|
16
|
-
// Exit 3 until the run settles. A
|
|
17
|
-
// empties the event loop: the runtime then exits with the code it holds, which must never be 0 with no result.
|
|
16
|
+
// Exit 3 until the run settles. A config module whose loading never finishes leaves nothing to wait
|
|
17
|
+
// for and empties the event loop: the runtime then exits with the code it holds, which must never be 0 with no result.
|
|
18
18
|
process.exitCode = EXIT.runtime;
|
|
19
19
|
let settled = false;
|
|
20
20
|
process.once('beforeExit', () => {
|
package/dist/cli/main.d.ts
CHANGED
|
@@ -5,10 +5,12 @@ export { errorMessage, EXIT };
|
|
|
5
5
|
export type { CliIO };
|
|
6
6
|
/**
|
|
7
7
|
* Run one review and return the process exit code. Never throws: usage and configuration problems
|
|
8
|
-
* exit 2, runtime failures exit 3, an interrupt exits
|
|
9
|
-
* failed votes left undecided, or a
|
|
10
|
-
* was kept by a gating profile, so CI can tell a broken run
|
|
11
|
-
* (exit 1) and from a clean one (exit 0). `
|
|
8
|
+
* exit 2, a gating profile's failed research and other runtime failures exit 3, an interrupt exits
|
|
9
|
+
* 130, and an incomplete review (a finding its failed votes left undecided, or a gap) of a gating
|
|
10
|
+
* profile exits 4 unless a `high` finding was kept by a gating profile, so CI can tell a broken run
|
|
11
|
+
* and a partial review from a real finding (exit 1) and from a clean one (exit 0). `--allow-incomplete`
|
|
12
|
+
* turns that exit 4 into 0 only when every failed vote behind it failed in passing (see `exitCodeFor`).
|
|
13
|
+
* `exitCodeFor` maps the result; advisory flags never change it.
|
|
12
14
|
*
|
|
13
15
|
* It reads at most 16 MiB of diff, from stdin or a file, counted in bytes: a larger diff is a usage error,
|
|
14
16
|
* and reading stops at the chunk that crosses the cap.
|
|
@@ -40,22 +42,40 @@ export interface SignalSource {
|
|
|
40
42
|
}
|
|
41
43
|
/**
|
|
42
44
|
* The signal of one review, aborted by the first SIGINT or SIGTERM. The next one calls `exit(130)`: the
|
|
43
|
-
* way out
|
|
45
|
+
* way out when what a provider still runs, after the review settled its call, keeps the process alive.
|
|
44
46
|
*/
|
|
45
47
|
export declare function interruptSignal(source: SignalSource, exit: (code: number) => void): AbortSignal;
|
|
46
48
|
/**
|
|
47
49
|
* The exit code of a finished review: 1 when a kept finding is `high` and its profile gates, 4 when
|
|
48
|
-
* an unreviewed finding or a gap belongs to a gating profile
|
|
49
|
-
*
|
|
50
|
-
*
|
|
51
|
-
* as
|
|
50
|
+
* an unreviewed finding or a gap belongs to a gating profile, 0 otherwise. `allowIncomplete` turns
|
|
51
|
+
* that 4 into 0 only when every gap of a gating profile is a `vote-failed` one (an unreviewed finding
|
|
52
|
+
* is no gap) and, for each gating profile that has a gap, every failed vote of its own kept and
|
|
53
|
+
* unreviewed findings failed in passing, as `failedInPassing` says: a dropped connection, a timeout,
|
|
54
|
+
* or an HTTP 408, 409, 429 or 5xx. A gating profile with no gap blocks no other profile's lift. A gap
|
|
55
|
+
* with no failed vote behind it lifts nothing: an unreviewed finding whose votes hold none, or a
|
|
56
|
+
* `vote-failed` gap beside kept findings that hold none, which no review writes.
|
|
57
|
+
* A profile gates unless `meta.profiles` holds it with `gating: false`, so a profile missing from the
|
|
58
|
+
* result counts as gating and a gap never hides.
|
|
52
59
|
*
|
|
53
|
-
*
|
|
54
|
-
* request
|
|
55
|
-
*
|
|
56
|
-
*
|
|
57
|
-
*
|
|
60
|
+
* Why so narrow: a pull request causes the other gaps directly (ADR-0012). It pads the diff until a
|
|
61
|
+
* request is refused as too large, floods the research with decoy findings past `maxFindings`, or
|
|
62
|
+
* plants text that a voter refuses or that cuts its reply off or leaves it unusable, and one
|
|
63
|
+
* `.gitattributes` line, one NUL byte or a submodule's bump leaves a file's change out of the diff.
|
|
64
|
+
* The flag lifts only what the code names, so a reason added later is out of its reach by default,
|
|
65
|
+
* and so is what a result built by hand or by an older quorum leaves unsaid: a failed vote with no
|
|
66
|
+
* `failure`, a verdict whose `votes` is no array, a reason no quorum writes. A gating profile's
|
|
67
|
+
* `research-failed` gap, which no review records, is out too: a review that read nothing is never
|
|
68
|
+
* turned into exit 0. Planted text can still draw out a reply that ends in a lifted failure, a
|
|
69
|
+
* timeout of its own, so on a gating profile do not use the flag for a pull request you do not trust.
|
|
58
70
|
*/
|
|
59
71
|
export declare function exitCodeFor(result: QuorumResult, options?: {
|
|
60
72
|
readonly allowIncomplete?: boolean;
|
|
61
73
|
}): number;
|
|
74
|
+
/**
|
|
75
|
+
* True for a vote's failure that is infrastructure's: its own data `kind` is `network` or `timeout`, or
|
|
76
|
+
* `http` with an own data `status` of 408, 409, 429 or an integer from 500 to 599. Anything else is false:
|
|
77
|
+
* a refusal, a reply cut off or unusable, a request too large, any other status, a status that is no
|
|
78
|
+
* number, a kind no provider has, `null` and a value that is no object. It reads own data only, never
|
|
79
|
+
* runs a getter and never throws.
|
|
80
|
+
*/
|
|
81
|
+
export declare function failedInPassing(failure: unknown): boolean;
|