@tobi-or-not/quorum 0.4.0 → 0.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (71) hide show
  1. package/README.md +1 -1
  2. package/dist/bench/fingerprint.d.ts +7 -5
  3. package/dist/bench/fingerprint.js +22 -9
  4. package/dist/bench/records.d.ts +70 -0
  5. package/dist/bench/records.js +30 -0
  6. package/dist/bench/runner.d.ts +18 -0
  7. package/dist/bench/runner.js +29 -3
  8. package/dist/cli/bench.d.ts +11 -4
  9. package/dist/cli/bench.js +34 -17
  10. package/dist/cli/bin.js +2 -2
  11. package/dist/cli/main.d.ts +34 -14
  12. package/dist/cli/main.js +99 -20
  13. package/dist/cli/print-config.d.ts +8 -6
  14. package/dist/cli/print-config.js +21 -8
  15. package/dist/cli/render.d.ts +2 -2
  16. package/dist/cli/render.js +21 -2
  17. package/dist/config/env.d.ts +32 -19
  18. package/dist/config/env.js +95 -32
  19. package/dist/config/from-config.js +15 -3
  20. package/dist/config/resolve.d.ts +9 -1
  21. package/dist/config/resolve.js +17 -1
  22. package/dist/config/schema.d.ts +141 -5
  23. package/dist/config/schema.js +125 -12
  24. package/dist/core/clock.d.ts +17 -0
  25. package/dist/core/clock.js +14 -0
  26. package/dist/core/coverage.d.ts +49 -0
  27. package/dist/core/coverage.js +144 -0
  28. package/dist/core/diff.d.ts +24 -0
  29. package/dist/core/diff.js +63 -18
  30. package/dist/core/errors.d.ts +12 -3
  31. package/dist/core/errors.js +21 -2
  32. package/dist/core/exclude.d.ts +42 -0
  33. package/dist/core/exclude.js +189 -0
  34. package/dist/core/limits.d.ts +82 -0
  35. package/dist/core/limits.js +163 -0
  36. package/dist/core/output.d.ts +3 -2
  37. package/dist/core/output.js +3 -2
  38. package/dist/core/ports.d.ts +57 -2
  39. package/dist/core/ports.js +36 -1
  40. package/dist/core/quorum.d.ts +24 -8
  41. package/dist/core/quorum.js +187 -56
  42. package/dist/core/retry.d.ts +36 -0
  43. package/dist/core/retry.js +46 -0
  44. package/dist/core/stages/context.d.ts +118 -23
  45. package/dist/core/stages/context.js +304 -46
  46. package/dist/core/stages/panel.d.ts +5 -4
  47. package/dist/core/stages/panel.js +30 -11
  48. package/dist/core/stages/research.d.ts +15 -6
  49. package/dist/core/stages/research.js +75 -18
  50. package/dist/core/tokens.d.ts +31 -0
  51. package/dist/core/tokens.js +81 -0
  52. package/dist/core/types.d.ts +77 -10
  53. package/dist/index.d.ts +1 -1
  54. package/dist/providers/anthropic.d.ts +2 -1
  55. package/dist/providers/anthropic.js +8 -3
  56. package/dist/providers/claude-code.d.ts +51 -11
  57. package/dist/providers/claude-code.js +60 -25
  58. package/dist/providers/gemini.d.ts +4 -1
  59. package/dist/providers/gemini.js +63 -11
  60. package/dist/providers/http.d.ts +49 -4
  61. package/dist/providers/http.js +155 -14
  62. package/dist/providers/ollama.d.ts +3 -1
  63. package/dist/providers/ollama.js +11 -5
  64. package/dist/providers/openai-compatible.d.ts +3 -1
  65. package/dist/providers/openai-compatible.js +11 -5
  66. package/dist/providers/options.d.ts +12 -3
  67. package/dist/providers/options.js +30 -10
  68. package/dist/providers/size.d.ts +10 -14
  69. package/dist/providers/size.js +15 -69
  70. package/package.json +1 -1
  71. package/schema/quorum.schema.json +34 -1
package/README.md CHANGED
@@ -58,7 +58,7 @@ quorum ./my.diff > result.json
58
58
 
59
59
  Requires Claude Code 2.1.311 or newer. [`claudeCode`](./docs/20-using/30-providers.md#claudecode) has the command line, the isolation and the auth modes, and [Claude Code](./docs/20-using/20-configuration.md#claude-code) the setup, in CI too.
60
60
 
61
- The JSON result goes to stdout, and the exit code says what to do with it: `0` for a complete review in which no gating profile kept a `high` finding, `1` when one did, complete or not, `2` for a usage or configuration error, `3` when the research call failed, `4` when a gating profile's review is incomplete otherwise, and `130` when the run was interrupted. [Quick start](./docs/10-getting-started/20-quickstart.md) walks one review through, and [Start here](./docs/01-readme.md) lays out four tracks: review a diff, use it from code or review something else, know what it protects against, and measure configurations.
61
+ The JSON result goes to stdout, and the exit code says what to do with it: `0` for a complete review in which no gating profile kept a `high` finding, `1` when one did, complete or not, `2` for a usage or configuration error, `3` when a gating profile's research call failed, `4` when a gating profile's review is incomplete otherwise, and `130` when the run was interrupted. [Quick start](./docs/10-getting-started/20-quickstart.md) walks one review through, and [Start here](./docs/01-readme.md) lays out four tracks: review a diff, use it from code or review something else, know what it protects against, and measure configurations.
62
62
 
63
63
  ## Documentation
64
64
 
@@ -6,11 +6,13 @@
6
6
  * The form holds every setting a review uses but the key: the profiles that run, in run order, each with
7
7
  * its id, prompts, lenses, policy and gating, and its research provider and each lens's provider written
8
8
  * out in place of their names; every field of a provider defined as data but its key, the values of
9
- * `headers`, `query` and `extraBody` and the whole base URL included; a provider object as its config key
10
- * and its model; `concurrency` and `maxFindings` as the review applies them, defaults filled in. It holds
11
- * no config-file path, no warning and no provider name, so the same settings under other names are one
12
- * configuration, and a rotated key keeps a run. A `claude-code` provider's resolved command path is one
13
- * of its settings, so it is in the form.
9
+ * `headers`, `query` and `extraBody` and the whole base URL included, `timeoutMs` and `retries` among them;
10
+ * a provider object as its config key, its model and the `timeoutMs` and `retries` it states, so one that
11
+ * states none keeps the form it had, and never the `worstCase` it states, which is code like its `complete`;
12
+ * `concurrency`, `maxFindings`, `maxReviewTokens` and `reviewTimeoutMs` as the review applies them, defaults
13
+ * filled in, and `exclude` when the config lists a pattern. It holds no config-file path, no warning and no
14
+ * provider name, so the same settings under other names are one configuration, and a rotated key keeps a
15
+ * run. A `claude-code` provider's resolved command path is one of its settings, so it is in the form.
14
16
  *
15
17
  * The form can hold a secret (a credential kept in a header, a query parameter or the base path), so it
16
18
  * never leaves this module: not written, not logged, not put in an error message. Only the HMAC does.
@@ -6,11 +6,13 @@
6
6
  * The form holds every setting a review uses but the key: the profiles that run, in run order, each with
7
7
  * its id, prompts, lenses, policy and gating, and its research provider and each lens's provider written
8
8
  * out in place of their names; every field of a provider defined as data but its key, the values of
9
- * `headers`, `query` and `extraBody` and the whole base URL included; a provider object as its config key
10
- * and its model; `concurrency` and `maxFindings` as the review applies them, defaults filled in. It holds
11
- * no config-file path, no warning and no provider name, so the same settings under other names are one
12
- * configuration, and a rotated key keeps a run. A `claude-code` provider's resolved command path is one
13
- * of its settings, so it is in the form.
9
+ * `headers`, `query` and `extraBody` and the whole base URL included, `timeoutMs` and `retries` among them;
10
+ * a provider object as its config key, its model and the `timeoutMs` and `retries` it states, so one that
11
+ * states none keeps the form it had, and never the `worstCase` it states, which is code like its `complete`;
12
+ * `concurrency`, `maxFindings`, `maxReviewTokens` and `reviewTimeoutMs` as the review applies them, defaults
13
+ * filled in, and `exclude` when the config lists a pattern. It holds no config-file path, no warning and no
14
+ * provider name, so the same settings under other names are one configuration, and a rotated key keeps a
15
+ * run. A `claude-code` provider's resolved command path is one of its settings, so it is in the form.
14
16
  *
15
17
  * The form can hold a secret (a credential kept in a header, a query parameter or the base path), so it
16
18
  * never leaves this module: not written, not logged, not put in an error message. Only the HMAC does.
@@ -20,18 +22,24 @@
20
22
  */
21
23
  import { createHmac, randomBytes } from 'node:crypto';
22
24
  import { isPlainObject } from "../config/load.js";
23
- import { isProvider } from "../config/schema.js";
25
+ import { isProvider, statedFacts } from "../config/schema.js";
24
26
  import { DEFAULT_CONCURRENCY } from "../core/concurrency.js";
27
+ import { DEFAULT_MAX_REVIEW_TOKENS, DEFAULT_REVIEW_TIMEOUT_MS } from "../core/limits.js";
25
28
  import { DEFAULT_MAX_FINDINGS } from "../core/stages/research.js";
26
29
  const SALT = /^[0-9a-f]{64}$/;
27
30
  /** A fresh salt: 32 random bytes as 64 lowercase hex digits. */
28
31
  export function newSalt() {
29
32
  return randomBytes(32).toString('hex');
30
33
  }
31
- /** A provider as the form holds it: a provider object by its config key and model, a spec by its fields less the key. */
34
+ /** A provider as the form holds it: a provider object by its config key, its model and the facts it states, a spec by its fields less the key. */
32
35
  function providerForm(provider) {
33
- if (isProvider(provider))
34
- return { object: provider.id, model: provider.model };
36
+ if (isProvider(provider)) {
37
+ return {
38
+ object: provider.id,
39
+ model: provider.model,
40
+ ...statedFacts(provider).stated,
41
+ };
42
+ }
35
43
  return Object.fromEntries(Object.entries(provider).filter(([field]) => field !== 'apiKey'));
36
44
  }
37
45
  /** The form of the provider a profile binds by `name`; a name the config holds no provider under is a RangeError, never a silent gap. */
@@ -64,6 +72,11 @@ export function canonicalSettings(config) {
64
72
  return JSON.stringify({
65
73
  concurrency: config.concurrency ?? DEFAULT_CONCURRENCY,
66
74
  maxFindings: config.maxFindings ?? DEFAULT_MAX_FINDINGS,
75
+ maxReviewTokens: config.maxReviewTokens ?? DEFAULT_MAX_REVIEW_TOKENS,
76
+ reviewTimeoutMs: config.reviewTimeoutMs ?? DEFAULT_REVIEW_TIMEOUT_MS,
77
+ ...config.exclude === undefined
78
+ ? {}
79
+ : { exclude: config.exclude },
67
80
  profiles,
68
81
  }, sortKeys);
69
82
  }
@@ -106,6 +106,25 @@ export declare const resultSchema: z.ZodObject<{
106
106
  reasoning: z.ZodString;
107
107
  }, z.core.$strict>>;
108
108
  error: z.ZodNullable<z.ZodString>;
109
+ failure: z.ZodNullable<z.ZodObject<{
110
+ kind: z.ZodEnum<{
111
+ http: "http";
112
+ network: "network";
113
+ process: "process";
114
+ timeout: "timeout";
115
+ aborted: "aborted";
116
+ refusal: "refusal";
117
+ truncated: "truncated";
118
+ "invalid-response": "invalid-response";
119
+ unsupported: "unsupported";
120
+ "too-large": "too-large";
121
+ "token-cap": "token-cap";
122
+ "review-deadline": "review-deadline";
123
+ "unusable-reply": "unusable-reply";
124
+ unknown: "unknown";
125
+ }>;
126
+ status: z.ZodOptional<z.ZodInt>;
127
+ }, z.core.$strict>>;
109
128
  }, z.core.$strict>>;
110
129
  }, z.core.$strict>>;
111
130
  discarded: z.ZodArray<z.ZodObject<{
@@ -131,6 +150,25 @@ export declare const resultSchema: z.ZodObject<{
131
150
  reasoning: z.ZodString;
132
151
  }, z.core.$strict>>;
133
152
  error: z.ZodNullable<z.ZodString>;
153
+ failure: z.ZodNullable<z.ZodObject<{
154
+ kind: z.ZodEnum<{
155
+ http: "http";
156
+ network: "network";
157
+ process: "process";
158
+ timeout: "timeout";
159
+ aborted: "aborted";
160
+ refusal: "refusal";
161
+ truncated: "truncated";
162
+ "invalid-response": "invalid-response";
163
+ unsupported: "unsupported";
164
+ "too-large": "too-large";
165
+ "token-cap": "token-cap";
166
+ "review-deadline": "review-deadline";
167
+ "unusable-reply": "unusable-reply";
168
+ unknown: "unknown";
169
+ }>;
170
+ status: z.ZodOptional<z.ZodInt>;
171
+ }, z.core.$strict>>;
134
172
  }, z.core.$strict>>;
135
173
  }, z.core.$strict>>;
136
174
  unreviewed: z.ZodArray<z.ZodObject<{
@@ -156,6 +194,25 @@ export declare const resultSchema: z.ZodObject<{
156
194
  reasoning: z.ZodString;
157
195
  }, z.core.$strict>>;
158
196
  error: z.ZodNullable<z.ZodString>;
197
+ failure: z.ZodNullable<z.ZodObject<{
198
+ kind: z.ZodEnum<{
199
+ http: "http";
200
+ network: "network";
201
+ process: "process";
202
+ timeout: "timeout";
203
+ aborted: "aborted";
204
+ refusal: "refusal";
205
+ truncated: "truncated";
206
+ "invalid-response": "invalid-response";
207
+ unsupported: "unsupported";
208
+ "too-large": "too-large";
209
+ "token-cap": "token-cap";
210
+ "review-deadline": "review-deadline";
211
+ "unusable-reply": "unusable-reply";
212
+ unknown: "unknown";
213
+ }>;
214
+ status: z.ZodOptional<z.ZodInt>;
215
+ }, z.core.$strict>>;
159
216
  }, z.core.$strict>>;
160
217
  }, z.core.$strict>>;
161
218
  meta: z.ZodObject<{
@@ -189,15 +246,28 @@ export declare const resultSchema: z.ZodObject<{
189
246
  incomplete: z.ZodArray<z.ZodObject<{
190
247
  profile: z.ZodString;
191
248
  reason: z.ZodEnum<{
249
+ "token-cap": "token-cap";
250
+ "review-deadline": "review-deadline";
192
251
  "research-unusable": "research-unusable";
252
+ "research-failed": "research-failed";
193
253
  "findings-capped": "findings-capped";
194
254
  "finding-unreadable": "finding-unreadable";
195
255
  "vote-failed": "vote-failed";
196
256
  "request-too-large": "request-too-large";
257
+ binary: "binary";
258
+ submodule: "submodule";
197
259
  }>;
198
260
  file: z.ZodOptional<z.ZodString>;
199
261
  detail: z.ZodString;
200
262
  }, z.core.$strict>>;
263
+ leftOut: z.ZodArray<z.ZodObject<{
264
+ file: z.ZodString;
265
+ reason: z.ZodEnum<{
266
+ binary: "binary";
267
+ submodule: "submodule";
268
+ excluded: "excluded";
269
+ }>;
270
+ }, z.core.$strict>>;
201
271
  flags: z.ZodArray<z.ZodObject<{
202
272
  code: z.ZodEnum<{
203
273
  "hidden-characters": "hidden-characters";
@@ -41,10 +41,30 @@ const vote = z.strictObject({
41
41
  keep: z.boolean(),
42
42
  reasoning: z.string(),
43
43
  });
44
+ const voteFailure = z.strictObject({
45
+ kind: z.enum([
46
+ 'http',
47
+ 'network',
48
+ 'process',
49
+ 'timeout',
50
+ 'aborted',
51
+ 'refusal',
52
+ 'truncated',
53
+ 'invalid-response',
54
+ 'unsupported',
55
+ 'too-large',
56
+ 'token-cap',
57
+ 'review-deadline',
58
+ 'unusable-reply',
59
+ 'unknown',
60
+ ]),
61
+ status: z.int().min(100).max(599).optional(),
62
+ });
44
63
  const lensVote = z.strictObject({
45
64
  angle: z.string(),
46
65
  vote: vote.nullable(),
47
66
  error: z.string().nullable(),
67
+ failure: voteFailure.nullable(),
48
68
  });
49
69
  const verdict = z.strictObject({
50
70
  id: z.string(),
@@ -87,14 +107,23 @@ const incompleteEntry = z.strictObject({
87
107
  profile: z.string(),
88
108
  reason: z.enum([
89
109
  'research-unusable',
110
+ 'research-failed',
90
111
  'findings-capped',
91
112
  'finding-unreadable',
92
113
  'vote-failed',
93
114
  'request-too-large',
115
+ 'token-cap',
116
+ 'review-deadline',
117
+ 'binary',
118
+ 'submodule',
94
119
  ]),
95
120
  file: z.string().optional(),
96
121
  detail: z.string(),
97
122
  });
123
+ const leftOutFile = z.strictObject({
124
+ file: z.string(),
125
+ reason: z.enum(['excluded', 'binary', 'submodule']),
126
+ });
98
127
  const flag = z.strictObject({
99
128
  code: z.enum(['hidden-characters', 'ai-addressed-text']),
100
129
  file: z.string().optional(),
@@ -117,6 +146,7 @@ export const resultSchema = z.strictObject({
117
146
  profiles: z.record(z.string(), profileMeta),
118
147
  usage: usage.optional(),
119
148
  incomplete: z.array(incompleteEntry),
149
+ leftOut: z.array(leftOutFile),
120
150
  flags: z.array(flag),
121
151
  }),
122
152
  });
@@ -44,12 +44,21 @@ export interface RunReviewsOptions {
44
44
  readonly events: (event: RunnerEvent) => void;
45
45
  /** The most a line may hold, its line break not counted, in bytes; `MAX_RECORD_BYTES` when left out. */
46
46
  readonly maxRecordBytes?: number;
47
+ /**
48
+ * Once the run's reviews have spent this many tokens, as each review's cap counts them, no review starts.
49
+ * No cap when left out.
50
+ */
51
+ readonly maxRunTokens?: number;
47
52
  }
48
53
  export interface RunReviewsOutcome {
49
54
  /** The reviews written to the record. */
50
55
  readonly recorded: number;
51
56
  /** The reviews that failed, a rejection that is no denial, or whose record was too large to write. */
52
57
  readonly failed: number;
58
+ /** The reviews that did not start because the run had spent `maxRunTokens`. */
59
+ readonly notRun: number;
60
+ /** The tokens the run's `call` events counted against their reviews' caps, as `maxRunTokens` reads them. */
61
+ readonly spent: number;
53
62
  /** Whether the caller's signal was aborted by the time the run ended. */
54
63
  readonly interrupted: boolean;
55
64
  }
@@ -68,6 +77,8 @@ export declare function planReviews(options: {
68
77
  * other rejection (an HTTP, network, timeout or process failure, a reply the server shaped wrong) says
69
78
  * nothing of the model's answer, so the review fails and runs again. The classes are matched with
70
79
  * `instanceof`: an error from a second copy of quorum is no denial, so it fails, which fails closed.
80
+ * Only a gating profile's failed research rejects a review: an advisory profile's, the call or its
81
+ * reply, resolves it with a `research-failed` gap, recorded with its result, neither failed nor denied.
71
82
  */
72
83
  export declare function denialOf(error: unknown): Denial | undefined;
73
84
  /**
@@ -87,6 +98,13 @@ export declare function denialOf(error: unknown): Denial | undefined;
87
98
  * nothing for that review, cuts the others, and the run rejects with the error of the write once every
88
99
  * review has settled. A callback of `events` that throws does the same.
89
100
  *
101
+ * The run counts what each review's cap counts: the `spent` of every `call` event of every review, one that
102
+ * failed or was denied included. Before a review starts, and after the abort's check, the run checks that
103
+ * the total is below `maxRunTokens`: once it is not, the review does not start and counts as `notRun`, the
104
+ * reviews in flight finish and are recorded, and the outcome says what was spent. The check is made before
105
+ * each review starts, so the reviews in flight can pass the cap by up to `jobs` reviews' own caps. The calls
106
+ * a failing review's own abort cut send no event, so the count misses them.
107
+ *
90
108
  * @throws the error of the first `append` (or `events`) that threw
91
109
  */
92
110
  export declare function runReviews(options: RunReviewsOptions): Promise<RunReviewsOutcome>;
@@ -66,6 +66,8 @@ function tallyCalls() {
66
66
  * other rejection (an HTTP, network, timeout or process failure, a reply the server shaped wrong) says
67
67
  * nothing of the model's answer, so the review fails and runs again. The classes are matched with
68
68
  * `instanceof`: an error from a second copy of quorum is no denial, so it fails, which fails closed.
69
+ * Only a gating profile's failed research rejects a review: an advisory profile's, the call or its
70
+ * reply, resolves it with a `research-failed` gap, recorded with its result, neither failed nor denied.
69
71
  */
70
72
  export function denialOf(error) {
71
73
  if (error instanceof OutputError)
@@ -91,10 +93,17 @@ export function denialOf(error) {
91
93
  * nothing for that review, cuts the others, and the run rejects with the error of the write once every
92
94
  * review has settled. A callback of `events` that throws does the same.
93
95
  *
96
+ * The run counts what each review's cap counts: the `spent` of every `call` event of every review, one that
97
+ * failed or was denied included. Before a review starts, and after the abort's check, the run checks that
98
+ * the total is below `maxRunTokens`: once it is not, the review does not start and counts as `notRun`, the
99
+ * reviews in flight finish and are recorded, and the outcome says what was spent. The check is made before
100
+ * each review starts, so the reviews in flight can pass the cap by up to `jobs` reviews' own caps. The calls
101
+ * a failing review's own abort cut send no event, so the count misses them.
102
+ *
94
103
  * @throws the error of the first `append` (or `events`) that threw
95
104
  */
96
105
  export async function runReviews(options) {
97
- const { suite, configurations, reviews, jobs, env, append, events, maxRecordBytes = MAX_RECORD_BYTES } = options;
106
+ const { suite, configurations, reviews, jobs, env, append, events, maxRecordBytes = MAX_RECORD_BYTES, maxRunTokens = Infinity } = options;
98
107
  const own = new AbortController();
99
108
  const signal = options.signal === undefined
100
109
  ? own.signal
@@ -105,6 +114,8 @@ export async function runReviews(options) {
105
114
  ]));
106
115
  let recorded = 0;
107
116
  let failed = 0;
117
+ let notRun = 0;
118
+ let spent = 0;
108
119
  let stopped;
109
120
  const stop = (error) => {
110
121
  stopped ??= { error };
@@ -128,7 +139,12 @@ export async function runReviews(options) {
128
139
  if (configuration === undefined || diff === undefined)
129
140
  throw new Error(UNKNOWN_REVIEW);
130
141
  const tally = tallyCalls();
131
- const quorum = await fromConfig(configuration.config, { env, events: tally.sink });
142
+ const sink = (event) => {
143
+ if (event.type === 'call')
144
+ spent += event.spent;
145
+ tally.sink(event);
146
+ };
147
+ const quorum = await fromConfig(configuration.config, { env, events: sink });
132
148
  const started = performance.now();
133
149
  const facts = () => ({
134
150
  fingerprint: configuration.fingerprint,
@@ -160,6 +176,10 @@ export async function runReviews(options) {
160
176
  async function runOne(review) {
161
177
  if (signal.aborted)
162
178
  return;
179
+ if (spent >= maxRunTokens) {
180
+ notRun += 1;
181
+ return;
182
+ }
163
183
  let record;
164
184
  let line;
165
185
  try {
@@ -191,5 +211,11 @@ export async function runReviews(options) {
191
211
  await Promise.all(reviews.map(review => limit(() => runOne(review))));
192
212
  if (stopped !== undefined)
193
213
  throw stopped.error;
194
- return { recorded, failed, interrupted: options.signal?.aborted === true };
214
+ return {
215
+ recorded,
216
+ failed,
217
+ notRun,
218
+ spent,
219
+ interrupted: options.signal?.aborted === true,
220
+ };
195
221
  }
@@ -31,12 +31,19 @@ export declare const BENCH_USAGE: string;
31
31
  * `results.jsonl` to append, so a cut last line stays, and warns of it. It refuses a run directory that
32
32
  * holds no run, and `--retry-incomplete`, which runs reviews.
33
33
  *
34
+ * `--max-run-tokens <n>` stops the run from starting reviews once its reviews have spent `n` tokens, counted
35
+ * from each call's `spent`, the tokens its review's own cap counted (`maxReviewTokens`): the reviews in flight
36
+ * finish and are recorded, the reports are written, one stderr line says how many reviews were not run, and
37
+ * the next run runs them. The check is made before each review starts, so the reviews in flight can pass
38
+ * the cap by up to `--jobs` reviews' own caps, and the calls a failing review's abort cut are not counted.
39
+ *
34
40
  * Exit codes: 0 when every planned review is recorded, a denied one included, or `--report-only` wrote the
35
41
  * reports, 2 for a usage, suite, configuration or run directory error, 3 when the run stops (a file cannot
36
- * be written or read), 4 when a review failed (its research call failed, or its record was too large to
37
- * write: the others are recorded, and the next run runs the failed ones), 130 when interrupted. The
38
- * reports are written on 0, 4 and 130, so a stopped run shows what it recorded; an interrupt during the
39
- * checks and the plan leaves the run directory as it was, even one that does not exist yet.
42
+ * be written or read), 4 when a review failed (its gating profile's research call failed, or its record
43
+ * was too large to write: the others are recorded, and the next run runs the failed ones) or
44
+ * `--max-run-tokens` left reviews unrun, 130 when interrupted. The reports are written on 0, 4 and 130,
45
+ * so a stopped run shows what it recorded; an interrupt during the checks and the plan leaves the run
46
+ * directory as it was, even one that does not exist yet.
40
47
  *
41
48
  * Every line it writes is inert (invariant I3).
42
49
  */
package/dist/cli/bench.js CHANGED
@@ -26,22 +26,23 @@ export const BENCH_USAGE = [
26
26
  'Review every case of a suite with every config, --trials times each, and report what each caught.',
27
27
  '',
28
28
  'Options:',
29
- ' -c, --config <path> A config file to compare; repeat it, and the first is the reference',
30
- ' -p, --profile <id> Run only this profile; repeat it to run several',
31
- ' --trials <n> Reviews of each case per config: the suite\'s trials, else 1',
32
- ' --case <id> Run only this case; repeat it to run several',
33
- ' -j, --jobs <n> The reviews run at once (default 1)',
34
- ' -o, --out <dir> The run directory (default quorum-bench/<suite name>)',
35
- ' --retry-incomplete Run again the recorded reviews that came back incomplete',
36
- ' --report-only Write the reports of what is recorded, and run no review',
37
- ' --dry-run Check the suite and the configs, print the plan, and exit',
38
- ' -h, --help Print this help and exit',
29
+ ' -c, --config <path> A config file to compare; repeat it, and the first is the reference',
30
+ ' -p, --profile <id> Run only this profile; repeat it to run several',
31
+ ' --trials <n> Reviews of each case per config: the suite\'s trials, else 1',
32
+ ' --case <id> Run only this case; repeat it to run several',
33
+ ' -j, --jobs <n> The reviews run at once (default 1)',
34
+ ' --max-run-tokens <n> Start no review once the run has spent <n> tokens; the next run runs the rest',
35
+ ' -o, --out <dir> The run directory (default quorum-bench/<suite name>)',
36
+ ' --retry-incomplete Run again the recorded reviews that came back incomplete',
37
+ ' --report-only Write the reports of what is recorded, and run no review',
38
+ ' --dry-run Check the suite and the configs, print the plan, and exit',
39
+ ' -h, --help Print this help and exit',
39
40
  '',
40
41
  ].join('\n');
41
42
  /**
42
43
  * The bench's own exit codes (Decision 18), not the review's `EXIT`: `failed` means reviews failed and were
43
- * not recorded, not that a review left a gap or that a model's reply denied it, and `stopped` means the
44
- * run could not go on.
44
+ * not recorded, or that `--max-run-tokens` left reviews unrun, not that a review left a gap or that a
45
+ * model's reply denied it, and `stopped` means the run could not go on.
45
46
  */
46
47
  const BENCH_EXIT = {
47
48
  ok: 0,
@@ -61,6 +62,7 @@ const OPTIONS = {
61
62
  trials: { type: 'string' },
62
63
  case: { type: 'string', multiple: true },
63
64
  jobs: { type: 'string', short: 'j' },
65
+ 'max-run-tokens': { type: 'string' },
64
66
  out: { type: 'string', short: 'o' },
65
67
  'retry-incomplete': { type: 'boolean' },
66
68
  'report-only': { type: 'boolean' },
@@ -97,6 +99,7 @@ function checkOptions(values, positionals, cwd) {
97
99
  throw new Error(`unexpected arguments: ${extra.join(' ')}`);
98
100
  const trials = wholeNumber('--trials', values.trials, MAX_TRIALS);
99
101
  const jobs = wholeNumber('--jobs', values.jobs, MAX_JOBS);
102
+ const maxRunTokens = wholeNumber('--max-run-tokens', values['max-run-tokens'], Number.MAX_SAFE_INTEGER);
100
103
  const configs = values.config ?? [];
101
104
  if (configs.includes(''))
102
105
  throw usageIssue('--config', 'must name a file');
@@ -121,6 +124,7 @@ function checkOptions(values, positionals, cwd) {
121
124
  trials,
122
125
  cases,
123
126
  jobs: jobs ?? 1,
127
+ maxRunTokens,
124
128
  out: values.out,
125
129
  retryIncomplete: values['retry-incomplete'] === true,
126
130
  reportOnly: values['report-only'] === true,
@@ -355,7 +359,7 @@ async function writeRun({ out, state, runFile }) {
355
359
  * writer is closed whatever happens; when the run rejects, with the error of a failed write, that error
356
360
  * is the one thrown.
357
361
  */
358
- async function execute(io, { checked, scores, records, toRun, jobs }) {
362
+ async function execute(io, { checked, scores, records, toRun, jobs, maxRunTokens }) {
359
363
  const { suite, loaded, byId, out, runFile } = checked;
360
364
  await writeRun(checked);
361
365
  const writer = await openRecords(out, records);
@@ -367,6 +371,7 @@ async function execute(io, { checked, scores, records, toRun, jobs }) {
367
371
  configurations: loaded.map(({ config }, index) => ({ config, fingerprint: runFile.configurations[index].fingerprint })),
368
372
  reviews: toRun,
369
373
  jobs,
374
+ maxRunTokens,
370
375
  env: io.env,
371
376
  signal: io.signal,
372
377
  append: writer.append,
@@ -458,12 +463,17 @@ async function bench(io, options) {
458
463
  records,
459
464
  toRun,
460
465
  jobs: options.jobs,
466
+ maxRunTokens: options.maxRunTokens,
461
467
  });
462
468
  await writeReports(io, checked, trials, scores);
463
469
  if (outcome.interrupted) {
464
470
  io.stderr.write('interrupted\n');
465
471
  return BENCH_EXIT.interrupted;
466
472
  }
473
+ if (outcome.notRun > 0 && options.maxRunTokens !== undefined) {
474
+ writeLines(io.stderr, [`--max-run-tokens: the run reached its cap of ${options.maxRunTokens} tokens (it spent ${outcome.spent}), so ${outcome.notRun} review(s) were not run; the next run runs them`]);
475
+ return BENCH_EXIT.failed;
476
+ }
467
477
  return outcome.failed > 0
468
478
  ? BENCH_EXIT.failed
469
479
  : BENCH_EXIT.ok;
@@ -499,12 +509,19 @@ function usageError(io, error) {
499
509
  * `results.jsonl` to append, so a cut last line stays, and warns of it. It refuses a run directory that
500
510
  * holds no run, and `--retry-incomplete`, which runs reviews.
501
511
  *
512
+ * `--max-run-tokens <n>` stops the run from starting reviews once its reviews have spent `n` tokens, counted
513
+ * from each call's `spent`, the tokens its review's own cap counted (`maxReviewTokens`): the reviews in flight
514
+ * finish and are recorded, the reports are written, one stderr line says how many reviews were not run, and
515
+ * the next run runs them. The check is made before each review starts, so the reviews in flight can pass
516
+ * the cap by up to `--jobs` reviews' own caps, and the calls a failing review's abort cut are not counted.
517
+ *
502
518
  * Exit codes: 0 when every planned review is recorded, a denied one included, or `--report-only` wrote the
503
519
  * reports, 2 for a usage, suite, configuration or run directory error, 3 when the run stops (a file cannot
504
- * be written or read), 4 when a review failed (its research call failed, or its record was too large to
505
- * write: the others are recorded, and the next run runs the failed ones), 130 when interrupted. The
506
- * reports are written on 0, 4 and 130, so a stopped run shows what it recorded; an interrupt during the
507
- * checks and the plan leaves the run directory as it was, even one that does not exist yet.
520
+ * be written or read), 4 when a review failed (its gating profile's research call failed, or its record
521
+ * was too large to write: the others are recorded, and the next run runs the failed ones) or
522
+ * `--max-run-tokens` left reviews unrun, 130 when interrupted. The reports are written on 0, 4 and 130,
523
+ * so a stopped run shows what it recorded; an interrupt during the checks and the plan leaves the run
524
+ * directory as it was, even one that does not exist yet.
508
525
  *
509
526
  * Every line it writes is inert (invariant I3).
510
527
  */
package/dist/cli/bin.js CHANGED
@@ -13,8 +13,8 @@ function fail(error) {
13
13
  }
14
14
  process.on('uncaughtException', fail);
15
15
  process.on('unhandledRejection', fail);
16
- // Exit 3 until the run settles. A provider whose promise never settles, and holds no timer or socket,
17
- // empties the event loop: the runtime then exits with the code it holds, which must never be 0 with no result.
16
+ // Exit 3 until the run settles. A config module whose loading never finishes leaves nothing to wait
17
+ // for and empties the event loop: the runtime then exits with the code it holds, which must never be 0 with no result.
18
18
  process.exitCode = EXIT.runtime;
19
19
  let settled = false;
20
20
  process.once('beforeExit', () => {
@@ -5,10 +5,12 @@ export { errorMessage, EXIT };
5
5
  export type { CliIO };
6
6
  /**
7
7
  * Run one review and return the process exit code. Never throws: usage and configuration problems
8
- * exit 2, runtime failures exit 3, an interrupt exits 130, and an incomplete review (a finding its
9
- * failed votes left undecided, or a research gap) of a gating profile exits 4 unless a `high` finding
10
- * was kept by a gating profile, so CI can tell a broken run and a partial review from a real finding
11
- * (exit 1) and from a clean one (exit 0). `exitCodeFor` maps the result; advisory flags never change it.
8
+ * exit 2, a gating profile's failed research and other runtime failures exit 3, an interrupt exits
9
+ * 130, and an incomplete review (a finding its failed votes left undecided, or a gap) of a gating
10
+ * profile exits 4 unless a `high` finding was kept by a gating profile, so CI can tell a broken run
11
+ * and a partial review from a real finding (exit 1) and from a clean one (exit 0). `--allow-incomplete`
12
+ * turns that exit 4 into 0 only when every failed vote behind it failed in passing (see `exitCodeFor`).
13
+ * `exitCodeFor` maps the result; advisory flags never change it.
12
14
  *
13
15
  * It reads at most 16 MiB of diff, from stdin or a file, counted in bytes: a larger diff is a usage error,
14
16
  * and reading stops at the chunk that crosses the cap.
@@ -40,22 +42,40 @@ export interface SignalSource {
40
42
  }
41
43
  /**
42
44
  * The signal of one review, aborted by the first SIGINT or SIGTERM. The next one calls `exit(130)`: the
43
- * way out of a provider that ignores its signal, since no timer waits for it.
45
+ * way out when what a provider still runs, after the review settled its call, keeps the process alive.
44
46
  */
45
47
  export declare function interruptSignal(source: SignalSource, exit: (code: number) => void): AbortSignal;
46
48
  /**
47
49
  * The exit code of a finished review: 1 when a kept finding is `high` and its profile gates, 4 when
48
- * an unreviewed finding or a gap belongs to a gating profile (unless `allowIncomplete` turns it into
49
- * 0, which it does for every gap but a `request-too-large` one), 0 otherwise. A profile gates
50
- * unless `meta.profiles` holds it with `gating: false`, so a profile missing from the result counts
51
- * as gating and a gap never hides.
50
+ * an unreviewed finding or a gap belongs to a gating profile, 0 otherwise. `allowIncomplete` turns
51
+ * that 4 into 0 only when every gap of a gating profile is a `vote-failed` one (an unreviewed finding
52
+ * is no gap) and, for each gating profile that has a gap, every failed vote of its own kept and
53
+ * unreviewed findings failed in passing, as `failedInPassing` says: a dropped connection, a timeout,
54
+ * or an HTTP 408, 409, 429 or 5xx. A gating profile with no gap blocks no other profile's lift. A gap
55
+ * with no failed vote behind it lifts nothing: an unreviewed finding whose votes hold none, or a
56
+ * `vote-failed` gap beside kept findings that hold none, which no review writes.
57
+ * A profile gates unless `meta.profiles` holds it with `gating: false`, so a profile missing from the
58
+ * result counts as gating and a gap never hides.
52
59
  *
53
- * A `request-too-large` gap of a gating profile exits 4 whatever `allowIncomplete` says: a pull
54
- * request can cause that gap by padding the diff until a request is refused, so no flag lifts it
55
- * (ADR-0012). The flag lifts every other gap, including gaps a pull request can cause: decoy findings
56
- * that push a real one past `maxFindings`, or text that trips a hosted voter's content filter. On a
57
- * gating profile, do not use it for a pull request you do not trust.
60
+ * Why so narrow: a pull request causes the other gaps directly (ADR-0012). It pads the diff until a
61
+ * request is refused as too large, floods the research with decoy findings past `maxFindings`, or
62
+ * plants text that a voter refuses or that cuts its reply off or leaves it unusable, and one
63
+ * `.gitattributes` line, one NUL byte or a submodule's bump leaves a file's change out of the diff.
64
+ * The flag lifts only what the code names, so a reason added later is out of its reach by default,
65
+ * and so is what a result built by hand or by an older quorum leaves unsaid: a failed vote with no
66
+ * `failure`, a verdict whose `votes` is no array, a reason no quorum writes. A gating profile's
67
+ * `research-failed` gap, which no review records, is out too: a review that read nothing is never
68
+ * turned into exit 0. Planted text can still draw out a reply that ends in a lifted failure, a
69
+ * timeout of its own, so on a gating profile do not use the flag for a pull request you do not trust.
58
70
  */
59
71
  export declare function exitCodeFor(result: QuorumResult, options?: {
60
72
  readonly allowIncomplete?: boolean;
61
73
  }): number;
74
+ /**
75
+ * True for a vote's failure that is infrastructure's: its own data `kind` is `network` or `timeout`, or
76
+ * `http` with an own data `status` of 408, 409, 429 or an integer from 500 to 599. Anything else is false:
77
+ * a refusal, a reply cut off or unusable, a request too large, any other status, a status that is no
78
+ * number, a kind no provider has, `null` and a value that is no object. It reads own data only, never
79
+ * runs a getter and never throws.
80
+ */
81
+ export declare function failedInPassing(failure: unknown): boolean;