scene-capability-engine 3.3.16 → 3.3.17
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +21 -0
- package/README.md +1 -1
- package/README.zh.md +1 -1
- package/docs/agent-runtime/orchestrator-rate-limit-profiles.md +66 -0
- package/docs/command-reference.md +29 -1
- package/lib/commands/orchestrate.js +223 -3
- package/lib/orchestrator/orchestration-engine.js +44 -7
- package/lib/orchestrator/orchestrator-config.js +91 -2
- package/package.json +2 -1
- package/template/.sce/config/orchestrator.json +24 -0
package/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [3.3.17] - 2026-02-26
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
- Orchestrator rate-limit profile management commands:
|
|
14
|
+
- `sce orchestrate profile list`
|
|
15
|
+
- `sce orchestrate profile show`
|
|
16
|
+
- `sce orchestrate profile set <conservative|balanced|aggressive> [--reset-overrides]`
|
|
17
|
+
- Runtime one-shot profile override:
|
|
18
|
+
- `sce orchestrate run --rate-limit-profile <profile>`
|
|
19
|
+
- New anti-429 regression shortcut:
|
|
20
|
+
- `npm run test:orchestrator-429`
|
|
21
|
+
- Added default orchestrator baseline config files:
|
|
22
|
+
- `.sce/config/orchestrator.json`
|
|
23
|
+
- `template/.sce/config/orchestrator.json`
|
|
24
|
+
- Added profile runbook:
|
|
25
|
+
- `docs/agent-runtime/orchestrator-rate-limit-profiles.md`
|
|
26
|
+
|
|
27
|
+
### Changed
|
|
28
|
+
- Orchestration engine now supports runtime config overrides for one execution without mutating persisted config.
|
|
29
|
+
- Command reference updated with profile workflow and recommended anti-429 usage.
|
|
30
|
+
|
|
10
31
|
## [3.3.16] - 2026-02-26
|
|
11
32
|
|
|
12
33
|
### Added
|
package/README.md
CHANGED
package/README.zh.md
CHANGED
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
# Orchestrator Rate-Limit Profiles
|
|
2
|
+
|
|
3
|
+
This document defines the default anti-429 presets used by SCE multi-agent orchestration.
|
|
4
|
+
|
|
5
|
+
## Profiles
|
|
6
|
+
|
|
7
|
+
| Profile | Positioning | Best for |
|
|
8
|
+
|---|---|---|
|
|
9
|
+
| `conservative` | strongest throttling, safest | unstable provider quota windows, repeated `429` spikes |
|
|
10
|
+
| `balanced` | default baseline | normal daily multi-agent runs |
|
|
11
|
+
| `aggressive` | higher throughput, lower safety margin | stable quota windows with strict delivery deadlines |
|
|
12
|
+
|
|
13
|
+
## Effective Preset Values
|
|
14
|
+
|
|
15
|
+
| Key | conservative | balanced | aggressive |
|
|
16
|
+
|---|---:|---:|---:|
|
|
17
|
+
| `rateLimitMaxRetries` | 10 | 8 | 6 |
|
|
18
|
+
| `rateLimitBackoffBaseMs` | 2200 | 1500 | 1000 |
|
|
19
|
+
| `rateLimitBackoffMaxMs` | 90000 | 60000 | 30000 |
|
|
20
|
+
| `rateLimitCooldownMs` | 60000 | 45000 | 20000 |
|
|
21
|
+
| `rateLimitLaunchBudgetPerMinute` | 4 | 8 | 16 |
|
|
22
|
+
| `rateLimitSignalWindowMs` | 45000 | 30000 | 20000 |
|
|
23
|
+
| `rateLimitSignalThreshold` | 2 | 3 | 4 |
|
|
24
|
+
| `rateLimitSignalExtraHoldMs` | 5000 | 3000 | 2000 |
|
|
25
|
+
| `rateLimitDynamicBudgetFloor` | 1 | 1 | 2 |
|
|
26
|
+
|
|
27
|
+
## Usage
|
|
28
|
+
|
|
29
|
+
Persistent (writes `.sce/config/orchestrator.json`):
|
|
30
|
+
|
|
31
|
+
```bash
|
|
32
|
+
sce orchestrate profile set conservative
|
|
33
|
+
sce orchestrate profile set balanced --reset-overrides
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
One-shot for a single run (does not change file):
|
|
37
|
+
|
|
38
|
+
```bash
|
|
39
|
+
sce orchestrate run --specs "spec-a,spec-b,spec-c" --rate-limit-profile conservative
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
Inspect current effective state:
|
|
43
|
+
|
|
44
|
+
```bash
|
|
45
|
+
sce orchestrate profile show --json
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
## Validation Checklist
|
|
49
|
+
|
|
50
|
+
Run anti-429 regression:
|
|
51
|
+
|
|
52
|
+
```bash
|
|
53
|
+
npm run test:orchestrator-429
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
Run full suite before release:
|
|
57
|
+
|
|
58
|
+
```bash
|
|
59
|
+
npm test -- --runInBand
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
Release readiness criteria:
|
|
63
|
+
|
|
64
|
+
1. No failing test in orchestrator/rate-limit scope.
|
|
65
|
+
2. `orchestrate profile show --json` returns expected profile and effective values.
|
|
66
|
+
3. Multi-agent run no longer stalls under sustained `429`; launch budget and hold telemetry progress over time.
|
|
@@ -295,15 +295,29 @@ sce repo health [--json]
|
|
|
295
295
|
# Start orchestration for multiple specs
|
|
296
296
|
sce orchestrate run --specs "spec-a,spec-b,spec-c" --max-parallel 3
|
|
297
297
|
|
|
298
|
+
# One-shot anti-429 profile override (without editing orchestrator.json)
|
|
299
|
+
sce orchestrate run --specs "spec-a,spec-b,spec-c" --rate-limit-profile conservative
|
|
300
|
+
|
|
298
301
|
# Show orchestration status
|
|
299
302
|
sce orchestrate status [--json]
|
|
300
303
|
|
|
301
304
|
# Stop all running sub-agents
|
|
302
305
|
sce orchestrate stop
|
|
306
|
+
|
|
307
|
+
# List/show/set persistent rate-limit profile
|
|
308
|
+
sce orchestrate profile list
|
|
309
|
+
sce orchestrate profile show --json
|
|
310
|
+
sce orchestrate profile set conservative
|
|
311
|
+
sce orchestrate profile set balanced --reset-overrides
|
|
303
312
|
```
|
|
304
313
|
|
|
305
314
|
When you pass `--specs` to `sce spec bootstrap|pipeline run|gate run`, sce now defaults to this orchestrate mode automatically.
|
|
306
315
|
|
|
316
|
+
Rate-limit profiles:
|
|
317
|
+
- `conservative`: strongest anti-429 throttling (recommended for unstable quota windows)
|
|
318
|
+
- `balanced`: default profile for normal multi-agent runs
|
|
319
|
+
- `aggressive`: higher throughput with lower protection margins
|
|
320
|
+
|
|
307
321
|
### Studio Workflow
|
|
308
322
|
|
|
309
323
|
```bash
|
|
@@ -408,6 +422,7 @@ Contract/baseline files:
|
|
|
408
422
|
- `docs/agent-runtime/capability-mapping-report.schema.json`
|
|
409
423
|
- `docs/agent-runtime/agent-result-summary-contract.schema.json`
|
|
410
424
|
- `docs/agent-runtime/multi-agent-coordination-policy-baseline.json`
|
|
425
|
+
- `docs/agent-runtime/orchestrator-rate-limit-profiles.md`
|
|
411
426
|
|
|
412
427
|
Multi-agent merge governance default:
|
|
413
428
|
- `sce orchestrate run` loads `docs/agent-runtime/multi-agent-coordination-policy-baseline.json`.
|
|
@@ -1327,6 +1342,7 @@ Recommended `.sce/config/orchestrator.json`:
|
|
|
1327
1342
|
"maxParallel": 3,
|
|
1328
1343
|
"timeoutSeconds": 900,
|
|
1329
1344
|
"maxRetries": 2,
|
|
1345
|
+
"rateLimitProfile": "balanced",
|
|
1330
1346
|
"rateLimitMaxRetries": 8,
|
|
1331
1347
|
"rateLimitBackoffBaseMs": 1500,
|
|
1332
1348
|
"rateLimitBackoffMaxMs": 60000,
|
|
@@ -1335,13 +1351,25 @@ Recommended `.sce/config/orchestrator.json`:
|
|
|
1335
1351
|
"rateLimitCooldownMs": 45000,
|
|
1336
1352
|
"rateLimitLaunchBudgetPerMinute": 8,
|
|
1337
1353
|
"rateLimitLaunchBudgetWindowMs": 60000,
|
|
1354
|
+
"rateLimitSignalWindowMs": 30000,
|
|
1355
|
+
"rateLimitSignalThreshold": 3,
|
|
1356
|
+
"rateLimitSignalExtraHoldMs": 3000,
|
|
1357
|
+
"rateLimitDynamicBudgetFloor": 1,
|
|
1338
1358
|
"apiKeyEnvVar": "CODEX_API_KEY",
|
|
1339
1359
|
"codexArgs": ["--skip-git-repo-check"],
|
|
1340
1360
|
"codexCommand": "npx @openai/codex"
|
|
1341
1361
|
}
|
|
1342
1362
|
```
|
|
1343
1363
|
|
|
1344
|
-
`
|
|
1364
|
+
`rateLimitProfile` applies preset anti-429 behavior (`conservative|balanced|aggressive`). Any explicit `rateLimit*` field in `orchestrator.json` overrides the selected profile value.
|
|
1365
|
+
|
|
1366
|
+
`rateLimit*` settings provide dedicated retry/backoff and adaptive throttling when providers return 429 / too-many-requests errors. Engine retry honors `Retry-After` / `try again in ...` hints from provider error messages and clamps final retry waits by `rateLimitBackoffMaxMs` to avoid unbounded pause windows. During active backoff windows, new pending spec launches are paused to reduce request bursts (launch hold remains active even if adaptive parallel throttling is disabled). Sustained 429 spikes are additionally controlled by:
|
|
1367
|
+
- `rateLimitSignalWindowMs`: rolling signal window for spike detection
|
|
1368
|
+
- `rateLimitSignalThreshold`: signals required inside window before escalation
|
|
1369
|
+
- `rateLimitSignalExtraHoldMs`: extra launch hold per escalation unit
|
|
1370
|
+
- `rateLimitDynamicBudgetFloor`: lowest dynamic launch budget allowed during sustained pressure
|
|
1371
|
+
|
|
1372
|
+
`orchestrate stop` interrupts pending retry waits immediately so long backoff windows do not look like deadlocks.
|
|
1345
1373
|
|
|
1346
1374
|
Codex sub-agent permission defaults:
|
|
1347
1375
|
- `--sandbox danger-full-access` is always injected by orchestrator runtime.
|
|
@@ -13,6 +13,26 @@ const fs = require('fs-extra');
|
|
|
13
13
|
|
|
14
14
|
const SPECS_DIR = '.sce/specs';
|
|
15
15
|
const STATUS_FILE = '.sce/config/orchestration-status.json';
|
|
16
|
+
const ORCHESTRATOR_CONFIG_FILE = '.sce/config/orchestrator.json';
|
|
17
|
+
const FALLBACK_RATE_LIMIT_PROFILE_PRESETS = Object.freeze({
|
|
18
|
+
conservative: true,
|
|
19
|
+
balanced: true,
|
|
20
|
+
aggressive: true,
|
|
21
|
+
});
|
|
22
|
+
const RATE_LIMIT_FIELD_KEYS = Object.freeze([
|
|
23
|
+
'rateLimitMaxRetries',
|
|
24
|
+
'rateLimitBackoffBaseMs',
|
|
25
|
+
'rateLimitBackoffMaxMs',
|
|
26
|
+
'rateLimitAdaptiveParallel',
|
|
27
|
+
'rateLimitParallelFloor',
|
|
28
|
+
'rateLimitCooldownMs',
|
|
29
|
+
'rateLimitLaunchBudgetPerMinute',
|
|
30
|
+
'rateLimitLaunchBudgetWindowMs',
|
|
31
|
+
'rateLimitSignalWindowMs',
|
|
32
|
+
'rateLimitSignalThreshold',
|
|
33
|
+
'rateLimitSignalExtraHoldMs',
|
|
34
|
+
'rateLimitDynamicBudgetFloor',
|
|
35
|
+
]);
|
|
16
36
|
|
|
17
37
|
/**
|
|
18
38
|
* Run orchestration programmatically.
|
|
@@ -21,6 +41,7 @@ const STATUS_FILE = '.sce/config/orchestration-status.json';
|
|
|
21
41
|
* @param {string} [options.specs] - Comma separated spec names
|
|
22
42
|
* @param {string[]} [options.specNames] - Explicit spec names array
|
|
23
43
|
* @param {number} [options.maxParallel]
|
|
44
|
+
* @param {string} [options.rateLimitProfile]
|
|
24
45
|
* @param {boolean} [options.json]
|
|
25
46
|
* @param {boolean} [options.silent]
|
|
26
47
|
* @param {object} dependencies
|
|
@@ -42,12 +63,21 @@ async function runOrchestration(options = {}, dependencies = {}) {
|
|
|
42
63
|
throw new Error('--max-parallel must be >= 1');
|
|
43
64
|
}
|
|
44
65
|
|
|
66
|
+
const rawRateLimitProfile = options.rateLimitProfile === undefined || options.rateLimitProfile === null
|
|
67
|
+
? null
|
|
68
|
+
: `${options.rateLimitProfile}`.trim().toLowerCase();
|
|
69
|
+
|
|
45
70
|
const missing = await _validateSpecs(workspaceRoot, specNames);
|
|
46
71
|
if (missing.length > 0) {
|
|
47
72
|
throw new Error(`Specs not found: ${missing.join(', ')}`);
|
|
48
73
|
}
|
|
49
74
|
|
|
50
|
-
const
|
|
75
|
+
const orchestratorConfigModule = require('../orchestrator/orchestrator-config');
|
|
76
|
+
const {
|
|
77
|
+
OrchestratorConfig,
|
|
78
|
+
RATE_LIMIT_PROFILE_PRESETS,
|
|
79
|
+
buildRateLimitProfileConfig
|
|
80
|
+
} = orchestratorConfigModule;
|
|
51
81
|
const { BootstrapPromptBuilder } = require('../orchestrator/bootstrap-prompt-builder');
|
|
52
82
|
const { AgentSpawner } = require('../orchestrator/agent-spawner');
|
|
53
83
|
const { StatusMonitor } = require('../orchestrator/status-monitor');
|
|
@@ -69,6 +99,20 @@ async function runOrchestration(options = {}, dependencies = {}) {
|
|
|
69
99
|
|
|
70
100
|
const orchestratorConfig = new OrchestratorConfig(workspaceRoot);
|
|
71
101
|
const config = await orchestratorConfig.getConfig();
|
|
102
|
+
const availableProfiles = Object.keys(
|
|
103
|
+
RATE_LIMIT_PROFILE_PRESETS || FALLBACK_RATE_LIMIT_PROFILE_PRESETS
|
|
104
|
+
);
|
|
105
|
+
const selectedRateLimitProfile = rawRateLimitProfile || null;
|
|
106
|
+
if (selectedRateLimitProfile && !availableProfiles.includes(selectedRateLimitProfile)) {
|
|
107
|
+
throw new Error(
|
|
108
|
+
`--rate-limit-profile must be one of: ${availableProfiles.join(', ')}`
|
|
109
|
+
);
|
|
110
|
+
}
|
|
111
|
+
const runtimeRateLimitOverrides = selectedRateLimitProfile
|
|
112
|
+
? (typeof buildRateLimitProfileConfig === 'function'
|
|
113
|
+
? buildRateLimitProfileConfig(selectedRateLimitProfile)
|
|
114
|
+
: { rateLimitProfile: selectedRateLimitProfile })
|
|
115
|
+
: null;
|
|
72
116
|
const effectiveMaxParallel = maxParallel || config.maxParallel;
|
|
73
117
|
|
|
74
118
|
const bootstrapPromptBuilder = new BootstrapPromptBuilder(workspaceRoot, orchestratorConfig);
|
|
@@ -98,7 +142,7 @@ async function runOrchestration(options = {}, dependencies = {}) {
|
|
|
98
142
|
if (!options.silent && !options.json) {
|
|
99
143
|
console.log(
|
|
100
144
|
chalk.blue('🚀'),
|
|
101
|
-
`Starting orchestration for ${specNames.length} spec(s) (max-parallel: ${effectiveMaxParallel})...`
|
|
145
|
+
`Starting orchestration for ${specNames.length} spec(s) (max-parallel: ${effectiveMaxParallel})${selectedRateLimitProfile ? `, rate-limit-profile: ${selectedRateLimitProfile}` : ''}...`
|
|
102
146
|
);
|
|
103
147
|
}
|
|
104
148
|
|
|
@@ -146,7 +190,10 @@ async function runOrchestration(options = {}, dependencies = {}) {
|
|
|
146
190
|
|
|
147
191
|
let result;
|
|
148
192
|
try {
|
|
149
|
-
result = await engine.start(specNames, {
|
|
193
|
+
result = await engine.start(specNames, {
|
|
194
|
+
maxParallel: effectiveMaxParallel,
|
|
195
|
+
configOverrides: runtimeRateLimitOverrides
|
|
196
|
+
});
|
|
150
197
|
} finally {
|
|
151
198
|
clearInterval(statusTimer);
|
|
152
199
|
await persistStatus();
|
|
@@ -189,6 +236,10 @@ function registerOrchestrateCommands(program) {
|
|
|
189
236
|
.description('Start orchestration for specified Specs')
|
|
190
237
|
.requiredOption('--specs <specs>', 'Comma-separated list of Spec names')
|
|
191
238
|
.option('--max-parallel <n>', 'Maximum parallel agents', parseInt)
|
|
239
|
+
.option(
|
|
240
|
+
'--rate-limit-profile <profile>',
|
|
241
|
+
'Rate-limit profile (conservative|balanced|aggressive)'
|
|
242
|
+
)
|
|
192
243
|
.option('--json', 'Output in JSON format')
|
|
193
244
|
.action(async (options) => {
|
|
194
245
|
try {
|
|
@@ -256,6 +307,157 @@ function registerOrchestrateCommands(program) {
|
|
|
256
307
|
process.exit(1);
|
|
257
308
|
}
|
|
258
309
|
});
|
|
310
|
+
|
|
311
|
+
// ── sce orchestrate profile * ────────────────────────────────────
|
|
312
|
+
const profile = orchestrate
|
|
313
|
+
.command('profile')
|
|
314
|
+
.description('Manage orchestrator rate-limit profiles');
|
|
315
|
+
|
|
316
|
+
profile
|
|
317
|
+
.command('list')
|
|
318
|
+
.description('List available rate-limit profiles')
|
|
319
|
+
.option('--json', 'Output in JSON format')
|
|
320
|
+
.action(async (options) => {
|
|
321
|
+
try {
|
|
322
|
+
const orchestratorConfigModule = require('../orchestrator/orchestrator-config');
|
|
323
|
+
const presets = orchestratorConfigModule.RATE_LIMIT_PROFILE_PRESETS || FALLBACK_RATE_LIMIT_PROFILE_PRESETS;
|
|
324
|
+
const profiles = Object.keys(presets);
|
|
325
|
+
|
|
326
|
+
if (options.json) {
|
|
327
|
+
console.log(JSON.stringify({ profiles }, null, 2));
|
|
328
|
+
return;
|
|
329
|
+
}
|
|
330
|
+
|
|
331
|
+
console.log(chalk.bold('Rate-limit profiles'));
|
|
332
|
+
for (const item of profiles) {
|
|
333
|
+
console.log(` - ${item}`);
|
|
334
|
+
}
|
|
335
|
+
} catch (err) {
|
|
336
|
+
_errorAndExit(err.message, options.json);
|
|
337
|
+
}
|
|
338
|
+
});
|
|
339
|
+
|
|
340
|
+
profile
|
|
341
|
+
.command('show')
|
|
342
|
+
.description('Show active rate-limit profile and effective anti-429 settings')
|
|
343
|
+
.option('--json', 'Output in JSON format')
|
|
344
|
+
.action(async (options) => {
|
|
345
|
+
try {
|
|
346
|
+
const workspaceRoot = process.cwd();
|
|
347
|
+
const orchestratorConfigModule = require('../orchestrator/orchestrator-config');
|
|
348
|
+
const { OrchestratorConfig, RATE_LIMIT_PROFILE_PRESETS, resolveRateLimitProfileName } = orchestratorConfigModule;
|
|
349
|
+
const presets = RATE_LIMIT_PROFILE_PRESETS || FALLBACK_RATE_LIMIT_PROFILE_PRESETS;
|
|
350
|
+
const availableProfiles = Object.keys(presets);
|
|
351
|
+
|
|
352
|
+
const orchestratorConfig = new OrchestratorConfig(workspaceRoot);
|
|
353
|
+
const effectiveConfig = await orchestratorConfig.getConfig();
|
|
354
|
+
const rawConfig = await _readRawOrchestratorConfig(workspaceRoot);
|
|
355
|
+
const activeProfile = typeof resolveRateLimitProfileName === 'function'
|
|
356
|
+
? resolveRateLimitProfileName(effectiveConfig.rateLimitProfile, 'balanced')
|
|
357
|
+
: `${effectiveConfig.rateLimitProfile || 'balanced'}`;
|
|
358
|
+
const preset = presets[activeProfile] || {};
|
|
359
|
+
const overrides = [];
|
|
360
|
+
for (const field of RATE_LIMIT_FIELD_KEYS) {
|
|
361
|
+
if (
|
|
362
|
+
Object.prototype.hasOwnProperty.call(rawConfig, field) &&
|
|
363
|
+
Object.prototype.hasOwnProperty.call(preset, field) &&
|
|
364
|
+
rawConfig[field] !== preset[field]
|
|
365
|
+
) {
|
|
366
|
+
overrides.push(field);
|
|
367
|
+
}
|
|
368
|
+
}
|
|
369
|
+
const effectiveRateLimit = {};
|
|
370
|
+
for (const field of RATE_LIMIT_FIELD_KEYS) {
|
|
371
|
+
effectiveRateLimit[field] = effectiveConfig[field];
|
|
372
|
+
}
|
|
373
|
+
|
|
374
|
+
const payload = {
|
|
375
|
+
profile: activeProfile,
|
|
376
|
+
available_profiles: availableProfiles,
|
|
377
|
+
explicit_overrides: overrides,
|
|
378
|
+
effective: effectiveRateLimit,
|
|
379
|
+
};
|
|
380
|
+
|
|
381
|
+
if (options.json) {
|
|
382
|
+
console.log(JSON.stringify(payload, null, 2));
|
|
383
|
+
return;
|
|
384
|
+
}
|
|
385
|
+
|
|
386
|
+
console.log(chalk.bold(`Active rate-limit profile: ${activeProfile}`));
|
|
387
|
+
if (overrides.length > 0) {
|
|
388
|
+
console.log(chalk.yellow(`Explicit overrides: ${overrides.join(', ')}`));
|
|
389
|
+
} else {
|
|
390
|
+
console.log(chalk.gray('Explicit overrides: none'));
|
|
391
|
+
}
|
|
392
|
+
for (const field of RATE_LIMIT_FIELD_KEYS) {
|
|
393
|
+
console.log(` ${field}: ${effectiveRateLimit[field]}`);
|
|
394
|
+
}
|
|
395
|
+
} catch (err) {
|
|
396
|
+
_errorAndExit(err.message, options.json);
|
|
397
|
+
}
|
|
398
|
+
});
|
|
399
|
+
|
|
400
|
+
profile
|
|
401
|
+
.command('set')
|
|
402
|
+
.description('Set persistent rate-limit profile in .sce/config/orchestrator.json')
|
|
403
|
+
.argument('<profile>', 'Profile name (conservative|balanced|aggressive)')
|
|
404
|
+
.option('--reset-overrides', 'Reset explicit rateLimit* overrides to profile defaults')
|
|
405
|
+
.option('--json', 'Output in JSON format')
|
|
406
|
+
.action(async (profileName, options) => {
|
|
407
|
+
try {
|
|
408
|
+
const workspaceRoot = process.cwd();
|
|
409
|
+
const orchestratorConfigModule = require('../orchestrator/orchestrator-config');
|
|
410
|
+
const {
|
|
411
|
+
OrchestratorConfig,
|
|
412
|
+
RATE_LIMIT_PROFILE_PRESETS,
|
|
413
|
+
resolveRateLimitProfileName,
|
|
414
|
+
buildRateLimitProfileConfig
|
|
415
|
+
} = orchestratorConfigModule;
|
|
416
|
+
const presets = RATE_LIMIT_PROFILE_PRESETS || FALLBACK_RATE_LIMIT_PROFILE_PRESETS;
|
|
417
|
+
const availableProfiles = Object.keys(presets);
|
|
418
|
+
const normalizedInput = `${profileName || ''}`.trim().toLowerCase();
|
|
419
|
+
if (!availableProfiles.includes(normalizedInput)) {
|
|
420
|
+
throw new Error(`profile must be one of: ${availableProfiles.join(', ')}`);
|
|
421
|
+
}
|
|
422
|
+
|
|
423
|
+
const resolvedProfile = typeof resolveRateLimitProfileName === 'function'
|
|
424
|
+
? resolveRateLimitProfileName(normalizedInput, 'balanced')
|
|
425
|
+
: normalizedInput;
|
|
426
|
+
const orchestratorConfig = new OrchestratorConfig(workspaceRoot);
|
|
427
|
+
let updated;
|
|
428
|
+
|
|
429
|
+
if (options.resetOverrides && typeof buildRateLimitProfileConfig === 'function') {
|
|
430
|
+
updated = await orchestratorConfig.updateConfig(buildRateLimitProfileConfig(resolvedProfile));
|
|
431
|
+
} else {
|
|
432
|
+
updated = await orchestratorConfig.updateConfig({ rateLimitProfile: resolvedProfile });
|
|
433
|
+
}
|
|
434
|
+
|
|
435
|
+
const payload = {
|
|
436
|
+
profile: resolvedProfile,
|
|
437
|
+
reset_overrides: !!options.resetOverrides,
|
|
438
|
+
config_file: ORCHESTRATOR_CONFIG_FILE,
|
|
439
|
+
effective: RATE_LIMIT_FIELD_KEYS.reduce((acc, key) => {
|
|
440
|
+
acc[key] = updated[key];
|
|
441
|
+
return acc;
|
|
442
|
+
}, {})
|
|
443
|
+
};
|
|
444
|
+
|
|
445
|
+
if (options.json) {
|
|
446
|
+
console.log(JSON.stringify(payload, null, 2));
|
|
447
|
+
return;
|
|
448
|
+
}
|
|
449
|
+
|
|
450
|
+
console.log(chalk.green(`Set rate-limit profile: ${resolvedProfile}`));
|
|
451
|
+
if (options.resetOverrides) {
|
|
452
|
+
console.log(chalk.gray('Explicit rateLimit* overrides reset to profile defaults.'));
|
|
453
|
+
} else {
|
|
454
|
+
console.log(chalk.gray('Existing explicit rateLimit* overrides (if any) are preserved.'));
|
|
455
|
+
}
|
|
456
|
+
console.log(chalk.gray(`Config updated: ${ORCHESTRATOR_CONFIG_FILE}`));
|
|
457
|
+
} catch (err) {
|
|
458
|
+
_errorAndExit(err.message, options.json);
|
|
459
|
+
}
|
|
460
|
+
});
|
|
259
461
|
}
|
|
260
462
|
|
|
261
463
|
// ── Helpers ──────────────────────────────────────────────────────────
|
|
@@ -303,6 +505,24 @@ async function _writeStatus(workspaceRoot, status) {
|
|
|
303
505
|
await fs.writeJson(statusPath, status, { spaces: 2 });
|
|
304
506
|
}
|
|
305
507
|
|
|
508
|
+
/**
|
|
509
|
+
* Read raw orchestrator config file (without default merge).
|
|
510
|
+
* @param {string} workspaceRoot
|
|
511
|
+
* @returns {Promise<object>}
|
|
512
|
+
*/
|
|
513
|
+
async function _readRawOrchestratorConfig(workspaceRoot) {
|
|
514
|
+
const configPath = path.join(workspaceRoot, ORCHESTRATOR_CONFIG_FILE);
|
|
515
|
+
try {
|
|
516
|
+
const payload = await fs.readJson(configPath);
|
|
517
|
+
if (!payload || typeof payload !== 'object' || Array.isArray(payload)) {
|
|
518
|
+
return {};
|
|
519
|
+
}
|
|
520
|
+
return payload;
|
|
521
|
+
} catch (_err) {
|
|
522
|
+
return {};
|
|
523
|
+
}
|
|
524
|
+
}
|
|
525
|
+
|
|
306
526
|
/**
|
|
307
527
|
* Print a human-readable orchestration result.
|
|
308
528
|
* @param {object} result
|
|
@@ -132,6 +132,10 @@ class OrchestrationEngine extends EventEmitter {
|
|
|
132
132
|
this._rateLimitSignalWindowMs = DEFAULT_RATE_LIMIT_SIGNAL_WINDOW_MS;
|
|
133
133
|
/** @type {number} number of rate-limit signals inside window that triggers escalation */
|
|
134
134
|
this._rateLimitSignalThreshold = DEFAULT_RATE_LIMIT_SIGNAL_THRESHOLD;
|
|
135
|
+
/** @type {number} additional launch hold applied per escalation step */
|
|
136
|
+
this._rateLimitSignalExtraHoldMs = DEFAULT_RATE_LIMIT_SIGNAL_EXTRA_HOLD_MS;
|
|
137
|
+
/** @type {number} minimum dynamic launch budget floor under sustained pressure */
|
|
138
|
+
this._rateLimitDynamicBudgetFloor = DEFAULT_RATE_LIMIT_DYNAMIC_BUDGET_FLOOR;
|
|
135
139
|
/** @type {number[]} timestamps (ms) of recent spec launches for rolling budget accounting */
|
|
136
140
|
this._rateLimitLaunchTimestamps = [];
|
|
137
141
|
/** @type {number} last launch-budget hold telemetry emission timestamp (ms) */
|
|
@@ -172,6 +176,7 @@ class OrchestrationEngine extends EventEmitter {
|
|
|
172
176
|
* @param {string[]} specNames - Specs to orchestrate
|
|
173
177
|
* @param {object} [options]
|
|
174
178
|
* @param {number} [options.maxParallel] - Override max parallel from config
|
|
179
|
+
* @param {object} [options.configOverrides] - Runtime config overrides for this execution only
|
|
175
180
|
* @returns {Promise<object>} OrchestrationResult
|
|
176
181
|
*/
|
|
177
182
|
async start(specNames, options = {}) {
|
|
@@ -235,11 +240,18 @@ class OrchestrationEngine extends EventEmitter {
|
|
|
235
240
|
|
|
236
241
|
// Get config for maxParallel and maxRetries
|
|
237
242
|
const config = await this._orchestratorConfig.getConfig();
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
const
|
|
242
|
-
|
|
243
|
+
const configOverrides = options && typeof options.configOverrides === 'object' && !Array.isArray(options.configOverrides)
|
|
244
|
+
? options.configOverrides
|
|
245
|
+
: null;
|
|
246
|
+
const effectiveConfig = configOverrides
|
|
247
|
+
? { ...config, ...configOverrides }
|
|
248
|
+
: config;
|
|
249
|
+
|
|
250
|
+
this._applyRetryPolicyConfig(effectiveConfig);
|
|
251
|
+
await this._applyCoordinationPolicyConfig(effectiveConfig);
|
|
252
|
+
this._agentWaitTimeoutMs = this._resolveAgentWaitTimeoutMs(effectiveConfig);
|
|
253
|
+
const maxParallel = options.maxParallel || effectiveConfig.maxParallel || 3;
|
|
254
|
+
const maxRetries = effectiveConfig.maxRetries || 2;
|
|
243
255
|
this._initializeAdaptiveParallel(maxParallel);
|
|
244
256
|
|
|
245
257
|
// Step 5: Execute batches (Req 3.4)
|
|
@@ -1063,6 +1075,22 @@ class OrchestrationEngine extends EventEmitter {
|
|
|
1063
1075
|
config && config.rateLimitLaunchBudgetWindowMs,
|
|
1064
1076
|
DEFAULT_RATE_LIMIT_LAUNCH_BUDGET_WINDOW_MS
|
|
1065
1077
|
);
|
|
1078
|
+
this._rateLimitSignalWindowMs = this._toPositiveInteger(
|
|
1079
|
+
config && config.rateLimitSignalWindowMs,
|
|
1080
|
+
DEFAULT_RATE_LIMIT_SIGNAL_WINDOW_MS
|
|
1081
|
+
);
|
|
1082
|
+
this._rateLimitSignalThreshold = this._toPositiveInteger(
|
|
1083
|
+
config && config.rateLimitSignalThreshold,
|
|
1084
|
+
DEFAULT_RATE_LIMIT_SIGNAL_THRESHOLD
|
|
1085
|
+
);
|
|
1086
|
+
this._rateLimitSignalExtraHoldMs = this._toPositiveInteger(
|
|
1087
|
+
config && config.rateLimitSignalExtraHoldMs,
|
|
1088
|
+
DEFAULT_RATE_LIMIT_SIGNAL_EXTRA_HOLD_MS
|
|
1089
|
+
);
|
|
1090
|
+
this._rateLimitDynamicBudgetFloor = this._toPositiveInteger(
|
|
1091
|
+
config && config.rateLimitDynamicBudgetFloor,
|
|
1092
|
+
DEFAULT_RATE_LIMIT_DYNAMIC_BUDGET_FLOOR
|
|
1093
|
+
);
|
|
1066
1094
|
}
|
|
1067
1095
|
|
|
1068
1096
|
/**
|
|
@@ -1570,7 +1598,10 @@ class OrchestrationEngine extends EventEmitter {
|
|
|
1570
1598
|
const escalationUnits = signalCount - threshold + 1;
|
|
1571
1599
|
const extraHoldMs = Math.min(
|
|
1572
1600
|
maxHoldMs,
|
|
1573
|
-
escalationUnits *
|
|
1601
|
+
escalationUnits * this._toPositiveInteger(
|
|
1602
|
+
this._rateLimitSignalExtraHoldMs,
|
|
1603
|
+
DEFAULT_RATE_LIMIT_SIGNAL_EXTRA_HOLD_MS
|
|
1604
|
+
)
|
|
1574
1605
|
);
|
|
1575
1606
|
if (extraHoldMs > 0) {
|
|
1576
1607
|
const currentHoldUntil = this._toNonNegativeInteger(this._rateLimitLaunchHoldUntil, 0);
|
|
@@ -1593,7 +1624,13 @@ class OrchestrationEngine extends EventEmitter {
|
|
|
1593
1624
|
);
|
|
1594
1625
|
const budgetFloor = Math.max(
|
|
1595
1626
|
1,
|
|
1596
|
-
Math.min(
|
|
1627
|
+
Math.min(
|
|
1628
|
+
configuredBudget,
|
|
1629
|
+
this._toPositiveInteger(
|
|
1630
|
+
this._rateLimitDynamicBudgetFloor,
|
|
1631
|
+
DEFAULT_RATE_LIMIT_DYNAMIC_BUDGET_FLOOR
|
|
1632
|
+
)
|
|
1633
|
+
)
|
|
1597
1634
|
);
|
|
1598
1635
|
const nextBudget = Math.max(budgetFloor, Math.floor(currentBudget / 2));
|
|
1599
1636
|
if (nextBudget >= currentBudget) {
|
|
@@ -24,6 +24,7 @@ const KNOWN_KEYS = new Set([
|
|
|
24
24
|
'maxParallel',
|
|
25
25
|
'timeoutSeconds',
|
|
26
26
|
'maxRetries',
|
|
27
|
+
'rateLimitProfile',
|
|
27
28
|
'rateLimitMaxRetries',
|
|
28
29
|
'rateLimitBackoffBaseMs',
|
|
29
30
|
'rateLimitBackoffMaxMs',
|
|
@@ -32,18 +33,85 @@ const KNOWN_KEYS = new Set([
|
|
|
32
33
|
'rateLimitCooldownMs',
|
|
33
34
|
'rateLimitLaunchBudgetPerMinute',
|
|
34
35
|
'rateLimitLaunchBudgetWindowMs',
|
|
36
|
+
'rateLimitSignalWindowMs',
|
|
37
|
+
'rateLimitSignalThreshold',
|
|
38
|
+
'rateLimitSignalExtraHoldMs',
|
|
39
|
+
'rateLimitDynamicBudgetFloor',
|
|
35
40
|
'apiKeyEnvVar',
|
|
36
41
|
'bootstrapTemplate',
|
|
37
42
|
'codexArgs',
|
|
38
43
|
'codexCommand',
|
|
39
44
|
]);
|
|
40
45
|
|
|
46
|
+
const RATE_LIMIT_PROFILE_PRESETS = Object.freeze({
|
|
47
|
+
conservative: Object.freeze({
|
|
48
|
+
rateLimitMaxRetries: 10,
|
|
49
|
+
rateLimitBackoffBaseMs: 2200,
|
|
50
|
+
rateLimitBackoffMaxMs: 90000,
|
|
51
|
+
rateLimitAdaptiveParallel: true,
|
|
52
|
+
rateLimitParallelFloor: 1,
|
|
53
|
+
rateLimitCooldownMs: 60000,
|
|
54
|
+
rateLimitLaunchBudgetPerMinute: 4,
|
|
55
|
+
rateLimitLaunchBudgetWindowMs: 60000,
|
|
56
|
+
rateLimitSignalWindowMs: 45000,
|
|
57
|
+
rateLimitSignalThreshold: 2,
|
|
58
|
+
rateLimitSignalExtraHoldMs: 5000,
|
|
59
|
+
rateLimitDynamicBudgetFloor: 1,
|
|
60
|
+
}),
|
|
61
|
+
balanced: Object.freeze({
|
|
62
|
+
rateLimitMaxRetries: 8,
|
|
63
|
+
rateLimitBackoffBaseMs: 1500,
|
|
64
|
+
rateLimitBackoffMaxMs: 60000,
|
|
65
|
+
rateLimitAdaptiveParallel: true,
|
|
66
|
+
rateLimitParallelFloor: 1,
|
|
67
|
+
rateLimitCooldownMs: 45000,
|
|
68
|
+
rateLimitLaunchBudgetPerMinute: 8,
|
|
69
|
+
rateLimitLaunchBudgetWindowMs: 60000,
|
|
70
|
+
rateLimitSignalWindowMs: 30000,
|
|
71
|
+
rateLimitSignalThreshold: 3,
|
|
72
|
+
rateLimitSignalExtraHoldMs: 3000,
|
|
73
|
+
rateLimitDynamicBudgetFloor: 1,
|
|
74
|
+
}),
|
|
75
|
+
aggressive: Object.freeze({
|
|
76
|
+
rateLimitMaxRetries: 6,
|
|
77
|
+
rateLimitBackoffBaseMs: 1000,
|
|
78
|
+
rateLimitBackoffMaxMs: 30000,
|
|
79
|
+
rateLimitAdaptiveParallel: true,
|
|
80
|
+
rateLimitParallelFloor: 1,
|
|
81
|
+
rateLimitCooldownMs: 20000,
|
|
82
|
+
rateLimitLaunchBudgetPerMinute: 16,
|
|
83
|
+
rateLimitLaunchBudgetWindowMs: 60000,
|
|
84
|
+
rateLimitSignalWindowMs: 20000,
|
|
85
|
+
rateLimitSignalThreshold: 4,
|
|
86
|
+
rateLimitSignalExtraHoldMs: 2000,
|
|
87
|
+
rateLimitDynamicBudgetFloor: 2,
|
|
88
|
+
}),
|
|
89
|
+
});
|
|
90
|
+
|
|
91
|
+
function resolveRateLimitProfileName(profileName, fallback = 'balanced') {
|
|
92
|
+
const normalized = `${profileName || ''}`.trim().toLowerCase();
|
|
93
|
+
if (normalized && Object.prototype.hasOwnProperty.call(RATE_LIMIT_PROFILE_PRESETS, normalized)) {
|
|
94
|
+
return normalized;
|
|
95
|
+
}
|
|
96
|
+
return fallback;
|
|
97
|
+
}
|
|
98
|
+
|
|
99
|
+
function buildRateLimitProfileConfig(profileName) {
|
|
100
|
+
const resolvedProfile = resolveRateLimitProfileName(profileName, 'balanced');
|
|
101
|
+
const preset = RATE_LIMIT_PROFILE_PRESETS[resolvedProfile] || RATE_LIMIT_PROFILE_PRESETS.balanced;
|
|
102
|
+
return {
|
|
103
|
+
...preset,
|
|
104
|
+
rateLimitProfile: resolvedProfile,
|
|
105
|
+
};
|
|
106
|
+
}
|
|
107
|
+
|
|
41
108
|
/** @type {import('./orchestrator-config').OrchestratorConfigData} */
|
|
42
109
|
const DEFAULT_CONFIG = Object.freeze({
|
|
43
110
|
agentBackend: 'codex',
|
|
44
111
|
maxParallel: 3,
|
|
45
112
|
timeoutSeconds: 600,
|
|
46
113
|
maxRetries: 2,
|
|
114
|
+
rateLimitProfile: 'balanced',
|
|
47
115
|
rateLimitMaxRetries: 8,
|
|
48
116
|
rateLimitBackoffBaseMs: 1500,
|
|
49
117
|
rateLimitBackoffMaxMs: 60000,
|
|
@@ -52,6 +120,10 @@ const DEFAULT_CONFIG = Object.freeze({
|
|
|
52
120
|
rateLimitCooldownMs: 45000,
|
|
53
121
|
rateLimitLaunchBudgetPerMinute: 8,
|
|
54
122
|
rateLimitLaunchBudgetWindowMs: 60000,
|
|
123
|
+
rateLimitSignalWindowMs: 30000,
|
|
124
|
+
rateLimitSignalThreshold: 3,
|
|
125
|
+
rateLimitSignalExtraHoldMs: 3000,
|
|
126
|
+
rateLimitDynamicBudgetFloor: 1,
|
|
55
127
|
apiKeyEnvVar: 'CODEX_API_KEY',
|
|
56
128
|
bootstrapTemplate: null,
|
|
57
129
|
codexArgs: [],
|
|
@@ -142,7 +214,17 @@ class OrchestratorConfig {
|
|
|
142
214
|
*/
|
|
143
215
|
_mergeWithDefaults(data) {
|
|
144
216
|
const filtered = this._filterKnownKeys(data);
|
|
145
|
-
|
|
217
|
+
const rateLimitProfile = resolveRateLimitProfileName(
|
|
218
|
+
filtered.rateLimitProfile,
|
|
219
|
+
DEFAULT_CONFIG.rateLimitProfile
|
|
220
|
+
);
|
|
221
|
+
const profileDefaults = buildRateLimitProfileConfig(rateLimitProfile);
|
|
222
|
+
return {
|
|
223
|
+
...DEFAULT_CONFIG,
|
|
224
|
+
...profileDefaults,
|
|
225
|
+
...filtered,
|
|
226
|
+
rateLimitProfile,
|
|
227
|
+
};
|
|
146
228
|
}
|
|
147
229
|
|
|
148
230
|
/**
|
|
@@ -170,4 +252,11 @@ class OrchestratorConfig {
|
|
|
170
252
|
}
|
|
171
253
|
}
|
|
172
254
|
|
|
173
|
-
module.exports = {
|
|
255
|
+
module.exports = {
|
|
256
|
+
OrchestratorConfig,
|
|
257
|
+
DEFAULT_CONFIG,
|
|
258
|
+
KNOWN_KEYS,
|
|
259
|
+
RATE_LIMIT_PROFILE_PRESETS,
|
|
260
|
+
resolveRateLimitProfileName,
|
|
261
|
+
buildRateLimitProfileConfig,
|
|
262
|
+
};
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "scene-capability-engine",
|
|
3
|
-
"version": "3.3.
|
|
3
|
+
"version": "3.3.17",
|
|
4
4
|
"description": "SCE (Scene Capability Engine) - A CLI tool and npm package for spec-driven development with AI coding assistants.",
|
|
5
5
|
"main": "index.js",
|
|
6
6
|
"bin": {
|
|
@@ -30,6 +30,7 @@
|
|
|
30
30
|
"test:unit": "npx jest tests/unit",
|
|
31
31
|
"test:integration": "npx jest tests/integration",
|
|
32
32
|
"test:properties": "npx jest tests/properties",
|
|
33
|
+
"test:orchestrator-429": "npx jest tests/orchestrator/orchestration-engine.test.js tests/orchestrator/orchestrate-command.status-events.test.js --runInBand",
|
|
33
34
|
"test:handles": "npx jest --config=jest.config.js --runInBand --detectOpenHandles",
|
|
34
35
|
"test:interactive-loop-smoke": "node scripts/interactive-loop-smoke.js --json",
|
|
35
36
|
"test:interactive-flow-smoke": "node scripts/interactive-flow-smoke.js --json",
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
{
|
|
2
|
+
"agentBackend": "codex",
|
|
3
|
+
"maxParallel": 3,
|
|
4
|
+
"timeoutSeconds": 900,
|
|
5
|
+
"maxRetries": 2,
|
|
6
|
+
"rateLimitProfile": "balanced",
|
|
7
|
+
"rateLimitMaxRetries": 8,
|
|
8
|
+
"rateLimitBackoffBaseMs": 1500,
|
|
9
|
+
"rateLimitBackoffMaxMs": 60000,
|
|
10
|
+
"rateLimitAdaptiveParallel": true,
|
|
11
|
+
"rateLimitParallelFloor": 1,
|
|
12
|
+
"rateLimitCooldownMs": 45000,
|
|
13
|
+
"rateLimitLaunchBudgetPerMinute": 8,
|
|
14
|
+
"rateLimitLaunchBudgetWindowMs": 60000,
|
|
15
|
+
"rateLimitSignalWindowMs": 30000,
|
|
16
|
+
"rateLimitSignalThreshold": 3,
|
|
17
|
+
"rateLimitSignalExtraHoldMs": 3000,
|
|
18
|
+
"rateLimitDynamicBudgetFloor": 1,
|
|
19
|
+
"apiKeyEnvVar": "CODEX_API_KEY",
|
|
20
|
+
"codexArgs": [
|
|
21
|
+
"--skip-git-repo-check"
|
|
22
|
+
],
|
|
23
|
+
"codexCommand": "npx @openai/codex"
|
|
24
|
+
}
|