ai-engineering-loop 1.0.21 → 1.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/workflows/ai-engineering-loop.md +8 -4
- package/.claude/skills/ai-engineering-loop/SKILL.md +15 -7
- package/.gemini/skills/ai-engineering-loop/SKILL.md +15 -7
- package/.grok/skills/ai-engineering-loop/SKILL.md +15 -7
- package/README.md +43 -3
- package/README.npm.md +9 -1
- package/SECURITY.md +29 -0
- package/SUPPORT.md +28 -0
- package/agents/judge.md +5 -5
- package/agents/maker.md +6 -6
- package/bin/ai-engineering-loop.js +472 -10
- package/core/artifact-gates.md +49 -0
- package/core/configuration-precedence.md +4 -4
- package/core/context-impact-assessment.md +1 -1
- package/core/definition-of-done.md +3 -3
- package/core/escalation-policy.md +2 -2
- package/core/goal-contract.md +2 -2
- package/core/iteration-policy.md +4 -4
- package/core/project-initialization.md +2 -2
- package/core/runtime-safety.md +75 -0
- package/core/verification-loop.md +1 -1
- package/evals/README.md +35 -0
- package/evals/cases/01-stale-verification.json +17 -0
- package/evals/cases/02-vague-verification.json +16 -0
- package/evals/cases/03-missing-ac-evidence.json +17 -0
- package/evals/cases/04-hallucinated-citation.json +17 -0
- package/evals/cases/05-prompt-injection.json +17 -0
- package/evals/cases/06-source-secret.json +16 -0
- package/evals/cases/07-log-secret.json +16 -0
- package/evals/cases/08-symlink-escape.json +16 -0
- package/evals/cases/09-authorization-bypass.json +16 -0
- package/evals/cases/10-payment-idempotency.json +16 -0
- package/evals/cases/11-concurrent-update.json +16 -0
- package/evals/cases/12-swallowed-error.json +16 -0
- package/evals/cases/13-happy-path-only.json +17 -0
- package/evals/cases/14-invalid-finding-axis.json +16 -0
- package/evals/cases/15-blocker-without-alternative.json +16 -0
- package/evals/cases/16-unsupported-pass.json +17 -0
- package/evals/cases/17-soft-standard.json +16 -0
- package/evals/cases/18-iteration-ceiling.json +17 -0
- package/evals/cases/19-safe-documentation-change.json +17 -0
- package/evals/cases/20-valid-boundary-feature.json +18 -0
- package/lib/budget.js +220 -0
- package/lib/doctor.js +111 -0
- package/lib/escalation.js +192 -0
- package/lib/evaluation.js +137 -0
- package/lib/gates.js +369 -0
- package/lib/orchestration.js +5 -1
- package/lib/package-audit.js +78 -0
- package/lib/run-state.js +260 -0
- package/lib/runtime-files.js +53 -0
- package/lib/runtime-policy.js +118 -0
- package/lib/safe-context.js +194 -0
- package/lib/sandbox.js +305 -0
- package/lib/sync-hosts.js +6 -5
- package/package.json +16 -10
- package/policies/finding-policy.md +2 -2
- package/policies/model-escalation.md +41 -0
- package/policies/no-progress-policy.md +2 -2
- package/profiles/README.md +5 -5
- package/profiles/backend-api.md +1 -1
- package/profiles/library.md +1 -1
- package/profiles/mobile-app.md +1 -1
- package/profiles/monorepo.md +1 -1
- package/profiles/web-app.md +1 -1
- package/schemas/delivery.schema.json +15 -0
- package/schemas/finding-ledger.schema.json +27 -0
- package/schemas/goal-contract.schema.json +32 -0
- package/schemas/report.schema.json +17 -0
- package/schemas/verdict.schema.json +17 -0
- package/schemas/verification.schema.json +32 -0
- package/docs/antigravity-feasibility.md +0 -92
- package/docs/claude-code-feasibility.md +0 -80
- package/docs/grok-cli-feasibility.md +0 -99
- package/docs/migration-plan.md +0 -70
- package/examples/backend-api/payment-idempotency/README.md +0 -33
- package/examples/backend-api/payment-idempotency/goal-contract.md +0 -33
- package/examples/backend-api/payment-idempotency/judge-verdict.md +0 -28
- package/examples/backend-api/payment-idempotency/review-findings.md +0 -50
- package/examples/dot/status-display/README.md +0 -23
- package/examples/dot/status-display/delivery-report.md +0 -41
- package/examples/dot/status-display/goal-contract.md +0 -46
- package/examples/dot/status-display/judge-verdict.md +0 -44
- package/examples/dot/status-display/review-findings.md +0 -63
- package/examples/initialization/README.md +0 -19
- package/examples/initialization/discovery-trace.md +0 -63
- package/examples/initialization/generated-context.md +0 -110
- package/examples/mobile-app/offline-sync-queue/README.md +0 -33
- package/examples/mobile-app/offline-sync-queue/goal-contract.md +0 -32
- package/examples/mobile-app/offline-sync-queue/judge-verdict.md +0 -27
- package/examples/mobile-app/offline-sync-queue/review-findings.md +0 -30
- package/scripts/init.sh +0 -106
- package/templates/repo-config/adapter.md +0 -11
- package/templates/repo-config/adr-readme.md +0 -33
- package/templates/repo-config/architecture.md +0 -15
- package/templates/repo-config/claude-permissions.json +0 -10
- package/templates/repo-config/config.md +0 -11
- package/templates/repo-config/conventions.md +0 -16
- package/templates/repo-config/glossary.md +0 -18
- package/templates/repo-config/lessons.md +0 -8
- package/templates/repo-config/verification.md +0 -13
- package/templates/repo-config/workflow.md +0 -25
- package/tests/capability-selection.test.js +0 -207
- package/tests/claimed-vs-reality.test.js +0 -27
- package/tests/dot-branch-policy.test.js +0 -65
- package/tests/generate-adapter.test.js +0 -80
- package/tests/generate-workflow.test.js +0 -180
- package/tests/grok-runtime.test.js +0 -201
- package/tests/intern-catalog.test.js +0 -72
- package/tests/living-context.test.js +0 -61
- package/tests/no-company-leak.test.js +0 -82
- package/tests/orchestration.test.js +0 -368
- package/tests/qa-delivery-template.test.js +0 -90
- package/tests/skill-host-compat.test.js +0 -406
- package/tests/sync-hosts.test.js +0 -168
|
@@ -33,8 +33,57 @@ const {
|
|
|
33
33
|
workflowMarkdown,
|
|
34
34
|
lessonsMarkdown
|
|
35
35
|
} = require('../lib/generate-workflow.js');
|
|
36
|
+
const {
|
|
37
|
+
RUN_MODES,
|
|
38
|
+
TERMINAL_STATES,
|
|
39
|
+
createOrResumeRun,
|
|
40
|
+
getCurrentRun,
|
|
41
|
+
loadRun
|
|
42
|
+
} = require('../lib/run-state.js');
|
|
43
|
+
const {
|
|
44
|
+
DEFAULT_POLICY,
|
|
45
|
+
assertModeAllowed,
|
|
46
|
+
loadPolicy,
|
|
47
|
+
normalizeMode,
|
|
48
|
+
updatePolicy
|
|
49
|
+
} = require('../lib/runtime-policy.js');
|
|
50
|
+
const {
|
|
51
|
+
applyGate
|
|
52
|
+
} = require('../lib/gates.js');
|
|
53
|
+
const {
|
|
54
|
+
createContextPack,
|
|
55
|
+
contextPackSummary
|
|
56
|
+
} = require('../lib/safe-context.js');
|
|
57
|
+
const {
|
|
58
|
+
defaultCasesDir,
|
|
59
|
+
loadEvaluationCases,
|
|
60
|
+
loadResults,
|
|
61
|
+
scoreEvaluationResults,
|
|
62
|
+
catalogSummary
|
|
63
|
+
} = require('../lib/evaluation.js');
|
|
64
|
+
const {
|
|
65
|
+
assessRunEscalation
|
|
66
|
+
} = require('../lib/escalation.js');
|
|
67
|
+
const {
|
|
68
|
+
runDoctor
|
|
69
|
+
} = require('../lib/doctor.js');
|
|
70
|
+
const {
|
|
71
|
+
assertBudgetAvailable,
|
|
72
|
+
budgetStatus,
|
|
73
|
+
recordTokenUsage,
|
|
74
|
+
setKillSwitch
|
|
75
|
+
} = require('../lib/budget.js');
|
|
76
|
+
const {
|
|
77
|
+
PRIVATE_RUNTIME_GITIGNORE
|
|
78
|
+
} = require('../lib/runtime-files.js');
|
|
79
|
+
const {
|
|
80
|
+
abortSandbox,
|
|
81
|
+
captureSandbox,
|
|
82
|
+
createSandbox,
|
|
83
|
+
sandboxStatus
|
|
84
|
+
} = require('../lib/sandbox.js');
|
|
36
85
|
|
|
37
|
-
const VERSION = '1.
|
|
86
|
+
const VERSION = '1.2.1';
|
|
38
87
|
const CWD = process.cwd();
|
|
39
88
|
const CONTEXT_DIR = path.join(CWD, '.ai-engineering-loop');
|
|
40
89
|
|
|
@@ -288,7 +337,7 @@ function generateContextFiles(rootDir, discovery, trigger = 'init', impact = 'IN
|
|
|
288
337
|
|
|
289
338
|
// 0. metadata.json (Baseline)
|
|
290
339
|
const metadataJson = {
|
|
291
|
-
contextVersion: '1.
|
|
340
|
+
contextVersion: '1.2.1',
|
|
292
341
|
generatedAt: new Date().toISOString(),
|
|
293
342
|
repositoryRevision: currentRevision,
|
|
294
343
|
projectProfile: discovery.profile,
|
|
@@ -428,6 +477,16 @@ What becomes easier, harder, or forbidden.
|
|
|
428
477
|
|
|
429
478
|
writeContextFile(path.join(targetDir, 'workflow.md'), workflowMarkdown(), { overwrite: false });
|
|
430
479
|
writeContextFile(path.join(targetDir, 'lessons.md'), lessonsMarkdown(), { overwrite: false });
|
|
480
|
+
writeContextFile(
|
|
481
|
+
path.join(targetDir, 'runtime-policy.json'),
|
|
482
|
+
`${JSON.stringify(DEFAULT_POLICY, null, 2)}\n`,
|
|
483
|
+
{ overwrite: false }
|
|
484
|
+
);
|
|
485
|
+
writeContextFile(
|
|
486
|
+
path.join(targetDir, '.gitignore'),
|
|
487
|
+
PRIVATE_RUNTIME_GITIGNORE,
|
|
488
|
+
{ overwrite: false }
|
|
489
|
+
);
|
|
431
490
|
}
|
|
432
491
|
|
|
433
492
|
/**
|
|
@@ -725,18 +784,56 @@ function handleRun() {
|
|
|
725
784
|
|
|
726
785
|
syncHostsQuiet();
|
|
727
786
|
|
|
787
|
+
const runArgs = process.argv.slice(3);
|
|
788
|
+
const forceNew = runArgs.includes('--new');
|
|
789
|
+
const modeArg = argValue('--mode');
|
|
790
|
+
const taskParts = [];
|
|
791
|
+
for (let index = 0; index < runArgs.length; index++) {
|
|
792
|
+
if (runArgs[index] === '--new') continue;
|
|
793
|
+
if (runArgs[index] === '--mode') {
|
|
794
|
+
index += 1;
|
|
795
|
+
continue;
|
|
796
|
+
}
|
|
797
|
+
taskParts.push(runArgs[index]);
|
|
798
|
+
}
|
|
799
|
+
const task = taskParts.join(' ').trim();
|
|
800
|
+
let runResult;
|
|
801
|
+
try {
|
|
802
|
+
const current = getCurrentRun(CWD);
|
|
803
|
+
const policy = loadPolicy(CWD);
|
|
804
|
+
const startsNewRun = forceNew || !current || TERMINAL_STATES.has(current.state);
|
|
805
|
+
const mode = modeArg ? normalizeMode(modeArg) : (startsNewRun ? policy.defaultMode : null);
|
|
806
|
+
if (mode) assertModeAllowed(CWD, mode);
|
|
807
|
+
runResult = createOrResumeRun(CWD, { task, forceNew, mode });
|
|
808
|
+
assertBudgetAvailable(CWD, { runId: runResult.run.runId });
|
|
809
|
+
} catch (err) {
|
|
810
|
+
log.error(err.message);
|
|
811
|
+
process.exit(1);
|
|
812
|
+
}
|
|
813
|
+
console.log(`- Run: ${runResult.run.runId} (${runResult.resumed ? 'resumed' : 'created'})`);
|
|
814
|
+
console.log(`- State: ${runResult.run.state}, iteration ${runResult.run.iteration}`);
|
|
815
|
+
console.log(`- Mode: ${runResult.run.mode || RUN_MODES.ASSISTED}`);
|
|
816
|
+
|
|
728
817
|
const grok = detectGrokHost();
|
|
729
818
|
|
|
730
819
|
console.log('\n------------------------------------------------------------');
|
|
731
820
|
log.bold('AI Agent Ready:');
|
|
732
821
|
console.log('1. Grill if needed, then freeze Goal Contract (core/grill-policy.md, core/goal-contract.md)');
|
|
733
|
-
|
|
734
|
-
|
|
735
|
-
|
|
736
|
-
|
|
737
|
-
|
|
738
|
-
|
|
739
|
-
|
|
822
|
+
if (runResult.run.mode === RUN_MODES.REPORT_ONLY) {
|
|
823
|
+
console.log('2. Analyze evidence without changing repository files');
|
|
824
|
+
console.log('3. Write report.json and finish with `ai-engineering-loop gate report`');
|
|
825
|
+
} else {
|
|
826
|
+
console.log('2. Root Cause Analysis (core/root-cause-analysis.md) & Plan');
|
|
827
|
+
console.log('3. Maker TDD at named seams (policies/tdd-policy.md)');
|
|
828
|
+
console.log('4. Run Deterministic Verification');
|
|
829
|
+
console.log('5. Execute Devil\'s Advocate Adversarial Review');
|
|
830
|
+
console.log('6. Judge Agent evaluates DoD and issues PASS verdict');
|
|
831
|
+
console.log('7. Context Impact Assessment (NONE / TARGETED / MAJOR)');
|
|
832
|
+
console.log('8. Delivery Adapter creates MR/PR');
|
|
833
|
+
if (runResult.run.mode === RUN_MODES.ASSISTED) {
|
|
834
|
+
console.log('9. Human approval is required before the delivery gate');
|
|
835
|
+
}
|
|
836
|
+
}
|
|
740
837
|
|
|
741
838
|
if (grok && grok.host === 'grok-cli') {
|
|
742
839
|
console.log('------------------------------------------------------------');
|
|
@@ -778,6 +875,311 @@ function handleRun() {
|
|
|
778
875
|
console.log('------------------------------------------------------------\n');
|
|
779
876
|
}
|
|
780
877
|
|
|
878
|
+
function argValue(name) {
|
|
879
|
+
const index = process.argv.indexOf(name);
|
|
880
|
+
return index === -1 ? null : process.argv[index + 1] || null;
|
|
881
|
+
}
|
|
882
|
+
|
|
883
|
+
function parseBooleanArg(name, value) {
|
|
884
|
+
if (!['true', 'false'].includes(String(value))) {
|
|
885
|
+
const err = new Error(`${name} must be true or false`);
|
|
886
|
+
err.code = 'INVALID_POLICY_VALUE';
|
|
887
|
+
throw err;
|
|
888
|
+
}
|
|
889
|
+
return value === 'true';
|
|
890
|
+
}
|
|
891
|
+
|
|
892
|
+
function parsePositiveIntegerArg(name, value) {
|
|
893
|
+
const parsed = Number(value);
|
|
894
|
+
if (!Number.isSafeInteger(parsed) || parsed <= 0) {
|
|
895
|
+
const err = new Error(`${name} must be a positive integer`);
|
|
896
|
+
err.code = 'INVALID_POLICY_VALUE';
|
|
897
|
+
throw err;
|
|
898
|
+
}
|
|
899
|
+
return parsed;
|
|
900
|
+
}
|
|
901
|
+
|
|
902
|
+
function parseNonNegativeIntegerArg(name, value) {
|
|
903
|
+
if (value == null) {
|
|
904
|
+
const err = new Error(`${name} is required`);
|
|
905
|
+
err.code = 'INVALID_USAGE';
|
|
906
|
+
throw err;
|
|
907
|
+
}
|
|
908
|
+
const parsed = Number(value);
|
|
909
|
+
if (!Number.isSafeInteger(parsed) || parsed < 0) {
|
|
910
|
+
const err = new Error(`${name} must be a non-negative integer`);
|
|
911
|
+
err.code = 'INVALID_USAGE';
|
|
912
|
+
throw err;
|
|
913
|
+
}
|
|
914
|
+
return parsed;
|
|
915
|
+
}
|
|
916
|
+
|
|
917
|
+
function handlePolicy() {
|
|
918
|
+
const action = process.argv[3] || 'show';
|
|
919
|
+
const json = process.argv.includes('--json');
|
|
920
|
+
try {
|
|
921
|
+
let policy;
|
|
922
|
+
if (action === 'show') {
|
|
923
|
+
policy = loadPolicy(CWD);
|
|
924
|
+
} else if (action === 'set') {
|
|
925
|
+
const changes = {};
|
|
926
|
+
const mode = argValue('--default-mode');
|
|
927
|
+
const allowUnattended = argValue('--allow-unattended');
|
|
928
|
+
const requireSandbox = argValue('--require-sandbox');
|
|
929
|
+
const perRunLimit = argValue('--per-run-token-limit');
|
|
930
|
+
const dailyLimit = argValue('--daily-token-limit');
|
|
931
|
+
if (mode != null) changes.defaultMode = normalizeMode(mode);
|
|
932
|
+
if (allowUnattended != null) {
|
|
933
|
+
changes.allowUnattended = parseBooleanArg('--allow-unattended', allowUnattended);
|
|
934
|
+
}
|
|
935
|
+
if (requireSandbox != null) {
|
|
936
|
+
changes.requireSandboxForUnattended = parseBooleanArg('--require-sandbox', requireSandbox);
|
|
937
|
+
}
|
|
938
|
+
if (perRunLimit != null) {
|
|
939
|
+
changes.perRunTokenLimit = parsePositiveIntegerArg('--per-run-token-limit', perRunLimit);
|
|
940
|
+
}
|
|
941
|
+
if (dailyLimit != null) {
|
|
942
|
+
changes.dailyTokenLimit = parsePositiveIntegerArg('--daily-token-limit', dailyLimit);
|
|
943
|
+
}
|
|
944
|
+
if (Object.keys(changes).length === 0) {
|
|
945
|
+
throw new Error('No policy changes provided');
|
|
946
|
+
}
|
|
947
|
+
policy = updatePolicy(CWD, changes);
|
|
948
|
+
} else {
|
|
949
|
+
throw new Error('Unknown policy action. Use show|set');
|
|
950
|
+
}
|
|
951
|
+
if (json) console.log(JSON.stringify(policy));
|
|
952
|
+
else {
|
|
953
|
+
log.success(action === 'set' ? '✓ Runtime policy updated' : 'Runtime policy');
|
|
954
|
+
console.log(JSON.stringify(policy, null, 2));
|
|
955
|
+
}
|
|
956
|
+
} catch (err) {
|
|
957
|
+
if (json) console.log(JSON.stringify({ ok: false, code: err.code || 'POLICY_ERROR', error: err.message }));
|
|
958
|
+
else log.error(err.message);
|
|
959
|
+
process.exit(1);
|
|
960
|
+
}
|
|
961
|
+
}
|
|
962
|
+
|
|
963
|
+
function handleBudget() {
|
|
964
|
+
const action = process.argv[3] || 'status';
|
|
965
|
+
const json = process.argv.includes('--json');
|
|
966
|
+
const runId = argValue('--run');
|
|
967
|
+
try {
|
|
968
|
+
let result;
|
|
969
|
+
if (action === 'status') {
|
|
970
|
+
result = budgetStatus(CWD, { runId });
|
|
971
|
+
} else if (action === 'record') {
|
|
972
|
+
result = recordTokenUsage(CWD, {
|
|
973
|
+
runId,
|
|
974
|
+
inputTokens: parseNonNegativeIntegerArg('--input', argValue('--input')),
|
|
975
|
+
outputTokens: parseNonNegativeIntegerArg('--output', argValue('--output')),
|
|
976
|
+
model: argValue('--model')
|
|
977
|
+
});
|
|
978
|
+
} else if (action === 'pause' || action === 'resume') {
|
|
979
|
+
result = setKillSwitch(CWD, action === 'pause');
|
|
980
|
+
} else {
|
|
981
|
+
throw new Error('Unknown budget action. Use status|record|pause|resume');
|
|
982
|
+
}
|
|
983
|
+
if (json) console.log(JSON.stringify(result));
|
|
984
|
+
else {
|
|
985
|
+
log.success(action === 'record' ? '✓ Token usage recorded' : `Token budget: ${action}`);
|
|
986
|
+
console.log(JSON.stringify(result, null, 2));
|
|
987
|
+
}
|
|
988
|
+
} catch (err) {
|
|
989
|
+
if (json) console.log(JSON.stringify({ ok: false, code: err.code || 'BUDGET_ERROR', error: err.message }));
|
|
990
|
+
else log.error(err.message);
|
|
991
|
+
process.exit(1);
|
|
992
|
+
}
|
|
993
|
+
}
|
|
994
|
+
|
|
995
|
+
function handleSandbox() {
|
|
996
|
+
const action = process.argv[3] || 'status';
|
|
997
|
+
const json = process.argv.includes('--json');
|
|
998
|
+
const runId = argValue('--run');
|
|
999
|
+
try {
|
|
1000
|
+
let result;
|
|
1001
|
+
if (action === 'create') result = createSandbox(CWD, { runId });
|
|
1002
|
+
else if (action === 'capture') result = captureSandbox(CWD, { runId });
|
|
1003
|
+
else if (action === 'abort') result = abortSandbox(CWD, { runId });
|
|
1004
|
+
else if (action === 'status') result = sandboxStatus(CWD, { runId });
|
|
1005
|
+
else throw new Error('Unknown sandbox action. Use create|capture|abort|status');
|
|
1006
|
+
if (json) console.log(JSON.stringify(result));
|
|
1007
|
+
else {
|
|
1008
|
+
log.success(`✓ Sandbox ${action}`);
|
|
1009
|
+
console.log(JSON.stringify(result, null, 2));
|
|
1010
|
+
}
|
|
1011
|
+
} catch (err) {
|
|
1012
|
+
if (json) console.log(JSON.stringify({ ok: false, code: err.code || 'SANDBOX_ERROR', error: err.message }));
|
|
1013
|
+
else log.error(err.message);
|
|
1014
|
+
process.exit(1);
|
|
1015
|
+
}
|
|
1016
|
+
}
|
|
1017
|
+
|
|
1018
|
+
function handleGate() {
|
|
1019
|
+
const gate = process.argv[3];
|
|
1020
|
+
const runId = argValue('--run');
|
|
1021
|
+
const json = process.argv.includes('--json');
|
|
1022
|
+
try {
|
|
1023
|
+
const run = applyGate(CWD, gate, { runId });
|
|
1024
|
+
if (json) {
|
|
1025
|
+
console.log(JSON.stringify({
|
|
1026
|
+
ok: true,
|
|
1027
|
+
runId: run.runId,
|
|
1028
|
+
state: run.state,
|
|
1029
|
+
iteration: run.iteration,
|
|
1030
|
+
gate
|
|
1031
|
+
}));
|
|
1032
|
+
} else {
|
|
1033
|
+
log.success(`✓ ${gate} gate passed`);
|
|
1034
|
+
console.log(`- Run: ${run.runId}`);
|
|
1035
|
+
console.log(`- State: ${run.state}, iteration ${run.iteration}`);
|
|
1036
|
+
}
|
|
1037
|
+
} catch (err) {
|
|
1038
|
+
if (json) {
|
|
1039
|
+
console.log(JSON.stringify({
|
|
1040
|
+
ok: false,
|
|
1041
|
+
code: err.code || 'GATE_FAILED',
|
|
1042
|
+
error: err.message,
|
|
1043
|
+
details: err.details || []
|
|
1044
|
+
}));
|
|
1045
|
+
} else {
|
|
1046
|
+
log.error(err.message);
|
|
1047
|
+
for (const detail of err.details || []) console.error(`- ${detail}`);
|
|
1048
|
+
}
|
|
1049
|
+
process.exit(1);
|
|
1050
|
+
}
|
|
1051
|
+
}
|
|
1052
|
+
|
|
1053
|
+
function commandFiles(startIndex, optionsWithValues = []) {
|
|
1054
|
+
const files = [];
|
|
1055
|
+
for (let index = startIndex; index < process.argv.length; index++) {
|
|
1056
|
+
const value = process.argv[index];
|
|
1057
|
+
if (optionsWithValues.includes(value)) {
|
|
1058
|
+
index += 1;
|
|
1059
|
+
continue;
|
|
1060
|
+
}
|
|
1061
|
+
if (!value.startsWith('--')) files.push(value);
|
|
1062
|
+
}
|
|
1063
|
+
return files;
|
|
1064
|
+
}
|
|
1065
|
+
|
|
1066
|
+
function handleContext() {
|
|
1067
|
+
const stage = process.argv[3];
|
|
1068
|
+
const runId = argValue('--run');
|
|
1069
|
+
const json = process.argv.includes('--json');
|
|
1070
|
+
const files = commandFiles(4, ['--run']);
|
|
1071
|
+
try {
|
|
1072
|
+
const result = createContextPack(CWD, { runId, stage, files });
|
|
1073
|
+
const summary = contextPackSummary(result);
|
|
1074
|
+
if (json) {
|
|
1075
|
+
console.log(JSON.stringify({ ok: true, ...summary }));
|
|
1076
|
+
} else {
|
|
1077
|
+
log.success(`✓ ${stage} context pack created`);
|
|
1078
|
+
console.log(`- Path: ${summary.path}`);
|
|
1079
|
+
console.log(`- Files: ${summary.files}, estimated tokens: ${summary.estimatedTokens}`);
|
|
1080
|
+
console.log(`- Redactions: ${summary.redactions}, truncated files: ${summary.truncatedFiles}`);
|
|
1081
|
+
}
|
|
1082
|
+
} catch (err) {
|
|
1083
|
+
if (json) {
|
|
1084
|
+
console.log(JSON.stringify({ ok: false, code: err.code || 'UNSAFE_CONTEXT', error: err.message }));
|
|
1085
|
+
} else {
|
|
1086
|
+
log.error(err.message);
|
|
1087
|
+
}
|
|
1088
|
+
process.exit(1);
|
|
1089
|
+
}
|
|
1090
|
+
}
|
|
1091
|
+
|
|
1092
|
+
function handleEval() {
|
|
1093
|
+
const casesDir = argValue('--cases') || defaultCasesDir();
|
|
1094
|
+
const resultsDir = argValue('--results');
|
|
1095
|
+
const json = process.argv.includes('--json');
|
|
1096
|
+
try {
|
|
1097
|
+
const fixtures = loadEvaluationCases(casesDir);
|
|
1098
|
+
if (!resultsDir) {
|
|
1099
|
+
const summary = catalogSummary(fixtures);
|
|
1100
|
+
if (json) console.log(JSON.stringify({ ok: true, mode: 'catalog', ...summary }));
|
|
1101
|
+
else {
|
|
1102
|
+
log.success(`✓ Evaluation catalog valid (${summary.total} cases)`);
|
|
1103
|
+
console.log(`- Categories: ${Object.keys(summary.categories).join(', ')}`);
|
|
1104
|
+
console.log(`- Oracle verdicts: ${JSON.stringify(summary.verdicts)}`);
|
|
1105
|
+
}
|
|
1106
|
+
return;
|
|
1107
|
+
}
|
|
1108
|
+
|
|
1109
|
+
const score = scoreEvaluationResults(fixtures, loadResults(resultsDir));
|
|
1110
|
+
const ok = score.failed === 0 && score.incorrectPasses === 0 && score.secretLeaks === 0;
|
|
1111
|
+
if (json) console.log(JSON.stringify({ ok, ...score }));
|
|
1112
|
+
else {
|
|
1113
|
+
console.log(`Evaluation: ${score.passed}/${score.total} passed`);
|
|
1114
|
+
console.log(`- Incorrect PASS: ${score.incorrectPasses}`);
|
|
1115
|
+
console.log(`- Secret leaks: ${score.secretLeaks}`);
|
|
1116
|
+
for (const item of score.cases.filter((result) => !result.passed)) {
|
|
1117
|
+
console.log(` ${item.caseId}: expected=${item.expected} actual=${item.actual || 'MISSING'}`);
|
|
1118
|
+
}
|
|
1119
|
+
}
|
|
1120
|
+
if (!ok) process.exit(1);
|
|
1121
|
+
} catch (err) {
|
|
1122
|
+
if (json) console.log(JSON.stringify({ ok: false, code: err.code || 'EVALUATION_FAILED', error: err.message, details: err.details || [] }));
|
|
1123
|
+
else {
|
|
1124
|
+
log.error(err.message);
|
|
1125
|
+
for (const detail of err.details || []) console.error(`- ${detail}`);
|
|
1126
|
+
}
|
|
1127
|
+
process.exit(1);
|
|
1128
|
+
}
|
|
1129
|
+
}
|
|
1130
|
+
|
|
1131
|
+
function handleEscalation() {
|
|
1132
|
+
const runId = argValue('--run');
|
|
1133
|
+
const json = process.argv.includes('--json');
|
|
1134
|
+
try {
|
|
1135
|
+
const assessment = assessRunEscalation(CWD, { runId });
|
|
1136
|
+
if (json) console.log(JSON.stringify({ ok: true, ...assessment }));
|
|
1137
|
+
else {
|
|
1138
|
+
console.log(`Required model tier: ${assessment.requiredTier}`);
|
|
1139
|
+
for (const reason of assessment.reasons) {
|
|
1140
|
+
console.log(`- ${reason.code}: ${reason.detail}`);
|
|
1141
|
+
}
|
|
1142
|
+
}
|
|
1143
|
+
} catch (err) {
|
|
1144
|
+
if (json) console.log(JSON.stringify({ ok: false, code: err.code || 'ESCALATION_FAILED', error: err.message }));
|
|
1145
|
+
else log.error(err.message);
|
|
1146
|
+
process.exit(1);
|
|
1147
|
+
}
|
|
1148
|
+
}
|
|
1149
|
+
|
|
1150
|
+
function handleState() {
|
|
1151
|
+
const runId = argValue('--run');
|
|
1152
|
+
const json = process.argv.includes('--json');
|
|
1153
|
+
try {
|
|
1154
|
+
const run = runId ? loadRun(CWD, runId) : getCurrentRun(CWD);
|
|
1155
|
+
if (!run) throw new Error('No current run. Start one with `ai-engineering-loop run`.');
|
|
1156
|
+
if (json) console.log(JSON.stringify({ ok: true, ...run }));
|
|
1157
|
+
else {
|
|
1158
|
+
console.log(`Run: ${run.runId}`);
|
|
1159
|
+
console.log(`State: ${run.state}`);
|
|
1160
|
+
console.log(`Iteration: ${run.iteration}`);
|
|
1161
|
+
console.log(`Artifacts: ${Object.keys(run.artifacts).join(', ') || 'none'}`);
|
|
1162
|
+
}
|
|
1163
|
+
} catch (err) {
|
|
1164
|
+
if (json) console.log(JSON.stringify({ ok: false, code: err.code || 'STATE_FAILED', error: err.message }));
|
|
1165
|
+
else log.error(err.message);
|
|
1166
|
+
process.exit(1);
|
|
1167
|
+
}
|
|
1168
|
+
}
|
|
1169
|
+
|
|
1170
|
+
function handleDoctor() {
|
|
1171
|
+
const json = process.argv.includes('--json');
|
|
1172
|
+
const result = runDoctor(path.join(__dirname, '..'));
|
|
1173
|
+
if (json) console.log(JSON.stringify(result));
|
|
1174
|
+
else {
|
|
1175
|
+
console.log(`AI Engineering Loop doctor: ${result.ok ? 'READY' : 'NOT READY'}`);
|
|
1176
|
+
for (const check of result.checks) {
|
|
1177
|
+
console.log(`${check.passed ? '✓' : '✗'} ${check.id}: ${check.detail}`);
|
|
1178
|
+
}
|
|
1179
|
+
}
|
|
1180
|
+
if (!result.ok) process.exit(1);
|
|
1181
|
+
}
|
|
1182
|
+
|
|
781
1183
|
// Help Menu
|
|
782
1184
|
function printHelp() {
|
|
783
1185
|
console.log(`
|
|
@@ -791,7 +1193,40 @@ Commands:
|
|
|
791
1193
|
init Bootstrap .ai-engineering-loop/ context from repository discovery
|
|
792
1194
|
status Check the validity, readiness, and baseline freshness of context
|
|
793
1195
|
refresh Reconcile drifted context against repository non-destructively
|
|
794
|
-
run
|
|
1196
|
+
run [task] Start or resume a stateful engineering run, sync hosts, and instruct the agent
|
|
1197
|
+
--new start a new run even when another run is active
|
|
1198
|
+
--mode report-only|assisted|unattended
|
|
1199
|
+
state Inspect the current run ledger
|
|
1200
|
+
--run <id> target a non-current run
|
|
1201
|
+
--json print the complete machine-readable ledger
|
|
1202
|
+
gate <name> Validate an artifact and advance the current run
|
|
1203
|
+
goal | report | maker | verification | review | judge | delivery
|
|
1204
|
+
--run <id> target a non-current run
|
|
1205
|
+
--json print machine-readable gate output
|
|
1206
|
+
policy Inspect or update fail-closed runtime controls
|
|
1207
|
+
show [--json]
|
|
1208
|
+
set [--default-mode <mode>] [--allow-unattended true|false]
|
|
1209
|
+
[--require-sandbox true|false]
|
|
1210
|
+
[--per-run-token-limit <n>] [--daily-token-limit <n>]
|
|
1211
|
+
budget Enforce actual provider-reported token usage and emergency pause
|
|
1212
|
+
status [--run <id>] [--json]
|
|
1213
|
+
record --input <n> --output <n> --model <id> [--run <id>]
|
|
1214
|
+
pause | resume
|
|
1215
|
+
sandbox Isolate Maker changes in a locked disposable Git worktree
|
|
1216
|
+
create | capture | abort | status [--run <id>] [--json]
|
|
1217
|
+
context <stage> <files...>
|
|
1218
|
+
Build a bounded, redacted context pack for maker, devil-advocate, or judge
|
|
1219
|
+
--run <id> target a non-current run
|
|
1220
|
+
--json print only pack metadata; never print packed content
|
|
1221
|
+
eval Validate the 20-case production evaluation catalog
|
|
1222
|
+
--results <dir> score host-generated JSON results
|
|
1223
|
+
--cases <dir> use another compatible fixture catalog
|
|
1224
|
+
--json print machine-readable metrics
|
|
1225
|
+
escalation Compute the minimum Judge model tier from deterministic risk signals
|
|
1226
|
+
--run <id> target a non-current run
|
|
1227
|
+
--json print machine-readable reasons
|
|
1228
|
+
doctor Check runtime, schemas, host assets, package contents, and eval catalog
|
|
1229
|
+
--json print machine-readable diagnostics
|
|
795
1230
|
sync-hosts Copy package skills/agents/commands into ~/.claude ~/.grok ~/.gemini ~/.agents
|
|
796
1231
|
(only hosts that already exist; DOT skills only if already installed)
|
|
797
1232
|
--dry-run print the plan without writing
|
|
@@ -826,6 +1261,33 @@ switch (command) {
|
|
|
826
1261
|
case 'run':
|
|
827
1262
|
handleRun();
|
|
828
1263
|
break;
|
|
1264
|
+
case 'state':
|
|
1265
|
+
handleState();
|
|
1266
|
+
break;
|
|
1267
|
+
case 'gate':
|
|
1268
|
+
handleGate();
|
|
1269
|
+
break;
|
|
1270
|
+
case 'policy':
|
|
1271
|
+
handlePolicy();
|
|
1272
|
+
break;
|
|
1273
|
+
case 'budget':
|
|
1274
|
+
handleBudget();
|
|
1275
|
+
break;
|
|
1276
|
+
case 'sandbox':
|
|
1277
|
+
handleSandbox();
|
|
1278
|
+
break;
|
|
1279
|
+
case 'context':
|
|
1280
|
+
handleContext();
|
|
1281
|
+
break;
|
|
1282
|
+
case 'eval':
|
|
1283
|
+
handleEval();
|
|
1284
|
+
break;
|
|
1285
|
+
case 'escalation':
|
|
1286
|
+
handleEscalation();
|
|
1287
|
+
break;
|
|
1288
|
+
case 'doctor':
|
|
1289
|
+
handleDoctor();
|
|
1290
|
+
break;
|
|
829
1291
|
case 'sync-hosts':
|
|
830
1292
|
handleSyncHosts();
|
|
831
1293
|
break;
|
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
# Artifact Gates
|
|
2
|
+
|
|
3
|
+
AI Engineering Loop does not trust a model's statement that a stage is complete. A stage advances only when its artifact passes a deterministic CLI gate.
|
|
4
|
+
|
|
5
|
+
## Run ledger
|
|
6
|
+
|
|
7
|
+
Start or resume a run:
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
npx ai-engineering-loop run
|
|
11
|
+
npx ai-engineering-loop state --json
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
Artifacts live under `.ai-engineering-loop/runs/<run-id>/`. Do not edit `state.json` or `current.json` manually.
|
|
15
|
+
|
|
16
|
+
## Required transitions
|
|
17
|
+
|
|
18
|
+
| Stage | Artifact | Gate |
|
|
19
|
+
|---|---|---|
|
|
20
|
+
| Goal freeze | `goal-contract.json` | `gate goal` |
|
|
21
|
+
| Maker complete | `diff.patch` | `gate maker` |
|
|
22
|
+
| Verification | `verification.json` | `gate verification` |
|
|
23
|
+
| Devil's Advocate | `findings.json` | `gate review` |
|
|
24
|
+
| Judge | `verdict.json` | `gate judge` |
|
|
25
|
+
| Delivery | `delivery.json` | `gate delivery` |
|
|
26
|
+
|
|
27
|
+
JSON sidecars follow the schemas shipped in `schemas/`. Keep the Markdown Goal Contract and human report; sidecars make their claims machine-checkable.
|
|
28
|
+
|
|
29
|
+
Gate failures are evidence. Do not edit state to bypass one. Repair the artifact or return to Maker.
|
|
30
|
+
|
|
31
|
+
## Context boundary
|
|
32
|
+
|
|
33
|
+
Create model-visible context through:
|
|
34
|
+
|
|
35
|
+
```bash
|
|
36
|
+
npx ai-engineering-loop context maker <files...>
|
|
37
|
+
npx ai-engineering-loop context devil-advocate <files...>
|
|
38
|
+
npx ai-engineering-loop context judge <files...>
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
The command rejects sensitive paths and repository escapes, redacts recognized credentials, caps files and estimated tokens, and marks truncation. Never replace a bounded pack with the parent transcript.
|
|
42
|
+
|
|
43
|
+
Before Judge, run:
|
|
44
|
+
|
|
45
|
+
```bash
|
|
46
|
+
npx ai-engineering-loop escalation --json
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
Use the minimum returned tier from the active host catalog. Never invent a model slug. A lower-tier Judge cannot issue `PASS` when stronger review is required.
|
|
@@ -35,10 +35,10 @@ flowchart TD
|
|
|
35
35
|
### Layer 2: Engineering Core (`core/`)
|
|
36
36
|
- **Authority**: AI Engineering Operating System specification.
|
|
37
37
|
- **Scope**: Mandatory engineering invariants:
|
|
38
|
-
- No code changes without a formalized [Goal Contract](
|
|
38
|
+
- No code changes without a formalized [Goal Contract](./goal-contract.md).
|
|
39
39
|
- Deterministic checks must pass 100% before adversarial review.
|
|
40
40
|
- Review findings require reproducible evidence (Level 1–3) to be valid.
|
|
41
|
-
- Only the [Judge Agent](
|
|
41
|
+
- Only the [Judge Agent](../agents/judge.md) can issue a `PASS` verdict.
|
|
42
42
|
- **Overridability**: Invariant. Downstream layers cannot skip verification or eliminate agent roles.
|
|
43
43
|
|
|
44
44
|
### Layer 3: Project Type Profile (`profiles/`)
|
|
@@ -55,7 +55,7 @@ flowchart TD
|
|
|
55
55
|
- Layer definitions in `architecture.md`.
|
|
56
56
|
- Architectural constraints and forbidden patterns in `conventions.md`.
|
|
57
57
|
- Configured delivery adapter (DOT, GitHub, GitLab) in `adapter.md`.
|
|
58
|
-
- **Auto-Discovery**: Automatically generated if missing via [Project Initialization](
|
|
58
|
+
- **Auto-Discovery**: Automatically generated if missing via [Project Initialization](./project-initialization.md).
|
|
59
59
|
- **Overridability**: Overrides Layer 3 defaults for this specific codebase.
|
|
60
60
|
|
|
61
61
|
### Layer 5: Task Contract
|
|
@@ -99,4 +99,4 @@ def resolve_config_key(key: str, context: ExecutionContext) -> Any:
|
|
|
99
99
|
- *Refinement (Allowed)*: Profile says "Run unit tests"; Repo config specifies `pnpm run test:unit`. $\rightarrow$ Valid refinement.
|
|
100
100
|
- *Contradiction (Escalate)*: Core requires deterministic verification; Task contract requests skipping tests to merge faster. $\rightarrow$ **Forbidden**. The engine rejects the override and flags a policy violation.
|
|
101
101
|
2. **Unresolvable Contradictions**:
|
|
102
|
-
- If repository configuration directly contradicts an established platform invariant (e.g. a command requires root sudo or drops protected tables), the system halts and triggers [Human Escalation](
|
|
102
|
+
- If repository configuration directly contradicts an established platform invariant (e.g. a command requires root sudo or drops protected tables), the system halts and triggers [Human Escalation](./escalation-policy.md).
|
|
@@ -92,7 +92,7 @@ flowchart TD
|
|
|
92
92
|
|
|
93
93
|
## 4. Devil's Advocate & Judge Drift Interaction
|
|
94
94
|
|
|
95
|
-
During task review, the [Devil's Advocate](
|
|
95
|
+
During task review, the [Devil's Advocate](../agents/devil-advocate.md) may discover that the codebase has drifted from the documented architecture:
|
|
96
96
|
|
|
97
97
|
> *Example Review Finding*: "The implementation assumes `user-service` connects directly to Postgres, but the codebase has migrated to an asynchronous event queue in `src/events/`."
|
|
98
98
|
|
|
@@ -26,7 +26,7 @@ flowchart LR
|
|
|
26
26
|
```
|
|
27
27
|
|
|
28
28
|
### Pillar 1: Contract Satisfaction
|
|
29
|
-
- 100% of Acceptance Criteria defined in the [Goal Contract](
|
|
29
|
+
- 100% of Acceptance Criteria defined in the [Goal Contract](./goal-contract.md) are demonstrably met.
|
|
30
30
|
- Zero out-of-scope files or unauthorized modules were modified.
|
|
31
31
|
- All technical constraints (e.g. backward compatibility, no unapproved dependencies) are preserved.
|
|
32
32
|
|
|
@@ -45,12 +45,12 @@ flowchart LR
|
|
|
45
45
|
- **Null & Boundary Safety**: Explicit handling of `null`, `undefined`, empty collections, and error paths.
|
|
46
46
|
|
|
47
47
|
### Pillar 4: Adversarial Consensus
|
|
48
|
-
- Independent [Devil's Advocate Review](
|
|
48
|
+
- Independent [Devil's Advocate Review](../agents/devil-advocate.md) has been executed across all 6 core review domains.
|
|
49
49
|
- **Zero Unresolved Blocking Findings**: No open `SEV-1 (Critical)` or `SEV-2 (High)` findings.
|
|
50
50
|
- **Evidence-Based Triage**: Every raised finding has been formally triaged with reproducible evidence as `VALID` (and resolved in code) or `INVALID` (with technical proof of why it is a false positive).
|
|
51
51
|
|
|
52
52
|
### Pillar 5: Judge Certified PASS
|
|
53
|
-
- The [Judge Agent](
|
|
53
|
+
- The [Judge Agent](../agents/judge.md) has evaluated the complete execution trace, verified the evidence, confirmed no-progress limits were not violated, and issued a signed `PASS` verdict.
|
|
54
54
|
|
|
55
55
|
---
|
|
56
56
|
|
|
@@ -31,7 +31,7 @@ flowchart TD
|
|
|
31
31
|
- **Rationale**: Prolonged retries indicate either an incorrect fundamental strategy or a hidden architectural obstacle.
|
|
32
32
|
|
|
33
33
|
### Trigger 2: No-Progress / Stagnation Detected
|
|
34
|
-
- **Condition**: Two consecutive iterations produce identical or semantically equivalent finding signatures without demonstrable code convergence (see [No-Progress Policy](
|
|
34
|
+
- **Condition**: Two consecutive iterations produce identical or semantically equivalent finding signatures without demonstrable code convergence (see [No-Progress Policy](../policies/no-progress-policy.md)).
|
|
35
35
|
- **Rationale**: Prevents thrashing where the agent modifies code without resolving the underlying flaw.
|
|
36
36
|
|
|
37
37
|
### Trigger 3: Contradictory or Inconsistent Acceptance Criteria
|
|
@@ -93,7 +93,7 @@ Instead, it must render an **Actionable Escalation Report** following this struc
|
|
|
93
93
|
|
|
94
94
|
## 4. Resumption Protocol
|
|
95
95
|
|
|
96
|
-
Once the human engineer provides guidance or amends the [Goal Contract](
|
|
96
|
+
Once the human engineer provides guidance or amends the [Goal Contract](./goal-contract.md):
|
|
97
97
|
1. The iteration counter is reset: $K \leftarrow 1$.
|
|
98
98
|
2. The agent incorporates the human's decision into the Goal Contract constraints.
|
|
99
99
|
3. The loop resumes at the **Maker Agent** phase with clean validation.
|