@tea-agent/loop-agent 0.20.1 → 0.21.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +26 -0
- package/dist/commands/init.js +7 -0
- package/dist/executors/dag-pi-executor.js +24 -0
- package/dist/executors/pi-executor.js +111 -36
- package/dist/executors/pi-sdk-executor.js +105 -29
- package/dist/executors/shell-executor.js +54 -11
- package/dist/worker/loop-agent/loop-agent-client.js +43 -9
- package/dist/worker/observability/read-model.js +7 -1
- package/dist/worker/observe/static/constants.js +5 -0
- package/dist/worker/observe/static/format-pool.js +22 -3
- package/dist/worker/observe/static/styles.css +32 -3
- package/dist/worker/observe/static/views/dag-inspector.js +2 -2
- package/dist/worker/run-task/run-task.js +16 -6
- package/dist/workflows/dag/backend-test-markdown-workflow.js +291 -97
- package/dist/workflows/dag/backend-test-result-contract.js +10 -4
- package/dist/workflows/dag/init-hybrid.js +27 -16
- package/dist/workflows/dag/lifecycle.js +60 -4
- package/dist/workflows/dag/liveness-policy.js +250 -0
- package/dist/workflows/dag/node-execution.js +49 -0
- package/dist/workflows/dag/runner.js +21 -1
- package/dist/workflows/dag/types.js +5 -0
- package/docs/README.md +5 -6
- package/docs/architecture/dag-execution.md +11 -0
- package/docs/architecture/facts-and-state.md +1 -0
- package/docs/architecture/worker-and-feature.md +10 -0
- package/docs/templates/backend-test-dag.generate-pytest.prompt.md +5 -2
- package/docs/templates/backend-test-dag.json +15 -15
- package/harness.json +1 -1
- package/package.json +1 -1
- package/skills/loop-agent/references/command-reference.md +2 -0
- package/skills/loop-agent/references/hybrid-dag.md +2 -2
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,24 @@
|
|
|
2
2
|
|
|
3
3
|
## [Unreleased]
|
|
4
4
|
|
|
5
|
+
## [0.21.0] - 2026-07-24
|
|
6
|
+
|
|
7
|
+
### 重点更新
|
|
8
|
+
|
|
9
|
+
- DAG/Pi 自适应活跃度监管:精准区分传输活动与真实执行进展,受控处置静默卡死,避免误杀。
|
|
10
|
+
- 后端测试执行范围收窄:仅运行最终 Markdown 明确映射且通过安全校验的脚本,杜绝无关测试混入。
|
|
11
|
+
- 后端测试 HTML 报告全面升级:采用清晰的用例卡片,直观展示脱敏的请求与响应详情。
|
|
12
|
+
|
|
13
|
+
### 新增
|
|
14
|
+
|
|
15
|
+
- DAG/Pi 自适应 liveness 与非误杀监管:区分 runner lease、transport 活动与 Provider/tool/output 真实进展;节点可投影 active/quiet/suspected-stall/probing/needs-attention;静默 stall 会受控终止,退出无法确认时禁止自动重试;Pi absolute max 对齐 4h;Worker 默认不再给 `run-dag` 设置外层 wall-clock,显式 `worker.timeout_ms` 仍为 hard timeout。
|
|
16
|
+
|
|
17
|
+
### 改进
|
|
18
|
+
|
|
19
|
+
- backend-test 第 7 节点现在只执行最终 Markdown `自动化映射` 明确声明且安全校验通过的 pytest 脚本,避免历史或无关 `test_*.py` 混入本轮 JUnit/HTML;空映射、缺失脚本或越界路径会 fail-closed。
|
|
20
|
+
- backend-test HTML 报告升级为更清晰的用例卡片:保留测试结论、质量状态和失败概览,直接展示简洁测试场景、自动化用例名称、结果与耗时,并为通过/失败用例都展示同次执行中脱敏且有界的接口请求参数与响应结果;HTML 不再附带技术证据章节,审计路径与哈希继续保留在 facts/JUnit。
|
|
21
|
+
- 新初始化项目的 harness 指南同步 adaptive liveness、4h absolute max、stall 状态与 `termination-unconfirmed` 处置;旧项目可由 `init check-update` 识别未修改的受管指南,并通过 `init update --apply-safe` 安全刷新。
|
|
22
|
+
|
|
5
23
|
## [0.20.1] - 2026-07-24
|
|
6
24
|
|
|
7
25
|
### 新增
|
|
@@ -17,6 +35,9 @@
|
|
|
17
35
|
|
|
18
36
|
### 改进
|
|
19
37
|
|
|
38
|
+
- backend-test 第 4 个 Markdown 校验节点只将前置条件、操作步骤和预期结果作为必需章节;需求来源引用有效性不再阻断后续测试。普通质量问题保持 advisory,但 secret-shaped 值会先生成脱敏安全报告再 fail-closed,避免进入 pytest、HTML/facts 和最终报告。
|
|
39
|
+
- backend-test 生成的真实 HTTP pytest 现在必须输出脱敏且有界的请求/响应日志,包含 method、URL/path、请求参数摘要、响应状态和响应 body 摘要;追溯节点会在 Markdown 映射脚本范围内把日志合同缺口记录为 advisory。
|
|
40
|
+
- backend-test 第 7 节点的 HTML/facts 按测试概览、质量校验、失败概览、用例执行明细和技术证据分层展示;第 8 节点使用固定简洁的中文收口结构,减少重复原文并优先展示结论和风险。
|
|
20
41
|
- Operator Console「恢复」面板自动拉取 DAG 运行与节点候选,优先展示失败项并以选择器填入,仍可手动输入
|
|
21
42
|
- 从节点重跑预检失败时展示真实安全资格阻断原因,不再误报为「无法获取 planHash」
|
|
22
43
|
- 恢复页长操作(完整重跑等)会回写终态成功/失败;不再只提示「已接受」后无下文
|
|
@@ -33,6 +54,11 @@
|
|
|
33
54
|
- frontend-test 环境预检:在生成用例前对冻结的非生产 `baseUrl` 做 curl 可达性探测;不可达或缺少 curl 时以 `frontend-base-url-unreachable` / `curl-unavailable` 阻断 generate/map,并写入 `environmentProbe` 事实。节点 5 manifest 物化改为 ruleId 失败分类、map 配额预检与 temp+rename 原子写入;生成/清单/执行强制 playwright-cli-only,U/D 场景要求当前用户数据归属或 create/Mock/blocked。
|
|
34
55
|
- 前端测试 DAG 的证据自检改为按用例降级:个别缺失、空白或格式错误的证据会如实标为 blocked 并继续生成复盘报告;只有全部用例都未产生任何证据内容或证据路径不安全时才失败。
|
|
35
56
|
|
|
57
|
+
### 修复
|
|
58
|
+
|
|
59
|
+
- Observe 总览 KPI 问号提示框宽度与窄卡片解耦:提示框使用稳定的 240–280px 视口安全宽度,并在五列、三列和两列布局的右侧列向内对齐,避免宽屏下被压缩成约 78px 的窄条;hover 与键盘焦点行为保持不变。
|
|
60
|
+
- 节点检查器的规范证据文件预览会渲染 Markdown 标题、列表、表格与代码块,不再把正文作为带语法标记的纯文本显示。
|
|
61
|
+
|
|
36
62
|
## [0.20.0] - 2026-07-23
|
|
37
63
|
|
|
38
64
|
### 新增
|
package/dist/commands/init.js
CHANGED
|
@@ -2057,6 +2057,13 @@ function buildTargetLoopAgentHarness(input) {
|
|
|
2057
2057
|
"",
|
|
2058
2058
|
"Use `loop-agent new-task`, `loop-agent dag run-task`, `loop-agent dag validate`, and `loop-agent run-dag` for non-trivial implementation work.",
|
|
2059
2059
|
"",
|
|
2060
|
+
"## Adaptive Liveness",
|
|
2061
|
+
"",
|
|
2062
|
+
"- Healthy Pi nodes are no longer stopped by a fixed 30-minute deadline. The default 4h absolute max is a final safety bound and cannot be extended by synthetic heartbeats.",
|
|
2063
|
+
"- Runner heartbeat proves only the runner lease. Provider, tool, or output activity records meaningful progress; prolonged inactivity can surface as `quiet`, `suspected-stall`, `probing`, or `needs-attention` in `dag status`, `dag doctor`, and Observe.",
|
|
2064
|
+
"- A silent transport is aborted in a controlled way. `termination-unconfirmed` means the old attempt may still exist, so loop-agent fails closed and does not start an automatic retry; inspect the run before any operator recovery.",
|
|
2065
|
+
"- `agent-worker` does not set an outer `run-dag` wall-clock by default. An explicit `worker.timeout_ms` remains a hard timeout.",
|
|
2066
|
+
"",
|
|
2060
2067
|
"## Operator Recovery",
|
|
2061
2068
|
"",
|
|
2062
2069
|
"After a run, use `loop-agent dag report --run-id <run-id> --markdown` to read canonical run facts and `loop-agent dag doctor --run-id <run-id> --markdown` to diagnose failed or paused runs. Failed DAG runs should produce a failure handoff via `loop-agent dag closeout-draft --run-id <run-id>` instead of a successful closeout.",
|
|
@@ -263,6 +263,17 @@ export async function executeDagPiNode(input, meta, piStepFn = executePiStep) {
|
|
|
263
263
|
beforePathFingerprints = undefined;
|
|
264
264
|
}
|
|
265
265
|
}
|
|
266
|
+
const reportActivity = input.reportActivity;
|
|
267
|
+
const attempt = input.attempt ?? 1;
|
|
268
|
+
const bridgeActivity = reportActivity
|
|
269
|
+
? (kind, at) => {
|
|
270
|
+
reportActivity({
|
|
271
|
+
attempt,
|
|
272
|
+
kind,
|
|
273
|
+
at: at ?? new Date().toISOString(),
|
|
274
|
+
});
|
|
275
|
+
}
|
|
276
|
+
: undefined;
|
|
266
277
|
const result = await piStepFn({
|
|
267
278
|
attachedFiles: [],
|
|
268
279
|
modelConfig: resolveDagPiModelConfig(input.model),
|
|
@@ -272,6 +283,19 @@ export async function executeDagPiNode(input, meta, piStepFn = executePiStep) {
|
|
|
272
283
|
toolNames: resolveDagPiToolNames(input.task),
|
|
273
284
|
userMessage: buildDagPiUserMessage(input.task, persona, step),
|
|
274
285
|
sessionEventsPath: path.join(meta.runDir, input.task.id, "session-events.jsonl"),
|
|
286
|
+
// Absolute max wall clock from adaptive liveness policy (defaults to 4h).
|
|
287
|
+
timeoutMs: input.timeoutMs,
|
|
288
|
+
stallTimeoutMs: input.stallTimeoutMs,
|
|
289
|
+
abortGraceMs: input.abortGraceMs,
|
|
290
|
+
onActivity: bridgeActivity
|
|
291
|
+
? (activity) => {
|
|
292
|
+
if (activity.kind === "lease"
|
|
293
|
+
|| activity.kind === "synthetic-heartbeat") {
|
|
294
|
+
return;
|
|
295
|
+
}
|
|
296
|
+
bridgeActivity(activity.kind, activity.at);
|
|
297
|
+
}
|
|
298
|
+
: undefined,
|
|
275
299
|
});
|
|
276
300
|
const context = {
|
|
277
301
|
channel: "dag",
|
|
@@ -47,8 +47,13 @@ export function extractSubagentStats(stdout) {
|
|
|
47
47
|
parts.push(`agents=[${Array.from(agents).sort().join(',')}]`);
|
|
48
48
|
return parts.join(' | ');
|
|
49
49
|
}
|
|
50
|
-
|
|
51
|
-
export const
|
|
50
|
+
/** Absolute max wall clock for a single Pi attempt (4h). Not renewed by empty heartbeats. */
|
|
51
|
+
export const DEFAULT_TIMEOUT_MS = 14_400_000;
|
|
52
|
+
export const MAX_ALLOWED_TIMEOUT_MS = 14_400_000;
|
|
53
|
+
/** Max provider/CLI transport silence before initiating a controlled abort (15m). */
|
|
54
|
+
export const DEFAULT_STALL_TIMEOUT_MS = 900_000;
|
|
55
|
+
/** Grace after SIGTERM before SIGKILL on absolute max. */
|
|
56
|
+
export const DEFAULT_ABORT_GRACE_MS = 30_000;
|
|
52
57
|
const PI_BACKEND_ENV = 'CODE_AGENT_PI_BACKEND';
|
|
53
58
|
const OUTPUT_PREVIEW_HEAD_CHARS = 64_000;
|
|
54
59
|
const OUTPUT_PREVIEW_TAIL_CHARS = 64_000;
|
|
@@ -515,7 +520,9 @@ async function runCliOrchestration(options, meta) {
|
|
|
515
520
|
}
|
|
516
521
|
const primaryHardFail = !primaryResult.ok;
|
|
517
522
|
const primaryInvalidOutput = primaryResult.ok && primaryIssues.length > 0;
|
|
518
|
-
const shouldFallback = primary.fallback && ((primaryHardFail
|
|
523
|
+
const shouldFallback = primary.fallback && ((primaryHardFail
|
|
524
|
+
&& primaryResult.failureCategory !== 'termination-unconfirmed'
|
|
525
|
+
&& shouldRetryWithFallback(primaryResult.stderr, primaryResult.stdout))
|
|
519
526
|
|| primaryInvalidOutput);
|
|
520
527
|
if (shouldFallback) {
|
|
521
528
|
const fallbackResult = await executeSingleCliAttempt(options, primary.fallback);
|
|
@@ -590,7 +597,9 @@ async function runCliOrchestration(options, meta) {
|
|
|
590
597
|
function shouldFallbackSdkToCli(sdkResult, validationIssues) {
|
|
591
598
|
if (sdkResult.ok && validationIssues.length > 0)
|
|
592
599
|
return true;
|
|
593
|
-
if (!sdkResult.ok && (sdkResult.failureCategory === 'auth'
|
|
600
|
+
if (!sdkResult.ok && (sdkResult.failureCategory === 'auth'
|
|
601
|
+
|| sdkResult.failureCategory === 'timeout'
|
|
602
|
+
|| sdkResult.failureCategory === 'termination-unconfirmed')) {
|
|
594
603
|
return false;
|
|
595
604
|
}
|
|
596
605
|
if (!sdkResult.ok && shouldRetryWithFallback(sdkResult.stderr, sdkResult.stdout))
|
|
@@ -655,9 +664,15 @@ async function executeSingleCliAttempt(options, modelConfig) {
|
|
|
655
664
|
? `${modelConfig.provider}/${modelConfig.model}`
|
|
656
665
|
: modelConfig.model ?? 'default';
|
|
657
666
|
const timeoutMs = options.timeoutMs ?? modelConfig.timeoutMs ?? DEFAULT_TIMEOUT_MS;
|
|
667
|
+
const stallTimeoutMs = options.stallTimeoutMs ?? DEFAULT_STALL_TIMEOUT_MS;
|
|
668
|
+
const abortGraceMs = options.abortGraceMs ?? DEFAULT_ABORT_GRACE_MS;
|
|
658
669
|
let timedOut = false;
|
|
670
|
+
let terminationUnconfirmed = false;
|
|
671
|
+
let supervisionStderr = '';
|
|
659
672
|
let timeoutHandle;
|
|
673
|
+
let stallHandle;
|
|
660
674
|
let sigkillHandle;
|
|
675
|
+
let exitConfirmationHandle;
|
|
661
676
|
// Build pi CLI arguments (same as before).
|
|
662
677
|
// We invoke pi via `node <pi-cli-path>` instead of relying on `#!/usr/bin/env node`
|
|
663
678
|
// to avoid provider resolution issues when spawned from tsx-managed processes.
|
|
@@ -698,39 +713,39 @@ async function executeSingleCliAttempt(options, modelConfig) {
|
|
|
698
713
|
const stdoutPreview = new BoundedTextPreview('stdout');
|
|
699
714
|
const stderrPreview = new BoundedTextPreview('stderr', 16_000, 16_000);
|
|
700
715
|
const stdoutCollector = new PiJsonlStreamCollector();
|
|
716
|
+
let settled = false;
|
|
701
717
|
child.stdout.setEncoding('utf-8');
|
|
702
718
|
child.stderr.setEncoding('utf-8');
|
|
703
|
-
|
|
704
|
-
stdoutPreview.append(chunk);
|
|
705
|
-
stdoutCollector.append(chunk);
|
|
706
|
-
});
|
|
707
|
-
child.stderr.on('data', (chunk) => {
|
|
708
|
-
stderrPreview.append(chunk);
|
|
709
|
-
});
|
|
710
|
-
child.on('error', (err) => {
|
|
711
|
-
if (timeoutHandle)
|
|
712
|
-
clearTimeout(timeoutHandle);
|
|
713
|
-
if (sigkillHandle)
|
|
714
|
-
clearTimeout(sigkillHandle);
|
|
715
|
-
reject(err);
|
|
716
|
-
});
|
|
717
|
-
child.on('close', (code) => {
|
|
719
|
+
const clearSupervisionTimers = () => {
|
|
718
720
|
if (timeoutHandle)
|
|
719
721
|
clearTimeout(timeoutHandle);
|
|
722
|
+
if (stallHandle)
|
|
723
|
+
clearTimeout(stallHandle);
|
|
720
724
|
if (sigkillHandle)
|
|
721
725
|
clearTimeout(sigkillHandle);
|
|
726
|
+
if (exitConfirmationHandle)
|
|
727
|
+
clearTimeout(exitConfirmationHandle);
|
|
728
|
+
};
|
|
729
|
+
const finish = (code) => {
|
|
730
|
+
if (settled)
|
|
731
|
+
return;
|
|
732
|
+
settled = true;
|
|
733
|
+
clearSupervisionTimers();
|
|
722
734
|
const durationMs = Date.now() - startedAt;
|
|
723
735
|
const stdout = stdoutPreview.text();
|
|
724
|
-
const
|
|
736
|
+
const capturedStderr = stderrPreview.text();
|
|
737
|
+
const stderr = [supervisionStderr, capturedStderr].filter(Boolean).join('\n');
|
|
725
738
|
const collected = stdoutCollector.finish();
|
|
726
|
-
const failureCategory =
|
|
727
|
-
|
|
728
|
-
|
|
729
|
-
|
|
730
|
-
|
|
731
|
-
|
|
732
|
-
|
|
733
|
-
|
|
739
|
+
const failureCategory = terminationUnconfirmed
|
|
740
|
+
? 'termination-unconfirmed'
|
|
741
|
+
: classifyPiFailure({
|
|
742
|
+
assistantText: collected.assistantText,
|
|
743
|
+
exitCode: code,
|
|
744
|
+
stderr,
|
|
745
|
+
stdout,
|
|
746
|
+
timedOut,
|
|
747
|
+
outputTooLarge: collected.outputTooLarge,
|
|
748
|
+
});
|
|
734
749
|
resolve({
|
|
735
750
|
assistantText: collected.assistantText,
|
|
736
751
|
command: ['pi', ...piArgs],
|
|
@@ -738,7 +753,7 @@ async function executeSingleCliAttempt(options, modelConfig) {
|
|
|
738
753
|
exitCode: code,
|
|
739
754
|
failureCategory,
|
|
740
755
|
modelDisplay,
|
|
741
|
-
ok: code === 0 && collected.assistantText.length > 0 && !collected.outputTooLarge,
|
|
756
|
+
ok: !timedOut && code === 0 && collected.assistantText.length > 0 && !collected.outputTooLarge,
|
|
742
757
|
parsedEvents: collected.parsedEvents,
|
|
743
758
|
stderr,
|
|
744
759
|
stdout,
|
|
@@ -749,16 +764,76 @@ async function executeSingleCliAttempt(options, modelConfig) {
|
|
|
749
764
|
tokensUsed: collected.tokensUsed,
|
|
750
765
|
subagentStats: collected.subagentStats,
|
|
751
766
|
});
|
|
767
|
+
};
|
|
768
|
+
const hasConfirmedExitCode = () => child.exitCode !== null && child.exitCode !== undefined;
|
|
769
|
+
const beginControlledTermination = (reason) => {
|
|
770
|
+
if (timedOut || settled)
|
|
771
|
+
return;
|
|
772
|
+
timedOut = true;
|
|
773
|
+
supervisionStderr = reason;
|
|
774
|
+
child.kill('SIGTERM');
|
|
775
|
+
sigkillHandle = setTimeout(() => {
|
|
776
|
+
if (settled)
|
|
777
|
+
return;
|
|
778
|
+
if (hasConfirmedExitCode()) {
|
|
779
|
+
finish(child.exitCode);
|
|
780
|
+
return;
|
|
781
|
+
}
|
|
782
|
+
child.kill('SIGKILL');
|
|
783
|
+
exitConfirmationHandle = setTimeout(() => {
|
|
784
|
+
if (settled)
|
|
785
|
+
return;
|
|
786
|
+
if (hasConfirmedExitCode()) {
|
|
787
|
+
finish(child.exitCode);
|
|
788
|
+
return;
|
|
789
|
+
}
|
|
790
|
+
terminationUnconfirmed = true;
|
|
791
|
+
supervisionStderr += `\npi CLI exit was not confirmed within ${abortGraceMs}ms after SIGKILL`;
|
|
792
|
+
finish(null);
|
|
793
|
+
}, abortGraceMs);
|
|
794
|
+
}, abortGraceMs);
|
|
795
|
+
};
|
|
796
|
+
const armStallWatchdog = () => {
|
|
797
|
+
if (stallTimeoutMs <= 0 || timedOut || settled)
|
|
798
|
+
return;
|
|
799
|
+
if (stallHandle)
|
|
800
|
+
clearTimeout(stallHandle);
|
|
801
|
+
stallHandle = setTimeout(() => {
|
|
802
|
+
beginControlledTermination(`pi CLI step stalled after ${stallTimeoutMs}ms with no stdout/stderr activity`);
|
|
803
|
+
}, stallTimeoutMs);
|
|
804
|
+
};
|
|
805
|
+
const reportOutputActivity = () => {
|
|
806
|
+
armStallWatchdog();
|
|
807
|
+
try {
|
|
808
|
+
options.onActivity?.({ kind: 'output', at: new Date().toISOString() });
|
|
809
|
+
}
|
|
810
|
+
catch {
|
|
811
|
+
// best-effort: never fail the pi step
|
|
812
|
+
}
|
|
813
|
+
};
|
|
814
|
+
child.stdout.on('data', (chunk) => {
|
|
815
|
+
stdoutPreview.append(chunk);
|
|
816
|
+
stdoutCollector.append(chunk);
|
|
817
|
+
if (chunk.length > 0)
|
|
818
|
+
reportOutputActivity();
|
|
819
|
+
});
|
|
820
|
+
child.stderr.on('data', (chunk) => {
|
|
821
|
+
stderrPreview.append(chunk);
|
|
822
|
+
if (chunk.length > 0)
|
|
823
|
+
reportOutputActivity();
|
|
752
824
|
});
|
|
825
|
+
child.on('error', (err) => {
|
|
826
|
+
clearSupervisionTimers();
|
|
827
|
+
settled = true;
|
|
828
|
+
reject(err);
|
|
829
|
+
});
|
|
830
|
+
child.on('close', (code) => {
|
|
831
|
+
finish(code);
|
|
832
|
+
});
|
|
833
|
+
armStallWatchdog();
|
|
753
834
|
if (timeoutMs > 0) {
|
|
754
835
|
timeoutHandle = setTimeout(() => {
|
|
755
|
-
|
|
756
|
-
child.kill('SIGTERM');
|
|
757
|
-
sigkillHandle = setTimeout(() => {
|
|
758
|
-
if (child.exitCode === null) {
|
|
759
|
-
child.kill('SIGKILL');
|
|
760
|
-
}
|
|
761
|
-
}, 5_000);
|
|
836
|
+
beginControlledTermination(`pi CLI step timed out after ${timeoutMs}ms`);
|
|
762
837
|
}, timeoutMs);
|
|
763
838
|
}
|
|
764
839
|
});
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
import { appendFile, mkdir } from 'node:fs/promises';
|
|
2
2
|
import path from 'node:path';
|
|
3
|
-
import { BoundedTextPreview, classifyPiFailure, createPiJsonlStreamCollector, DEFAULT_TIMEOUT_MS, extractAssistantTextFromPiJson, } from './pi-executor.js';
|
|
3
|
+
import { BoundedTextPreview, classifyPiFailure, createPiJsonlStreamCollector, DEFAULT_ABORT_GRACE_MS, DEFAULT_STALL_TIMEOUT_MS, DEFAULT_TIMEOUT_MS, extractAssistantTextFromPiJson, } from './pi-executor.js';
|
|
4
4
|
import { serializeSessionEvent } from './pi-event-serializer.js';
|
|
5
5
|
let sdkSessionFactoryOverride;
|
|
6
6
|
let sdkImportOverrideForTests;
|
|
@@ -169,6 +169,21 @@ export function shouldPersistSessionEvent(event) {
|
|
|
169
169
|
}
|
|
170
170
|
return true;
|
|
171
171
|
}
|
|
172
|
+
function classifySdkActivityKind(event) {
|
|
173
|
+
if (!event || typeof event !== 'object')
|
|
174
|
+
return 'provider';
|
|
175
|
+
const type = typeof event.type === 'string'
|
|
176
|
+
? String(event.type)
|
|
177
|
+
: '';
|
|
178
|
+
if (type === 'tool_start' || type === 'tool_end'
|
|
179
|
+
|| type === 'tool_execution_start' || type === 'tool_execution_end') {
|
|
180
|
+
return 'tool';
|
|
181
|
+
}
|
|
182
|
+
if (type === 'thinking_delta' || type === 'message_update') {
|
|
183
|
+
return null;
|
|
184
|
+
}
|
|
185
|
+
return 'provider';
|
|
186
|
+
}
|
|
172
187
|
function createSessionEventAppender(filePath, onSessionEvent) {
|
|
173
188
|
let chain = Promise.resolve();
|
|
174
189
|
let dirEnsured = false;
|
|
@@ -181,6 +196,9 @@ function createSessionEventAppender(filePath, onSessionEvent) {
|
|
|
181
196
|
dirEnsured = true;
|
|
182
197
|
}
|
|
183
198
|
await appendFile(filePath, `${line}\n`, 'utf-8');
|
|
199
|
+
// Only after successful persistence: surface the persisted session event.
|
|
200
|
+
// Transport activity is reported synchronously by the subscription so
|
|
201
|
+
// filtered deltas and slow disk writes cannot trip the stall watchdog.
|
|
184
202
|
onSessionEvent?.(line, event);
|
|
185
203
|
}
|
|
186
204
|
catch {
|
|
@@ -271,6 +289,8 @@ export async function executeSingleSdkAttempt(options) {
|
|
|
271
289
|
? `${modelConfig.provider}/${modelConfig.model}`
|
|
272
290
|
: modelConfig.model ?? 'default';
|
|
273
291
|
const timeoutMs = options.timeoutMs ?? modelConfig.timeoutMs ?? DEFAULT_TIMEOUT_MS;
|
|
292
|
+
const stallTimeoutMs = options.stallTimeoutMs ?? DEFAULT_STALL_TIMEOUT_MS;
|
|
293
|
+
const abortGraceMs = options.abortGraceMs ?? DEFAULT_ABORT_GRACE_MS;
|
|
274
294
|
const piSdkArgs = [
|
|
275
295
|
'--provider', modelConfig.provider ?? '(default)',
|
|
276
296
|
'--model', modelConfig.model ?? '(default)',
|
|
@@ -282,15 +302,46 @@ export async function executeSingleSdkAttempt(options) {
|
|
|
282
302
|
];
|
|
283
303
|
const startedAt = Date.now();
|
|
284
304
|
let timedOut = false;
|
|
305
|
+
let terminationConfirmed = true;
|
|
285
306
|
let stderr = '';
|
|
286
307
|
const stdoutPreview = new BoundedTextPreview('stdout');
|
|
287
308
|
const stdoutCollector = createPiJsonlStreamCollector();
|
|
288
309
|
const usageSamples = [];
|
|
289
310
|
let session;
|
|
311
|
+
let unsubscribe;
|
|
290
312
|
let timeoutHandle;
|
|
313
|
+
let stallHandle;
|
|
314
|
+
let disposeAttempted = false;
|
|
291
315
|
const sessionEventAppender = options.sessionEventsPath
|
|
292
316
|
? createSessionEventAppender(options.sessionEventsPath, options.onSessionEvent)
|
|
293
317
|
: undefined;
|
|
318
|
+
const appendStderr = (message) => {
|
|
319
|
+
stderr = stderr ? `${stderr}\n${message}` : message;
|
|
320
|
+
};
|
|
321
|
+
const runSessionActionWithGrace = async (label, action) => {
|
|
322
|
+
let graceHandle;
|
|
323
|
+
const outcome = await Promise.race([
|
|
324
|
+
Promise.resolve()
|
|
325
|
+
.then(action)
|
|
326
|
+
.then(() => ({ ok: true }))
|
|
327
|
+
.catch((error) => ({ ok: false, error })),
|
|
328
|
+
new Promise((resolve) => {
|
|
329
|
+
graceHandle = setTimeout(() => resolve({ ok: false, timedOut: true }), abortGraceMs);
|
|
330
|
+
}),
|
|
331
|
+
]);
|
|
332
|
+
if (graceHandle)
|
|
333
|
+
clearTimeout(graceHandle);
|
|
334
|
+
if (outcome.ok)
|
|
335
|
+
return true;
|
|
336
|
+
if ('timedOut' in outcome) {
|
|
337
|
+
appendStderr(`pi SDK ${label} was not confirmed within ${abortGraceMs}ms`);
|
|
338
|
+
}
|
|
339
|
+
else {
|
|
340
|
+
const message = outcome.error instanceof Error ? outcome.error.message : String(outcome.error);
|
|
341
|
+
appendStderr(`pi SDK ${label} failed: ${message}`);
|
|
342
|
+
}
|
|
343
|
+
return false;
|
|
344
|
+
};
|
|
294
345
|
try {
|
|
295
346
|
const createSession = await resolveSdkSessionFactory(options.reuseScope);
|
|
296
347
|
session = await createSession({
|
|
@@ -301,7 +352,32 @@ export async function executeSingleSdkAttempt(options) {
|
|
|
301
352
|
model: modelConfig.model,
|
|
302
353
|
thinking: modelConfig.thinking,
|
|
303
354
|
});
|
|
304
|
-
|
|
355
|
+
let resolveStall;
|
|
356
|
+
const stallPromise = stallTimeoutMs > 0
|
|
357
|
+
? new Promise((resolve) => {
|
|
358
|
+
resolveStall = resolve;
|
|
359
|
+
})
|
|
360
|
+
: null;
|
|
361
|
+
const armStallWatchdog = () => {
|
|
362
|
+
if (!stallPromise || !resolveStall)
|
|
363
|
+
return;
|
|
364
|
+
if (stallHandle)
|
|
365
|
+
clearTimeout(stallHandle);
|
|
366
|
+
stallHandle = setTimeout(() => resolveStall?.('stall'), stallTimeoutMs);
|
|
367
|
+
};
|
|
368
|
+
unsubscribe = session.subscribe((event) => {
|
|
369
|
+
// Every real SDK event proves transport activity, including noisy deltas
|
|
370
|
+
// intentionally excluded from persisted JSONL and meaningful DAG progress.
|
|
371
|
+
armStallWatchdog();
|
|
372
|
+
const activityKind = classifySdkActivityKind(event);
|
|
373
|
+
if (activityKind) {
|
|
374
|
+
try {
|
|
375
|
+
options.onActivity?.({ kind: activityKind, at: new Date().toISOString() });
|
|
376
|
+
}
|
|
377
|
+
catch {
|
|
378
|
+
// best-effort: activity must never change Pi result
|
|
379
|
+
}
|
|
380
|
+
}
|
|
305
381
|
const usageSample = extractSdkUsageSample(event);
|
|
306
382
|
if (usageSample)
|
|
307
383
|
usageSamples.push(usageSample);
|
|
@@ -317,44 +393,44 @@ export async function executeSingleSdkAttempt(options) {
|
|
|
317
393
|
? `${filePrefix}\n${options.userMessage}`
|
|
318
394
|
: options.userMessage;
|
|
319
395
|
const promptPromise = session.prompt(promptMessage);
|
|
396
|
+
armStallWatchdog();
|
|
320
397
|
const timeoutPromise = timeoutMs > 0
|
|
321
398
|
? new Promise((resolve) => {
|
|
322
399
|
timeoutHandle = setTimeout(() => {
|
|
323
|
-
|
|
324
|
-
void session?.abort();
|
|
325
|
-
resolve('timeout');
|
|
400
|
+
resolve('absolute-timeout');
|
|
326
401
|
}, timeoutMs);
|
|
327
402
|
})
|
|
328
403
|
: null;
|
|
329
|
-
|
|
330
|
-
|
|
331
|
-
|
|
332
|
-
|
|
333
|
-
|
|
334
|
-
|
|
335
|
-
|
|
336
|
-
|
|
404
|
+
const raced = await Promise.race([
|
|
405
|
+
promptPromise.then(() => 'done'),
|
|
406
|
+
...(timeoutPromise ? [timeoutPromise] : []),
|
|
407
|
+
...(stallPromise ? [stallPromise] : []),
|
|
408
|
+
]);
|
|
409
|
+
if (raced !== 'done') {
|
|
410
|
+
timedOut = true;
|
|
411
|
+
appendStderr(raced === 'stall'
|
|
412
|
+
? `pi SDK step stalled after ${stallTimeoutMs}ms with no provider activity`
|
|
413
|
+
: `pi SDK step timed out after ${timeoutMs}ms`);
|
|
414
|
+
const abortConfirmed = await runSessionActionWithGrace('abort', () => session.abort());
|
|
415
|
+
disposeAttempted = true;
|
|
416
|
+
const disposeConfirmed = await runSessionActionWithGrace('dispose', () => session.dispose());
|
|
417
|
+
terminationConfirmed = abortConfirmed && disposeConfirmed;
|
|
337
418
|
}
|
|
338
|
-
else {
|
|
339
|
-
await promptPromise;
|
|
340
|
-
}
|
|
341
|
-
unsubscribe();
|
|
342
419
|
}
|
|
343
420
|
catch (error) {
|
|
344
421
|
const message = error instanceof Error ? error.message : String(error);
|
|
345
|
-
|
|
422
|
+
appendStderr(message);
|
|
346
423
|
}
|
|
347
424
|
finally {
|
|
348
425
|
if (timeoutHandle)
|
|
349
426
|
clearTimeout(timeoutHandle);
|
|
350
|
-
if (
|
|
351
|
-
|
|
352
|
-
|
|
353
|
-
|
|
354
|
-
|
|
355
|
-
|
|
356
|
-
|
|
357
|
-
}
|
|
427
|
+
if (stallHandle)
|
|
428
|
+
clearTimeout(stallHandle);
|
|
429
|
+
unsubscribe?.();
|
|
430
|
+
if (session && !disposeAttempted) {
|
|
431
|
+
disposeAttempted = true;
|
|
432
|
+
const disposeConfirmed = await runSessionActionWithGrace('dispose', () => session.dispose());
|
|
433
|
+
terminationConfirmed = terminationConfirmed && disposeConfirmed;
|
|
358
434
|
}
|
|
359
435
|
if (sessionEventAppender) {
|
|
360
436
|
try {
|
|
@@ -378,14 +454,14 @@ export async function executeSingleSdkAttempt(options) {
|
|
|
378
454
|
? collected.parsedEvents
|
|
379
455
|
: fallbackParsed?.parsedEvents ?? 0;
|
|
380
456
|
const tokensUsed = aggregateSdkTokenUsage(usageSamples);
|
|
381
|
-
const failureCategory = classifyPiFailure({
|
|
457
|
+
const failureCategory = terminationConfirmed ? classifyPiFailure({
|
|
382
458
|
assistantText,
|
|
383
459
|
exitCode: timedOut ? 1 : stderr ? 1 : 0,
|
|
384
460
|
outputTooLarge: collected.outputTooLarge,
|
|
385
461
|
stderr,
|
|
386
462
|
stdout,
|
|
387
463
|
timedOut,
|
|
388
|
-
});
|
|
464
|
+
}) : 'termination-unconfirmed';
|
|
389
465
|
return {
|
|
390
466
|
assistantText,
|
|
391
467
|
backend: 'sdk',
|
|
@@ -394,7 +470,7 @@ export async function executeSingleSdkAttempt(options) {
|
|
|
394
470
|
exitCode: timedOut || stderr ? 1 : 0,
|
|
395
471
|
failureCategory,
|
|
396
472
|
modelDisplay,
|
|
397
|
-
ok: !timedOut && !stderr && assistantText.length > 0 && !collected.outputTooLarge,
|
|
473
|
+
ok: terminationConfirmed && !timedOut && !stderr && assistantText.length > 0 && !collected.outputTooLarge,
|
|
398
474
|
parsedEvents,
|
|
399
475
|
stderr,
|
|
400
476
|
stdout,
|
|
@@ -17,7 +17,7 @@ import { formatFrontendReviewContextStdout, runFrontendReviewContextGate, } from
|
|
|
17
17
|
import { formatTraceabilityGateStdout, materializeBackendTestCaseManifest, runBackendTestTraceabilityGate, } from "../workflows/dag/backend-test-case-manifest.js";
|
|
18
18
|
import { materializeBackendTestExecutionContract } from "../workflows/dag/backend-test-execution-contract.js";
|
|
19
19
|
import { materializeBackendTestResultFromRunDir, parseJunitXml } from "../workflows/dag/backend-test-result-contract.js";
|
|
20
|
-
import { collectBackendTestHumanCaseCatalog, inspectBackendTestEnvironment, requiredBackendMarkdownCaseAcIds, renderBackendTestFacts, renderBackendTestHtml, redactBackendTestOutput, validateBackendMarkdownCases, validateBackendMarkdownTraceability, writeRunReport, } from "../workflows/dag/backend-test-markdown-workflow.js";
|
|
20
|
+
import { collectBackendTestHumanCaseCatalog, collectBackendTestMappedPytestScripts, hasBlockingBackendMarkdownSafetyFindings, inspectBackendTestEnvironment, requiredBackendMarkdownCaseAcIds, renderBackendTestFacts, renderBackendTestHtml, redactBackendTestOutput, validateBackendMarkdownCases, validateBackendMarkdownTraceability, writeRunReport, } from "../workflows/dag/backend-test-markdown-workflow.js";
|
|
21
21
|
import { buildBackendTestCanonicalResultFromInitialShellSnippet, materializeBackendTestClassification, } from "../workflows/dag/backend-test-classification-contract.js";
|
|
22
22
|
import { backendTestSemanticReviewSchema, materializeBackendTestSemanticReview, } from "../workflows/dag/backend-test-semantic-review-contract.js";
|
|
23
23
|
import { pathsChangedDuringRun, readGitStatusPorcelain, snapshotGitStatusPorcelain, validateShellWriteGuard, } from "./shell-write-guard.js";
|
|
@@ -304,6 +304,14 @@ async function runShellWriteGuard(input) {
|
|
|
304
304
|
forbiddenPaths: input.task.forbiddenPaths,
|
|
305
305
|
});
|
|
306
306
|
}
|
|
307
|
+
async function readRequiredRunReport(reportsDir, filename) {
|
|
308
|
+
try {
|
|
309
|
+
return await readFile(path.join(reportsDir, filename), "utf8");
|
|
310
|
+
}
|
|
311
|
+
catch {
|
|
312
|
+
throw new Error(`missing required upstream report: reports/${filename}`);
|
|
313
|
+
}
|
|
314
|
+
}
|
|
307
315
|
async function executeBackendTestPipeline(input, meta) {
|
|
308
316
|
const pipeline = input.task.shell?.backendTestPipeline;
|
|
309
317
|
const started = Date.now();
|
|
@@ -332,22 +340,54 @@ async function executeBackendTestPipeline(input, meta) {
|
|
|
332
340
|
meta.spec.objective ?? "",
|
|
333
341
|
...(meta.spec.successCriteria ?? []),
|
|
334
342
|
]);
|
|
335
|
-
|
|
336
|
-
|
|
337
|
-
|
|
338
|
-
|
|
339
|
-
|
|
340
|
-
|
|
343
|
+
let report;
|
|
344
|
+
try {
|
|
345
|
+
report = await validateBackendMarkdownCases({
|
|
346
|
+
workspaceRoot: input.cwd,
|
|
347
|
+
environmentMarkdown,
|
|
348
|
+
requiredRequirementIds,
|
|
349
|
+
sourceBinding: meta.spec.sourceBinding,
|
|
350
|
+
});
|
|
351
|
+
}
|
|
352
|
+
catch (error) {
|
|
353
|
+
report = `# Backend Markdown Case Validation\n\n## Status\n\nFAIL\n\n## Findings\n\n- Validator could not complete: ${error instanceof Error ? error.message : String(error)}\n`;
|
|
354
|
+
}
|
|
341
355
|
const reportPath = await writeRunReport(meta.runDir, "backend-md-case-validation.md", report);
|
|
342
356
|
outputs.push(`caseValidation=${reportPath}`, report);
|
|
357
|
+
if (hasBlockingBackendMarkdownSafetyFindings(report)) {
|
|
358
|
+
return {
|
|
359
|
+
ok: false,
|
|
360
|
+
stdout: outputs.join("\n"),
|
|
361
|
+
stderr: "backend Markdown safety gate blocked: secret-shaped values detected",
|
|
362
|
+
failureCategory: "invalid-output",
|
|
363
|
+
durationMs: Date.now() - started,
|
|
364
|
+
};
|
|
365
|
+
}
|
|
343
366
|
}
|
|
344
367
|
else if (pipeline === "markdown-traceability") {
|
|
345
|
-
|
|
368
|
+
let report;
|
|
369
|
+
try {
|
|
370
|
+
report = await validateBackendMarkdownTraceability(input.cwd);
|
|
371
|
+
}
|
|
372
|
+
catch (error) {
|
|
373
|
+
report = `# Backend Test Traceability\n\n## Status\n\nFAIL\n\n## Findings\n\n- Traceability validation could not complete: ${error instanceof Error ? error.message : String(error)}\n`;
|
|
374
|
+
}
|
|
346
375
|
const reportPath = await writeRunReport(meta.runDir, "backend-test-traceability.md", report);
|
|
347
376
|
outputs.push(`traceability=${reportPath}`, report);
|
|
348
377
|
}
|
|
349
378
|
else if (pipeline === "markdown-execute-html") {
|
|
350
|
-
const
|
|
379
|
+
const mappedScripts = await collectBackendTestMappedPytestScripts(input.cwd);
|
|
380
|
+
const shellQuote = (value) => `'${value.replaceAll("'", `'"'"'`)}'`;
|
|
381
|
+
const pytestTargets = mappedScripts.map(shellQuote).join(" ");
|
|
382
|
+
const pytestCommand = [
|
|
383
|
+
'mkdir -p "${HARNESS_DAG_RUN_DIR}/reports"',
|
|
384
|
+
`PYTHONUTF8=1 PYTHONIOENCODING=utf-8 PYTHONDONTWRITEBYTECODE=1 python -m pytest ${pytestTargets} -v -p no:cacheprovider -o junit_logging=all -o junit_log_passing_tests=true --junitxml="\${HARNESS_DAG_RUN_DIR}/reports/backend-test.junit.xml"`,
|
|
385
|
+
"STATUS=$?",
|
|
386
|
+
'printf "%s" "${STATUS}" > "${HARNESS_DAG_RUN_DIR}/reports/backend-test-pytest-exit.txt"',
|
|
387
|
+
'if { [ "${STATUS}" -eq 0 ] || [ "${STATUS}" -eq 1 ]; } && [ -s "${HARNESS_DAG_RUN_DIR}/reports/backend-test.junit.xml" ]; then exit 0; fi',
|
|
388
|
+
'exit "${STATUS}"',
|
|
389
|
+
].join("; ");
|
|
390
|
+
const results = await executePipelineCommands(input, meta, [pytestCommand]);
|
|
351
391
|
if (!results.every((result) => result.ok)) {
|
|
352
392
|
const failure = results.find((result) => !result.ok);
|
|
353
393
|
return { ok: false, stdout: results.map((result) => result.stdout).join("\n"), stderr: failure.stderr, failureCategory: failure.failureCategory, durationMs: Date.now() - started };
|
|
@@ -359,15 +399,18 @@ async function executeBackendTestPipeline(input, meta) {
|
|
|
359
399
|
throw new Error(`pytest did not complete with a reportable exit code: ${pytestExitCode}`);
|
|
360
400
|
const parsed = parseJunitXml(junitContent);
|
|
361
401
|
const cases = await collectBackendTestHumanCaseCatalog(input.cwd);
|
|
402
|
+
const caseValidationSummary = await readRequiredRunReport(reportsDir, "backend-md-case-validation.md");
|
|
403
|
+
const traceabilitySummary = await readRequiredRunReport(reportsDir, "backend-test-traceability.md");
|
|
362
404
|
const htmlContent = renderBackendTestHtml({
|
|
363
405
|
title: meta.spec.title,
|
|
364
406
|
parsed,
|
|
365
407
|
cases,
|
|
366
408
|
environmentSummary: await readFile(path.join(reportsDir, "backend-test-environment.md"), "utf8"),
|
|
367
|
-
|
|
409
|
+
caseValidationSummary,
|
|
410
|
+
traceabilitySummary,
|
|
368
411
|
});
|
|
369
412
|
const htmlPath = await writeRunReport(meta.runDir, "backend-test.html", htmlContent);
|
|
370
|
-
const facts = renderBackendTestFacts({ parsed, cases, pytestExitCode, junitRelativePath: "reports/backend-test.junit.xml", htmlRelativePath: "reports/backend-test.html", junitContent, htmlContent });
|
|
413
|
+
const facts = renderBackendTestFacts({ parsed, cases, pytestExitCode, junitRelativePath: "reports/backend-test.junit.xml", htmlRelativePath: "reports/backend-test.html", junitContent, htmlContent, caseValidationSummary, traceabilitySummary });
|
|
371
414
|
const factsPath = await writeRunReport(meta.runDir, "backend-test-facts.md", facts);
|
|
372
415
|
const sanitizedOutputs = results.map((result) => redactBackendTestOutput(result.stdout));
|
|
373
416
|
outputs.push(...sanitizedOutputs, `html=${htmlPath}`, `facts=${factsPath}`, facts);
|