agent-workflow-inspector 0.0.0-stage → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +23 -0
- package/CODE_OF_CONDUCT.md +5 -0
- package/CONTRIBUTING.md +9 -0
- package/LICENSE +21 -0
- package/README.md +341 -2
- package/SECURITY.md +5 -0
- package/awi.config.example.json +39 -0
- package/dist/cli/index.d.ts +1 -0
- package/dist/cli/index.js +1338 -0
- package/dist/cli/index.js.map +1 -0
- package/dist/index.d.ts +256 -0
- package/dist/index.js +1159 -0
- package/dist/index.js.map +1 -0
- package/docs/architecture.md +21 -0
- package/docs/assets/architecture.svg +139 -0
- package/docs/assets/banner.svg +105 -0
- package/docs/assets/detection-patterns.svg +134 -0
- package/docs/assets/terminal-preview.svg +93 -0
- package/docs/audit.md +91 -0
- package/docs/ci-integration.md +45 -0
- package/docs/configuration.md +17 -0
- package/docs/detection-rules.md +38 -0
- package/docs/getting-started.md +70 -0
- package/docs/limitations.md +15 -0
- package/docs/releases/v1.0.0.md +32 -0
- package/docs/releasing.md +23 -0
- package/docs/report-schema.md +29 -0
- package/docs/schemas/config.schema.json +183 -0
- package/docs/schemas/report.schema.json +158 -0
- package/docs/schemas/trace.schema.json +98 -0
- package/docs/security.md +34 -0
- package/docs/trace-formats.md +41 -0
- package/docs/troubleshooting.md +12 -0
- package/docs/verification.md +44 -0
- package/examples/argument-drift.config.json +9 -0
- package/examples/healthy/basic.json +30 -0
- package/examples/opentelemetry/tools.json +210 -0
- package/examples/suspicious/argument-drift.json +62 -0
- package/examples/suspicious/cycle.json +78 -0
- package/examples/suspicious/retries.json +116 -0
- package/package.json +78 -3
|
@@ -0,0 +1,134 @@
|
|
|
1
|
+
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 940 320" width="100%" height="100%">
|
|
2
|
+
<defs>
|
|
3
|
+
<linearGradient id="bg2" x1="0%" y1="0%" x2="100%" y2="100%">
|
|
4
|
+
<stop offset="0%" stop-color="#0d1117"/>
|
|
5
|
+
<stop offset="100%" stop-color="#161b22"/>
|
|
6
|
+
</linearGradient>
|
|
7
|
+
<linearGradient id="pCardGrad" x1="0%" y1="0%" x2="0%" y2="100%">
|
|
8
|
+
<stop offset="0%" stop-color="#1c2128"/>
|
|
9
|
+
<stop offset="100%" stop-color="#12161c"/>
|
|
10
|
+
</linearGradient>
|
|
11
|
+
<filter id="pCardShadow" x="-5%" y="-5%" width="110%" height="110%">
|
|
12
|
+
<feDropShadow dx="0" dy="4" stdDeviation="8" flood-color="#000000" flood-opacity="0.35"/>
|
|
13
|
+
</filter>
|
|
14
|
+
</defs>
|
|
15
|
+
|
|
16
|
+
<style>
|
|
17
|
+
.font-sans { font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif; }
|
|
18
|
+
.font-mono { font-family: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, monospace; }
|
|
19
|
+
</style>
|
|
20
|
+
|
|
21
|
+
<rect width="940" height="320" rx="14" fill="url(#bg2)" stroke="#30363d" stroke-width="1"/>
|
|
22
|
+
|
|
23
|
+
<!-- Pattern 1: Retry Storm & Repeated Errors -->
|
|
24
|
+
<g transform="translate(24, 25)" filter="url(#pCardShadow)">
|
|
25
|
+
<rect width="206" height="270" rx="10" fill="url(#pCardGrad)" stroke="#f85149" stroke-width="1.2"/>
|
|
26
|
+
<rect width="206" height="38" rx="10" fill="#2d1519"/>
|
|
27
|
+
<rect y="28" width="206" height="10" fill="#2d1519"/>
|
|
28
|
+
<text x="14" y="24" class="font-sans" font-size="12" font-weight="700" fill="#ff7b72">AWI002 & AWI004</text>
|
|
29
|
+
<text x="14" y="58" class="font-sans" font-size="13" font-weight="700" fill="#f0f6fc">Retry Storms</text>
|
|
30
|
+
<text x="14" y="76" class="font-sans" font-size="11" fill="#8b949e">Repeated failing calls</text>
|
|
31
|
+
|
|
32
|
+
<!-- Visual Sequence -->
|
|
33
|
+
<g transform="translate(14, 96)">
|
|
34
|
+
<rect width="178" height="30" rx="4" fill="#381a1f" stroke="#f85149" stroke-width="0.8"/>
|
|
35
|
+
<text x="10" y="20" class="font-mono" font-size="11" fill="#ff7b72">tool(search) ❌ FAIL</text>
|
|
36
|
+
|
|
37
|
+
<path d="M 89 32 L 89 42" stroke="#f85149" stroke-width="1.5" stroke-dasharray="2 2"/>
|
|
38
|
+
|
|
39
|
+
<rect y="44" width="178" height="30" rx="4" fill="#381a1f" stroke="#f85149" stroke-width="0.8"/>
|
|
40
|
+
<text x="10" y="64" class="font-mono" font-size="11" fill="#ff7b72">tool(search) ❌ FAIL</text>
|
|
41
|
+
|
|
42
|
+
<path d="M 89 76 L 89 86" stroke="#f85149" stroke-width="1.5" stroke-dasharray="2 2"/>
|
|
43
|
+
|
|
44
|
+
<rect y="88" width="178" height="30" rx="4" fill="#381a1f" stroke="#f85149" stroke-width="0.8"/>
|
|
45
|
+
<text x="10" y="108" class="font-mono" font-size="11" fill="#ff7b72">tool(search) ❌ FAIL</text>
|
|
46
|
+
|
|
47
|
+
<text x="0" y="146" class="font-sans" font-size="10.5" fill="#fca5a5">💥 Rapid consecutive errors with identical arguments.</text>
|
|
48
|
+
</g>
|
|
49
|
+
</g>
|
|
50
|
+
|
|
51
|
+
<!-- Pattern 2: Execution Cycles -->
|
|
52
|
+
<g transform="translate(252, 25)" filter="url(#pCardShadow)">
|
|
53
|
+
<rect width="206" height="270" rx="10" fill="url(#pCardGrad)" stroke="#d29922" stroke-width="1.2"/>
|
|
54
|
+
<rect width="206" height="38" rx="10" fill="#2f2413"/>
|
|
55
|
+
<rect y="28" width="206" height="10" fill="#2f2413"/>
|
|
56
|
+
<text x="14" y="24" class="font-sans" font-size="12" font-weight="700" fill="#e3b341">AWI003</text>
|
|
57
|
+
<text x="14" y="58" class="font-sans" font-size="13" font-weight="700" fill="#f0f6fc">Execution Cycles</text>
|
|
58
|
+
<text x="14" y="76" class="font-sans" font-size="11" fill="#8b949e">Periodic tool loops (A-B-C-A)</text>
|
|
59
|
+
|
|
60
|
+
<!-- Visual Sequence -->
|
|
61
|
+
<g transform="translate(14, 96)">
|
|
62
|
+
<rect width="178" height="26" rx="4" fill="#1c2433" stroke="#58a6ff" stroke-width="0.8"/>
|
|
63
|
+
<text x="10" y="17" class="font-mono" font-size="10.5" fill="#79c0ff">1. Tool: read_file</text>
|
|
64
|
+
|
|
65
|
+
<rect y="32" width="178" height="26" rx="4" fill="#1c2433" stroke="#bc8cff" stroke-width="0.8"/>
|
|
66
|
+
<text x="10" y="49" class="font-mono" font-size="10.5" fill="#d2a8ff">2. Tool: edit_code</text>
|
|
67
|
+
|
|
68
|
+
<rect y="64" width="178" height="26" rx="4" fill="#1c2433" stroke="#3fb950" stroke-width="0.8"/>
|
|
69
|
+
<text x="10" y="81" class="font-mono" font-size="10.5" fill="#7ee787">3. Tool: run_test</text>
|
|
70
|
+
|
|
71
|
+
<!-- Loop Indicator -->
|
|
72
|
+
<path d="M 184 77 Q 198 48 184 20" fill="none" stroke="#d29922" stroke-width="2"/>
|
|
73
|
+
<polygon points="184,17 188,24 180,24" fill="#d29922"/>
|
|
74
|
+
|
|
75
|
+
<text x="0" y="122" class="font-sans" font-size="11" font-weight="600" fill="#e3b341">🔁 Repeated 3+ times</text>
|
|
76
|
+
<text x="0" y="146" class="font-sans" font-size="10.5" fill="#fde047">Agent trapped in circular non-converging logic.</text>
|
|
77
|
+
</g>
|
|
78
|
+
</g>
|
|
79
|
+
|
|
80
|
+
<!-- Pattern 3: Excessive Depth & Complexity -->
|
|
81
|
+
<g transform="translate(480, 25)" filter="url(#pCardShadow)">
|
|
82
|
+
<rect width="206" height="270" rx="10" fill="url(#pCardGrad)" stroke="#a371f7" stroke-width="1.2"/>
|
|
83
|
+
<rect width="206" height="38" rx="10" fill="#241936"/>
|
|
84
|
+
<rect y="28" width="206" height="10" fill="#241936"/>
|
|
85
|
+
<text x="14" y="24" class="font-sans" font-size="12" font-weight="700" fill="#d2a8ff">AWI005</text>
|
|
86
|
+
<text x="14" y="58" class="font-sans" font-size="13" font-weight="700" fill="#f0f6fc">Runaway Tree</text>
|
|
87
|
+
<text x="14" y="76" class="font-sans" font-size="11" fill="#8b949e">Excess depth or operations</text>
|
|
88
|
+
|
|
89
|
+
<!-- Visual Tree -->
|
|
90
|
+
<g transform="translate(14, 96)">
|
|
91
|
+
<rect width="178" height="22" rx="4" fill="#261b3d" stroke="#8957e5"/>
|
|
92
|
+
<text x="8" y="15" class="font-mono" font-size="10" fill="#d2a8ff">Root Coordinator</text>
|
|
93
|
+
|
|
94
|
+
<g transform="translate(14, 28)">
|
|
95
|
+
<path d="M 0 0 L 0 14 L 10 14" stroke="#8957e5" stroke-width="1.5" fill="none"/>
|
|
96
|
+
<rect x="12" width="152" height="22" rx="4" fill="#261b3d" stroke="#8957e5"/>
|
|
97
|
+
<text x="20" y="15" class="font-mono" font-size="10" fill="#d2a8ff">Subagent A</text>
|
|
98
|
+
</g>
|
|
99
|
+
|
|
100
|
+
<g transform="translate(28, 56)">
|
|
101
|
+
<path d="M 0 0 L 0 14 L 10 14" stroke="#8957e5" stroke-width="1.5" fill="none"/>
|
|
102
|
+
<rect x="12" width="138" height="22" rx="4" fill="#261b3d" stroke="#8957e5"/>
|
|
103
|
+
<text x="20" y="15" class="font-mono" font-size="10" fill="#d2a8ff">Sub-task ... [Depth > 20]</text>
|
|
104
|
+
</g>
|
|
105
|
+
|
|
106
|
+
<text x="0" y="122" class="font-mono" font-size="11" font-weight="600" fill="#d2a8ff">> 100 total operations</text>
|
|
107
|
+
<text x="0" y="146" class="font-sans" font-size="10.5" fill="#e9d5ff">Catches exponential subagent recursion bugs.</text>
|
|
108
|
+
</g>
|
|
109
|
+
</g>
|
|
110
|
+
|
|
111
|
+
<!-- Pattern 4: Resource & Cost Burn -->
|
|
112
|
+
<g transform="translate(708, 25)" filter="url(#pCardShadow)">
|
|
113
|
+
<rect width="206" height="270" rx="10" fill="url(#pCardGrad)" stroke="#3fb950" stroke-width="1.2"/>
|
|
114
|
+
<rect width="206" height="38" rx="10" fill="#142b1a"/>
|
|
115
|
+
<rect y="28" width="206" height="10" fill="#142b1a"/>
|
|
116
|
+
<text x="14" y="24" class="font-sans" font-size="12" font-weight="700" fill="#7ee787">AWI006</text>
|
|
117
|
+
<text x="14" y="58" class="font-sans" font-size="13" font-weight="700" fill="#f0f6fc">Resource Burn</text>
|
|
118
|
+
<text x="14" y="76" class="font-sans" font-size="11" fill="#8b949e">Tokens & cost pinned to errors</text>
|
|
119
|
+
|
|
120
|
+
<!-- Visual Burn -->
|
|
121
|
+
<g transform="translate(14, 96)">
|
|
122
|
+
<rect width="178" height="38" rx="4" fill="#132317" stroke="#238636"/>
|
|
123
|
+
<text x="10" y="18" class="font-mono" font-size="11" fill="#7ee787">Wasted Tokens: +84,200</text>
|
|
124
|
+
<text x="10" y="32" class="font-sans" font-size="10" fill="#8b949e">Pinned to culprit leaves</text>
|
|
125
|
+
|
|
126
|
+
<rect y="46" width="178" height="38" rx="4" fill="#132317" stroke="#238636"/>
|
|
127
|
+
<text x="10" y="66" class="font-mono" font-size="11" fill="#7ee787">Wasted Cost: $4.21 USD</text>
|
|
128
|
+
<text x="10" y="80" class="font-sans" font-size="10" fill="#8b949e">Direct currency tracking</text>
|
|
129
|
+
|
|
130
|
+
<text x="0" y="122" class="font-sans" font-size="11" font-weight="600" fill="#7ee787">💰 Leaf Attribution</text>
|
|
131
|
+
<text x="0" y="146" class="font-sans" font-size="10.5" fill="#bbf7d0">Pinpoints financial waste directly to flawed loops.</text>
|
|
132
|
+
</g>
|
|
133
|
+
</g>
|
|
134
|
+
</svg>
|
|
@@ -0,0 +1,93 @@
|
|
|
1
|
+
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 920 540" width="100%" height="100%">
|
|
2
|
+
<defs>
|
|
3
|
+
<linearGradient id="terminalBg" x1="0%" y1="0%" x2="0%" y2="100%">
|
|
4
|
+
<stop offset="0%" stop-color="#161b22"/>
|
|
5
|
+
<stop offset="100%" stop-color="#0d1117"/>
|
|
6
|
+
</linearGradient>
|
|
7
|
+
<filter id="shadow" x="-5%" y="-5%" width="110%" height="110%">
|
|
8
|
+
<feDropShadow dx="0" dy="12" stdDeviation="16" flood-color="#000000" flood-opacity="0.5"/>
|
|
9
|
+
</filter>
|
|
10
|
+
</defs>
|
|
11
|
+
|
|
12
|
+
<style>
|
|
13
|
+
.window { fill: url(#terminalBg); stroke: #30363d; stroke-width: 1; rx: 10px; }
|
|
14
|
+
.header { fill: #161b22; stroke: #30363d; stroke-width: 1; }
|
|
15
|
+
.title { fill: #8b949e; font-family: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, monospace; font-size: 13px; font-weight: 500; text-anchor: middle; }
|
|
16
|
+
.text { font-family: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, monospace; font-size: 13px; line-height: 20px; }
|
|
17
|
+
.cmd { fill: #58a6ff; font-weight: 600; }
|
|
18
|
+
.prompt { fill: #7ee787; font-weight: 600; }
|
|
19
|
+
.dim { fill: #8b949e; }
|
|
20
|
+
.info { fill: #c9d1d9; }
|
|
21
|
+
.high-badge { fill: #f85149; font-weight: bold; }
|
|
22
|
+
.high-title { fill: #ff7b72; font-weight: 600; }
|
|
23
|
+
.med-badge { fill: #d29922; font-weight: bold; }
|
|
24
|
+
.med-title { fill: #e3b341; font-weight: 600; }
|
|
25
|
+
.desc { fill: #8b949e; }
|
|
26
|
+
.evidence { fill: #79c0ff; }
|
|
27
|
+
.summary { fill: #f85149; font-weight: 600; }
|
|
28
|
+
.exit-code { fill: #ff7b72; font-weight: bold; }
|
|
29
|
+
</style>
|
|
30
|
+
|
|
31
|
+
<g filter="url(#shadow)">
|
|
32
|
+
<!-- Terminal Window -->
|
|
33
|
+
<rect x="20" y="20" width="880" height="500" rx="10" class="window" />
|
|
34
|
+
|
|
35
|
+
<!-- Title Bar -->
|
|
36
|
+
<path d="M 20 30 Q 20 20 30 20 L 890 20 Q 900 20 900 30 L 900 56 L 20 56 Z" fill="#161b22" stroke="#30363d" stroke-width="1" />
|
|
37
|
+
<line x1="20" y1="56" x2="900" y2="56" stroke="#30363d" stroke-width="1"/>
|
|
38
|
+
|
|
39
|
+
<!-- Mac-style Window Controls -->
|
|
40
|
+
<circle cx="42" cy="38" r="6" fill="#ff5f56" stroke="#e0443e" stroke-width="0.5"/>
|
|
41
|
+
<circle cx="62" cy="38" r="6" fill="#ffbd2e" stroke="#dea123" stroke-width="0.5"/>
|
|
42
|
+
<circle cx="82" cy="38" r="6" fill="#27c93f" stroke="#1aab29" stroke-width="0.5"/>
|
|
43
|
+
|
|
44
|
+
<!-- Title -->
|
|
45
|
+
<text x="460" y="42" class="title">awi inspect examples/suspicious/retries.json</text>
|
|
46
|
+
|
|
47
|
+
<!-- Terminal Content -->
|
|
48
|
+
<g transform="translate(42, 86)">
|
|
49
|
+
<!-- Command line -->
|
|
50
|
+
<text x="0" y="0" class="text">
|
|
51
|
+
<tspan class="prompt">$ </tspan>
|
|
52
|
+
<tspan class="cmd">awi inspect examples/suspicious/retries.json</tspan>
|
|
53
|
+
</text>
|
|
54
|
+
|
|
55
|
+
<text x="0" y="26" class="text info">Agent Workflow Inspector — Find loops, retries, and wasted work in AI agent traces.</text>
|
|
56
|
+
<text x="0" y="46" class="text dim">Trace retries: 5 operations, 5 tool calls, depth 1; duration 9000 ms; recorded leaf usage {"inputTokens":50}</text>
|
|
57
|
+
|
|
58
|
+
<!-- Finding 1: AWI002 -->
|
|
59
|
+
<text x="0" y="80" class="text">
|
|
60
|
+
<tspan class="high-badge">[HIGH] </tspan>
|
|
61
|
+
<tspan class="high-title">AWI002 Repeated identical failing tool call (high evidence)</tspan>
|
|
62
|
+
</text>
|
|
63
|
+
<text x="16" y="100" class="text dim">Trace: retries; operations: op-0, op-1, op-2, op-3, op-4</text>
|
|
64
|
+
<text x="16" y="120" class="text info">Equivalent recorded tool calls failed with matching error evidence.</text>
|
|
65
|
+
<text x="16" y="140" class="text evidence">Evidence: {"occurrences":5,"identicalArguments":true,"equivalentFailureOutcomes":true}</text>
|
|
66
|
+
|
|
67
|
+
<!-- Finding 2: AWI004 -->
|
|
68
|
+
<text x="0" y="174" class="text">
|
|
69
|
+
<tspan class="high-badge">[HIGH] </tspan>
|
|
70
|
+
<tspan class="high-title">AWI004 Retry storm (moderate evidence)</tspan>
|
|
71
|
+
</text>
|
|
72
|
+
<text x="16" y="194" class="text dim">Trace: retries; operations: op-0, op-1, op-2, op-3, op-4</text>
|
|
73
|
+
<text x="16" y="214" class="text info">Rapid equivalent attempts follow explicit failures in a 30s window.</text>
|
|
74
|
+
<text x="16" y="234" class="text evidence">Evidence: {"attempts":5,"windowSeconds":30,"priorAttemptsExplicitlyFailed":true}</text>
|
|
75
|
+
|
|
76
|
+
<!-- Finding 3: AWI006 -->
|
|
77
|
+
<text x="0" y="268" class="text">
|
|
78
|
+
<tspan class="med-badge">[MEDIUM] </tspan>
|
|
79
|
+
<tspan class="med-title">AWI006 Recorded resource usage associated with suspicious execution</tspan>
|
|
80
|
+
</text>
|
|
81
|
+
<text x="16" y="288" class="text dim">Trace: retries; operations: op-0, op-1, op-2, op-3, op-4</text>
|
|
82
|
+
<text x="16" y="308" class="text info">Recorded resources belong directly to operations in suspicious patterns.</text>
|
|
83
|
+
<text x="16" y="328" class="text evidence">Evidence: {"recordedUsage":{"inputTokens":50},"recordedCosts":{"USD":0.005},"relatedRules":["AWI002","AWI004"]}</text>
|
|
84
|
+
|
|
85
|
+
<!-- Footer summary -->
|
|
86
|
+
<line x1="0" y1="354" x2="836" y2="354" stroke="#21262d" stroke-width="1"/>
|
|
87
|
+
<text x="0" y="380" class="text">
|
|
88
|
+
<tspan class="summary">✖ 3 findings (2 high, 1 medium); analysis complete.</tspan>
|
|
89
|
+
<tspan class="dim"> [Exit code: </tspan><tspan class="exit-code">2</tspan><tspan class="dim">]</tspan>
|
|
90
|
+
</text>
|
|
91
|
+
</g>
|
|
92
|
+
</g>
|
|
93
|
+
</svg>
|
package/docs/audit.md
ADDED
|
@@ -0,0 +1,91 @@
|
|
|
1
|
+
# Pre-release audit — AWI V1
|
|
2
|
+
|
|
3
|
+
Historical audit record for the local audit commit. Its hosted-CI gate was subsequently satisfied for d2514af by [run 39208844432](https://github.com/jigar48/awi/actions/runs/39208844432). Later release-preparation commits require their own CI. See the current [verification record](verification.md) and [release procedure](releasing.md); the original findings and audit-time recommendation below are preserved.
|
|
4
|
+
|
|
5
|
+
Date: October 19, 2026. Baseline: commit `9bc61c1`, branch `codex/awi-v1`. This audit inspected the implementation, original supplied brief, all original 37 tests, documentation, packaged artifacts and CI configuration. The brief ends mid-sentence in report redaction; no absent requirements are claimed. Changes fix V1 defects and release verification gaps; no new product features or remote actions were introduced.
|
|
6
|
+
|
|
7
|
+
## Recommendation
|
|
8
|
+
|
|
9
|
+
**NO-GO for publishing until the complete hosted CI matrix passes, especially macOS.** The exercised final implementation has no known remaining critical/high defect from this audit, but local Windows and WSL evidence does not prove every advertised platform/runtime combination. Complete the configured nine Windows/macOS/Linux × Node 22/24/26 jobs before publishing. Publication also requires the maintainer's explicit authorization; none was performed here.
|
|
10
|
+
|
|
11
|
+
## Findings and fixes
|
|
12
|
+
|
|
13
|
+
The following are finding groups, not a count of every affected assertion. All listed implementation defects are fixed locally. Residual design limitations follow separately.
|
|
14
|
+
|
|
15
|
+
| ID | Severity | Reproduction / observed defect | Fix and regression evidence |
|
|
16
|
+
| --- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
17
|
+
| H01 | High | 2,000 spans sharing a 20,000-element OTLP resource attribute exhausted a 96 MiB child heap (baseline exit 134, heap-out-of-memory). Wide JSON and repeated native string references also allowed disproportionate work; long IDs amplified default reports. Long ancestor keys with wide descendants required potentially multi-gigabyte ancestor-path serialization for a sub-megabyte input (baseline subprocess timed out). | Decode resource attributes once; apply raw/normalized node and escaped-content budgets, use bounded-depth descriptor-at-a-time validation, cap default report content before serialization. Compile/reuse bounded ignore-path maps, cap path depth, and safely invalidate changed caller options. The 96 MiB subprocess now exits normally with controlled limits for shared resources, a million-element JSON array and repeated strings; report amplification and long-key CPU amplification have separate regressions, including cache mutation/accessor safety. |
|
|
18
|
+
| H02 | High | Direct `analyze()` accepted `source.index={password:"AUDIT_SECRET"}` and copied it into an ordinary report. It validated a stripped copy but analyzed original values, skipped defaults and evaluated getters before validation. Malformed native input could throw an uncontrolled TypeError. | Security preflight before schema access; strict source schema; analyze parsed/defaulted data; controlled AwiError failures. Known optional native fields normalize explicit undefined to missing without accepting undefined inside arbitrary JSON argument/metadata values. Regressions cover source leakage, getters, malformed inputs and unknown-status defaults. This exploit affected the library boundary; canonical-file CLI ingestion already rejected source fields. |
|
|
19
|
+
| M01 | Medium | Fractional timestamps lost submillisecond precision and made overlapping calls appear sequential. Reversed nanosecond endpoints could pass. The first member of an overlap cluster could remain in a qualifying run. Known interleaved model/generic spans also hid qualifying repeated tools by resetting the tool window. | Preserve OTLP nanoseconds; use exact integer comparisons for normalization, overlap and retry windows; exclude every overlap-cluster member and warn; project known tool calls across interleaved non-tool spans while retaining timing/parent constraints. Tests cover true/false overlap, valid sequential short spans, reversed endpoints, timezone offsets and window boundaries. |
|
|
20
|
+
| M02 | Medium | `error,error,error,error,ok,error` inside the retry window yielded no AWI004; the later attempt erased the qualifying recovery chain. Blank error fields could satisfy AWI002 failure equivalence. | Stop retry chains at first non-error; require nonblank failure evidence. Positive/negative regressions preserve explicit-failure and inferred-retry distinctions. |
|
|
21
|
+
| M03 | Medium | OTLP intValue null/empty values became zero usage, absent AnyValue arguments became null, arrays could classify as model operations, and negative numeric timestamps were accepted. Recorded error status messages were ignored. | Validate known AnyValue types and integer ranges, omit unknown/incomplete argument evidence, prevent unnamed display placeholders from establishing tool identity, classify only strings, bound unsigned nanosecond timestamps, preserve explicit error status messages without printing them. Tests exercise malformed/unknown/nested values and status evidence, plus binary/text distinctions and duplicate unknown attributes. |
|
|
22
|
+
| M04 | Medium | AWI002 default threshold changed from 3 to 4 when `{enabled:true}` or `{}` was supplied. Thresholds above the fixed 100-call window silently made repetition rules impossible to trigger. | Rule-specific defaults; reject thresholds above 100. Configuration, generated schema and documentation agree. |
|
|
23
|
+
| M05 | Medium | Reordering fragments of one trace changed AWI006 operation/reference ordering and could change which findings survived the finding cap. | Stable operation identity ordering before grouping, accounting and selection. Tests compare reversed fragments, capped results, resource evidence and caller-data immutability. |
|
|
24
|
+
| M06 | Medium | An input/configuration/report path below a junction or directory symlink bypassed the advertised no-symlink policy. Aggregate input accounting used pre-read sizes. | Validate path components, use O_NOFOLLOW where available, compare opened-file identity, count actual read bytes and bound directory enumeration/sorting. Use a fixed-length temporary name so valid long report filenames do not exceed filesystem component limits. Tests cover input/config/output ancestors, final output aliases, valid long filenames, depth, file count and aggregate byte limits. Parent replacement races still require trusted directories. |
|
|
25
|
+
| M07 | Medium | The public humanReport renderer emitted raw terminal escapes from a schema-valid report warning. Internally generated CLI warnings were constant, so this was a public-renderer boundary issue. | Sanitize report strings including warnings/titles/explanations; keep trusted color escapes separate. Existing CLI tests also cover ANSI/bidi identifiers and JSON escaping. |
|
|
26
|
+
| M08 | Medium | Malformed UTF-8 inside identifiers was silently replaced with U+FFFD, changing recorded evidence and potentially collapsing distinct values. UTF-8 BOM exports failed unexpectedly. | Fatal UTF-8 decoding with BOM support; regression tests reject invalid bytes and accept a BOM export. |
|
|
27
|
+
| M09 | Medium | A valid argument object containing two 40,000-character fields failed the input single-string limit after canonicalization, despite neither input string exceeding 65,536 units. | Hash the exact ordered V1 envelope in chunks without revalidating generated canonical JSON as an input field. Existing fixed hash vector and an independent reference envelope test confirm unchanged V1 bytes. |
|
|
28
|
+
| L01 | Low | Direct fingerprint/canonicalize callers bypassed configuration ignore-path syntax restrictions; direct fingerprint getters could execute. | Enforce safe paths and preflight JSON descriptors at public fingerprint boundaries; malformed-path/getter regressions. |
|
|
29
|
+
| L02 | Low | A closed stdout pipe caused an uncaught EPIPE stack trace including installation paths. | Controlled output-stream failure with exit 1; actual closed-pipe subprocess regression. |
|
|
30
|
+
| L03 | Low | Maintainer documents linked from the README were absent from the package. The old consumer test lived below the repository and could inherit development dependencies; it did not verify all examples, tar permissions or declarations. Broad test discovery also collected scratch checkouts. | Include maintainer documents; use a clean OS temporary consumer outside the repository; verify allowlist/bin mode, all examples/options, ESM exports and strict consumer TypeScript. Scope Vitest discovery to tests/. CLI setup builds only runtime JS; full builds/package checks still verify declarations. |
|
|
31
|
+
| L04 | Low | CI used mutable major tags and retained checkout credentials by default. Original setup actions used a Node 20 action runtime. | Pin verified upstream Node 24 action commits, disable persisted checkout credentials and automatic package-manager caching, retain read-only permissions and add a job timeout. Package engine ranges now match the supported 22/24/26 major lines rather than implying unverified future-major support. No hosted workflow was executed by this audit. |
|
|
32
|
+
|
|
33
|
+
No critical finding was identified. None of the tests or reviews establishes absence of every possible defect.
|
|
34
|
+
|
|
35
|
+
## Rule completeness review
|
|
36
|
+
|
|
37
|
+
| Rule | Audit conclusion after fixes |
|
|
38
|
+
| ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
39
|
+
| AWI001 | Fingerprints preserve object key order normalization, array order, types, Unicode code points, null vs absent and explicit ignored paths. Threshold 4, disjoint 100-call windows, same trace/parent, reliable sequential evidence. Intentional polling remains a documented false positive. |
|
|
40
|
+
| AWI002 | Default 3; requires explicit error status and matching nonblank recorded error information. Matching hashes are internal only; more specific findings suppress AWI001 for their exact covered operations. Unknown status and differing outcomes do not qualify. |
|
|
41
|
+
| AWI003 | Exact contiguous bounded cycles; default 3 repeats, maximum default length 6/configuration cap 16; shortest matching period wins and consumed sequences are not rescanned. Single-call repetition is excluded. Concurrent/missing evidence cannot establish a cycle. |
|
|
42
|
+
| AWI004 | Default 5 attempts/30 seconds; preceding attempts must explicitly fail. A terminal success/unknown stops the chain; later calls cannot erase an earlier match. Exact time boundaries; absent timing excludes sequences. Retry intent remains inferred and may coexist with AWI002 for distinct timing evidence. |
|
|
43
|
+
| AWI005 | Separately counts all operations and observed parent depth (roots=1). Thresholds are policy, not proof of bugs. Missing parents warn and give lower bounds; duplicate IDs, parent cycles and reversed endpoints fail explicitly. |
|
|
44
|
+
| AWI006 | Associates only directly implicated leaf operations with recorded usage/cost, unions related IDs once, keeps input/output/total independent and currencies separate. Missing values remain absent; parent totals and unrelated model usage are not attributed. Empty usage does not fabricate a finding. |
|
|
45
|
+
|
|
46
|
+
Cycle scan lengths and repeat thresholds are capped; failed candidate scans have a bounded repeat count before the configured minimum would qualify and consume the sequence. No combinatorial subsequence search is present. The existing 50,000-operation regression guard remains; tests did not relax detector performance assertions. Findings cap at 1,000 with complete:false and exit 1. Content budgets fail explicitly rather than returning a falsely clean partial analysis.
|
|
47
|
+
|
|
48
|
+
## Ingestion, interfaces and reporting
|
|
49
|
+
|
|
50
|
+
Canonical v1 trace/collection input is strict, supports nested operations/handoffs via parent structure and metadata, preserves generic spans and optional resource evidence, and groups by trace identity across files. The OTLP adapter supports only the documented protobuf JSON subset. Format ambiguity and malformed required fields fail; missing/unknown evidence is not invented. Unsupported AnyValue encodings are omitted from interpretation; this is not universal exporter compatibility.
|
|
51
|
+
|
|
52
|
+
All required commands/options, ESM public imports, configuration defaults/rejections and exit codes were exercised. Exit 1 takes precedence over severity; exit 2 means completed analysis meets fail-on; fail-on none does not suppress operational failures. JSON stdout contains only JSON; error paths use stderr without raw trace content. Report version and fingerprint version remain 1. Evidence references identify trace/operation IDs and validated adapter source indices. Schema files were regenerated; global resource/graph checks require ingest/analyze, not structural schemas alone.
|
|
53
|
+
|
|
54
|
+
## Test quality and security evidence
|
|
55
|
+
|
|
56
|
+
Baseline: 37 passing tests. Final suite: **69 tests**, adding 32 targeted cases plus stronger assertions in existing CLI tests. The additions focus on failures reproduced during the audit, resource budgets, exact boundary behavior, deterministic capped selection and public/native input validation. No thresholds/assertions were weakened to accept a product defect. Test setup uses a separate bounded compilation budget and runtime-only bundling because WSL declaration compilation from an NTFS mount exceeded the previous 10-second setup hook; declaration validation remains in full builds and consumer typechecking. CLI end-to-end cases have a separate 15-second I/O envelope after measuring WSL cold-start timeouts in cases launching multiple Node processes. Individual CLI subprocesses are also bounded at 15 seconds. Functional assertions and the 50,000-operation detector timing assertion remain unchanged.
|
|
57
|
+
|
|
58
|
+
The hostile-content subprocess blocks network, DNS, fetch and subprocess APIs, then analyzes URLs, path traversal strings, embedded instructions, tool/name/error/metadata secrets. It returns the expected finding exit code without any attempted external effect or those secrets in default output. Identifier strings deliberately remain visible by contract. JSON prototype keys, unsafe native shapes, nesting/string limits, oversized files, symlink aliases, overwrite/hardlink protection, private POSIX output mode and broken pipes are covered. Heap-budget tests are specific adversarial regression vectors, not a promise that every valid maximum-size input fits a 96 MiB process.
|
|
59
|
+
|
|
60
|
+
## Package verification
|
|
61
|
+
|
|
62
|
+
The actual package is built and packed, checked as a tar archive, installed with lifecycle scripts disabled into a new OS temporary consumer outside the repository, and removed after testing. Verification checks allowlisted contents, no traversal/.git/node_modules/scratch payloads, executable mode and shebang for the awi bin, correct ESM/type export mappings, presence of README/license/maintainer docs, strict TypeScript imports and the public API. No undeclared development dependency is inherited from the project.
|
|
63
|
+
|
|
64
|
+
| Installed example | Expected result |
|
|
65
|
+
| ------------------------ | ------------------------------------------------------- |
|
|
66
|
+
| healthy/basic.json | exit 0, no findings |
|
|
67
|
+
| suspicious/retries.json | exit 2, AWI002/AWI004 and recorded resource association |
|
|
68
|
+
| suspicious/cycle.json | exit 0 at default high threshold, AWI003 present |
|
|
69
|
+
| opentelemetry/tools.json | exit 2, OTLP failure/retry findings |
|
|
70
|
+
|
|
71
|
+
Installed checks also exercise help/version, explicit/auto formats, config, directory/multi-example inspection, JSON/human output, fail-on none, output creation, no-color and opt-in content. A local tarball is prepared; no npm publication or global installation was performed.
|
|
72
|
+
|
|
73
|
+
## Verification status
|
|
74
|
+
|
|
75
|
+
See [verification record](verification.md) for final platform/runtime results. Windows Node 24.19.0 passed the final 69 tests, strict typecheck, lint, full build, formatting, frozen install, full dependency audit, schemas and isolated package smoke. Linux WSL Node 22.18.0, 24.19.0 and 26.0.0 each passed all 69 tests, strict typecheck, lint, full build and isolated package smoke. These runs use native Linux Node binaries with official SHA-256 manifest verification and an isolated checkout/dependency tree. macOS and the hosted GitHub matrix are unavailable/unexecuted in this session.
|
|
76
|
+
|
|
77
|
+
## Remaining limitations and release gates
|
|
78
|
+
|
|
79
|
+
- **Medium, known privacy limitation:** trace/operation identifiers may themselves contain secrets and are included by the evidence contract. Default reports are not a general anonymizer. Export opaque identifiers and review reports before sharing.
|
|
80
|
+
- **Medium, explicit opt-in risk:** show-content can expose arbitrary strings, tool names, IDs and error codes. Key-based redaction catches selected structured keys, not every credential or private value. No guarantee of secret-free opt-in output is made.
|
|
81
|
+
- **Low, filesystem assumptions:** parent-component replacement races require trusted directories; existing symlink aliases must be resolved first. Hard-link report publication must be supported. POSIX mode 0600 is checked; Windows uses inherited ACLs. Cleanup is best-effort under OS failures/crashes.
|
|
82
|
+
- **Detection limitations:** disjoint windows miss boundary-spanning patterns; incomplete/untimed/concurrent groups can hide genuine problems; exact fingerprints are not semantic equivalence or proof of real-world effects. Leaf-only resource association undercounts some exports. No framework certification or universal OTLP support is claimed.
|
|
83
|
+
- **Verification limits:** macOS and hosted runner jobs remain release gates; functional tests are not an exhaustive fuzzing campaign, formal proof, independent penetration test, or Node/OS vulnerability assessment. Dependency audit results are time-dependent.
|
|
84
|
+
|
|
85
|
+
## Git and external actions
|
|
86
|
+
|
|
87
|
+
Changes remain on `codex/awi-v1` and are saved in the local audit commit containing this report; the exact commit is reported by git log after commit creation. The starting tree was clean at 9bc61c1. No remote push, publication, release, tag, deployment, trace upload or remote repository modification occurred. Scratch runtimes/checkouts live under ignored .awi-audit/; consumer directories are temporary and cleaned up on ordinary test completion.
|
|
88
|
+
|
|
89
|
+
## Upstream action evidence
|
|
90
|
+
|
|
91
|
+
Pinned action runtimes and inputs were checked against the upstream repositories: [checkout action metadata](https://github.com/actions/checkout/blob/d23441a48e516b6c34aea4fa41551a30e30af803/action.yml), [setup-node action metadata](https://github.com/actions/setup-node/blob/249970729cb0ef3589644e2896645e5dc5ba9c38/action.yml), [pnpm setup action metadata](https://github.com/pnpm/action-setup/blob/a8198c4bff370c8506180b035930dea56dbd5288/action.yml). This inspection is not a substitute for running the workflow.
|
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
# CI integration
|
|
2
|
+
|
|
3
|
+
AWI analyzes local JSON and sends no telemetry. Use synthetic traces for examples. Review reports before sharing because workflow measurements, source indices and relationships can be sensitive. See the [security and privacy documentation](security.md).
|
|
4
|
+
|
|
5
|
+
## Inside the AWI repository
|
|
6
|
+
|
|
7
|
+
After checkout, use the repository's pinned pnpm version and build its source:
|
|
8
|
+
|
|
9
|
+
```yaml
|
|
10
|
+
- uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
|
|
11
|
+
with:
|
|
12
|
+
persist-credentials: false
|
|
13
|
+
- uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6
|
|
14
|
+
with:
|
|
15
|
+
node-version: '24.11.0'
|
|
16
|
+
package-manager-cache: false
|
|
17
|
+
- uses: pnpm/action-setup@a8198c4bff370c8506180b035930dea56dbd5288 # v5
|
|
18
|
+
- run: pnpm install --frozen-lockfile
|
|
19
|
+
- run: pnpm build
|
|
20
|
+
- run: pnpm verify:quickstart
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
The verifier expects exit 0 for the healthy fixture and exit 2 with `AWI002`, `AWI004`, and `AWI006` for retries. It exits 0 only when both match, so expected findings do not fail the job.
|
|
24
|
+
|
|
25
|
+
Repository CI runs this verification alongside typecheck, lint, 69 tests, build, formatting, schema drift, production audit and package smoke checks on Windows/macOS/Linux × Node 22.18.0/24.11.0/26. Earlier hosted results are recorded in [verification](verification.md); a fresh matrix is needed for these changes.
|
|
26
|
+
|
|
27
|
+
## Inside another repository
|
|
28
|
+
|
|
29
|
+
Install the published CLI rather than expecting AWI's `dist/` and `examples/` paths in your application. Version 1.0.0 was confirmed in the [npm registry](https://registry.npmjs.org/agent-workflow-inspector/1.0.0) on October 20, 2026. The following steps assume your checkout contains `test-traces/` with supported JSON exports:
|
|
30
|
+
|
|
31
|
+
```yaml
|
|
32
|
+
- uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
|
|
33
|
+
with:
|
|
34
|
+
persist-credentials: false
|
|
35
|
+
- uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6
|
|
36
|
+
with:
|
|
37
|
+
node-version: '24.11.0'
|
|
38
|
+
package-manager-cache: false
|
|
39
|
+
- run: npm install --global --ignore-scripts agent-workflow-inspector@1.0.0
|
|
40
|
+
- run: awi inspect ./test-traces --format json --fail-on high --output awi-report.json
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
Set workflow permissions to `contents: read`. For lockfile-controlled dependencies, add AWI as an exact dev dependency in your application, commit its lockfile, and use your package manager's local executable instead of a global install.
|
|
44
|
+
|
|
45
|
+
The gate exits 0 below threshold, 2 for completed analysis meeting the threshold, or 1 for errors/incomplete analysis. `--fail-on none` never masks errors. Do not suppress exit 2 when you want findings to block CI. JSON warnings remain structured in stdout. `--output` requires a new filename in an existing directory; keep reports outside input directories and remove old reports before rerunning.
|
|
@@ -0,0 +1,17 @@
|
|
|
1
|
+
# Configuration
|
|
2
|
+
|
|
3
|
+
Use --config path.json. No automatic home lookup, executable settings, remote loading or environment substitution. Partial objects receive deterministic defaults. Unknown keys, versions, unsafe paths and invalid thresholds fail. See [example](../awi.config.example.json) and [schema](schemas/config.schema.json).
|
|
4
|
+
|
|
5
|
+
Version 1; every rule supports enabled. AWI001/002 threshold 2–1,000 (defaults 4/3). AWI003 minRepetitions 2–100 (default 3), maxCycleLength 2–16 (default 6). AWI004 threshold 2–1,000 (default 5), windowSeconds 1–3,600 (default 30). AWI005 maxOperations 1–50,000 (default 100), maxDepth 1–1,000 (default 20). AWI006 reports recorded association, so needs no spending threshold.
|
|
6
|
+
|
|
7
|
+
fingerprint.ignorePaths accepts at most 100 dot-separated ASCII identifiers, e.g. metadata.trace_id, applied to arguments only, with at most 64 path segments. Arrays preserve order and are not traversed by ignore paths. The public canonicalize/fingerprint helpers enforce the same path syntax. No wildcards, expressions, indices, `constructor`, `prototype` or `__proto__`. Missing ignored paths have no effect. Literal dotted JSON keys do not match nested paths. IDs and timestamps are never automatically ignored.
|
|
8
|
+
|
|
9
|
+
## Fingerprint v1
|
|
10
|
+
|
|
11
|
+
awi-sha256-v1 hashes operation type, toolName (fallback name), argument presence and canonical argument JSON. Keys sort lexically by UTF-16 code unit. Arrays keep order, JSON types remain distinct, missing differs from null. Unicode has no normalization. Numbers use finite JavaScript double semantics: -0 becomes 0; large numeric integers may lose precision, so encode precision-sensitive integers as strings. Duplicate keys follow JSON.parse last-value-wins before hashing. Unsupported values (BigInt, undefined as a value, functions, class instances, sparse arrays, symbol keys, accessors), nonfinite numbers, unsafe keys and oversized structures fail. There is no executable canonicalizer.
|
|
12
|
+
|
|
13
|
+
Generated canonical JSON is not subject to the per-input-string limit; argument objects containing multiple individually valid strings remain supported within the content budget. The v1 byte contract is unchanged. Ignored paths are compiled into bounded maps, safely reused while unchanged, and never matched by repeatedly serializing full ancestor paths.
|
|
14
|
+
|
|
15
|
+
Identical fingerprints mean identical recorded values under these rules, not identical effects. Hashes are sensitive correlators, not anonymization, and are absent from ordinary reports.
|
|
16
|
+
|
|
17
|
+
AWI007 defaults to `{ "enabled": false, "threshold": 4, "minDistinctArguments": 3 }`. Threshold accepts integers 4–100; minDistinctArguments accepts integers 3–100. Both are minimums over a maximal sequential failure run, so the distinct minimum can exceed the call threshold. Unknown fields and non-integer/out-of-range values fail strict validation. Enable with [the opt-in example](../examples/argument-drift.config.json).
|
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
# Detection rules
|
|
2
|
+
|
|
3
|
+
Diagnostics separate observed evidence from heuristic explanation. High evidence means recorded equivalence/failure or policy counts are directly established. Moderate means pattern/association inference. Categories are not statistical probabilities; no rule establishes intent, actual defects or wasted money.
|
|
4
|
+
|
|
5
|
+
Identity rules require recorded tool/operation identity (not an adapter display placeholder), recorded arguments (null is present), positive-duration timed siblings under one parent, and non-overlap. Untimed/zero-duration groups cannot establish sequences and are excluded. Known model/generic spans are projected out of tool sequences, while their timing still constrains overlap and eligibility. Missing tool arguments/identity interrupt tool segments. All members of overlapping clusters are excluded, splitting runs; unrelated traces/parents never combine. Incomplete exports can therefore hide genuine problems.
|
|
6
|
+
|
|
7
|
+
| ID | Default criterion | Limitations |
|
|
8
|
+
| ------ | ------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------- |
|
|
9
|
+
| AWI001 | Same fingerprint at least 4 times in disjoint 100-call window | Intentional polling matches; window boundaries can miss repetition |
|
|
10
|
+
| AWI002 | 3 same-fingerprint explicit errors with matching error object; nonblank code or message required | Different failure text splits equivalence; unknown is not failure |
|
|
11
|
+
| AWI003 | Contiguous cycle length 2–6 repeating 3 times, at least two distinct fingerprints | Shortest matching period wins; consumed cycles not rescanned |
|
|
12
|
+
| AWI004 | 5 equivalent attempts in 30 seconds from first attempt, every prior attempt explicitly failed | Retry intent inferred; successes, unknown status and slow attempts do not qualify |
|
|
13
|
+
| AWI005 | More than 100 total operations OR observed depth over 20 | One trace-wide finding, not a tool-count limit; complexity is not a defect |
|
|
14
|
+
| AWI006 | Recorded usage or cost on implicated leaf operations from AWI001–004 or enabled AWI007 | Direct association only; no inferred price or unrelated model attribution |
|
|
15
|
+
|
|
16
|
+
AWI001/002/004 use disjoint 100-call windows after segmentation; AWI004 accepts disjoint time groups within each fingerprint group, ending a retry chain at its first non-error outcome. Later attempts cannot erase an earlier qualifying storm. Boundary-spanning patterns may be missed. AWI003 scans the full eligible segment, with configurable length capped at 16. Single-fingerprint repetition is AWI001/002 rather than a cycle.
|
|
17
|
+
|
|
18
|
+
## Interactions
|
|
19
|
+
|
|
20
|
+
AWI002 suppresses AWI001 for its exact covered failure operations. Remaining repeated operations can qualify separately. AWI004 can coexist with AWI002: rapid failed attempts and matching failure outcomes provide distinct evidence. AWI003 adds ordering evidence. One AWI006 per trace unions related IDs and counts each leaf once. AWI005 uses empty operationIds for trace-wide evidence.
|
|
21
|
+
|
|
22
|
+
## Resource accounting
|
|
23
|
+
|
|
24
|
+
Summaries sum recorded leaf usage, excluding parent aggregates. Input/output/total fields are independently summed and never added to each other. Cost sums remain separate per currency. AWI006 uses only implicated leaves, never nearby/enclosing model spans. Missing resources remain absent. Explicit zero is valid recorded evidence. Parent-only usage is omitted conservatively, so absence of AWI006 does not imply free execution. Wall duration is not CPU time or wasted time.
|
|
25
|
+
|
|
26
|
+
## Limits
|
|
27
|
+
|
|
28
|
+
16 MiB/file; 64 MiB aggregate CLI inputs; 256 files; 2,048 directory entries; directory depth 16; 50,000 total operations; 50,000 traces; JSON depth 64; 65,536 UTF-16 code units/string or key; 1,000,000 JSON values and a 64 MiB escaped-content budget per raw or normalized structure (shared values count per occurrence); 1,000 findings. Structural limits fail with exit 1. Default report data is subject to the same node/content budget before serialization; excessive evidence amplification fails with an explicit report-limit error rather than attempting unbounded output. Finding overflow returns complete:false, warning, exit 1 even with fail-on none. At most 100 fingerprint paths. Config schemas bound thresholds/windows/cycle lengths. A 50,000-span test uses a generous 15-second regression ceiling, not a cross-hardware benchmark claim.
|
|
29
|
+
|
|
30
|
+
## AWI007: Repeated failure with argument drift
|
|
31
|
+
|
|
32
|
+
Opt-in; medium severity, moderate confidence. Requires at least four consecutive eligible tool failures in one trace and parent, the same toolName (fallback name), an exactly matching nonblank explicit error code, and at least three distinct existing v1 fingerprints. No case folding or semantic argument comparison occurs. Fingerprint ignore paths apply. Missing arguments, unknown identity, missing/blank codes, success, unknown status, or a different tool/code interrupt a sequence. Non-tool siblings constrain timing but are projected out, as with existing sequence rules. Missing/zero-duration sibling timing excludes the sibling group; overlapping operations split runs and are excluded.
|
|
33
|
+
|
|
34
|
+
One finding covers each maximal qualifying run, without sliding-window duplicates. Thresholds are minimum counts, not fixed window sizes. The scan adds linear time and memory after existing timing sorting, bounded by the 50,000-operation input limit and existing report budgets.
|
|
35
|
+
|
|
36
|
+
AWI002 compares equivalent arguments and complete recorded error objects; AWI004 counts rapid equivalent attempts after failures within its time window. AWI007 requires varying fingerprints and a shared error code, without a retry-time window. Messages may differ and are never used by AWI007. Other rules may independently report the same operations; AWI006 can associate recorded leaf resources when AWI007 is enabled.
|
|
37
|
+
|
|
38
|
+
Legitimate parameter exploration, validation attempts, pagination, and fallback strategies can match. Repeated recorded failures do not prove a lack of progress or reveal intent. Review the error code and retry policy before deciding whether to bound retries or escalate.
|
|
@@ -0,0 +1,70 @@
|
|
|
1
|
+
# Getting started
|
|
2
|
+
|
|
3
|
+
Use Node.js **22.18+ within 22.x, 24.11+ within 24.x, or 26.x** and **pnpm 11.25.0**, matching `package.json`. Check `node --version` and `pnpm --version` first. If pnpm is missing, install the pinned version with `npm install --global pnpm@11.25.0` using a supported Node version.
|
|
4
|
+
|
|
5
|
+
From a new checkout (PowerShell, macOS, or Linux shell):
|
|
6
|
+
|
|
7
|
+
```sh
|
|
8
|
+
git clone https://github.com/jigar48/awi.git
|
|
9
|
+
cd awi
|
|
10
|
+
pnpm install --frozen-lockfile
|
|
11
|
+
pnpm build
|
|
12
|
+
pnpm verify:quickstart
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
This uses the checked-out source, including the verifier added after v1.0.0; the original release tag does not contain that script. For a reproducible repeat, record `git rev-parse HEAD` and use that commit. Clone, toolchain setup, and dependency installation need network access; build, verification, and analysis run locally with no telemetry.
|
|
16
|
+
|
|
17
|
+
Expected verification output:
|
|
18
|
+
|
|
19
|
+
```text
|
|
20
|
+
examples/healthy/basic.json: exit 0; rules none
|
|
21
|
+
examples/suspicious/retries.json: exit 2; rules AWI002, AWI004, AWI006
|
|
22
|
+
Quickstart verification passed.
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
The verifier runs the built CLI on synthetic, checked-in examples and itself exits 0 when both cases match. Rebuild after changing source. To run the CLI yourself:
|
|
26
|
+
|
|
27
|
+
```sh
|
|
28
|
+
node dist/cli/index.js --help
|
|
29
|
+
node dist/cli/index.js inspect examples/healthy/basic.json
|
|
30
|
+
node dist/cli/index.js inspect examples/suspicious/retries.json --format json
|
|
31
|
+
node dist/cli/index.js inspect examples/opentelemetry/tools.json --input-format otel
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
The retry and OTLP examples intentionally exit 2 at the default high threshold; a shell/CI job may stop there. Exit 1 means an operational error or incomplete analysis. `--fail-on none` disables finding-based failure only. AWI analyzes exports, not live executions.
|
|
35
|
+
|
|
36
|
+
For contributor checks, run `pnpm check` and `pnpm format:check`.
|
|
37
|
+
|
|
38
|
+
For use outside this checkout, `agent-workflow-inspector@1.0.0` is [published on npm](https://www.npmjs.com/package/agent-workflow-inspector/v/1.0.0) (verified October 20, 2026):
|
|
39
|
+
|
|
40
|
+
```sh
|
|
41
|
+
npm install --global agent-workflow-inspector@1.0.0
|
|
42
|
+
awi --version
|
|
43
|
+
awi --help
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
Save the following synthetic example as `trace.json`, then run `awi inspect trace.json` (exit 0). Checkout-relative `examples/` and `dist/` paths apply only inside AWI's source repository.
|
|
47
|
+
|
|
48
|
+
A minimal canonical trace is:
|
|
49
|
+
|
|
50
|
+
```json
|
|
51
|
+
{
|
|
52
|
+
"version": 1,
|
|
53
|
+
"traceId": "example",
|
|
54
|
+
"operations": [
|
|
55
|
+
{
|
|
56
|
+
"id": "one",
|
|
57
|
+
"type": "tool",
|
|
58
|
+
"name": "lookup",
|
|
59
|
+
"arguments": { "q": "demo" },
|
|
60
|
+
"status": "ok",
|
|
61
|
+
"startTime": "2026-01-01T00:00:00Z",
|
|
62
|
+
"endTime": "2026-01-01T00:00:01Z"
|
|
63
|
+
}
|
|
64
|
+
]
|
|
65
|
+
}
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
Supply several files or a directory. Files sharing traceId merge; duplicate operation IDs then fail validation. Distinct trace IDs never share sequence detection. Output is withheld on ordinary input failures.
|
|
69
|
+
|
|
70
|
+
Use --format json for tooling. --output must name a new file in an existing directory. Keep reports outside input directories, as they would be treated as traces on the next scan.
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
# Limitations
|
|
2
|
+
|
|
3
|
+
Rules evaluate recorded facts and bounded heuristics, not intention or correctness. Polling/retries/cycles may be intentional. Missing arguments/timing, zero durations, overlaps, missing parent structure and instrumentation differences can hide patterns. Missing parents yield lower-bound depth. Neither graph structure nor temporal order establishes causality. Timing rules assume comparable recorded clocks; AWI does not correct distributed clock skew.
|
|
4
|
+
|
|
5
|
+
Disjoint 100-call windows can miss boundary-spanning repetition. Cycles are exact, contiguous and bounded. Different arguments do not semantically match. No LLM, embeddings, live tracing or distributed causal reconstruction. Failure comparison is exact recorded-object matching.
|
|
6
|
+
|
|
7
|
+
Only recorded leaf resources are counted. Parent aggregates and unrelated model usage are not attributed. Separate leaves can still duplicate values in poorly instrumented exports; absent accounting semantics cannot be recovered. Costs use finite JSON/JavaScript number arithmetic, are explicit, and never mix currencies; they are not a billing ledger. Wall duration is never labeled waste.
|
|
8
|
+
|
|
9
|
+
OTLP support is the documented subset, not every export/framework. Duplicate JSON keys are last-wins; numbers use double precision. Strings preserve large integers exactly. Timestamp ordering retains up to nanosecond precision; duration summaries use floating-point milliseconds.
|
|
10
|
+
|
|
11
|
+
Windows and Ubuntu WSL checks do not establish macOS or hosted-runner compatibility; the full CI matrix must run. Regression timing is not a universal speed promise. Trusted directories remain required against concurrent replacement of parent components. Legitimate symlink paths, including package-manager aliases and some OS temporary paths, must be resolved to real paths before inspection. Report publication requires hard-link support. Opt-in key-based redaction is not general anonymization.
|
|
12
|
+
|
|
13
|
+
AWI007 is disabled by default. Legitimate argument exploration and fallback attempts may produce the same code repeatedly. Fingerprint differences do not measure semantic change or progress; ignored paths can hide differences. Exact error-code equality can miss equivalent failures with different codes. Missing codes/arguments/status/timing can hide runs. AWI007 emits maximal runs rather than using the disjoint repetition windows of AWI001/002/004.
|
|
14
|
+
|
|
15
|
+
AWI007 evidence contains the failure count, distinct fingerprint count and operation references. Its tool name and shared error code are used internally. `--show-content` can expose recorded identifiers, labels and argument content despite key-based redaction. See the [security and privacy documentation](security.md).
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
# Agent Workflow Inspector v1.0.0 — Deterministic AI Agent Trace Diagnostics
|
|
2
|
+
|
|
3
|
+
Published on [GitHub](https://github.com/jigar48/awi/releases/tag/v1.0.0) October 19, 2026. [npm version 1.0.0](https://registry.npmjs.org/agent-workflow-inspector/1.0.0) confirmed October 20, 2026.
|
|
4
|
+
|
|
5
|
+
Agent Workflow Inspector is a local-first, deterministic linter for AI agent execution traces. It highlights recorded patterns for investigation; it does not understand agent intentions or prove a workflow is stuck.
|
|
6
|
+
|
|
7
|
+
## Included in V1
|
|
8
|
+
|
|
9
|
+
- Six bounded rules covering repeated tool calls, repeated failures, execution cycles, rapid retry attempts, operation/depth limits and associated recorded leaf resource usage.
|
|
10
|
+
- Versioned AWI JSON and the [documented OTLP JSON subset](https://github.com/jigar48/awi/blob/v1.0.0/docs/trace-formats.md); no blanket framework compatibility claim.
|
|
11
|
+
- CLI and ESM/TypeScript library, configurable thresholds, human-readable and structured JSON reports, stable evidence references and CI exit codes.
|
|
12
|
+
- Local analysis without telemetry or an LLM. Input budgets, terminal escaping, symlink checks and exclusive output creation.
|
|
13
|
+
|
|
14
|
+
## Install
|
|
15
|
+
|
|
16
|
+
```sh
|
|
17
|
+
npm install --global agent-workflow-inspector@1.0.0
|
|
18
|
+
awi --help
|
|
19
|
+
awi inspect trace.json --format json
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
Requires Node 22.18+ (22.x), 24.11+ (24.x), or 26.x. Exit 0: completed below threshold; 1: invalid input, operational failure or incomplete analysis; 2: completed with findings meeting the configured threshold (high by default).
|
|
23
|
+
|
|
24
|
+
The audited source passed 69 tests and nine hosted Windows/macOS/Linux × Node 22/24/26 jobs. These historical results do not validate later documentation or verification-script changes; see the [verification record](https://github.com/jigar48/awi/blob/v1.0.0/docs/verification.md).
|
|
25
|
+
|
|
26
|
+
## Privacy and limitations
|
|
27
|
+
|
|
28
|
+
Default reports omit raw arguments, results, messages and metadata. Trace/operation identifiers remain visible and can themselves contain sensitive data. Opt-in content display uses limited key-based redaction and can expose secrets. Review reports before sharing.
|
|
29
|
+
|
|
30
|
+
Intentional polling/retries can trigger findings. Missing timestamps, concurrency, incomplete exports, bounded windows and leaf-only resource attribution limit detection and accounting. Resource estimates do not establish causality or financial waste.
|
|
31
|
+
|
|
32
|
+
See [getting started](https://github.com/jigar48/awi/blob/v1.0.0/docs/getting-started.md), [rules](https://github.com/jigar48/awi/blob/v1.0.0/docs/detection-rules.md), [configuration](https://github.com/jigar48/awi/blob/v1.0.0/docs/configuration.md), [security](https://github.com/jigar48/awi/blob/v1.0.0/docs/security.md), [limitations](https://github.com/jigar48/awi/blob/v1.0.0/docs/limitations.md) and [changelog](https://github.com/jigar48/awi/blob/v1.0.0/CHANGELOG.md). MIT licensed; maintained by [Jigar Prajapati](https://github.com/jigar48).
|