agentguard-lint 0.3.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- agentguard_lint-0.3.0/LICENSE +21 -0
- agentguard_lint-0.3.0/PKG-INFO +260 -0
- agentguard_lint-0.3.0/README.md +228 -0
- agentguard_lint-0.3.0/agentguard/__init__.py +3 -0
- agentguard_lint-0.3.0/agentguard/__main__.py +3 -0
- agentguard_lint-0.3.0/agentguard/analyzer.py +1154 -0
- agentguard_lint-0.3.0/agentguard/cli.py +216 -0
- agentguard_lint-0.3.0/agentguard/config.py +51 -0
- agentguard_lint-0.3.0/agentguard/crawler.py +74 -0
- agentguard_lint-0.3.0/agentguard/data/prices.json +46 -0
- agentguard_lint-0.3.0/agentguard/estimator.py +177 -0
- agentguard_lint-0.3.0/agentguard/mcp_server.py +79 -0
- agentguard_lint-0.3.0/agentguard/pricing.py +107 -0
- agentguard_lint-0.3.0/agentguard/py.typed +0 -0
- agentguard_lint-0.3.0/agentguard/reporter.py +299 -0
- agentguard_lint-0.3.0/agentguard/rules.py +78 -0
- agentguard_lint-0.3.0/agentguard_lint.egg-info/PKG-INFO +260 -0
- agentguard_lint-0.3.0/agentguard_lint.egg-info/SOURCES.txt +32 -0
- agentguard_lint-0.3.0/agentguard_lint.egg-info/dependency_links.txt +1 -0
- agentguard_lint-0.3.0/agentguard_lint.egg-info/entry_points.txt +3 -0
- agentguard_lint-0.3.0/agentguard_lint.egg-info/requires.txt +18 -0
- agentguard_lint-0.3.0/agentguard_lint.egg-info/top_level.txt +1 -0
- agentguard_lint-0.3.0/pyproject.toml +77 -0
- agentguard_lint-0.3.0/setup.cfg +4 -0
- agentguard_lint-0.3.0/tests/test_cli.py +150 -0
- agentguard_lint-0.3.0/tests/test_crawler.py +53 -0
- agentguard_lint-0.3.0/tests/test_detection.py +139 -0
- agentguard_lint-0.3.0/tests/test_estimator.py +152 -0
- agentguard_lint-0.3.0/tests/test_growth.py +177 -0
- agentguard_lint-0.3.0/tests/test_interprocedural.py +81 -0
- agentguard_lint-0.3.0/tests/test_loops.py +151 -0
- agentguard_lint-0.3.0/tests/test_mcp_server.py +72 -0
- agentguard_lint-0.3.0/tests/test_pricing.py +62 -0
- agentguard_lint-0.3.0/tests/test_rules_misc.py +119 -0
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Muaz
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,260 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: agentguard-lint
|
|
3
|
+
Version: 0.3.0
|
|
4
|
+
Summary: Find LLM cost risks in Python agent code - unbounded loops, quadratic history growth - and estimate what a run costs.
|
|
5
|
+
Author: Muaz
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
Project-URL: Homepage, https://agent-guard-six.vercel.app
|
|
8
|
+
Project-URL: Source, https://github.com/muaz-q/AgentGuard
|
|
9
|
+
Project-URL: Issues, https://github.com/muaz-q/AgentGuard/issues
|
|
10
|
+
Keywords: llm,agents,cost,static-analysis,linter,openai,anthropic,langchain
|
|
11
|
+
Classifier: Development Status :: 4 - Beta
|
|
12
|
+
Classifier: Environment :: Console
|
|
13
|
+
Classifier: Intended Audience :: Developers
|
|
14
|
+
Classifier: Programming Language :: Python :: 3
|
|
15
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
16
|
+
Classifier: Topic :: Software Development :: Quality Assurance
|
|
17
|
+
Classifier: Typing :: Typed
|
|
18
|
+
Requires-Python: >=3.9
|
|
19
|
+
Description-Content-Type: text/markdown
|
|
20
|
+
License-File: LICENSE
|
|
21
|
+
Requires-Dist: tomli>=1.1; python_version < "3.11"
|
|
22
|
+
Provides-Extra: mcp
|
|
23
|
+
Requires-Dist: mcp<3,>=2.3; python_version >= "3.10" and extra == "mcp"
|
|
24
|
+
Provides-Extra: dev
|
|
25
|
+
Requires-Dist: pytest>=8; extra == "dev"
|
|
26
|
+
Requires-Dist: pytest-cov>=5; extra == "dev"
|
|
27
|
+
Requires-Dist: pytest-asyncio>=1.0; python_version >= "3.10" and extra == "dev"
|
|
28
|
+
Requires-Dist: mcp<3,>=2.3; python_version >= "3.10" and extra == "dev"
|
|
29
|
+
Requires-Dist: ruff>=0.6; extra == "dev"
|
|
30
|
+
Requires-Dist: mypy>=1.10; extra == "dev"
|
|
31
|
+
Dynamic: license-file
|
|
32
|
+
|
|
33
|
+
# AgentGuard
|
|
34
|
+
|
|
35
|
+
[](https://github.com/muaz-q/AgentGuard/actions/workflows/ci.yml)
|
|
36
|
+
[](https://pypi.org/project/agentguard-lint/)
|
|
37
|
+
[](https://pypi.org/project/agentguard-lint/)
|
|
38
|
+
[](LICENSE)
|
|
39
|
+
|
|
40
|
+
**A static analyser that finds the code patterns that make LLM agents expensive - before they run.**
|
|
41
|
+
|
|
42
|
+
Agent bills rarely blow up because of one expensive call. They blow up because of loops: an
|
|
43
|
+
agent that stops only when the model says so, a conversation history that is re-sent in full
|
|
44
|
+
on every turn, a call buried three functions deep inside a loop over a dataset. AgentGuard
|
|
45
|
+
reads your Python (it never runs it or calls an API), finds those patterns with file and line,
|
|
46
|
+
and estimates what a run costs.
|
|
47
|
+
|
|
48
|
+
```console
|
|
49
|
+
$ agentguard examples/research_agent.py
|
|
50
|
+
examples/research_agent.py:21:13: AG006 [low] LLM call (openai.chat) sets no output-token cap
|
|
51
|
+
examples/research_agent.py:31:17: AG001 [high] LLM call (anthropic.messages) inside `while` loop (line 28) with no iteration cap
|
|
52
|
+
examples/research_agent.py:31:17: AG002 [high] LLM call (anthropic.messages) re-sends a history that grows every iteration; input tokens grow quadratically
|
|
53
|
+
examples/research_agent.py:31:17: AG007 [medium] ~2,260-token static prompt re-sent each iteration with no `cache_control`
|
|
54
|
+
examples/research_agent.py:47:13: AG003 [medium] call to `summarise` inside 2 nested loops (lines 45, 46); calls multiply
|
|
55
|
+
|
|
56
|
+
5 finding(s) (2 high, 2 medium, 1 low) in 1 file(s), 3 LLM call site(s).
|
|
57
|
+
|
|
58
|
+
Estimated cost per run
|
|
59
|
+
loops with no constant bound evaluated at N = 1, 10, 100
|
|
60
|
+
N=1 $0.0082 - $0.02 (2 calls, 5,760 tokens at most)
|
|
61
|
+
N=10 $0.21 - $0.49 (110 calls, 305,100 tokens at most)
|
|
62
|
+
N=100 $14.88 - $36.40 (10,100 calls, 27,801,000 tokens at most)
|
|
63
|
+
|
|
64
|
+
Largest cost drivers (at N=100, high end)
|
|
65
|
+
$20.00 examples/research_agent.py:47 -> summarise()
|
|
66
|
+
$16.40 examples/research_agent.py:31 anthropic.messages [claude-sonnet-5-5]
|
|
67
|
+
|
|
68
|
+
assumptions: 500 system + 500 input tokens per call, 200-1000 output tokens; default model claude-sonnet-5-5; prices as of 2026-10-11
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
(Each finding also prints its fix; trimmed here.) [`examples/research_agent_fixed.py`](examples/research_agent_fixed.py)
|
|
72
|
+
is the same agent with every finding fixed.
|
|
73
|
+
|
|
74
|
+
## Does it work?
|
|
75
|
+
|
|
76
|
+
Checked two ways, both reproducible from [`bench/`](bench/):
|
|
77
|
+
|
|
78
|
+
**Findings on real code.** AgentGuard was run on 22 well-known open-source agent projects
|
|
79
|
+
(1.9M lines) at pinned commits, and **every finding was checked by hand** against the source
|
|
80
|
+
([`bench/labels.csv`](bench/labels.csv)). The projects were split in two: 14 were used while
|
|
81
|
+
developing the rules, and false positives found there were fixed; the other 8 were only
|
|
82
|
+
looked at after the rules were frozen. The hold-out number is the honest one.
|
|
83
|
+
|
|
84
|
+
| | Projects | Lines | Findings | Correct |
|
|
85
|
+
|---|---|---|---|---|
|
|
86
|
+
| Tuning set (first review, before fixes) | 14 | 803k | 50 | 44 (88%) |
|
|
87
|
+
| **Hold-out set (rules frozen)** | **8** | **1.11M** | **36** | **33 (92%)** |
|
|
88
|
+
|
|
89
|
+
Correct findings include the history growth in OpenAI's own [swarm](https://github.com/openai/swarm/blob/6af0b4caf37dca4526dfd98e9fbd8ce36e7eeb22/swarm/core.py#L260)
|
|
90
|
+
`run()` loop, whose turn limit defaults to infinity. The misses, and what each rule gets
|
|
91
|
+
right, are in [docs/benchmarks.md](docs/benchmarks.md).
|
|
92
|
+
|
|
93
|
+
**The cost model against measured tokens.** A real chat agent that keeps its whole history
|
|
94
|
+
([`bench/ollama_agent.py`](bench/ollama_agent.py)) was run for 40 turns on a local model, and
|
|
95
|
+
AgentGuard - given only the first two turns' sizes - predicted the cumulative input tokens of
|
|
96
|
+
every later turn:
|
|
97
|
+
|
|
98
|
+
| Run | Turns | Measured input tokens | AgentGuard | Naive (N x first call) |
|
|
99
|
+
|---|---|---|---|---|
|
|
100
|
+
| Constant sizes (every reply hits a 64-token cap) | 10 | 4,495 | 4,495 (+0.0%) | 580 (-87.1%) |
|
|
101
|
+
| Constant sizes (every reply hits a 64-token cap) | 40 | 70,180 | 70,180 (+0.0%) | 2,320 (-96.7%) |
|
|
102
|
+
| Constant sizes (every reply hits a 128-token cap) | 10 | 7,430 | 7,430 (+0.0%) | 590 (-92.1%) |
|
|
103
|
+
| Constant sizes (every reply hits a 128-token cap) | 40 | 120,920 | 120,920 (+0.0%) | 2,360 (-98.0%) |
|
|
104
|
+
| Varied sizes (questions of different lengths, replies end naturally) | 10 | 13,833 | 12,360 (-10.7%) | 570 (-95.9%) |
|
|
105
|
+
| Varied sizes (questions of different lengths, replies end naturally) | 40 | 232,760 | 206,640 (-11.2%) | 2,280 (-99.0%) |
|
|
106
|
+
|
|
107
|
+
In the realistic run AgentGuard was 11% low at 40 turns (mean error 10.9% over turns 3-40,
|
|
108
|
+
worst 24.3% early on), the error coming from two calibration turns that were shorter than
|
|
109
|
+
average. The naive estimate - "N calls times the first call's size", what you get from a
|
|
110
|
+
spreadsheet - was 99% low: it is linear in the number of turns, while turn *k* re-sends all
|
|
111
|
+
*k - 1* earlier turns.
|
|
112
|
+
|
|
113
|
+
## Install
|
|
114
|
+
|
|
115
|
+
```bash
|
|
116
|
+
pip install agentguard-lint
|
|
117
|
+
agentguard . # scan the current project
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
Python 3.9+. No runtime dependencies (`tomli` on 3.9/3.10).
|
|
121
|
+
|
|
122
|
+
## Use it in CI
|
|
123
|
+
|
|
124
|
+
**GitHub Action** - findings appear in the Security tab and as PR annotations, the report in
|
|
125
|
+
the job summary:
|
|
126
|
+
|
|
127
|
+
```yaml
|
|
128
|
+
# .github/workflows/agentguard.yml
|
|
129
|
+
on: [push, pull_request]
|
|
130
|
+
jobs:
|
|
131
|
+
agentguard:
|
|
132
|
+
runs-on: ubuntu-latest
|
|
133
|
+
permissions: { contents: read, security-events: write }
|
|
134
|
+
steps:
|
|
135
|
+
- uses: actions/checkout@v5
|
|
136
|
+
- uses: muaz-q/AgentGuard@v0.3.0
|
|
137
|
+
with:
|
|
138
|
+
fail-on: high # high | medium | low | never
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
**pre-commit:**
|
|
142
|
+
|
|
143
|
+
```yaml
|
|
144
|
+
- repo: https://github.com/muaz-q/AgentGuard
|
|
145
|
+
rev: v0.3.0
|
|
146
|
+
hooks:
|
|
147
|
+
- id: agentguard
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
Exit status is 1 when a finding is at or above `--fail-on` (default `high`), 0 otherwise, 2
|
|
151
|
+
on an error.
|
|
152
|
+
|
|
153
|
+
## Use it from a coding agent (MCP)
|
|
154
|
+
|
|
155
|
+
AI coding agents write a lot of agent code. AgentGuard ships an MCP server so they can check
|
|
156
|
+
their own work: a `scan` tool that returns the findings and cost estimate as JSON, and
|
|
157
|
+
`explain_rule`. Both are read-only.
|
|
158
|
+
|
|
159
|
+
```bash
|
|
160
|
+
pip install "agentguard-lint[mcp]" # Python 3.10+
|
|
161
|
+
claude mcp add agentguard -- agentguard-mcp # Claude Code; any MCP client works the same way
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
## Rules
|
|
165
|
+
|
|
166
|
+
| Rule | Severity | Finds |
|
|
167
|
+
|---|---|---|
|
|
168
|
+
| [AG001](docs/rules.md#ag001) | high | an LLM call in a loop with no iteration cap |
|
|
169
|
+
| [AG002](docs/rules.md#ag002) | high | a history that grows each iteration and is re-sent (quadratic input) |
|
|
170
|
+
| [AG003](docs/rules.md#ag003) | medium | an LLM call inside nested open-ended loops |
|
|
171
|
+
| [AG004](docs/rules.md#ag004) | low | one LLM call per item of a collection |
|
|
172
|
+
| [AG005](docs/rules.md#ag005) | high | a function that makes LLM calls and recurses with no depth limit |
|
|
173
|
+
| [AG006](docs/rules.md#ag006) | low | a call with no output-token cap |
|
|
174
|
+
| [AG007](docs/rules.md#ag007) | medium | a large static prompt sent to Anthropic in a loop without caching |
|
|
175
|
+
|
|
176
|
+
What it understands: the OpenAI, Anthropic, Gemini (`google-genai`, `google-generativeai`),
|
|
177
|
+
LiteLLM and Ollama SDKs; LangChain chat models, chains (`prompt | llm`) and `bind_tools`;
|
|
178
|
+
calls reached through helper functions, methods and imports in other files; loop bounds
|
|
179
|
+
from literals, module constants, step counters, countdowns and retry helpers; and history
|
|
180
|
+
trimming (`messages[-k:]`, summarisation) that makes growth safe.
|
|
181
|
+
|
|
182
|
+
Silence a finding with `# agentguard: ignore[AG006]` on its line, or a whole file with
|
|
183
|
+
`# agentguard: ignore-file`.
|
|
184
|
+
|
|
185
|
+
## The cost estimate
|
|
186
|
+
|
|
187
|
+
Each call site is priced at the model named in the code (falling back to `--model`). Loops
|
|
188
|
+
with a constant bound use it; loops whose size is only known at run time are evaluated at
|
|
189
|
+
several assumed iteration counts (`--depths`, default 1, 10, 100). For a history that grows,
|
|
190
|
+
turn *k* re-sends the *k - 1* earlier turns, so over *n* turns:
|
|
191
|
+
|
|
192
|
+
```
|
|
193
|
+
input = n*S + U*n(n+1)/2 + O*n(n-1)/2 S = static prompt, U = new input per turn,
|
|
194
|
+
output = n*O O = output per turn
|
|
195
|
+
```
|
|
196
|
+
|
|
197
|
+
`S` is measured from the prompt in the code when it is a static string; otherwise `S`, `U`
|
|
198
|
+
and the `O` range are assumptions (`--system-tokens`, `--input-tokens`, `--output-tokens 200:1000`).
|
|
199
|
+
Anthropic calls that use `cache_control` have the static prefix billed at the cache-read rate.
|
|
200
|
+
Prices are bundled with an `as_of` date (`agentguard --list-models`); pass `--prices` with a
|
|
201
|
+
file in the same format or [LiteLLM's price map](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json)
|
|
202
|
+
to update them.
|
|
203
|
+
|
|
204
|
+
The estimate is a range for planning, not a bill: it cannot know how often a branch is taken
|
|
205
|
+
or how long a real reply is. The findings are the main output.
|
|
206
|
+
|
|
207
|
+
## Output and configuration
|
|
208
|
+
|
|
209
|
+
`--format text | json | sarif | markdown`, `-o FILE`. JSON has a versioned schema; SARIF 2.1.0
|
|
210
|
+
works with GitHub code scanning and most IDE SARIF viewers.
|
|
211
|
+
|
|
212
|
+
```toml
|
|
213
|
+
# pyproject.toml
|
|
214
|
+
[tool.agentguard]
|
|
215
|
+
exclude = ["tests/*", "scripts/*"]
|
|
216
|
+
ignore = ["AG004"]
|
|
217
|
+
fail_on = "medium"
|
|
218
|
+
model = "gpt-4.1-mini"
|
|
219
|
+
```
|
|
220
|
+
|
|
221
|
+
## How it works
|
|
222
|
+
|
|
223
|
+
```mermaid
|
|
224
|
+
flowchart LR
|
|
225
|
+
A[crawl .py files] --> B[pass 1: imports, functions,<br/>direct LLM calls]
|
|
226
|
+
B --> C[call graph fixpoint:<br/>which functions reach an LLM]
|
|
227
|
+
C --> D[pass 2: call sites with<br/>enclosing loops + history state]
|
|
228
|
+
D --> E[rules AG001-AG007]
|
|
229
|
+
D --> F[cost model]
|
|
230
|
+
E & F --> G[text / JSON / SARIF / Markdown]
|
|
231
|
+
```
|
|
232
|
+
|
|
233
|
+
Pure `ast`, two passes per file. The first pass learns each module's imports, functions,
|
|
234
|
+
client objects (`client = OpenAI()`, `llm = ChatOpenAI(...)`, `chain = prompt | llm`) and
|
|
235
|
+
which functions call a model directly; a fixpoint over the project call graph then finds
|
|
236
|
+
every function that reaches a model through others. The second pass records each call site
|
|
237
|
+
with its enclosing loops - classified as fixed, capped, data-driven, interactive or
|
|
238
|
+
unbounded - and tracks which variables grow, and which are trimmed, inside each loop.
|
|
239
|
+
About 12,000 lines per second on a laptop; files that cannot reach a model skip the second pass.
|
|
240
|
+
|
|
241
|
+
## Limitations
|
|
242
|
+
|
|
243
|
+
- Python only. Calls made through dynamic dispatch (`getattr`, plugin registries, a
|
|
244
|
+
provider chosen at run time) are not followed; frameworks that wrap the SDK in their own
|
|
245
|
+
abstractions are seen only where the wrapper makes the SDK call.
|
|
246
|
+
- A loop paced by external events - a websocket server, a queue consumer - is reported as
|
|
247
|
+
unbounded. `input()` and prompt libraries are recognised; servers are not.
|
|
248
|
+
- The cost model assumes every call site runs once per scenario; it does not model branch
|
|
249
|
+
probabilities.
|
|
250
|
+
|
|
251
|
+
## Development
|
|
252
|
+
|
|
253
|
+
```bash
|
|
254
|
+
pip install -e ".[dev]"
|
|
255
|
+
pytest && ruff check . && mypy
|
|
256
|
+
python bench/real_world.py --set holdout # re-run the precision benchmark
|
|
257
|
+
python bench/validate_cost_model.py # needs a local Ollama with llama3.2
|
|
258
|
+
```
|
|
259
|
+
|
|
260
|
+
See [CONTRIBUTING.md](CONTRIBUTING.md). MIT licensed.
|
|
@@ -0,0 +1,228 @@
|
|
|
1
|
+
# AgentGuard
|
|
2
|
+
|
|
3
|
+
[](https://github.com/muaz-q/AgentGuard/actions/workflows/ci.yml)
|
|
4
|
+
[](https://pypi.org/project/agentguard-lint/)
|
|
5
|
+
[](https://pypi.org/project/agentguard-lint/)
|
|
6
|
+
[](LICENSE)
|
|
7
|
+
|
|
8
|
+
**A static analyser that finds the code patterns that make LLM agents expensive - before they run.**
|
|
9
|
+
|
|
10
|
+
Agent bills rarely blow up because of one expensive call. They blow up because of loops: an
|
|
11
|
+
agent that stops only when the model says so, a conversation history that is re-sent in full
|
|
12
|
+
on every turn, a call buried three functions deep inside a loop over a dataset. AgentGuard
|
|
13
|
+
reads your Python (it never runs it or calls an API), finds those patterns with file and line,
|
|
14
|
+
and estimates what a run costs.
|
|
15
|
+
|
|
16
|
+
```console
|
|
17
|
+
$ agentguard examples/research_agent.py
|
|
18
|
+
examples/research_agent.py:21:13: AG006 [low] LLM call (openai.chat) sets no output-token cap
|
|
19
|
+
examples/research_agent.py:31:17: AG001 [high] LLM call (anthropic.messages) inside `while` loop (line 28) with no iteration cap
|
|
20
|
+
examples/research_agent.py:31:17: AG002 [high] LLM call (anthropic.messages) re-sends a history that grows every iteration; input tokens grow quadratically
|
|
21
|
+
examples/research_agent.py:31:17: AG007 [medium] ~2,260-token static prompt re-sent each iteration with no `cache_control`
|
|
22
|
+
examples/research_agent.py:47:13: AG003 [medium] call to `summarise` inside 2 nested loops (lines 45, 46); calls multiply
|
|
23
|
+
|
|
24
|
+
5 finding(s) (2 high, 2 medium, 1 low) in 1 file(s), 3 LLM call site(s).
|
|
25
|
+
|
|
26
|
+
Estimated cost per run
|
|
27
|
+
loops with no constant bound evaluated at N = 1, 10, 100
|
|
28
|
+
N=1 $0.0082 - $0.02 (2 calls, 5,760 tokens at most)
|
|
29
|
+
N=10 $0.21 - $0.49 (110 calls, 305,100 tokens at most)
|
|
30
|
+
N=100 $14.88 - $36.40 (10,100 calls, 27,801,000 tokens at most)
|
|
31
|
+
|
|
32
|
+
Largest cost drivers (at N=100, high end)
|
|
33
|
+
$20.00 examples/research_agent.py:47 -> summarise()
|
|
34
|
+
$16.40 examples/research_agent.py:31 anthropic.messages [claude-sonnet-5-5]
|
|
35
|
+
|
|
36
|
+
assumptions: 500 system + 500 input tokens per call, 200-1000 output tokens; default model claude-sonnet-5-5; prices as of 2026-10-11
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
(Each finding also prints its fix; trimmed here.) [`examples/research_agent_fixed.py`](examples/research_agent_fixed.py)
|
|
40
|
+
is the same agent with every finding fixed.
|
|
41
|
+
|
|
42
|
+
## Does it work?
|
|
43
|
+
|
|
44
|
+
Checked two ways, both reproducible from [`bench/`](bench/):
|
|
45
|
+
|
|
46
|
+
**Findings on real code.** AgentGuard was run on 22 well-known open-source agent projects
|
|
47
|
+
(1.9M lines) at pinned commits, and **every finding was checked by hand** against the source
|
|
48
|
+
([`bench/labels.csv`](bench/labels.csv)). The projects were split in two: 14 were used while
|
|
49
|
+
developing the rules, and false positives found there were fixed; the other 8 were only
|
|
50
|
+
looked at after the rules were frozen. The hold-out number is the honest one.
|
|
51
|
+
|
|
52
|
+
| | Projects | Lines | Findings | Correct |
|
|
53
|
+
|---|---|---|---|---|
|
|
54
|
+
| Tuning set (first review, before fixes) | 14 | 803k | 50 | 44 (88%) |
|
|
55
|
+
| **Hold-out set (rules frozen)** | **8** | **1.11M** | **36** | **33 (92%)** |
|
|
56
|
+
|
|
57
|
+
Correct findings include the history growth in OpenAI's own [swarm](https://github.com/openai/swarm/blob/6af0b4caf37dca4526dfd98e9fbd8ce36e7eeb22/swarm/core.py#L260)
|
|
58
|
+
`run()` loop, whose turn limit defaults to infinity. The misses, and what each rule gets
|
|
59
|
+
right, are in [docs/benchmarks.md](docs/benchmarks.md).
|
|
60
|
+
|
|
61
|
+
**The cost model against measured tokens.** A real chat agent that keeps its whole history
|
|
62
|
+
([`bench/ollama_agent.py`](bench/ollama_agent.py)) was run for 40 turns on a local model, and
|
|
63
|
+
AgentGuard - given only the first two turns' sizes - predicted the cumulative input tokens of
|
|
64
|
+
every later turn:
|
|
65
|
+
|
|
66
|
+
| Run | Turns | Measured input tokens | AgentGuard | Naive (N x first call) |
|
|
67
|
+
|---|---|---|---|---|
|
|
68
|
+
| Constant sizes (every reply hits a 64-token cap) | 10 | 4,495 | 4,495 (+0.0%) | 580 (-87.1%) |
|
|
69
|
+
| Constant sizes (every reply hits a 64-token cap) | 40 | 70,180 | 70,180 (+0.0%) | 2,320 (-96.7%) |
|
|
70
|
+
| Constant sizes (every reply hits a 128-token cap) | 10 | 7,430 | 7,430 (+0.0%) | 590 (-92.1%) |
|
|
71
|
+
| Constant sizes (every reply hits a 128-token cap) | 40 | 120,920 | 120,920 (+0.0%) | 2,360 (-98.0%) |
|
|
72
|
+
| Varied sizes (questions of different lengths, replies end naturally) | 10 | 13,833 | 12,360 (-10.7%) | 570 (-95.9%) |
|
|
73
|
+
| Varied sizes (questions of different lengths, replies end naturally) | 40 | 232,760 | 206,640 (-11.2%) | 2,280 (-99.0%) |
|
|
74
|
+
|
|
75
|
+
In the realistic run AgentGuard was 11% low at 40 turns (mean error 10.9% over turns 3-40,
|
|
76
|
+
worst 24.3% early on), the error coming from two calibration turns that were shorter than
|
|
77
|
+
average. The naive estimate - "N calls times the first call's size", what you get from a
|
|
78
|
+
spreadsheet - was 99% low: it is linear in the number of turns, while turn *k* re-sends all
|
|
79
|
+
*k - 1* earlier turns.
|
|
80
|
+
|
|
81
|
+
## Install
|
|
82
|
+
|
|
83
|
+
```bash
|
|
84
|
+
pip install agentguard-lint
|
|
85
|
+
agentguard . # scan the current project
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
Python 3.9+. No runtime dependencies (`tomli` on 3.9/3.10).
|
|
89
|
+
|
|
90
|
+
## Use it in CI
|
|
91
|
+
|
|
92
|
+
**GitHub Action** - findings appear in the Security tab and as PR annotations, the report in
|
|
93
|
+
the job summary:
|
|
94
|
+
|
|
95
|
+
```yaml
|
|
96
|
+
# .github/workflows/agentguard.yml
|
|
97
|
+
on: [push, pull_request]
|
|
98
|
+
jobs:
|
|
99
|
+
agentguard:
|
|
100
|
+
runs-on: ubuntu-latest
|
|
101
|
+
permissions: { contents: read, security-events: write }
|
|
102
|
+
steps:
|
|
103
|
+
- uses: actions/checkout@v5
|
|
104
|
+
- uses: muaz-q/AgentGuard@v0.3.0
|
|
105
|
+
with:
|
|
106
|
+
fail-on: high # high | medium | low | never
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
**pre-commit:**
|
|
110
|
+
|
|
111
|
+
```yaml
|
|
112
|
+
- repo: https://github.com/muaz-q/AgentGuard
|
|
113
|
+
rev: v0.3.0
|
|
114
|
+
hooks:
|
|
115
|
+
- id: agentguard
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
Exit status is 1 when a finding is at or above `--fail-on` (default `high`), 0 otherwise, 2
|
|
119
|
+
on an error.
|
|
120
|
+
|
|
121
|
+
## Use it from a coding agent (MCP)
|
|
122
|
+
|
|
123
|
+
AI coding agents write a lot of agent code. AgentGuard ships an MCP server so they can check
|
|
124
|
+
their own work: a `scan` tool that returns the findings and cost estimate as JSON, and
|
|
125
|
+
`explain_rule`. Both are read-only.
|
|
126
|
+
|
|
127
|
+
```bash
|
|
128
|
+
pip install "agentguard-lint[mcp]" # Python 3.10+
|
|
129
|
+
claude mcp add agentguard -- agentguard-mcp # Claude Code; any MCP client works the same way
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
## Rules
|
|
133
|
+
|
|
134
|
+
| Rule | Severity | Finds |
|
|
135
|
+
|---|---|---|
|
|
136
|
+
| [AG001](docs/rules.md#ag001) | high | an LLM call in a loop with no iteration cap |
|
|
137
|
+
| [AG002](docs/rules.md#ag002) | high | a history that grows each iteration and is re-sent (quadratic input) |
|
|
138
|
+
| [AG003](docs/rules.md#ag003) | medium | an LLM call inside nested open-ended loops |
|
|
139
|
+
| [AG004](docs/rules.md#ag004) | low | one LLM call per item of a collection |
|
|
140
|
+
| [AG005](docs/rules.md#ag005) | high | a function that makes LLM calls and recurses with no depth limit |
|
|
141
|
+
| [AG006](docs/rules.md#ag006) | low | a call with no output-token cap |
|
|
142
|
+
| [AG007](docs/rules.md#ag007) | medium | a large static prompt sent to Anthropic in a loop without caching |
|
|
143
|
+
|
|
144
|
+
What it understands: the OpenAI, Anthropic, Gemini (`google-genai`, `google-generativeai`),
|
|
145
|
+
LiteLLM and Ollama SDKs; LangChain chat models, chains (`prompt | llm`) and `bind_tools`;
|
|
146
|
+
calls reached through helper functions, methods and imports in other files; loop bounds
|
|
147
|
+
from literals, module constants, step counters, countdowns and retry helpers; and history
|
|
148
|
+
trimming (`messages[-k:]`, summarisation) that makes growth safe.
|
|
149
|
+
|
|
150
|
+
Silence a finding with `# agentguard: ignore[AG006]` on its line, or a whole file with
|
|
151
|
+
`# agentguard: ignore-file`.
|
|
152
|
+
|
|
153
|
+
## The cost estimate
|
|
154
|
+
|
|
155
|
+
Each call site is priced at the model named in the code (falling back to `--model`). Loops
|
|
156
|
+
with a constant bound use it; loops whose size is only known at run time are evaluated at
|
|
157
|
+
several assumed iteration counts (`--depths`, default 1, 10, 100). For a history that grows,
|
|
158
|
+
turn *k* re-sends the *k - 1* earlier turns, so over *n* turns:
|
|
159
|
+
|
|
160
|
+
```
|
|
161
|
+
input = n*S + U*n(n+1)/2 + O*n(n-1)/2 S = static prompt, U = new input per turn,
|
|
162
|
+
output = n*O O = output per turn
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
`S` is measured from the prompt in the code when it is a static string; otherwise `S`, `U`
|
|
166
|
+
and the `O` range are assumptions (`--system-tokens`, `--input-tokens`, `--output-tokens 200:1000`).
|
|
167
|
+
Anthropic calls that use `cache_control` have the static prefix billed at the cache-read rate.
|
|
168
|
+
Prices are bundled with an `as_of` date (`agentguard --list-models`); pass `--prices` with a
|
|
169
|
+
file in the same format or [LiteLLM's price map](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json)
|
|
170
|
+
to update them.
|
|
171
|
+
|
|
172
|
+
The estimate is a range for planning, not a bill: it cannot know how often a branch is taken
|
|
173
|
+
or how long a real reply is. The findings are the main output.
|
|
174
|
+
|
|
175
|
+
## Output and configuration
|
|
176
|
+
|
|
177
|
+
`--format text | json | sarif | markdown`, `-o FILE`. JSON has a versioned schema; SARIF 2.1.0
|
|
178
|
+
works with GitHub code scanning and most IDE SARIF viewers.
|
|
179
|
+
|
|
180
|
+
```toml
|
|
181
|
+
# pyproject.toml
|
|
182
|
+
[tool.agentguard]
|
|
183
|
+
exclude = ["tests/*", "scripts/*"]
|
|
184
|
+
ignore = ["AG004"]
|
|
185
|
+
fail_on = "medium"
|
|
186
|
+
model = "gpt-4.1-mini"
|
|
187
|
+
```
|
|
188
|
+
|
|
189
|
+
## How it works
|
|
190
|
+
|
|
191
|
+
```mermaid
|
|
192
|
+
flowchart LR
|
|
193
|
+
A[crawl .py files] --> B[pass 1: imports, functions,<br/>direct LLM calls]
|
|
194
|
+
B --> C[call graph fixpoint:<br/>which functions reach an LLM]
|
|
195
|
+
C --> D[pass 2: call sites with<br/>enclosing loops + history state]
|
|
196
|
+
D --> E[rules AG001-AG007]
|
|
197
|
+
D --> F[cost model]
|
|
198
|
+
E & F --> G[text / JSON / SARIF / Markdown]
|
|
199
|
+
```
|
|
200
|
+
|
|
201
|
+
Pure `ast`, two passes per file. The first pass learns each module's imports, functions,
|
|
202
|
+
client objects (`client = OpenAI()`, `llm = ChatOpenAI(...)`, `chain = prompt | llm`) and
|
|
203
|
+
which functions call a model directly; a fixpoint over the project call graph then finds
|
|
204
|
+
every function that reaches a model through others. The second pass records each call site
|
|
205
|
+
with its enclosing loops - classified as fixed, capped, data-driven, interactive or
|
|
206
|
+
unbounded - and tracks which variables grow, and which are trimmed, inside each loop.
|
|
207
|
+
About 12,000 lines per second on a laptop; files that cannot reach a model skip the second pass.
|
|
208
|
+
|
|
209
|
+
## Limitations
|
|
210
|
+
|
|
211
|
+
- Python only. Calls made through dynamic dispatch (`getattr`, plugin registries, a
|
|
212
|
+
provider chosen at run time) are not followed; frameworks that wrap the SDK in their own
|
|
213
|
+
abstractions are seen only where the wrapper makes the SDK call.
|
|
214
|
+
- A loop paced by external events - a websocket server, a queue consumer - is reported as
|
|
215
|
+
unbounded. `input()` and prompt libraries are recognised; servers are not.
|
|
216
|
+
- The cost model assumes every call site runs once per scenario; it does not model branch
|
|
217
|
+
probabilities.
|
|
218
|
+
|
|
219
|
+
## Development
|
|
220
|
+
|
|
221
|
+
```bash
|
|
222
|
+
pip install -e ".[dev]"
|
|
223
|
+
pytest && ruff check . && mypy
|
|
224
|
+
python bench/real_world.py --set holdout # re-run the precision benchmark
|
|
225
|
+
python bench/validate_cost_model.py # needs a local Ollama with llama3.2
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
See [CONTRIBUTING.md](CONTRIBUTING.md). MIT licensed.
|