agentguard-lint 0.3.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (34) hide show
  1. agentguard_lint-0.3.0/LICENSE +21 -0
  2. agentguard_lint-0.3.0/PKG-INFO +260 -0
  3. agentguard_lint-0.3.0/README.md +228 -0
  4. agentguard_lint-0.3.0/agentguard/__init__.py +3 -0
  5. agentguard_lint-0.3.0/agentguard/__main__.py +3 -0
  6. agentguard_lint-0.3.0/agentguard/analyzer.py +1154 -0
  7. agentguard_lint-0.3.0/agentguard/cli.py +216 -0
  8. agentguard_lint-0.3.0/agentguard/config.py +51 -0
  9. agentguard_lint-0.3.0/agentguard/crawler.py +74 -0
  10. agentguard_lint-0.3.0/agentguard/data/prices.json +46 -0
  11. agentguard_lint-0.3.0/agentguard/estimator.py +177 -0
  12. agentguard_lint-0.3.0/agentguard/mcp_server.py +79 -0
  13. agentguard_lint-0.3.0/agentguard/pricing.py +107 -0
  14. agentguard_lint-0.3.0/agentguard/py.typed +0 -0
  15. agentguard_lint-0.3.0/agentguard/reporter.py +299 -0
  16. agentguard_lint-0.3.0/agentguard/rules.py +78 -0
  17. agentguard_lint-0.3.0/agentguard_lint.egg-info/PKG-INFO +260 -0
  18. agentguard_lint-0.3.0/agentguard_lint.egg-info/SOURCES.txt +32 -0
  19. agentguard_lint-0.3.0/agentguard_lint.egg-info/dependency_links.txt +1 -0
  20. agentguard_lint-0.3.0/agentguard_lint.egg-info/entry_points.txt +3 -0
  21. agentguard_lint-0.3.0/agentguard_lint.egg-info/requires.txt +18 -0
  22. agentguard_lint-0.3.0/agentguard_lint.egg-info/top_level.txt +1 -0
  23. agentguard_lint-0.3.0/pyproject.toml +77 -0
  24. agentguard_lint-0.3.0/setup.cfg +4 -0
  25. agentguard_lint-0.3.0/tests/test_cli.py +150 -0
  26. agentguard_lint-0.3.0/tests/test_crawler.py +53 -0
  27. agentguard_lint-0.3.0/tests/test_detection.py +139 -0
  28. agentguard_lint-0.3.0/tests/test_estimator.py +152 -0
  29. agentguard_lint-0.3.0/tests/test_growth.py +177 -0
  30. agentguard_lint-0.3.0/tests/test_interprocedural.py +81 -0
  31. agentguard_lint-0.3.0/tests/test_loops.py +151 -0
  32. agentguard_lint-0.3.0/tests/test_mcp_server.py +72 -0
  33. agentguard_lint-0.3.0/tests/test_pricing.py +62 -0
  34. agentguard_lint-0.3.0/tests/test_rules_misc.py +119 -0
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Muaz
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,260 @@
1
+ Metadata-Version: 2.4
2
+ Name: agentguard-lint
3
+ Version: 0.3.0
4
+ Summary: Find LLM cost risks in Python agent code - unbounded loops, quadratic history growth - and estimate what a run costs.
5
+ Author: Muaz
6
+ License-Expression: MIT
7
+ Project-URL: Homepage, https://agent-guard-six.vercel.app
8
+ Project-URL: Source, https://github.com/muaz-q/AgentGuard
9
+ Project-URL: Issues, https://github.com/muaz-q/AgentGuard/issues
10
+ Keywords: llm,agents,cost,static-analysis,linter,openai,anthropic,langchain
11
+ Classifier: Development Status :: 4 - Beta
12
+ Classifier: Environment :: Console
13
+ Classifier: Intended Audience :: Developers
14
+ Classifier: Programming Language :: Python :: 3
15
+ Classifier: Programming Language :: Python :: 3 :: Only
16
+ Classifier: Topic :: Software Development :: Quality Assurance
17
+ Classifier: Typing :: Typed
18
+ Requires-Python: >=3.9
19
+ Description-Content-Type: text/markdown
20
+ License-File: LICENSE
21
+ Requires-Dist: tomli>=1.1; python_version < "3.11"
22
+ Provides-Extra: mcp
23
+ Requires-Dist: mcp<3,>=2.3; python_version >= "3.10" and extra == "mcp"
24
+ Provides-Extra: dev
25
+ Requires-Dist: pytest>=8; extra == "dev"
26
+ Requires-Dist: pytest-cov>=5; extra == "dev"
27
+ Requires-Dist: pytest-asyncio>=1.0; python_version >= "3.10" and extra == "dev"
28
+ Requires-Dist: mcp<3,>=2.3; python_version >= "3.10" and extra == "dev"
29
+ Requires-Dist: ruff>=0.6; extra == "dev"
30
+ Requires-Dist: mypy>=1.10; extra == "dev"
31
+ Dynamic: license-file
32
+
33
+ # AgentGuard
34
+
35
+ [![CI](https://github.com/muaz-q/AgentGuard/actions/workflows/ci.yml/badge.svg)](https://github.com/muaz-q/AgentGuard/actions/workflows/ci.yml)
36
+ [![PyPI](https://img.shields.io/pypi/v/agentguard-lint)](https://pypi.org/project/agentguard-lint/)
37
+ [![Python](https://img.shields.io/pypi/pyversions/agentguard-lint)](https://pypi.org/project/agentguard-lint/)
38
+ [![License: MIT](https://img.shields.io/badge/license-MIT-blue)](LICENSE)
39
+
40
+ **A static analyser that finds the code patterns that make LLM agents expensive - before they run.**
41
+
42
+ Agent bills rarely blow up because of one expensive call. They blow up because of loops: an
43
+ agent that stops only when the model says so, a conversation history that is re-sent in full
44
+ on every turn, a call buried three functions deep inside a loop over a dataset. AgentGuard
45
+ reads your Python (it never runs it or calls an API), finds those patterns with file and line,
46
+ and estimates what a run costs.
47
+
48
+ ```console
49
+ $ agentguard examples/research_agent.py
50
+ examples/research_agent.py:21:13: AG006 [low] LLM call (openai.chat) sets no output-token cap
51
+ examples/research_agent.py:31:17: AG001 [high] LLM call (anthropic.messages) inside `while` loop (line 28) with no iteration cap
52
+ examples/research_agent.py:31:17: AG002 [high] LLM call (anthropic.messages) re-sends a history that grows every iteration; input tokens grow quadratically
53
+ examples/research_agent.py:31:17: AG007 [medium] ~2,260-token static prompt re-sent each iteration with no `cache_control`
54
+ examples/research_agent.py:47:13: AG003 [medium] call to `summarise` inside 2 nested loops (lines 45, 46); calls multiply
55
+
56
+ 5 finding(s) (2 high, 2 medium, 1 low) in 1 file(s), 3 LLM call site(s).
57
+
58
+ Estimated cost per run
59
+ loops with no constant bound evaluated at N = 1, 10, 100
60
+ N=1 $0.0082 - $0.02 (2 calls, 5,760 tokens at most)
61
+ N=10 $0.21 - $0.49 (110 calls, 305,100 tokens at most)
62
+ N=100 $14.88 - $36.40 (10,100 calls, 27,801,000 tokens at most)
63
+
64
+ Largest cost drivers (at N=100, high end)
65
+ $20.00 examples/research_agent.py:47 -> summarise()
66
+ $16.40 examples/research_agent.py:31 anthropic.messages [claude-sonnet-5-5]
67
+
68
+ assumptions: 500 system + 500 input tokens per call, 200-1000 output tokens; default model claude-sonnet-5-5; prices as of 2026-10-11
69
+ ```
70
+
71
+ (Each finding also prints its fix; trimmed here.) [`examples/research_agent_fixed.py`](examples/research_agent_fixed.py)
72
+ is the same agent with every finding fixed.
73
+
74
+ ## Does it work?
75
+
76
+ Checked two ways, both reproducible from [`bench/`](bench/):
77
+
78
+ **Findings on real code.** AgentGuard was run on 22 well-known open-source agent projects
79
+ (1.9M lines) at pinned commits, and **every finding was checked by hand** against the source
80
+ ([`bench/labels.csv`](bench/labels.csv)). The projects were split in two: 14 were used while
81
+ developing the rules, and false positives found there were fixed; the other 8 were only
82
+ looked at after the rules were frozen. The hold-out number is the honest one.
83
+
84
+ | | Projects | Lines | Findings | Correct |
85
+ |---|---|---|---|---|
86
+ | Tuning set (first review, before fixes) | 14 | 803k | 50 | 44 (88%) |
87
+ | **Hold-out set (rules frozen)** | **8** | **1.11M** | **36** | **33 (92%)** |
88
+
89
+ Correct findings include the history growth in OpenAI's own [swarm](https://github.com/openai/swarm/blob/6af0b4caf37dca4526dfd98e9fbd8ce36e7eeb22/swarm/core.py#L260)
90
+ `run()` loop, whose turn limit defaults to infinity. The misses, and what each rule gets
91
+ right, are in [docs/benchmarks.md](docs/benchmarks.md).
92
+
93
+ **The cost model against measured tokens.** A real chat agent that keeps its whole history
94
+ ([`bench/ollama_agent.py`](bench/ollama_agent.py)) was run for 40 turns on a local model, and
95
+ AgentGuard - given only the first two turns' sizes - predicted the cumulative input tokens of
96
+ every later turn:
97
+
98
+ | Run | Turns | Measured input tokens | AgentGuard | Naive (N x first call) |
99
+ |---|---|---|---|---|
100
+ | Constant sizes (every reply hits a 64-token cap) | 10 | 4,495 | 4,495 (+0.0%) | 580 (-87.1%) |
101
+ | Constant sizes (every reply hits a 64-token cap) | 40 | 70,180 | 70,180 (+0.0%) | 2,320 (-96.7%) |
102
+ | Constant sizes (every reply hits a 128-token cap) | 10 | 7,430 | 7,430 (+0.0%) | 590 (-92.1%) |
103
+ | Constant sizes (every reply hits a 128-token cap) | 40 | 120,920 | 120,920 (+0.0%) | 2,360 (-98.0%) |
104
+ | Varied sizes (questions of different lengths, replies end naturally) | 10 | 13,833 | 12,360 (-10.7%) | 570 (-95.9%) |
105
+ | Varied sizes (questions of different lengths, replies end naturally) | 40 | 232,760 | 206,640 (-11.2%) | 2,280 (-99.0%) |
106
+
107
+ In the realistic run AgentGuard was 11% low at 40 turns (mean error 10.9% over turns 3-40,
108
+ worst 24.3% early on), the error coming from two calibration turns that were shorter than
109
+ average. The naive estimate - "N calls times the first call's size", what you get from a
110
+ spreadsheet - was 99% low: it is linear in the number of turns, while turn *k* re-sends all
111
+ *k - 1* earlier turns.
112
+
113
+ ## Install
114
+
115
+ ```bash
116
+ pip install agentguard-lint
117
+ agentguard . # scan the current project
118
+ ```
119
+
120
+ Python 3.9+. No runtime dependencies (`tomli` on 3.9/3.10).
121
+
122
+ ## Use it in CI
123
+
124
+ **GitHub Action** - findings appear in the Security tab and as PR annotations, the report in
125
+ the job summary:
126
+
127
+ ```yaml
128
+ # .github/workflows/agentguard.yml
129
+ on: [push, pull_request]
130
+ jobs:
131
+ agentguard:
132
+ runs-on: ubuntu-latest
133
+ permissions: { contents: read, security-events: write }
134
+ steps:
135
+ - uses: actions/checkout@v5
136
+ - uses: muaz-q/AgentGuard@v0.3.0
137
+ with:
138
+ fail-on: high # high | medium | low | never
139
+ ```
140
+
141
+ **pre-commit:**
142
+
143
+ ```yaml
144
+ - repo: https://github.com/muaz-q/AgentGuard
145
+ rev: v0.3.0
146
+ hooks:
147
+ - id: agentguard
148
+ ```
149
+
150
+ Exit status is 1 when a finding is at or above `--fail-on` (default `high`), 0 otherwise, 2
151
+ on an error.
152
+
153
+ ## Use it from a coding agent (MCP)
154
+
155
+ AI coding agents write a lot of agent code. AgentGuard ships an MCP server so they can check
156
+ their own work: a `scan` tool that returns the findings and cost estimate as JSON, and
157
+ `explain_rule`. Both are read-only.
158
+
159
+ ```bash
160
+ pip install "agentguard-lint[mcp]" # Python 3.10+
161
+ claude mcp add agentguard -- agentguard-mcp # Claude Code; any MCP client works the same way
162
+ ```
163
+
164
+ ## Rules
165
+
166
+ | Rule | Severity | Finds |
167
+ |---|---|---|
168
+ | [AG001](docs/rules.md#ag001) | high | an LLM call in a loop with no iteration cap |
169
+ | [AG002](docs/rules.md#ag002) | high | a history that grows each iteration and is re-sent (quadratic input) |
170
+ | [AG003](docs/rules.md#ag003) | medium | an LLM call inside nested open-ended loops |
171
+ | [AG004](docs/rules.md#ag004) | low | one LLM call per item of a collection |
172
+ | [AG005](docs/rules.md#ag005) | high | a function that makes LLM calls and recurses with no depth limit |
173
+ | [AG006](docs/rules.md#ag006) | low | a call with no output-token cap |
174
+ | [AG007](docs/rules.md#ag007) | medium | a large static prompt sent to Anthropic in a loop without caching |
175
+
176
+ What it understands: the OpenAI, Anthropic, Gemini (`google-genai`, `google-generativeai`),
177
+ LiteLLM and Ollama SDKs; LangChain chat models, chains (`prompt | llm`) and `bind_tools`;
178
+ calls reached through helper functions, methods and imports in other files; loop bounds
179
+ from literals, module constants, step counters, countdowns and retry helpers; and history
180
+ trimming (`messages[-k:]`, summarisation) that makes growth safe.
181
+
182
+ Silence a finding with `# agentguard: ignore[AG006]` on its line, or a whole file with
183
+ `# agentguard: ignore-file`.
184
+
185
+ ## The cost estimate
186
+
187
+ Each call site is priced at the model named in the code (falling back to `--model`). Loops
188
+ with a constant bound use it; loops whose size is only known at run time are evaluated at
189
+ several assumed iteration counts (`--depths`, default 1, 10, 100). For a history that grows,
190
+ turn *k* re-sends the *k - 1* earlier turns, so over *n* turns:
191
+
192
+ ```
193
+ input = n*S + U*n(n+1)/2 + O*n(n-1)/2 S = static prompt, U = new input per turn,
194
+ output = n*O O = output per turn
195
+ ```
196
+
197
+ `S` is measured from the prompt in the code when it is a static string; otherwise `S`, `U`
198
+ and the `O` range are assumptions (`--system-tokens`, `--input-tokens`, `--output-tokens 200:1000`).
199
+ Anthropic calls that use `cache_control` have the static prefix billed at the cache-read rate.
200
+ Prices are bundled with an `as_of` date (`agentguard --list-models`); pass `--prices` with a
201
+ file in the same format or [LiteLLM's price map](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json)
202
+ to update them.
203
+
204
+ The estimate is a range for planning, not a bill: it cannot know how often a branch is taken
205
+ or how long a real reply is. The findings are the main output.
206
+
207
+ ## Output and configuration
208
+
209
+ `--format text | json | sarif | markdown`, `-o FILE`. JSON has a versioned schema; SARIF 2.1.0
210
+ works with GitHub code scanning and most IDE SARIF viewers.
211
+
212
+ ```toml
213
+ # pyproject.toml
214
+ [tool.agentguard]
215
+ exclude = ["tests/*", "scripts/*"]
216
+ ignore = ["AG004"]
217
+ fail_on = "medium"
218
+ model = "gpt-4.1-mini"
219
+ ```
220
+
221
+ ## How it works
222
+
223
+ ```mermaid
224
+ flowchart LR
225
+ A[crawl .py files] --> B[pass 1: imports, functions,<br/>direct LLM calls]
226
+ B --> C[call graph fixpoint:<br/>which functions reach an LLM]
227
+ C --> D[pass 2: call sites with<br/>enclosing loops + history state]
228
+ D --> E[rules AG001-AG007]
229
+ D --> F[cost model]
230
+ E & F --> G[text / JSON / SARIF / Markdown]
231
+ ```
232
+
233
+ Pure `ast`, two passes per file. The first pass learns each module's imports, functions,
234
+ client objects (`client = OpenAI()`, `llm = ChatOpenAI(...)`, `chain = prompt | llm`) and
235
+ which functions call a model directly; a fixpoint over the project call graph then finds
236
+ every function that reaches a model through others. The second pass records each call site
237
+ with its enclosing loops - classified as fixed, capped, data-driven, interactive or
238
+ unbounded - and tracks which variables grow, and which are trimmed, inside each loop.
239
+ About 12,000 lines per second on a laptop; files that cannot reach a model skip the second pass.
240
+
241
+ ## Limitations
242
+
243
+ - Python only. Calls made through dynamic dispatch (`getattr`, plugin registries, a
244
+ provider chosen at run time) are not followed; frameworks that wrap the SDK in their own
245
+ abstractions are seen only where the wrapper makes the SDK call.
246
+ - A loop paced by external events - a websocket server, a queue consumer - is reported as
247
+ unbounded. `input()` and prompt libraries are recognised; servers are not.
248
+ - The cost model assumes every call site runs once per scenario; it does not model branch
249
+ probabilities.
250
+
251
+ ## Development
252
+
253
+ ```bash
254
+ pip install -e ".[dev]"
255
+ pytest && ruff check . && mypy
256
+ python bench/real_world.py --set holdout # re-run the precision benchmark
257
+ python bench/validate_cost_model.py # needs a local Ollama with llama3.2
258
+ ```
259
+
260
+ See [CONTRIBUTING.md](CONTRIBUTING.md). MIT licensed.
@@ -0,0 +1,228 @@
1
+ # AgentGuard
2
+
3
+ [![CI](https://github.com/muaz-q/AgentGuard/actions/workflows/ci.yml/badge.svg)](https://github.com/muaz-q/AgentGuard/actions/workflows/ci.yml)
4
+ [![PyPI](https://img.shields.io/pypi/v/agentguard-lint)](https://pypi.org/project/agentguard-lint/)
5
+ [![Python](https://img.shields.io/pypi/pyversions/agentguard-lint)](https://pypi.org/project/agentguard-lint/)
6
+ [![License: MIT](https://img.shields.io/badge/license-MIT-blue)](LICENSE)
7
+
8
+ **A static analyser that finds the code patterns that make LLM agents expensive - before they run.**
9
+
10
+ Agent bills rarely blow up because of one expensive call. They blow up because of loops: an
11
+ agent that stops only when the model says so, a conversation history that is re-sent in full
12
+ on every turn, a call buried three functions deep inside a loop over a dataset. AgentGuard
13
+ reads your Python (it never runs it or calls an API), finds those patterns with file and line,
14
+ and estimates what a run costs.
15
+
16
+ ```console
17
+ $ agentguard examples/research_agent.py
18
+ examples/research_agent.py:21:13: AG006 [low] LLM call (openai.chat) sets no output-token cap
19
+ examples/research_agent.py:31:17: AG001 [high] LLM call (anthropic.messages) inside `while` loop (line 28) with no iteration cap
20
+ examples/research_agent.py:31:17: AG002 [high] LLM call (anthropic.messages) re-sends a history that grows every iteration; input tokens grow quadratically
21
+ examples/research_agent.py:31:17: AG007 [medium] ~2,260-token static prompt re-sent each iteration with no `cache_control`
22
+ examples/research_agent.py:47:13: AG003 [medium] call to `summarise` inside 2 nested loops (lines 45, 46); calls multiply
23
+
24
+ 5 finding(s) (2 high, 2 medium, 1 low) in 1 file(s), 3 LLM call site(s).
25
+
26
+ Estimated cost per run
27
+ loops with no constant bound evaluated at N = 1, 10, 100
28
+ N=1 $0.0082 - $0.02 (2 calls, 5,760 tokens at most)
29
+ N=10 $0.21 - $0.49 (110 calls, 305,100 tokens at most)
30
+ N=100 $14.88 - $36.40 (10,100 calls, 27,801,000 tokens at most)
31
+
32
+ Largest cost drivers (at N=100, high end)
33
+ $20.00 examples/research_agent.py:47 -> summarise()
34
+ $16.40 examples/research_agent.py:31 anthropic.messages [claude-sonnet-5-5]
35
+
36
+ assumptions: 500 system + 500 input tokens per call, 200-1000 output tokens; default model claude-sonnet-5-5; prices as of 2026-10-11
37
+ ```
38
+
39
+ (Each finding also prints its fix; trimmed here.) [`examples/research_agent_fixed.py`](examples/research_agent_fixed.py)
40
+ is the same agent with every finding fixed.
41
+
42
+ ## Does it work?
43
+
44
+ Checked two ways, both reproducible from [`bench/`](bench/):
45
+
46
+ **Findings on real code.** AgentGuard was run on 22 well-known open-source agent projects
47
+ (1.9M lines) at pinned commits, and **every finding was checked by hand** against the source
48
+ ([`bench/labels.csv`](bench/labels.csv)). The projects were split in two: 14 were used while
49
+ developing the rules, and false positives found there were fixed; the other 8 were only
50
+ looked at after the rules were frozen. The hold-out number is the honest one.
51
+
52
+ | | Projects | Lines | Findings | Correct |
53
+ |---|---|---|---|---|
54
+ | Tuning set (first review, before fixes) | 14 | 803k | 50 | 44 (88%) |
55
+ | **Hold-out set (rules frozen)** | **8** | **1.11M** | **36** | **33 (92%)** |
56
+
57
+ Correct findings include the history growth in OpenAI's own [swarm](https://github.com/openai/swarm/blob/6af0b4caf37dca4526dfd98e9fbd8ce36e7eeb22/swarm/core.py#L260)
58
+ `run()` loop, whose turn limit defaults to infinity. The misses, and what each rule gets
59
+ right, are in [docs/benchmarks.md](docs/benchmarks.md).
60
+
61
+ **The cost model against measured tokens.** A real chat agent that keeps its whole history
62
+ ([`bench/ollama_agent.py`](bench/ollama_agent.py)) was run for 40 turns on a local model, and
63
+ AgentGuard - given only the first two turns' sizes - predicted the cumulative input tokens of
64
+ every later turn:
65
+
66
+ | Run | Turns | Measured input tokens | AgentGuard | Naive (N x first call) |
67
+ |---|---|---|---|---|
68
+ | Constant sizes (every reply hits a 64-token cap) | 10 | 4,495 | 4,495 (+0.0%) | 580 (-87.1%) |
69
+ | Constant sizes (every reply hits a 64-token cap) | 40 | 70,180 | 70,180 (+0.0%) | 2,320 (-96.7%) |
70
+ | Constant sizes (every reply hits a 128-token cap) | 10 | 7,430 | 7,430 (+0.0%) | 590 (-92.1%) |
71
+ | Constant sizes (every reply hits a 128-token cap) | 40 | 120,920 | 120,920 (+0.0%) | 2,360 (-98.0%) |
72
+ | Varied sizes (questions of different lengths, replies end naturally) | 10 | 13,833 | 12,360 (-10.7%) | 570 (-95.9%) |
73
+ | Varied sizes (questions of different lengths, replies end naturally) | 40 | 232,760 | 206,640 (-11.2%) | 2,280 (-99.0%) |
74
+
75
+ In the realistic run AgentGuard was 11% low at 40 turns (mean error 10.9% over turns 3-40,
76
+ worst 24.3% early on), the error coming from two calibration turns that were shorter than
77
+ average. The naive estimate - "N calls times the first call's size", what you get from a
78
+ spreadsheet - was 99% low: it is linear in the number of turns, while turn *k* re-sends all
79
+ *k - 1* earlier turns.
80
+
81
+ ## Install
82
+
83
+ ```bash
84
+ pip install agentguard-lint
85
+ agentguard . # scan the current project
86
+ ```
87
+
88
+ Python 3.9+. No runtime dependencies (`tomli` on 3.9/3.10).
89
+
90
+ ## Use it in CI
91
+
92
+ **GitHub Action** - findings appear in the Security tab and as PR annotations, the report in
93
+ the job summary:
94
+
95
+ ```yaml
96
+ # .github/workflows/agentguard.yml
97
+ on: [push, pull_request]
98
+ jobs:
99
+ agentguard:
100
+ runs-on: ubuntu-latest
101
+ permissions: { contents: read, security-events: write }
102
+ steps:
103
+ - uses: actions/checkout@v5
104
+ - uses: muaz-q/AgentGuard@v0.3.0
105
+ with:
106
+ fail-on: high # high | medium | low | never
107
+ ```
108
+
109
+ **pre-commit:**
110
+
111
+ ```yaml
112
+ - repo: https://github.com/muaz-q/AgentGuard
113
+ rev: v0.3.0
114
+ hooks:
115
+ - id: agentguard
116
+ ```
117
+
118
+ Exit status is 1 when a finding is at or above `--fail-on` (default `high`), 0 otherwise, 2
119
+ on an error.
120
+
121
+ ## Use it from a coding agent (MCP)
122
+
123
+ AI coding agents write a lot of agent code. AgentGuard ships an MCP server so they can check
124
+ their own work: a `scan` tool that returns the findings and cost estimate as JSON, and
125
+ `explain_rule`. Both are read-only.
126
+
127
+ ```bash
128
+ pip install "agentguard-lint[mcp]" # Python 3.10+
129
+ claude mcp add agentguard -- agentguard-mcp # Claude Code; any MCP client works the same way
130
+ ```
131
+
132
+ ## Rules
133
+
134
+ | Rule | Severity | Finds |
135
+ |---|---|---|
136
+ | [AG001](docs/rules.md#ag001) | high | an LLM call in a loop with no iteration cap |
137
+ | [AG002](docs/rules.md#ag002) | high | a history that grows each iteration and is re-sent (quadratic input) |
138
+ | [AG003](docs/rules.md#ag003) | medium | an LLM call inside nested open-ended loops |
139
+ | [AG004](docs/rules.md#ag004) | low | one LLM call per item of a collection |
140
+ | [AG005](docs/rules.md#ag005) | high | a function that makes LLM calls and recurses with no depth limit |
141
+ | [AG006](docs/rules.md#ag006) | low | a call with no output-token cap |
142
+ | [AG007](docs/rules.md#ag007) | medium | a large static prompt sent to Anthropic in a loop without caching |
143
+
144
+ What it understands: the OpenAI, Anthropic, Gemini (`google-genai`, `google-generativeai`),
145
+ LiteLLM and Ollama SDKs; LangChain chat models, chains (`prompt | llm`) and `bind_tools`;
146
+ calls reached through helper functions, methods and imports in other files; loop bounds
147
+ from literals, module constants, step counters, countdowns and retry helpers; and history
148
+ trimming (`messages[-k:]`, summarisation) that makes growth safe.
149
+
150
+ Silence a finding with `# agentguard: ignore[AG006]` on its line, or a whole file with
151
+ `# agentguard: ignore-file`.
152
+
153
+ ## The cost estimate
154
+
155
+ Each call site is priced at the model named in the code (falling back to `--model`). Loops
156
+ with a constant bound use it; loops whose size is only known at run time are evaluated at
157
+ several assumed iteration counts (`--depths`, default 1, 10, 100). For a history that grows,
158
+ turn *k* re-sends the *k - 1* earlier turns, so over *n* turns:
159
+
160
+ ```
161
+ input = n*S + U*n(n+1)/2 + O*n(n-1)/2 S = static prompt, U = new input per turn,
162
+ output = n*O O = output per turn
163
+ ```
164
+
165
+ `S` is measured from the prompt in the code when it is a static string; otherwise `S`, `U`
166
+ and the `O` range are assumptions (`--system-tokens`, `--input-tokens`, `--output-tokens 200:1000`).
167
+ Anthropic calls that use `cache_control` have the static prefix billed at the cache-read rate.
168
+ Prices are bundled with an `as_of` date (`agentguard --list-models`); pass `--prices` with a
169
+ file in the same format or [LiteLLM's price map](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json)
170
+ to update them.
171
+
172
+ The estimate is a range for planning, not a bill: it cannot know how often a branch is taken
173
+ or how long a real reply is. The findings are the main output.
174
+
175
+ ## Output and configuration
176
+
177
+ `--format text | json | sarif | markdown`, `-o FILE`. JSON has a versioned schema; SARIF 2.1.0
178
+ works with GitHub code scanning and most IDE SARIF viewers.
179
+
180
+ ```toml
181
+ # pyproject.toml
182
+ [tool.agentguard]
183
+ exclude = ["tests/*", "scripts/*"]
184
+ ignore = ["AG004"]
185
+ fail_on = "medium"
186
+ model = "gpt-4.1-mini"
187
+ ```
188
+
189
+ ## How it works
190
+
191
+ ```mermaid
192
+ flowchart LR
193
+ A[crawl .py files] --> B[pass 1: imports, functions,<br/>direct LLM calls]
194
+ B --> C[call graph fixpoint:<br/>which functions reach an LLM]
195
+ C --> D[pass 2: call sites with<br/>enclosing loops + history state]
196
+ D --> E[rules AG001-AG007]
197
+ D --> F[cost model]
198
+ E & F --> G[text / JSON / SARIF / Markdown]
199
+ ```
200
+
201
+ Pure `ast`, two passes per file. The first pass learns each module's imports, functions,
202
+ client objects (`client = OpenAI()`, `llm = ChatOpenAI(...)`, `chain = prompt | llm`) and
203
+ which functions call a model directly; a fixpoint over the project call graph then finds
204
+ every function that reaches a model through others. The second pass records each call site
205
+ with its enclosing loops - classified as fixed, capped, data-driven, interactive or
206
+ unbounded - and tracks which variables grow, and which are trimmed, inside each loop.
207
+ About 12,000 lines per second on a laptop; files that cannot reach a model skip the second pass.
208
+
209
+ ## Limitations
210
+
211
+ - Python only. Calls made through dynamic dispatch (`getattr`, plugin registries, a
212
+ provider chosen at run time) are not followed; frameworks that wrap the SDK in their own
213
+ abstractions are seen only where the wrapper makes the SDK call.
214
+ - A loop paced by external events - a websocket server, a queue consumer - is reported as
215
+ unbounded. `input()` and prompt libraries are recognised; servers are not.
216
+ - The cost model assumes every call site runs once per scenario; it does not model branch
217
+ probabilities.
218
+
219
+ ## Development
220
+
221
+ ```bash
222
+ pip install -e ".[dev]"
223
+ pytest && ruff check . && mypy
224
+ python bench/real_world.py --set holdout # re-run the precision benchmark
225
+ python bench/validate_cost_model.py # needs a local Ollama with llama3.2
226
+ ```
227
+
228
+ See [CONTRIBUTING.md](CONTRIBUTING.md). MIT licensed.
@@ -0,0 +1,3 @@
1
+ """AgentGuard: find LLM cost risks in Python agent code before they reach production."""
2
+
3
+ __version__ = "0.3.0"
@@ -0,0 +1,3 @@
1
+ from agentguard.cli import entry
2
+
3
+ entry()