agent-budget-semantics 0.1.0__tar.gz

agent_budget_semantics-0.1.0/.gitignore

@@ -0,0 +1,16 @@
+__pycache__/
+*.py[cod]
+*$py.class
+*.egg-info/
+dist/
+build/
+.eggs/
+*.egg
+.venv/
+venv/
+env/
+.pytest_cache/
+.mypy_cache/
+results/*.json
+.DS_Store
+.env

agent_budget_semantics-0.1.0/DIMENSIONS.md

@@ -0,0 +1,125 @@
+# Budget Semantics Divergence Dimensions
+
+24 dimensions of divergence discovered across 11 AI agent frameworks.
+
+## Core Iteration Semantics (D1-D4)
+
+| Dim | Name | Question | Scenarios |
+|-----|------|----------|-----------|
+| D1 | Iteration unit | What counts as one iteration? | S1, S2 |
+| D2 | Token accounting | Which tokens are counted? | S2, S6 |
+| D3 | Enforcement point | Pre-call or post-call budget check? | S2 |
+| D4 | Exhaustion behavior | What happens when budget runs out? | S2 |
+
+## Multi-Agent (D5-D6)
+
+| Dim | Name | Question | Scenarios |
+|-----|------|----------|-----------|
+| D5 | Budget delegation | How is budget shared with sub-agents? | S3, S10 |
+| D6 | Delegation cost | Does delegating itself consume budget? | S3, S10 |
+
+## Parallel Execution (D7-D8)
+
+| Dim | Name | Question | Scenarios |
+|-----|------|----------|-----------|
+| D7 | Parallel tool counting | N parallel tools = how many units? | S4 |
+| D8 | Partial batch execution | If budget=2 and batch=3, run 2 or 0? | S4 |
+
+## Error Handling (D9-D10)
+
+| Dim | Name | Question | Scenarios |
+|-----|------|----------|-----------|
+| D9 | Error/retry counting | Failed + retry = 1 or 2 units? | S5 |
+| D10 | Error propagation | How is the error communicated to LLM? | S5 |
+
+## Token Budget (D11-D12)
+
+| Dim | Name | Question | Scenarios |
+|-----|------|----------|-----------|
+| D11 | Token budget enforcement | Cumulative token tracking? | S6 |
+| D12 | Token counting method | Total, completion-only, or prompt-only? | S6 |
+
+## Streaming (D13-D14)
+
+| Dim | Name | Question | Scenarios |
+|-----|------|----------|-----------|
+| D13 | Streaming chunk counting | Streamed response = 1 or N budget units? | S7 |
+| D14 | Streaming token attribution | Single usage record or incremental? | S7 |
+
+## Context Growth (D15-D16)
+
+| Dim | Name | Question | Scenarios |
+|-----|------|----------|-----------|
+| D15 | Tool output token counting | Large tool output counts in budget? | S8 |
+| D16 | Context growth attribution | Growing prompt attributed to which span? | S8 |
+
+## System Prompt (D17-D18)
+
+| Dim | Name | Question | Scenarios |
+|-----|------|----------|-----------|
+| D17 | System prompt attribution | Counted once, every turn, or never? | S9 |
+| D18 | System prompt telemetry | How represented in OTel spans? | S9 |
+
+## Deep Delegation (D19-D20)
+
+| Dim | Name | Question | Scenarios |
+|-----|------|----------|-----------|
+| D19 | Nested inheritance | Budget across 3+ delegation levels? | S10 |
+| D20 | Delegation cost visibility | Parent see child's consumption? | S10 |
+
+## Fault Tolerance (D21-D22)
+
+| Dim | Name | Question | Scenarios |
+|-----|------|----------|-----------|
+| D21 | Timeout budget impact | Timed-out call consumes budget? | S11 |
+| D22 | Timeout token attribution | Tokens from failed request counted? | S11 |
+
+## Dynamic Budget (D23-D24)
+
+| Dim | Name | Question | Scenarios |
+|-----|------|----------|-----------|
+| D23 | Dynamic budget support | Can budget change mid-execution? | S12 |
+| D24 | Budget modification telemetry | How to represent changes in OTel? | S12 |
+
+## Framework Coverage Matrix
+
+| Framework | D1 | D5 | D7 | D9 | D11 | D13 | D15 | D17 | D19 | D21 | D23 |
+|-----------|----|----|----|----|-----|-----|-----|-----|-----|-----|-----|
+| AutoGen | msg | shared | N | counts | callback | transparent | configurable | every-turn | shared-pool | counts | callback |
+| OpenAI Agents | llm | shared | 1 | counts | none | transparent | no | every-turn | shared | counts | no |
+| LangChain | cycle | n/a | 1 | configurable | none | transparent | no | every-turn | n/a | configurable | no |
+| LangGraph | node | subtract | 1 | counts | none | transparent | indirect | every-turn | subtract | counts | no |
+| CrewAI | cycle | independent | n/a | free | none | n/a | no | every-turn | independent | free | no |
+| ADK | loop | remaining | 1 | counts | configurable | transparent | configurable | once | remaining | counts | no |
+| Semantic Kernel | invoke | n/a | 1 | free | none | transparent | no | every-turn | n/a | counts | mutable |
+| Anthropic | client | n/a | client | client | per-call | transparent | indirect | per-call | n/a | client | n/a |
+| Swarm | msg | shared | 2N | counts | none | transparent | no | every-turn | shared | counts | mutable |
+| LlamaIndex | step | n/a | separate | counts | none | transparent | no | every-turn | n/a | counts | no |
+| Agno | cycle | team-pool | 1 | counts | output-only | transparent | no | every-turn | team-pool | counts | no |
+
+## Key Finding
+
+For `gen_ai.agent.iteration_budget.consumed` with the SAME execution
+(4 LLM calls, 3 tool calls):
+
+```
+consumed = [3, 4, 7, 10]
+```
+
+depending on which framework. 4 different values. Not edge cases or
+implementation bugs. Fundamental design disagreements about what "one iteration"
+means.
+
+## Impact on OTel Semantic Conventions
+
+Without mandatory counting semantics metadata, the budget attributes in
+PR #439 are:
+
+1. Not comparable across frameworks (the primary use case for OTel)
+2. Meaningless for multi-framework dashboards
+3. Incorrect for cost attribution and chargeback
+4. Unstable for alert thresholds (same execution, different consumed values)
+5. Misleading for capacity planning
+
+Proposed fix: mandatory `gen_ai.agent.iteration_budget.counting_method` enum
+that classifies the framework's counting approach.

agent_budget_semantics-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Elankumaran Srinivasan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

agent_budget_semantics-0.1.0/PKG-INFO

@@ -0,0 +1,320 @@
+Metadata-Version: 2.5
+Name: agent-budget-semantics
+Version: 0.1.0
+Summary: Differential testing of budget enforcement semantics across AI agent frameworks
+Project-URL: Homepage, https://github.com/elang2/agent-budget-semantics
+Project-URL: Repository, https://github.com/elang2/agent-budget-semantics
+Project-URL: Issues, https://github.com/elang2/agent-budget-semantics/issues
+Project-URL: Documentation, https://github.com/elang2/agent-budget-semantics#readme
+Author: Elankumaran Srinivasan
+License-Expression: Apache-2.0
+License-File: LICENSE
+Keywords: agent,anthropic,autogen,budget,conformance,crewai,differential-testing,genai,google-adk,langchain,langgraph,llm,openai-agents,opentelemetry,semantic-kernel,telemetry
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.11
+Requires-Dist: httpx>=0.27
+Requires-Dist: pyyaml>=6.0
+Provides-Extra: adk
+Requires-Dist: google-adk>=1.0; extra == 'adk'
+Provides-Extra: agno
+Requires-Dist: agno>=1.0; extra == 'agno'
+Provides-Extra: all
+Requires-Dist: agno>=1.0; extra == 'all'
+Requires-Dist: anthropic>=0.34; extra == 'all'
+Requires-Dist: autogen-agentchat>=0.4; extra == 'all'
+Requires-Dist: autogen-ext>=0.4; extra == 'all'
+Requires-Dist: crewai>=0.100; extra == 'all'
+Requires-Dist: google-adk>=1.0; extra == 'all'
+Requires-Dist: langchain-openai>=0.3; extra == 'all'
+Requires-Dist: langchain>=0.3; extra == 'all'
+Requires-Dist: langgraph>=0.3; extra == 'all'
+Requires-Dist: llama-index-core>=0.11; extra == 'all'
+Requires-Dist: llama-index-llms-openai>=0.3; extra == 'all'
+Requires-Dist: openai-agents>=0.1; extra == 'all'
+Requires-Dist: openai-swarm>=0.1; extra == 'all'
+Requires-Dist: semantic-kernel>=1.0; extra == 'all'
+Provides-Extra: anthropic
+Requires-Dist: anthropic>=0.34; extra == 'anthropic'
+Provides-Extra: autogen
+Requires-Dist: autogen-agentchat>=0.4; extra == 'autogen'
+Requires-Dist: autogen-ext>=0.4; extra == 'autogen'
+Provides-Extra: crewai
+Requires-Dist: crewai>=0.100; extra == 'crewai'
+Provides-Extra: dev
+Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Provides-Extra: langchain
+Requires-Dist: langchain-openai>=0.3; extra == 'langchain'
+Requires-Dist: langchain>=0.3; extra == 'langchain'
+Provides-Extra: langgraph
+Requires-Dist: langgraph>=0.3; extra == 'langgraph'
+Provides-Extra: llamaindex
+Requires-Dist: llama-index-core>=0.11; extra == 'llamaindex'
+Requires-Dist: llama-index-llms-openai>=0.3; extra == 'llamaindex'
+Provides-Extra: openai-agents
+Requires-Dist: openai-agents>=0.1; extra == 'openai-agents'
+Provides-Extra: semantic-kernel
+Requires-Dist: semantic-kernel>=1.0; extra == 'semantic-kernel'
+Provides-Extra: swarm
+Requires-Dist: openai-swarm>=0.1; extra == 'swarm'
+Description-Content-Type: text/markdown
+
+# agent-budget-semantics
+
+Differential testing of budget enforcement semantics across 11 AI agent frameworks.
+
+## The Problem
+
+```
+gen_ai.agent.iteration_budget.consumed = [3, 4, 7, 10]
+```
+
+Same work. Same LLM calls. Same tokens consumed. Four different telemetry values depending on which framework is instrumented. Setting `budget=3` means something fundamentally different across frameworks.
+
+## The Evidence
+
+| Framework | `budget=3` means | Parallel 3 tools | Error retry | Final answer |
+|-----------|-----------------|------------------|-------------|--------------|
+| AutoGen | 3 messages (LLM + tool mixed) | 3 budget units | Counts | Counts |
+| OpenAI Agents | 3 LLM invocations | 1 budget unit | Counts | Counts |
+| LangChain | 3 tool-call cycles | 1 budget unit | Configurable | Free |
+| LangGraph | 3 node executions | 1 budget unit | Counts | Counts |
+| CrewAI | 3 tool-use cycles | N/A | Free | Free extra call |
+| Google ADK | 3 full agent loops | 1 budget unit | Counts | Part of last |
+| Semantic Kernel | 3 auto-invoke attempts | 1 budget unit | Free | Not counted |
+| Anthropic | Client-defined | Client decides | Client decides | Client decides |
+| Swarm | Messages in history | 2N budget units | Counts | Counts |
+| LlamaIndex | 3 ReAct steps | Separate budget | Counts | Free extra |
+| Agno | 3 tool-use cycles | 1 budget unit | Counts | Part of flow |
+
+## Install
+
+```bash
+pip install agent-budget-semantics
+```
+
+Or with Docker (no dependencies):
+
+```bash
+docker run --rm ghcr.io/elang2/agent-budget-semantics compare
+docker run --rm ghcr.io/elang2/agent-budget-semantics cost
+docker run --rm ghcr.io/elang2/agent-budget-semantics spans
+```
+
+## Quick Start
+
+```bash
+# Show the iteration divergence matrix
+agent-budget-semantics compare
+
+# Show cost divergence ($97K/year spread at scale)
+agent-budget-semantics cost --daily-runs 1000
+
+# Show OTel telemetry divergence (what your dashboard would show)
+agent-budget-semantics spans
+
+# Generate full report suite (markdown + JSON)
+agent-budget-semantics report --output reports/
+
+# Run differential tests against a specific framework
+pip install "agent-budget-semantics[autogen]"
+agent-budget-semantics run --scenario scenarios/s2_budget_exhaustion.yaml --frameworks autogen
+
+# Run all frameworks
+pip install "agent-budget-semantics[all]"
+agent-budget-semantics run --all
+```
+
+## What It Produces
+
+### Iteration divergence (the headline finding)
+
+```
+Framework          consumed   utilization   Counting method
+--------------------------------------------------------------------------------
+autogen            7          233%          Each message (LLM response OR tool result)
+openai_agents      4          133%          Each full LLM invocation
+langchain          3          100%          Each tool-use cycle
+langgraph          7          233%          Each graph node execution
+swarm              10         333%          Messages added to history
+```
+
+### Cost divergence (makes it tangible)
+
+```
+Monthly Cost Projection (1000 runs/day)
+----------------------------------------------------------------------
+langchain        $5,850/mo     baseline
+openai_agents    $6,750/mo     +$900 (+15%)
+autogen          $10,350/mo    +$4,500 (+77%)
+swarm            $13,950/mo    +$8,100 (+138%)
+
+Annual spread: $97,200 — from iteration counting alone.
+```
+
+### OTel span structure (what your dashboard shows)
+
+```
+Framework        Spans   consumed   util%    Structure
+------------------------------------------------------------------------------------------
+autogen          8       7          233%     root → 4 llm → 3 tool
+langchain        6       3          100%     root → 4 llm → 1 batch
+swarm            8       10         333%     root → 4 llm → 3 tool
+```
+
+## 12 Scenarios, 24 Dimensions
+
+| Scenario | Tests | Dimensions |
+|----------|-------|-----------|
+| S1: Simple tool loop | Baseline behavior | D1-D4 |
+| S2: Budget exhaustion | Enforcement boundaries | D1-D4 |
+| S3: Multi-agent delegation | Budget sharing | D5-D6 |
+| S4: Parallel tools | Batch counting | D7-D8 |
+| S5: Error/retry | Retry budget impact | D9-D10 |
+| S6: Token budget | Cumulative token tracking | D11-D12 |
+| S7: Streaming | Chunk counting | D13-D14 |
+| S8: Tool output explosion | Large response attribution | D15-D16 |
+| S9: System prompt | Repeated prompt tokens | D17-D18 |
+| S10: Nested delegation | 3-level inheritance | D19-D20 |
+| S11: Timeout/cancellation | Failed call budget impact | D21-D22 |
+| S12: Dynamic budget | Mid-run modification | D23-D24 |
+
+See [DIMENSIONS.md](DIMENSIONS.md) for the full taxonomy with per-framework behavior.
+
+## Use in CI
+
+Drop into `.github/workflows/budget-conformance.yml`:
+
+```yaml
+name: Budget Semantics Check
+on: [push, pull_request]
+
+jobs:
+  check:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+
+      - name: Install
+        run: pip install agent-budget-semantics
+
+      - name: Run comparison
+        run: |
+          agent-budget-semantics compare
+          agent-budget-semantics cost
+          agent-budget-semantics report --output budget-report/
+
+      - name: Upload report
+        uses: actions/upload-artifact@v4
+        with:
+          name: budget-divergence-report
+          path: budget-report/
+```
+
+Or with Docker (no Python setup needed):
+
+```yaml
+jobs:
+  check:
+    runs-on: ubuntu-latest
+    container:
+      image: ghcr.io/elang2/agent-budget-semantics:latest
+    steps:
+      - run: agent-budget-semantics compare
+      - run: agent-budget-semantics cost --daily-runs 500
+```
+
+## How It Works
+
+```
+┌─────────────┐     ┌──────────────┐     ┌───────────────┐
+│  Scenario   │────▶│  Mock LLM    │◀────│  Framework    │
+│  (YAML)     │     │  (ledger)    │     │  Runner       │
+└─────────────┘     └──────┬───────┘     └───────────────┘
+                           │
+                           ▼
+                    Ground Truth
+                    (actual calls,
+                     actual tokens)
+```
+
+A deterministic mock LLM with a request ledger serves as ground truth. Scripted scenarios force tool-calling loops of known depth. Each framework runner executes the same scenario against the same mock. The harness compares what each framework reports vs. what actually happened.
+
+No real LLM API keys needed. No flaky network calls. Fully reproducible.
+
+## Three Architectural Models
+
+Testing revealed three fundamentally different approaches to budget enforcement:
+
+1. **Client-side only** (Anthropic) — No server-side budget concept. The client library decides when to stop. The API has no awareness of iteration limits.
+
+2. **Framework-enforced** (9 frameworks) — The framework wraps the LLM API and applies its own budget logic. Each framework counts differently, producing the 4-value divergence.
+
+3. **Server-side opaque** (AWS Bedrock) — The server enforces budget internally. The client cannot observe or control the counting mechanism.
+
+## Relevance to OTel GenAI Conventions
+
+This project provides empirical evidence for the budget governance discussion in the OpenTelemetry semantic conventions. Without mandatory counting semantics metadata, `gen_ai.agent.iteration_budget.consumed` is not comparable across frameworks.
+
+Proposed fix: mandatory `counting_method` enum that classifies the framework's approach:
+
+```
+gen_ai.agent.iteration_budget.counting_method
+  Values: llm_calls | tool_cycles | graph_nodes | messages
+```
+
+Related PRs/Issues:
+- open-telemetry/semantic-conventions #439 (budget governance attributes)
+- open-telemetry/semantic-conventions #451 (turn count)
+- open-telemetry/semantic-conventions #447 (agent delegation)
+- open-telemetry/semantic-conventions #4025 (retry counting)
+
+## Project Structure
+
+```
+cli.py                 CLI entry point
+harness.py             Test orchestrator
+otel_comparison.py     Iteration divergence analysis
+otel_span_capture.py   OTel telemetry simulation
+cost_divergence.py     Cost impact calculator
+report_generator.py    Markdown/JSON report suite
+DIMENSIONS.md          24-dimension taxonomy
+
+mock-llm/              Deterministic mock LLM server
+  server.py            OpenAI-compatible API with request ledger
+
+runners/               Per-framework adapters (11 frameworks)
+  runner_autogen.py
+  runner_openai_agents.py
+  runner_langchain.py
+  runner_langgraph.py
+  runner_crewai.py
+  runner_adk.py
+  runner_semantic_kernel.py
+  runner_anthropic.py
+  runner_swarm.py
+  runner_llamaindex.py
+  runner_agno.py
+
+scenarios/             YAML-defined test scenarios (12 scenarios)
+  S1-S12               Covering 24 divergence dimensions
+
+tests/                 Unit tests
+reports/               Generated report artifacts
+```
+
+## License
+
+Apache-2.0