agenticlens 0.1.0__tar.gz

agenticlens-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,38 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.10", "3.11", "3.12", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v3
+        with:
+          enable-cache: true
+
+      - name: Set up Python ${{ matrix.python-version }}
+        run: uv python install ${{ matrix.python-version }}
+
+      - name: Install dependencies
+        run: uv sync --extra dev --python ${{ matrix.python-version }}
+
+      - name: Lint (ruff check)
+        run: uv run ruff check .
+
+      - name: Format check (ruff format)
+        run: uv run ruff format --check .
+
+      - name: Type check (mypy)
+        run: uv run mypy
+
+      - name: Test (pytest)
+        run: uv run pytest

agenticlens-0.1.0/.gitignore

@@ -0,0 +1,14 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+.eggs/
+.venv/
+.mypy_cache/
+.ruff_cache/
+.pytest_cache/
+htmlcov/
+.coverage
+dist/
+build/
+site/
+.env

agenticlens-0.1.0/AgenticLens_Spec.md

@@ -0,0 +1,463 @@
+# AgenticLens — Project Master Specification (MVP)
+
+---
+
+## Project Overview
+
+**AgenticLens** is an open-source Python library that helps developers **profile, analyze, and optimize token consumption** in LLM-powered applications and agentic workflows.
+
+Unlike traditional observability tools that only display token usage and cost, AgenticLens explains:
+
+- **Where** tokens were consumed
+- **Why** they were consumed
+- **Which** workflow step was responsible
+- **Which** tool calls were expensive
+- **How** developers can reduce token usage and cost
+
+> **Design principle:** Framework-agnostic and provider-agnostic.
+
+---
+
+## Vision
+
+Modern AI applications consist of multiple LLM calls, planning agents, memory systems, retrieval pipelines, tool calls, MCP servers, and multi-agent workflows.
+
+Current observability tools show traces and token counts but provide **limited guidance on optimization**.
+
+AgenticLens should become the **"performance profiler" for AI applications**, similar to:
+
+| Analogy | Domain |
+|---|---|
+| `cProfile` | Python runtime |
+| Chrome DevTools | Browser performance |
+| TensorBoard | ML training |
+
+### The library should ultimately answer three questions
+
+1. Where did the tokens go?
+2. Why were they spent?
+3. How can developers reduce them?
+
+---
+
+## Initial Scope (MVP)
+
+The first version focuses on **profiling and reporting**.
+
+> **Out of scope for MVP:** dashboards, databases, web applications, enterprise features.
+
+Keep the MVP lightweight and suitable for publishing as a PyPI package.
+
+---
+
+## Primary Features
+
+### 1. Workflow Profiler
+
+Profile an entire workflow with a single context manager:
+
+```python
+from agenticlens import profile
+
+with profile("Customer Support"):
+    agent.run(question)
+```
+
+The profiler establishes the workflow context (start/end time, id) but does **not** auto-capture LLM calls — see [Instrumentation Model](#instrumentation-model) below.
+
+---
+
+### 2. Step Profiling
+
+Every workflow consists of multiple steps. Each step should have independent metrics, and is wrapped **explicitly** by the developer:
+
+```python
+from agenticlens import profile, step
+
+with profile("Customer Support"):
+    with step("Planner", type="planner") as s:
+        plan = planner_llm.invoke(prompt)
+        s.record(plan)  # attaches token usage from the provider response
+
+    with step("Retriever", type="retriever") as s:
+        chunks = retriever.search(query)
+
+    with step("Final Response", type="llm_call") as s:
+        answer = response_llm.invoke(context)
+        s.record(answer)
+```
+
+`step()` yields a handle whose `.record(response)` extracts token usage from a provider response (auto-detected via the provider registry) and attaches it to the step. Steps are recorded against the currently active workflow via a `contextvar`-based stack, so calling `step()` outside of a `profile()` block raises a clear error.
+
+**Example step types:**
+
+- Planner
+- Retriever
+- Tool Call
+- LLM Call
+- Memory
+- Final Response
+
+#### Instrumentation Model
+
+> **Decision:** AgenticLens uses an **explicit step API**, not automatic SDK monkey-patching.
+
+Rationale:
+
+- Reliable across provider SDK versions — no breakage when OpenAI/Anthropic change internal client internals.
+- Testable without mocking global state.
+- Step boundaries (Planner vs. Retriever vs. Tool Call) are a workflow concept the library cannot infer reliably; the developer already knows them.
+
+Trade-off accepted: integrating AgenticLens requires adding `with step(...):` around LLM/tool calls. This cost is acceptable for an MVP and can be revisited post-MVP with optional auto-instrumentation adapters (e.g. for LangChain callbacks) once the core model is stable.
+
+---
+
+### 3. Token Metrics
+
+Collected per step:
+
+| Metric | Description |
+|---|---|
+| `prompt_tokens` | Tokens in the input prompt |
+| `completion_tokens` | Tokens in the model output |
+| `total_tokens` | Sum of prompt + completion |
+
+---
+
+### 4. Cost Metrics
+
+| Metric | Description |
+|---|---|
+| `input_cost` | Cost of prompt tokens |
+| `output_cost` | Cost of completion tokens |
+| `total_cost` | Total spend for the step |
+
+**Pricing source of truth:** a static pricing table bundled at `src/agenticlens/config/pricing.yaml`, keyed by `provider:model`, kept current via periodic manual updates (community PRs welcome). Resolution order:
+
+1. User-supplied pricing override (via `pyproject.toml` / YAML config / env var) — always wins.
+2. Bundled `pricing.yaml` entry for the exact `provider:model`.
+3. Unknown model → cost fields are `None` and a `UnknownModelPricingWarning` is emitted. **Never silently report `$0.00`** for an unpriced model — that's misleading, not a graceful fallback.
+
+---
+
+### 5. Performance Metrics
+
+| Metric | Description |
+|---|---|
+| `latency` | Total step duration |
+| `ttft` | Time To First Token. `float \| None` — only populated when the wrapped call is a streaming call; `None` for non-streaming calls (the common case). Not an error or missing-data condition. |
+| `tps` | Tokens Per Second (`completion_tokens / latency`) |
+
+---
+
+### 6. Workflow Summary Output
+
+```
+╔══════════════════════════════╗
+║   Customer Support Agent     ║
+╠══════════════════════════════╣
+║  Total Tokens    │  24,581   ║
+║  Total Cost      │  $0.24    ║
+║  Latency         │  18.2 sec ║
+╚══════════════════════════════╝
+```
+
+---
+
+### 7. Step Breakdown Output
+
+```
+Planner
+───────────────────────────────
+  Prompt Tokens      850
+  Completion Tokens  210
+  Cost               $0.02
+  Latency            1.1 sec
+```
+
+Repeated for every workflow step.
+
+---
+
+## Recommendation Engine (Rule-Based)
+
+AgenticLens does not stop at reporting — it **analyzes** the workflow and generates actionable optimization suggestions.
+
+> Initial recommendations use **simple heuristics**, not AI-generated analysis.
+
+### MVP Heuristic Rules
+
+Each rule below detects a pattern and estimates the tokens it would save (`tokens_saved`), which feeds the savings calculation.
+
+| Rule | Detection logic | `tokens_saved` estimate |
+|---|---|---|
+| **Repeated system prompt** | Hash the first N tokens (configurable, default 50) of each step's prompt. If the same hash appears in ≥2 steps, flag all occurrences after the first. | Sum of token counts of the repeated prefix across the duplicate occurrences. |
+| **Excessive retrieved chunks** | Retriever step returns more than `max_chunks` (configurable, default 8) chunks. | `(chunk_count - max_chunks) × avg_tokens_per_chunk`. |
+| **Long conversation history** | A step's `prompt_tokens` attributable to history/memory content exceeds `history_token_limit` (configurable, default 4000). | `prompt_tokens_from_history - history_token_limit`. |
+| **Duplicate tool calls** | Two or more Tool Call steps in the same workflow share an identical `(tool_name, arguments)` signature. | Sum of `prompt_tokens + completion_tokens` for every duplicate occurrence after the first. |
+
+All thresholds live in `RecommenderConfig` and are user-overridable; defaults above are starting points, not hard-coded constants.
+
+### Estimated Savings Formula
+
+```
+estimated_savings_pct = min(100, (sum(tokens_saved for all triggered rules) / workflow.total_tokens) * 100)
+```
+
+Each `Recommendation` carries its own `estimated_savings` (per-rule), and the workflow summary reports the aggregate using the formula above.
+
+**Example output:**
+
+```
+Optimization Suggestions
+────────────────────────
+  ✓ Planner repeated system prompt
+  ✓ Retrieved 12 chunks
+    └─ Only 4 appear to be useful
+
+Estimated Savings: 31%
+```
+
+---
+
+## Architecture
+
+The project is modular, with each package having a single responsibility.
+
+```
+src/agenticlens/
+├── profiler/       # Workflow and step profiling logic
+├── metrics/        # Token, cost, and performance collection
+├── providers/      # LLM provider integrations
+├── analyzers/      # Workflow analysis and pattern detection
+├── recommenders/   # Rule-based optimization recommendations
+├── exporters/      # JSON and CSV export
+├── cli/            # Typer-based CLI
+├── config/         # Configuration loading
+├── models/         # Pydantic data models
+└── utils/          # Shared utilities
+```
+
+---
+
+## Core Data Models
+
+All models are implemented with **Pydantic v2**.
+
+### `Workflow`
+
+| Field | Type | Description |
+|---|---|---|
+| `id` | `str` | Unique identifier |
+| `name` | `str` | Workflow name |
+| `start_time` | `datetime` | Execution start |
+| `end_time` | `datetime` | Execution end |
+| `total_tokens` | `int` | Aggregate token count |
+| `total_cost` | `float` | Aggregate cost |
+| `latency` | `float` | Wall-clock duration |
+| `steps` | `list[Step]` | All profiled steps |
+
+---
+
+### `Step`
+
+| Field | Type | Description |
+|---|---|---|
+| `id` | `str` | Unique identifier |
+| `name` | `str` | Step label |
+| `type` | `StepType` | Enum (planner, retriever, tool, etc.) |
+| `provider` | `str` | LLM provider name |
+| `model` | `str` | Model identifier |
+| `metrics` | `Metrics` | Step-level metrics |
+
+---
+
+### `Metrics`
+
+| Field | Type | Description |
+|---|---|---|
+| `prompt_tokens` | `int` | Input token count |
+| `completion_tokens` | `int` | Output token count |
+| `total_tokens` | `int` | Combined count |
+| `latency` | `float` | Duration in seconds |
+| `ttft` | `float \| None` | Time To First Token |
+| `cost` | `float` | Calculated cost |
+
+---
+
+### `Recommendation`
+
+| Field | Type | Description |
+|---|---|---|
+| `title` | `str` | Short recommendation title |
+| `description` | `str` | Detailed explanation |
+| `severity` | `Severity` | Enum: `info`, `warning`, `critical` |
+| `estimated_savings` | `float \| None` | Projected % token reduction |
+
+---
+
+## Provider Architecture
+
+Providers are independent modules behind an **abstract interface**, so additional providers can be added without modifying the profiler.
+
+### Initial providers
+
+| Provider | Status |
+|---|---|
+| OpenAI | ✅ MVP |
+| Anthropic | ✅ MVP |
+
+### Future providers
+
+| Provider | Status |
+|---|---|
+| Gemini | 🔜 Roadmap |
+| Ollama | 🔜 Roadmap |
+| vLLM | 🔜 Roadmap |
+| LiteLLM | 🔜 Roadmap |
+| Azure OpenAI | 🔜 Roadmap |
+
+---
+
+## CLI
+
+Built with **Typer** and **Rich** for terminal output.
+
+```bash
+# Profile a script
+agenticlens profile app.py
+
+# Display a saved report
+agenticlens report report.json
+
+# Analyze a saved workflow
+agenticlens analyze workflow.json
+```
+
+---
+
+## Exporters
+
+| Format | Status |
+|---|---|
+| JSON | ✅ MVP |
+| CSV | ✅ MVP |
+
+---
+
+## Configuration
+
+Configuration is supported through any of:
+
+- `pyproject.toml`
+- YAML file
+- Environment variables
+
+---
+
+## Technology Stack
+
+| Concern | Choice |
+|---|---|
+| Language | Python 3.10+ |
+| Package Manager | `uv` |
+| Packaging | `pyproject.toml` |
+| Testing | `pytest` |
+| Linting | `ruff` |
+| Formatting | `ruff format` |
+| Type Checking | `mypy` |
+| Documentation | MkDocs Material |
+| CLI | Typer |
+| Terminal Output | Rich |
+| Data Models | Pydantic v2 |
+
+---
+
+## Coding Standards
+
+- Full **type hints** throughout
+- **Async-friendly** architecture
+- **Modular design** — single responsibility per module
+- No global mutable state
+- Unit tests for every module
+- Comprehensive docstrings
+- Clean separation between providers, metrics, and analyzers
+
+---
+
+## Repository Structure
+
+```
+agenticlens/
+├── README.md
+├── LICENSE
+├── ROADMAP.md
+├── pyproject.toml
+├── src/
+│   └── agenticlens/
+├── tests/
+├── examples/
+├── docs/
+└── .github/
+    └── workflows/
+```
+
+---
+
+## MVP Deliverables
+
+The initial implementation includes:
+
+- [ ] Repository scaffold
+- [ ] Project configuration (`pyproject.toml`, `ruff`, `mypy`)
+- [ ] Complete package structure
+- [ ] Data models (Pydantic v2)
+- [ ] Provider abstraction (abstract base class)
+- [ ] Profiler skeleton
+- [ ] Metrics engine skeleton
+- [ ] CLI skeleton (Typer)
+- [ ] Unit test setup (pytest)
+- [ ] GitHub Actions CI pipeline
+- [ ] Documentation structure (MkDocs)
+
+> Business logic is implemented incrementally after the scaffold is complete.
+
+---
+
+## Scaffold-First Instructions
+
+> **The first task is NOT to implement the complete library.**
+
+In order:
+
+1. Scaffold the complete repository
+2. Create all directories
+3. Configure tooling (ruff, mypy, pytest)
+4. Configure CI (GitHub Actions)
+5. Configure packaging (pyproject.toml, uv)
+6. Create all base classes and interfaces
+7. Create placeholder implementations where appropriate
+8. Ensure the project installs successfully
+9. Ensure linting, formatting, typing, and tests pass
+10. **Do not implement optimization algorithms until project structure is complete**
+
+> **Objective:** Create a production-quality open-source project foundation that can be incrementally expanded.
+
+---
+
+## Future Roadmap
+
+| Feature | Notes |
+|---|---|
+| LangGraph integration | Native graph-step profiling |
+| CrewAI integration | Multi-agent workflow support |
+| OpenAI Agents SDK integration | Tool call + handoff tracing |
+| MCP profiling | Server-level token attribution |
+| RAG analysis | Chunk utility scoring |
+| Prompt optimization | Automated prompt compression |
+| Context utilization metrics | Effective context window usage |
+| Evaluation framework | Quality vs. cost tradeoffs |
+| Dashboard | Visual workflow explorer |
+| Enterprise reporting | Team-level aggregation and export |
+
+> All items above are **out of scope for MVP**.

agenticlens-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 DeepAgentLabs
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

agenticlens-0.1.0/PKG-INFO

@@ -0,0 +1,86 @@
+Metadata-Version: 2.4
+Name: agenticlens
+Version: 0.1.0
+Summary: Profile, analyze, and optimize token consumption in LLM-powered applications and agentic workflows.
+Project-URL: Homepage, https://github.com/agenticlens/agenticlens
+Project-URL: Issues, https://github.com/agenticlens/agenticlens/issues
+Author: AgenticLens Contributors
+License: MIT License
+        
+        Copyright (c) 2026 DeepAgentLabs
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+License-File: LICENSE
+Keywords: agents,cost,llm,observability,profiling,tokens
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.10
+Requires-Dist: pydantic<3,>=2.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich>=13.0
+Requires-Dist: typer>=0.12
+Provides-Extra: dev
+Requires-Dist: mypy>=1.10; extra == 'dev'
+Requires-Dist: pytest-cov>=5.0; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: ruff>=0.6; extra == 'dev'
+Requires-Dist: types-pyyaml>=6.0; extra == 'dev'
+Provides-Extra: docs
+Requires-Dist: mkdocs-material>=9.5; extra == 'docs'
+Description-Content-Type: text/markdown
+
+# AgenticLens
+
+An open-source profiler for AI agents that analyzes token usage, cost, latency, and optimization opportunities across LLM workflows.
+
+> **Status:** early scaffold. Core data models, provider abstraction, and the explicit `profile()`/`step()` instrumentation API are in place. The recommendation engine's heuristic rules are not yet implemented — see [AgenticLens_Spec.md](AgenticLens_Spec.md).
+
+## Install (development)
+
+```bash
+uv sync --extra dev
+```
+
+## Usage
+
+```python
+from agenticlens import profile, step
+
+with profile("Customer Support"):
+    with step("Planner", type="planner") as s:
+        response = planner_llm.invoke(prompt)
+        s.record(response)
+```
+
+## Development
+
+```bash
+uv run pytest          # tests
+uv run ruff check .    # lint
+uv run ruff format .   # format
+uv run mypy            # type check
+```
+
+See [AgenticLens_Spec.md](AgenticLens_Spec.md) for the full project specification and [ROADMAP.md](ROADMAP.md) for what's planned beyond the MVP.

agenticlens-0.1.0/README.md

@@ -0,0 +1,33 @@
+# AgenticLens
+
+An open-source profiler for AI agents that analyzes token usage, cost, latency, and optimization opportunities across LLM workflows.
+
+> **Status:** early scaffold. Core data models, provider abstraction, and the explicit `profile()`/`step()` instrumentation API are in place. The recommendation engine's heuristic rules are not yet implemented — see [AgenticLens_Spec.md](AgenticLens_Spec.md).
+
+## Install (development)
+
+```bash
+uv sync --extra dev
+```
+
+## Usage
+
+```python
+from agenticlens import profile, step
+
+with profile("Customer Support"):
+    with step("Planner", type="planner") as s:
+        response = planner_llm.invoke(prompt)
+        s.record(response)
+```
+
+## Development
+
+```bash
+uv run pytest          # tests
+uv run ruff check .    # lint
+uv run ruff format .   # format
+uv run mypy            # type check
+```
+
+See [AgenticLens_Spec.md](AgenticLens_Spec.md) for the full project specification and [ROADMAP.md](ROADMAP.md) for what's planned beyond the MVP.

agenticlens-0.1.0/ROADMAP.md

@@ -0,0 +1,30 @@
+# Roadmap
+
+## MVP (current)
+
+- [x] Repository scaffold
+- [x] Project configuration (`pyproject.toml`, `ruff`, `mypy`)
+- [x] Complete package structure
+- [x] Data models (Pydantic v2)
+- [x] Provider abstraction (abstract base class) + OpenAI/Anthropic
+- [x] Profiler skeleton (`profile()` / `step()`)
+- [x] Metrics engine skeleton (cost calculation, pricing resolution)
+- [x] CLI skeleton (Typer)
+- [x] Unit test setup (pytest)
+- [x] GitHub Actions CI pipeline
+- [x] Recommendation engine heuristic rules (repeated system prompt, excessive chunks, long history, duplicate tool calls)
+- [x] CLI `profile`/`report`/`analyze` business logic
+- [ ] Documentation structure (MkDocs)
+
+## Post-MVP
+
+See "Future Roadmap" in [AgenticLens_Spec.md](AgenticLens_Spec.md):
+
+- LangGraph / CrewAI / OpenAI Agents SDK integrations
+- MCP server-level token attribution
+- RAG chunk-utility scoring
+- Automated prompt compression
+- Context utilization metrics
+- Evaluation framework (quality vs. cost)
+- Dashboard / visual workflow explorer
+- Enterprise reporting

agenticlens-0.1.0/docs/index.md

@@ -0,0 +1,14 @@
+# AgenticLens
+
+Profile, analyze, and optimize token consumption in LLM-powered applications and agentic workflows.
+
+See the [project specification](../AgenticLens_Spec.md) for the full design.
+
+```python
+from agenticlens import profile, step
+
+with profile("Customer Support"):
+    with step("Planner", type="planner") as s:
+        response = planner_llm.invoke(prompt)
+        s.record(response)
+```