agentloop-profiler 0.5.0__tar.gz → 0.7.0__tar.gz

agentloop_profiler-0.7.0/PKG-INFO

@@ -0,0 +1,475 @@
+Metadata-Version: 2.4
+Name: agentloop-profiler
+Version: 0.7.0
+Summary: Profiler and optimization layer for AI agent execution graphs
+Author: Dipesh Babu
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/dipeshbabu/agentloop
+Project-URL: Documentation, https://github.com/dipeshbabu/agentloop#readme
+Project-URL: Repository, https://github.com/dipeshbabu/agentloop.git
+Project-URL: Issues, https://github.com/dipeshbabu/agentloop/issues
+Project-URL: Changelog, https://github.com/dipeshbabu/agentloop/blob/main/CHANGELOG.md
+Keywords: ai-agents,agent-evaluation,observability,optimization,profiling,research,tracing
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Debuggers
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+License-File: THIRD_PARTY_LICENSES.md
+Requires-Dist: typer>=0.12.0
+Requires-Dist: rich>=13.0.0
+Provides-Extra: dashboard
+Requires-Dist: streamlit>=1.34.0; extra == "dashboard"
+Requires-Dist: pandas>=2.0.0; extra == "dashboard"
+Provides-Extra: server
+Requires-Dist: fastapi>=0.110.0; extra == "server"
+Requires-Dist: uvicorn>=0.27.0; extra == "server"
+Provides-Extra: postgres
+Requires-Dist: psycopg[binary]>=3.3.4; extra == "postgres"
+Provides-Extra: instrumentation
+Requires-Dist: openai>=1.0.0; extra == "instrumentation"
+Provides-Extra: all
+Requires-Dist: streamlit>=1.34.0; extra == "all"
+Requires-Dist: pandas>=2.0.0; extra == "all"
+Requires-Dist: fastapi>=0.110.0; extra == "all"
+Requires-Dist: uvicorn>=0.27.0; extra == "all"
+Requires-Dist: psycopg[binary]>=3.3.4; extra == "all"
+Requires-Dist: openai>=1.0.0; extra == "all"
+Dynamic: license-file
+
+<h1 align="center">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="docs/assets/agentloop-logo-dark.svg">
+    <img src="docs/assets/agentloop-logo-light.svg" alt="AgentLoop" width="440">
+  </picture>
+</h1>
+
+<p align="center">
+  <strong>Find wasted work in AI agents. Verify each change.</strong><br>
+  Trace calls, inspect findings, and compare performance with your quality checks.
+</p>
+
+<p align="center">
+  <a href="https://github.com/dipeshbabu/agentloop/actions/workflows/ci.yml"><img src="https://github.com/dipeshbabu/agentloop/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
+  <a href="https://pypi.org/project/agentloop-profiler/"><img src="https://img.shields.io/pypi/v/agentloop-profiler.svg?cacheSeconds=300" alt="PyPI"></a>
+  <a href="https://pepy.tech/projects/agentloop-profiler"><img src="https://static.pepy.tech/badge/agentloop-profiler/month" alt="Downloads/month"></a>
+  <a href="https://www.python.org/downloads/"><img src="https://img.shields.io/badge/python-3.10%2B-3776AB.svg" alt="Python 3.10 or newer"></a>
+  <a href="LICENSE"><img src="https://img.shields.io/badge/license-Apache--2.0-blue.svg" alt="License: Apache-2.0"></a>
+</p>
+
+<p align="center">
+  <a href="#quickstart">Quickstart</a> ·
+  <a href="#without-and-with-agentloop">Why AgentLoop</a> ·
+  <a href="#compare-before-and-after">Compare runs</a> ·
+  <a href="#framework-integrations">Integrations</a> ·
+  <a href="docs/FIRST_USE.md">Guide</a>
+</p>
+
+AgentLoop records model calls, tool calls, and retries in an agent workflow. Use
+its reports to inspect latency, token usage, and estimated model costs, find
+repeated work, and compare runs before and after a change.
+
+Tracing and analysis run locally. You can work with JSON files without a database
+or hosted account. The dashboard and HTTP API are optional.
+
+## Without and with AgentLoop
+
+<picture>
+  <source media="(max-width: 600px)" srcset="docs/assets/workflow-comparison-mobile.svg">
+  <source media="(prefers-reduced-motion: reduce)" srcset="docs/assets/workflow-comparison.svg">
+  <img src="docs/assets/workflow-comparison.gif" width="100%" alt="Architecture comparison: your agent feeds custom timing, usage parsing, retry analysis, output checks, and report code without AgentLoop. With AgentLoop, the same execution evidence feeds a shared tracing, analysis, and replay workflow.">
+</picture>
+
+<details>
+<summary>Read the comparison in text</summary>
+
+| Question | Without AgentLoop | With AgentLoop |
+|---|---|---|
+| Where did the time go? | Add timers and connect logs to calls. | Inspect recorded model, tool, and retry spans. |
+| What should I investigate? | Look through calls and logs for repeated work. | Follow findings to affected spans and suggested changes. |
+| Did the change help? | Write your own before/after comparison. | Compare saved runs with `agentloop replay`. |
+| Is the output still acceptable? | Combine task-specific checks with the performance comparison. | Supply quality fixtures and evaluate them alongside performance gates. |
+
+</details>
+
+AgentLoop supplies the tracing and comparison workflow. You choose the change,
+define what a correct output looks like, and decide which checks must pass.
+
+## How AgentLoop fits your stack
+
+<picture>
+  <source media="(max-width: 600px)" srcset="docs/assets/framework-comparison-mobile.svg">
+  <source media="(prefers-reduced-motion: reduce)" srcset="docs/assets/framework-comparison.svg">
+  <img src="docs/assets/framework-comparison.gif" width="100%" alt="Architecture: your application runs the agent and calls models and tools. AgentLoop decorators and adapters record spans into trace JSON. Analyze produces metrics and findings; replay compares a candidate with its baseline and your quality fixtures. Both produce reports for review or CI.">
+</picture>
+
+Your application runs the agent and chooses its framework and models.
+AgentLoop's [decorators and adapters](#framework-integrations) record the
+instrumented calls. Exported traces feed analysis, baseline/candidate replay,
+and reports for review or CI.
+
+When you change the framework, model, prompt, or execution strategy, record the
+same task again. [Compare both runs](#compare-before-and-after) with the same
+[quality fixtures](#quality-gates) to check the effect of your change.
+
+## Quickstart
+
+Requires Python 3.10 or newer:
+
+```bash
+python -m pip install agentloop-profiler
+agentloop quickstart
+```
+
+The package name is `agentloop-profiler`; the Python import and CLI command are
+both `agentloop`.
+
+`quickstart` creates `runs/agentloop_quickstart.json` and prints findings from
+synthetic data. Use it to check the installation and learn the report format.
+The demo itself makes no network or model API calls.
+
+Inspect the saved trace and export the complete analysis:
+
+```bash
+agentloop analyze runs/agentloop_quickstart.json --json-out runs/quickstart_analysis.json
+```
+
+The analysis JSON contains the trace, its metrics, findings, and optimization
+suggestions. See [First useful result](docs/FIRST_USE.md) for more options.
+
+Add `--html runs/report.html` to `analyze` for a single offline report with a
+timeline and linked finding evidence. See [HTML reports](docs/HTML_REPORTS.md).
+
+## Record your own run
+
+Save this complete example as `example_agent.py`. It traces a few local function
+calls so you can try instrumentation without a model SDK or API key. Replace the
+function body with your own work when adapting it to an agent.
+
+```python
+import agentloop
+
+NOTES = {
+    "tracing": "Record the steps in a workflow.",
+    "optimization": "Investigate repeated or expensive work.",
+    "replay": "Compare saved baseline and candidate traces.",
+}
+
+
+@agentloop.trace_tool(name="lookup_note")
+def lookup_note(topic: str) -> str:
+    return NOTES[topic]
+
+
+with agentloop.trace_agent("research_agent") as trace:
+    output = {"notes": [lookup_note(topic) for topic in NOTES]}
+    trace.metadata["output"] = output
+
+trace.export_json("runs/research_agent.json")
+print(output)
+```
+
+Run it, then analyze the file it created:
+
+```bash
+python example_agent.py
+agentloop analyze runs/research_agent.json --json-out runs/research_agent_analysis.json
+```
+
+The decorator records each function call's duration and status. The example also
+explicitly saves the final output in trace metadata so the quality checks below
+can read it. Decide which outputs are appropriate to retain when adapting this
+example to real data.
+
+For model calls, use an [SDK integration](#framework-integrations) to capture
+provider token usage. The [integration guide](docs/INTEGRATIONS.md) also covers
+model decorators, context managers, streaming, and cancellation.
+
+## Inspect findings
+
+Start with `analyze`. It combines metrics, findings, and proposed optimizations
+in one report. AgentLoop looks for patterns such as repeated context, retry loops,
+repeated model calls, large steps, and recurring tool calls.
+
+On `main`, reports and optimization use the same [versioned finding rules](docs/FINDING_RULES.md).
+If a rule fails, its diagnostics remain visible and analysis is marked incomplete.
+CI requires complete analysis before recommending a merge.
+
+Treat a suggestion as a change to investigate. For example, repeated tool names
+alone do not establish that calls are safe to run concurrently. The development
+version on `main` exposes the evidence and assumptions behind
+[parallelization candidates](docs/PARALLELIZATION.md).
+
+Use individual commands when you need a specific artifact:
+
+| Command | Output |
+|---|---|
+| `agentloop report runs/research_agent.json` | Metrics for one run. |
+| `agentloop diagnose --path runs/research_agent.json` | Findings with affected spans and validation criteria. |
+| `agentloop optimize --path runs/research_agent.json` | Suggested changes and estimated savings. |
+| `agentloop patch --path runs/research_agent.json --repo .` | Proposed edits for the repository at `.`, ready for you to review and apply. |
+
+Change one thing at a time, such as reducing repeated context or changing a retry
+policy, so you can attribute the result to that change.
+
+## Compare before and after
+
+Run your agent before the change and export `runs/baseline.json`. Run the changed
+agent on the same task and export `runs/candidate.json`. `replay` compares these
+saved traces:
+
+```bash
+agentloop replay --baseline runs/baseline.json --candidate runs/candidate.json --min-latency-improvement-pct 10
+```
+
+This requires at least a 10% latency improvement. The report also compares costs,
+tokens, retries, and call counts. It writes `runs/replay_report.md` and exits
+nonzero when a required gate fails. Use `agentloop replay --help` to choose other
+thresholds or a JSON output path.
+
+A result applies to the runs and checks you supplied. Repeat the comparison across
+representative tasks before drawing broader conclusions. Add output checks when
+you need evidence that the changed agent still meets its task requirements.
+
+## Quality gates
+
+Quality fixtures define checks against the recorded output. For the example
+above, save this as `fixtures.json`:
+
+```json
+{
+  "fixtures": [
+    {
+      "id": "notes_present",
+      "scorer": {"type": "required_fields", "required": ["notes"]}
+    }
+  ]
+}
+```
+
+Run the check against the example trace:
+
+```bash
+agentloop quality-report fixtures.json --candidate runs/research_agent.json
+```
+
+This checks that the final output has a nonempty `notes` field. Choose expectations
+that express correctness for your own task. Fixtures can provide outputs directly
+or use a trace's `metadata.output`; the scorer can also use the last model call's
+recorded output text when available.
+
+| Scorer | Check |
+|---|---|
+| `exact_match` | Output exactly matches an expected value, including its type. |
+| `contains` | Output contains specified text. |
+| `glob` | Output matches a bounded wildcard pattern. |
+| `required_fields` | A JSON object contains the specified nonempty fields. |
+| `json_subset` | A JSON object contains specified top-level keys with matching values. |
+| `custom` | A trusted local Python function, configured as `module:function`, scores the output. |
+
+Every candidate case must pass. If you set `--min-quality-score` in `replay`/`ci`
+(or `--min-score` in `quality-report`), the average score must also meet that threshold.
+Invalid suites fail closed.
+
+Custom scorers run Python code and should be used only with trusted local fixtures.
+The HTTP endpoint rejects them. See the [changelog](CHANGELOG.md) for `regex` and
+`json_schema` migration notes.
+
+Once you have baseline and candidate traces, use the same fixtures in CI:
+
+```bash
+agentloop ci --baseline runs/baseline.json --candidate runs/candidate.json --quality-fixtures fixtures.json
+```
+
+`ci` writes a Markdown report suitable for a pull request and exits nonzero when
+a required gate fails. It evaluates the traces your application supplies.
+
+The repository's required replay check uses synthetic demos to test AgentLoop's
+CI machinery. For comparisons using your application's artifacts, see
+[CI self-tests and application comparisons](docs/CI.md).
+
+## Understand the results
+
+These notes describe the current `main` branch. Check [Unreleased changes](CHANGELOG.md#unreleased)
+for behavior that has not yet shipped in the published package.
+
+| Result | How to interpret it |
+|---|---|
+| Runtime and call counts | Recorded execution data. `total_runtime_ms` is elapsed time; `cumulative_span_time_ms` sums spans and can be larger when calls overlap or nest. |
+| Operation kinds | Agent, workflow, model, tool, retrieval, and other roles recorded alongside legacy event categories. See [Operation kinds](docs/OPERATIONS.md). |
+| Token counts | Counts identify their source: a provider, a tokenizer, user code, or a word-count estimate. Missing usage is marked unavailable. |
+| Model cost | A calculation from token counts and configured pricing. Unknown prices or estimated usage limit the comparisons you can make. |
+| Optimization savings | Uncalibrated predictions with versioned formulas, coefficients, and assumptions. Measure a candidate run to establish the actual effect. |
+| Quality scores | Results of the checks you supplied. Their coverage depends on your fixtures. |
+
+Cost gates can be marked `indeterminate` when pricing or usable token counts are
+missing. An indeterminate cost comparison does not fail the default gate; requiring
+a positive `--min-cost-improvement-pct` makes unverifiable cost improvement fail.
+See [Pricing](docs/PRICING.md) and [Trace compatibility](docs/TRACE_SCHEMA.md) for
+the details, including how older traces are handled.
+
+The optimizer avoids double-counting suggestions that affect the same spans.
+Reports say whether the chosen combination is exact or approximate. See
+[Savings selection](docs/SAVINGS_SELECTION.md).
+For individual predictions, see [Estimator provenance](docs/ESTIMATORS.md).
+The [intervention ledger](docs/INTERVENTIONS.md) links finding predictions to
+measured candidate results and preserves quality checks in an exportable record.
+For experiments across tasks and repetitions, use [paired study summaries](docs/STUDIES.md).
+
+Keep your Markdown renderer's safe mode or HTML sanitizer enabled when viewing
+reports from untrusted traces.
+
+## Framework integrations
+
+For existing telemetry, [OTLP conformance](docs/TELEMETRY_CONFORMANCE.md) covers
+GenAI, OpenInference, and MCP JSON, including operation kinds and evidence metadata.
+
+| Integration | How it connects |
+|---|---|
+| Custom Python agents | Decorators and context managers around your functions. |
+| OpenAI SDK | Wrap a client or callable to capture calls, streams, and reported usage. |
+| OpenAI Agents SDK | Attach an AgentLoop tracing processor. |
+| LangGraph | Instrument the builder before adding nodes, then wrap the compiled runnable. |
+| CrewAI | Wrap crew, task, or agent execution methods. |
+| OpenTelemetry / Vercel AI SDK telemetry | Import supported telemetry into AgentLoop traces. |
+
+The adapters are included in AgentLoop. Install the framework's own SDK separately
+when needed; `python -m pip install "agentloop-profiler[instrumentation]"` includes
+the OpenAI Python SDK. See [Framework integrations](docs/INTEGRATIONS.md) for
+setup and supported behavior.
+
+## Dashboard and persistence
+
+The dashboard reads stored traces. From a [source checkout](#source-checkout),
+load the synthetic example and start Streamlit:
+
+```bash
+uv run --frozen --all-extras agentloop quickstart
+uv run --frozen --all-extras agentloop store-trace --path runs/agentloop_quickstart.json --project-id default
+uv run --frozen --all-extras streamlit run dashboard/app.py
+```
+
+Open the URL printed by Streamlit and leave the sidebar's Project field set to
+`default`. To inspect your own data, store your trace file with `store-trace`.
+
+AgentLoop uses SQLite by default and supports Postgres for shared storage. The
+optional HTTP API accepts traces and serves project-scoped reports and findings.
+Application endpoints use `/v1`. Existing unversioned aliases remain available
+throughout 0.x; see [API versioning](docs/API_VERSIONING.md) for the migration path.
+See [Dashboard setup](docs/DASHBOARD.md) and [Production deployment](docs/PRODUCTION.md)
+for database configuration, API authentication, and deployment instructions.
+
+## Other installation options
+
+### Source checkout
+
+Use a source checkout for the dashboard or the latest unreleased changes.
+Install [uv](https://docs.astral.sh/uv/getting-started/installation/), then run:
+
+```bash
+git clone https://github.com/dipeshbabu/agentloop.git
+cd agentloop
+uv sync --locked --all-extras
+uv run --frozen --all-extras agentloop quickstart
+```
+
+### Standalone CLI
+
+The [GitHub Releases page](https://github.com/dipeshbabu/agentloop/releases) has
+executables for Linux x86-64, Windows x86-64, macOS Intel, and macOS Apple silicon.
+They bundle Python and the core CLI. Use the Python package for SDK instrumentation,
+the dashboard, API server, or Postgres support.
+
+Use the downloaded executable wherever the examples show `agentloop`.
+
+<details>
+<summary>Download and verify a standalone executable</summary>
+
+Download the matching file and `SHA256SUMS` from the same release. Replace `X.Y.Z`
+with its version number.
+
+Linux (Bash):
+
+```bash
+set -euo pipefail
+version=X.Y.Z
+asset="agentloop-v${version}-linux-x86_64"
+curl -LO "https://github.com/dipeshbabu/agentloop/releases/download/v${version}/${asset}"
+curl -LO "https://github.com/dipeshbabu/agentloop/releases/download/v${version}/SHA256SUMS"
+mapfile -t checksum_lines < <(awk -v asset="$asset" '$2 == asset { print }' SHA256SUMS)
+if [ "${#checksum_lines[@]}" -ne 1 ]; then
+  echo "Expected exactly one checksum for ${asset}" >&2
+  exit 1
+fi
+printf '%s\n' "${checksum_lines[0]}" | sha256sum --check --strict
+chmod +x "$asset"
+./"$asset" --help
+```
+
+Windows PowerShell:
+
+```powershell
+$Version = "X.Y.Z"
+$Asset = "agentloop-v$Version-windows-x86_64.exe"
+Invoke-WebRequest "https://github.com/dipeshbabu/agentloop/releases/download/v$Version/$Asset" -OutFile agentloop.exe
+Invoke-WebRequest "https://github.com/dipeshbabu/agentloop/releases/download/v$Version/SHA256SUMS" -OutFile SHA256SUMS
+$ChecksumLines = @(Get-Content SHA256SUMS | Where-Object {
+    $_ -match "^[0-9a-fA-F]{64}  $([regex]::Escape($Asset))$"
+})
+if ($ChecksumLines.Count -ne 1) { throw "Expected exactly one checksum for $Asset" }
+$ExpectedHash = ($ChecksumLines[0] -split "  ", 2)[0]
+$ActualHash = (Get-FileHash .\agentloop.exe -Algorithm SHA256).Hash
+if ($ActualHash -ne $ExpectedHash) { throw "Checksum verification failed for $Asset" }
+.\agentloop.exe --help
+```
+
+Each release includes platform-specific third-party notices. macOS executables
+are ad-hoc signed but not notarized; Gatekeeper may require approval on first launch.
+
+</details>
+
+## Research use
+
+For experiments, record one trace per task attempt and condition. Attach metadata
+such as task ID, seed, model, and source commit, and repeat comparisons across your
+task set. The [research guide](docs/RESEARCH.md) explains this workflow. From a
+source checkout, run the offline paired example with:
+
+```bash
+uv run --frozen --all-extras python examples/research_experiment_demo.py
+```
+
+## Help and contributions
+
+Run `agentloop --help` for all commands, or `agentloop COMMAND --help` for a specific
+command. Use `agentloop doctor` to inspect your installation and available integrations.
+
+- [Support](SUPPORT.md): usage help and troubleshooting.
+- [Issue forms](https://github.com/dipeshbabu/agentloop/issues/new/choose): bugs and feature requests.
+- [Contributing](CONTRIBUTING.md): development setup and required checks.
+- [Roadmap](docs/ROADMAP.md) and [changelog](CHANGELOG.md): project direction and release changes.
+- [Security](SECURITY.md): private vulnerability reports.
+- [Governance](GOVERNANCE.md) and [Code of Conduct](CODE_OF_CONDUCT.md): project decisions and participation.
+
+AgentLoop is under active pre-1.0 development. Use a tagged release when you need
+a fixed version, and check the changelog before upgrading. Maintainers can use the
+[launch checklist](docs/OPEN_SOURCE_CHECKLIST.md) when preparing a public release.
+
+## License
+
+Copyright 2026 Dipesh Tharu Mahato.
+
+AgentLoop is licensed under the [Apache License 2.0](LICENSE). Dependencies retain
+their own terms; see [THIRD_PARTY_LICENSES.md](THIRD_PARTY_LICENSES.md).