agent-learning-kit 0.1.0__tar.gz → 0.2.0__tar.gz

agent_learning_kit-0.2.0/CHANGELOG.md

@@ -0,0 +1,57 @@
+# Changelog
+
+All notable release changes for Agent Learning Kit are tracked here.
+
+## Unreleased
+
+Post-rc.1 increments on the release branch. Each one adds gates on top of the
+66 proved at rc.1; the rc.1 entry below is historical and unchanged.
+
+- Docs corpus: the full `docs/` tree (quickstarts, per-track guides, framework
+  pages, reference material, and the `docs/llms.txt` machine index) with
+  machine-checkable page metadata, enforced by the new `docs_executability`
+  release gate (67 gates).
+- Live lanes: opt-in live execution lanes (LiveKit, Pipecat, LangGraph, MCP,
+  A2A) behind per-framework extras, with the engine/public boundary enforced
+  by the new `live_lane_boundary` release gate (68 gates).
+- Optimizer expansion: optimizer portfolio routing, frozen capability
+  profiles, and apply plans, enforced by the two new Phase-4 gates
+  `optimizer_profile_matrix_readiness` and `capability_profile_freeze_readiness`
+  (70 gates).
+- Hosted Vapi and Retell calls now carry the target agent's provider call ID,
+  native end reason, tokens, cost, and latency on result receipts instead of
+  dropping that evidence; simulator usage remains separate.
+- Hosted voice workers preserve stdout/stderr in the parent diagnostics stream,
+  with per-call correlation, turn-state transitions, transcription dispatch,
+  and simulator generation lifecycle events.
+
+## v1.0.0-rc.1 — 2026-06-10
+
+First locally-cut v1 release candidate. Package labels: Python
+`agent-learning-kit==0.1.0`, TypeScript `@future-agi/agent-learning-kit==0.2.0`
+(decision records D1/D2; the tag, not the semver, names the product milestone).
+
+- One SDK and CLI (`fi.alk` / `agent-learn`) consolidating the
+  `simulate`, `evals`, and `opt` engines — three engines, four workflows
+  (test, simulate, red-team, optimize).
+- 66 executable release gates behind `agent-learn release-check`, proved by
+  `agent-learn release-proof` (`agent-learning.release-proof.v1`) on the cut
+  commit.
+- Distribution hygiene: the sdist now ships only `src/`, `tests/`, `examples/`,
+  `docs/`, and the standard release files — `internal-docs/`, `uv.lock`, the
+  roadmap, internal guides, the `typescript/` workspace, and build artifacts no
+  longer leak; enforced by the new `package_distribution_hygiene` gate.
+- `Development Status` classifier moved to `4 - Beta` (D3); `uv.lock` tracked
+  in git and excluded from the sdist (D4).
+- README claims reconciled with executable proof (LlamaIndex listed,
+  `AGENT_LEARNING_API_KEY` named, OpenEnv positioning deduplicated with the
+  robustness bar defined, install framing honest pre-publish, probe-promoted
+  vs runtime-simulated coverage distinguished).
+- Prepared v1 release-candidate documentation, Apache-2.0 licensing artifacts,
+  community files, and release-proof handoff notes.
+- Added a developer-first README opening with install, quickstart, workflow,
+  release proof, repository map, and community links.
+- Hardened `agent-learn release-proof` timeout handling and raised the default
+  per-command timeout to 7200s for the expanded v1 proof suite (the full pytest
+  suite, which executes every release gate inside the milestone test, exceeds
+  the previous 2400s budget).

agent_learning_kit-0.2.0/PKG-INFO

@@ -0,0 +1,382 @@
+Metadata-Version: 2.5
+Name: agent-learning-kit
+Version: 0.2.0
+Summary: Unified Future AGI SDK for agent learning workflows.
+Project-URL: Homepage, https://futureagi.com
+Project-URL: Documentation, https://docs.futureagi.com
+Project-URL: Repository, https://github.com/future-agi/agent-learning-kit
+Project-URL: Issues, https://github.com/future-agi/agent-learning-kit/issues
+Project-URL: Changelog, https://github.com/future-agi/agent-learning-kit/blob/main/CHANGELOG.md
+Author-email: Future AGI <hello@futureagi.io>
+License: Apache-2.0
+License-File: LICENSE
+License-File: NOTICE
+Keywords: agent-evaluation,agent-optimization,agent-simulation,agent-testing,ai-agents,future-agi,red-teaming
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.10
+Requires-Dist: claude-agent-sdk<0.3,>=0.2.140
+Requires-Dist: fi-instrumentation-otel>=0.1.16
+Requires-Dist: gepa>=0.0.17
+Requires-Dist: google-adk<2.8,>=2.7.1
+Requires-Dist: httpx>=0.24.0
+Requires-Dist: jsonschema<5,>=4.25.1
+Requires-Dist: levenshtein>=0.25.0
+Requires-Dist: litellm<2,>=1.80.0
+Requires-Dist: nltk>=3.9.0
+Requires-Dist: numpy>=1.26.4
+Requires-Dist: openai<3,>=1.109.1
+Requires-Dist: opentelemetry-api<2,>=1.39.1
+Requires-Dist: opentelemetry-exporter-otlp<2,>=1.39.1
+Requires-Dist: opentelemetry-sdk<2,>=1.39.1
+Requires-Dist: optuna>=3.6.1
+Requires-Dist: pandas>=2.0.0
+Requires-Dist: pydantic<3,>=2.0
+Requires-Dist: python-dotenv>=1.0.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: requests-futures>=1.0.0
+Requires-Dist: requests<3,>=2.32.5
+Requires-Dist: retell-sdk<6,>=5.64
+Requires-Dist: rich>=13.0.0
+Requires-Dist: rouge-score>=0.1.2
+Requires-Dist: sqlalchemy[asyncio]<3,>=2.0
+Requires-Dist: typer<1.0.0,>=0.9.0
+Provides-Extra: a2a
+Requires-Dist: a2a-sdk[http-server]>=1.1.0; extra == 'a2a'
+Provides-Extra: all
+Requires-Dist: aiohttp>=3.10; extra == 'all'
+Requires-Dist: audioop-lts>=0.2.1; (python_version >= '3.13') and extra == 'all'
+Requires-Dist: chromadb>=0.4.0; extra == 'all'
+Requires-Dist: livekit-agents[cartesia,deepgram,google,openai,silero]<2,>=1.8.3; extra == 'all'
+Requires-Dist: livekit-plugins-elevenlabs<2,>=1.8.3; extra == 'all'
+Requires-Dist: sentence-transformers<6,>=5.2.3; extra == 'all'
+Requires-Dist: torch<3,>=2.10.0; extra == 'all'
+Requires-Dist: transformers<6,>=5.2.0; extra == 'all'
+Provides-Extra: embeddings
+Requires-Dist: sentence-transformers<6,>=5.2.3; extra == 'embeddings'
+Provides-Extra: evaluation
+Provides-Extra: feedback
+Requires-Dist: chromadb>=0.4.0; extra == 'feedback'
+Provides-Extra: harness-stores
+Requires-Dist: psycopg[binary]>=3.2; extra == 'harness-stores'
+Provides-Extra: harness-ui
+Requires-Dist: fastapi<1,>=0.115; extra == 'harness-ui'
+Requires-Dist: uvicorn<1,>=0.30; extra == 'harness-ui'
+Provides-Extra: langchain
+Requires-Dist: langchain-core<2,>=1.4.6; extra == 'langchain'
+Requires-Dist: langgraph-checkpoint-sqlite>=3.1.0; extra == 'langchain'
+Requires-Dist: langgraph<2,>=1.2.4; extra == 'langchain'
+Provides-Extra: livekit
+Requires-Dist: aiohttp>=3.10; extra == 'livekit'
+Requires-Dist: audioop-lts>=0.2.1; (python_version >= '3.13') and extra == 'livekit'
+Requires-Dist: livekit-agents[cartesia,deepgram,google,openai,silero]<2,>=1.8.3; extra == 'livekit'
+Requires-Dist: livekit-plugins-elevenlabs<2,>=1.8.3; extra == 'livekit'
+Provides-Extra: mcp
+Requires-Dist: mcp<2,>=1.27; extra == 'mcp'
+Provides-Extra: nli
+Requires-Dist: torch<3,>=2.10.0; extra == 'nli'
+Requires-Dist: transformers<6,>=5.2.0; extra == 'nli'
+Provides-Extra: notebook
+Requires-Dist: ipykernel>=6; extra == 'notebook'
+Requires-Dist: nbformat>=5; extra == 'notebook'
+Provides-Extra: optimize
+Provides-Extra: pipecat
+Requires-Dist: pipecat-ai>=0.0.108; extra == 'pipecat'
+Provides-Extra: simulate
+Provides-Extra: trinity
+Requires-Dist: aiohttp>=3.10; extra == 'trinity'
+Requires-Dist: audioop-lts>=0.2.1; (python_version >= '3.13') and extra == 'trinity'
+Requires-Dist: livekit-agents[cartesia,deepgram,google,openai,silero]<2,>=1.8.3; extra == 'trinity'
+Requires-Dist: livekit-plugins-elevenlabs<2,>=1.8.3; extra == 'trinity'
+Description-Content-Type: text/markdown
+
+<p align="center">
+  <img src="https://raw.githubusercontent.com/future-agi/agent-learning-kit/main/docs/assets/futureagi-mark-email.png" alt="Future AGI" width="72" />
+</p>
+
+<h1 align="center">Agent Learning Kit</h1>
+
+<p align="center">
+  Local-first testing, simulation, red teaming, and optimization for AI agents.
+</p>
+
+<p align="center">
+  <a href="https://github.com/future-agi/agent-learning-kit/blob/main/LICENSE">Apache-2.0</a>
+  ·
+  <a href="https://github.com/future-agi/agent-learning-kit/blob/main/docs/index.md">Docs</a>
+  ·
+  <a href="https://github.com/future-agi/agent-learning-kit/blob/main/CONTRIBUTING.md">Contributing</a>
+  ·
+  <a href="https://github.com/future-agi/agent-learning-kit/blob/main/SECURITY.md">Security</a>
+  ·
+  <a href="https://github.com/future-agi/agent-learning-kit/blob/main/ROADMAP.md">V1 roadmap</a>
+  ·
+  <a href="https://github.com/future-agi/agent-learning-kit/blob/main/LIBRARIES.md">Library inventory</a>
+</p>
+
+![Agent Learning lifecycle blueprint](https://raw.githubusercontent.com/future-agi/agent-learning-kit/main/docs/assets/hero-agent-blueprint.jpg)
+
+Agent Learning Kit is the local-first SDK and CLI for testing, simulating,
+red-teaming, and optimizing AI agents.
+
+It brings the three core Future AGI engines into one public developer surface —
+three engines, four workflows: red-teaming rides on the `simulate` and `evals`
+engines rather than being a fourth engine:
+
+- `simulate`: run local worlds, tasks, framework-shaped adapters, replays, and
+  regression artifacts.
+- `evals`: evaluate prompts, task outputs, runtime contracts, traces, memory,
+  retrieval, safety, and robustness evidence.
+- `optimize`: search over prompts, agents, framework adapters, worlds,
+  multi-agent interactions, memory layers, workflows, and red-team scenarios.
+
+Use it when you want one reproducible loop:
+
+1. Simulate an agent or framework workflow.
+2. Evaluate the behavior and runtime evidence.
+3. Optimize the weak layer.
+4. Promote the result into a replayable artifact.
+5. Prove release readiness with local gates.
+
+### The harness: point it at an agent and talk to it
+
+`src/fi/alk/harness/` builds all of the above **for** an agent instead of asking you to write it.
+Point it at an agent's source and it reads what that agent verifiably is, builds a real world its
+tools act on, and writes test scenarios that are each proved before they are kept. It is driven
+as a conversation, in a terminal or on a web page.
+
+- **[Start here](https://github.com/future-agi/agent-learning-kit/blob/main/src/fi/alk/harness/README.md)**: setup from nothing, then how to use it
+- **[The web page](https://github.com/future-agi/agent-learning-kit/blob/main/harness-ui/README.md)**: the same harness as a chat, on `localhost:8777`
+- **[How it works](https://github.com/future-agi/agent-learning-kit/blob/main/src/fi/alk/harness/HOW-IT-WORKS.md)** and
+  **[why it is shaped this way](https://github.com/future-agi/agent-learning-kit/blob/main/src/fi/alk/harness/DESIGN.md)**
+
+OpenEnv/Gymnasium shapes are compatibility inputs, not the product center.
+Agent Learning Kit is the primary runtime and release contract, and the bar is
+the executable `environment_10x_robustness` release gate.
+OpenEnv/Gymnasium-shaped traces remain compatibility evidence inside that bar.
+
+## Install
+
+Install from PyPI:
+
+```bash
+pip install agent-learning-kit
+```
+
+To develop against source (contributors):
+
+```bash
+git clone https://github.com/future-agi/agent-learning-kit
+cd agent-learning-kit
+uv sync          # or: pip install -e .
+```
+
+(npm publishing of the TypeScript SDK lands at the v1 launch.)
+
+Optional Python extras:
+
+```bash
+pip install "agent-learning-kit[livekit]"
+pip install "agent-learning-kit[nli]"
+pip install "agent-learning-kit[all]"
+```
+
+TypeScript evaluation package (npm at launch; today build from
+[`typescript/agent-learning-kit`](https://github.com/future-agi/agent-learning-kit/blob/main/typescript/agent-learning-kit)):
+
+```bash
+pnpm add @future-agi/agent-learning-kit
+```
+
+## Quickstart
+
+Everything below runs fully offline — no API key, no network. Start with the
+local doctor:
+
+```bash
+agent-learn doctor
+```
+
+Then run the golden path against the bundled example manifests. The
+`AGENT_LEARNING_*_EXAMPLE_KEY` prefixes satisfy each manifest's
+`required_env` list — that list is CI wiring metadata, not a provider
+credential, so any placeholder value works.
+
+> Prefer the SDK spine over the CLI?
+> [Spec + Runner](https://github.com/future-agi/agent-learning-kit/blob/main/docs/simulate/spec-and-runner.md) runs the same simulation as
+> one `SimulationSpec` fed to one `SimulationRunner` — the plug-and-play surface
+> behind every simulation.
+
+Evaluate a suite:
+
+```bash
+agent-learn eval examples/eval_suite.json \
+  --output artifacts/eval.json
+```
+
+Simulate a run manifest:
+
+```bash
+AGENT_LEARNING_RUN_EXAMPLE_KEY=offline-demo-key \
+  agent-learn run examples/run_manifest.json \
+  --no-eval \
+  --output artifacts/run.json
+```
+
+Optimize an agent workflow:
+
+```bash
+AGENT_LEARNING_OPTIMIZE_EXAMPLE_KEY=offline-demo-key \
+  agent-learn optimize examples/optimization_manifest.json \
+  --output artifacts/optimization.json
+```
+
+Run a red-team campaign:
+
+```bash
+AGENT_LEARNING_REDTEAM_EXAMPLE_KEY=offline-demo-key \
+  agent-learn redteam examples/redteam_manifest.json \
+  --output artifacts/redteam.json
+```
+
+Each command prints a `wrote <path>` line; relative `--output` paths resolve
+against your current working directory.
+
+Optional platform mode: to use Future AGI platform-backed evaluation, set
+`AGENT_LEARNING_API_KEY` (it takes precedence over the `FUTURE_AGI_API_KEY`
+and `FI_API_KEY` aliases), or call `configure(api_key="...")` from
+`fi.alk`. See
+[docs/reference/configure.md](https://github.com/future-agi/agent-learning-kit/blob/main/docs/reference/configure.md).
+
+Cut local release proof:
+
+```bash
+agent-learn release-check --project-root .
+agent-learn release-proof \
+  --project-root . \
+  --output /tmp/agent-learning-release-proof.json \
+  --quiet
+```
+
+## TypeScript
+
+```typescript
+import { Evaluator } from "@future-agi/agent-learning-kit";
+import { LocalEvaluator } from "@future-agi/agent-learning-kit/evals/local";
+```
+
+## What You Can Build
+
+- Prompt and response evaluations.
+- Local task and world simulations.
+- Framework adapter probes (probe-promoted coverage) for LangChain, LangGraph,
+  LlamaIndex, AutoGen, CrewAI, LiveKit, Pipecat, Browser Use, MCP, A2A, and
+  custom orchestration objects.
+- Runtime-simulated coverage for PydanticAI (multi-framework runtime
+  simulation) and OpenAI Agents (handoff-transcript promotion).
+- Runtime-contract and trace-quality checks.
+- Multi-agent coordination and handoff tests.
+- Retrieval and memory quality checks.
+- Voice, realtime, browser/CUA, workflow, lifecycle, and protocol traces.
+- Red-team corpus, campaign, adaptive-loop, and persistent-state checks.
+- Optimizer governance, candidate lineage, rollback, and release proof.
+
+## Why It Exists
+
+Most agent stacks split testing, simulation, optimization, and safety review
+across separate tools. Agent Learning Kit keeps those steps in one artifact
+model so a developer can inspect what happened, score it, improve it, and replay
+it in CI.
+
+The public SDK is `agent-learning-kit`, the Python namespace is
+`fi.alk`, the CLI is `agent-learn`, and the TypeScript package is
+`@future-agi/agent-learning-kit`.
+
+The active `ai-evaluation` code is included here under `src/fi/evals`, with its
+TypeScript SDK source under `typescript/agent-learning-kit/src`. The
+`simulate-sdk` and `agent-opt` engine code is included under `src/fi/simulate`
+and `src/fi/opt`. See [LIBRARIES.md](https://github.com/future-agi/agent-learning-kit/blob/main/LIBRARIES.md) for the complete source map.
+The ai-evaluation source inventory used by `agent-learn release-check` lives at
+the ai-evaluation source inventory (maintained in the internal-docs repo).
+
+## Repository Map
+
+- [`examples/`](https://github.com/future-agi/agent-learning-kit/blob/main/examples): runnable cookbooks and manifests.
+- [`src/fi/alk`](https://github.com/future-agi/agent-learning-kit/blob/main/src/fi/alk): public Python SDK facade and CLI.
+- [`src/fi/evals`](https://github.com/future-agi/agent-learning-kit/blob/main/src/fi/evals): active `ai-evaluation` engine code.
+- [`src/fi/simulate`](https://github.com/future-agi/agent-learning-kit/blob/main/src/fi/simulate): migrated `simulate-sdk` engine code.
+- [`src/fi/opt`](https://github.com/future-agi/agent-learning-kit/blob/main/src/fi/opt): migrated `agent-opt` engine code.
+- [`typescript/agent-learning-kit`](https://github.com/future-agi/agent-learning-kit/blob/main/typescript/agent-learning-kit): public
+  TypeScript package, including the active evaluation SDK source.
+- [`docs/index.md`](https://github.com/future-agi/agent-learning-kit/blob/main/docs/index.md): full documentation index.
+- [`ROADMAP.md`](https://github.com/future-agi/agent-learning-kit/blob/main/ROADMAP.md): public v1 roadmap and post-v1 extensions.
+- [`LIBRARIES.md`](https://github.com/future-agi/agent-learning-kit/blob/main/LIBRARIES.md): source map for the consolidated engines.
+- [`CONTRIBUTING.md`](https://github.com/future-agi/agent-learning-kit/blob/main/CONTRIBUTING.md): local development and PR workflow.
+- [`SECURITY.md`](https://github.com/future-agi/agent-learning-kit/blob/main/SECURITY.md): vulnerability reporting policy.
+- [`LICENSE`](https://github.com/future-agi/agent-learning-kit/blob/main/LICENSE): Apache-2.0 license.
+- [`NOTICE`](https://github.com/future-agi/agent-learning-kit/blob/main/NOTICE): Apache notice metadata.
+
+## Development
+
+New public SDK development belongs here. See [DEVELOPMENT.md](https://github.com/future-agi/agent-learning-kit/blob/main/DEVELOPMENT.md)
+for the boundary between this package and the backing engine repos.
+
+```bash
+uv sync
+uv run ruff check .
+uv run pytest -q
+uv run python -m build
+pnpm --dir typescript --filter @future-agi/agent-learning-kit build
+pnpm --dir typescript --filter @future-agi/agent-learning-kit test -- --runInBand
+```
+
+For the heavier release cut, run `agent-learn release-proof --project-root .`.
+It emits `agent-learning.release-proof.v1` with command evidence for the full
+local proof stack.
+
+Before a release:
+
+```bash
+uv run python -m fi.alk.cli release-proof \
+  --project-root . \
+  --output /tmp/agent-learning-release-proof.json \
+  --quiet
+```
+
+`release-proof` includes release-check, full-repo ruff, pytest, Python package
+build, TypeScript package build/test, and `git diff --check`. Use
+`--only <check>` for partial proof during development or `--dry-run` to emit the
+exact command plan without executing commands.
+
+## Project Status
+
+The v1 release gate is local-first and executable. It covers SDK consolidation,
+promptfoo-style CLI usage, native optimizer evidence, docs/examples, schema
+kinds, packaging metadata, red-team corpus/campaign coverage, Future AGI
+UI/action/report artifacts, framework/provider compatibility, environment
+robustness, regression replay, and release proof.
+
+All v1 gates are green on the proved release commit (see the release-proof
+artifact). Roadmap milestones marked "mostly complete" or "in progress" are
+extend-only: the v1 contract those gates assert is frozen and proved; the named
+extensions land post-v1 without weakening any gate.
+
+## Community
+
+- Contributions: [CONTRIBUTING.md](https://github.com/future-agi/agent-learning-kit/blob/main/CONTRIBUTING.md)
+- Code of conduct: [CODE_OF_CONDUCT.md](https://github.com/future-agi/agent-learning-kit/blob/main/CODE_OF_CONDUCT.md)
+- Security reports: [SECURITY.md](https://github.com/future-agi/agent-learning-kit/blob/main/SECURITY.md)
+- License: [Apache-2.0](https://github.com/future-agi/agent-learning-kit/blob/main/LICENSE)
+
+## Deep Dive
+
+The full documentation set — quickstarts, per-track guides, framework pages,
+and reference material — starts at [docs/index.md](https://github.com/future-agi/agent-learning-kit/blob/main/docs/index.md).