agentframe-py 0.2.0__tar.gz → 0.3.0__tar.gz

agentframe_py-0.3.0/.github/workflows/base-image.yml

@@ -0,0 +1,62 @@
+# Weekly check that the python:3.13-slim digest pinned by `agentframe compose`
+# (src/agentframe/compose.py, tests/unit/golden/Dockerfile and
+# docs/guides/development.md) is still the tag's current digest. When it is
+# not, it opens one issue and fails, so a maintainer refreshes the pin
+# deliberately; it never writes to the repository.
+name: Base image digest
+
+on:
+  schedule:
+    - cron: "0 6 * * 1"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  check:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      issues: write
+    env:
+      IMAGE: python
+      TAG: 3.13-slim
+      ISSUE_TITLE: Refresh the pinned python:3.13-slim digest
+    steps:
+      - uses: actions/checkout@v7
+      - name: Compare the pinned and current digests
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          set -euo pipefail
+          pinned=$(sed -n 's/^FROM .*@//p' tests/unit/golden/Dockerfile)
+          token=$(curl -fsS "https://auth.docker.io/token?service=registry.docker.io&scope=repository:library/${IMAGE}:pull" | jq -r .token)
+          current=$(curl -fsSI \
+            -H "Authorization: Bearer ${token}" \
+            -H "Accept: application/vnd.oci.image.index.v1+json" \
+            -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" \
+            "https://registry-1.docker.io/v2/library/${IMAGE}/manifests/${TAG}" \
+            | tr -d '\r' | sed -n 's/^docker-content-digest: //Ip')
+          if [[ ! "${current}" =~ ^sha256:[0-9a-f]{64}$ ]]; then
+            echo "::error::Could not read the current ${IMAGE}:${TAG} digest."
+            exit 1
+          fi
+          echo "Pinned:  ${pinned}"
+          echo "Current: ${current}"
+          if [[ "${pinned}" == "${current}" ]]; then
+            exit 0
+          fi
+          open=$(gh issue list --state open --search "\"${ISSUE_TITLE}\" in:title" --json number --jq length)
+          if [[ "${open}" == "0" ]]; then
+            gh issue create --title "${ISSUE_TITLE}" --body "$(cat <<EOF
+          \`${IMAGE}:${TAG}\` now resolves to \`${current}\`; \`agentframe compose\` pins \`${pinned}\`.
+
+          After testing the new image, update the digest in \`src/agentframe/compose.py\` (\`BASE_IMAGE\`), \`tests/unit/golden/Dockerfile\` and \`docs/guides/development.md\`. See the Containers section of the development guide.
+
+          Opened by the [Base image digest workflow](${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}).
+          EOF
+          )"
+          fi
+          echo "::error::${IMAGE}:${TAG} moved to ${current}; refresh the pinned digest."
+          exit 1

{agentframe_py-0.2.0 → agentframe_py-0.3.0}/AGENTS.md

@@ -14,10 +14,11 @@ This repository contains the M0 development tooling, M1 model-free
 authoring/local `Runtime`, M2 native A2A hosting, M3 Ollama/Foundry text
 generation, M4 typed structured outputs, M5 bounded tool calling, M6 built-in
 opt-in model routing and M7 hardening alongside design documentation. Version
-0.1.0, the first usable single-agent release, is published on PyPI as
-`agentframe-py` (import `agentframe`). Pre-1.0 there is no stable API. The
-M8+ multi-agent stories (#83-#97) are complete and form the 0.2.0
-multi-agent release; later features (MCP, telemetry, streaming) are not implemented. Keep design API examples beyond
+0.1.0, the first usable single-agent release, and version 0.2.0, the M8+
+multi-agent release (#83-#97), are published on PyPI as `agentframe-py`
+(import `agentframe`). Pre-1.0 there is no stable API. M9 evaluation (#164, #152-#163)
+and M10 model-planned orchestration (#177, #166-#176) are complete and form
+version 0.3.0; later features (MCP, telemetry, streaming) are not implemented. Keep design API examples beyond
 the implemented surface labeled as unimplemented; explicitly approved
 names/defaults are decisions, not still-unresolved recommendations. Do not
 scaffold implementation or add dependencies merely to make design snippets
@@ -88,24 +89,27 @@ Folders stay one level deep. Put new code in the group that owns the concept:
 | Path | Contents |
 | --- | --- |
 | `src/agentframe/__init__.py` | Public API re-exports; every public name is importable from `agentframe`. |
-| `src/agentframe/` (root) | `errors.py`, `testing.py` (`ScriptedModel`), `cli.py` (`agentframe run` / `dev` / `compose`), `project.py` (`agentframe.yaml`), `dev.py` (`dev_settings()`), `dev_server.py` (`agentframe dev` supervisor), `compose.py` (`agentframe compose` generator). |
+| `src/agentframe/` (root) | `errors.py`, `testing.py` (`ScriptedModel`), `cli.py` (`agentframe run` / `dev` / `compose` / `eval`), `project.py` (`agentframe.yaml`), `dev.py` (`dev_settings()`), `dev_server.py` (`agentframe dev` supervisor), `compose.py` (`agentframe compose` generator). |
 | `src/agentframe/authoring/` | `Agent`, `@agent` / `@skill`, definitions, `RunContext`, `Inject` / `Services`. |
 | `src/agentframe/runtime/` | `Runtime`, generation, `RunLimits`, `RunOptions`, `Usage`. |
 | `src/agentframe/models/` | `base.py` model protocol and settings; provider adapters `ollama.py`, `foundry.py`; `strict_schema.py` (OpenAI strict JSON-schema converter). |
 | `src/agentframe/hosting/` | `AgentApp`, `HostConfig`, Agent Card, A2A executor, routing. |
 | `src/agentframe/security/` | `Principal`, authentication, authorization. |
+| `src/agentframe/orchestration/` | `RemoteSkill`, `SkillCatalog`, bounding of untrusted card text, `RemoteTool` (remote entries as model tools, with per-entry permissions and tool checks), `Plan` / `PlanStep` with static checks and reference substitution (`plan.py`), the `ctx.plan` planning call (`planning.py`) and the plan executor behind `ctx.run_plan` (`execution.py`, `PlanResult`, `StepResult`) (M10). |
 | `src/agentframe/discovery/` | `AgentRegistry`, `AgentReference`, `AgentQuery`, file and environment registries (M8+). |
+| `src/agentframe/evaluation/` | Evaluation harness (M9): `cases.py` (`Case`, `load_cases`), `targets.py` (`Target`, `TargetResult`, `LocalSkill`, `A2ATarget`), `scoring.py` (`Score`, scorer calls, built-ins `equals`, `contains`, `valid`), `judge.py` (`Judge`, the rubric scorer's model-backed run and its budget holds), `report.py` (`CaseResult`, `EvalReport`, versioned JSON, `require`), `budget.py` (`TokenBudget`, evaluation-wide token holds), `harness.py` (`Evaluation`). Imported as `agentframe.evaluation`, not re-exported from `agentframe`, so its `__init__.py` re-exports its own public names. |
 | `tests/unit/<group>/` | Unit tests mirroring the source groups; root-module tests in `tests/unit/`. |
 | `tests/integration/` | Interoperability, real providers, success criterion, README example. |
-| `tests/support/` | Shared helpers, on pytest's `pythonpath` (`from local_server import ...`). |
-| `examples/` | Runnable examples, each with `create_app()`; `smoke.py` checks them all against the installed wheel in CI. |
+| `tests/support/` | Shared helpers, on pytest's `pythonpath` (`from local_server import ...`); `agent_processes.py` starts agents (`agentframe run`) and the plain A2A peer (`peer_process.py`) as separate processes for release tests. |
+| `examples/` | Runnable examples, each with `create_app()` (`evaluation.py` has `create_evaluation()` instead; `multi_agent/planner_evaluation.py` has `create_planner_evaluation(model, runtime)`); `smoke.py` checks them all against the installed wheel in CI. `multi_agent/` holds the research, writer and planner agents (hand-written `plan`, model-planned `propose` / `planned`, M10). |
 | `docs/roadmap.md` | Product boundaries, decisions and milestones. |
 | `docs/guides/` | How to use AgentFrame and how its behavior works. |
 | `docs/design/` | Rationale and proposed designs. |
 | `docs/api/` | Public surface reference. |
 
 Keep subpackage `__init__.py` files to a short docstring; add public exports to
-the top-level `__init__.py` only. Import internal modules by full path, for
+the top-level `__init__.py` only (except `agentframe.evaluation`, which
+`import agentframe` must not load). Import internal modules by full path, for
 example `from agentframe.runtime.limits import RunLimits`. Keep test basenames
 unique, since test folders are not packages.
 
@@ -118,7 +122,20 @@ clear error for an unknown model/deployment, a hosted structured call through
 the official A2A client, a validated typed `output=` result and the example's
 `summarize` skill returning a typed data part over A2A, a one-tool round trip
 run locally and through `examples/orders.py` over A2A (skipped if the model
-lacks tool support), a Foundry one-tool round trip that replays encrypted
+lacks tool support), a model calling a localhost agent's skill as a remote
+tool (M10 #167, checking only that the entry was called; skipped without tool
+support), a rubric `Judge` grading one model-free case in an `Evaluation`
+by majority of `votes=3` (M9 #160, #161), `examples/evaluation.py`'s answers
+graded by a rubric judge (M9 #163; xfails for Ollama when the small judge
+fails correct answers), a model planning one unambiguous
+two-step goal over two localhost agents' cards with `ctx.plan` (M10 #170,
+checking only the entries and the dependency; `llama3.2:3b` often writes
+an invalid or different plan, an Ollama xfail), the `examples/multi_agent` planner planning and running its
+post with the real model and the planner evaluation with `report.require`
+(M10 #176, `-k real_planner`; specialists use stand-in models; for Ollama only
+model-quality outcomes xfail: an invalid or empty plan, a plan the example's
+checks refuse, or more than half the evaluation's plans erroring that way;
+any other failure fails), a Foundry one-tool round trip that replays encrypted
 reasoning items and one unambiguous free-text request routed by
 `examples/routed_assistant.py` over A2A (checking only the selected skill and
 that its argument is present, since routing is nondeterministic). They are marked
@@ -158,7 +175,10 @@ agent and fetches that agent's card, read-only. It runs only with
 roles are in `docs/api/discovery.md#foundryagentregistry`. The same agent is the
 real non-AgentFrame server for #94: one test `ask`s it once, another checks
 `call` is refused before sending. In CI, `test_multi_agent_interop.py` proves
-both directions against a plain SDK peer (`tests/support/a2a_peer.py`).
+both directions against a plain SDK peer (`tests/support/a2a_peer.py`). For
+#159 the same file evaluates the agent with one `A2ATarget` `ask` case and
+checks a `skill=` case is refused before sending; `test_evaluation_interop.py`
+does the same against the plain SDK peer in CI.
 
 `tests/integration/test_entra_interoperability.py` has one opt-in `entra` test
 using a real token from `az account get-access-token`, enabled with
@@ -448,7 +468,9 @@ Model deployments are not agent endpoints; no AgentFrame-to-Foundry deployment
 or tested registry integration is claimed.
 
 Implemented (#84): `AgentDirectory(sources=[...], card_ttl_seconds=300,
-allow_stale=False, allow_http_hosts=(), http_client=None)` owns SDK card
+allow_stale=False, allow_http_hosts=(), http_client=None,
+max_reply_bytes=8 MiB)` (#150: `Remote` replies streamed, over it -> safe
+`RemoteAgentError`) owns SDK card
 fetch/validation/cache/refresh via an internal fetcher (no public
 `AgentCardResolver`/`CardCacheConfig`). Sources read on every `get`; cards
 cached per card URL, refetched when stale; failed refresh raises unless
@@ -538,7 +560,9 @@ reservations. Default max 5 remote hops, excess fails. These controls use the
 same A2A extension; detailed config/wire/usage trust remains open, not base
 protocol fields or hard billing guarantees.
 No orchestration helpers in the first multi-agent release: plain await,
-asyncio.TaskGroup and directory.find. Later optional `Remote[ResearchAgent]`
+asyncio.TaskGroup and directory.find. M10 (0.3.0) adds opt-in remote skills as model
+tools and checked plans (`docs/guides/orchestration.md`); plain await stays
+the default. Later optional `Remote[ResearchAgent]`
 typed proxy always uses structured selection, never prompt routing. Prove two or more
 independently deployed AgentFrame agents, an orchestrator discovering/calling a
 specialist with propagated context, and non-AgentFrame A2A client/server interop.
@@ -605,7 +629,11 @@ failed runs log ID, class and safe message; skill/tool/router tracebacks stay in
 operator logs with the ID, so authors keep secrets out of exception messages.
 `tests/integration/test_hardening.py` is the M7 end-to-end acceptance suite;
 `tests/integration/test_multi_agent_release.py` is M8+'s, running
-`examples/multi_agent` as separate `agentframe run` processes.
+`examples/multi_agent` as separate `agentframe run` processes;
+`tests/integration/test_orchestration_release.py` is M10's (#176), running its
+research, writer and planner agents, the plain SDK peer and a registry-wide
+coordinator (`tests/support/release_targets.py`) as separate processes with a
+scripted planning model, plus the planner evaluation.
 Network host examples must include clearly labeled application-supplied
 development authentication/authorization placeholders where appropriate;
 loopback/omission is not access policy or a production-safe default.
@@ -673,7 +701,7 @@ remain open; tracing/accounting/enforcement work independently. See
 
 Read `README.md` and `docs/README.md` first. Keep `docs/roadmap.md`, the
 guides (`docs/guides/annotations.md`, `agent-examples.md`, `limits.md`,
-`authorization.md`, `development.md`) and the design docs
+`authorization.md`, `development.md`, `evaluation.md`, `orchestration.md`) and the design docs
 (`docs/design/base-classes.md`, `tracing.md`, `mcp-skills.md`,
 `prompt-safeguarding.md`, `discovery.md`, `streaming.md`, `evaluation.md`,
 `orchestration.md`) consistent, including discovery/streaming API pages and