ageval-cli 0.7.1__tar.gz → 0.7.2__tar.gz

ageval_cli-0.7.2/.agents/skills/ageval-cli/SKILL.md

@@ -0,0 +1,47 @@
+---
+name: ageval-cli
+description: >
+  Operate ageval CLI (lock/run/plugin/view/publish/release/executors/campaign/evidence/status/cancel/jobs/results):
+  flags, --set pointers, exit codes, --probe, offline agent path (AGEVAL_OFFLINE_AGENT),
+  ACP entry readiness, plugin install, suite upload. Use when running ageval lock/run, PASS/FAIL/ERROR,
+  trajectory export, Hub upload. Do not invent flags.
+---
+
+# ageval CLI
+
+```bash
+uv sync --frozen --all-packages
+uv run ageval --help
+```
+
+Public commands: `ageval lock` `ageval run` `ageval campaign` `tasks` `jobs` `view` `plugin` `evidence` `ageval status` `ageval cancel` `executors` `publish` `release` `login` `agent` `registry` `cache` `results`. No `submit`. Path arguments are **dataset** roots.
+
+```bash
+uv run ageval lock examples/core --task config-minimal
+uv run ageval run  examples/core --task acp-local-min
+uv run ageval run  examples/core --task acp-docker-min --profiles examples/core/profiles.docker.yaml
+uv run ageval run  examples/datasets/minimal-demo --task terminal-jsonl-agg
+uv run ageval run  official/demo@0.1.0 --dir tmp
+uv run ageval view official/demo@0.1.0
+uv run ageval run  … --probe          # lock + preflight, no Agent
+uv run ageval executors -v
+uv run ageval plugin install plugins/nooa
+```
+
+`--profiles` replaces the dataset job document. `--agent` and `--profiles` are mutually exclusive. `--agent pi` (and other shipped short ids) needs no `agent install`. `--model` requires `--agent` and overrides this run’s `binding.model` (not `ageval results --model`). `--set` allowlist: `/parameters/seed`, `/parameters/active_profile`, `/bindings/<role>/{model,executor,api_key,base_url,options/<key>}`. Not `limits.*`.
+
+TTY prints a short recap (no digest / sha256). A pipe keeps JSON. `ageval run --json` forces the run document.
+
+Exit: 0 PASS / probe ready; 1 FAIL / probe not ready; 2 ERROR.
+
+Single Attempt evidence: `<dataset>/.ageval/runs/<id>/` (`lock.json`, `result.json`, `trajectory.jsonl`; optional `evaluation/observation.jsonl` for evaluate-phase SDK invoke). Trajectory ≠ PASS.
+
+`--probe` missing `E2B_API_KEY` or ssh locator → ready false, started false.
+
+Offline: `AGEVAL_OFFLINE_AGENT=1` must not PASS agent tasks.
+
+Hub Leaderboard needs suite upload (`ageval results upload-suite`), not a lone `results upload`.
+
+Flags: `uv run ageval <cmd> --help`. Source: `src/ageval/cli/main.py`.
+
+Detail: [references/commands.md](references/commands.md), [references/failures.md](references/failures.md).

ageval_cli-0.7.2/.agents/skills/ageval-cli/references/commands.md

@@ -0,0 +1,108 @@
+# CLI command notes
+
+Box kind is `environment:` on the profiles document. `--probe` is `ageval run --probe` (lock + preflight). Dataset root. `submit` does not exist. Trust `uv run ageval --help`.
+
+## `ageval registry`
+
+- `list` / `show <id@version|@sha256:…>`: visible releases; `show` prints digest / size / visibility
+- Org ops (`org-create` / `org-add-member` / `org-set-role` / `org-transfer` / `org-remove-member`): owner or admin; the last owner cannot leave or be demoted
+- `set-visibility <id@version> --visibility public|private`: owner or admin
+- `set-description <org/name> --description '…'`: marketplace description override, no republish; empty string clears the override (falls back to the package manifest description). Server enforces org owner
+- `delete <id@version> --yes`: owner or admin
+
+## `ageval executors`
+
+Two product facts (+ ACP entry inventory):
+
+1. **What ageval supports** — `agent_profiles[].executor` kinds on this install
+2. **What the host can run** — binaries / ACP entries ready (no secrets)
+
+Stdout JSON (high level):
+
+| Key | Meaning |
+| --- | --- |
+| `supported` | Kind names valid for `agent_profiles[].executor` (e.g. `acp`, `openai-http`, `anthropic-http`) |
+| `host_ready` | Subset of kinds that can be constructed on this host |
+| `executors[]` | Per kind: `execution_mode` (from `describe()` when published), `host_ready` |
+| `acp_entries[]` | Per ACP `entry_id`: `acp_command`, `engine_ready`, `acp_entry_ready`, `host_ready`, credential env *names*; `-v` adds `credential_missing` / `keyless_auth` |
+
+- ACP registry: `ageval.plugins.contrib.acp.registry` (static pins)
+- Plugin `host_ready` uses declared `host_requires` / reachable `describe()`
+- Child env is allowlisted (`PATH` / `HOME` / `LANG` + entry credential names + binding locators + `fixed_env`). Undeclared host tokens do not reach the entry
+- No package path; no secrets; exit 0
+
+**Author packages with:** `executor: acp` + `- plugin: acp` / `options.entry: <entry_id from acp_entries>`.
+
+## `ageval lock`
+
+- Deterministic JSON on stdout (digest, `dataset_id`, task_id, resolution). No `database_id`.
+- No secret values.
+- Does not create Run/Attempt or start Agent.
+- Rejects unknown format (`invalid_format` at `/format`), unknown `executor` kinds, and ACP profiles missing `- plugin: acp` / `options.entry`.
+- `--probe` on **run** (not a second lock mode that starts an environment): lock plus observational readiness. Exit non-zero when the selected `environment` path is unsatisfied. Checks declared `host_requires`, Docker daemon when kind is docker, locator **names** (never values). ACP entries add `credential_missing` (if the entry requires a key, the check fails and the run does not start; warning-only when `keyless_auth`). HTTP executors report `credential_missing` only when the locked `base_url` is not loopback. `AGEVAL_OFFLINE_AGENT=1` is reported; probe still does not spawn.
+
+## `ageval run`
+
+- `--probe` (requires `--task`): same feasibility report; does not start an Attempt.
+- TTY: short recap on stdout (per-task PASS/FAIL, `summary` / `logs`, `ageval view` next). Full document stays on disk. `--json` or a non-TTY stdout keeps one JSON object.
+- One foreground Attempt via production composition root **when** `--task` is set, `-k` defaults to 1, and no `--resume-suite`.
+- Evidence under dataset root `.ageval/runs/<id>/`. `logs` / `evidence_path` are portable relative to the dataset root.
+- **Always-k** (`--n-attempts` / `-k`, integer ≥1): fixed k independent Attempts per task. CLI/job only — not `task.yaml`.
+- **Suite**: omit `--task` → all members.
+- **`--max-concurrent-tasks`**: speeds wall time only; does not change k or PASS.
+- **`--resume-suite` / `--replace-slot` / `--attempt-index`**: suite resume; see `ageval run --help`.
+- **`--keep-workspace`**: after cleanup, do not delete the box work root (host residual may still be named `l1-work/` — historical directory name, not an isolation tier). Default off. Docker volumes are still removed. Upload packs never include `l1-work/**`.
+- **`--keep-vendor-raw`**: after a successful trajectory seal, keep invocation `backend_raw/` / layer B (`request.json`, `events.jsonl`, `final-response.json`, `metadata.json`) and `evaluation/evaluator_raw.json`. Default off: those files are dropped; Hub archives skip them even if residuals remain. `evaluation/observation.jsonl` (evaluate-phase SDK invoke) is **kept** either way. Independent of `--keep-workspace`.
+- `--profiles` replaces the dataset job document. `--agent` and `--profiles` are mutually exclusive. `--agent pi` resolves the shipped catalog tree (no install). `--model` requires `--agent` and writes this run’s overlay model; omit it to keep the package default. Not `ageval results --model`.
+- `--dir <path>`: only with a registry ref. Looks at `<path>/<dataset_id>/` (example: `--dir tmp` + `official/demo@0.1.0` → `tmp/official/demo`). Reuse that child if it already matches; otherwise fetch into it and run. Relative paths are from cwd. Local path + `--dir` is `invalid_override`.
+- Per invocation: Core writes `trajectory.jsonl` (`ageval.trajectory.event/1`) for **run-phase** Agent turns. Evaluate-phase SDK invoke (LLM-as-judge) writes `evaluation/observation.jsonl` instead. Neither file is PASS.
+- Docker kind uses the docker environment winner + `attach_stdio`; coding entries stay on the parent ACP client.
+
+## `ageval evidence`
+
+- Read-only export of **sealed** invocations.
+- Refuses unsealed (running) metadata.
+- Does not change evaluation score.
+
+## `ageval campaign`
+
+- Foreground serial matrix; allowlisted `/parameters/*` axes.
+- Not a parallel scheduler. Does not merge with `run.py` inner loops.
+
+## Control surface
+
+- `status` / `cancel` operate on ControlStore records.
+- Suite jobs may take `--dataset` to read `progress.json` or write `cancel.requested`.
+- There is no `submit`.
+
+## Always-k vs campaign
+
+- **Always-k** (`ageval run -k`): repeat independent Attempts for pass@k samples.
+- **Campaign** (`ageval campaign --matrix`): sweep allowlisted parameters / bindings on one task.
+
+## `ageval results upload` / `upload-suite`
+
+- Single Attempt upload does **not** create a suite Leaderboard row.
+- Prefer `upload-suite` after `ageval run <dataset>` **without** `--task`.
+- Dataset arg is a path or registry ref (`id@version`); a unique verified cache hit does not contact Hub to find the local suite-runs.
+- Hub Leaderboard **Public** lists complete, release-bound suites only.
+
+### Hub path checklist
+
+```text
+ageval publish <dataset> --org <org> --draft
+ageval release <org/dataset>
+ageval run <dataset> [--profiles …]          # omit --task → suite
+ageval results upload-suite <dataset> --suite-run <8-hex> --public [--with-attempts]
+```
+
+## `ageval publish` / `ageval release`
+
+- Draft slot overwrites for that `dataset_id`. Reserved version name `draft` is not a release.
+- `ageval release <dataset_id>` promotes the current draft.
+- Plugin packages do **not** use the draft slot.
+
+## `ageval view` / `ageval jobs delete`
+
+- Local dataset UI. Path or registry ref (`id@version` / `@sha256:…`). A unique verified cache hit does not contact Hub.
+- `jobs delete --local <dataset> --job <id> --yes` is local Job delete, not Registry `results delete`.

ageval_cli-0.7.2/.agents/skills/ageval-cli/references/failures.md

@@ -0,0 +1,23 @@
+# CLI failure diagnosis
+
+| Symptom | Check |
+| --- | --- |
+| exit 2, empty stdout on lock | Config error on stderr (`unknown_profile`, schema, `invalid_format` at `/format`) |
+| lock `unsupported_capability` / `unsupported executor` | Kind not in `ageval executors` `.supported` — coding agents need `executor: acp` |
+| lock `options.entry required` | ACP profile missing `- plugin: acp` / `options.entry` |
+| Agent ERROR offline | Expected under `AGEVAL_OFFLINE_AGENT=1` |
+| executor unbound on docker | Plugin did not bind to the environment (`attach_stdio` / in-environment worker). Do not silently fall back to the host |
+| `image_contribute_unsatisfied` / missing bake | Bound external executor but `config.image_layers` / `Dockerfile.bake` empty — `$ageval-plugin` |
+| `nooa_package_missing` / `No module named 'nooa'` | Host needs `uv sync --extra nooa`; docker writes the package into the image |
+| ACP entry not ready | `ageval executors -v` → that `entry_id` `host_ready` / install pin; no invoke-time `npm i` |
+| `credential_missing` | Declared credential env names unset and no `api_key` locator. Required ACP entries fail at `--probe` / session-open; keyless (OAuth) warn only. HTTP executors (`openai-http` / `anthropic-http` / `dsh` / `nooa` / `miniswe`) skip this when locked `base_url` host is `127.0.0.1` / `localhost` / `::1` |
+| e2b / ssh probe not ready | Missing `E2B_API_KEY` or SSH locator — the check fails and the run does not start; `started: false`. Skip is not a pass |
+| PASS without real model | Forbidden — do not use fixtures as public proof |
+| Trajectory empty | Non-empty `agent_profiles` + `run.py` `Agent.session`/`invoke` |
+| Resume skipped an ERROR | Default `--resume-suite` skips finished PASS / FAIL / ERROR. Use `--replace-slot --task T` |
+| Export fails `unsealed_invocation` | Attempt still running or metadata not terminal |
+| Export fails `secret_residual` | Fix source evidence; do not strip secrets by hand in export dir |
+| Docker ERROR | Docker daemon, image build, network/creds projection; read result / agent meta under `.ageval/runs/` |
+| Invoke / wall / verifier timeout | FAIL (`metrics.reason=timeout`), exit 1 — not ERROR. Environment start failure stays ERROR |
+
+Design: `docs/design/05-runtime/evidence.md`, `docs/design/07-budget-evaluation-failure.md`.

ageval_cli-0.7.2/.agents/skills/ageval-config-package/references/isolation.md

@@ -0,0 +1,73 @@
+# Isolation notes for package authors
+
+Authority: `docs/design/06-capability-adapter-visibility.md`, `docs/design/05-runtime/environment.md`.
+
+## Gold / evaluation material
+
+- Put hidden labels under `evaluation/` (paths never mounted to Agent).
+- Gold is uploaded at the start of evaluate, not during run. This is a cut in **time**, not `path_views`.
+- Evaluator receives allowlisted staged inputs only after writer barrier.
+- Optional LLM-as-judge: `Agent.session` is allowed **after** that upload. Solver profiles that already ran must not invoke. Judge prompts stay out of `evaluation/observation.jsonl` user rows.
+
+## Box kind
+
+- Job `environment: local | docker | e2b | ssh | daytona`.
+- docker / e2b share `environment/Dockerfile` (or an image option). Official base:
+  `FROM ageval-attempt:base` (`docker/attempt/`). Secrets never baked into layers.
+- Coding agents: `executor: acp` + `options.entry`; parent ACP + `host.attach_stdio`.
+- `setup.sh` is the last environment slot (`environment_setup`). No provision phase.
+
+## Dockerfile depth (conversion)
+
+| Tier | When | Pattern |
+| --- | --- | --- |
+| **Thin** | Official base already has engines + ACP entries | `FROM ageval-attempt:base` |
+| **Heavy** | Extra apt/pip/lang | Same base **+ migrated `RUN`** at **build** time |
+
+**Forbid as parity default:** runtime `apt-get` / floating `pip install` / `npm i` / `npx`
+inside the Attempt after start. Bake deps in the image.
+
+### Official base is not the upstream runtime
+
+The official Attempt image is **CPython 3.12 + baked ACP entries**. It is **not**
+the upstream bench’s Python or pinned wheels. If the package Dockerfile `FROM`s this base:
+
+1. **Do not** copy upstream `pip install foo` unpinned.
+2. **Pin to what the checkout imports.**
+3. **Gate at image build**, not first eval: `python -c "from …"` so a bad pin fails `docker build`.
+4. Collection / conftest ImportError is eval ERROR, not FAIL.
+5. `host_requires` / `--probe` do not catch eval deps inside the image — those are the package Dockerfile’s job.
+
+### `data/` seed
+
+Put agent-visible files under `tasks/<id>/data/`. The environment phase uploads them to `/attempt/workspace`. Prefer this over baking content into image layers.
+
+### `solution/` (offline fixture)
+
+- Default: **not** seeded to Agent.
+- Offline helper flags may copy `solution/*` into workspace — author-level only, not production default.
+
+## Dataset `shared/`
+
+- Import contract: Runtime injects **`[task_dir, dataset_root]`** — **not** the `shared/lib` leaf. Authors write `from shared.lib…`.
+- `shared/lib` is for `run.py` / evaluator import only — **not** default Agent mount.
+- Gold stays under `tasks/*/evaluation/` only; **forbid** gold / `.env` under `shared/`.
+- Task members must **not** own top-level name `shared`.
+- docker images: **no** Core implicit COPY of `shared/`. The same `evaluator.py` with `from shared.lib…` can pass on `environment: local` and ImportError on docker unless the Dockerfile `COPY`s `shared/` and puts the **dataset root** on `PYTHONPATH`.
+
+```dockerfile
+FROM ageval-attempt:base
+COPY shared/ /attempt/shared/
+ENV PYTHONPATH=/attempt:/attempt/task:${PYTHONPATH}
+```
+
+Do **not** `COPY` `tasks/*/evaluation/` gold into Agent-visible layers.
+
+## Do not
+
+- Rely on “delete field from yaml” as isolation.
+- Mount gold into `run.py` “for convenience”.
+- Put credentials in the package tree.
+- Expect Core to auto-COPY `shared/` into docker images.
+- Use runtime package installs as the default parity path.
+- Treat `FROM ageval-attempt:base` + unpinned `pip install` as equivalent to the upstream instance image.

ageval_cli-0.7.2/.agents/skills/ageval-platform/references/authority.md

@@ -0,0 +1,27 @@
+# Authority & conflict resolution
+
+## Chain
+
+```text
+docs/design/*  →  ARCHITECTURE.md  →  GitHub Issues  →  code/tests/examples
+```
+
+`docs/` is self-contained. Do not read an external BRIEF / vault as design truth.
+
+`ARCHITECTURE.md` 拥有 Current 源码树、依赖图、五个阶段状态机、emit 图、数据流表、失败归属。施工红线与 CI 命令在根 `AGENTS.md`。不要在 skill 里复制那两份图。
+
+`website/` is reader-facing product documentation only — never design truth.  
+`apps/*` and `services/*` READMEs own SPA/service **dev** detail, not product tutorials.
+
+## Conflict handling
+
+1. Stop the conflicting implementation.
+2. Classify: design change | structure change | delivery scope | implementation drift.
+3. Edit the **highest** authority first.
+4. Sync README / AGENTS / website / tests in the same change when claims change.
+
+## What skills are not
+
+- Not a second design doc.
+- Not a license to claim unfinished surfaces as available.
+- Not allowed to invent `ageval` subcommands not present in `src/ageval/cli/main.py`.

ageval_cli-0.7.2/.agents/skills/ageval-platform/references/shipped-surfaces.md

@@ -0,0 +1,27 @@
+# Shipped public surfaces (orientation)
+
+Prefer mechanism packages under `examples/core/`. Full list: `examples/README.md`.
+Evidence grade is **limited `runnable-mvp`**. Do not claim `isolated` from one happy path.
+
+| Package | Demonstrates |
+| --- | --- |
+| `examples/core --task config-minimal` | `ageval lock` success (`dataset_id`, no `database_id`) |
+| `examples/core --task acp-local-min` | local ACP public run |
+| `examples/core --task acp-docker-min` | docker ACP public run (`--profiles examples/core/profiles.docker.yaml`) |
+| `examples/core --task sdk-agent-session` | Host multi-invoke + independent PASS |
+| `examples/core --task plugin-agent-executor` | Second executor mechanism (`openai-http`) |
+| `examples/datasets/minimal-demo --task terminal-jsonl-agg` | Named demo task |
+| `examples/datasets/minimal-demo --task env-postgres-min` | Sidecar / compose path |
+| docker topology `sdk-session-single-actor` | lock 有 topology 即可 |
+
+e2b / ssh / daytona: code exists; missing key → `--probe` `ready: false`. Skip is not a pass.
+
+## Install (repo root)
+
+```bash
+uv sync --frozen --all-packages
+uv run ageval --help
+uv run ageval executors -v
+```
+
+Design: `docs/design/00-overview-and-product.md`, `docs/design/01-ageval-core.md`, `ARCHITECTURE.md`.

ageval_cli-0.7.2/.agents/skills/ageval-plugin/references/authoring.md

@@ -0,0 +1,246 @@
+# Author a `ageval.plugin/1`
+
+Reference implementations: `plugins/nooa` (executor), `plugins/dsh`
+(DeepSeek JSON-RPC, not ACP), `plugins/home-files` (chain). Design:
+`docs/design/11-extension-plugins.md`.
+
+Slots are **exclusive** and **chain** only. Do not invent extra slot kinds.
+
+## Package layout
+
+```text
+plugins/my-mech/
+├── plugin.yaml                 # format: ageval.plugin/1
+├── README.md                   # required: capability + parameter tables
+├── docker/Dockerfile.bake      # optional: docker image layers
+├── src/my_mech/
+│   ├── __init__.py             # must not import ageval Core at package import
+│   ├── factory.py
+│   ├── hooks.py
+│   └── trajectory.py           # optional: native → ageval.trajectory.event/1
+└── worker/                     # optional: in-environment entry
+```
+
+```yaml
+format: ageval.plugin/1
+plugin_id: my-mech
+version: "0.1.0"
+description: "One paragraph Hub shows above Install (CLI). Links: [text](https://example.com)."
+host_requires:
+  - import: my_vendor_sdk
+    hint: "uv sync --extra my-mech"
+slots:
+  exclusive:
+    - id: executor
+      priority: 110
+      entry: "my_mech.factory:build_executor"
+  chain:
+    - id: trajectory_collect
+      priority: 110
+      entry: "my_mech.hooks:trajectory_collect"
+inject:
+  - service: environment
+    capabilities: [exec, upload]   # ACP uses attach_stdio instead
+config:
+  image_layers: docker/Dockerfile.bake
+```
+
+Hub: `package_kind=plugin`. Dataset vs plugin is rejected.
+`description` is optional Hub copy (one paragraph above Install (CLI)). Empty / non-string fails closed. Markdown links (`[text](https://…)`) render on Hub; other block syntax does not.
+
+README is the Hub detail contract (previewed next to `plugin.yaml`). A plugin that
+omits capability or parameter tables is unfinished. See **README contract** below.
+
+`host_requires` allowlist keys: `import`, `file`, `hint`. Unknown keys are rejected.
+`import:` is `importlib.util.find_spec` (no spawn). Core does not map plugin-id → pip extra.
+docker kind does **not** consume `host_requires` (image bake does).
+
+An environment plugin fills exclusive slot `environment` and implements
+`src/ageval/environments/protocol.py`. Do not import docker/e2b/ssh from ACP.
+
+## Executor
+
+Host factory: `build_executor(**kwargs)`. Common kwargs: `options`, `profile_id`,
+`model`, `base_url`, `api_key` (locator name), `host`, `placement`, `package_root`.
+
+ACP attaches with `host.attach_stdio`. dsh / nooa run a baked (or uploaded)
+worker with `host.exec` and `host.upload`. Missing capability → lock fails.
+Core must not reconstruct a container executor by kind. Do not silently fall back to the host.
+
+`describe()` keys already in production (copy semantics):
+
+```text
+execution_mode, tools, structured_output, session, stream,
+credential_env_names, binary
+```
+
+## Trajectory
+
+Layer B: each event row has `schema: ageval.trajectory.event/1`, `source` = this plugin id, `session_id`.
+trajectory.jsonl: the `trajectory_seal` winner writes `trajectory.jsonl` (engine default). Plugins must not emit layer-C rows unless they won that slot.
+
+`trajectory_collect` may map **this** plugin's vendor dump into layer B.
+Never stamp `trajectory_source` onto another plugin's `source`. Never emit ACP `session_update`.
+
+When the container does `import my_mech.trajectory`, package `__init__.py` must
+not import the Core-touching factory.
+
+## Image layers (docker bake)
+
+Not a timeline slot. The environment winner reads `config.image_layers` at `host.start()`.
+`executor:` alone does not bake. Context = installed plugin root (first-party
+contribs use the package next to the plugin). Pin wheels / npm at image build.
+No invoke-time `npm i` / floating pip. First-party ACP uses the same contract
+and bakes only the bound `options.entry` (pins from `acp_entries.json`).
+
+`${BASE_IMAGE}` is usually `ageval-attempt:base` (CPython 3.12).
+Declare `ARG PIP_INDEX_URL=` after `FROM` so a parent `AGEVAL_PIP_INDEX` is
+consumed. Empty means pip's default index. Do not `ENV` a blank `PIP_INDEX_URL`.
+A `RUN pip` should `unset PIP_INDEX_URL` when the ARG is empty.
+
+## Hook shape
+
+```python
+async def trajectory_collect(ctx, value, nxt):
+    out = await nxt(value)
+    return out
+```
+
+## Job binding (profiles, not task.yaml)
+
+```yaml
+format: ageval.profiles/1
+environment: docker
+agent_profiles:
+  solver:
+    executor: my-mech
+    extensions:
+      - plugin: my-mech
+        options:
+          agent: "lib.agents:MyAgent"
+          method: "run"
+    model: openai/glm-5.2
+    api_key: ${litellm_api_key}
+```
+
+`--set /bindings/<role>/options/<key>=…` writes the executor plugin row
+(ACP still rejects `command` / engine keys). YAML key is `agent_profiles`.
+
+## README contract
+
+Hub previews this file. Authors bind against it. Keep it in the same language
+as the rest of the package README. **No GitHub Issue numbers.**
+
+Two tables are required. Copy the columns; fill from this plugin's code, not
+from a sibling plugin.
+
+### Capabilities
+
+What this plugin **is** on the Attempt graph, and what it **needs**.
+
+| | Value |
+| --- | --- |
+| export | exclusive slot it wins, if any (`environment` / `executor` / …). Exclusive winners register that slot name as a service. |
+| inject | `service` + **capabilities** this plugin calls. Capability names are only those in `src/ageval/environments/protocol.py` `CAPABILITY_NAMES`: `exec`, `upload`, `download`, `attach_stdio`, `uid_gid`, `path_views`, `compose`. |
+| chain | chain slots it fills (`after_environment_ready`, `trajectory_collect`, …) |
+| bake | `config.image_layers` path, or omit the row |
+
+Environment winners also list the caps they **export** (the Protocol methods they actually implement). Declaring a cap the kind cannot deliver is a bug.
+
+Do not invent slot names. Do not write `inject: {plugin_id: e2b}`. Missing a declared cap fails at **lock**, not mid-invoke.
+
+### Parameters
+
+Every knob this plugin **reads**. One row per name. Columns:
+
+| Name | Default | Purpose |
+| --- | --- | --- |
+| `options.agent` | *(required)* | … |
+| `model` | `openai/gpt-4.1-mini` | … |
+
+**Where the name lives** (prefix it, do not dump a second table of sources):
+
+| Prefix | Who writes it | Who reads it |
+| --- | --- | --- |
+| `environment_options.<key>` | job `profiles.yaml` | environment exclusive winner |
+| `options.<key>` | profile `options` then this plugin's `extensions` row; last wins | executor factory / chain hook |
+| `model` / `base_url` / `api_key` | role profile (job axis) | executor factory kwargs. `api_key` is an **env locator name**, never the secret |
+| `E2B_API_KEY` (host env) | host environment | preflight / invoke locators this plugin itself looks up |
+
+Rules:
+
+- If the plugin reads it, the row exists. If it does not read it, do not list it.
+- Required knobs use `*(required)*` as Default. Omit / blank / `null` behaviour goes in Purpose.
+- Allowed values and rejected cases go in Purpose, not a prose dump above the table.
+- Rejected keys (ACP `command` / `engine_command` / …) get their own short table, or a Purpose note “rejected”.
+- `config.image_layers` is bake input, not a job parameter — list it under Capabilities, not here.
+- Do not document Core-owned fields (`environment:`, `executor:`) as this plugin's parameters.
+- Secrets stay locators. Never show a real key as a default.
+
+Template (executor):
+
+```markdown
+## Capabilities
+
+| | Value |
+| --- | --- |
+| export | exclusive `executor` |
+| inject | `environment`: `exec`, `upload` |
+| chain | `trajectory_collect` |
+| bake | `docker/Dockerfile.bake` |
+
+## Parameters
+
+| Name | Default | Purpose |
+| --- | --- | --- |
+| `options.agent` | *(required)* | Package-local `module:Class`. Missing → lock `extension_materialize_failed`. |
+| `options.method` | `run` | Method invoked on that class. |
+| `model` | `openai/gpt-4.1-mini` | Model id projected into the worker. |
+| `api_key` | `OPENAI_API_KEY` | Env **locator name**. Value never enters the lock. |
+| `base_url` | `OPENAI_BASE_URL` / `litellm_base_url` | OpenAI-compatible base. |
+```
+
+Template (environment):
+
+```markdown
+## Capabilities
+
+| | Value |
+| --- | --- |
+| export | exclusive `environment` |
+| capabilities | `exec`, `upload`, `download`, `attach_stdio`: yes. `uid_gid`, `path_views`, `compose`: no |
+| inject | — |
+
+## Parameters
+
+Job knobs are `environment_options` (not `extensions[].options`).
+
+| Name | Default | Purpose |
+| --- | --- | --- |
+| `environment_options.host` | *(required)* | … |
+```
+
+## Typed failures
+
+| Signal | Typical cause |
+| --- | --- |
+| `unknown_extension_slot` | `plugin.yaml` names a slot not in `slots.py` |
+| `extension_materialize_failed` | factory/options invalid (e.g. ACP missing `options.entry`) |
+| executor unbound | no in-environment bind on docker |
+| bake unsatisfied | bound external executor but no `image_layers` file |
+| `unsupported executor` | not installed |
+
+Recognition ≠ this host can run ≠ image baked.
+
+## Checklist
+
+```bash
+uv run ageval plugin install plugins/my-mech
+uv run ageval plugin list
+uv run ageval executors
+uv run ageval lock <dataset> --task <id> --profiles path/to/profiles.yaml
+uv run ageval run <dataset> --task <id> --profiles path/to/profiles.yaml --probe
+```
+
+README: Capabilities table matches `plugin.yaml` inject/slots and Protocol caps.
+Parameters table matches factory / host reads (name, default, purpose). No Issue numbers.

ageval_cli-0.7.2/.agents/skills/ageval-plugin/references/mechanism.md

@@ -0,0 +1,48 @@
+# Plugin mechanism
+
+Design authority: `docs/design/11-extension-plugins.md`. This note is for
+authoring agents only; it does not own new contract.
+
+## Registry
+
+- First-party contrib: `acp`, `openai-http`, `anthropic-http`, `local`, `docker`, `e2b`, `ssh`, `daytona`. No product mock executor.
+- External `ageval plugin install` joins the same table.
+- `profiles.executor` / `environment` pick exclusive winners. Chain slots are opt-in via `extensions`.
+- `extensions:` is the opt-in list (`- plugin: nooa`, or `slots: [...]`, or `{slot, plugin}`). Installed-but-unlisted plugins stay off MULTI chains and off the image.
+- Local path install and `ageval run` use the short `plugin.yaml` id. Hub publish/install uses `org/name`.
+- README is the public contract Hub previews: export / inject capabilities, and the parameters this plugin reads. `plugin.yaml` is the machine manifest; the README tables are what authors bind against.
+- Resolve: explicit binding > lower priority wins; a tie with no explicit pick is rejected.
+- `ageval lock` writes the resolved graph as `extension_bindings` (plugin_id / slot / source / priority / digest).
+
+Do not revive the deleted `ageval.agent_executors` entry-point bypass.
+
+## Slots
+
+Ids live in `src/ageval/plugins/slots.py`. Exclusive: `environment`, `executor`, `evaluation_runtime`, `trajectory_seal`. Chain: environment ready/setup, run bookends, agent open/invoke/close, evaluate bookends, `trajectory_collect` / `trajectory_enrich`, `summary_enrich`, `cleanup_report`.
+
+`evaluation_runtime` / `trajectory_seal` have engine defaults (`plugin_id: default`). No job-field sugar; replace only with an explicit `extensions` row. PASS still enters only through `bind_evaluation`.
+
+A public slot must have a production emit, or design / ARCHITECTURE must mark it non-public. No silent dead SPI.
+
+Agent emit chain:
+
+```text
+open_session → pin graph → before/after_agent_open
+invoke       → before_agent_invoke → executor.invoke → after_agent_invoke
+             → normalize_agent_result
+record       → trajectory_collect → enrich → trajectory_seal writes trajectory.jsonl
+             → summary_enrich (later steps still run if this hook fails; Attempt summary.extra)
+close        → before_agent_close → executor.close → after_agent_close
+```
+
+## Coexists with ACP
+
+The default coding-agent inlet remains first-party `executor: acp` + `- plugin: acp` / `options.entry`.
+An external plugin is an optional mechanism. The **same `run.py`** switches via
+profiles. ACP is not the trajectory schema authority.
+
+## Priority convention
+
+`DEFAULT_PRIORITY = 1000`. **Lower number runs first (chain) / wins first (exclusive).**
+First-party ACP is about `100`; the nooa example uses `110`. Explicit profile
+binding beats the number.

ageval_cli-0.7.2/.agents/skills/ageval-sdk-harness/references/antipatterns.md

@@ -0,0 +1,13 @@
+# run.py antipatterns
+
+| Antipattern | Do instead |
+| --- | --- |
+| Treat `completed` as PASS | Independent evaluator only |
+| Treat judge `invoke` text / `observation.jsonl` as PASS | Return `{status, score, metrics}`; engine binds |
+| `if executor == "codex":` / `if entry == "pi":` for Core policy | Switch `agent_profiles` / `active_profile` / `parameters.roles` |
+| Write `executor: codex` in package yaml | `executor: acp` + `- plugin: acp` / `options.entry: codex` (etc.) |
+| Read `~/.codex/auth.json` or print secrets | Rely on Runtime projection |
+| Write training JSON by hand under package | Use Core `trajectory.jsonl` under Result.logs |
+| Soft CallLimit as the only hard ceiling | `limits.agent_invocations` in yaml (Runtime) |
+| Import evaluation gold into `run.py` | Keep gold unmounted; upload after the run stops writing |
+| Branch on task_id / benchmark name in Core path | Mechanism-only adapters |