ageval-cli 0.7.1__tar.gz

ageval_cli-0.7.1/.agents/skills/ageval-cli/SKILL.md

@@ -0,0 +1,47 @@
+---
+name: ageval-cli
+description: >
+  Operate ageval CLI (lock/run/plugin/view/publish/release/executors/campaign/evidence/status/cancel/jobs/results):
+  flags, --set pointers, exit codes, --probe, offline fail-closed (AGEVAL_OFFLINE_AGENT),
+  ACP entry readiness, plugin install, suite upload. Use when running ageval lock/run, PASS/FAIL/ERROR,
+  trajectory export, Hub upload. Do not invent flags.
+---
+
+# ageval CLI
+
+```bash
+uv sync --frozen --all-packages
+uv run ageval --help
+```
+
+Public commands: `ageval lock` `ageval run` `ageval campaign` `tasks` `jobs` `view` `plugin` `evidence` `ageval status` `ageval cancel` `executors` `publish` `release` `login` `agent` `registry` `cache` `results`. No `submit`. Path arguments are **dataset** roots.
+
+```bash
+uv run ageval lock examples/core --task config-minimal
+uv run ageval run  examples/core --task acp-local-min
+uv run ageval run  examples/core --task acp-docker-min --profiles examples/core/profiles.docker.yaml
+uv run ageval run  examples/datasets/minimal-demo --task terminal-jsonl-agg
+uv run ageval run  official/demo@0.1.0 --dir tmp
+uv run ageval view official/demo@0.1.0
+uv run ageval run  … --probe          # lock + preflight, no Agent
+uv run ageval executors -v
+uv run ageval plugin install plugins/nooa
+```
+
+`--profiles` replaces the dataset job document. `--agent` and `--profiles` are mutually exclusive. `--agent pi` (and other shipped short ids) needs no `agent install`. `--model` requires `--agent` and overrides this run’s `binding.model` (not `ageval results --model`). `--set` allowlist: `/parameters/seed`, `/parameters/active_profile`, `/bindings/<role>/{model,executor,api_key,base_url,options/<key>}`. Not `limits.*`.
+
+TTY prints a short recap (no digest / sha256). A pipe keeps JSON. `ageval run --json` forces the run document.
+
+Exit: 0 PASS / probe ready; 1 FAIL / probe not ready; 2 ERROR.
+
+Single Attempt evidence: `<dataset>/.ageval/runs/<id>/` (`lock.json`, `result.json`, `trajectory.jsonl`; optional `evaluation/observation.jsonl` for evaluate-phase SDK invoke). Trajectory ≠ PASS.
+
+`--probe` missing `E2B_API_KEY` or ssh locator → ready false, started false.
+
+Offline: `AGEVAL_OFFLINE_AGENT=1` must not PASS agent tasks.
+
+Hub Leaderboard needs suite upload (`ageval results upload-suite`), not a lone `results upload`.
+
+Flags: `uv run ageval <cmd> --help`. Source: `src/ageval/cli/main.py`.
+
+Detail: [references/commands.md](references/commands.md), [references/failures.md](references/failures.md).

ageval_cli-0.7.1/.agents/skills/ageval-cli/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "ageval CLI"
+  short_description: "lock/run/evidence and exit codes"
+  default_prompt: "Use $ageval-cli to run the correct ageval lock/run/evidence commands and interpret Result.logs without inventing flags."

ageval_cli-0.7.1/.agents/skills/ageval-cli/references/commands.md

@@ -0,0 +1,108 @@
+# CLI command notes
+
+Box kind is `environment:` on the profiles document. `--probe` is `ageval run --probe` (lock + preflight). Dataset root. `submit` does not exist. Trust `uv run ageval --help`.
+
+## `ageval registry`
+
+- `list` / `show <id@version|@sha256:…>`: visible releases; `show` prints digest / size / visibility
+- Org ops (`org-create` / `org-add-member` / `org-set-role` / `org-transfer` / `org-remove-member`): owner or admin; the last owner cannot leave or be demoted
+- `set-visibility <id@version> --visibility public|private`: owner or admin
+- `set-description <org/name> --description '…'`: marketplace description override, no republish; empty string clears the override (falls back to the package manifest description). Server enforces org owner
+- `delete <id@version> --yes`: owner or admin
+
+## `ageval executors`
+
+Two product facts (+ ACP entry inventory):
+
+1. **What ageval supports** — `agent_profiles[].executor` kinds on this install
+2. **What the host can run** — binaries / ACP entries ready (no secrets)
+
+Stdout JSON (high level):
+
+| Key | Meaning |
+| --- | --- |
+| `supported` | Kind names valid for `agent_profiles[].executor` (e.g. `acp`, `openai-http`, `anthropic-http`) |
+| `host_ready` | Subset of kinds that can be constructed on this host |
+| `executors[]` | Per kind: `execution_mode` (from `describe()` when published), `host_ready` |
+| `acp_entries[]` | Per ACP `entry_id`: `acp_command`, `engine_ready`, `acp_entry_ready`, `host_ready`, credential env *names*; `-v` adds `credential_missing` / `keyless_auth` |
+
+- ACP registry: `ageval.plugins.contrib.acp.registry` (static pins)
+- Plugin `host_ready` uses declared `host_requires` / reachable `describe()`
+- Child env is allowlisted (`PATH` / `HOME` / `LANG` + entry credential names + binding locators + `fixed_env`). Undeclared host tokens do not reach the entry
+- No package path; no secrets; exit 0
+
+**Author packages with:** `executor: acp` + `- plugin: acp` / `options.entry: <entry_id from acp_entries>`.
+
+## `ageval lock`
+
+- Deterministic JSON on stdout (digest, `dataset_id`, task_id, resolution). No `database_id`.
+- No secret values.
+- Does not create Run/Attempt or start Agent.
+- Rejects unknown format (`invalid_format` at `/format`), unknown `executor` kinds, and ACP profiles missing `- plugin: acp` / `options.entry`.
+- `--probe` on **run** (not a second lock mode that starts a box): lock plus observational readiness. Exit non-zero when the selected `environment` path is unsatisfied. Checks declared `host_requires`, Docker daemon when kind is docker, locator **names** (never values). ACP entries add `credential_missing` (fail-closed when the entry requires a key; warning-only when `keyless_auth`). HTTP executors report `credential_missing` only when the locked `base_url` is not loopback. `AGEVAL_OFFLINE_AGENT=1` is reported; probe still does not spawn.
+
+## `ageval run`
+
+- `--probe` (requires `--task`): same feasibility report; does not start an Attempt.
+- TTY: short recap on stdout (per-task PASS/FAIL, `summary` / `logs`, `ageval view` next). Full document stays on disk. `--json` or a non-TTY stdout keeps one JSON object.
+- One foreground Attempt via production composition root **when** `--task` is set, `-k` defaults to 1, and no `--resume-suite`.
+- Evidence under dataset root `.ageval/runs/<id>/`. `logs` / `evidence_path` are portable relative to the dataset root.
+- **Always-k** (`--n-attempts` / `-k`, integer ≥1): fixed k independent Attempts per task. CLI/job only — not `task.yaml`.
+- **Suite**: omit `--task` → all members.
+- **`--max-concurrent-tasks`**: speeds wall time only; does not change k or PASS.
+- **`--resume-suite` / `--replace-slot` / `--attempt-index`**: suite resume; see `ageval run --help`.
+- **`--keep-workspace`**: after cleanup, do not delete the box work root (host residual may still be named `l1-work/` — historical directory name, not an isolation tier). Default off. Docker volumes are still removed. Upload packs never include `l1-work/**`.
+- **`--keep-vendor-raw`**: after a successful trajectory seal, keep invocation `backend_raw/` / layer B (`request.json`, `events.jsonl`, `final-response.json`, `metadata.json`) and `evaluation/evaluator_raw.json`. Default off: those files are dropped; Hub archives skip them even if residuals remain. `evaluation/observation.jsonl` (evaluate-phase SDK invoke) is **kept** either way. Independent of `--keep-workspace`.
+- `--profiles` replaces the dataset job document. `--agent` and `--profiles` are mutually exclusive. `--agent pi` resolves the shipped catalog tree (no install). `--model` requires `--agent` and writes this run’s overlay model; omit it to keep the package default. Not `ageval results --model`.
+- `--dir <path>`: only with a registry ref. Looks at `<path>/<dataset_id>/` (example: `--dir tmp` + `official/demo@0.1.0` → `tmp/official/demo`). Reuse that child if it already matches; otherwise fetch into it and run. Relative paths are from cwd. Local path + `--dir` is `invalid_override`.
+- Per invocation: Core writes `trajectory.jsonl` (`ageval.trajectory.event/1`) for **run-phase** Agent turns. Evaluate-phase SDK invoke (LLM-as-judge) writes `evaluation/observation.jsonl` instead. Neither file is PASS.
+- Docker kind uses the docker environment winner + `attach_stdio`; coding entries stay on the parent ACP client.
+
+## `ageval evidence`
+
+- Read-only export of **sealed** invocations.
+- Refuses unsealed (running) metadata.
+- Does not change evaluation score.
+
+## `ageval campaign`
+
+- Foreground serial matrix; allowlisted `/parameters/*` axes.
+- Not a parallel scheduler. Does not merge with `run.py` inner loops.
+
+## Control surface
+
+- `status` / `cancel` operate on ControlStore records.
+- Suite jobs may take `--dataset` to read `progress.json` or write `cancel.requested`.
+- There is no `submit`.
+
+## Always-k vs campaign
+
+- **Always-k** (`ageval run -k`): repeat independent Attempts for pass@k samples.
+- **Campaign** (`ageval campaign --matrix`): sweep allowlisted parameters / bindings on one task.
+
+## `ageval results upload` / `upload-suite`
+
+- Single Attempt upload does **not** create a suite Leaderboard row.
+- Prefer `upload-suite` after `ageval run <dataset>` **without** `--task`.
+- Dataset arg is a path or registry ref (`id@version`); a unique verified cache hit does not contact Hub to find the local suite-runs.
+- Hub Leaderboard **Public** lists complete, release-bound suites only.
+
+### Hub path checklist
+
+```text
+ageval publish <dataset> --org <org> --draft
+ageval release <org/dataset>
+ageval run <dataset> [--profiles …]          # omit --task → suite
+ageval results upload-suite <dataset> --suite-run <8-hex> --public [--with-attempts]
+```
+
+## `ageval publish` / `ageval release`
+
+- Draft slot overwrites for that `dataset_id`. Reserved version name `draft` is not a release.
+- `ageval release <dataset_id>` promotes the current draft.
+- Plugin packages do **not** use the draft slot.
+
+## `ageval view` / `ageval jobs delete`
+
+- Local dataset UI. Path or registry ref (`id@version` / `@sha256:…`). A unique verified cache hit does not contact Hub.
+- `jobs delete --local <dataset> --job <id> --yes` is local Job delete, not Registry `results delete`.

ageval_cli-0.7.1/.agents/skills/ageval-cli/references/failures.md

@@ -0,0 +1,23 @@
+# CLI failure diagnosis
+
+| Symptom | Check |
+| --- | --- |
+| exit 2, empty stdout on lock | Config error on stderr (`unknown_profile`, schema, `invalid_format` at `/format`) |
+| lock `unsupported_capability` / `unsupported executor` | Kind not in `ageval executors` `.supported` — coding agents need `executor: acp` |
+| lock `options.entry required` | ACP profile missing `- plugin: acp` / `options.entry` |
+| Agent ERROR offline | Expected under `AGEVAL_OFFLINE_AGENT=1` |
+| executor unbound on docker | Plugin did not bind to the box (`attach_stdio` / in-box worker). Do not silent host-fallback |
+| `image_contribute_unsatisfied` / missing bake | Bound external executor but `config.image_layers` / `Dockerfile.bake` empty — `$ageval-plugin` |
+| `nooa_package_missing` / `No module named 'nooa'` | Host needs `uv sync --extra nooa`; docker image bake installs it in-image |
+| ACP entry not ready | `ageval executors -v` → that `entry_id` `host_ready` / install pin; no invoke-time `npm i` |
+| `credential_missing` | Declared credential env names unset and no `api_key` locator. Required ACP entries fail at `--probe` / session-open; keyless (OAuth) warn only. HTTP executors (`openai-http` / `anthropic-http` / `dsh` / `nooa` / `miniswe`) skip this when locked `base_url` host is `127.0.0.1` / `localhost` / `::1` |
+| e2b / ssh probe not ready | Missing `E2B_API_KEY` or SSH locator — fail-closed, `started: false`. Skip ≠ pass |
+| PASS without real model | Forbidden — do not use fixtures as public proof |
+| Trajectory empty | Non-empty `agent_profiles` + `run.py` `Agent.session`/`invoke` |
+| Resume skipped an ERROR | Default `--resume-suite` skips finished PASS / FAIL / ERROR. Use `--replace-slot --task T` |
+| Export fails `unsealed_invocation` | Attempt still running or metadata not terminal |
+| Export fails `secret_residual` | Fix source evidence; do not strip secrets by hand in export dir |
+| Docker ERROR | Docker daemon, image build, network/creds projection; read result / agent meta under `.ageval/runs/` |
+| Invoke / wall / verifier timeout | FAIL (`metrics.reason=timeout`), exit 1 — not ERROR. Environment start failure stays ERROR |
+
+Design: `docs/design/05-runtime/evidence.md`, `docs/design/07-budget-evaluation-failure.md`.

ageval_cli-0.7.1/.agents/skills/ageval-config-package/SKILL.md

@@ -0,0 +1,71 @@
+---
+name: ageval-config-package
+description: >
+  Author ageval datasets (ageval.yaml + tasks/*/task.yaml, run.py/evaluator.py,
+  profiles.yaml, environment kinds local|docker|e2b|ssh|daytona, limits, gold isolation,
+  script vs LLM-as-judge scoring). Triggers: ageval.yaml, task.yaml, profiles.yaml,
+  dataset, environment kind, evaluator.py. Never secrets in yaml. Never provider.kind.
+---
+
+# Config / dataset + task
+
+Config Core is the only normative reader. Delivery unit is a **dataset** root.
+
+```text
+my-dataset/                 # CLI path (ageval.dataset/1)
+├── ageval.yaml
+├── profiles.yaml           # job: environment + agent_profiles
+├── env.example
+├── shared/lib/             # optional; import shared.lib.*
+└── tasks/<task_id>/
+    ├── task.yaml           # ageval.task/1 — roles + intent
+    ├── run.py              # async def run(ctx)
+    ├── evaluator.py
+    ├── environment/        # Dockerfile, setup.sh, compose
+    ├── evaluation/         # gold — not for Agent
+    └── data/               # seed
+```
+
+CLI: `ageval lock|run <dataset-root> --task <id>`.
+
+Files present are recognized. Do not re-declare `run.py` entrypoints in yaml. Do not put `executor` / `api_key` on the member task. Do not write `provider.kind` or `assurance`.
+
+```yaml
+# ageval.yaml
+format: ageval.dataset/1
+dataset_id: org/my-suite
+version: "0.1.0"
+tasks:
+  root: tasks
+```
+
+```yaml
+# profiles.yaml
+format: ageval.profiles/1
+environment: docker
+agent_profiles:
+  solver:
+    executor: acp
+    api_key: ${ZHIPU_API_KEY}
+    options: { entry: pi }
+    extensions:
+      - plugin: acp
+      - plugin: docker
+```
+
+Gold lives in `evaluation/` and is uploaded at evaluate, not before. `setup.sh` is the last environment slot.
+
+Two scoring styles in `evaluator.py` (same barrier; PASS still the return value):
+
+| Style | When | Evidence |
+| --- | --- | --- |
+| Deterministic script | Default. Compare artifacts to gold. No `Agent.session`. | `result.json`. No `evaluation/observation.jsonl`. |
+| LLM-as-judge | Extra role on the **same** `profiles.yaml` (e.g. `judge`). After gold lands, `Agent.session("judge").invoke`. | `evaluation/observation.jsonl` (no `user` rows). Not merged into Agent `trajectory.jsonl`. |
+
+Judge binding stays in `profiles.yaml` (not on the member task). Budget `limits.agent_invocations` for solver **and** judge. SDK session into the evaluator process is projected on `environment: local`. No `Agent.session` = script path.
+
+`--profiles` swaps the job. `--agent` is mutually exclusive with `--profiles`. `--model` requires `--agent`.
+
+Hub comparability still uses suite fingerprint / `upload-suite`. Do not add `n_attempts` to yaml.
+
+Field catalog: [references/ageval-yaml.md](references/ageval-yaml.md). Isolation: [references/isolation.md](references/isolation.md). Conversion: [references/conversion.md](references/conversion.md).

ageval_cli-0.7.1/.agents/skills/ageval-config-package/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "ageval Config Package"
+  short_description: "ageval.yaml and package layout rules"
+  default_prompt: "Use $ageval-config-package to author or review a Task Package ageval.yaml without secrets or ownership violations."

ageval_cli-0.7.1/.agents/skills/ageval-config-package/references/ageval-yaml.md

@@ -0,0 +1,15 @@
+# Field catalog
+
+Formats: `ageval.dataset/1`, `ageval.task/1`, `ageval.profiles/1`. Unknown format → `invalid_format` at `/format`.
+
+Dataset root keys: `format`, `dataset_id`, `version`, `description`, `tasks`. No `database_id`.
+
+Task: `format`, `task_id`, `parameters`, `agent_profiles` (role ids only), `limits`, `artifacts`, `evaluation`, `provenance`, `requires`. Files present (`run.py`, `evaluator.py`, Dockerfile, `setup.sh`) are recognized. No `harness:` block, no `provider.kind`, no `assurance`.
+
+Profiles: `format`, `environment`, `environment_options`, `agent_profiles`. Role rows may have `executor`, `model`, `api_key` (locator), `options.entry`, `extensions`.
+
+Plugin knobs (`environment_options.*`, `extensions[].options`, role `model` / `api_key` / `base_url`) live in **that plugin's README** Parameters table (name, default, purpose). Capabilities (export / inject) are the other required table. Do not copy keys from a sibling plugin.
+
+`api_key: ${ENV}` is a locator name. Value never enters lock.
+
+`--set` allowlist lives in Config Core, not here.

ageval_cli-0.7.1/.agents/skills/ageval-config-package/references/conversion.md

@@ -0,0 +1,76 @@
+# Conversion & multi-task package authoring
+
+Author-facing patterns for porting upstream benchmarks into an ageval **dataset**.
+Authority: `docs/design/08-conversion-security-testing.md`, `docs/design/02-task-package-and-config.md`.
+
+## When to use a generator
+
+| Signal | Action |
+| --- | --- |
+| ≥ ~5 isomorphic members (same `run.py` / eval shape) | Prefer `scripts/generate_package.py` (or equivalent) |
+| Members differ only by scenario id / params / data | Thin `task.yaml` + `data/` / `evaluation/` per task; shared orchestration |
+| One-off demo task | Inline `run.py` is fine; still respect gold isolation |
+
+### Generator shape (reference)
+
+1. Read upstream assets (often under `shared/assets/` or an external path).
+2. Emit `tasks/<task_id>/task.yaml` with `provenance` filled.
+3. Emit thin `run.py` / `evaluator.py` that import **`shared.lib.*`**.
+4. Write per-task `data/` (agent-visible) and `evaluation/` (gold) as needed.
+5. Stay idempotent: re-run regenerates members without hand-editing fifty files.
+
+Do not write `provider.kind` / `assurance`. Generate `run.py`.
+
+### Import migration
+
+| Old (leaf inject) | New (dataset root on path) |
+| --- | --- |
+| `from harness_core import run` | `from shared.lib.harness_core import run` |
+| `from bridge import make_environment` | `from shared.lib.bridge import make_environment` |
+| `from evaluator_core import evaluate` | `from shared.lib.evaluator_core import evaluate` |
+
+Shared glue lives under `shared/lib`; the task entry is `run.py`.
+
+## Upstream → ageval owner map
+
+| Upstream concept | Put under | Consumer | Never |
+| --- | --- | --- | --- |
+| Instruction / prompt / agent-visible files | `tasks/<id>/data/` | Agent seed / `ctx.params` | Gold under `data/` |
+| Hidden tests / labels / expected answers | `tasks/<id>/evaluation/` | Evaluator only | `shared/`, Agent mount |
+| Container image / apt / system deps | `tasks/<id>/environment/Dockerfile` | docker / e2b `host.start` | Runtime `apt`/`pip` as parity default |
+| Domain tools / bridge / dual-agent loop | `shared/lib/` (multi-task) or `tasks/<id>/lib/` | `run.py` + evaluator import | Gold files |
+| Shared policies / DB dumps / static assets | `shared/assets/` | Code paths | Agent default mount of gold |
+| Offline known-good workspace files | `tasks/<id>/solution/` | Human / CI / offline flag only | Default Agent seed |
+| Job binding (executor / entry / model) | dataset `profiles.yaml` | Runtime / job overlay | Secrets in yaml |
+
+## Scenario cut
+
+- One upstream scenario / one stable role topology → one dataset (`ageval.yaml`).
+- Vary task business parameters and `data/` / `evaluation/` per member, not role topology.
+- Different scenarios → separate datasets.
+
+## Thin multi-task template
+
+```text
+shared/
+  lib/
+    run_core.py          # async run(ctx, *, task_dir)
+    evaluator_core.py
+    bridge.py
+tasks/task-00/
+  run.py                 # from shared.lib.run_core import run as _run
+  evaluator.py
+  data/ … evaluation/
+```
+
+## Official base `ageval-attempt:base`
+
+When the port `FROM`s the official tag (historical name, CPython 3.12) instead of the vendor instance image, re-check every migrated `RUN pip` / `apt` against this base. A green `docker build` that never imports the hidden tests still ERROR at collection.
+
+Details: `references/isolation.md`.
+
+## Evidence / claims
+
+- Conversion completeness, Hub package publish, or `ageval lock` success **do not** upgrade evidence grade.
+- PASS only from the independent evaluator after barrier.
+- Fill `provenance` for ports; known gaps stay honest.

ageval_cli-0.7.1/.agents/skills/ageval-config-package/references/isolation.md

@@ -0,0 +1,73 @@
+# Isolation notes for package authors
+
+Authority: `docs/design/06-capability-adapter-visibility.md`, `docs/design/05-runtime/environment.md`.
+
+## Gold / evaluation material
+
+- Put hidden labels under `evaluation/` (paths never mounted to Agent).
+- Gold is uploaded at the start of evaluate, not during run. This is a cut in **time**, not `path_views`.
+- Evaluator receives allowlisted staged inputs only after writer barrier.
+- Optional LLM-as-judge: `Agent.session` is allowed **after** that upload. Solver profiles that already ran must not invoke. Judge prompts stay out of `evaluation/observation.jsonl` user rows.
+
+## Box kind
+
+- Job `environment: local | docker | e2b | ssh | daytona`.
+- docker / e2b share `environment/Dockerfile` (or an image option). Official base:
+  `FROM ageval-attempt:base` (`docker/attempt/`). Secrets never baked into layers.
+- Coding agents: `executor: acp` + `options.entry`; parent ACP + `host.attach_stdio`.
+- `setup.sh` is the last environment slot (`environment_setup`). No provision phase.
+
+## Dockerfile depth (conversion)
+
+| Tier | When | Pattern |
+| --- | --- | --- |
+| **Thin** | Official base already has engines + ACP entries | `FROM ageval-attempt:base` |
+| **Heavy** | Extra apt/pip/lang | Same base **+ migrated `RUN`** at **build** time |
+
+**Forbid as parity default:** runtime `apt-get` / floating `pip install` / `npm i` / `npx`
+inside the Attempt after start. Bake deps in the image.
+
+### Official base is not the upstream runtime
+
+The official Attempt image is **CPython 3.12 + baked ACP entries**. It is **not**
+the upstream bench’s Python or pinned wheels. If the package Dockerfile `FROM`s this base:
+
+1. **Do not** copy upstream `pip install foo` unpinned.
+2. **Pin to what the checkout imports.**
+3. **Gate at image build**, not first eval: `python -c "from …"` so a bad pin fails `docker build`.
+4. Collection / conftest ImportError is eval ERROR, not FAIL.
+5. `host_requires` / `--probe` do not catch in-image eval deps — those are the package Dockerfile’s job.
+
+### `data/` seed
+
+Put agent-visible files under `tasks/<id>/data/`. The environment phase uploads them to `/attempt/workspace`. Prefer this over baking content into image layers.
+
+### `solution/` (offline fixture)
+
+- Default: **not** seeded to Agent.
+- Offline helper flags may copy `solution/*` into workspace — author-level only, not production default.
+
+## Dataset `shared/`
+
+- Import contract: Runtime injects **`[task_dir, dataset_root]`** — **not** the `shared/lib` leaf. Authors write `from shared.lib…`.
+- `shared/lib` is for `run.py` / evaluator import only — **not** default Agent mount.
+- Gold stays under `tasks/*/evaluation/` only; **forbid** gold / `.env` under `shared/`.
+- Task members must **not** own top-level name `shared`.
+- docker images: **no** Core implicit COPY of `shared/`. The same `evaluator.py` with `from shared.lib…` can pass on `environment: local` and ImportError on docker unless the Dockerfile `COPY`s `shared/` and puts the **dataset root** on `PYTHONPATH`.
+
+```dockerfile
+FROM ageval-attempt:base
+COPY shared/ /attempt/shared/
+ENV PYTHONPATH=/attempt:/attempt/task:${PYTHONPATH}
+```
+
+Do **not** `COPY` `tasks/*/evaluation/` gold into Agent-visible layers.
+
+## Do not
+
+- Rely on “delete field from yaml” as isolation.
+- Mount gold into `run.py` “for convenience”.
+- Put credentials in the package tree.
+- Expect Core to auto-COPY `shared/` into docker images.
+- Use runtime package installs as the default parity path.
+- Treat `FROM ageval-attempt:base` + unpinned `pip install` as equivalent to the upstream instance image.

ageval_cli-0.7.1/.agents/skills/ageval-design-system/SKILL.md

@@ -0,0 +1,106 @@
+---
+name: ageval-design-system
+description: >
+  ageval web UI style rules for coding agents editing website (landing + docs),
+  apps/hub, or apps/viewer: canonical color tokens (cool paper/ink + IKB
+  #1B54E8/#5B7BFF), font stacks (Geist/Anton wordmark-only), radius,
+  role-based focus (scan chrome keeps hairline; edit fields use IKB border),
+  reuse of shipped components, the ten invariants, and the machine check
+  scripts/check_design_tokens.py. Use when adding or restyling UI in any of the
+  three web surfaces, picking colors or fonts, adding icons, or when a change
+  touches raw hex, focus rings, buttons, search fields, or the owl brand assets.
+  Authority is docs/design/13-web-ui-tokens.md; apps/viewer/DESIGN.md YAML is
+  the SPA token listing (Hub inherits it); SPA DESIGN.md Taste is anti-slop
+  (product chrome, not landing) plus the reuse map.
+  This skill only routes and summarizes — it is not a page inventory.
+---
+
+# ageval web UI design system
+
+Three surfaces, one language: website landing, website docs (fumadocs),
+hub SPA, viewer SPA. Authority: `docs/design/13-web-ui-tokens.md`
+(constitution: tokens, focus roles, motion — **not** a page inventory).
+
+On Hub / Viewer, also read `apps/viewer/DESIGN.md` YAML (shared theme constants),
+that SPA's `DESIGN.md` **Taste** (anti-slop) and role table, and `AGENTS.md`
+(scope). Copy a shipped instance. Do not restyle a shadcn primitive default
+and call it on-brand.
+
+Hub and Viewer are **product consoles**. Do not apply landing/portfolio
+playbooks (`frontend-design`, `design-taste-frontend` hero / bento / GSAP /
+magnetic / "invent a new identity"). Identity is already locked: cool paper
++ IKB + Geist. Spend the skill budget on not looking like default shadcn.
+
+## Where tokens live
+
+| Surface      | Token file                                                              |
+| ------------ | ----------------------------------------------------------------------- |
+| constitution | `docs/design/13-web-ui-tokens.md`                                       |
+| SPA listing  | `apps/viewer/DESIGN.md` YAML (Hub inherits; do not fork)                |
+| hub / viewer | `apps/{hub,viewer}/src/index.css` (`--viewer-*` + `@theme`)             |
+| docs         | `website/src/app/global.css` (`--color-fd-*`, `--ageval-link*`)           |
+| landing      | Token block at the top of `website/src/components/landing/landing.css`  |
+
+Change order: edit the table in `docs/design/13` first → sync script `CANONICAL`
+→ sync `apps/viewer/DESIGN.md` YAML and the surface CSS → run the machine check.
+If they disagree, fix the copies to match docs/13 + the script.
+
+## Quick rules (full list in docs/design/13)
+
+- Hex values live only in token files and owl brand assets. App code uses semantic
+  names (`text-ink`, `border-hairline`, …).
+- IKB (`link` / `primary`) is for links, focus, primary CTAs, and brand slots.
+  Do not use it as a large fill. Functional accents (error, warning, star gold)
+  have their own tokens; the UI is not ink/paper/IKB-only.
+- `mute` is never body text. `canvas*` is surface, `hairline` is line,
+  `ink` / `body` / `mute` are type.
+- Anton is wordmark-only. Body is Geist (+ CJK fallback). Mono is Geist Mono.
+- Radii are 8 / 10 / 14px only. Primary CTA (SPA Button `default`) is IKB fill +
+  `rounded-[8px]` + mono 13px + pop shadow + `focus-visible:ring-2 ring-link/70`.
+  Search is stadium. Do not use clip-path chamfer on buttons.
+- Section tabs use `UnderlineTabs` (sans `text-sm`, liquid-gooey Move thumb, fill
+  `canvas-soft-2`). Do not draw an IKB underline. Page heads use `PageHead`
+  (h1 + optional sub + hairline; no numbered kicker). One tab strip per view; a
+  second exclusive choice is `Select`. Wrapping chips use `Chip`, not `bg-link/10`.
+- Operator-facing controls and table column labels use body-sm 14px. Caption is
+  timestamps and mute meta, not column names. Do not invent a smaller clickable size. A button group is one hairline box; selected fill
+  is `canvas-soft-2`. New chrome joins the toolbar already on the page — do not open a
+  vacant band for a single control. Layout: SPA DESIGN.md **Composition**.
+- List / table row text (non-numeric): always default sans. Mono only for
+  `<code>` / `<pre>` / command strips, and numeric `tabular-nums` alignment.
+- Plugin / agent marketplace lists are `CatalogCard` grids. Datasets are
+  org-grouped hairline tables; jobs, leaderboard, and members are flat
+  hairline tables. Do not put slot/binding tags
+  on cards.
+- Motion is CSS only on hub/viewer, plus two named exceptions in docs/13:
+  `ThinkingLogo` canvas and `liquid-gooey` Move (tabs / Hub sidebar). Default
+  `200ms` / `ease-smooth`. Also `--ease-spring` (toast, star burst, squish
+  release), `--ease-glide` (thumb CSS), `--t-press` 80ms. Morph does not ship.
+  Website docs/landing do not take the library. Landing may stagger the hero and
+  reveal sections 8px; no GSAP/Motion, pin/scrub, magnetic hover, or cursor trail.
+- Focus is role-based (docs/13), not "every field goes IKB":
+  buttons / links / cards `ring-2 ring-link/70`; **edit** fields 1px
+  `border-link`; **scan** chrome (search, filter, Select trigger) keeps
+  `border-hairline` — no border color change. Landing 3px outline.
+  Hub search copies `CatalogScopeBar`, not `Input`'s default.
+- Popover / card shadow is `--viewer-shadow-pop` only (tight pop; no
+  `--viewer-shadow-liquid`). Phase charts use `--viewer-phase-1..6` only.
+  Hub shell is left/right (`canvas-soft` aside, `canvas` main). Viewer header is
+  `canvas-soft`; docs sidebar matches the aside.
+- Icons: product brand uses owl; function icons use lucide; plugin/agent entity
+  marks default to the uploader GitHub avatar, or a closed color catalog /
+  GitHub login override. Do not add a third-party logo component library as a
+  runtime dependency. Catalog hex lives only in `brand-marks/assets/`.
+- SPA anti-slop lives in that SPA's `DESIGN.md` **Taste**. Do not copy it here.
+
+## Machine check
+
+```bash
+python3 scripts/check_design_tokens.py
+```
+
+Checks: docs/13 table ↔ script `CANONICAL` ↔ Viewer DESIGN.md YAML; mapped
+variables stay inside the canonical set; no raw hex in app code (outside the
+allowlist).
+CI job `design-tokens` runs the same command. Run it locally after token or
+style edits.

ageval_cli-0.7.1/.agents/skills/ageval-platform/SKILL.md

@@ -0,0 +1,52 @@
+---
+name: ageval-platform
+description: >
+  ageval platform map: authority (docs/design vs ARCHITECTURE vs Issues), Core vs
+  dataset ownership, red lines (trajectory≠PASS, no secrets, mechanism-named
+  adapters), evidence grades, which sibling skill to load. Use when entering the
+  repo, asking "what is ageval", "who owns PASS", "read order", or "evidence grade".
+  Not a substitute for docs/design.
+---
+
+# ageval platform
+
+Runtime owns lock, Attempt, the box, hard ceilings, trajectory, independent evaluation. The dataset owns the business loop in `run.py`.
+
+## Read order
+
+1. `docs/design/` — product + mechanism (self-contained; no external BRIEF)
+2. `ARCHITECTURE.md` — modules
+3. GitHub Issues — delivery
+4. Code / tests / examples — what is shipped
+
+`website/` is reader-facing. Conflict → fix `docs/` first.
+
+## Ownership
+
+| Owns | Does not own |
+| --- | --- |
+| **Core:** lock, Attempt phases, box Protocol, Agent Service, hard ceilings, bind PASS | Task loop, scoring algorithm |
+| **`run.py`:** loop, tools, publish | PASS, credentials, opening the box |
+| **evaluator:** truth (script, or optional `Agent.session` after gold) | Opening the box; binding PASS from trajectory |
+
+## Red lines
+
+1. Trajectory ≠ PASS (`trajectory.jsonl` and `evaluation/observation.jsonl`).
+2. No secrets in lock / yaml / evidence. Env vars are locators.
+3. Adapters named by mechanism (`acp`, `docker`, `e2b`, `ssh`). Never by benchmark.
+4. Kind is `environment: local|docker|e2b|ssh|daytona`.
+5. Unknown format → one error.
+6. Coding agents: `executor: acp` + `options.entry`. Not `executor: pi`.
+7. Plugins: exclusive / chain slots only. `ageval plugin install` never rewrites profiles.
+8. Do not claim `isolated` / `real-benchmark-verified` from one happy path.
+
+## Skills
+
+| Task | Load |
+| --- | --- |
+| CLI | `$ageval-cli` |
+| `ageval.yaml` / task layout | `$ageval-config-package` |
+| `run.py` / AgentSession | `$ageval-sdk-harness` |
+| `ageval.plugin/1` | `$ageval-plugin` |
+
+Detail: [references/authority.md](references/authority.md), [references/red-lines.md](references/red-lines.md), [references/shipped-surfaces.md](references/shipped-surfaces.md).

ageval_cli-0.7.1/.agents/skills/ageval-platform/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "ageval Platform"
+  short_description: "Authority, ownership, red lines map"
+  default_prompt: "Use $ageval-platform to orient on ageval authority order, Core vs package ownership, and red lines before changing this repo."

ageval_cli-0.7.1/.agents/skills/ageval-platform/references/authority.md

@@ -0,0 +1,27 @@
+# Authority & conflict resolution
+
+## Chain
+
+```text
+docs/design/*  →  ARCHITECTURE.md  →  GitHub Issues  →  code/tests/examples
+```
+
+`docs/` is self-contained. Do not read an external BRIEF / vault as design truth.
+
+`ARCHITECTURE.md` 拥有 Current 源码树、依赖图、五相位状态机、emit 图、数据流表、失败归属。施工红线与 CI 命令在根 `AGENTS.md`。不要在 skill 里复制那两份图。
+
+`website/` is reader-facing product documentation only — never design truth.  
+`apps/*` and `services/*` READMEs own SPA/service **dev** detail, not product tutorials.
+
+## Conflict handling
+
+1. Stop the conflicting implementation.
+2. Classify: design change | structure change | delivery scope | implementation drift.
+3. Edit the **highest** authority first.
+4. Sync README / AGENTS / website / tests in the same change when claims change.
+
+## What skills are not
+
+- Not a second design doc.
+- Not a license to claim unfinished surfaces as available.
+- Not allowed to invent `ageval` subcommands not present in `src/ageval/cli/main.py`.