ageval-cli 0.7.3__tar.gz → 0.8.1__tar.gz

ageval_cli-0.8.1/.agents/skills/ageval-cli/SKILL.md

@@ -0,0 +1,47 @@
+---
+name: ageval-cli
+description: >
+  Operate ageval CLI (lock/run/plugin/view/publish/release/executors/campaign/evidence/status/cancel/jobs/results):
+  flags, --set pointers, exit codes, --probe, offline agent path (AGEVAL_OFFLINE_AGENT),
+  ACP entry readiness, plugin install, suite upload. Use when running ageval lock/run, PASS/FAIL/ERROR,
+  trajectory export, Hub upload. Do not invent flags.
+---
+
+# ageval CLI
+
+```bash
+uv sync --frozen --all-packages
+uv run ageval --help
+```
+
+Public commands: `ageval lock` `ageval run` `ageval campaign` `tasks` `jobs` `view` `plugin` `evidence` `ageval status` `ageval cancel` `executors` `publish` `release` `login` `agent` `registry` `cache` `results`. No `submit`. Path arguments are **dataset** roots.
+
+```bash
+uv run ageval lock examples/core --task config-minimal
+uv run ageval run  examples/core --task acp-local-min
+uv run ageval run  examples/core --task acp-docker-min --profiles examples/core/profiles.docker.yaml
+uv run ageval run  examples/datasets/minimal-demo --task terminal-jsonl-agg
+uv run ageval run  official/demo@0.1.0 --dir tmp
+uv run ageval view official/demo@0.1.0
+uv run ageval run  … --probe          # lock + preflight, no Agent
+uv run ageval executors -v
+uv run ageval plugin install plugins/nooa
+```
+
+`--profiles` replaces the dataset job document. `--agent` and `--profiles` are mutually exclusive. `--agent pi` (and other shipped short ids) needs no `agent install`. `--model` requires `--agent` and overrides this run’s `binding.model` (not `ageval results --model`). `--set` allowlist: `/parameters/seed`, `/parameters/active_profile`, `/bindings/<role>/{model,executor,api_key,base_url,options/<key>}`. Not `limits.*`.
+
+TTY prints a short recap (no digest / sha256). A pipe keeps JSON. `ageval run --json` forces the run document.
+
+Exit: 0 PASS / probe ready; 1 FAIL / probe not ready; 2 ERROR.
+
+Single Attempt evidence: `<dataset>/.ageval/runs/<id>/` (`lock.json`, `result.json`, `trajectory.jsonl`; optional `evaluation/observation.jsonl` for evaluate-phase SDK invoke; optional `evaluation/checks.json` for evaluator `checks`).
+
+`--probe` missing `E2B_API_KEY` or ssh locator → ready false, started false.
+
+Offline: `AGEVAL_OFFLINE_AGENT=1` must not PASS agent tasks.
+
+Hub Leaderboard needs suite upload (`ageval results upload-suite`), not a lone `results upload`.
+
+Flags: `uv run ageval <cmd> --help`. Source: `src/ageval/cli/main.py`.
+
+Detail: [references/commands.md](references/commands.md), [references/failures.md](references/failures.md).

ageval_cli-0.8.1/.agents/skills/ageval-cli/references/commands.md

@@ -0,0 +1,108 @@
+# CLI command notes
+
+Box kind is `environment:` on the profiles document. `--probe` is `ageval run --probe` (lock + preflight). Dataset root. `submit` does not exist. Trust `uv run ageval --help`.
+
+## `ageval registry`
+
+- `list` / `show <id@version|@sha256:…>`: visible releases; `show` prints digest / size / visibility
+- Org ops (`org-create` / `org-add-member` / `org-set-role` / `org-transfer` / `org-remove-member`): owner or admin; the last owner cannot leave or be demoted
+- `set-visibility <id@version> --visibility public|private`: owner or admin
+- `set-description <org/name> --description '…'`: marketplace description override, no republish; empty string clears the override (falls back to the package manifest description). Server enforces org owner
+- `delete <id@version> --yes`: owner or admin
+
+## `ageval executors`
+
+Two product facts (+ ACP entry inventory):
+
+1. **What ageval supports** — `agent_profiles[].executor` kinds on this install
+2. **What the host can run** — binaries / ACP entries ready (no secrets)
+
+Stdout JSON (high level):
+
+| Key | Meaning |
+| --- | --- |
+| `supported` | Kind names valid for `agent_profiles[].executor` (e.g. `acp`, `openai-http`, `anthropic-http`) |
+| `host_ready` | Subset of kinds that can be constructed on this host |
+| `executors[]` | Per kind: `execution_mode` (from `describe()` when published), `host_ready` |
+| `acp_entries[]` | Per ACP `entry_id`: `acp_command`, `engine_ready`, `acp_entry_ready`, `host_ready`, credential env *names*; `-v` adds `credential_missing` / `keyless_auth` |
+
+- ACP registry: `ageval.plugins.contrib.acp.registry` (static pins)
+- Plugin `host_ready` uses declared `host_requires` / reachable `describe()`
+- Child env is allowlisted (`PATH` / `HOME` / `LANG` + entry credential names + binding locators + `fixed_env`). Undeclared host tokens do not reach the entry
+- No package path; no secrets; exit 0
+
+**Author packages with:** `executor: acp` + `- plugin: acp` / `options.entry: <entry_id from acp_entries>`.
+
+## `ageval lock`
+
+- Deterministic JSON on stdout (digest, `dataset_id`, task_id, resolution). No `database_id`.
+- No secret values.
+- Does not create Run/Attempt or start Agent.
+- Rejects unknown format (`invalid_format` at `/format`), unknown `executor` kinds, and ACP profiles missing `- plugin: acp` / `options.entry`.
+- `--probe` on **run** (not a second lock mode that starts an environment): lock plus observational readiness. Exit non-zero when the selected `environment` path is unsatisfied. Checks declared `host_requires`, Docker daemon when kind is docker, locator **names** (never values). ACP entries add `credential_missing` (if the entry requires a key, the check fails and the run does not start; warning-only when `keyless_auth`). HTTP executors report `credential_missing` only when the locked `base_url` is not loopback. `AGEVAL_OFFLINE_AGENT=1` is reported; probe still does not spawn.
+
+## `ageval run`
+
+- `--probe` (requires `--task`): same feasibility report; does not start an Attempt.
+- TTY: short recap on stdout (per-task PASS/FAIL, `summary` / `logs`, `ageval view` next). Full document stays on disk. `--json` or a non-TTY stdout keeps one JSON object.
+- One foreground Attempt via production composition root **when** `--task` is set, `-k` defaults to 1, and no `--resume-suite`.
+- Evidence under dataset root `.ageval/runs/<id>/`. `logs` / `evidence_path` are portable relative to the dataset root.
+- **Always-k** (`--n-attempts` / `-k`, integer ≥1): fixed k independent Attempts per task. CLI/job only — not `task.yaml`.
+- **Suite**: omit `--task` → all members.
+- **`--max-concurrent-tasks`**: speeds wall time only; does not change k or PASS.
+- **`--resume-suite` / `--replace-slot` / `--attempt-index`**: suite resume; see `ageval run --help`.
+- **`--keep-workspace`**: after cleanup, do not delete the box work root (host residual may still be named `l1-work/` — historical directory name, not an isolation tier). Default off. Docker volumes are still removed. Upload packs never include `l1-work/**`.
+- **`--keep-vendor-raw`**: after a successful trajectory seal, keep invocation `backend_raw/` / layer B (`request.json`, `events.jsonl`, `final-response.json`, `metadata.json`) and `evaluation/evaluator_raw.json`. Default off: those files are dropped; Hub archives skip them even if residuals remain. `evaluation/observation.jsonl` (evaluate-phase SDK invoke) and `evaluation/checks.json` (evaluator `checks`) are **kept** either way. Independent of `--keep-workspace`.
+- `--profiles` replaces the dataset job document. `--agent` and `--profiles` are mutually exclusive. `--agent pi` resolves the shipped catalog tree (no install). `--model` requires `--agent` and writes this run’s overlay model; omit it to keep the package default. Not `ageval results --model`.
+- `--dir <path>`: only with a registry ref. Looks at `<path>/<dataset_id>/` (example: `--dir tmp` + `official/demo@0.1.0` → `tmp/official/demo`). Reuse that child if it already matches; otherwise fetch into it and run. Relative paths are from cwd. Local path + `--dir` is `invalid_override`.
+- Per invocation: Core writes `trajectory.jsonl` (`ageval.trajectory.event/1`) for **run-phase** Agent turns. Evaluate-phase SDK invoke (LLM-as-judge) writes `evaluation/observation.jsonl` instead. Neither file is PASS.
+- Docker kind uses the docker environment winner + `attach_stdio`; coding entries stay on the parent ACP client.
+
+## `ageval evidence`
+
+- Read-only export of **sealed** invocations.
+- Refuses unsealed (running) metadata.
+- Does not change evaluation score.
+
+## `ageval campaign`
+
+- Foreground serial matrix; allowlisted `/parameters/*` axes.
+- Not a parallel scheduler. Does not merge with `run.py` inner loops.
+
+## Control surface
+
+- `status` / `cancel` operate on ControlStore records.
+- Suite jobs may take `--dataset` to read `progress.json` or write `cancel.requested`.
+- There is no `submit`.
+
+## Always-k vs campaign
+
+- **Always-k** (`ageval run -k`): repeat independent Attempts for pass@k samples.
+- **Campaign** (`ageval campaign --matrix`): sweep allowlisted parameters / bindings on one task.
+
+## `ageval results upload` / `upload-suite`
+
+- Single Attempt upload does **not** create a suite Leaderboard row.
+- Prefer `upload-suite` after `ageval run <dataset>` **without** `--task`.
+- Dataset arg is a path or registry ref (`id@version`); a unique verified cache hit does not contact Hub to find the local suite-runs.
+- Hub Leaderboard **Public** lists complete, release-bound suites only.
+
+### Hub path checklist
+
+```text
+ageval publish <dataset> --org <org> --draft
+ageval release <org/dataset>
+ageval run <dataset> [--profiles …]          # omit --task → suite
+ageval results upload-suite <dataset> --suite-run <8-hex> --public [--with-attempts]
+```
+
+## `ageval publish` / `ageval release`
+
+- Draft slot overwrites for that `dataset_id`. Reserved version name `draft` is not a release.
+- `ageval release <dataset_id>` promotes the current draft.
+- Plugin packages do **not** use the draft slot.
+
+## `ageval view` / `ageval jobs delete`
+
+- Local dataset UI. Path or registry ref (`id@version` / `@sha256:…`). A unique verified cache hit does not contact Hub.
+- `jobs delete --local <dataset> --job <id> --yes` is local Job delete, not Registry `results delete`.

ageval_cli-0.8.1/.agents/skills/ageval-config-package/SKILL.md

@@ -0,0 +1,86 @@
+---
+name: ageval-config-package
+description: >
+  Author ageval datasets (ageval.yaml + tasks/*/task.yaml, run.py/evaluator.py,
+  profiles.yaml, environment kinds local|docker|e2b|ssh|daytona, limits, gold isolation,
+  script vs LLM-as-judge scoring). Dataset README Run commands start with `ageval`
+  (CLI already installed). Never `uv run ageval` or `.venv/bin/ageval`. Triggers:
+  ageval.yaml, task.yaml, profiles.yaml, dataset, environment kind, evaluator.py.
+  Never secrets in yaml. Never provider.kind.
+---
+
+# Config / dataset + task
+
+Config Core is the only normative reader. Delivery unit is a **dataset** root.
+
+```text
+my-dataset/                 # CLI path (ageval.dataset/1)
+├── ageval.yaml
+├── profiles.yaml           # THE job (lock/run default). ageval publish also packs every root profiles*.yaml
+├── env.example
+├── shared/lib/             # optional; import shared.lib.*
+└── tasks/<task_id>/
+    ├── task.yaml           # ageval.task/1 — roles + intent
+    ├── run.py              # async def run(ctx)
+    ├── evaluator.py
+    ├── environment/        # Dockerfile, setup.sh, compose
+    ├── evaluation/         # gold — not for Agent
+    └── data/               # seed
+```
+
+CLI: `ageval lock|run <dataset-root> --task <id>`.
+
+**Reader-facing README / docs:** commands start with `ageval`. The reader already has the CLI. Do not write `uv run ageval`, `uv run ageval lock`, `.venv/bin/ageval`, or `AGEVAL=/path/to/...` wrappers. Agent-internal notes (this skill, AGENTS.md) may name a venv binary; dataset README may not.
+
+Dataset README introduces **this dataset** (identity, `profiles.yaml`, layout, `ageval lock|run`, conversion, known gaps). It does **not** document local `.ageval/suite-runs` ids, P/F/E counts, or leaderboard rows. Those are `upload-suite` later, not package copy.
+
+Files present are recognized. Do not re-declare `run.py` entrypoints in yaml. Do not put `executor` / `api_key` on the member task. Do not write `provider.kind` or `assurance`.
+
+```yaml
+# ageval.yaml
+format: ageval.dataset/1
+dataset_id: org/my-suite
+version: "0.1.0"
+tasks:
+  root: tasks
+```
+
+```yaml
+# profiles.yaml
+format: ageval.profiles/1
+environment: docker
+agent_profiles:
+  solver:
+    executor: acp
+    api_key: ${ZHIPU_API_KEY}
+    options: { entry: pi }
+    extensions:
+      - plugin: acp
+      - plugin: docker
+```
+
+Gold lives in `evaluation/` and is uploaded at evaluate, not before. `setup.sh` is the last environment slot.
+
+Two scoring styles in `evaluator.py` (same barrier; PASS still the return value):
+
+| Style | When | Evidence |
+| --- | --- | --- |
+| Deterministic script | Default. Compare artifacts to gold. No `Agent.session`. | `result.json`. No `evaluation/observation.jsonl`. Optional `checks` via `evaluation_check(...)` → `evaluation/checks.json`. |
+| LLM-as-judge | Extra role on the **same** `profiles.yaml` (e.g. `judge`). After gold lands, `Agent.session("judge").invoke`. | `evaluation/observation.jsonl` (no `user` rows). Not merged into Agent `trajectory.jsonl`. May also return `checks`. |
+
+Judge binding stays in `profiles.yaml` (not on the member task). Budget `limits.agent_invocations` for solver **and** judge. SDK session into the evaluator process is projected on `environment: local`. No `Agent.session` = script path.
+
+`--profiles` swaps the job for **this** lock/run. `--agent` is mutually exclusive with `--profiles`. `--model` requires `--agent`.
+
+### `profiles.yaml` vs extra `profiles*.yaml`
+
+| | Default job (`ageval run <root>`) | `ageval publish` / packageDigest |
+| --- | --- | --- |
+| `profiles.yaml` | **this file only** | included |
+| sibling `profiles*.yaml` at dataset root | ignored unless `--profiles` | **included** (`member_paths_for_digest` / tarball). Overlays those docs declare also enter. |
+
+Hub consumers who `ageval run official/…@ver` without `--profiles` get **`profiles.yaml`**. Extra `profiles.acp.*.yaml` at the root still **ship in the package** and change digest. Experiment jobs that must not be on Hub do not live as `profiles*.yaml` under the dataset root.
+
+Hub comparability still uses suite fingerprint / `upload-suite`. Do not add `n_attempts` to yaml.
+
+Field catalog: [references/ageval-yaml.md](references/ageval-yaml.md). Isolation: [references/isolation.md](references/isolation.md). Conversion: [references/conversion.md](references/conversion.md).

ageval_cli-0.8.1/.agents/skills/ageval-config-package/references/conversion.md

@@ -0,0 +1,76 @@
+# Conversion & multi-task package authoring
+
+Author-facing patterns for porting upstream benchmarks into an ageval **dataset**.
+Authority: `docs/design/08-conversion-security-testing.md`, `docs/design/02-task-package-and-config.md`.
+
+## When to use a generator
+
+| Signal | Action |
+| --- | --- |
+| ≥ ~5 isomorphic members (same `run.py` / eval shape) | Prefer `scripts/generate_package.py` (or equivalent) |
+| Members differ only by scenario id / params / data | Thin `task.yaml` + `data/` / `evaluation/` per task; shared orchestration |
+| One-off demo task | Inline `run.py` is fine; still respect gold isolation |
+
+### Generator shape (reference)
+
+1. Read upstream assets (often under `shared/assets/` or an external path).
+2. Emit `tasks/<task_id>/task.yaml` with `provenance` filled.
+3. Emit thin `run.py` / `evaluator.py` that import **`shared.lib.*`**.
+4. Write per-task `data/` (agent-visible) and `evaluation/` (gold) as needed.
+5. Stay idempotent: re-run regenerates members without hand-editing fifty files.
+
+Do not write `provider.kind` / `assurance`. Generate `run.py`.
+
+### Import migration
+
+| Old (leaf inject) | New (dataset root on path) |
+| --- | --- |
+| `from harness_core import run` | `from shared.lib.harness_core import run` |
+| `from bridge import make_environment` | `from shared.lib.bridge import make_environment` |
+| `from evaluator_core import evaluate` | `from shared.lib.evaluator_core import evaluate` |
+
+Shared glue lives under `shared/lib`; the task entry is `run.py`.
+
+## Upstream → ageval owner map
+
+| Upstream concept | Put under | Consumer | Never |
+| --- | --- | --- | --- |
+| Instruction / prompt / agent-visible files | `tasks/<id>/data/` | Agent seed / `ctx.params` | Gold under `data/` |
+| Hidden tests / labels / expected answers | `tasks/<id>/evaluation/` | Evaluator only | `shared/`, Agent mount |
+| Container image / apt / system deps | `tasks/<id>/environment/Dockerfile` | docker / e2b `host.start` | Runtime `apt`/`pip` as parity default |
+| Domain tools / bridge / dual-agent loop | `shared/lib/` (multi-task) or `tasks/<id>/lib/` | `run.py` + evaluator import | Gold files |
+| Shared policies / DB dumps / static assets | `shared/assets/` | Code paths | Agent default mount of gold |
+| Offline known-good workspace files | `tasks/<id>/solution/` | Human / CI / offline flag only | Default Agent seed |
+| Job binding (executor / entry / model) | dataset `profiles.yaml` (default job) | Runtime / job overlay | Secrets in yaml; extra `profiles*.yaml` at dataset root also enter `ageval publish` / packageDigest |
+
+## Scenario cut
+
+- One upstream scenario / one stable role topology → one dataset (`ageval.yaml`).
+- Vary task business parameters and `data/` / `evaluation/` per member, not role topology.
+- Different scenarios → separate datasets.
+
+## Thin multi-task template
+
+```text
+shared/
+  lib/
+    run_core.py          # async run(ctx, *, task_dir)
+    evaluator_core.py
+    bridge.py
+tasks/task-00/
+  run.py                 # from shared.lib.run_core import run as _run
+  evaluator.py
+  data/ … evaluation/
+```
+
+## Official base `ageval-attempt:base`
+
+When the port `FROM`s the official tag (historical name, CPython 3.12) instead of the vendor instance image, re-check every migrated `RUN pip` / `apt` against this base. A green `docker build` that never imports the hidden tests still ERROR at collection.
+
+Details: `references/isolation.md`.
+
+## Evidence / claims
+
+- Conversion completeness, Hub package publish, or `ageval lock` success **do not** upgrade evidence grade.
+- PASS only from the independent evaluator after barrier.
+- Fill `provenance` for ports; known gaps stay honest.

ageval_cli-0.8.1/.agents/skills/ageval-design-system/SKILL.md

@@ -0,0 +1,106 @@
+---
+name: ageval-design-system
+description: >
+  ageval web UI style rules for coding agents editing website (landing + docs),
+  apps/hub, or apps/viewer: canonical color tokens (cool paper/ink + IKB
+  #1B54E8/#5B7BFF), font stacks (Geist/Anton wordmark-only), radius,
+  role-based focus (scan chrome keeps hairline; edit fields use IKB border),
+  reuse of shipped components, the ten invariants, and the machine check
+  scripts/check_design_tokens.py. Use when adding or restyling UI in any of the
+  three web surfaces, picking colors or fonts, adding icons, or when a change
+  touches raw hex, focus rings, buttons, search fields, or the owl brand assets.
+  Authority is docs/design/13-web-ui-tokens.md; apps/shared/DESIGN.md YAML is
+  the SPA token listing (Hub/Viewer inherit it); SPA DESIGN.md Taste is anti-slop
+  (product chrome, not landing) plus the reuse map.
+  This skill only routes and summarizes — it is not a page inventory.
+---
+
+# ageval web UI design system
+
+Three surfaces, one language: website landing, website docs (fumadocs),
+hub SPA, viewer SPA. Authority: `docs/design/13-web-ui-tokens.md`
+(constitution: tokens, focus roles, motion — **not** a page inventory).
+
+On Hub / Viewer, also read `apps/shared/DESIGN.md` YAML (shared theme constants),
+that SPA's `DESIGN.md` **Taste** (anti-slop) and role table, and `AGENTS.md`
+(scope). Copy a shipped instance. Do not restyle a shadcn primitive default
+and call it on-brand.
+
+Hub and Viewer are **product consoles**. Do not apply landing/portfolio
+playbooks (`frontend-design`, `design-taste-frontend` hero / bento / GSAP /
+magnetic / "invent a new identity"). Identity is already locked: cool paper
++ IKB + Geist. Spend the skill budget on not looking like default shadcn.
+
+## Where tokens live
+
+| Surface      | Token file                                                              |
+| ------------ | ----------------------------------------------------------------------- |
+| constitution | `docs/design/13-web-ui-tokens.md`                                       |
+| SPA listing  | `apps/shared/DESIGN.md` YAML (Hub/Viewer inherit; do not fork)          |
+| hub / viewer | `apps/shared/css/tokens.css` (`--viewer-*` + `@theme`; each `index.css` `@import`s it) |
+| docs         | `website/src/app/global.css` (`--color-fd-*`, `--ageval-link*`)           |
+| landing      | Token block at the top of `website/src/components/landing/landing.css`  |
+
+Change order: edit the table in `docs/design/13` first → sync script `CANONICAL`
+→ sync `apps/shared/DESIGN.md` YAML and `apps/shared/css/tokens.css` → run the machine check.
+If they disagree, fix the copies to match docs/13 + the script.
+
+## Quick rules (full list in docs/design/13)
+
+- Hex values live only in token files and owl brand assets. App code uses semantic
+  names (`text-ink`, `border-hairline`, …).
+- IKB (`link` / `primary`) is for links, focus, primary CTAs, and brand slots.
+  Do not use it as a large fill. Functional accents (error, warning, star gold)
+  have their own tokens; the UI is not ink/paper/IKB-only.
+- `mute` is never body text. `canvas*` is surface, `hairline` is line,
+  `ink` / `body` / `mute` are type.
+- Anton is wordmark-only. Body is Geist (+ CJK fallback). Mono is Geist Mono.
+- Radii are 8 / 10 / 14px only. Primary CTA (SPA Button `default`) is IKB fill +
+  `rounded-[8px]` + mono 13px + pop shadow + `focus-visible:ring-2 ring-link/70`.
+  Search is stadium. Do not use clip-path chamfer on buttons.
+- Section tabs use `UnderlineTabs` (sans `text-sm`, liquid-gooey Move thumb, fill
+  `canvas-soft-2`). Do not draw an IKB underline. Page heads use `PageHead`
+  (h1 + optional sub + hairline; no numbered kicker). One tab strip per view; a
+  second exclusive choice is `Select`. Wrapping chips use `Chip`, not `bg-link/10`.
+- Operator-facing controls and table column labels use body-sm 14px. Caption is
+  timestamps and mute meta, not column names. Do not invent a smaller clickable size. A button group is one hairline box; selected fill
+  is `canvas-soft-2`. New chrome joins the toolbar already on the page — do not open a
+  vacant band for a single control. Layout: SPA DESIGN.md **Composition**.
+- List / table row text (non-numeric): always default sans. Mono only for
+  `<code>` / `<pre>` / command strips, and numeric `tabular-nums` alignment.
+- Plugin / agent marketplace lists are `CatalogCard` grids. Datasets are
+  org-grouped hairline tables; jobs, leaderboard, and members are flat
+  hairline tables. Do not put slot/binding tags
+  on cards.
+- Motion is CSS only on hub/viewer, plus two named exceptions in docs/13:
+  `ThinkingLogo` canvas and `liquid-gooey` Move (tabs / Hub sidebar). Default
+  `200ms` / `ease-smooth`. Also `--ease-spring` (toast, star burst, squish
+  release), `--ease-glide` (thumb CSS), `--t-press` 80ms. Morph does not ship.
+  Website docs/landing do not take the library. Landing may stagger the hero and
+  reveal sections 8px; no GSAP/Motion, pin/scrub, magnetic hover, or cursor trail.
+- Focus is role-based (docs/13), not "every field goes IKB":
+  buttons / links / cards `ring-2 ring-link/70`; **edit** fields 1px
+  `border-link`; **scan** chrome (search, filter, Select trigger) keeps
+  `border-hairline` — no border color change. Landing 3px outline.
+  Hub search copies `CatalogScopeBar`, not `Input`'s default.
+- Popover / card shadow is `--viewer-shadow-pop` only (tight pop; no
+  `--viewer-shadow-liquid`). Phase charts use `--viewer-phase-1..6` only.
+  Hub shell is left/right (`canvas-soft` aside, `canvas` main). Viewer header is
+  `canvas-soft`; docs sidebar matches the aside.
+- Icons: product brand uses owl; function icons use lucide; plugin/agent entity
+  marks default to the uploader GitHub avatar, or a closed color catalog /
+  GitHub login override. Do not add a third-party logo component library as a
+  runtime dependency. Catalog hex lives only in `apps/shared/lib/brand-marks/assets/`.
+- SPA anti-slop lives in that SPA's `DESIGN.md` **Taste**. Do not copy it here.
+
+## Machine check
+
+```bash
+python3 scripts/check_design_tokens.py
+```
+
+Checks: docs/13 table ↔ script `CANONICAL` ↔ `apps/shared/DESIGN.md` YAML; mapped
+variables stay inside the canonical set; no raw hex in app code (outside the
+allowlist).
+CI job `design-tokens` runs the same command. Run it locally after token or
+style edits.

ageval_cli-0.8.1/.agents/skills/ageval-sdk-harness/SKILL.md

@@ -0,0 +1,36 @@
+---
+name: ageval-sdk-harness
+description: >
+  Write dataset run.py with ageval_sdk (RunContext, Agent/AgentSession, ToolSet,
+  RunTerminal, publish). Use for task run.py, sessions, tool guards, optional
+  evaluator.py LLM-as-judge. Triggers: AgentSession, RunTerminal,
+  ctx.publish_json, write run.py, evaluator.py. SDK never decides PASS or holds
+  host credentials.
+---
+
+# SDK / run.py
+
+```python
+from ageval_sdk import RunContext, RunTerminal
+
+async def run(ctx: RunContext) -> RunTerminal:
+    async with ctx.agent.session("solver", max_turns=1) as session:
+        reply = await session.invoke(ctx.params["instruction"])
+    ctx.publish_json("reply", {"ok": bool(reply.get("ok")), "text": reply.get("text") or ""})
+    if not reply.get("ok"):
+        return RunTerminal.failed(str(reply.get("error") or "invoke_failed"))
+    return RunTerminal.completed("ok")
+```
+
+`RunTerminal.completed` is not PASS. Evaluator is a separate process.
+
+Default `evaluator.py` is a script: read artifacts + gold, return `{status, score, metrics}`. Optional extra key `checks` (use `evaluation_check(...)`) — parent writes `evaluation/checks.json`; bind ignores it. Optional LLM-as-judge: after gold is uploaded, `Agent.session(<role>).invoke` on a role declared in the same `profiles.yaml`. Parent injects `inputs["agent"]` when the socket is projected (`environment: local`). Still return `{status, score, metrics}`. `invoke` kwargs must not override `profile_id` / executor.
+
+| May | Must not |
+| --- | --- |
+| Open sessions, tools, publish | Decide PASS |
+| Use `ctx.params` | Re-read lock or secrets |
+| Return completed/failed | Raise Core ceilings |
+| Evaluator `Agent.session` after gold | Bind PASS from judge prose / observation.jsonl |
+
+API: [references/api.md](references/api.md). Antipatterns: [references/antipatterns.md](references/antipatterns.md).

ageval_cli-0.8.1/.agents/skills/ageval-sdk-harness/references/api.md

@@ -0,0 +1,55 @@
+# ageval_sdk public surface (shipped)
+
+Package: `ageval_sdk` (see `src/ageval_sdk/`, same distribution as `ageval`).
+
+## Types
+
+| Symbol | Role |
+| --- | --- |
+| `RunContext` | params, workspace, artifact_dir, agent, publish |
+| `RunScope` | attempt/trial/run identity (parent-owned) |
+| `Agent` | factory for sessions bound to attempt |
+| `AgentSession` | `open` / `invoke` / `close` via parent socket |
+| `Tool` / `ToolSet` | local tools |
+| `AllowList` / `CallLimit` | soft local guards |
+| `RunTerminal` | completed / failed — not PASS |
+| `bounded_gather` / `collect_results` / `first_success` | workflow helpers |
+| `evaluation_check` | optional `checks` row for the evaluator return. `evaluation_check(id, *, title, status, score, script, environment, exit_code, stdout, stderr, exec_result)` — `id` required; `exec_result` fills exit/stdout/stderr when those kwargs are omitted |
+
+## AgentSession.invoke return keys (parent path)
+
+Includes `ok`, `error`, `text`, `structured`, `provider_session_handle`, `invocation_id`, `evidence_relative`, and `tool_calls` (list of `{id, name, arguments}`; empty when the turn is text-only).
+
+Optional invoke kwargs: `tools` (OpenAI-style catalog) and `messages` (chat history). Omit both for prompt-only invoke. Profile / workspace / executor overrides stay rejected. ACP ignores the catalog.
+
+## Offline
+
+When `AGEVAL_OFFLINE_AGENT=1`, invoke fails closed (`offline_forced`) — never stub PASS on public paths.
+
+## Evaluator (optional)
+
+`evaluator.py` may construct `Agent(attempt_id=…)` or use injected `inputs["agent"]` when `AGEVAL_AGENT_SERVICE_SOCK` is set. Same `Agent.session(profile_id).invoke` as `run.py`. Evaluate-phase turns fold to `evaluation/observation.jsonl` (omit `user`). Optional `checks` on the return persist as `evaluation/checks.json`. Bind still reads only `{status, score, metrics}`.
+
+```python
+from ageval_sdk import evaluation_check
+
+return {
+    "status": "PASS",
+    "score": 1.0,
+    "metrics": {},
+    "checks": [
+        evaluation_check(
+            "audit",
+            title="schema audit",
+            status="PASS" if audit.exit_code == 0 else "FAIL",
+            script="evaluation/audit.py",
+            environment="audit",
+            exec_result=audit,
+        ),
+    ],
+}
+```
+
+Omit `checks` → no `evaluation/checks.json`. A FAIL row with a PASS return is legal. `script` is a dataset locator (member-relative or `tasks/<id>/…`); parent does not copy the script into the archive.
+
+Design: `docs/design/03-task-run-and-sdk.md`.