@amd-gaia/gaia 0.2.0 → 0.3.0

package/README.md

@@ -11,8 +11,10 @@ verifies both against a checksum manifest that ships inside this package, and dr
 you into the terminal UI. No Python to install, no repo to clone, no build step.
 Local inference is the default. The TUI can explicitly select Fireworks AI or AMD
 LLM Gateway through Lemonade; remote chat sends conversation history to that provider. The
-terminal UI's `--use-claude` flag, which would send a conversation to the
-Anthropic API, is refused for the agent this package installs — see
+terminal UI's *launch* flag `--use-claude`, which would pin the Anthropic API for
+the life of the process, is refused for the agent this package installs. That is a
+limit on the launch flag, not on the agent: a caller can still ask for Anthropic
+per request with `/query`'s `provider` field (contract ≥ 2.14) — see
 [`SPEC.md` §5.5](./SPEC.md#55-other-transports).
 
 The terminal UI you get here is the published **`terminal-hub`** component — the
@@ -37,9 +39,9 @@ however you arrive at the terminal UI, it behaves identically.
 - **Node.js 18+** (for the built-in `fetch`).
 - **[Lemonade Server](https://amd-gaia.ai/docs/reference/dev)** running locally —
   it hosts the model the agent thinks with. GAIA tells you if it isn't up.
-- The `gaia` Python CLI **0.23.1 or newer** on `PATH` for the daemon the terminal
-  UI starts. Earlier cores start a daemon that has no entry for this agent, so the
-  UI comes up with nothing behind it. Install it with
+- The `gaia` Python CLI **0.25.0 or newer** on `PATH` for the daemon the terminal
+  UI starts. An earlier core cannot run this agent, so the UI comes up with
+  nothing behind it. Install it with
   `curl -fsSL https://amd-gaia.ai/install.sh | sh` (Windows:
   `irm https://amd-gaia.ai/install.ps1 | iex`).
 
@@ -182,10 +184,16 @@ await shutdown(proc);
 **Reuse the same `session_id` for every turn in a conversation.** It is what
 lets a document you had it index, or a skill you had it load, survive to the
 next question — drop it (or mint a new one per call) and the agent still
-answers, but it forgets everything from the previous turn. See
+answers, but it forgets everything from the previous turn. An id must be 1–128
+characters from `A-Z a-z 0-9 . _ -` (a UUID works); anything else is a 400. See
 [`SPEC.md` §5.2](./SPEC.md#52-session_id-and-agent-retention) for the retry
 and eviction behavior.
 
+**Attaching files to a conversation? Send them in `documents`** (contract ≥
+2.16): the absolute paths, the whole set on every turn. The agent indexes any it
+has not read yet before answering, and can read those exact files and nothing
+else beside them. See [`SPEC.md` §5.1](./SPEC.md#51-endpoints).
+
 Every failure throws a typed error (`IntegrityError`, `PlatformError`,
 `HealthTimeoutError`, `VersionMismatchError`, `BinaryNotFoundError`) with a message
 that names what failed and what to do about it.
@@ -211,17 +219,31 @@ what it must hash to. The release pipeline regenerates it from the artifacts
 actually being served — the sidecars it just published, and the terminal-hub
 artifacts it downloaded and cross-checked against the hub's own recorded hashes.
 
-Between releases the lock carries `PENDING-replace-with-real-sha256` placeholders.
-**A placeholder blocks the fetch outright** — before any network call — so an
-unverifiable binary can never be downloaded, let alone executed. If you need to run
-against a locally built binary, build it yourself and point the lifecycle helpers
-at it directly; the fetcher will not be talked into it.
+A committed release's lock, like this one, always carries real hashes — the
+regenerate step in the release pipeline never publishes placeholders. Between
+releases, an unreleased working tree's lock instead carries
+`PENDING-replace-with-real-sha256` placeholders, and **a placeholder blocks the
+fetch outright** — before any network call — so an unverifiable binary can never
+be downloaded, let alone executed. If you need to run against a locally built
+binary, build it yourself and point the lifecycle helpers at it directly; the
+fetcher will not be talked into it.
 
 ## Links
 
 - Guide: <https://amd-gaia.ai/docs/guides/gaia>
 - Technical reference: [`SPEC.md`](./SPEC.md)
+- Eval scorecard: [`SCORECARD.md`](./SCORECARD.md) — the agent's measured
+  judged-scenario pass rate across the 13-category eval corpus, with the exact
+  recipe to reproduce it
 - Changes: [`CHANGELOG.md`](./CHANGELOG.md)
 - Issues: <https://github.com/amd/gaia/issues>
 
 MIT licensed. © 2024-2026 Advanced Micro Devices, Inc.
+
+## Developer engineering mode
+
+For GAIA contributors: with `--developer-mode` on, you can report a problem you hit
+while using GAIA and hand a snapshot of it, which you approve, to your own Claude
+Code or Codex app. That app diagnoses it and works on a fix in a separate git
+worktree. The mode is off unless you turn it on, and ordinary sessions never
+expose it. See the [usage guide](https://amd-gaia.ai/docs/guides/harness-engineering).

package/SCORECARD.md

@@ -0,0 +1,180 @@
+---
+schema_version: 1
+agent:
+  name: GAIA (flagship agent)
+  version: 0.1.1
+recipe:
+  dataset:
+    reference: eval/scenarios/
+    description: 'Judged multi-turn scenario corpus for the flagship gaia agent (13
+      gaia_* categories: core conversation/memory tiers through adversarial, RAG,
+      files, data, web, shell gate, skill lifecycle + all 12 skills, honesty floor,
+      tool selection, code index, skill capture); deterministic fixtures under tests/fixtures/gaia/,
+      planted-fact ground truth per eval/scenarios/GAIA_FIXTURE_VALUES.md'
+    size: 103
+  methodology: 'gaia eval agent --agent-type gaia: each scenario is driven against
+    the Agent UI backend (REST/SSE) by the eval driver and scored by the Claude judge
+    on planted-fact ground truth + success criteria. Aggregate = judged_pass_rate
+    over judged scenarios (PASS/FAIL/BLOCKED_BY_ARCHITECTURE; infra failures are excluded
+    from the rate and fail the run''s integrity gate instead). Reported secondaries
+    (weight 0): the judge''s 0-10 avg_score normalized to [0,1], and per-category
+    pass rates. Thresholds/enforcement: tests/fixtures/gaia/quality_gate_thresholds.json'
+  config:
+    harness: gaia eval agent
+    run_id: eval-20260824-104200
+    agent_type: gaia
+    eval_model: claude-sonnet-4-6
+    budget_per_scenario_usd: 5.0
+    categories:
+    - gaia_core
+  environment:
+    gaia_commit: d104b3b1
+    model: claude-haiku-4-5
+    ctx_size: 200000
+    hardware: developer workstation (x86-64, agent on the Claude API — no Lemonade,
+      no AMD NPU/GPU inference)
+    eval_model: claude-sonnet-4-6
+    note: 'HARNESS-VALIDATION card, pending first runner baseline: agent on claude-haiku-4-5
+      over the gaia_core category only (11 scenarios, tags local_blocked_no_embedder
+      excluded). NOT the product baseline — that is Gemma-4-E4B-it-GGUF on the self-hosted
+      runner, captured by the first full gaia_scorecard_refresh.yml dispatch (rebaseline=true).'
+results:
+  test_cases_run: 11
+  metrics:
+  - name: judged_pass_rate
+    value: 0.7272727272727273
+    weight: 1.0
+  - name: avg_score_normalized
+    value: 0.885
+    weight: 0.0
+  - name: gaia_core_pass_rate
+    value: 0.7273
+    weight: 0.0
+  breakdown:
+    per_category:
+    - category: gaia_core
+      total: 11
+      correct: 8
+      accuracy: 0.7273
+  performance:
+    ttft_s: 7.254
+    throughput_tps: 38.7
+    total_input_tokens: 90710
+    total_output_tokens: 810
+    scenarios_with_data: 2
+aggregate:
+  name: judged_pass_rate
+  formula: round(100 * sum(weight_i * value_i) / sum(weight_i), 2)
+  components:
+  - metric: judged_pass_rate
+    value: 0.7272727272727273
+    weight: 1.0
+  - metric: avg_score_normalized
+    value: 0.885
+    weight: 0.0
+  - metric: gaia_core_pass_rate
+    value: 0.7273
+    weight: 0.0
+  value: 72.73
+generated_at: '2026-08-24T11:25:31.271112+00:00'
+inherited_from: null
+---
+# GAIA (flagship agent) — Eval Scorecard v0.1.0
+
+**Aggregate score: 72.73** (out of 100)
+
+## Recipe
+
+| Field | Value |
+|-------|-------|
+| Dataset | [eval/scenarios/](eval/scenarios/) |
+| Description | Judged multi-turn scenario corpus for the flagship gaia agent (13 gaia_* categories: core conversation/memory tiers through adversarial, RAG, files, data, web, shell gate, skill lifecycle + all 12 skills, honesty floor, tool selection, code index, skill capture); deterministic fixtures under tests/fixtures/gaia/, planted-fact ground truth per eval/scenarios/GAIA_FIXTURE_VALUES.md |
+| Dataset size | 103 labeled examples |
+| Test cases run | 11 |
+| Methodology | gaia eval agent --agent-type gaia: each scenario is driven against the Agent UI backend (REST/SSE) by the eval driver and scored by the Claude judge on planted-fact ground truth + success criteria. Aggregate = judged_pass_rate over judged scenarios (PASS/FAIL/BLOCKED_BY_ARCHITECTURE; infra failures are excluded from the rate and fail the run's integrity gate instead). Reported secondaries (weight 0): the judge's 0-10 avg_score normalized to [0,1], and per-category pass rates. Thresholds/enforcement: tests/fixtures/gaia/quality_gate_thresholds.json |
+
+## Metrics
+
+  - **judged_pass_rate**: 0.7273 × 1.0
+  - **avg_score_normalized**: 0.8850 × 0.0
+  - **gaia_core_pass_rate**: 0.7273 × 0.0
+
+## Aggregate score recomputation
+
+Formula: `round(100 × Σ(weightᵢ × valueᵢ) / Σ(weightᵢ), 2)`
+
+Worked example:
+
+```
+round(100 × ((0.7273 × 1.0) + (0.8850 × 0.0) + (0.7273 × 0.0)) / 1.0, 2) = 72.73
+```
+
+A reader can reproduce this value from the `aggregate.components` in the front
+matter alone — no eval-harness access needed.
+
+## Reproduction
+
+Run the following commands from the repository root:
+
+```sh
+# Prerequisites: install the eval extras + this repo's chat/gaia hub
+# packages, start a Lemonade Server with the model on AMD Ryzen AI
+# hardware, and have the Claude Code CLI on PATH (the eval driver).
+uv pip install -e ".[dev,eval,ui,api]" -e hub/agents/chat/python -e hub/agents/gaia/python
+gaia init   # installs and starts Lemonade Server; it must stay running
+
+# Step 0: stage fixtures + start the fixture server (see
+# tests/fixtures/gaia/README.md for the staging contract)
+python tests/fixtures/gaia/prepare_fixture_hub.py --skills-root <skills-root>
+python tests/fixtures/gaia/serve_fixtures.py --port 8765   # separate shell
+
+# Step 1: start the Agent UI backend (separate shell; NOT port 4001)
+python -m gaia.ui.server --port 4200 --host 127.0.0.1
+
+# Step 2: run every gaia category serially (never in parallel)
+for c in gaia_core gaia_memory gaia_rag gaia_files gaia_data gaia_web \
+         gaia_shell gaia_skills_lifecycle gaia_skills_tasks \
+         gaia_skills_capture gaia_honesty gaia_tool_selection \
+         gaia_code; do
+  gaia eval agent --agent-type gaia --category $c \
+    --backend http://127.0.0.1:4200 --budget 5.00 --exclude-tag live
+done
+
+# Step 3: generate this scorecard from ONE combined run's output dir
+python hub/agents/gaia/python/packaging/gen_scorecard.py \
+    --run-dir <run-dir-printed-by-the-eval> \
+    --model claude-haiku-4-5 \
+    --ctx-size 200000 \
+    --hardware "<hardware class>"
+```
+
+See [eval-scorecard docs](https://amd-gaia.ai/docs/reference/eval-scorecard) and the [`adding-eval-scorecard` skill](.claude/skills/adding-eval-scorecard/SKILL.md) for the full setup guide.
+
+## Environment
+
+| Field | Value |
+|-------|-------|
+| gaia_commit | d104b3b1 |
+| model | claude-haiku-4-5 |
+| ctx_size | 200000 |
+| hardware | developer workstation (x86-64, agent on the Claude API — no Lemonade, no AMD NPU/GPU inference) |
+| eval_model | claude-sonnet-4-6 |
+| note | HARNESS-VALIDATION card, pending first runner baseline: agent on claude-haiku-4-5 over the gaia_core category only (11 scenarios, tags local_blocked_no_embedder excluded). NOT the product baseline — that is Gemma-4-E4B-it-GGUF on the self-hosted runner, captured by the first full gaia_scorecard_refresh.yml dispatch (rebaseline=true). |
+
+## Category breakdown
+
+| Category | Total | Correct | Accuracy |
+|----------|-------|---------|----------|
+| gaia_core | 11 | 8 | 0.7273 |
+
+## Performance
+
+_Measured on the run environment above (model / hardware / gaia_commit / corpus size); the perf gate is report-only, so these are observed values, not pass/fail bars (see `tests/fixtures/email/perf_gate_thresholds.json`)._
+
+| Metric | Value |
+|--------|-------|
+| ttft_s | 7.254 |
+| throughput_tps | 38.7 |
+| total_input_tokens | 90710 |
+| total_output_tokens | 810 |
+| scenarios_with_data | 2 |