@amd-gaia/gaia 0.2.0 → 0.3.2

package/README.md

@@ -11,8 +11,10 @@ verifies both against a checksum manifest that ships inside this package, and dr
 you into the terminal UI. No Python to install, no repo to clone, no build step.
 Local inference is the default. The TUI can explicitly select Fireworks AI or AMD
 LLM Gateway through Lemonade; remote chat sends conversation history to that provider. The
-terminal UI's `--use-claude` flag, which would send a conversation to the
-Anthropic API, is refused for the agent this package installs — see
+terminal UI's *launch* flag `--use-claude`, which would pin the Anthropic API for
+the life of the process, is refused for the agent this package installs. That is a
+limit on the launch flag, not on the agent: a caller can still ask for Anthropic
+per request with `/query`'s `provider` field (contract ≥ 2.14) — see
 [`SPEC.md` §5.5](./SPEC.md#55-other-transports).
 
 The terminal UI you get here is the published **`terminal-hub`** component — the
@@ -37,9 +39,12 @@ however you arrive at the terminal UI, it behaves identically.
 - **Node.js 18+** (for the built-in `fetch`).
 - **[Lemonade Server](https://amd-gaia.ai/docs/reference/dev)** running locally —
   it hosts the model the agent thinks with. GAIA tells you if it isn't up.
-- The `gaia` Python CLI **0.23.1 or newer** on `PATH` for the daemon the terminal
-  UI starts. Earlier cores start a daemon that has no entry for this agent, so the
-  UI comes up with nothing behind it. Install it with
+- **On Linux x64, glibc 2.34 or newer** for the agent sidecar (Ubuntu 22.04,
+  Debian 12 and RHEL 9 or later). An older distro cannot start the agent. Check
+  with `ldd --version`.
+- The `gaia` Python CLI **0.25.2 or newer** on `PATH` for the daemon the terminal
+  UI starts. An earlier core cannot run this agent, so the UI comes up with
+  nothing behind it. Install it with
   `curl -fsSL https://amd-gaia.ai/install.sh | sh` (Windows:
   `irm https://amd-gaia.ai/install.ps1 | iex`).
 
@@ -92,8 +97,10 @@ Common options:
 | `--component <name>`  | Fetch only `sidecar` or `tui`                                  | `fetch` |
 | `--port <n>`          | Sidecar bind port (default `8141`)                             | `serve` |
 
-A flag a command does not read is **refused**, not ignored — `gaia run --port
-9000` exits 2 rather than silently coming up on the default port.
+A flag a command does not read, or an option the launcher does not know, is
+**refused**, not ignored — `gaia run --port 9000` and `gaia --forse` exit 2
+rather than silently doing something else. Arguments intended for the terminal
+UI must come after a bare `--`, for example `gaia -- --debug`.
 
 Set `DEBUG=gaia` for download, spawn, and sidecar output on stderr. Diagnostics
 never touch stdout, which the terminal UI owns.
@@ -182,10 +189,79 @@ await shutdown(proc);
 **Reuse the same `session_id` for every turn in a conversation.** It is what
 lets a document you had it index, or a skill you had it load, survive to the
 next question — drop it (or mint a new one per call) and the agent still
-answers, but it forgets everything from the previous turn. See
+answers, but it forgets everything from the previous turn. An id must be 1–128
+characters from `A-Z a-z 0-9 . _ -` (a UUID works); anything else is a 400. See
 [`SPEC.md` §5.2](./SPEC.md#52-session_id-and-agent-retention) for the retry
 and eviction behavior.
 
+**Attaching files to a conversation? Send them in `documents`** (contract ≥
+2.16): the absolute paths, the whole set on every turn. The agent indexes any it
+has not read yet before answering, and can read those exact files and nothing
+else beside them. See [`SPEC.md` §5.1](./SPEC.md#51-endpoints).
+
+**Private chat, or memory off? Send `memory` every turn** (contract ≥ 2.18):
+`"private"` or `"off"` stops the turn saving anything to memory, including the
+errors and lessons the agent records on its own; omit it and memory stays as the
+agent has it. Send `allowed_paths` with the folders your UI shows the chat can
+read, so the agent works in the same scope. Send `working_directory` (contract
+≥ 2.24) with the absolute chat workspace so bare file names resolve in the same
+folder the UI shows; it is separate from the access allowlist. Binaries older
+than 2.18 reject `allowed_paths` and older than 2.24 reject `working_directory`,
+so probe `/version` first. See
+[`SPEC.md` §5.1](./SPEC.md#51-endpoints).
+
+**Internet switched off for the chat? Send `internet_off`** (contract ≥
+2.23) every turn: `true` refuses web search, page fetches, the browser,
+connector, mailbox and Agent Hub tools and commands that contact another host;
+`false` allows them.
+Omit it and the agent keeps its own setting. Binaries older than 2.23 reject
+the field with `422`, so probe `/version` first, do not answer an
+internet-off turn on an older one, and cancel its running turn when the switch
+goes off. If the user flips the switch while a turn is
+running, `POST /v1/gaia/sessions/{session_id}/internet` with
+`{"internet_off": true}` so the next tool call follows it rather than the next
+turn. Post it only once `/query` has answered: before then the turn's own
+`internet_off` replaces it, so re-read the switch at that point and post it if
+it changed. See [`SPEC.md` §5.1](./SPEC.md#51-endpoints).
+
+**Showing what a turn changed, with undo? Send `changes_turn_id`** (contract ≥
+2.19) along with `session_id`: the agent copies each file before it writes it,
+into the snapshot store under its GAIA home, keyed by the session and that turn
+id. The copies are complete before the stream's terminal event, so you can read
+and restore them as soon as the stream ends. Omit it and nothing is copied. See
+[`SPEC.md` §5.1](./SPEC.md#51-endpoints).
+
+**Approving tool calls by policy? Match on `args`.** A `needs_confirmation`
+event's `args` are exactly the arguments the model sent for the gated call,
+each value capped at 64,000 characters; `args_truncated: true` means one was
+cut, so don't auto-approve that call. What the agent adds to its own prompt —
+the folder a shell call runs in, a risk note, the full path of a relative
+write — comes in `prompt_context`, never
+in `args`. See [`SPEC.md` §5.1](./SPEC.md#51-endpoints).
+
+**Serving the turn from the Agent UI? Send `surface: "agent_ui"`** (contract ≥
+2.20), and the agent points recurring requests at the **Scheduled tasks** page
+instead of saying it cannot schedule. Older binaries reject it with `422`.
+
+**Calling it headless?** From contract 2.20 a failed model request names the
+host it went through, and its advice points at the model server the GAIA host
+uses (`LEMONADE_BASE_URL`) rather than at the TUI's settings. A front-end with
+its own settings screen sends `surface` (`"tui"`, `"agent_ui"` or `"cli"`) to
+get its own wording; a headless caller omits it. See
+[`SPEC.md` §5.1](./SPEC.md#51-endpoints).
+
+**Showing the model's reasoning?** Send `reasoning: true` (contract ≥ 2.22)
+and it arrives as `reasoning` events with a `delta`, apart from `status` lines
+and the answer. Join the deltas as they come. Older binaries reject the field
+with `422`. See [`SPEC.md` §5.1](./SPEC.md#51-endpoints).
+
+**Showing how full the model's window is?** From contract 2.21 the `final`
+event carries `context`: the window size, the tokens the next call will take,
+and what takes them (prompt, tools, documents, history). Older binaries leave
+it out. It also carries `performance`, the turn's model and tool timings and
+each model call's token counts, when the binary sends it. See
+[`SPEC.md` §5.1](./SPEC.md#51-endpoints).
+
 Every failure throws a typed error (`IntegrityError`, `PlatformError`,
 `HealthTimeoutError`, `VersionMismatchError`, `BinaryNotFoundError`) with a message
 that names what failed and what to do about it.
@@ -211,17 +287,31 @@ what it must hash to. The release pipeline regenerates it from the artifacts
 actually being served — the sidecars it just published, and the terminal-hub
 artifacts it downloaded and cross-checked against the hub's own recorded hashes.
 
-Between releases the lock carries `PENDING-replace-with-real-sha256` placeholders.
-**A placeholder blocks the fetch outright** — before any network call — so an
-unverifiable binary can never be downloaded, let alone executed. If you need to run
-against a locally built binary, build it yourself and point the lifecycle helpers
-at it directly; the fetcher will not be talked into it.
+A committed release's lock, like this one, always carries real hashes — the
+regenerate step in the release pipeline never publishes placeholders. Between
+releases, an unreleased working tree's lock instead carries
+`PENDING-replace-with-real-sha256` placeholders, and **a placeholder blocks the
+fetch outright** — before any network call — so an unverifiable binary can never
+be downloaded, let alone executed. If you need to run against a locally built
+binary, build it yourself and point the lifecycle helpers at it directly; the
+fetcher will not be talked into it.
 
 ## Links
 
 - Guide: <https://amd-gaia.ai/docs/guides/gaia>
 - Technical reference: [`SPEC.md`](./SPEC.md)
+- Eval scorecard: [`SCORECARD.md`](./SCORECARD.md) — the agent's measured
+  judged-scenario pass rate across the 13-category eval corpus, with the exact
+  recipe to reproduce it
 - Changes: [`CHANGELOG.md`](./CHANGELOG.md)
 - Issues: <https://github.com/amd/gaia/issues>
 
 MIT licensed. © 2024-2026 Advanced Micro Devices, Inc.
+
+## Developer engineering mode
+
+For GAIA contributors: with `--developer-mode` on, you can report a problem you hit
+while using GAIA and hand a snapshot of it, which you approve, to your own Claude
+Code or Codex app. That app diagnoses it and works on a fix in a separate git
+worktree. The mode is off unless you turn it on, and ordinary sessions never
+expose it. See the [usage guide](https://amd-gaia.ai/docs/guides/harness-engineering).

package/SCORECARD.md

@@ -0,0 +1,180 @@
+---
+schema_version: 1
+agent:
+  name: GAIA (flagship agent)
+  version: 0.1.1
+recipe:
+  dataset:
+    reference: eval/scenarios/
+    description: 'Judged multi-turn scenario corpus for the flagship gaia agent (13
+      gaia_* categories: core conversation/memory tiers through adversarial, RAG,
+      files, data, web, shell gate, skill lifecycle + all 12 skills, honesty floor,
+      tool selection, code index, skill capture); deterministic fixtures under tests/fixtures/gaia/,
+      planted-fact ground truth per eval/scenarios/GAIA_FIXTURE_VALUES.md'
+    size: 103
+  methodology: 'gaia eval agent --agent-type gaia: each scenario is driven against
+    the Agent UI backend (REST/SSE) by the eval driver and scored by the Claude judge
+    on planted-fact ground truth + success criteria. Aggregate = judged_pass_rate
+    over judged scenarios (PASS/FAIL/BLOCKED_BY_ARCHITECTURE; infra failures are excluded
+    from the rate and fail the run''s integrity gate instead). Reported secondaries
+    (weight 0): the judge''s 0-10 avg_score normalized to [0,1], and per-category
+    pass rates. Thresholds/enforcement: tests/fixtures/gaia/quality_gate_thresholds.json'
+  config:
+    harness: gaia eval agent
+    run_id: eval-20260824-104200
+    agent_type: gaia
+    eval_model: claude-sonnet-4-6
+    budget_per_scenario_usd: 5.0
+    categories:
+    - gaia_core
+  environment:
+    gaia_commit: d104b3b1
+    model: claude-haiku-4-5
+    ctx_size: 200000
+    hardware: developer workstation (x86-64, agent on the Claude API — no Lemonade,
+      no AMD NPU/GPU inference)
+    eval_model: claude-sonnet-4-6
+    note: 'HARNESS-VALIDATION card, pending first runner baseline: agent on claude-haiku-4-5
+      over the gaia_core category only (11 scenarios, tags local_blocked_no_embedder
+      excluded). NOT the product baseline — that is Gemma-4-E4B-it-GGUF on the self-hosted
+      runner, captured by the first full gaia_scorecard_refresh.yml dispatch (rebaseline=true).'
+results:
+  test_cases_run: 11
+  metrics:
+  - name: judged_pass_rate
+    value: 0.7272727272727273
+    weight: 1.0
+  - name: avg_score_normalized
+    value: 0.885
+    weight: 0.0
+  - name: gaia_core_pass_rate
+    value: 0.7273
+    weight: 0.0
+  breakdown:
+    per_category:
+    - category: gaia_core
+      total: 11
+      correct: 8
+      accuracy: 0.7273
+  performance:
+    ttft_s: 7.254
+    throughput_tps: 38.7
+    total_input_tokens: 90710
+    total_output_tokens: 810
+    scenarios_with_data: 2
+aggregate:
+  name: judged_pass_rate
+  formula: round(100 * sum(weight_i * value_i) / sum(weight_i), 2)
+  components:
+  - metric: judged_pass_rate
+    value: 0.7272727272727273
+    weight: 1.0
+  - metric: avg_score_normalized
+    value: 0.885
+    weight: 0.0
+  - metric: gaia_core_pass_rate
+    value: 0.7273
+    weight: 0.0
+  value: 72.73
+generated_at: '2026-08-24T11:25:31.271112+00:00'
+inherited_from: null
+---
+# GAIA (flagship agent) — Eval Scorecard v0.1.0
+
+**Aggregate score: 72.73** (out of 100)
+
+## Recipe
+
+| Field | Value |
+|-------|-------|
+| Dataset | [eval/scenarios/](eval/scenarios/) |
+| Description | Judged multi-turn scenario corpus for the flagship gaia agent (13 gaia_* categories: core conversation/memory tiers through adversarial, RAG, files, data, web, shell gate, skill lifecycle + all 12 skills, honesty floor, tool selection, code index, skill capture); deterministic fixtures under tests/fixtures/gaia/, planted-fact ground truth per eval/scenarios/GAIA_FIXTURE_VALUES.md |
+| Dataset size | 103 labeled examples |
+| Test cases run | 11 |
+| Methodology | gaia eval agent --agent-type gaia: each scenario is driven against the Agent UI backend (REST/SSE) by the eval driver and scored by the Claude judge on planted-fact ground truth + success criteria. Aggregate = judged_pass_rate over judged scenarios (PASS/FAIL/BLOCKED_BY_ARCHITECTURE; infra failures are excluded from the rate and fail the run's integrity gate instead). Reported secondaries (weight 0): the judge's 0-10 avg_score normalized to [0,1], and per-category pass rates. Thresholds/enforcement: tests/fixtures/gaia/quality_gate_thresholds.json |
+
+## Metrics
+
+  - **judged_pass_rate**: 0.7273 × 1.0
+  - **avg_score_normalized**: 0.8850 × 0.0
+  - **gaia_core_pass_rate**: 0.7273 × 0.0
+
+## Aggregate score recomputation
+
+Formula: `round(100 × Σ(weightᵢ × valueᵢ) / Σ(weightᵢ), 2)`
+
+Worked example:
+
+```
+round(100 × ((0.7273 × 1.0) + (0.8850 × 0.0) + (0.7273 × 0.0)) / 1.0, 2) = 72.73
+```
+
+A reader can reproduce this value from the `aggregate.components` in the front
+matter alone — no eval-harness access needed.
+
+## Reproduction
+
+Run the following commands from the repository root:
+
+```sh
+# Prerequisites: install the eval extras + this repo's chat/gaia hub
+# packages, start a Lemonade Server with the model on AMD Ryzen AI
+# hardware, and have the Claude Code CLI on PATH (the eval driver).
+uv pip install -e ".[dev,eval,ui,api]" -e hub/agents/chat/python -e hub/agents/gaia/python
+gaia init   # installs and starts Lemonade Server; it must stay running
+
+# Step 0: stage fixtures + start the fixture server (see
+# tests/fixtures/gaia/README.md for the staging contract)
+python tests/fixtures/gaia/prepare_fixture_hub.py --skills-root <skills-root>
+python tests/fixtures/gaia/serve_fixtures.py --port 8765   # separate shell
+
+# Step 1: start the Agent UI backend (separate shell; NOT port 4001)
+python -m gaia.ui.server --port 4200 --host 127.0.0.1
+
+# Step 2: run every gaia category serially (never in parallel)
+for c in gaia_core gaia_memory gaia_rag gaia_files gaia_data gaia_web \
+         gaia_shell gaia_skills_lifecycle gaia_skills_tasks \
+         gaia_skills_capture gaia_honesty gaia_tool_selection \
+         gaia_code; do
+  gaia eval agent --agent-type gaia --category $c \
+    --backend http://127.0.0.1:4200 --budget 5.00 --exclude-tag live
+done
+
+# Step 3: generate this scorecard from ONE combined run's output dir
+python hub/agents/gaia/python/packaging/gen_scorecard.py \
+    --run-dir <run-dir-printed-by-the-eval> \
+    --model claude-haiku-4-5 \
+    --ctx-size 200000 \
+    --hardware "<hardware class>"
+```
+
+See [eval-scorecard docs](https://amd-gaia.ai/docs/reference/eval-scorecard) and the [`adding-eval-scorecard` skill](.claude/skills/adding-eval-scorecard/SKILL.md) for the full setup guide.
+
+## Environment
+
+| Field | Value |
+|-------|-------|
+| gaia_commit | d104b3b1 |
+| model | claude-haiku-4-5 |
+| ctx_size | 200000 |
+| hardware | developer workstation (x86-64, agent on the Claude API — no Lemonade, no AMD NPU/GPU inference) |
+| eval_model | claude-sonnet-4-6 |
+| note | HARNESS-VALIDATION card, pending first runner baseline: agent on claude-haiku-4-5 over the gaia_core category only (11 scenarios, tags local_blocked_no_embedder excluded). NOT the product baseline — that is Gemma-4-E4B-it-GGUF on the self-hosted runner, captured by the first full gaia_scorecard_refresh.yml dispatch (rebaseline=true). |
+
+## Category breakdown
+
+| Category | Total | Correct | Accuracy |
+|----------|-------|---------|----------|
+| gaia_core | 11 | 8 | 0.7273 |
+
+## Performance
+
+_Measured on the run environment above (model / hardware / gaia_commit / corpus size); the perf gate is report-only, so these are observed values, not pass/fail bars (see `tests/fixtures/email/perf_gate_thresholds.json`)._
+
+| Metric | Value |
+|--------|-------|
+| ttft_s | 7.254 |
+| throughput_tps | 38.7 |
+| total_input_tokens | 90710 |
+| total_output_tokens | 810 |
+| scenarios_with_data | 2 |