@amd-gaia/gaia 0.1.1 → 0.3.0

package/README.md

@@ -9,7 +9,13 @@ npx @amd-gaia/gaia
 That fetches the two binaries GAIA needs — the agent sidecar and the terminal UI —
 verifies both against a checksum manifest that ships inside this package, and drops
 you into the terminal UI. No Python to install, no repo to clone, no build step.
-Everything runs locally on your machine; nothing you type or index leaves it.
+Local inference is the default. The TUI can explicitly select Fireworks AI or AMD
+LLM Gateway through Lemonade; remote chat sends conversation history to that provider. The
+terminal UI's *launch* flag `--use-claude`, which would pin the Anthropic API for
+the life of the process, is refused for the agent this package installs. That is a
+limit on the launch flag, not on the agent: a caller can still ask for Anthropic
+per request with `/query`'s `provider` field (contract ≥ 2.14) — see
+[`SPEC.md` §5.5](./SPEC.md#55-other-transports).
 
 The terminal UI you get here is the published **`terminal-hub`** component — the
 exact same binary a full GAIA install runs as `gaia tui`, not a separate build. So
@@ -33,8 +39,10 @@ however you arrive at the terminal UI, it behaves identically.
 - **Node.js 18+** (for the built-in `fetch`).
 - **[Lemonade Server](https://amd-gaia.ai/docs/reference/dev)** running locally —
   it hosts the model the agent thinks with. GAIA tells you if it isn't up.
-- The `gaia` Python CLI on `PATH` for the daemon the terminal UI starts. Install it
-  with `curl -fsSL https://amd-gaia.ai/install.sh | sh` (Windows:
+- The `gaia` Python CLI **0.25.0 or newer** on `PATH` for the daemon the terminal
+  UI starts. An earlier core cannot run this agent, so the UI comes up with
+  nothing behind it. Install it with
+  `curl -fsSL https://amd-gaia.ai/install.sh | sh` (Windows:
   `irm https://amd-gaia.ai/install.ps1 | iex`).
 
 ## Supported platforms
@@ -75,14 +83,19 @@ npx @amd-gaia/gaia -- --debug
 
 Common options:
 
-| Flag                  | Meaning                                                       |
-| --------------------- | ------------------------------------------------------------- |
-| `--base-url <url>`    | Override the download base URL from `binaries.lock.json`       |
-| `--cache-dir <dir>`   | Where to cache the terminal UI binary                          |
-| `--sidecar-dir <dir>` | Where to install the agent sidecar (default `~/.gaia/agents/gaia`) |
-| `--platform <key>`    | Fetch for another platform (`fetch` only)                      |
-| `--force`             | Re-download even when a verified binary is already cached      |
-| `--port <n>`          | Sidecar bind port for `serve` (default `8141`)                 |
+| Flag                  | Meaning                                                       | Accepted by |
+| --------------------- | ------------------------------------------------------------- | ----------- |
+| `--base-url <url>`    | Override the download base URL from `binaries.lock.json`. Must be `https:` | `run`, `fetch`, `serve` |
+| `--allow-insecure-base-url` | Permit a non-`https` `--base-url` (a trusted local mirror) | `run`, `fetch`, `serve` |
+| `--sidecar-dir <dir>` | Where to install the agent sidecar (default `~/.gaia/agents/gaia`) | `run`, `fetch`, `serve` |
+| `--cache-dir <dir>`   | Where to cache the terminal UI binary                          | `run`, `fetch` |
+| `--force`             | Re-download even when a verified binary is already cached      | `run`, `fetch`, `serve` |
+| `--platform <key>`    | Fetch for another platform                                     | `fetch` |
+| `--component <name>`  | Fetch only `sidecar` or `tui`                                  | `fetch` |
+| `--port <n>`          | Sidecar bind port (default `8141`)                             | `serve` |
+
+A flag a command does not read is **refused**, not ignored — `gaia run --port
+9000` exits 2 rather than silently coming up on the default port.
 
 Set `DEBUG=gaia` for download, spawn, and sidecar output on stderr. Diagnostics
 never touch stdout, which the terminal UI owns.
@@ -92,6 +105,7 @@ never touch stdout, which the terminal UI owns.
 | What            | Path                                     |
 | --------------- | ---------------------------------------- |
 | Agent sidecar   | `~/.gaia/agents/gaia/gaia-agent[.exe]`   |
+| Install record  | `~/.gaia/agents/gaia/.installed`         |
 | Terminal UI     | `~/.gaia/npm-cache/gaia-<version>/gaia-tui[.exe]` |
 
 The sidecar goes into the GAIA daemon's own cache directory on purpose: the daemon
@@ -99,6 +113,13 @@ is what spawns and supervises it, and it does its own SHA-256 check on the way.
 putting an already-verified binary there we save a second download rather than
 racing one.
 
+The `.installed` record next to it is what the daemon and the terminal UI read to
+know the agent is installed — without it the UI would run the sidecar as its own
+stdio child instead of letting the daemon supervise it. It is rewritten even when
+the binary was already cached, so an install left by an earlier version repairs
+itself the next time you run. A `--platform` fetch stages a binary for a
+*different* machine, so it deliberately leaves no record.
+
 The terminal UI is installed as `gaia-tui`, **never** as `gaia` — a file named
 `gaia` in a cache directory would shadow the `gaia` shim npm puts on your `PATH`.
 
@@ -126,6 +147,12 @@ curl http://127.0.0.1:8141/health
 It waits for `GET /health`, checks the contract version, and tears the whole
 process tree down on Ctrl+C. See [`SPEC.md`](./SPEC.md) for the endpoints.
 
+The sidecar normally requires a per-session bearer token on `/v1/gaia/*`, but
+neither `serve` nor `startSidecar` mints one, so both leave it in dev mode — the
+token check off, `Host`/`Origin` still enforced. This agent has shell and file
+tools, so before you expose it to anything, supply a token of your own: see
+[`SPEC.md` §5.4](./SPEC.md#54-caller-authentication).
+
 ## Programmatic use
 
 ```ts
@@ -157,10 +184,16 @@ await shutdown(proc);
 **Reuse the same `session_id` for every turn in a conversation.** It is what
 lets a document you had it index, or a skill you had it load, survive to the
 next question — drop it (or mint a new one per call) and the agent still
-answers, but it forgets everything from the previous turn. See
+answers, but it forgets everything from the previous turn. An id must be 1–128
+characters from `A-Z a-z 0-9 . _ -` (a UUID works); anything else is a 400. See
 [`SPEC.md` §5.2](./SPEC.md#52-session_id-and-agent-retention) for the retry
 and eviction behavior.
 
+**Attaching files to a conversation? Send them in `documents`** (contract ≥
+2.16): the absolute paths, the whole set on every turn. The agent indexes any it
+has not read yet before answering, and can read those exact files and nothing
+else beside them. See [`SPEC.md` §5.1](./SPEC.md#51-endpoints).
+
 Every failure throws a typed error (`IntegrityError`, `PlatformError`,
 `HealthTimeoutError`, `VersionMismatchError`, `BinaryNotFoundError`) with a message
 that names what failed and what to do about it.
@@ -186,17 +219,31 @@ what it must hash to. The release pipeline regenerates it from the artifacts
 actually being served — the sidecars it just published, and the terminal-hub
 artifacts it downloaded and cross-checked against the hub's own recorded hashes.
 
-Between releases the lock carries `PENDING-replace-with-real-sha256` placeholders.
-**A placeholder blocks the fetch outright** — before any network call — so an
-unverifiable binary can never be downloaded, let alone executed. If you need to run
-against a locally built binary, build it yourself and point the lifecycle helpers
-at it directly; the fetcher will not be talked into it.
+A committed release's lock, like this one, always carries real hashes — the
+regenerate step in the release pipeline never publishes placeholders. Between
+releases, an unreleased working tree's lock instead carries
+`PENDING-replace-with-real-sha256` placeholders, and **a placeholder blocks the
+fetch outright** — before any network call — so an unverifiable binary can never
+be downloaded, let alone executed. If you need to run against a locally built
+binary, build it yourself and point the lifecycle helpers at it directly; the
+fetcher will not be talked into it.
 
 ## Links
 
 - Guide: <https://amd-gaia.ai/docs/guides/gaia>
 - Technical reference: [`SPEC.md`](./SPEC.md)
+- Eval scorecard: [`SCORECARD.md`](./SCORECARD.md) — the agent's measured
+  judged-scenario pass rate across the 13-category eval corpus, with the exact
+  recipe to reproduce it
 - Changes: [`CHANGELOG.md`](./CHANGELOG.md)
 - Issues: <https://github.com/amd/gaia/issues>
 
 MIT licensed. © 2024-2026 Advanced Micro Devices, Inc.
+
+## Developer engineering mode
+
+For GAIA contributors: with `--developer-mode` on, you can report a problem you hit
+while using GAIA and hand a snapshot of it, which you approve, to your own Claude
+Code or Codex app. That app diagnoses it and works on a fix in a separate git
+worktree. The mode is off unless you turn it on, and ordinary sessions never
+expose it. See the [usage guide](https://amd-gaia.ai/docs/guides/harness-engineering).

package/SCORECARD.md

@@ -0,0 +1,180 @@
+---
+schema_version: 1
+agent:
+  name: GAIA (flagship agent)
+  version: 0.1.1
+recipe:
+  dataset:
+    reference: eval/scenarios/
+    description: 'Judged multi-turn scenario corpus for the flagship gaia agent (13
+      gaia_* categories: core conversation/memory tiers through adversarial, RAG,
+      files, data, web, shell gate, skill lifecycle + all 12 skills, honesty floor,
+      tool selection, code index, skill capture); deterministic fixtures under tests/fixtures/gaia/,
+      planted-fact ground truth per eval/scenarios/GAIA_FIXTURE_VALUES.md'
+    size: 103
+  methodology: 'gaia eval agent --agent-type gaia: each scenario is driven against
+    the Agent UI backend (REST/SSE) by the eval driver and scored by the Claude judge
+    on planted-fact ground truth + success criteria. Aggregate = judged_pass_rate
+    over judged scenarios (PASS/FAIL/BLOCKED_BY_ARCHITECTURE; infra failures are excluded
+    from the rate and fail the run''s integrity gate instead). Reported secondaries
+    (weight 0): the judge''s 0-10 avg_score normalized to [0,1], and per-category
+    pass rates. Thresholds/enforcement: tests/fixtures/gaia/quality_gate_thresholds.json'
+  config:
+    harness: gaia eval agent
+    run_id: eval-20260824-104200
+    agent_type: gaia
+    eval_model: claude-sonnet-4-6
+    budget_per_scenario_usd: 5.0
+    categories:
+    - gaia_core
+  environment:
+    gaia_commit: d104b3b1
+    model: claude-haiku-4-5
+    ctx_size: 200000
+    hardware: developer workstation (x86-64, agent on the Claude API — no Lemonade,
+      no AMD NPU/GPU inference)
+    eval_model: claude-sonnet-4-6
+    note: 'HARNESS-VALIDATION card, pending first runner baseline: agent on claude-haiku-4-5
+      over the gaia_core category only (11 scenarios, tags local_blocked_no_embedder
+      excluded). NOT the product baseline — that is Gemma-4-E4B-it-GGUF on the self-hosted
+      runner, captured by the first full gaia_scorecard_refresh.yml dispatch (rebaseline=true).'
+results:
+  test_cases_run: 11
+  metrics:
+  - name: judged_pass_rate
+    value: 0.7272727272727273
+    weight: 1.0
+  - name: avg_score_normalized
+    value: 0.885
+    weight: 0.0
+  - name: gaia_core_pass_rate
+    value: 0.7273
+    weight: 0.0
+  breakdown:
+    per_category:
+    - category: gaia_core
+      total: 11
+      correct: 8
+      accuracy: 0.7273
+  performance:
+    ttft_s: 7.254
+    throughput_tps: 38.7
+    total_input_tokens: 90710
+    total_output_tokens: 810
+    scenarios_with_data: 2
+aggregate:
+  name: judged_pass_rate
+  formula: round(100 * sum(weight_i * value_i) / sum(weight_i), 2)
+  components:
+  - metric: judged_pass_rate
+    value: 0.7272727272727273
+    weight: 1.0
+  - metric: avg_score_normalized
+    value: 0.885
+    weight: 0.0
+  - metric: gaia_core_pass_rate
+    value: 0.7273
+    weight: 0.0
+  value: 72.73
+generated_at: '2026-08-24T11:25:31.271112+00:00'
+inherited_from: null
+---
+# GAIA (flagship agent) — Eval Scorecard v0.1.0
+
+**Aggregate score: 72.73** (out of 100)
+
+## Recipe
+
+| Field | Value |
+|-------|-------|
+| Dataset | [eval/scenarios/](eval/scenarios/) |
+| Description | Judged multi-turn scenario corpus for the flagship gaia agent (13 gaia_* categories: core conversation/memory tiers through adversarial, RAG, files, data, web, shell gate, skill lifecycle + all 12 skills, honesty floor, tool selection, code index, skill capture); deterministic fixtures under tests/fixtures/gaia/, planted-fact ground truth per eval/scenarios/GAIA_FIXTURE_VALUES.md |
+| Dataset size | 103 labeled examples |
+| Test cases run | 11 |
+| Methodology | gaia eval agent --agent-type gaia: each scenario is driven against the Agent UI backend (REST/SSE) by the eval driver and scored by the Claude judge on planted-fact ground truth + success criteria. Aggregate = judged_pass_rate over judged scenarios (PASS/FAIL/BLOCKED_BY_ARCHITECTURE; infra failures are excluded from the rate and fail the run's integrity gate instead). Reported secondaries (weight 0): the judge's 0-10 avg_score normalized to [0,1], and per-category pass rates. Thresholds/enforcement: tests/fixtures/gaia/quality_gate_thresholds.json |
+
+## Metrics
+
+  - **judged_pass_rate**: 0.7273 × 1.0
+  - **avg_score_normalized**: 0.8850 × 0.0
+  - **gaia_core_pass_rate**: 0.7273 × 0.0
+
+## Aggregate score recomputation
+
+Formula: `round(100 × Σ(weightᵢ × valueᵢ) / Σ(weightᵢ), 2)`
+
+Worked example:
+
+```
+round(100 × ((0.7273 × 1.0) + (0.8850 × 0.0) + (0.7273 × 0.0)) / 1.0, 2) = 72.73
+```
+
+A reader can reproduce this value from the `aggregate.components` in the front
+matter alone — no eval-harness access needed.
+
+## Reproduction
+
+Run the following commands from the repository root:
+
+```sh
+# Prerequisites: install the eval extras + this repo's chat/gaia hub
+# packages, start a Lemonade Server with the model on AMD Ryzen AI
+# hardware, and have the Claude Code CLI on PATH (the eval driver).
+uv pip install -e ".[dev,eval,ui,api]" -e hub/agents/chat/python -e hub/agents/gaia/python
+gaia init   # installs and starts Lemonade Server; it must stay running
+
+# Step 0: stage fixtures + start the fixture server (see
+# tests/fixtures/gaia/README.md for the staging contract)
+python tests/fixtures/gaia/prepare_fixture_hub.py --skills-root <skills-root>
+python tests/fixtures/gaia/serve_fixtures.py --port 8765   # separate shell
+
+# Step 1: start the Agent UI backend (separate shell; NOT port 4001)
+python -m gaia.ui.server --port 4200 --host 127.0.0.1
+
+# Step 2: run every gaia category serially (never in parallel)
+for c in gaia_core gaia_memory gaia_rag gaia_files gaia_data gaia_web \
+         gaia_shell gaia_skills_lifecycle gaia_skills_tasks \
+         gaia_skills_capture gaia_honesty gaia_tool_selection \
+         gaia_code; do
+  gaia eval agent --agent-type gaia --category $c \
+    --backend http://127.0.0.1:4200 --budget 5.00 --exclude-tag live
+done
+
+# Step 3: generate this scorecard from ONE combined run's output dir
+python hub/agents/gaia/python/packaging/gen_scorecard.py \
+    --run-dir <run-dir-printed-by-the-eval> \
+    --model claude-haiku-4-5 \
+    --ctx-size 200000 \
+    --hardware "<hardware class>"
+```
+
+See [eval-scorecard docs](https://amd-gaia.ai/docs/reference/eval-scorecard) and the [`adding-eval-scorecard` skill](.claude/skills/adding-eval-scorecard/SKILL.md) for the full setup guide.
+
+## Environment
+
+| Field | Value |
+|-------|-------|
+| gaia_commit | d104b3b1 |
+| model | claude-haiku-4-5 |
+| ctx_size | 200000 |
+| hardware | developer workstation (x86-64, agent on the Claude API — no Lemonade, no AMD NPU/GPU inference) |
+| eval_model | claude-sonnet-4-6 |
+| note | HARNESS-VALIDATION card, pending first runner baseline: agent on claude-haiku-4-5 over the gaia_core category only (11 scenarios, tags local_blocked_no_embedder excluded). NOT the product baseline — that is Gemma-4-E4B-it-GGUF on the self-hosted runner, captured by the first full gaia_scorecard_refresh.yml dispatch (rebaseline=true). |
+
+## Category breakdown
+
+| Category | Total | Correct | Accuracy |
+|----------|-------|---------|----------|
+| gaia_core | 11 | 8 | 0.7273 |
+
+## Performance
+
+_Measured on the run environment above (model / hardware / gaia_commit / corpus size); the perf gate is report-only, so these are observed values, not pass/fail bars (see `tests/fixtures/email/perf_gate_thresholds.json`)._
+
+| Metric | Value |
+|--------|-------|
+| ttft_s | 7.254 |
+| throughput_tps | 38.7 |
+| total_input_tokens | 90710 |
+| total_output_tokens | 810 |
+| scenarios_with_data | 2 |