@flame0510/project-aether 1.9.1 → 1.11.0

package/docs/ARCHITECTURE.md

@@ -1,7 +1,7 @@
 # Rev4a Architecture — Design & Vision
 
 > **Status:** Active — `main` branch
-> **Last updated:** 2026-09-26
+> **Last updated:** 2026-10-01
 > **Goal:** Transform Rev4a from a monitoring dashboard into a central orchestrator for a distributed multi-container agency.
 
 ---
@@ -72,7 +72,7 @@
 
 **Credentials vault:** The `/credentials` page stores third-party service tokens in `credentials.db` and installs them into containers on explicit sync. Provider API keys live separately in `provider-keys.json`, managed from the Gateway UI.
 
-**Container management UI:** The `/containers` page lists all Docker containers, running or stopped, with status, image, IP and ports, and opens a web terminal into any running one. Fully functional. (Machine CPU, RAM and disk are on `/system`.)
+**Container management UI:** The `/containers` page lists all Docker containers, running or stopped, with status, image, IP and ports, and opens a web terminal into any running one. Fully functional. Each running container's CPU, memory and processes are on its row, with a link to its charts on `/system`, which also shows the machine's CPU, RAM and disk.
 
 **Workspace API:** Lazy-loaded file tree explorer with real-time reads (no caching), supports both host and container workspaces via `docker exec`.
 
@@ -136,7 +136,8 @@ The central container, running the Next.js dashboard + orchestration API.
   hand (identity, `enabled`, `deprecated`, and an optional `info` block for what no API
   publishes: the vendor's size claim, benchmarks, the docs link, notes).
   `model-pricing.json` and `model-details.json` are **generated** by
-  `npm run refresh:pricing` (`scripts/refresh-model-pricing.mjs`): prices from OpenRouter,
+  `npm run refresh:pricing` (`scripts/refresh-model-pricing.mjs`): prices from OpenRouter
+  (cache rates included; direct vendors' prices by hand from their pricing pages),
   and per model the description, architecture and benchmarks from OpenRouter plus the size,
   weight mix, licence and dates from the Hugging Face card. Two scripts back the curation
   of `info`: `npm run info:suggest` (`scripts/model-info-suggest.mjs`) prints the candidate
@@ -577,8 +578,12 @@ The Rev4a daemon (`daemon.js`) is a standalone Node.js process that bridges the
 3. Emit `spawn` / `complete` / `fail` / `spawn_timeout` events into the `events` table
 4. Sample the host machine (CPU, RAM, swap, storage) every 30 s into `metrics.db`, on a
    timer of its own
-5. Detect anomalies (CPU > 85%, RAM > 90%, a disk > 90%) and record them as events
-6. Manage DB lifecycle (WAL mode; events.db checkpointed after each poll cycle)
+5. Sample what each container consumes every 60 s, and the disk Docker holds every 10 min,
+   into `metrics.db` (see [Container Consumption](#container-consumption))
+6. Detect anomalies (CPU > 85%, RAM > 90%, a disk at 90%) and record them as events
+7. Read what each running agent spent, as its own OpenClaw priced it, every 5 min into
+   `costs.db`, and each vendor's balance or usage every hour (see [Agent Costs](#agent-costs))
+8. Manage DB lifecycle (WAL mode; events.db checkpointed after each poll cycle)
 
 ### Poll Interval
 
@@ -586,33 +591,12 @@ A fixed 30 s timer, whether or not sessions are working. When the OpenClaw CLI i
 the host, the session poll logs it once and stays idle; the machine metrics keep being
 sampled, since they have their own timer.
 
-### Cost Estimation
+### Session costs
 
-Costs are estimated from token counts using per-model pricing:
-
-| Model key | Match strategy | In (per 1M) | Out (per 1M) |
-|---|---|---|---|
-| `claude-sonnet-4` | substring | $3.00 | $15.00 |
-| `claude-opus-4` | substring | $15.00 | $75.00 |
-| `gpt-5-mini` | substring | $0.15 | $0.60 |
-| `codex` | substring | $3.00 | $15.00 |
-| `gemini` | substring | $0.075 | $0.30 |
-| `flash` | substring | $0.075 | $0.30 |
-| `deepseek` | substring | $0.55 | $2.19 |
-| `default` | fallback | $3.00 | $15.00 |
-
-**Aliases** (resolved before substring match):
-
-| Alias | Resolves to |
-|---|---|
-| `cheap` | `flash` |
-| `fast` | `claude-sonnet-4` |
-| `big` | `claude-opus-4` |
-| `coder` | `codex` |
-| `pro` | `gemini` |
-| `reason` | `deepseek` |
-
-Formula: `cost = (tokens_in / 1_000_000 * price_in) + (tokens_out / 1_000_000 * price_out)`
+The session poll writes no cost: `sessions.cost_usd` stays 0. It used to be estimated from a
+fixed 2025 rate table (`$3 / $15` per 1M for anything unknown), which the dashboard showed as
+spend. What the agents actually spend is priced by each agent's own OpenClaw and read by the
+Agent Costs timer — see [Agent Costs](#agent-costs).
 
 ### Upsert Logic
 
@@ -647,6 +631,63 @@ taken at load), then one every 30 s. Samples older than 30 days are pruned at ev
 it through `lib/metrics-db.ts` (read-only, `null` when there is nothing to read);
 `GET /api/metrics` serves the System page (`/system`) and the dashboard's Machine card.
 
+### Container Consumption
+
+The System page says *that* the machine is busy; this says *who*. On a timer of its own the
+daemon reads every running container from the Docker Engine API — `GET /containers/{id}/stats?stream=false&one-shot=true`,
+10–20 ms a container, where `docker stats --no-stream` waits a second each and returns formatted
+strings — and writes `container_metrics` into `metrics.db` (schema in `docs/dev/DATABASE.md`):
+
+- **CPU** from the cumulative CPU time (ns): the difference between two samples over the
+  elapsed time, in cores; the first sample after a start only primes the counters. A counter
+  that goes backwards (the container restarted) gives no figure for that sample, never a spike.
+- **Memory** is the working set — usage minus the inactive page cache (`inactive_file` on
+  cgroup v2, `total_inactive_file` on v1), what `docker stats` shows.
+- **Processes, network and block I/O** as gauges and per-second rates. A container that stops
+  leaves a gap; one that is gone takes its counters with it.
+- **Names:** an agent's `AGENT_NAME` (read once per container id from its inspect), else the
+  container's name. Every running container is sampled — agents, and the rest as "container".
+- **Docker's disk** (`/system/df`, every 10 min): each named volume — which gives each agent's
+  storage, through the volume names its mounts list —, each writable layer, the images and the
+  build cache with what no container/build uses. Sizes and "unused" are readings, not advice.
+- The shares need Docker's own cores and memory (`/info`, re-read every 10 min): on Docker
+  Desktop those are its VM's (8 GB of a 16 GB Mac), on the VPS the machine's.
+
+The socket path rules (env, `/var/run`, Docker Desktop's `~/.docker/run`, `docker context`) live
+once, in `lib/docker-socket-path.js`, required by both the daemon and `lib/docker-socket.ts`. A
+Docker that is down is logged once and recovered from by itself; a failed disk reading leaves the
+previous one in place. `lib/container-metrics.ts` reads it all for `GET /api/metrics/containers`
+(the System page's *Agents and containers* section, the agent panel's RESOURCES, the Containers
+page). Anomalies per container (processes, sustained memory) are not raised yet.
+
+### Agent Costs
+
+Rev4a computes no cost. Every model in the `rev4a` provider block synced into the agents
+carries its price (`cost`, from `model-pricing.json` through `priceAt()`), so each agent's
+OpenClaw prices every call itself — per call, session and day, cache split out. A vendor
+priced by time of day (DeepSeek off-peak) is re-synced at each change of band by
+`lib/price-schedule-sync.ts`, started from `instrumentation.ts`; OpenClaw keeps the cost
+recorded at call time. Details in `docs/dev/GATEWAY.md` (Costs).
+
+Every 5 min (first pass 20 s after start) the daemon first wakes every running agent with one
+short call — OpenClaw rebuilds its transcript index in about a minute after a restart or a
+price change, and woken together the agents rebuild in parallel — then asks each, one at a
+time, for its figures over the last 30 UTC days — `usage.cost` and
+`sessions.usage` through `docker exec … openclaw gateway call` on the agent's port 3000,
+the token resolved inside the container — and rewrites that window in `costs.db` (schema
+in `docs/dev/DATABASE.md`). An answer from an index OpenClaw is still rebuilding is never
+stored. The readers — `GET /api/costs/usage` (the Costs page, `/costs`), `GET /api/costs` and
+the SSE stream (the dashboard), the system-health cost check, and `GET /api/costs/agent` (the
+agent panel) — share one read-only handle (`lib/costs-db.ts`) and one set of spend queries
+(`lib/agent-costs.ts`); the vendor readings are read by `lib/cost-reconciliation.ts`.
+
+After a pass, at most once an hour, the daemon also reads each vendor's own figure —
+DeepSeek's balance, OpenRouter's lifetime usage (read-only calls, never billed) — and stores
+it next to the agents' cumulative priced total for that vendor's models, in the same row
+(`vendor_balance`). Between two rows, what the vendor billed and what the agents priced
+cover the same interval: the Costs page's *Against the bill* (`lib/cost-reconciliation.ts`).
+`costs.db` is opened in a try: if it cannot be, collection is off (`[COSTS] disabled`).
+
 ### Anomaly Detection
 
 After each sample the daemon inserts a `system_anomaly` event into `events.db`

package/docs/DESIGN-SYSTEM.md

@@ -2,7 +2,7 @@
 
 Source of truth: `app/design-system.ts` + CSS tokens in `app/globals.css`.
 
-> **Last updated:** 2026-09-26
+> **Last updated:** 2026-09-27
 
 ---
 
@@ -82,7 +82,7 @@ then use it — a one-off copy of markup is the thing this section exists to pre
 5. **CSS vars over hardcoded colors.** Never inline `#22c55e`, `#ef4444`, `#888` — use the CSS token.
 6. **Loading feedback.** Use `loading` prop on `Button` (shows inline spinner). Do not render separate loader elements.
 7. **Minimal animation.** Transitions on hover/active (0.1s–0.12s), spin on loading spinner. No bounce, shake, or decorative animations.
-8. **Data display: violet for the data, status colours for levels.** A chart series is `var(--violet)` (2px line, 10% wash), grid and axis text are `var(--border)` / `var(--text-dim)`, never the series colour. A `Meter` fill is violet when fine and `var(--yellow)` / `var(--red)` at its warning / critical level, with the level also written out — status never rests on colour alone. Focus on a chart is a 1px `outline` in `var(--violet-dim)`, not a shadow. See `TimeSeriesChart` and `Meter` in `docs/FRONTEND-ARCHITECTURE.md`.
+8. **Data display: violet for the data, status colours for levels.** A chart series is `var(--violet)` (2px line, 10% wash), grid and axis text are `var(--border)` / `var(--text-dim)`, never the series colour. A `Meter` fill is violet when fine and `var(--yellow)` / `var(--red)` at its warning / critical level, with the level also written out — status never rests on colour alone. Focus on a chart is a 1px `outline` in `var(--violet-dim)`, not a shadow. A figure that is only a share of a whole (the Costs page's agents and models) uses the same `Meter`, always at its `ok` level: it is a proportion, not a warning. See `TimeSeriesChart` and `Meter` in `docs/FRONTEND-ARCHITECTURE.md`.
 
 ### React hook
 

package/docs/FRONTEND-ARCHITECTURE.md

@@ -1,6 +1,6 @@
 # Rev4a Frontend Architecture
 
-> **Last updated:** 2026-09-26
+> **Last updated:** 2026-10-01
 
 ## Layering
 
@@ -39,7 +39,7 @@ All shared UI primitives live in `app/components/ui/` and are exported from `app
 | `Metric` | `Metric.tsx` | Metric card with title, value, subtitle, tone, and a `size` (`md` default, `sm` for a grid inside a dialog: smaller value, tighter spacing). |
 | `StatusCard` | `StatusCard.tsx` | Health/status report card. |
 | `Meter` / `meterLevel` | `Meter.tsx` | Horizontal 0-100 % fill whose colour carries the level (`ok` accent, `warning` yellow, `critical` red); the level is also written out beside it, never colour alone. `meterLevel(percent, warning, critical)` picks the level. ARIA `role="meter"`. |
-| `TimeSeriesChart` | `TimeSeriesChart.tsx` | One series over time in plain SVG measured to its container (text never scaled): a 2px accent line for the value, a 10 % wash up to the bucket's peak, a hairline grid at 0 / 50 / 100 %, the latest value labelled at the line's end. Points further apart than 1.5 buckets are not joined (a gap means no samples). Hover or keyboard focus (← → Esc) shows a crosshair snapped to the nearest point with value and peak; a **Table** disclosure holds the same data. `stale` dims the previous render while a new range loads. Used by the System page. |
+| `TimeSeriesChart` | `TimeSeriesChart.tsx` | One series over time in plain SVG measured to its container (text never scaled): a 2px accent line for the value, a 10 % wash up to the bucket's peak, a hairline grid at 0 / 50 / 100 %, the latest value labelled at the line's end. Points further apart than 1.5 buckets are not joined (a gap means no samples). Hover or keyboard focus (← → Esc) shows a crosshair snapped to the nearest point with value and peak; a **Table** disclosure holds the same data. `stale` dims the previous render while a new range loads. `yMax` (a number, or `'auto'`: a rounded ceiling just above the highest value or peak — for a series that is a small share of its natural scale, like one agent's CPU) / `unit` scale and label a percentage by default; `formatValue` replaces the label format everywhere (axis, end label, tooltip, table) — the Costs page passes dollars. The left and right margins grow with the longest axis label and the end label, so a longer format is never cut. Used by the System and Costs pages. |
 | `Surface` | `Surface.tsx` | Shared panel/card surface, variant prop. |
 | `Page` / `PageHeader` | `Page.tsx` | Full-page layout shell. |
 | `Icons` | `Icons.tsx` | SVG icons (`EyeIcon`, `EyeOffIcon`), 16/20px shared. |
@@ -47,7 +47,12 @@ All shared UI primitives live in `app/components/ui/` and are exported from `app
 | `UpdateSection` | `app/agents/UpdateSection.tsx` | OPENCLAW VERSION section of the agent detail panel: the version the agent runs, **Update to <version>** when a newer supported version is downloaded (confirm modal, disabled while the agent is stopped), the running update's steps with backup progress, the outcome, and **Roll back to <version>** after an update. Polls `/api/agents/[id]/update` every 2 s while an update or rollback runs; one action at a time (keyed busy state). |
 | `BackupSection` | inline in `app/agents/PageClient.tsx` | BACKUP section of the agent detail panel, on the cold backup and the restore. **Backup Now** starts `POST /api/agents/[id]/cold-backup`; while the job runs a banner shows the file and its live percent with **Cancel** (`DELETE /cold-backup`). **Restore** (after a confirm) starts `POST /restore` and a banner shows `Restoring <file>…` (no percent: the extract is a single `tar xzf`, and there is no Cancel). Both sections poll their `GET` every 2 s while running, and on mount pick up a job that is already running — a backup lives in a Docker helper, a restore in `agent_restores`, so reloading the page or navigating away never loses them nor allows a second one (the server answers 409 anyway). Delete per row; all actions disabled while one runs; keyed busy state `{ kind, file }` so only the row in action shows the spinner. On the agent list, an activity Badge (fed by `/api/agents/activity-summary`, polled at 2 s only while something runs, otherwise riding the 15 s list poll) reads `BACKUP nn%`, `RESTORING`, `RECREATING`, `EDITING` or `UPDATING`. |
 | `RecreateSection` | inline in `app/agents/PageClient.tsx` | RECREATE section of the agent detail panel. **Recreate Container** starts `POST /api/agents/[id]/recreate` (202) after a confirm; a banner then shows the phase — *Backing up … nn%* while the cold backup runs, *Recreating container…* while the container is rebuilt and the gateway starts. The section polls `GET /recreate` every 2 s, and on mount picks up a recreate that is already running, so a reload or navigation never loses it; it refetches the agent once the job reports `done`. |
-| System page | `app/system/PageClient.tsx` | `/system`: CPU, memory and storage of the host, from `GET /api/metrics`. Three cards (CPU with cores and load; memory with available and swap; one `Meter` per filesystem with its roles) — thresholds 85/95 % for CPU and memory, 80/90 % for storage — then a range switch (`Tabs`: 1h · 24h · 7d · 30d) scoping the charts below it: CPU and memory average with peak, one chart per filesystem. Polls every 30 s with an `AbortController` ref (a range change aborts the previous fetch). Times in the configured Rev4a timezone (`useRev4aTimezone`). Before the first answer the cards and charts are skeletons shaped like them (`app/system/SystemSkeleton.tsx`, also the route's `loading.tsx`), so nothing jumps; the hostname line is cut with an ellipsis and never widens the page. Says *No samples yet* before the daemon's first sample and *Not collecting* when the latest sample is older than three intervals. Two columns of charts from lg (992px) up, one below. |
+| `Accordion` | `Accordion.tsx` | A row that opens: the header is a real `<button type="button">` (`aria-expanded`, `aria-controls` while open) carrying a `title`, `summary` figures and an optional muted `detail` line, so the answer to "what is this now" is on screen closed; the body holds what takes room and is **mounted only while open**, so a chart inside fetches nothing while closed. Controlled (`open` / `onToggle`: the parent can deep-link or fetch on it), optional `id` for a link to land on. Square, flat, tokens only. On mobile the summary wraps under the title. Used by the System page's agents. |
+| System page | `app/system/PageClient.tsx` | `/system`: CPU, memory and storage of the host, from `GET /api/metrics`. Three cards (CPU with cores and load; memory with available and swap; one `Meter` per filesystem with its roles) — thresholds 85/95 % for CPU and memory, 80/90 % for storage — then a range switch (`Tabs`: 1h · 24h · 7d · 30d) scoping the charts below it: CPU and memory average with peak, one chart per filesystem. Polls every 30 s with an `AbortController` ref (a range change aborts the previous fetch). Times in the configured Rev4a timezone (`useRev4aTimezone`). Before the first answer the cards and charts are skeletons shaped like them (`app/system/SystemSkeleton.tsx`, also the route's `loading.tsx`), so nothing jumps; the hostname line is cut with an ellipsis and never widens the page. Says *No samples yet* before the daemon's first sample and *Not collecting* when the latest sample is older than three intervals. Two columns of charts from lg (992px) up, one below; an odd last chart spans both columns instead of leaving a hole. Under them, **Agents and containers** (`app/system/AgentsSection.tsx`, heading in the accent eyebrow with a rule above — `system-section-heading`): *Docker storage* (images, volumes, writable layers, build cache, each with what no container uses — wording neutral on purpose: an unused image may be a rollback target, an unused volume the cold backups), then **one `Accordion` per container**, sorted by CPU now (`GET /api/metrics/containers`, polled every 30 s with an `AbortController` ref). Closed: name, an *agent* / *container* pill, *stopped* when it is — or *no data* when Docker still lists it but its stats call fails — and CPU (share of Docker's cores), memory, storage, PIDs, with average · peak over the range tabs, cores, network and memory share on a muted line. Open (`AgentCharts.tsx`): the same three charts as the machine — CPU (share), Memory, Storage — for that container, fetched when it opens (`?container=`), polled every 30 s, aborted on close or range change; each with `yMax="auto"` (an agent is 0.01–2 % of Docker's cores) and sizes in MB/GB, on `subtle` surfaces inside the accordion. A last row, *Everything else*, is the machine minus the containers. `/system?agent=<container>` opens one and scrolls to it. Then **Recent alerts** (`RecentAlerts.tsx`, `GET /api/metrics/alerts`): the machine's latest 10 anomalies with time (configured timezone), metric pill and message. The section has its own skeleton (`SystemAgentsSkeleton`, also in the route's `loading.tsx`). |
+| Costs page | `app/costs/PageClient.tsx` | `/costs`: what each agent spent, as its own OpenClaw priced it, from `GET /api/costs/usage`. A range switch (`Tabs`: Today · 7 days · 30 days, UTC days) scopes everything below it: three cards (spent with tokens and days; the cost split into input / output / cache read / cache write; the tokens with the share of input served from cache), the daily spend (`TimeSeriesChart` with dollars, not on Today), then *By agent* and *By model* side by side — each row a name and figure over a share `Meter`, the pattern of the System storage card — *Against the bill* (per vendor, what it billed, from balance or usage readings, next to what the agents priced over the same readings, top-ups apart), and the most expensive sessions. Warnings above: *No figures yet* before the daemon's first pass, *Not current* for agents still being read whose last read failed or is older than three intervals (a stopped or deleted agent keeps its figures unflagged), and *Calls without a price* by model (a model with no price is also marked in *By model*, so a $0.00 is never read as free). The header shows the DeepSeek band now (peak / off-peak, until when, UTC) when a DeepSeek model is on offer. Polls every 60 s with an `AbortController` ref. Skeletons shaped like the content (`app/costs/CostsSkeleton.tsx`, also the route's `loading.tsx`). Costs under a cent keep four decimals. |
+| Agent costs | `app/agents/CostSection.tsx` | The COSTS section of the agent panel, after MODEL: today (UTC), last 7 and 30 days with tokens, the top model, calls without a price, when the agent was last read (a note when not recently: a stopped agent keeps its last figures), and a link to `/costs`. From `GET /api/costs/agent`, every 60 s with an `AbortController` ref. |
+| Agent resources | `app/agents/ResourceSection.tsx` | The RESOURCES section of the agent panel, after COSTS: CPU and memory now and over the last 24 h, processes, network, storage (volume and layer), from `GET /api/metrics/containers`, every 60 s with an `AbortController` ref, reset on agent switch; *Stopped* when the agent is not running, *Not collecting* when the newest sample is older than three intervals (the "now" figures then stand down), a refresh-failure line while the last answer stays, and a link to its charts on `/system?agent=<container>` — the history lives in one place. Rows are the shared `PanelRow` (`app/agents/PanelRow.tsx`, also used by COSTS). |
+| Containers page | `app/containers/ContainersClient.tsx` | `/containers`: one card per Docker container; a running one the daemon has sampled adds a line — CPU share, memory, PIDs — and *Charts →* to its accordion on `/system`. The figures are a bonus: a daemon that has not sampled yet, or a failed request, leaves the list as it was. |
 | Machine card | `app/components/SystemCockpit.tsx` | On the dashboard, next to Active sessions: CPU, RAM and the fullest disk now, each figure coloured only when past its threshold (same thresholds as the System page), the issues named in words, and a link to `/system`. It says *No samples yet* before the first sample, *Not collecting* when the newest is stale, and *Metrics unavailable* when a refresh fails (keeping the last figures). A `Surface`, not a `Metric`: a danger `Metric` paints its whole value red. |
 | Container terminal | `app/containers/terminal/[id]/TerminalClient.tsx` | Full-screen shell into a container: xterm.js 6 (`@xterm/xterm` + `addon-fit` refitted by a `ResizeObserver`, `addon-web-links`), loaded client-side only, palette from the design tokens. Keystrokes are sent raw (`onData`, `onBinary`), the size on connect and whenever the columns or rows change. A bar with `ui-kicker`, the container name, a status `Badge` (Connecting / Connected / Reconnecting / Closed) and `Button`s **Reconnect** / **Close**; styles are the `terminal-page` classes. Session id per container in `sessionStorage` (a UUID from `getRandomValues`: `randomUUID` is missing over plain HTTP), so a reload resumes the shell; after a drop it reconnects with `resume=1` and is told when the shell is gone; retries with backoff (six failed opens, about 40 s, then a message) and stops on the server's final close codes with the reason shown. **Close** while reconnecting still ends the waiting shell. Protocol and sessions: `docs/CONTAINER-TERMINAL.md`. |
 | `ImageDownloadBanner` | `app/agents/ImageDownloadBanner.tsx` | Agent image banner on the Agents page. Polls `/api/agents/image-status` every 2 s; offers **Download Image** when no supported version is downloaded, **Download <version>** when the registry publishes a newer one, and while a download runs shows the percent of layers finished and the latest line of docker's output (the bar is indeterminate until a percent can be computed). **Cancel** (`DELETE /api/agents/download-image`, own loading state) appears once `image-status` reports the download running; before that the button reads *Starting…* and is disabled. Every outcome — ready and cancelled for 3 s, a failure until the next action — comes from the server's `lastResult`, so one that ended while the page was closed or reloading is still reported if it is less than 30 s old (aged with `serverTime`). Each outcome is announced once per page load, although the Agents page mounts the banner in three places (mobile list, mobile detail, desktop). Downloading changes no agent. |
@@ -65,6 +70,12 @@ All shared UI primitives live in `app/components/ui/` and are exported from `app
 | `Badge` | `Badge.tsx` | Inline status tag with tone variants (success, danger, warning, neutral). Used for channel chips, pairing labels, error/success messages. |
 | `Skeleton` + shape helpers | `app/components/Skeleton.tsx` | Shimmer placeholders. `Skeleton` is the primitive; the rest mirror one specific layout each: `TreeSkeleton`, `CodeSkeleton`, `CronJobsSkeleton`, `CronRunsSkeleton`, `CredentialCardsSkeleton`, `AgentSyncRowsSkeleton`, `SkeletonLines`, `SkeletonMetric`, `CardRowSkeleton`. A route's `loading.tsx` sits in `RouteLoadingShell` (full width up to `maxWidth`, centred): the app shell's content area is a column flexbox, and a `margin: 0 auto` child without `width: 100%` shrinks to its content — every loader did, and its rows came out as wide as its 120 px title (measured: 120 px instead of 1040). `ListPageSkeleton` (a title over rows) is the list pages' loader. Create Agent (`app/agents/create/loading.tsx`) and the container terminal (`app/containers/terminal/[id]/loading.tsx`) have their own, shaped like the page, instead of inheriting the Agents / Containers rows; the Agents page prefetches `/agents/create` (its **+ New** uses `router.push`, which prefetches nothing) and the Containers **Terminal** is a `Link`, so both open without a full reload. A shape helper must match the real markup it stands in for — same row structure, same paddings, same element count where the count is known — so nothing reflows when data replaces it. Inside text (a `<p>`, a `<span>`) pass `as="span"`: a `<div>` there is invalid HTML, and the parser splits the paragraph in the server-rendered page. The System page keeps its own shapes in `app/system/SystemSkeleton.tsx`, each shimmer inside the class of the text it replaces so the line boxes are the real ones — measured to 0 px of movement on desktop and mobile. |
 
+The System page's container rows distinguish a recent Docker listing from fresh resource
+statistics. A row seen in Docker without figures says *no data*. When the newest statistics
+sample ages past three intervals, current CPU, memory and processes disappear; an older row
+says *not current*. The page advances its clock at each poll and recomputes age from
+`sampled_at`, so repeated failed refreshes cannot leave a previous answer labelled as current.
+
 ### Rules
 
 1. **Prefer existing primitives over new ad-hoc markup.** Every new component starts from the shared catalog.

package/docs/REV4A.md

@@ -1,6 +1,6 @@
 # Rev4a — VPS Dashboard
 
-> **Last updated:** 2026-09-26
+> **Last updated:** 2026-10-01
 
 A Next.js 16 dashboard for monitoring and managing the OpenClaw ecosystem.
 
@@ -99,10 +99,11 @@ When environment variables are saved via the Config page (Save & Restart):
 
 | Route | Description |
 |---|---|
-| `/` | System overview — health, sessions, the machine's CPU / RAM / disk (links to `/system`), cost, live feed |
+| `/` | System overview — health, sessions, the machine's CPU / RAM / disk (links to `/system`), what the agents spent today (as on `/costs`), live feed |
 | `/agents` | Running agents (Docker containers with `AGENT_ID`), gateway token management, agent creation wizard, channel manager (Telegram pairing) |
-| `/containers` | All Docker containers on the host |
-| `/system` | The host machine: CPU, memory and swap, storage per filesystem — current values and 1h / 24h / 7d / 30d history (from `metrics.db`, sampled by the daemon every 30 s) |
+| `/containers` | All Docker containers on the host, each running one with its CPU, memory and processes now (from the daemon's container samples) and a link to its charts |
+| `/costs` | What each agent spent, as its own OpenClaw priced it with the rates Rev4a syncs: today / 7 / 30 UTC days, by day, by agent, by model, the most expensive sessions, the calls that could not be priced, the DeepSeek band now, and a check against what DeepSeek and OpenRouter actually billed (read by the daemon every 5 min into `costs.db`; each agent's own figures are also in its panel, COSTS) |
+| `/system` | The host machine: CPU, memory and swap, storage per filesystem — current values and 1h / 24h / 7d / 30d history (from `metrics.db`, sampled by the daemon every 30 s); **Agents and containers**: what Docker holds on disk, and one row per container — CPU, memory, storage, processes — that opens to its own CPU / Memory / Storage charts (sampled every 60 s; a recently seen row without current statistics says "no data", and stale current figures are hidden; each agent's numbers are also in its panel, RESOURCES); and the machine's recent alerts |
 | `/wizard` | First-run setup wizard (Welcome → Providers → Agent → Ready) |
 | `/workspace` | File explorer with tree view + editor — VPS host or container workspaces |
 | `/lineage` | Agent lineage / orchestration tree |
@@ -307,7 +308,7 @@ Store third-party service credentials and install them into agent containers.
 ### Container management (`/containers`)
 View all running Docker containers with:
 - Name, image, status, ports, uptime
-- Resource usage (CPU, memory)
+- Resource usage now (CPU share, memory, processes) for each running container, with *Charts →* to its history on `/system`
 - Quick links to agent control UIs
 
 ### Agent management (`/agents`)

package/docs/dev/API-REFERENCE.md

@@ -1,6 +1,6 @@
 # Rev4a API Reference
 
-> **Last updated:** 2026-09-26
+> **Last updated:** 2026-10-01
 
 All routes are under `/api/`. Authentication is required on every endpoint
 unless otherwise noted.
@@ -469,12 +469,12 @@ Returns all sessions (up to 2000), newest first, joined with lineage labels.
     "model": "openai-codex/gpt-5.4",
     "tokens_in": 12500,
     "tokens_out": 3200,
-    "cost_usd": 0.085,
+    "cost_usd": 0,
     "status": "idle",
     "task_preview": "Run hygiene audit...",
-    "started_at": 1749200000,
+    "started_at": 1749200000000,
     "ended_at": null,
-    "updated_at": 1749201000,
+    "updated_at": 1749201000000,
     "trello_card_url": null,
     "lineage_label": null,
     "lineage_agent_name": null
@@ -517,8 +517,76 @@ One session with its events and any child sessions.
 
 ## Stats & Costs
 
+Agent costs come from `costs.db`: what each agent's own OpenClaw priced, copied by the
+daemon (the queries are in `lib/agent-costs.ts` (spend) and `lib/cost-reconciliation.ts` (`vendor_balance`), on a read-only handle from `lib/costs-db.ts`). `GET /api/costs/usage` feeds the Costs
+page, `GET /api/costs` the dashboard, `GET /api/costs/agent` the agent panel. `GET
+/api/stats` and `GET /api/stats-since` still read the `sessions` table of `events.db`,
+whose `cost_usd` is no longer estimated (always 0).
+
+### `GET /api/costs/usage?range=today|7d|30d`
+What the agents spent in a range of UTC days, today included (default `7d`), read from
+`costs.db` (written by the daemon every 5 minutes, see `docs/dev/DATABASE.md`). Rev4a
+computes no cost: every figure is the one OpenClaw recorded with the prices Rev4a syncs.
+
+**Auth:** browser cookie or Bearer
+
+**Response:**
+```json
+{
+  "available": true,
+  "range": "7d",
+  "startDate": "2026-09-21",
+  "endDate": "2026-09-27",
+  "intervalS": 300,
+  "pricing": { "vendor": "DeepSeek", "band": "off-peak", "nextChange": 1790557200000, "source": "https://api-docs.deepseek.com/quick_start/pricing" },
+  "totals": { "cost": 0.0086, "tokens": 114053, "input": 47436, "output": 2233, "cacheRead": 64384, "cacheWrite": 0,
+              "inputCost": 0.0071, "outputCost": 0.0013, "cacheReadCost": 0.0002, "cacheWriteCost": 0, "missing": 0 },
+  "daily": [ { "date": "2026-09-21", "cost": 0.0071, "tokens": 96806, "missing": 0 } ],
+  "byAgent": [ { "container": "agent_9253eee3", "agentId": "agent_9253eee3", "name": "test", "cost": 0.0077, "tokens": 96806,
+                 "missing": 0, "collectedAt": 1790526404843, "attemptedAt": 1790526404843, "error": null } ],
+  "byModel": [ { "provider": "rev4a", "model": "deepseek/deepseek-flash", "cost": 0.0086, "tokens": 114053, "calls": 13 } ],
+  "sessions": [ { "container": "agent_9253eee3", "agentName": "test", "sessionId": "…", "sessionKey": "agent:main:…", "label": "…",
+                  "agentId": "main", "model": "deepseek/deepseek-flash", "cost": 0.004, "tokens": 18681, "missing": 0, "lastActivity": 1790322536609 } ],
+  "unpriced": [ { "model": "rev4a/deepseek/deepseek-v4-flash", "calls": 84, "agents": ["test", "test 2"] } ],
+  "reconciliation": [ { "vendor": "deepseek", "currency": "USD", "from": 1790529518869, "to": 1790536317000,
+                        "billed": 0.02, "priced": 0.0171, "topUps": 0, "readings": 3, "comparable": true } ]
+}
+```
+
+- `daily` has one entry per day of the range, zeros included.
+- `missing` counts calls OpenClaw could not price (no price for their model): they are in
+  the tokens, not in the cost. `unpriced` lists them by model over each agent's last 30
+  days.
+- `byAgent` holds every agent read, running or not, with use in the range, an error, or
+  a read in the last 15 minutes; `error` is why its last read stored nothing (OpenClaw's
+  own reason, e.g. "Gateway not reachable…"). An agent still being read (`attemptedAt`
+  within 3 × `intervalS`) with an `error` or a `collectedAt` older than that is not
+  current; a stopped or deleted agent is no longer read and keeps its last figures.
+- `sessions`: the 20 most expensive sessions active in the range; a session's figures are
+  its total over the last 30 days.
+- `pricing`: the band DeepSeek bills at now and when it changes, when a DeepSeek model is
+  on offer; `null` otherwise.
+- `reconciliation`: per vendor with a configured key (DeepSeek, OpenRouter), what it
+  billed next to what the agents priced over the same hourly readings — `{ vendor,
+  currency, from, to, billed, priced, topUps, readings, comparable }`. `billed` is the
+  balance drop (DeepSeek; rises are `topUps`, not spend) or the usage growth (OpenRouter);
+  `priced` the growth of the agents' total for that vendor's models. `billed`/`priced`
+  are `null` until two readings exist; `comparable` is false for a non-USD balance.
+  `vendor` is the lowercase key (`deepseek`, `openrouter`); `pricing.vendor` is a display
+  name. Known gaps, shown rather than corrected: the same key may also pay for the
+  assistant or clients outside Rev4a; a top-up and spend in the same hour net out; an
+  agent missed by a pass joins the agents' total one pass later; calls never priced are
+  priced by OpenClaw when read, so the agents' total can grow for calls made earlier.
+- `available: false` (zeros, empty lists) until the daemon has created `costs.db`.
+
+**Errors:** `400` for another `range`; `500` when `costs.db` cannot be read.
+
+---
+
 ### `GET /api/stats`
-Current-month aggregate stats (tokens + cost) grouped by model.
+Current-month aggregate stats (tokens + cost) grouped by model, from the `sessions` table
+(`started_at` is in milliseconds). The costs are the `sessions.cost_usd` column, which is no
+longer estimated: 0, except rows written before 2026-09-27. Agent spend is `GET /api/costs/usage`.
 
 **Auth:** Bearer or browser cookie
 
@@ -526,13 +594,13 @@ Current-month aggregate stats (tokens + cost) grouped by model.
 ```json
 {
   "total": {
-    "total": 1.24,
+    "total": 0,
     "total_in": 850000,
     "total_out": 210000,
     "sessions": 47
   },
   "byModel": [
-    { "model": "openai-codex/gpt-5.4", "cost": 0.95, "tokens_in": 600000, "tokens_out": 150000, "sessions": 30 }
+    { "model": "rev4a/deepseek/deepseek-flash", "cost": 0, "tokens_in": 600000, "tokens_out": 150000, "sessions": 30 }
   ]
 }
 ```
@@ -540,7 +608,8 @@ Current-month aggregate stats (tokens + cost) grouped by model.
 ---
 
 ### `GET /api/stats-since?ts=<unix_ms>`
-Total cost (USD) since a given timestamp.
+Sum of `sessions.cost_usd` since a given timestamp — 0 for sessions polled since 2026-09-27
+(see `GET /api/stats`).
 
 **Auth:** Bearer or browser cookie
 
@@ -548,28 +617,49 @@ Total cost (USD) since a given timestamp.
 
 **Response:**
 ```json
-{ "total": 0.42 }
+{ "total": 0 }
 ```
 
 ---
 
 ### `GET /api/costs`
-Full cost breakdown: today, 7d, 30d, all-time, per-model, plus cost-override for the current month.
+The dashboard's figures: today's spend (UTC day), all-time spend, today's calls without a
+price, and today's spend by model — from `costs.db`. The SSE stream (`GET /api/stream`)
+sends the same `today`.
 
-**Auth:** browser cookie
+**Auth:** browser cookie or Bearer
 
 **Response:**
 ```json
 {
-  "today": 0.12,
-  "week": 1.85,
-  "month": 6.40,
-  "allTime": 42.10,
-  "override": null,
-  "byModel": [ ... ]
+  "today": 0.0112,
+  "allTime": 0.0559,
+  "allTimeSource": "agents_openclaw",
+  "missingToday": 0,
+  "byModel": [ { "model": "deepseek/deepseek-flash", "cost_usd": 0.0112, "tokens": 244157, "calls": 7 } ]
 }
 ```
 
+`byModel` lists the models that cost something today; the free and unpriced ones are on the
+Costs page.
+
+**Errors:** `500` when `costs.db` cannot be read.
+
+---
+
+### `GET /api/costs/agent?container=<name>`
+One agent's spend for its panel, from `costs.db`.
+
+**Auth:** browser cookie or Bearer
+
+**Response:** `{ "available": true, "intervalS": 300, "today": 0.011, "week": 0.019,
+"month": 0.022, "tokensMonth": 2409574, "topModel": { "model": "deepseek/deepseek-flash",
+"cost": 0.022 }, "unpricedMonth": 81, "collectedAt": 1790529516737, "error": null }` —
+UTC days (today, last 7, last 30); `collectedAt`/`error` from the agent's last read.
+`{ "available": false, "intervalS": 300 }` before `costs.db` exists.
+
+**Errors:** `400` for an invalid container name; `500` when `costs.db` cannot be read.
+
 ---
 
 ### `GET /api/cost-override?month=YYYY-MM`
@@ -619,13 +709,17 @@ Paginated event log.
 ---
 
 ### `GET /api/stream`
-Server-Sent Events stream. Pushes a combined payload every ~3 s.
+Server-Sent Events stream. Every 5 s it sends one message per kind, each `{ type, data }`:
+`events` (new rows of the `events` table, only when there are any), `sessions` (the 100 most
+recent sessions with their lineage label) and `costs` (`{ "today": … }`, what the agents spent
+today, UTC day — the same figure as `GET /api/costs`).
 
-**Auth:** browser cookie
+**Auth:** browser cookie or Bearer
 
 **Event format:**
 ```
-data: {"events":[...],"sessions":[...],"costs":{"today":0.12},"lineage":[...]}
+data: {"type":"sessions","data":[...]}
+data: {"type":"costs","data":{"today":0.0112}}
 ```
 
 Use `EventSource` in the browser or `curl -N` for testing.
@@ -1951,6 +2045,81 @@ average and peak: 60 s for `1h`, 720 s for `24h`, 5040 s for `7d`, 21 600 s for
   means the daemon is not sampling; the page says so.
 - `host` is read live by the API process, which runs on the same host.
 
+### `GET /api/metrics/containers?range=1h|24h|7d|30d`
+What every container consumes — now, and as an average and peak over the range — plus what is
+not a container and the disk Docker holds. From `metrics.db` (the daemon samples every 60 s,
+see `docs/dev/DATABASE.md`). Default range `1h`.
+
+**Auth:** browser cookie or Bearer
+
+**Response:**
+```json
+{
+  "available": true, "range": "1h", "interval_s": 60,
+  "host": { "ncpu": 10, "mem_total_mb": 7837 },
+  "sampled_at": 1790789760, "age_s": 10,
+  "containers": [
+    { "container": "agent_9253eee3", "name": "test", "agent_id": "agent_9253eee3", "is_agent": true,
+      "running": true, "last_seen": 1790789760,
+      "now": { "cpu_cores": 0.022, "cpu_percent": 0.22, "mem_mb": 668, "mem_percent": 8.5, "pids": 12,
+               "net_rx_bps": 0, "net_tx_bps": 0, "blk_read_bps": 0, "blk_write_bps": 0 },
+      "range": { "cpu_avg_percent": 1.02, "cpu_max_percent": 2.17, "mem_avg_mb": 679, "mem_max_mb": 701 },
+      "storage": { "volume_mb": 1544, "layer_mb": 10, "total_mb": 1554, "ts": 1790789645 } }
+  ],
+  "rest": { "cpu_percent": 42.26, "mem_mb": 11864 },
+  "docker_storage": { "ts": 1790789645, "images_mb": 4734, "images_reclaimable_mb": 3518,
+                      "build_cache_mb": 1597, "build_cache_reclaimable_mb": 1597,
+                      "volumes_mb": 10120, "unused_volumes_mb": 7337, "layers_mb": 30,
+                      "unused_volumes": [ { "name": "rev4a-backups", "size_mb": 7337 } ] }
+}
+```
+
+- `cpu_percent` is a share of Docker's cores (`host.ncpu`), `mem_percent` of Docker's memory —
+  on Docker Desktop that is its VM, on the VPS the machine. `cpu_cores` is the raw figure.
+- `now` is `null` for a container that is not running (no sample within 3 intervals of the
+  newest sample — running is judged against the newest sample, not the wall clock, so a
+  stopped daemon does not make every container look stopped);
+  `running: false` ones stay listed while they were seen in the range. A rate that is null in
+  the newest sample (the first after a start) is replaced by the previous reading.
+- The list is sorted running first, then by CPU now. `storage` is the named volumes plus the
+  writable layer, from the last Docker disk reading (every 10 min).
+- `rest` is the machine's newest sample minus the running containers (the containers' CPU
+  re-scaled to the machine's cores first, so on Docker Desktop the two shares still compare):
+  the host, Rev4a and other software (`null` without a machine sample or Docker's figures).
+- `docker_storage.*_reclaimable_mb` and `unused_volumes` (the 5 largest): what **no container
+  uses** — images of older agent versions you may still roll back to count as such; a volume
+  like the cold backups' is listed, not advised for removal. `volumes_mb` sums all volumes;
+  `unused_volumes_mb` is the total of what no container uses, `unused_volumes` only names the
+  five largest.
+- `available: false` until the daemon has created the tables; with the tables present and no
+  rows yet it answers `available: true` with an empty list.
+
+**With `&container=<name>`** — that container's history, bucketed like `GET /api/metrics`:
+```json
+{ "available": true, "range": "1h", "interval_s": 60, "host": { "ncpu": 10, "mem_total_mb": 7837 },
+  "container": "agent_9253eee3", "name": "test", "bucket_s": 120,
+  "history": [ { "ts": 1790789640, "cpu_avg": 1.41, "cpu_max": 2.17, "mem_avg": 685, "mem_max": 701 } ],
+  "storage_bucket_s": 1200, "storage_history": [ { "ts": 1790788800, "total_mb": 1554 } ] }
+```
+`cpu_*` are shares of Docker's cores, `mem_*` MB. A container that was not running has no
+points for that stretch (a gap). A name that matches the rules but was never sampled answers
+`available: true` with an empty history.
+
+**Errors:** `400` for another `range` or an invalid container name. A `metrics.db` that
+cannot be opened (missing, corrupt) answers `available: false`, not an error.
+
+### `GET /api/metrics/alerts?limit=10`
+The machine's latest `system_anomaly` events (CPU above 85 % or memory above 90 % on two
+samples in a row, a disk at 90 %), newest first, from `events.db`. `limit` 1–50, default 10.
+
+**Auth:** browser cookie or Bearer
+
+**Response:**
+```json
+{ "alerts": [ { "ts": 1790789625355, "metric": "disk", "message": "Disk high: / 94% used", "threshold": 90, "values": [94] } ] }
+```
+`ts` is in **milliseconds** (the `events` table's unit); `metric` is `cpu`, `ram`, `disk` or `null`.
+
 ---
 
 ## System Health
@@ -1973,14 +2142,16 @@ say so; it does not fail the others.
     { "id": "runtime.sessions", "label": "Runtime sessions", "health": "ok", "value": 1, "details": "last session activity 3m ago", "source": "runtime" }
   ],
   "recommendations": [
-    { "id": "cost.review-usage-based", "severity": "warning", "source": "cost", "title": "High usage-based cost", "details": "…", "actionHref": "/gateway", "dismissible": true, "createdAt": "2026-09-26T10:00:00.000Z" }
+    { "id": "cost.review-usage-based", "severity": "warning", "source": "cost", "title": "High spend today", "details": "…", "actionHref": "/costs", "dismissible": true, "createdAt": "2026-09-26T10:00:00.000Z" }
   ],
   "generatedAt": "2026-09-26T10:00:00.000Z"
 }
 ```
 
 Checks: `runtime.sessions`, `runtime.heartbeat`, `runtime.errors24h`,
-`cost.usageBasedToday`, `cron.jobs`. `health` (overall and per check) is `"ok"` /
+`cost.usageBasedToday` ("Spent today": the agents' spend today, UTC day, from `costs.db`;
+a warning above $10 or with calls without a price, each with its recommendation,
+`cost.review-usage-based` / `cost.unpriced-calls`), `cron.jobs`. `health` (overall and per check) is `"ok"` /
 `"warning"` / `"error"`; a recommendation's `severity` is `"info"` / `"warning"` /
 `"critical"`. `generatedAt` and `createdAt` are ISO strings.
 
@@ -2206,27 +2377,26 @@ Full key management documentation at [PROVIDERS.md](PROVIDERS.md).
 ## Containers
 
 ### `GET /api/containers`
-List all running Docker containers with resource usage.
+Every Docker container, running or stopped, with its identity and network details. What a
+running container consumes lives in `GET /api/metrics/containers`.
 
-**Auth:** browser cookie
+**Auth:** browser cookie or Bearer
 
-**Response:**
+**Response:** a bare array —
 ```json
-{
-  "containers": [
-    {
-      "name": "openclaw-atlas",
-      "image": "openclaw-agent-base:2026.9.3",
-      "status": "running",
-      "ports": ["0.0.0.0:3731->3000/tcp"],
-      "created": "2026-06-20T10:00:00Z",
-      "cpu_percent": 2.1,
-      "mem_usage_mb": 340
-    }
-  ]
-}
+[
+  { "id": "a1b2c3d4e5f6", "name": "openclaw-atlas", "image": "openclaw-agent-base:2026.9.3",
+    "status": "Up 3 hours", "state": "running", "ports": "0.0.0.0:3731->3000/tcp",
+    "ip": "172.18.0.2", "agentId": "agent_atlas", "displayName": "Atlas", "created": null }
+]
 ```
 
+- `name` is the daemon's `primaryName` (the name with one leading slash) — the same key the
+  metrics store and the `/system?agent=` link use.
+- `displayName` is the agent's `AGENT_NAME`, read from the inspect. If an inspect fails, the
+  row still comes from Docker's list (`ip` and `displayName` then `null`): a container is
+  never dropped, and never renamed, because one call failed.
+
 The container list is the only container route. There is no logs endpoint and no
 action endpoint: logs are read through the terminal WebSocket, and lifecycle
 operations on agent containers go through `/api/agents/[id]/lifecycle`.