@flame0510/project-aether 1.1.9

package/docs/dev/DATABASE.md

@@ -0,0 +1,224 @@
+# Rev4a — Database Reference
+
+> **Last updated:** 2026-06-30
+
+Rev4a uses a single SQLite file (`events.db`) in WAL mode.
+
+## Location
+
+| Environment | Path |
+|---|---|
+| VPS (default) | `/path/to/rev4a/data/events.db` |
+| Custom | Set `REV4A_DB` env var |
+
+## WAL Mode
+
+```sql
+PRAGMA journal_mode = WAL;
+PRAGMA synchronous = NORMAL;
+```
+
+- The daemon writes; the Next.js API routes open read-only connections
+- `PRAGMA wal_checkpoint(PASSIVE)` runs after each daemon poll cycle
+- WAL files: `events.db-shm`, `events.db-wal` — do not delete while daemon is running
+
+## Tables
+
+### `sessions` — primary session tracking
+
+```sql
+CREATE TABLE sessions (
+  session_id   TEXT PRIMARY KEY,
+  parent_id    TEXT,
+  label        TEXT,
+  model        TEXT,
+  tokens_in    INTEGER DEFAULT 0,
+  tokens_out   INTEGER DEFAULT 0,
+  cost_usd     REAL    DEFAULT 0,
+  status       TEXT    DEFAULT 'idle',
+  task_preview TEXT,
+  started_at   INTEGER,           -- Unix seconds
+  ended_at     INTEGER,           -- Unix seconds, null if active
+  updated_at   INTEGER,           -- Unix ms, updated only on real changes
+  trello_card_url TEXT
+);
+
+CREATE INDEX idx_sessions_updated ON sessions(updated_at);
+```
+
+**Status values:** `idle` / `working` / `completed` / `error`
+
+### `events` — lifecycle events
+
+```sql
+CREATE TABLE events (
+  id         INTEGER PRIMARY KEY AUTOINCREMENT,
+  ts         INTEGER NOT NULL,   -- Unix ms
+  session_id TEXT    NOT NULL,
+  type       TEXT    NOT NULL,   -- spawn | complete | error | tool_call
+  data       TEXT                -- JSON payload
+);
+
+CREATE INDEX idx_events_session ON events(session_id, ts);
+```
+
+### `lineage` — explicit parent→child declarations
+
+```sql
+CREATE TABLE lineage (
+  child_id    TEXT PRIMARY KEY,
+  parent_id   TEXT,
+  agent_name  TEXT,
+  declared_at INTEGER            -- Unix ms
+);
+```
+
+Populated by agents calling `lineage.js` at spawn time or via `POST /api/lineage`.
+
+### `cost_override` — manual monthly billing corrections
+
+```sql
+CREATE TABLE cost_override (
+  month      TEXT PRIMARY KEY,   -- YYYY-MM
+  amount     REAL,
+  note       TEXT,
+  updated_at INTEGER             -- Unix ms
+);
+```
+
+When a `cost_override` row exists for the current month, the UI displays the override value instead of the computed sum.
+
+### `system_metrics` — infrastructure metrics
+
+```sql
+CREATE TABLE system_metrics (
+  id           INTEGER PRIMARY KEY AUTOINCREMENT,
+  ts           INTEGER NOT NULL,
+  cpu_percent  REAL,
+  ram_used_mb  INTEGER,
+  ram_total_mb INTEGER,
+  disk_used_gb REAL,
+  disk_total_gb REAL,
+  load_avg_1m  REAL
+);
+
+CREATE INDEX idx_metrics_ts ON system_metrics(ts);
+```
+
+Rows older than 24 h are pruned automatically by the daemon.
+
+## Useful Queries
+
+```sql
+-- Total cost this month
+SELECT SUM(cost_usd) FROM sessions
+WHERE started_at >= strftime('%s', date('now', 'start of month'));
+
+-- Active sessions right now
+SELECT session_id, label, model, status, cost_usd
+FROM sessions WHERE status = 'working';
+
+-- Cost by model (all time)
+SELECT model, SUM(cost_usd) AS total, COUNT(*) AS sessions
+FROM sessions GROUP BY model ORDER BY total DESC;
+
+-- Last poll time (freshness check)
+SELECT datetime(MAX(ts)/1000, 'unixepoch', 'localtime') AS last_event
+FROM events;
+
+-- Lineage tree for a session
+WITH RECURSIVE tree(child_id, parent_id, agent_name, depth) AS (
+  SELECT child_id, parent_id, agent_name, 0 FROM lineage WHERE child_id = 'agent:ops:main'
+  UNION ALL
+  SELECT l.child_id, l.parent_id, l.agent_name, t.depth + 1
+  FROM lineage l JOIN tree t ON l.parent_id = t.child_id
+)
+SELECT * FROM tree;
+```
+
+## Backup
+
+```bash
+# Manual snapshot
+sqlite3 events.db ".backup events.db.bak-$(date +%s)"
+
+# Automated (via backup.sh in repo root)
+bash /data/.openclaw/workspace-ops/rev4a-next-ts/backup.sh
+```
+
+Backups are stored in `backups/` inside the repo directory.
+
+## Credentials Database (`credentials.db`)
+
+The credential hub uses a **separate SQLite file** (`credentials.db`) alongside
+events.db in the same data directory. It stores credentials that can be synced
+to agent containers via CLI auth or env files.
+
+**Location:** `data/credentials.db` (same directory as `events.db`)
+**Custom path:** Set `REV4A_CREDENTIALS_DB` env var to override.
+**WAL mode:** Enabled.
+
+### `credential_profiles` — credential hub metadata
+
+```sql
+CREATE TABLE credential_profiles (
+  id TEXT PRIMARY KEY,
+  provider_id TEXT NOT NULL,
+  label TEXT NOT NULL,
+  status TEXT NOT NULL DEFAULT 'active',
+  secret_masked TEXT,
+  created_at INTEGER NOT NULL,
+  updated_at INTEGER NOT NULL,
+  last_synced_at INTEGER
+);
+```
+
+### `credential_secrets` — stored secret payloads
+
+```sql
+CREATE TABLE credential_secrets (
+  profile_id TEXT PRIMARY KEY,
+  payload TEXT NOT NULL
+);
+```
+
+Payloads are stored as JSON strings. In a future release they will be AES-256-GCM encrypted.
+
+### `credential_audit_log` — append-only audit trail
+
+```sql
+CREATE TABLE credential_audit_log (
+  id INTEGER PRIMARY KEY AUTOINCREMENT,
+  ts INTEGER NOT NULL,
+  actor TEXT NOT NULL,
+  action TEXT NOT NULL,
+  profile_id TEXT,
+  result TEXT
+);
+```
+
+Actions: `create`, `update`, `delete`, `sync`, `reveal`.
+
+---
+
+## Migrations
+
+The daemon applies these migrations safely at startup:
+
+```sql
+ALTER TABLE sessions ADD COLUMN trello_card_url TEXT;  -- idempotent
+ALTER TABLE sessions ADD COLUMN ended_at INTEGER;       -- idempotent
+```
+
+For future schema changes, add a `try/catch` ALTER TABLE block in `daemon.js` following the same pattern.
+
+## Integrity Check
+
+```bash
+sqlite3 events.db "PRAGMA quick_check"
+# Expected output: ok
+```
+
+**Current location on VPS:** `/path/to/rev4a/data/events.db`
+
+The `REV4A_DB` env var is set in the systemd override file at `/etc/systemd/system/rev4a.service.d/env.conf`.

package/docs/dev/GATEWAY.md

@@ -0,0 +1,300 @@
+# Gateway Page
+
+> **Last updated:** 2026-07-31
+
+The Gateway page (`/gateway`) is the central control panel for managing provider
+configurations and agent model assignments across all Docker containers.
+
+The Gateway aggregates multiple upstream AI providers (DeepSeek, OpenAI, Anthropic,
+Kimi, GLM, Qwen, and others) behind a single Rev4a provider. Agent containers see
+only `rev4a` as their provider and route all LLM traffic through the Rev4a proxy
+at `/api/provider/v1/`.
+
+---
+
+## Overview
+
+```
+Gateway Page
+├── Provider Tab    → Manage provider API keys, enable/disable models, push sync
+└── Agents Tab      → Change primary model per agent container
+```
+
+The Gateway talks to each agent container directly via `docker exec`, reading and
+writing to `/root/.openclaw/openclaw.json` inside the container.
+
+---
+
+## Provider Tab
+
+Lists all AI providers from `models.config.json` in the project root. Each provider
+row shows:
+
+- Provider name and label
+- Configured API key (masked)
+- Enable/disable toggles per model
+- ADD / CHANGE / REMOVE KEY buttons
+
+### Provider Key Storage
+
+Provider API keys are stored in `data/provider-keys.json`:
+
+```json
+{
+  "deepseek": "sk-...",
+  "openrouter": "sk-...",
+  "rev4a": "ciao"
+}
+```
+
+The `rev4a` entry is the master gateway token. Agent containers authenticate
+against the Rev4a provider proxy using this token. See [`PROVIDERS.md`](PROVIDERS.md).
+
+### Sync Flow
+
+When a provider key is saved or a model toggle is changed, the Gateway:
+
+1. **Reads** `models.config.json` to get the full model catalogue
+2. **Filters** enabled models for providers that have a configured API key
+3. **Builds** a `models.providers.rev4a` config block (without `rev4a/` prefix
+   on model IDs)
+4. **Writes** the block into `/root/.openclaw/openclaw.json` on every agent container
+   (containers with `AGENT_ID` Docker label)
+5. **Cleans up** stale `models.json` and `auth-profiles.json` files inside each
+   container
+6. **Writes** the block into `/root/.openclaw/openclaw.json` on every agent container
+   — **does NOT touch** `agents.defaults.model` or `agents.list[].model`
+7. **Does NOT restart** the gateway — writes are live via file write
+
+### Model ID Convention
+
+Inside `models.providers.rev4a.models`, model IDs are stored **without** the
+`rev4a/` prefix:
+
+```json
+{
+  "models": {
+    "providers": {
+      "rev4a": {
+        "baseUrl": "http://host.docker.internal:3740/api/provider/v1",
+        "api": "openai-completions",
+        "apiKey": "***",
+        "models": [
+          { "id": "deepseek/deepseek-v4-flash", "name": "DeepSeek V4 Flash" },
+          { "id": "deepseek/deepseek-v4-pro", "name": "DeepSeek V4 Pro" }
+        ]
+      }
+    }
+  }
+}
+```
+
+OpenClaw automatically prefixes model IDs with the provider name at runtime,
+producing `rev4a/deepseek/deepseek-v4-flash`. The `mode` field is **not** set —
+OpenClaw defaults to merging config models with auto-discovered models.
+
+### Model Catalogue
+
+The model catalogue lives in [`models.config.json`](../models.config.json) at the project root.
+This file is **tracked in the repository** and serves as the default model configuration
+for anyone cloning the project. It defines all known models across all providers:
+
+```json
+{
+  "models": [
+    { "id": "openai/gpt-5.6-sol", "name": "GPT-5.6 Sol", "provider": "openai", "enabled": true },
+    { "id": "openai/gpt-5.6-terra", "name": "GPT-5.6 Terra", "provider": "openai", "enabled": true },
+    { "id": "openai/gpt-5.6-luna", "name": "GPT-5.6 Luna", "provider": "openai", "enabled": true },
+    { "id": "openai/gpt-5.5", "name": "GPT-5.5", "provider": "openai", "enabled": true },
+    { "id": "openai/gpt-5.4", "name": "GPT-5.4", "provider": "openai", "enabled": true },
+    { "id": "openai/gpt-5.4-mini", "name": "GPT-5.4 Mini", "provider": "openai", "enabled": true },
+    { "id": "anthropic/claude-fable-5", "name": "Claude Fable 5", "provider": "anthropic", "enabled": false },
+    { "id": "anthropic/claude-opus-5", "name": "Claude Opus 5", "provider": "anthropic", "enabled": false },
+    { "id": "anthropic/claude-sonnet-5", "name": "Claude Sonnet 5", "provider": "anthropic", "enabled": false },
+    { "id": "groq/llama-3.3-70b-versatile", "name": "Llama 3.3 70B (Groq)", "provider": "groq", "enabled": false },
+    { "id": "kimi/kimi-k3", "name": "Kimi K3", "provider": "kimi", "enabled": true },
+    { "id": "glm/glm-5.2", "name": "GLM-5.2", "provider": "glm", "enabled": false },
+    { "id": "qwen/qwen3.7-plus", "name": "Qwen 3.7 Plus", "provider": "qwen", "enabled": true }
+  ]
+}
+```
+
+Only enabled models for providers with a configured API key are synced to agent
+containers.
+
+---
+
+## Agents Tab
+
+Lists all agent containers discovered from Docker (containers with `AGENT_ID`
+label). Each agent row shows:
+
+- Agent name and container name
+- Current primary model
+- "Change Model" button
+
+### Change Model Flow
+
+Clicking "Change Model" opens an inline form with:
+
+1. **Primary model select** — dropdown of all enabled models (filtered to
+   configured providers only)
+
+On save:
+
+1. **PUT /api/gateway/agent** writes the model config to the container's
+   `openclaw.json`:
+   - `agents.defaults.model.primary` and `agents.list[0].model.primary` written
+     with format `rev4a/<provider>/<model>`
+   - No fallbacks are set unless explicitly provided
+2. **No restart** — writes are live via `docker exec node -e`
+
+### Model Select Filtering
+
+The model select in the Agents tab only shows models whose provider has a
+configured API key in `data/provider-keys.json`. This ensures users can only
+select models that are actually available.
+
+---
+
+## API Endpoints
+
+### `GET /api/gateway`
+
+Returns live Gateway status from all agent containers. Reads agent model configs
+via `openclaw models status --json` inside each container.
+
+**Response:**
+```json
+{
+  "agents": {
+    "total": 1,
+    "list": [
+      {
+        "containerName": "openclaw-atlas",
+        "agentId": "atlas",
+        "agentName": "Atlas",
+        "defaultModel": "rev4a/deepseek/deepseek-v4-flash",
+        "configured": true
+      }
+    ]
+  }
+}
+```
+
+### `PUT /api/gateway/provider`
+
+Triggers a full provider model sync to all agent containers. Only writes
+`models.providers.rev4a` — does NOT touch model references.
+
+**Body:** none (reads current provider state from `models.config.json` and
+`data/provider-keys.json`)
+
+**Response:**
+```json
+{
+  "status": "ok",
+  "activeModels": 2,
+  "results": ["Active models: 2 across 1 agent(s)"]
+}
+```
+
+### `PUT /api/gateway/agent`
+
+Updates model config for a single agent container (primary model only).
+
+**Body:**
+```json
+{
+  "containerName": "openclaw-atlas",
+  "model": "deepseek/deepseek-v4-flash"
+}
+```
+
+**Response:**
+```json
+{
+  "status": "ok",
+  "agent": "openclaw-atlas",
+  "model": {
+    "primary": "rev4a/deepseek/deepseek-v4-flash"
+  },
+  "verify": { "ok": true, "bytes": 2058 }
+}
+```
+
+### `GET /api/gateway/provider`
+
+Returns current provider configuration state.
+
+**Response:**
+```json
+{
+  "providers": [
+    {
+      "provider": "deepseek",
+      "label": "DeepSeek",
+      "configured": true,
+      "baseUrl": "https://api.deepseek.com",
+      "models": [
+        { "id": "deepseek/deepseek-v4-flash", "name": "DeepSeek V4 Flash", "enabled": true }
+      ]
+    }
+  ]
+}
+```
+
+---
+
+## Config Page (`/config`)
+
+A companion page at `/config` that allows editing environment variables and
+triggering a server restart.
+
+### `PUT /api/config/env`
+
+Update environment variables on the VPS host.
+
+**Body:** `{ "REV4A_PASSWORD": "newpass", ... }`
+
+**Response:** `{ "ok": true }`
+
+### `POST /api/config/restart`
+
+Restart the Rev4a server (via systemd).
+
+**Response:** `{ "ok": true }`
+
+---
+
+## Key Design Decisions
+
+1. **No `mode: 'replace'`** — The `mode` field is not written to the provider
+   config. OpenClaw uses its default merge behavior, combining config models with
+   auto-discovered models from the provider API.
+
+2. **Model IDs without `rev4a/` prefix in config** — Inside
+   `models.providers.rev4a.models`, model IDs do NOT include the `rev4a/`
+   prefix. OpenClaw adds it automatically at runtime.
+
+3. **Provider keys in `data/provider-keys.json`** — Not in env vars or
+   committed code. This file is gitignored and managed exclusively through the
+   Gateway UI.
+
+4. **No gateway restart after model sync** — The sync module writes `models.providers`
+   to the file directly without restarting the gateway.
+
+5. **Entrypoint does not generate openclaw.json** — The entrypoint resolves the
+   auth token and starts the gateway with `--allow-unconfigured`; OpenClaw
+   generates its own default config on first boot. All post-creation config
+   (controlUi, model refs, providers) is written by the create route
+   (`POST /api/agents/create`) after the gateway is fully ready.
+   The entrypoint also runs `openclaw doctor --non-interactive` **before** the
+   gateway, but only when the OpenClaw version changed since last boot
+   (tracked in `/root/.openclaw/.last-version`) — safe migrations only, no restart.
+
+6. **Model ref written after gateway readiness** — When the wizard creates an agent,
+   the create route waits for gateway health check, then writes
+   `agents.defaults.model.primary` + fallbacks and `gateway.controlUi.allowedOrigins`,
+   then calls `syncAgent(name)`. The Gateway sync (`PUT /api/gateway/provider`)
+   never touches model references — only `models.providers` is synced.