@flame0510/project-aether 1.1.9

package/docs/REV4A.md

@@ -0,0 +1,504 @@
+# Rev4a — VPS Dashboard
+
+> **Last updated:** 2026-07-27
+
+A Next.js 16 dashboard for monitoring and managing the OpenClaw ecosystem.
+
+**URL:** `http://<vps-ip>:3740` (or behind a reverse proxy of your choice)
+**Port:** 3740
+**Process manager:** `rev4a serve` (single command). Production: systemd service (`rev4a.service`).
+**Repository:** `github.com/Flame0510/rev4a.git` (branch: `main`)
+
+---
+
+## Overview
+
+| Field | Value |
+|---|---|
+| VPS IP | `187.77.156.41` |
+| OS | Ubuntu 24.04.4 LTS |
+| Resources | 4 CPU, 15 GB RAM, 193 GB disk |
+
+
+---
+
+## Architecture
+
+```
+                      Internet
+                         │
+                    ┌────┴────┐
+                    │ Traefik │  (Docker container, host network)
+                    │  :443   │
+                    └────┬────┘
+                         │
+              ┌──────────┴──────────┐
+              │  rev4a             │
+              │  systemd (rev4a.service)    │
+              │  :3740              │
+              │  Next.js 16         │
+              └─────────────────────┘
+                         │
+           ┌─────────────┴─────────────┐
+           │                           │
+   docker exec / API           docker exec / API
+           │                           │
+    ┌──────┴──────┐           ┌────────┴────────┐
+    │ openclaw-core│           │  openclaw-atlas │
+    │ :3700-3708   │           │  :3731 → 3000   │
+    │ AGENT_ID=core│           │  AGENT_ID=atlas  │
+    └──────────────┘           └─────────────────┘
+```
+
+### Environment
+
+Rev4a reads the following environment variables. Set them in `/config` (UI) or directly in `.env`.
+
+| Variable | Required | Description |
+|---|---|---|
+| `REV4A_PASSWORD` | Yes | Login password for the dashboard |
+| `REV4A_TOKEN` | Yes | Token used for programmatic API access |
+| `REV4A_JWT_SECRET` | Yes | Secret used to sign session JWT tokens |
+| `REV4A_DB` | No | Path to the events database (default: `data/events.db`) |
+| `REV4A_WORKSPACE_ROOT` | No | Filesystem path for the local file explorer |
+
+## Agent creation
+
+Every agent is created with:
+
+- A host port mapping (`-p <port>:3000`) for direct access
+- URL format: `http://<host-public-ip>:<port>#token=...`
+- The provider gateway at `http://host.docker.internal:3740/api/provider/v1`
+- `allowedOrigins: ['*']` for browser Control UI access
+
+The port is auto-assigned starting from 3000 (or user-specified).
+
+Domain and HTTPS routing (Traefik) is an external concern — operators can
+configure it on top of the base setup if needed, but Rev4a does not couple
+to any specific reverse proxy.
+
+## Config persistence
+
+When environment variables are saved via the Config page (Save & Restart):
+
+1. Values are written to `.env` in the project root
+2. Secrets are loaded via `EnvironmentFile=` pointing to `.env` (chmod 600). Only non-secret vars (`NODE_ENV`, `PATH`, `REV4A_DB`) are in the unit file. Other vars (passwords, non-Rev4a keys) stay in `.env` for the application to read at runtime.
+3. `systemctl daemon-reload` is run
+4. The server restarts with the new environment
+
+## Components
+
+- **Rev4a:** Next.js app serving the dashboard UI and REST API. Reads Docker state, container configs, and agent workspaces via `docker exec` and `fs` calls.
+- **Traefik:** External reverse proxy (optional). Rev4a does not depend on it — operators can use any reverse proxy or none at all.
+- **OpenClaw agents:** Docker containers with `AGENT_ID` label. Rev4a discovers them dynamically via `docker ps`.
+
+---
+
+## Pages
+
+| Route | Description |
+|---|---|
+| `/dashboard` | System overview — containers, resources, quick links |
+| `/agents` | Running agents (Docker containers with `AGENT_ID`), gateway token management, agent creation wizard, channel manager (Telegram pairing) |
+| `/containers` | All Docker containers on the host |
+| `/onboarding` | First-run setup wizard (Welcome → Providers → Agent → Ready) |
+| `/workspace` | File explorer with tree view + editor — VPS host or container workspaces |
+| `/lineage` | Agent lineage / orchestration tree |
+| `/memory` | Agent memory browser |
+| `/crons` | Scheduled cron jobs — auto-discovered from the host gateway and every running agent container (Docker containers with `AGENT_ID` label), on/off toggle, run history per job |
+| `/credentials` | Third-party tool credentials (GitHub, Trello, Vercel, Supabase) — JSON vault with audit trail, sync to agent containers via CLI auth (gh, vercel, supabase) or env files (trello) |
+| `/plugins` | Plugin manager |
+| `/plugins-skills` | Plugin skills |
+| `/skills` | Skill registry — browse/edit all skills (shared, per-agent workspace, bundled). Promote agent skills to shared with one click. |
+| `/tools` | Tool configuration |
+| `/gateway` | LLM provider sync + agent model configuration — see [GATEWAY.md](dev/GATEWAY.md) |
+| `/config` | Environment management and server restart |
+| `/vault` | Credential storage (per-agent permissions) — see [PROVIDERS.md](dev/PROVIDERS.md) |
+| `/login` | Authentication page |
+
+---
+
+## Authentication
+
+Rev4a uses password-based login with JWT cookies.
+
+**Env vars:**
+```
+REV4A_PASSWORD=***    # Login password
+REV4A_TOKEN=***       # API token (Bearer / query param / x-agent-token)
+REV4A_JWT_SECRET=***  # Secret for signing JWTs
+```
+
+**Auth flow:**
+1. User visits `/login`, enters password
+2. POST `/api/auth/login` validates password, returns a `rev4a_token` cookie (JWT, 7-day expiry)
+3. All subsequent API calls use the cookie (`credentials: 'same-origin'`)
+4. API-only auth fallbacks: `Authorization: Bearer <token>`, `?token=<token>`, or `x-agent-token` header
+5. Control UI bootstrap links should use `#token=<token>` so the browser client can import the shared secret before opening the socket
+
+---
+
+## Service Management
+
+```bash
+# Start
+sudo systemctl start rev4a.service
+
+# Stop
+sudo systemctl stop rev4a.service
+
+# Restart
+sudo systemctl restart rev4a.service
+
+# Status
+sudo systemctl status rev4a.service
+
+# Logs (follow)
+sudo journalctl -u rev4a -f
+```
+
+### Systemd environment override
+
+File: `/etc/systemd/system/rev4a-next.service` (EnvironmentFile)
+
+```ini
+[Service]
+Environment=REV4A_PASSWORD=***
+Environment=REV4A_TOKEN=***
+Environment=REV4A_JWT_SECRET=***
+Environment=REV4A_DB=/path/to/rev4a/data/events.db
+Environment=OPENCLAW_CONFIG_PATH=/path/to/openclaw-core.json
+Environment=SHARED_CONTEXT_DIR=/path/to/shared-context
+Environment=NODE_ENV=production
+Environment=NEXT_TELEMETRY_DISABLED=1
+```
+
+---
+
+## Build & Deploy
+
+```bash
+cd /path/to/'rev4a'
+
+# One command (builds + starts):
+rev4a serve
+
+# Manual build:
+npm run build
+
+# Full rebuild:
+rm -rf .next && npm run build
+
+# Production systemd (after setup script):
+sudo systemctl start rev4a.service
+```
+
+> **Important:** API routes (`route.ts`) are compiled at build time by Next.js. After editing any API route, a full `npm run build` is required. Restarting the server alone is not enough.
+
+---
+
+## Workspace API (`/api/workspace`)
+
+The workspace feature provides a file tree explorer + editor for both the VPS host and container workspaces.
+
+### Parameters
+
+| Method | Parameters | Description |
+|---|---|---|
+| `GET` | `?action=list` | List available workspaces |
+| `GET` | `?workspace=vps` | List root files in VPS host workspace |
+| `GET` | `?workspace=vps&path=<dir>` | List files in a directory |
+| `GET` | `?workspace=vps&path=<file>` | Read file content |
+| `PUT` | JSON `{workspace, path, content}` | Write/overwrite file |
+
+### Workspaces
+
+| ID | Label | Type | Path |
+|---|---|---|---|
+| `vps` | VPS Host (Nexus) | host | `/path/to/workspace/` |
+| `container-<name>` | `<agent_id> (<name>)` | container | `/root/.openclaw/` inside container |
+
+Container workspaces are discovered dynamically — any Docker container with an `AGENT_ID` label is listed.
+
+### Implementation notes
+
+- **Host:** uses `fs.readdirSync`, `fs.readFileSync`, `fs.writeFileSync` — real-time, no caching
+- **Container:** uses `docker exec` with `ls -1Ap`, `test -d`, `cat`, and heredoc writes
+- **Subdirectory expansion:** lazy-loaded via API when the user clicks a directory in the tree
+- **Binary files (images, PDFs):** detected by extension, served as-is or displayed inline
+
+---
+
+## Features (current)
+
+### Vault (`/vault`)
+Store and manage credentials (API keys, tokens) with per-agent permissions.
+
+- **SQLite storage:** `credentials.db` in the data directory
+- **Provider keys:** stored in `data/provider-keys.json`, managed via the Gateway UI
+- **Agent permissions:** each credential can be scoped to specific agents
+- **Service management:** add/remove provider API keys and service tokens via the UI
+
+### Container management (`/containers`)
+View all running Docker containers with:
+- Name, image, status, ports, uptime
+- Resource usage (CPU, memory)
+- Per-container logs
+- Quick links to agent control UIs
+
+### Agent management (`/agents`)
+Lists all agents (Docker containers with `AGENT_ID` label), with:
+- Name, image, template, status, ports, IP
+- TG channel status chip (green = connected, hidden if not configured)
+- **Shared gateway token** — text input with Save & Sync. Changing the token saves it to `data/agents-token.json` and pushes it to all running containers.
+- **Control UI link** — direct Traefik URL with `#token=<token>` hash, reads the token from `agents-token.json` (source of truth), not from the container's local config.
+- **Agent creation wizard** — multi-step form at `/agents/create`.
+- **Channel Manager** — each agent detail panel has a Channels section with Telegram configuration and pairing management.
+
+### Agent Creation Wizard (`/agents/create`)
+
+**Step 1 — Template:** Select an agent template from `agent-templates/`. Each template is a directory with AGENTS.md, SOUL.md, MEMORY.md, and optionally BOOTSTRAP.md files.
+
+Available templates:
+- **Atlas** — general-purpose development agent
+- **Argus** — security and audit agent
+- **Prometheus** — client-facing project delivery agent
+- **Custom** — blank slate. On first run, the agent runs a bootstrap ritual: introduces itself, learns who the user is, and together they define name, personality, tone, and boundaries. The agent writes SOUL.md, IDENTITY.md, and USER.md, then deletes BOOTSTRAP.md. No preset identity, fully user-defined.
+
+**Step 2 — Config:** Enter agent name, optional port, select a primary model from the available providers (pre-filtered by configured provider keys). The first model is pre-selected.
+
+**Step 3 — Deploy:** `POST /api/agents/create` creates the Docker container and returns the Traefik URL with the shared gateway token.
+
+**Post-creation pipeline:**
+1. The container boots with the **openclaw-agent-base:latest** image — see [Agent Templates](#agent-templates) below.
+2. OpenClaw gateway starts automatically with `--allow-unconfigured`, generating its own default config.
+3. The route waits for the gateway to be fully ready (health check poll, up to 60 s).
+4. Once ready, the route writes:
+   - `gateway.controlUi.allowedOrigins` — Rev4a dashboard URL + Traefik hostname (required for browser Control UI access)
+   - `agents.defaults.model.primary` + fallbacks — the primary model selected in the wizard
+5. `syncAgent(name)` writes `models.providers.rev4a` into the container.
+6. The agent's control UI is immediately accessible at `https://<name>.<your-domain>.com#token=<gateway-token>`.
+
+**Key difference from the old pipeline:** the entrypoint no longer generates `openclaw.json`. The bootstrap config (including `controlUi.allowedOrigins`) is written by the create route after the gateway has already started. This eliminates the race condition where the entrypoint would overwrite synced provider config on boot.
+
+---
+
+## Agent persistence & lifecycle
+
+Each agent is created with a **persistent Docker volume** `agent-<name>-data`
+mounted at `/root`. This cleanly separates two planes:
+
+- **Software → image.** OpenClaw, `gh`, `vercel`, `supabase`, `trello`, Node — all
+  live in `openclaw-agent-base:latest` (outside `/root`). Updated by rebuilding the
+  image and recreating the container.
+- **Data → volume.** Config (`openclaw.json`), state DB (`openclaw.sqlite`),
+  workspace files, and CLI credentials (`.config/gh`, `.local/share/com.vercel.cli`,
+  `.supabase`, …) live in `/root` and survive container removal.
+
+On create, template files are copied into the workspace **only if it is empty** —
+a re-created agent keeps the workspace the user edited.
+
+**Lifecycle actions** (see [API Reference](dev/API-REFERENCE.md) for endpoints):
+
+| Action | What it does |
+|---|---|
+| **Backup** | `tar` of `/root` (excl. npm cache) → `.tar.gz` in the `rev4a-backups` volume |
+| **Restore** | stop → replace `/root` with the backup (real wipe incl. dotfiles) → restart. Software/image untouched — only data goes back in time |
+| **Recreate** | auto-backup first (aborts if it fails) → rebuild from `openclaw-agent-base:latest` keeping the volume → picks up image/OpenClaw/CLI updates |
+| **Restart** | `docker restart`, nothing else |
+| **Delete** | **destructive**: removes container **+ volume + all backups**. Cannot be undone |
+
+Version migrations after an image update are handled by the entrypoint (see
+[Entrypoint behavior](#agent-templates)), not by `openclaw update` — updating an
+agent means rebuilding the image + Recreate, never `update` inside the container.
+
+---
+
+## Agent Templates
+
+### Base Image (`openclaw-agent-base:latest`)
+
+**Dockerfile:** `agent-templates/base-image/Dockerfile`
+
+Built from `node:24-bookworm-slim`, includes:
+- OpenClaw CLI + DeepSeek provider plugin
+- `openssl` (for local token generation)
+- Custom entrypoint `/agent-entrypoint.sh`
+
+**Entrypoint behavior:**
+- Resolves the auth token (file > env > local random)
+- Does NOT generate `openclaw.json` — OpenClaw creates its own default config on first boot
+- **Version-gated migration:** if the OpenClaw version differs from the one recorded in `/root/.openclaw/.last-version` (i.e. the image was rebuilt to a newer OpenClaw), runs `openclaw doctor --non-interactive` **before** starting the gateway. This applies only safe migrations (config normalization + on-disk state moves), skips service restarts, and aligns the persistent volume's state/config to the new binary. Best-effort — never blocks boot. Runs at most once per version change, not on every restart.
+- Starts OpenClaw gateway with `--allow-unconfigured`: `openclaw gateway --bind lan --port 3000 --auth token --token "<token>" --allow-unconfigured`
+- Everything else (controlUi, model refs, providers) is handled by the create route and the Gateway sync module
+
+### Template directories
+Located in `agent-templates/`. Each subdirectory (e.g. `atlas/`, `custom/`) contains:
+- `AGENTS.md`, `SOUL.md`, `IDENTITY.md`, `MEMORY.md`, `TOOLS.md`, `HEARTBEAT.md`
+- `BOOTSTRAP.md` (optional) — first-run ritual, deleted by the agent when complete
+
+At container creation these files are **copied** into `/root/.openclaw/workspace/`
+via `docker cp` (not bind-mounted), so agent edits never spill back to the Rev4a
+repo. The copy is skipped when the workspace is already populated (see
+[Agent persistence](#agent-persistence--lifecycle)).
+
+---
+
+## Gateway Page (`/gateway`)
+
+Two panels:
+
+### Provider Sync
+- Reads active models from `models.config.json` (file-based model catalogue)
+- Scans providers with keys in `data/provider-keys.json`
+- `PUT /api/gateway/provider` runs `syncAllAgents()` which writes `models.providers.rev4a` to every agent container
+- **Does NOT touch** `agents.defaults.model` or `agents.list[].model` — model references are managed per-container from the Agents Tab
+
+### Agent Model Config
+- Select a container and set its primary model
+- `PUT /api/gateway/agent` writes the model ref (`primary`, `fallbacks`) directly into the container's `openclaw.json`
+- Does NOT restart the gateway — writes are live via file write
+
+### Model Catalogue (`models.config.json`)
+File-based, tracked in git. Each entry:
+```json
+{ "id": "deepseek/deepseek-v4-flash", "name": "DeepSeek V4 Flash", "provider": "deepseek", "enabled": true }
+```
+Models with `enabled: false` are ignored.
+Only models whose provider has a key in `data/provider-keys.json` are synced.
+
+---
+
+## Containers
+
+### openclaw-atlas
+
+| Property | Value |
+|---|---|
+| Image | `openclaw-agent-base:latest` (Node 24-bookworm-slim + OpenClaw) |
+| Port | `0.0.0.0:3731 → 3000` |
+| IP | `172.19.0.3` |
+| AGENT_ID | `atlas` |
+| Config mount | `/docker/atlas-data/openclaw-fixed.json → /root/.openclaw/openclaw.json` |
+| Auth mode | `token` |
+
+**Control UI:** `http://<vps-ip>:3731` — enter the gateway token to log in.
+Rev4a-generated direct links should use `#token=<gateway-token>` for automatic Control UI sign-in.
+
+### openclaw-core
+
+| Property | Value |
+|---|---|
+| Image | `ghcr.io/hostinger/hvps-openclaw:latest` |
+| Ports | `:3711-3719` (mapped to container `:3700-3708`) |
+| AGENT_ID | `core` |
+
+### openclaw-giacomo
+
+| Property | Value |
+|---|---|
+| Image | `ghcr.io/hostinger/hvps-openclaw:latest` |
+| Port | `:3730` |
+| AGENT_ID | `giacomo` |
+
+### Other containers
+
+| Name | Image | Notes |
+|---|---|---|
+| `hermes` | `nousresearch/hermes-agent:latest` | Hermes agent |
+| `hermes-dashboard` | `nousresearch/hermes-agent:latest` | Hermes dashboard |
+| `traefik-traefik-1` | `traefik:latest` | Reverse proxy |
+
+---
+
+## Key Gotchas
+
+1. **`auth.mode: "none"` + `0.0.0.0:3000`** → OpenClaw refuses to bind. Must be `auth.mode: "token"`.
+2. **Control UI password** is the `gateway.auth.token`, not an env var.
+3. **API routes need full build** — after editing `route.ts`, run `npm run build`. Server restart alone is not enough.
+4. **Container directory listing** uses `docker exec` with `ls -1Ap` (single line per entry, trailing `/` on directories). Do NOT use `ls -la` with regex parsing.
+5. **Subdirectory detection** for containers: `docker exec … test -d "$path" && echo YES || echo NO` before deciding to list or read.
+6. **Service worker cache** (`public/sw.js`): if stale versions appear in the browser, delete this file and rebuild. Currently removed.
+7. **Browser cache on workspace page:** the API uses `cache: 'no-store'` but the browser may still cache the rendered page. Use hard refresh (Ctrl/Cmd+Shift+R) or incognito mode to verify.
+
+---
+
+## Troubleshooting
+
+### Workspace shows old files
+1. Hard refresh (Ctrl/Cmd+Shift+R)
+2. Open incognito/private tab
+3. Check API directly: `curl -H "authorization: Bearer <token>" 'http://127.0.0.1:3740/api/workspace?workspace=vps'`
+
+### API returns 500
+- For VPS: check file permissions on the workspace path
+- For containers: `docker exec` may fail on certain directories
+
+### "Empty workspace"
+- Verify the API returns files (test with curl)
+- Ensure the `workspace` parameter is a valid workspace ID
+
+### Old version persists after rebuild
+1. Kill process: `fuser -k 3740/tcp`
+2. Wait 2 seconds
+3. Restart: `sudo systemctl restart rev4a.service`
+4. Check `public/sw.js` exists (delete if so)
+
+---
+
+## API Test Commands
+
+```bash
+# Login (get cookie)
+curl -c /tmp/cookies.txt -X POST http://localhost:3740/api/auth/login \
+  -H 'Content-Type: application/json' \
+  -d '{"password":"***"}'
+
+# List workspaces
+curl -b /tmp/cookies.txt 'http://localhost:3740/api/workspace?action=list'
+
+# List VPS workspace files
+curl -b /tmp/cookies.txt 'http://localhost:3740/api/workspace?workspace=vps'
+
+# List Atlas container files
+curl -b /tmp/cookies.txt 'http://localhost:3740/api/workspace?workspace=container-openclaw-atlas'
+
+# Read a file
+curl -b /tmp/cookies.txt 'http://localhost:3740/api/workspace?workspace=vps&path=/path/to/workspace/SOUL.md'
+
+# Write a file
+curl -b /tmp/cookies.txt -X PUT http://localhost:3740/api/workspace \
+  -H 'Content-Type: application/json' \
+  -d '{"workspace":"vps","path":"/path/to/workspace/test.md","content":"hello"}'
+```
+
+---
+
+## Repository Structure
+
+```
+/path/to/rev4a/
+├── app/                     # Next.js app router pages & API routes
+│   ├── agents/
+│   ├── api/                 # API routes (workspace, agents, vault, auth, …)
+│   ├── components/          # Reusable React components
+│   ├── containers/
+│   ├── dashboard/
+│   ├── login/
+│   ├── memory/
+│   ├── plugins-skills/
+│   ├── skills/
+│   ├── tools/
+│   ├── vault/
+│   └── workspace/
+├── docs/                    # Project documentation
+│   ├── REV4A.md           # This file — operational docs
+│   └── ARCHITECTURE.md      # Design & future vision
+├── lib/                     # Shared utilities (apiFetch, vault, memory-context)
+├── public/                  # Static assets
+├── scripts/                 # Utility scripts (setup, helpers)
+├── .env                     # Local env vars (gitignored)
+├── proxy.ts                  # Auth proxy
+├── next.config.mjs
+└── package.json
+```