agent-usage-manager 0.1.3__tar.gz → 0.2.2__tar.gz

{agent_usage_manager-0.1.3 → agent_usage_manager-0.2.2}/.github/workflows/ci.yml

@@ -5,6 +5,9 @@ on:
     branches: [main]
   pull_request:
 
+permissions:
+  contents: read
+
 jobs:
   test:
     strategy:

{agent_usage_manager-0.1.3 → agent_usage_manager-0.2.2}/.gitignore

@@ -7,3 +7,4 @@ agents.local.yaml
 dist/
 build/
 *.egg-info/
+.gstack/

agent_usage_manager-0.2.2/PKG-INFO

@@ -0,0 +1,415 @@
+Metadata-Version: 2.4
+Name: agent-usage-manager
+Version: 0.2.2
+Summary: htop for AI agents — liveness, CPU/mem/GPU usage, and a kill switch for headless agents (openclaw, hermes, ollama, vllm, claude-code).
+Project-URL: Homepage, https://github.com/minglong51/agent-usage-manager
+Project-URL: Repository, https://github.com/minglong51/agent-usage-manager
+License: MIT
+License-File: LICENSE
+Keywords: ai-agents,gpu,llm,monitoring,observability,ollama,vllm
+Requires-Python: >=3.9
+Requires-Dist: fastapi>=0.110
+Requires-Dist: psutil>=5.9
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: uvicorn[standard]>=0.27
+Provides-Extra: dev
+Requires-Dist: httpx>=0.27; extra == 'dev'
+Requires-Dist: pytest>=7; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# agent-usage-manager
+
+A tiny, single-file web dashboard for **headless AI agents** running on a machine —
+OpenClaw, Hermes, Claude Code, Ollama, vLLM, llama.cpp, or anything you name. It
+shows which agents are alive and what they're costing you (CPU, memory, GPU), and
+gives you a **kill button** per agent. Think `htop`, scoped to just your agents —
+the [screenshot below](docs/dashboard.png) is a real run on a fleet node.
+
+No database, no auth framework (one static token file gates the kill switch),
+no dependencies beyond FastAPI + psutil. Runs on macOS and Linux. It is a
+per-node monitor and guarded local control panel: fleet schedulers may consume
+its read-only telemetry, but should own their own scheduling and actuation.
+
+![agent-usage-manager — live dashboard](docs/dashboard.png)
+
+*A real run: ten agents grouped by process tree (`+N` = children rolled up),
+per-agent CPU/memory/uptime, launchd-supervised jobs flagged, and a kill button
+per row.*
+
+```
+AGENT          PID    STATUS     CPU %   MEM MB   GPU MB   UPTIME   COMMAND          ┆
+openclaw +3    48213  ● running    62.4    1840     7320     2h 11m  openclaw serve … [kill] [force]
+claude-code +9 73590  ● running    97.4    7630        —     1h 02m  claude --chann … [kill] [force]
+hermes         49001  ● running    18.0     512        —     44m     hermes worker …  [kill] [force]
+ollama         50122  ● running     3.1    9210    14080     6h 02m  ollama runner …  [kill] [force]
+```
+(`+N` = child processes rolled up under the agent; CPU/mem/GPU are tree totals.)
+
+> The schematic above shows the GPU column (NVIDIA only); the screenshot is a real
+> run on Apple Silicon, where per-process GPU stats aren't available so that column
+> is hidden. The UI auto-refreshes every 3s.
+
+## What it does
+
+- **One row per agent.** Agents are grouped by process tree — the spawned children
+  of an agent (inference subprocesses, MCP servers, helpers) are rolled up under it
+  with a `+N` badge instead of cluttering the list as separate rows.
+- **Liveness** — green dot = running, red = zombie/dead. Status column shows the OS state.
+- **Usage** — CPU %, resident memory (MB), GPU memory (MB, NVIDIA only), and uptime,
+  refreshed every 3s. **CPU/mem/GPU are tree totals** — the agent's true cost including
+  everything it spawned.
+- **Kill the tree** — `kill` sends SIGTERM to the agent *and its children* (so spawned
+  helpers don't leak resources), `force` sends SIGKILL. SIGTERM auto-escalates to
+  SIGKILL after 3s. The confirm dialog tells you how many child processes will stop.
+- **Trends, not just snapshots.** Each row has a CPU sparkline (last ~20 min, sampled
+  in the background even with no browser open), plus a **`hot 5m+`** badge when an
+  agent has been pegged ≥90% CPU for 5+ minutes, an **`idle 10m+`** badge when a
+  long-running agent has done nothing for 10+ minutes, and a **`churn ×N`** badge when
+  the same agent has died young 3+ times in 10 minutes — the states worth investigating
+  (runaway, possibly wedged, crash-looping under a supervisor). Churn is what hot/idle
+  can't see: a crash-looping process is a fresh pid every poll, so no per-process
+  window ever fills. A **`leak?`** badge fires when an agent's memory ratchets up
+  ≥30% (and ≥128 MB) over 15 minutes without coming back down.
+- **Alerts.** A dashboard only helps while you're looking at it. Add an `alerts:`
+  block to `agents.yaml` and any badge appearing runs your command (desktop
+  notification, Telegram bot, pager — anything) with the details in `$AUM_*` env
+  vars. Fires once per transition with a cooldown, never from the `list` CLI.
+  By default only `hot`/`churn`/`leak` alert — `idle` is the normal state of an
+  agent fleet that waits for work, so it's opt-in.
+
+  ```yaml
+  alerts:
+    command: 'terminal-notifier -title agents -message "$AUM_MSG"'
+    cooldown: 600
+    flags: [hot, churn, leak]   # the default; add idle to opt in
+    leak_floor_mb: 1536         # optional: leak alerts only above this RSS
+  ```
+
+  `leak_floor_mb` mutes leak *alerts* (the badge still shows) until the agent's
+  absolute footprint crosses the floor — agents that accrue working state, like
+  a chat bot growing its session context, ratchet RSS exactly like leakers, so
+  relative growth alone can be noisy. Crossing the floor counts as the
+  appearance, so a genuine ratchet still alerts.
+- **Prometheus `/metrics`.** Per-agent CPU/mem/instances/restarts and badge states
+  in text exposition format, aggregated per label (no pid-churn series bloat) —
+  point Grafana or any Prometheus scraper at `http://127.0.0.1:8765/metrics`.
+- **Expand the tree.** Click the `+N` badge to unfold an agent's child processes
+  (per-child CPU/mem/command) — see what a kill would actually stop before clicking it.
+- **Config hot-reload.** Edits to `agents.yaml` apply on the next poll, no restart.
+  A broken edit keeps the last good config and shows the parse error in the header.
+- **`list` subcommand.** `agent-usage-manager list` (or `list --json`) prints a one-shot
+  table to stdout — no server, good for scripts and cron checks.
+- **Kill-safe table.** Rows keep a stable order (sorted by label) and never reorder
+  while your pointer is over the table, so the kill button can't shift under your
+  cursor mid-click.
+
+## Product boundary
+
+`agent-usage-manager` is intentionally **not** a fleet scheduler, dispatcher, or
+multi-host orchestrator. It answers local process questions: what agent process is
+running here, what resources is its process tree using, did it enter a suspicious
+state, and can this local operator safely stop it?
+
+If you run a separate fleet control plane, treat AUM as an optional read-only
+input. Scrape `list --json`, `/api/agents`, or `/metrics` for local OS facts, then
+make scheduling, budget, restart, and kill/retire decisions in your own
+deterministic control layer. Do not route irreversible fleet operations through
+AUM's kill endpoint as a central substrate.
+
+## Safety
+
+This is the important part — a web page that can kill processes needs guardrails:
+
+- **Allowlist only.** Only processes matching a pattern in `agents.yaml` are ever
+  listed *or* killable. The kill endpoint re-checks the match server-side before
+  sending any signal, so the dashboard can never be used to kill an arbitrary PID.
+- **Protected patterns.** Anything matching `protect:` in `agents.yaml` — plus the
+  monitor's own process and PID 1 — shows a disabled, greyed-out kill button and is
+  refused server-side.
+- **Secret redaction.** Command lines often carry tokens/keys in env vars or flags
+  (`FOO_TOKEN=...`, `--api-key ...`, `sk-...`, `ghp_...`, JWTs). The command column
+  redacts these to `***` before they ever reach the browser — safe to screenshot.
+- **Browser guard (CSRF + DNS rebinding).** Binding to localhost doesn't keep
+  browsers out — any web page you visit can `fetch()` a localhost port. Requests
+  whose `Host` is a non-local DNS name are refused (DNS-rebinding guard), and a
+  kill request carrying a foreign `Origin` is refused (CSRF guard) — so a
+  malicious page can't kill your agents or read your process list. `curl` and
+  the dashboard itself are unaffected.
+- **Kill requires a token (caller authorization).** The allowlist above says what
+  *may* be killed; the token says *who* may kill. The monitored agents are
+  themselves untrusted HTTP callers — a prompt-injected agent with an HTTP tool
+  and localhost reach could otherwise `POST /api/kill` and take down its
+  siblings (Origin headers are trivially forged outside a browser). The token is
+  auto-generated on first run into a `0600` file —
+  `~/Library/Application Support/agent-usage-manager/kill_token` on macOS,
+  `$XDG_STATE_HOME/agent-usage-manager/kill_token` (default
+  `~/.local/state/…`) elsewhere — and every kill must send it as an
+  `X-Kill-Token` header. It is **never served over HTTP** (anything that can
+  curl the dashboard could read it): the dashboard asks you to paste it once on
+  your first kill and keeps it in the browser's localStorage. Delete the file
+  to rotate the token.
+- **Action log.** Every kill attempt — success *and* every refusal — appends a
+  JSON line to `actions.log` next to the token file: timestamp, caller address,
+  target pid/command, outcome. "What was killed at 3am" and "what's been
+  probing the kill endpoint" both have an answer. Append-only, no rotation;
+  one line per attempt stays tiny.
+- **Non-loopback binds fail closed.** It listens on `127.0.0.1`; asking it to
+  bind anything else (`--host 0.0.0.0`, a LAN IP) refuses to start unless you
+  also pass `--unsafe-expose`. Exposing the port means one static token is all
+  that stands between the network and your agents — put real auth in front
+  (reverse proxy + basic auth, SSH tunnel, etc.) before using that flag.
+
+## Limits & known issues
+
+- **GPU column is NVIDIA-only.** Per-process GPU memory comes from
+  `nvidia-smi --query-compute-apps` — NVIDIA compute processes (CUDA), in
+  practice on Linux. AMD/Intel GPUs aren't read, graphics-only workloads don't
+  appear, and Apple Silicon has no per-process GPU accounting API at all, so
+  the column is hidden on Macs.
+- **Supervision detection is launchd-only (macOS, user domain).** Root
+  `LaunchDaemons` aren't flagged — that needs a privileged
+  `launchctl print system/…`. On Linux, systemd-supervised services
+  (`Restart=always`) aren't detected either, so killing one looks like it
+  failed when systemd respawns it — use `systemctl stop` for those.
+- **Same-user privileges only.** Signals are sent with the server's own
+  privileges. Agents running as another user (or root) are listed, but a kill
+  won't take (`killed: 0` in the response), and CPU/mem can read as 0 where
+  the OS denies access.
+- **History is in-memory.** Sparklines and the `hot`/`idle` flags (~20 min
+  window) rebuild from scratch after a server restart.
+- **Windows is untested.** Kill maps to `TerminateProcess` via psutil and may
+  work, but CI covers Linux + macOS only.
+
+## Quick start
+
+**Recommended — one command, nothing to install first:**
+
+```bash
+uvx agent-usage-manager
+# then open http://127.0.0.1:8765 (it also opens automatically)
+```
+
+`uvx` fetches and runs it in one step — no separate install, no virtualenv, no
+leftovers. Don't have [`uv`](https://github.com/astral-sh/uv) yet? One line:
+
+```bash
+curl -LsSf https://astral.sh/uv/install.sh | sh      # macOS / Linux
+# or: pip install uv
+```
+
+<details>
+<summary>Other ways to install</summary>
+
+```bash
+pipx install agent-usage-manager     # clean isolated global CLI (needs pipx)
+
+pip install agent-usage-manager      # universal; use inside a venv —
+                                     # system Python may refuse with
+                                     # "externally-managed-environment"
+```
+
+Then run `agent-usage-manager` (flags below).
+</details>
+
+From a clone (for hacking on it):
+
+```bash
+git clone <this-repo> && cd agent-usage-manager
+./run.sh                           # venv + editable install, serves on :8765
+```
+
+It opens the dashboard in your browser automatically. Flags: `--host`, `--port`,
+`--config /path/to/agents.yaml`, `--no-browser` (for headless/server use),
+`--unsafe-expose` (required for any non-loopback `--host` — see
+[Safety](#safety)).
+
+## Configure which processes are "agents"
+
+Edit `agents.yaml`:
+
+```yaml
+agents:
+  - label: openclaw           # shown as the badge in the UI
+    match: openclaw           # case-insensitive substring of the command line
+  - label: hermes
+    match: hermes
+  - label: claude-code
+    match: "claude(\\s|$|-code)"
+    regex: true               # treat `match` as a regex instead of substring
+
+protect:                      # matched + listed, but never killable
+  - uvicorn
+
+ignore:                       # never an agent: not listed, not killable
+  - crashpad                  # incidental processes that share a name/bundle
+  - shipit                    # path with a real agent (crash handlers,
+  - kiro-cli-term             # auto-updaters, integrated-terminal shells, …)
+```
+
+A process matches if the pattern hits its **executable basename + first few
+arguments** — deliberately not the whole command line, so a long embedded arg
+(e.g. a system prompt mentioning "claude") can't misclassify a wrapper. On macOS
+the outermost `.app` **bundle name** is also included, so GUI agents that launch
+a generically-named binary (Kiro.app → `Electron`) are still matched by app name.
+
+`protect:` keeps a matched process listed but refuses to kill it; `ignore:`
+drops it from agent classification entirely.
+
+**Telling identical agents apart (`tmux_labels:`)** — a fleet of same-binary
+agents (say five `claude` bots, one per tmux session) all hits one `agents:`
+entry and shows N indistinguishable rows; a distinguishing flag deeper in
+their command lines is invisible to matching *by design* (see above). When
+each instance runs in its own tmux session, the session name is its identity:
+
+```yaml
+tmux_labels: "^bot-(.+)$"   # session bot-coder_1 → row label coder_1
+```
+
+If a matched agent root (or an ancestor) is a tmux pane process whose session
+name matches the regex, the row's label becomes the first capture group (the
+whole session name if there's no group). Churn tracking, alert transitions,
+and `/metrics` series all use the derived label, so each instance gets its own
+state. Sessions that don't match the regex keep their `agents:` label, and the
+key is ignored where tmux isn't installed or running.
+
+**Which `agents.yaml` is used** — resolved once at startup, first hit wins:
+
+1. `AGENTS_CONFIG=/path/to/agents.yaml` env var (the `--config` flag sets this)
+2. `./agents.yaml` in the directory you launched from
+3. the default bundled with the package
+
+The dashboard header (and `list --json`) shows the resolved path, so you can
+always see which file is live. Hot-reload watches that one file. An
+`AGENTS_CONFIG` path that doesn't exist is an error at startup, not a silent
+fallback.
+
+## launchd-supervised agents (macOS)
+
+Some agents run as **launchd services** (a `~/Library/LaunchAgents/*.plist`, or
+anything started by `brew services`). If such a job sets `KeepAlive`, a signal
+can't stop it: the process dies, launchd immediately respawns it under a new PID,
+and the dashboard's "kill" looks like it silently failed.
+
+The dashboard detects these (via `launchctl list`) and marks them with a
+**`launchd`** badge. Instead of dead-end kill/force buttons it shows the command
+that actually stops the job — click to copy:
+
+```sh
+launchctl bootout gui/<uid>/<label>            # stop now
+launchctl disable gui/<uid>/<label>            # …and don't auto-start at login
+```
+
+The kill endpoint refuses signals for these jobs (HTTP 409) and returns the same
+guidance, so the API never lies about a kill that won't stick. The message is
+tailored to the job: `KeepAlive` jobs are told a signal won't stick at all;
+`RunAtLoad`-only jobs are told a signal works now but the job restarts at next
+login. *Limitation:* root `LaunchDaemons` aren't flagged — see
+[Limits & known issues](#limits--known-issues).
+
+## GPU notes
+
+Per-process GPU memory comes from `nvidia-smi` when it's on `PATH` (Linux / NVIDIA).
+**Apple Silicon has no per-process GPU accounting API**, so the GPU column stays blank
+on Macs — CPU and memory are the meaningful resource signals there.
+
+## API
+
+- `GET  /api/agents` → `{ api_version, aum_version, agents: [...], host, cpu_count, mem_total_mb, mem_used_pct, config_path, config_error, ts }`
+  — each agent includes read-only telemetry such as `pid`, `create_time`, `label`,
+  resource totals, recent CPU `trend`, flags (`hot`, `idle`, `churn`, `leak` when
+  present), protection state, and supervised-process guidance. Pair `pid` with
+  `create_time` when caching rows so PID reuse cannot alias two different agents.
+  This endpoint is suitable as an input to external tools, not as a fleet-control
+  contract.
+- `GET  /api/tree/{pid}` → the agent's process subtree (per-child pid/name/cpu/mem/cmdline);
+  only works on recognized agents, same authorization as kill
+- `POST /api/kill/{pid}?force=false` → SIGTERM (or SIGKILL with `force=true`).
+  Requires the `X-Kill-Token` header — the token lives in the `0600` file shown
+  in the 403 message (see [Safety](#safety)):
+
+  ```bash
+  curl -X POST -H "X-Kill-Token: $(cat ~/Library/Application\ Support/agent-usage-manager/kill_token)" \
+    http://127.0.0.1:8765/api/kill/48213
+  ```
+
+## Run as a service
+
+Linux (systemd), `~/.config/systemd/user/agent-usage-manager.service`:
+
+```ini
+[Unit]
+Description=agent usage manager
+[Service]
+ExecStart=%h/agent-usage-manager/.venv/bin/uvicorn app:app --port 8765
+WorkingDirectory=%h/agent-usage-manager
+Restart=on-failure
+[Install]
+WantedBy=default.target
+```
+
+```bash
+systemctl --user enable --now agent-usage-manager
+```
+
+## Development
+
+```bash
+git clone https://github.com/minglong51/agent-usage-manager && cd agent-usage-manager
+pip install -e ".[dev]"
+pytest -q
+```
+
+CI runs the test suite on Linux + macOS (Python 3.9 and 3.12) on every push and PR.
+Cross-platform note: kill uses psutil's `terminate()`/`kill()`, which map to
+SIGTERM/SIGKILL on POSIX and TerminateProcess on Windows.
+
+## Release notes
+
+### 0.2.2 — unreleased
+
+- Added `api_version` and `aum_version` to `/api/agents` and `list --json`.
+- Added per-agent `create_time` so external telemetry consumers can pair it with
+  `pid` and avoid PID-reuse aliasing.
+- Added `tmux_labels:` — derive per-instance row labels from tmux session names,
+  so a fleet of identical agents stops rendering as N indistinguishable rows.
+
+### 0.2.1 — security and verification hardening
+
+- Kill endpoint now requires caller authorization via the static token file.
+- Non-loopback binds fail closed unless `--unsafe-expose` is explicitly passed.
+- Kill attempts and refusals append to the local action log.
+- Added deterministic kill-path regression tests, including pid/create_time pins.
+- Added synthetic hot/idle/churn/leak trace fixtures.
+- Added adversarial matcher cases so lookalike process names stay test-covered.
+
+## Troubleshooting
+
+- **`pip install` fails building psutil** — no prebuilt wheel for your
+  Python/platform, so pip compiles it: you need a C toolchain and Python
+  headers (`xcode-select --install` on macOS; `apt install gcc python3-dev`
+  on Debian/Ubuntu). Or skip the problem with `uvx agent-usage-manager`.
+- **Dashboard is empty / "No matching agents running"** — first check which
+  config was picked up (resolution order above; the header shows the resolved
+  path). Then remember matching is against the executable basename + first few
+  arguments, not the full command line — a pattern that only appears deep in
+  the args won't match.
+- **Kill "doesn't work" — the agent comes back under a new PID** — it's
+  supervised. On macOS the row gets a `launchd` badge with the `launchctl
+  bootout` command that actually stops it; on Linux, systemd services aren't
+  detected (see limits) — `systemctl stop` them. A `killed: 0` in the kill
+  response means nothing was actually signaled (e.g. the agent runs as
+  another user).
+- **HTTP 403 on every request** — the DNS-rebinding guard refuses non-local
+  hostnames. Use `http://127.0.0.1:8765` (or a bare IP) instead of a custom
+  DNS name pointing at the box.
+- **HTTP 403 on kill only ("Kill requires the X-Kill-Token header")** — send
+  the token from the file named in the message. In the dashboard, the paste
+  prompt reappears on your next kill click (a stored stale token is forgotten
+  automatically when the server rejects it).
+- **"refusing to bind …" at startup** — non-loopback `--host` values fail
+  closed; add `--unsafe-expose` only with auth in front (see
+  [Safety](#safety)).
+
+## License
+
+MIT