eggpool 0.3.1__tar.gz → 0.3.3__tar.gz

{eggpool-0.3.1 → eggpool-0.3.3}/AGENTS.md

@@ -52,8 +52,12 @@ All four must pass with zero errors.
 
 ## Multi-Provider Architecture
 
+> See `architecture/README.md` and the `architecture` skill for the full design, worked examples, and MiniMax template details.
+
 - Provider-suffixed model IDs: `model-id/provider-id` (e.g., `claude-sonnet-4/opencode-go`)
-- `ProviderClientPool` manages per-provider `httpx.AsyncClient` with independent connection pools
+- `ProviderClientPool` manages per-provider `httpx.AsyncClient` with independent connection pools for upstream LLM forwarding and catalog model-list fetches
+- `OutboundClientManager` owns a shared `httpx.AsyncClient` for non-provider network paths (update checks, external catalog fetches). Initialized once at startup; `build_count` should stabilize at 1. Accepts `[network]` config for transport tuning and `[network.dns_cache]` for in-memory DNS caching (enabled by default, TTL 300s, max 50 entries). The DNS cache wraps the default httpcore transport so resolved entries are reused across requests. Exposes `snapshot()` with `build_count`, `request_count`, `error_count` for runtime diagnostics. `inject_client()` is the test escape hatch
+- Hot-path provider requests must **never** construct fresh HTTP clients. Background and CLI paths should use the shared outbound client. `warn_adhoc_clientConstruction()` emits a runtime warning after startup when fresh clients are built outside managed paths
 - Flat `[[accounts]]` configs auto-normalize to a default `opencode-go` provider
 - `parse_model_provider()` and `format_model_provider()` in `routing/provider.py`
 
@@ -85,7 +89,7 @@ Use the hierarchy in `errors.py`. Chain exceptions with `raise ... from err` or
 ## Gotchas
 
 - Configuration changes require a service restart; live reload is intentionally not supported
-- No CI workflows or pre-commit hooks are configured in this repo
+- No pre-commit hooks are configured in this repo; CI runs ruff, pyright, and pytest via GitHub Actions
 - `Database.vacuum()` is the only sanctioned path for `VACUUM` in production code
 - Every DML write must run inside `async with db.transaction():`
 - SQLite transactions are serialized across concurrent tasks via a single connection lock + ContextVar
@@ -96,11 +100,19 @@ Use the hierarchy in `errors.py`. Chain exceptions with `raise ... from err` or
 - **Streaming finalizer shielding**: streaming `_build_stream_generator` finalization runs under `asyncio.shield(asyncio.wait_for(..., timeout=10))` so ASGI task cancellation cannot kill the finalizer while it holds the DB lock. Leaks that escape this path are caught by the periodic `stale_request_finalizer` background task (`app._finalize_stale_requests`, runs every 60s) which force-finalizes any request that has been `pending` longer than `upstream.read_timeout_s`.
 - **Startup crash recovery**: `_crash_recovery` runs at every startup and recovers ALL pending requests and ALL active reservations with no time threshold. A process restart is a definitive boundary, so leaked state from the previous process is unconditionally cleaned up. If `Crash recovery: marked N stale requests` appears in logs, the safety net caught leaks from the previous run.
 - **Pricing resolution pipeline**: prices flow TOML override → upstream metadata → external catalog (OpenRouter / OpenCode Zen via the alias registry). `ResolvedPricing` records `source_detail` (`operator_override` / `provider_metadata` / `openrouter` / `opencode_zen`) and `source_confidence` (`exact_external_id` / `curated_alias` / `provider_metadata`). Cost rows are then labelled `derived` (every category trusted), `partial` (some categories filled by per-category fallback), `estimated` (no trusted rates), or `unknown` (no token usage). `partial_count` is a new exactness value exposed via `/api/stats/summary`, `/api/stats/accounts`, and `/api/stats/models`. The dashboard renders a per-row cost-exactness badge and a high-spend estimated warning banner (>$10 estimated) on the Accounts page.
+- **Provider-reported cost precedence**: when an upstream includes a billing field (e.g. `usage.cost`, `usage.cost_usd`, `usage.cost_microdollars`, `usage.billing.cost_usd`), `eggpool.proxy.cost_reporting.extract_provider_reported_cost()` parses it and the `RequestFinalizer` records it on the `requests` row as `provider_cost_microdollars` (with `provider_cost_source` storing the field path). That value takes precedence over locally computed rates — the cost precedence ladder is `provider_reported > derived/partial/exact > estimated (reservation fallback) > unknown (zero)`. Locally calculated cost is still persisted to `local_cost_microdollars` / `local_cost_exactness` for diagnostics, never overrides provider-reported. The new `provider_reported_count` and `provider_reported_cost_microdollars` fields are exposed via `/api/stats/summary`, `/api/stats/accounts`, and `/api/stats/models` and surface in the dashboard's exactness badge and Total-cost card. Migration `0033_request_provider_local_cost.sql` adds the four nullable audit columns. The parser is allowlisted (no arbitrary nested keys) and is defensive — `TypeError`/`ValueError`/`AttributeError`/`RecursionError` yield `None`, so finalization never breaks on a malformed response body.
 - **`eggpool stats recompute-costs [--dry-run|--apply] [--limit N]`**: walks the requests table in started_at DESC order, recomputes cost from the current price snapshots, and reports / applies the change. Default is `--dry-run`. Use after upgrading the resolver to fix inflated totals on cached-token-heavy models (e.g. MiMo 2.5). Implemented in `src/eggpool/cost_recompute.py` and reuses the live `CostCalculator` so the new values match what the finalizer would write today.
 - **Migration 0030 (`model_pricing_aliases`) + 0031 (`price_snapshot_provenance`)**: 0030 introduces the alias registry that maps upstream model IDs (e.g. `mimo-v2.5`) onto external catalog IDs (e.g. `xiaomi/mimo-v2.5`) with an `exact`/`curated_alias`/`ambiguous_skip` confidence enum. 0031 adds `source_detail`, `source_confidence`, `catalog_source` columns to `model_price_snapshots` so the dashboard can attribute prices back to the resolver that produced them. Seed data lives in `seed_default_aliases()` (`src/eggpool/catalog/pricing_aliases.py`) and runs idempotently at startup via `CatalogService.attach_pricing_resolvers()`.
+- **Migration 0032 (`usage_rollups`)**: introduces the rollup table for buffered analytics. Counter fields are designed for additive upserts (`INSERT ... ON CONFLICT DO UPDATE SET col = col + excluded.col`). Latency min/max use `CASE/WHEN` to converge monotonically within each bucket. The `UsageRollupRepository` provides `upsert_many()`, `query_timeseries()`, `query_summary()`, and `cleanup_old_rollups()`.
+- **Low-wear metrics buffering**: `MetricsWriteCoalescer` buffers lossy analytics events in memory and periodically flushes to `usage_rollups`. Correctness-critical writes (request state, reservations, routing) remain immediate. `write_mode = "balanced"` (default) uses 30s flush intervals; `write_mode = "low_wear"` uses 120s with coarser buckets. Buffered data may lose at most `flush_interval_s` seconds after abrupt power loss. The coalescer is wired into `RequestFinalizer.finalize()` and emits one `UsageMetricEvent` per terminal transition. Runtime diagnostics expose buffer health via `/api/stats/runtime`.
+- **Dispatch overhead and OS load average on `/runtime`**: `DispatchOverheadRecorder` (`src/eggpool/runtime_dispatch.py`) records `time.perf_counter_ns() - context.started_monotonic_ns` immediately before `client.send(...)` in both `_execute_non_streaming` and `_execute_streaming`, on every upstream attempt (so retries contribute). The recorder is bounded (`deque(maxlen=100)`, thread-safe), stores only integer nanoseconds (no body, model ID, account name, auth header, or client IP), and never writes to SQLite. `RuntimeMetricsService.snapshot()` exposes `dispatch_overhead` (avg/min/max/p50/p95) and `load` (`os.getloadavg` 1m/5m/15m + normalized per-core; `available: false` on platforms without it). The Runtime dashboard drops the configured-thread and process-count cards in favor of `Active threads`, `Load average`, and `Dispatch overhead`; process-count anomalies still surface as a warning-only panel
+- **Automatic backups**: in-process daily backups run by default under the `automatic_backup` supervised task (`src/eggpool/background/backup.py`). Uses stdlib `sqlite3.Connection.backup()` for consistent snapshots, atomic archive publication (write-to-temp + rename), and count-based retention (default 14). Controlled by `[backup]` config section. The `eggpool deploy backup-cron` path remains available for operators who prefer external scheduling.
+- **DNS cache**: `OutboundClientManager` and `ProviderClientPool` both integrate a `DnsNetworkBackend` that caches resolved DNS entries in memory. The cache reduces connection latency for repeated requests to the same upstream hosts. Controlled by `[network.dns_cache]` config. When a proxy is configured for an account, that account's client uses the proxy transport instead of the cached backend.
 
 ## Observability
 
+> Full API surface, attempt analytics, routing analytics, and latency phase details are in the `architecture` skill.
+
 - **Attempt analytics**: per-attempt aggregates including latency percentiles, byte totals, retry rate, and the `retry_category` distribution. Every `request_attempts` row carries `provider_id/model_id/protocol/retry_category/release_reason/bytes_received/latency_ms/streamed/is_retry_outcome`
 - **Routing analytics**: per-`(model, provider)` decision aggregates, account-level selection counts, and per-`(account, reason)` exclusion counts. Every routing decision is persisted as a `routing_decisions` row inside the same transaction as the `request_attempts` INSERT
 - **Latency phases**: decomposes each request into `upstream_connect_ms`, `upstream_read_ms`, and `coordinator_overhead_ms`
@@ -109,16 +121,43 @@ Use the hierarchy in `errors.py`. Chain exceptions with `raise ... from err` or
 - **Per-request trace**: parent request row, full attempt chain, and per-attempt routing decisions. Returns account name, model, protocol, status, error class (never raw error_detail), and timing. Auth-gated
 - **Recent request metadata**: bounded list of recent request rows with metadata only (no body, no auth headers, no error_detail). Auth-gated
 - **Cost/cache/reasoning exactness**: per-account and per-model `exact_count`, `partial_count`, `derived_count`, `estimated_count`, `cache_read_ratio`, `cache_write_ratio`, `reasoning_output_ratio`
+- **Metrics buffer health**: `metrics.write_mode`, `metrics.flush_interval_s`, `metrics.buffered_keys`, `metrics.buffered_events`, `metrics.total_events_received`, `metrics.total_events_flushed`, `metrics.total_events_dropped`, `metrics.last_flush_ts`, `metrics.last_flush_rows`, `metrics.last_flush_duration_ms`, `metrics.last_flush_error`. Exposed via `/api/stats/runtime` and `eggpool runtime-status`
+- **Network diagnostics** (`/api/network/diagnostics`): sanitized outbound client lifecycle (build count, request count, error count, per-scope builds, per-host requests/errors) and DNS cache behavior (hits, misses, negative hits, stale hits, evictions, per-host breakdown, resolution errors, per-entry metadata with state/TTL/staleness). Always auth-gated. Also displayed on the `/runtime` dashboard page and in `eggpool runtime-status` output. No API keys, auth headers, request bodies, or full URLs are exposed
 - Full API surface is documented in the `architecture` skill
 
 ## Dashboard
 
+> Full page list, chart lifecycle, grouped timeseries, tooltip system, and responsive/mobile details are in the `architecture` skill.
+
 - Server-rendered HTML pages in `src/eggpool/dashboard/render.py`
 - Overview page auto-refreshes in place (every `[dashboard].refresh_interval_s`); all other pages are static
 - Charts use bundled Chart.js v4 at `/static/chart.js` with `Cache-Control: public, max-age=86400`
 - New pages opt into Chart.js via `include_chart_js=True` in `_render_layout`
 - Frontend helpers in `src/eggpool/dashboard/static/dashboard.js` under `window.EggPoolDashboard`
 - Full page list and chart lifecycle details are in the `architecture` skill
+- **`header.topbar` is `position: sticky; top: 0; z-index: 5`** with a subtle backdrop blur so the page nav stays visible while scrolling on desktop. Mobile layout is unchanged — the topnav disclosure still wraps cleanly under 480px. Use `header.topbar` selectors when overriding; do not collapse the existing z-index above the value used by `body::before` (2) or the egg background (0)
+- **Footer update indicator**: `_render_update_indicator(update_info)` renders an empty string when `update_info is None` or `update_info.update_available is False`. The indicator is appended to the footer after `<span id="dashboard-updated">ready</span>` and contains a `data-update-command` markup hook consumed by `dashboard.js` `initUpdateCommandCopy()`. The hook uses Clipboard API with `document.execCommand("copy")` fallback for older browsers and surfaces a transient "copied!" indicator on Enter/Space/click
+
+### Update Checker
+
+- `src/eggpool/update_checker.py` is the single source of truth for "is there a newer eggpool release available?"
+- `UpdateChecker` (frozen `UpdateInfo` dataclass) holds the latest snapshot; `snapshot()` returns `dataclasses.replace(self._info)` so callers cannot mutate the cached state. Writes are serialized through an `asyncio.Lock` so the periodic background task and synchronous `snapshot()` calls from request handlers do not tear
+- `async_check_for_update()` is the shared helper used by both the dashboard background task and the `eggpool update` CLI — they MUST go through it instead of inlining their own PyPI lookup so the two paths cannot drift
+- Default check interval is 24h (`_DEFAULT_CHECK_INTERVAL_S = 24 * 60 * 60`), timeout 15s (`_CHECK_TIMEOUT_S = 15.0`)
+- The background task is registered in `src/eggpool/app.py` via `supervisor.register("update_checker", update_checker.run_periodic)` and uses the default supervisor `max_restarts` so transient PyPI failures do not give up
+- On PyPI failure, the checker preserves the previous `latest_version` so the indicator still surfaces a known-newer release during momentary outages
+- `/api/stats/update` (`src/eggpool/api/update.py`) returns the JSON snapshot — always auth-gated regardless of `dashboard.public`
+- Add new dashboard footer UI by extending `_UPDATE_INDICATOR_TEMPLATE` in `src/eggpool/dashboard/render.py:259`; ensure `_render_update_indicator` continues to return `""` whenever an update is not available so the footer does not show a stale or "no update" pill
+
+### Responsive / mobile
+
+- Dashboard targets a **320px minimum** viewport (iPhone SE / small Android). All 12 pages are mobile-equally
+- Topnav wraps page links in a `<details class="topnav-hamburger">` disclosure; below 480px the `<summary>` becomes a Menu chip and the 12 links collapse into it. Theme selector and refresh button stay **outside** the disclosure so they remain reachable on every viewport
+- Tables use `data-priority="N"` on every `<th>` and matching `<td>` to drive responsive column hiding: `P1` always shown, `P2` hidden below 480px, `P3` hidden below 760px. Renderers MUST use the `_th(label, *, priority=1)` and `_td_priority(content, priority, *, class_=None)` helpers in `src/eggpool/dashboard/render.py:460` — never emit a bare `<th>` for a tabular column
+- Chart.js canvases MUST sit inside a `<div class="chart-wrap" style="height: …">` (not an inline-style `<div style="position: relative; height: …">`). `.chart-wrap` sets `position: relative; width: 100%` so the canvas respects its parent panel's width on every viewport
+- `html { overflow-x: hidden }` is a deliberate safety net in `dashboard.css` to prevent chart-canvas resize races from causing horizontal scroll on the body. It hides real overflow bugs, so any new layout code should still design for `clientWidth == scrollWidth` even though it's masked
+- New tables: pick priority mappings that put operator-glance columns in P1, diagnostic core in P2, and deep-tail columns in P3. Update `tests/unit/test_dashboard.py` `TestResponsiveColumns` with a snapshot test
+- Adding a new topnav link requires updating both `_render_nav` and the breakpoints in `dashboard.css` (`@media (max-width: 480px)` and `topnav-hamburger > .topnav-links { … }`)
 
 ## Fast-Path CLI
 
@@ -186,7 +225,7 @@ Use the hierarchy in `errors.py`. Chain exceptions with `raise ... from err` or
 | `eggpool stats recompute-costs [--dry-run\|--apply] [--limit N]` | Recompute historical `cost_microdollars` from current price snapshots. Default `--dry-run`. |
 | `eggpool deploy systemd` | Print systemd unit; `--install` writes it (personal by default; `--production` for the dedicated-system layout; `--as-root` for a root-owned personal unit) |
 | `eggpool deploy cron` | Print / install / uninstall the watchdog crontab (`@reboot` + `*/N * * * *` `eggpool ensure-running`). `--interval N` (1-59, default 5) |
-| `eggpool deploy backup-cron` | Print / install / uninstall the daily backup cron (personal user cron or production `/etc/cron.d/`) |
+| `eggpool deploy backup-cron` | Print / install / uninstall the daily backup cron (personal user cron or production `/etc/cron.d/`). Optional — in-process automatic backup runs by default |
 | `eggpool deploy logrotate` | Print / install / logrotate config (validated via `logrotate -d`) |
 | `eggpool deploy all` | Print / install systemd + logrotate + watchdog cron (backup-cron is separate) |
 | `eggpool backup` | Create a timestamped `.zip` backup of config, `.env`, and database |

{eggpool-0.3.1 → eggpool-0.3.3}/CHANGELOG.md

@@ -7,6 +7,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- **Sticky dashboard topbar**: `header.topbar` is `position: sticky; top: 0; z-index: 5` with a subtle backdrop blur, so the page navigation stays visible while scrolling on desktop. Mobile layout is unchanged (the topnav disclosure still wraps cleanly under 480px).
+- **Footer update indicator**: periodic PyPI check (default 24h interval, 15s timeout) drives a footer pill that appears only when a newer `eggpool` release is available. The pill shows the current and latest versions side-by-side and the one-liner command (`eggpool update`) in an inline-code block. Clicking the command copies it to the clipboard via the bundled `dashboard.js` (Clipboard API with `execCommand("copy")` fallback); a transient "copied!" indicator confirms success. The new `src/eggpool/update_checker.py` module is the single source of truth for PyPI lookups — both the dashboard background task and the `eggpool update` CLI share `async_check_for_update()` so the two paths cannot drift.
+- **`/api/stats/update` endpoint**: auth-gated JSON snapshot of the latest `UpdateChecker` state (`current_version`, `latest_version`, `update_available`, `last_checked_at`, `last_error`). Returns an empty payload if the checker has not yet produced a snapshot. Always auth-gated regardless of `dashboard.public`.
+- **Runtime dispatch overhead and load metrics**: `DispatchOverheadRecorder` (`src/eggpool/runtime_dispatch.py`) records `time.perf_counter_ns() - context.started_monotonic_ns` immediately before `client.send(...)` in both `_execute_non_streaming` and `_execute_streaming`, on every upstream attempt (retries included). Bounded `deque(maxlen=100)`, thread-safe, integer-nanosecond storage — no body, model ID, account name, auth header, or client IP ever enters the buffer. `RuntimeMetricsService.snapshot()` gains two top-level sections: `dispatch_overhead` (avg/min/max/p50/p95 over the last 100 attempts) and `load` (`os.getloadavg` 1m/5m/15m + normalized per-core; `available: false` on platforms without it). The Runtime dashboard drops the configured-thread and process-count cards in favor of `Active threads`, `Load average`, and `Dispatch overhead`; process-count anomalies surface as a warning-only panel. `eggpool runtime-status` and `docs/deployment.md` document the new metrics.
+
 ## [0.3.0] - 2026-06-25
 
 ### Added

{eggpool-0.3.1 → eggpool-0.3.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: eggpool
-Version: 0.3.1
+Version: 0.3.3
 Summary: A lightweight proxy that aggregates multiple LLM provider accounts behind one OpenAI-compatible endpoint
 Project-URL: Homepage, https://github.com/eggstack/eggpool
 Project-URL: Repository, https://github.com/eggstack/eggpool
@@ -145,12 +145,25 @@ eggpool deploy cron --install
 
 This writes a `@reboot` + `*/5 * * * *` `eggpool ensure-running` block to the invoking user's crontab (or `SUDO_USER`'s crontab under sudo). Use `--interval N` to change the poll cadence. The block is bracketed by `# BEGIN EggPool watchdog` / `# END EggPool watchdog` markers so uninstall only strips the eggpool-owned lines. See [docs/deployment.md](docs/deployment.md) for the full design.
 
-### Backup and uninstall
+### Backup and restore
 
-EggPool ships lifecycle commands that mirror the install flow:
+EggPool creates **automatic daily backups** by default. The in-process `automatic_backup` supervised task produces restore-compatible `.zip` archives every 24 hours (after an initial 5-minute startup delay) and retains the last 14. Archives are stored under `$XDG_BACKUP_HOME/eggpool` or `~/backups/eggpool`.
+
+```toml
+# Optional — automatic backups are enabled by default
+[backup]
+enabled = true
+interval_s = 86400
+retain_count = 14
+startup_delay_s = 300
+# directory = "/path/to/backups"
+include_env = true
+```
+
+Manual backup and restore are also available:
 
 ```bash
-# Backup config + .env + database to ~/backups/eggpool/
+# Manual backup config + .env + database
 eggpool backup
 
 # Restore from a specific archive (or omit the path for an interactive menu)
@@ -163,7 +176,7 @@ eggpool uninstall --yes
 eggpool uninstall --yes --deploy-artifacts
 ```
 
-The uninstall command removes the binary, active config, `.env`, database, and `eggpool` shell-rc entries. Pass `--deploy-artifacts` to also remove the systemd unit, logrotate config, watchdog + backup cron blocks, and the personal backup script. Existing backups under `~/backups/eggpool/` are always left in place. See [docs/backup-restore.md](docs/backup-restore.md) for the full backup/restore workflow.
+`eggpool deploy backup-cron` is optional and mainly for operators who prefer external scheduling or want backups even when the server process is not running. See [docs/backup-restore.md](docs/backup-restore.md) for the full backup/restore workflow.
 
 ## CLI Commands
 
@@ -200,7 +213,7 @@ The uninstall command removes the binary, active config, `.env`, database, and `
 | `eggpool init-config` | Write bundled config.example.toml to current directory or TARGET |
 | `eggpool deploy systemd` | Print systemd unit; `--install` writes it (personal by default; `--production` for the dedicated-system layout; `--as-root` for a root-owned personal unit) |
 | `eggpool deploy cron` | Print / install / uninstall the **watchdog** crontab (`@reboot` + `*/N * * * *` `ensure-running`). `--interval N` (1-59, default 5) |
-| `eggpool deploy backup-cron` | Print / install / uninstall the daily backup cron (personal user cron or production `/etc/cron.d/`) |
+| `eggpool deploy backup-cron` | Print / install / uninstall the daily backup cron. Optional — in-process automatic backup runs by default |
 | `eggpool deploy logrotate` | Print / install the logrotate config (validated via `logrotate -d`) |
 | `eggpool deploy all` | Print / install systemd + logrotate + watchdog cron (backup-cron is separate) |
 | `eggpool backup` | Create a timestamped `.zip` backup (default `~/backups/eggpool/`) |
@@ -291,7 +304,7 @@ The dashboard includes:
 - Overview with request counts, error rates, costs, token usage, and a System Health row surfacing pending-request and reservation leaks
 - Reliability page (`/reliability`) with attempt success/retry breakdown, `retry_category` distribution, pending health, and operational events
 - Routing page (`/routing`) with per-`(model, provider)` decision aggregates, account selection counts, and exclusion taxonomy (suppressive vs advisory)
-- Runtime page (`/runtime`) with process topology, memory, background task status, database health, and in-flight request counts
+- Runtime page (`/runtime`) with RSS memory, open file descriptors, active thread count, OS load average, dispatch overhead (last 100 upstream attempts), background task status, database health, and in-flight request counts. Process count surfaces only when anomalous
 - Traces page (`/traces`) with auth-gated recent request metadata (no error_detail, no client_ip)
 - Account and model breakdowns with filtering, exactness columns, cache/reasoning ratios, and cost-per-1k-tokens
 - Latency metrics including time-to-first-token (TTFT) and connect/read/coordinator-overhead phase breakdown
@@ -317,8 +330,8 @@ appropriate cache headers.
 JSON stats endpoints are available under `/api/stats/*`, including summary,
 accounts, models, timeseries, errors, latency, pings, bandwidth, attempts,
 retries, routing, routing-selections, routing-exclusions, operational,
-pending-health, runtime, recent-requests, recent/{request_id}, and
-`/api/events`. The recent-requests, recent/{request_id}, pending-health,
+pending-health, runtime, recent-requests, recent/{request_id},
+`/api/stats/update`, and `/api/events`. The recent-requests, recent/{request_id}, pending-health,
 and runtime endpoints are always auth-gated (even when the dashboard is
 public) because they expose per-request metadata (model, prompt volume,
 error class), operational state (pending reservations, reserved cost),
@@ -362,17 +375,40 @@ dashboard's public/auth setting.
   no error_detail). Auth-gated.
 - **Cost/cache/reasoning exactness** (extended fields on `/api/stats/accounts`
   and `/api/stats/models`): per-account and per-model `exact_count`,
-  `partial_count`, `derived_count`, `estimated_count`, `cache_read_ratio`,
-  `cache_write_ratio`, `reasoning_output_ratio`, `estimated_cost_fraction`,
-  `avg_cost_per_request`, `avg_cost_per_1k_tokens`. Lets you see which
-  accounts/models/providers report exact usage versus partially-priced
-  versus locally estimated cost.
+  `partial_count`, `derived_count`, `estimated_count`, `provider_reported_count`,
+  `cache_read_ratio`, `cache_write_ratio`, `reasoning_output_ratio`,
+  `estimated_cost_fraction`, `avg_cost_per_request`, `avg_cost_per_1k_tokens`.
+  Lets you see which accounts/models/providers report exact usage versus
+  partially-priced versus locally estimated cost versus upstream-billed cost.
+- **Provider-reported cost precedence**: when an upstream provider includes a
+  billing field in its response (`usage.cost_microdollars`, `usage.cost_usd`,
+  `usage.billing.cost_usd`, OpenCode Go's `usage.cost`/`usage.total_cost`),
+  that value is recorded on the `requests` row in `provider_cost_microdollars`
+  (with `provider_cost_source` recording the field path) and the request
+  exactness is labelled `provider_reported`. Provider-reported cost is the
+  most-trusted tier in the cost precedence ladder — it precedes locally
+  derived rates (per-token pricing) and reservation-derived estimates
+  (`estimated`), and is never overridden by them. Locally calculated values
+  are still stored in `local_cost_microdollars` and `local_cost_exactness`
+  for diagnostics comparison.
 - **Pricing provenance** (`/api/stats/pricing-provenance`): per-`(model,
   provider)` snapshot breakdown of `source_detail` (operator override vs.
   upstream metadata vs. OpenRouter catalog vs. curated alias) and
   `source_confidence` (exact external ID vs. curated alias vs. unknown).
   Used by the dashboard to render the cost-exactness badge and the
   high-spend estimated warning.
+- **Update checker** (`/api/stats/update`): the server checks PyPI at startup
+  and approximately every 24 hours for newer eggpool releases. When an update
+  is found, the dashboard footer shows a non-intrusive indicator with the
+  current and latest versions and a copyable `eggpool update` command. The
+  JSON snapshot is also available at `GET /api/stats/update`.
+- **Network diagnostics** (`/api/network/diagnostics`): sanitized outbound
+  client lifecycle (build count, request count, error count, per-scope builds,
+  per-host requests/errors) and DNS cache behavior (hits, misses, negative
+  hits, stale hits, evictions, per-host breakdown, resolution errors,
+  per-entry metadata with state/TTL/staleness). Always auth-gated. Also
+  displayed on the `/runtime` dashboard page and in `eggpool runtime-status`
+  output. No API keys, auth headers, request bodies, or full URLs are exposed.
 
 ## Configuration
 
@@ -392,6 +428,7 @@ See `config.example.toml` for all available options.
 - `[security]` — Allowed hosts, CORS, header redaction
 - `[providers.*]` — Provider configurations with accounts and `routing_priority`
 - `[proxies.*]` — Named outbound proxy definitions (pproxy URI syntax)
+- `[network]` — Outbound client transport tuning and `[network.dns_cache]` for in-memory DNS caching (enabled by default; reduces connection latency for repeated upstream requests)
 - `[model_overrides.*]` — Per-model protocol or path overrides
 
 ### Provider Configuration
@@ -523,6 +560,29 @@ Merge the generated provider definition into your OpenCode configuration. OpenCo
 
 Model limit changes require a service restart.
 
+### Low-wear metrics buffering
+
+EggPool buffers lossy analytics writes (timeseries, bandwidth, token/cost aggregates) in memory and flushes them periodically to reduce microSD wear. Correctness-critical state (requests, reservations, routing) is never buffered.
+
+Three write modes are available:
+
+- **`immediate`** (default for debugging): existing direct-write behavior.
+- **`balanced`** (default): buffers analytics with 30s flush intervals.
+- **`low_wear`**: 120s flush intervals, coarser 300s buckets, 5% trace sampling, aggregate-only mode — designed for microSD / Raspberry Pi.
+
+Example low-wear configuration:
+
+```toml
+[metrics]
+write_mode = "low_wear"
+flush_interval_s = 120
+timeseries_bucket_s = 300
+trace_sample_rate = 0.05
+aggregate_only = true
+```
+
+Buffered analytics may lose at most `flush_interval_s` seconds of data after abrupt power loss. For sustained multi-session use on flash media, a high-endurance microSD or USB SSD is recommended.
+
 ## Development
 
 ```bash
@@ -556,11 +616,19 @@ src/eggpool/
 ├── __init__.py          # Package version
 ├── __main__.py          # python -m eggpool
 ├── app.py               # FastAPI application factory
-├── cli.py               # Click CLI commands
+├── cli.py               # CLI bootstrap entry point (tiny, dispatches fast-path then Click)
+├── cli_full.py          # Click CLI commands (heavy imports)
+├── fastcli.py           # Fast-path CLI (stdlib-only, croncheck/ensure-running)
 ├── auth.py              # Local API key authentication
 ├── constants.py         # Project-wide constants
 ├── errors.py            # Exception hierarchy
 ├── logging.py           # Structured logging setup
+├── runtime.py           # Process management (restart, stop, PID lifecycle)
+├── runtime_metrics.py   # Runtime/ops metrics: process, memory, DB, background tasks
+├── runtime_dispatch.py  # Bounded rolling-window dispatch overhead recorder
+├── runtime_paths.py     # PID file and log path resolution (stdlib-only)
+├── update_checker.py    # PyPI update checker (background + CLI)
+├── cost_recompute.py    # Cost recompute CLI command
 ├── onboard.py           # Interactive onboarding setup
 ├── models/
 │   ├── config.py        # Pydantic config models
@@ -597,6 +665,7 @@ src/eggpool/
 │   └── static/          # CSS, JavaScript, and favicon
 ├── integrations/        # External tool config generation (OpenCode, Claude Code)
 ├── security/            # Header redaction and security utilities
+├── lifecycle/           # Backup and uninstall orchestration
 ├── deploy/              # Bundled systemd/logrotate/cron snippets for CLI output
 └── _share/              # Bundled config examples and assets for pipx installs
 
@@ -623,7 +692,8 @@ docs/                    # Documentation
 ├── filesystem-layout.md # Filesystem layout reference
 ├── model-limits.md      # Model context limit configuration
 ├── providers.md         # Provider catalog and configuration guide
-└── proxy.md             # Per-account outbound proxy (pproxy)
+├── proxy.md             # Per-account outbound proxy (pproxy)
+└── network-diagnostics.md  # DNS cache and outbound client diagnostics
 
 config-examples/         # Editor-specific config snippets
 ├── opencode.jsonc       # OpenCode provider config (JSONC)
@@ -814,14 +884,32 @@ The worker is named `eggpool` in `ps` / `top` (via Granian's
 
 `eggpool runtime-status` calls the local `/api/stats/runtime` endpoint and
 prints a compact terminal summary of process topology, memory usage,
-background task health, database file/WAL sizes, and in-flight request
-counts. Use it to diagnose daemon/systemd/cron deployments without
+background task health, database file/WAL sizes, in-flight request
+counts, OS load average, and the recent upstream dispatch overhead
+distribution. Use it to diagnose daemon/systemd/cron deployments without
 inspecting logs.
 
 The `/api/stats/runtime` endpoint and the `/runtime` dashboard page expose
 the same data. Both are always auth-gated regardless of `dashboard.public`
 because they reveal operational details (PID, memory, DB path, process
 count). The endpoint is best-effort: probes that fail on a given platform
-(e.g., `/proc` on macOS) return `null` for the affected field.
+(e.g., `/proc` on macOS, `os.getloadavg` on Windows) return `null` for
+the affected field.
+
+#### Dispatch overhead
+
+The Runtime page surfaces a `Dispatch overhead` card sourced from an
+in-memory rolling window of the last 100 upstream attempts. It measures
+EggPool-local pre-dispatch work (request validation, routing selection,
+persistence, reservation accounting) and excludes provider connect/TTFT,
+streaming, body read, and finalization. Recording happens immediately
+before `client.send(...)` in both the streaming and non-streaming
+dispatch paths using `time.perf_counter_ns()`, so wall-clock skew never
+contaminates the measurement. The recorder is bounded (`deque(maxlen=100)`)
+and thread-safe; it stores only integer nanoseconds — no request body,
+auth header, account name, or client IP ever lands in the sample buffer.
+The same metric is available via `/api/stats/runtime` under
+`dispatch_overhead` (`window_size`, `sample_count`, `avg_ms`, `min_ms`,
+`max_ms`, `p50_ms`, `p95_ms`).
 
 See [CHANGELOG](CHANGELOG.md) for release history.

{eggpool-0.3.1 → eggpool-0.3.3}/README.md

@@ -101,12 +101,25 @@ eggpool deploy cron --install
 
 This writes a `@reboot` + `*/5 * * * *` `eggpool ensure-running` block to the invoking user's crontab (or `SUDO_USER`'s crontab under sudo). Use `--interval N` to change the poll cadence. The block is bracketed by `# BEGIN EggPool watchdog` / `# END EggPool watchdog` markers so uninstall only strips the eggpool-owned lines. See [docs/deployment.md](docs/deployment.md) for the full design.
 
-### Backup and uninstall
+### Backup and restore
 
-EggPool ships lifecycle commands that mirror the install flow:
+EggPool creates **automatic daily backups** by default. The in-process `automatic_backup` supervised task produces restore-compatible `.zip` archives every 24 hours (after an initial 5-minute startup delay) and retains the last 14. Archives are stored under `$XDG_BACKUP_HOME/eggpool` or `~/backups/eggpool`.
+
+```toml
+# Optional — automatic backups are enabled by default
+[backup]
+enabled = true
+interval_s = 86400
+retain_count = 14
+startup_delay_s = 300
+# directory = "/path/to/backups"
+include_env = true
+```
+
+Manual backup and restore are also available:
 
 ```bash
-# Backup config + .env + database to ~/backups/eggpool/
+# Manual backup config + .env + database
 eggpool backup
 
 # Restore from a specific archive (or omit the path for an interactive menu)
@@ -119,7 +132,7 @@ eggpool uninstall --yes
 eggpool uninstall --yes --deploy-artifacts
 ```
 
-The uninstall command removes the binary, active config, `.env`, database, and `eggpool` shell-rc entries. Pass `--deploy-artifacts` to also remove the systemd unit, logrotate config, watchdog + backup cron blocks, and the personal backup script. Existing backups under `~/backups/eggpool/` are always left in place. See [docs/backup-restore.md](docs/backup-restore.md) for the full backup/restore workflow.
+`eggpool deploy backup-cron` is optional and mainly for operators who prefer external scheduling or want backups even when the server process is not running. See [docs/backup-restore.md](docs/backup-restore.md) for the full backup/restore workflow.
 
 ## CLI Commands
 
@@ -156,7 +169,7 @@ The uninstall command removes the binary, active config, `.env`, database, and `
 | `eggpool init-config` | Write bundled config.example.toml to current directory or TARGET |
 | `eggpool deploy systemd` | Print systemd unit; `--install` writes it (personal by default; `--production` for the dedicated-system layout; `--as-root` for a root-owned personal unit) |
 | `eggpool deploy cron` | Print / install / uninstall the **watchdog** crontab (`@reboot` + `*/N * * * *` `ensure-running`). `--interval N` (1-59, default 5) |
-| `eggpool deploy backup-cron` | Print / install / uninstall the daily backup cron (personal user cron or production `/etc/cron.d/`) |
+| `eggpool deploy backup-cron` | Print / install / uninstall the daily backup cron. Optional — in-process automatic backup runs by default |
 | `eggpool deploy logrotate` | Print / install the logrotate config (validated via `logrotate -d`) |
 | `eggpool deploy all` | Print / install systemd + logrotate + watchdog cron (backup-cron is separate) |
 | `eggpool backup` | Create a timestamped `.zip` backup (default `~/backups/eggpool/`) |
@@ -247,7 +260,7 @@ The dashboard includes:
 - Overview with request counts, error rates, costs, token usage, and a System Health row surfacing pending-request and reservation leaks
 - Reliability page (`/reliability`) with attempt success/retry breakdown, `retry_category` distribution, pending health, and operational events
 - Routing page (`/routing`) with per-`(model, provider)` decision aggregates, account selection counts, and exclusion taxonomy (suppressive vs advisory)
-- Runtime page (`/runtime`) with process topology, memory, background task status, database health, and in-flight request counts
+- Runtime page (`/runtime`) with RSS memory, open file descriptors, active thread count, OS load average, dispatch overhead (last 100 upstream attempts), background task status, database health, and in-flight request counts. Process count surfaces only when anomalous
 - Traces page (`/traces`) with auth-gated recent request metadata (no error_detail, no client_ip)
 - Account and model breakdowns with filtering, exactness columns, cache/reasoning ratios, and cost-per-1k-tokens
 - Latency metrics including time-to-first-token (TTFT) and connect/read/coordinator-overhead phase breakdown
@@ -273,8 +286,8 @@ appropriate cache headers.
 JSON stats endpoints are available under `/api/stats/*`, including summary,
 accounts, models, timeseries, errors, latency, pings, bandwidth, attempts,
 retries, routing, routing-selections, routing-exclusions, operational,
-pending-health, runtime, recent-requests, recent/{request_id}, and
-`/api/events`. The recent-requests, recent/{request_id}, pending-health,
+pending-health, runtime, recent-requests, recent/{request_id},
+`/api/stats/update`, and `/api/events`. The recent-requests, recent/{request_id}, pending-health,
 and runtime endpoints are always auth-gated (even when the dashboard is
 public) because they expose per-request metadata (model, prompt volume,
 error class), operational state (pending reservations, reserved cost),
@@ -318,17 +331,40 @@ dashboard's public/auth setting.
   no error_detail). Auth-gated.
 - **Cost/cache/reasoning exactness** (extended fields on `/api/stats/accounts`
   and `/api/stats/models`): per-account and per-model `exact_count`,
-  `partial_count`, `derived_count`, `estimated_count`, `cache_read_ratio`,
-  `cache_write_ratio`, `reasoning_output_ratio`, `estimated_cost_fraction`,
-  `avg_cost_per_request`, `avg_cost_per_1k_tokens`. Lets you see which
-  accounts/models/providers report exact usage versus partially-priced
-  versus locally estimated cost.
+  `partial_count`, `derived_count`, `estimated_count`, `provider_reported_count`,
+  `cache_read_ratio`, `cache_write_ratio`, `reasoning_output_ratio`,
+  `estimated_cost_fraction`, `avg_cost_per_request`, `avg_cost_per_1k_tokens`.
+  Lets you see which accounts/models/providers report exact usage versus
+  partially-priced versus locally estimated cost versus upstream-billed cost.
+- **Provider-reported cost precedence**: when an upstream provider includes a
+  billing field in its response (`usage.cost_microdollars`, `usage.cost_usd`,
+  `usage.billing.cost_usd`, OpenCode Go's `usage.cost`/`usage.total_cost`),
+  that value is recorded on the `requests` row in `provider_cost_microdollars`
+  (with `provider_cost_source` recording the field path) and the request
+  exactness is labelled `provider_reported`. Provider-reported cost is the
+  most-trusted tier in the cost precedence ladder — it precedes locally
+  derived rates (per-token pricing) and reservation-derived estimates
+  (`estimated`), and is never overridden by them. Locally calculated values
+  are still stored in `local_cost_microdollars` and `local_cost_exactness`
+  for diagnostics comparison.
 - **Pricing provenance** (`/api/stats/pricing-provenance`): per-`(model,
   provider)` snapshot breakdown of `source_detail` (operator override vs.
   upstream metadata vs. OpenRouter catalog vs. curated alias) and
   `source_confidence` (exact external ID vs. curated alias vs. unknown).
   Used by the dashboard to render the cost-exactness badge and the
   high-spend estimated warning.
+- **Update checker** (`/api/stats/update`): the server checks PyPI at startup
+  and approximately every 24 hours for newer eggpool releases. When an update
+  is found, the dashboard footer shows a non-intrusive indicator with the
+  current and latest versions and a copyable `eggpool update` command. The
+  JSON snapshot is also available at `GET /api/stats/update`.
+- **Network diagnostics** (`/api/network/diagnostics`): sanitized outbound
+  client lifecycle (build count, request count, error count, per-scope builds,
+  per-host requests/errors) and DNS cache behavior (hits, misses, negative
+  hits, stale hits, evictions, per-host breakdown, resolution errors,
+  per-entry metadata with state/TTL/staleness). Always auth-gated. Also
+  displayed on the `/runtime` dashboard page and in `eggpool runtime-status`
+  output. No API keys, auth headers, request bodies, or full URLs are exposed.
 
 ## Configuration
 
@@ -348,6 +384,7 @@ See `config.example.toml` for all available options.
 - `[security]` — Allowed hosts, CORS, header redaction
 - `[providers.*]` — Provider configurations with accounts and `routing_priority`
 - `[proxies.*]` — Named outbound proxy definitions (pproxy URI syntax)
+- `[network]` — Outbound client transport tuning and `[network.dns_cache]` for in-memory DNS caching (enabled by default; reduces connection latency for repeated upstream requests)
 - `[model_overrides.*]` — Per-model protocol or path overrides
 
 ### Provider Configuration
@@ -479,6 +516,29 @@ Merge the generated provider definition into your OpenCode configuration. OpenCo
 
 Model limit changes require a service restart.
 
+### Low-wear metrics buffering
+
+EggPool buffers lossy analytics writes (timeseries, bandwidth, token/cost aggregates) in memory and flushes them periodically to reduce microSD wear. Correctness-critical state (requests, reservations, routing) is never buffered.
+
+Three write modes are available:
+
+- **`immediate`** (default for debugging): existing direct-write behavior.
+- **`balanced`** (default): buffers analytics with 30s flush intervals.
+- **`low_wear`**: 120s flush intervals, coarser 300s buckets, 5% trace sampling, aggregate-only mode — designed for microSD / Raspberry Pi.
+
+Example low-wear configuration:
+
+```toml
+[metrics]
+write_mode = "low_wear"
+flush_interval_s = 120
+timeseries_bucket_s = 300
+trace_sample_rate = 0.05
+aggregate_only = true
+```
+
+Buffered analytics may lose at most `flush_interval_s` seconds of data after abrupt power loss. For sustained multi-session use on flash media, a high-endurance microSD or USB SSD is recommended.
+
 ## Development
 
 ```bash
@@ -512,11 +572,19 @@ src/eggpool/
 ├── __init__.py          # Package version
 ├── __main__.py          # python -m eggpool
 ├── app.py               # FastAPI application factory
-├── cli.py               # Click CLI commands
+├── cli.py               # CLI bootstrap entry point (tiny, dispatches fast-path then Click)
+├── cli_full.py          # Click CLI commands (heavy imports)
+├── fastcli.py           # Fast-path CLI (stdlib-only, croncheck/ensure-running)
 ├── auth.py              # Local API key authentication
 ├── constants.py         # Project-wide constants
 ├── errors.py            # Exception hierarchy
 ├── logging.py           # Structured logging setup
+├── runtime.py           # Process management (restart, stop, PID lifecycle)
+├── runtime_metrics.py   # Runtime/ops metrics: process, memory, DB, background tasks
+├── runtime_dispatch.py  # Bounded rolling-window dispatch overhead recorder
+├── runtime_paths.py     # PID file and log path resolution (stdlib-only)
+├── update_checker.py    # PyPI update checker (background + CLI)
+├── cost_recompute.py    # Cost recompute CLI command
 ├── onboard.py           # Interactive onboarding setup
 ├── models/
 │   ├── config.py        # Pydantic config models
@@ -553,6 +621,7 @@ src/eggpool/
 │   └── static/          # CSS, JavaScript, and favicon
 ├── integrations/        # External tool config generation (OpenCode, Claude Code)
 ├── security/            # Header redaction and security utilities
+├── lifecycle/           # Backup and uninstall orchestration
 ├── deploy/              # Bundled systemd/logrotate/cron snippets for CLI output
 └── _share/              # Bundled config examples and assets for pipx installs
 
@@ -579,7 +648,8 @@ docs/                    # Documentation
 ├── filesystem-layout.md # Filesystem layout reference
 ├── model-limits.md      # Model context limit configuration
 ├── providers.md         # Provider catalog and configuration guide
-└── proxy.md             # Per-account outbound proxy (pproxy)
+├── proxy.md             # Per-account outbound proxy (pproxy)
+└── network-diagnostics.md  # DNS cache and outbound client diagnostics
 
 config-examples/         # Editor-specific config snippets
 ├── opencode.jsonc       # OpenCode provider config (JSONC)
@@ -770,14 +840,32 @@ The worker is named `eggpool` in `ps` / `top` (via Granian's
 
 `eggpool runtime-status` calls the local `/api/stats/runtime` endpoint and
 prints a compact terminal summary of process topology, memory usage,
-background task health, database file/WAL sizes, and in-flight request
-counts. Use it to diagnose daemon/systemd/cron deployments without
+background task health, database file/WAL sizes, in-flight request
+counts, OS load average, and the recent upstream dispatch overhead
+distribution. Use it to diagnose daemon/systemd/cron deployments without
 inspecting logs.
 
 The `/api/stats/runtime` endpoint and the `/runtime` dashboard page expose
 the same data. Both are always auth-gated regardless of `dashboard.public`
 because they reveal operational details (PID, memory, DB path, process
 count). The endpoint is best-effort: probes that fail on a given platform
-(e.g., `/proc` on macOS) return `null` for the affected field.
+(e.g., `/proc` on macOS, `os.getloadavg` on Windows) return `null` for
+the affected field.
+
+#### Dispatch overhead
+
+The Runtime page surfaces a `Dispatch overhead` card sourced from an
+in-memory rolling window of the last 100 upstream attempts. It measures
+EggPool-local pre-dispatch work (request validation, routing selection,
+persistence, reservation accounting) and excludes provider connect/TTFT,
+streaming, body read, and finalization. Recording happens immediately
+before `client.send(...)` in both the streaming and non-streaming
+dispatch paths using `time.perf_counter_ns()`, so wall-clock skew never
+contaminates the measurement. The recorder is bounded (`deque(maxlen=100)`)
+and thread-safe; it stores only integer nanoseconds — no request body,
+auth header, account name, or client IP ever lands in the sample buffer.
+The same metric is available via `/api/stats/runtime` under
+`dispatch_overhead` (`window_size`, `sample_count`, `avg_ms`, `min_ms`,
+`max_ms`, `p50_ms`, `p95_ms`).
 
 See [CHANGELOG](CHANGELOG.md) for release history.