rigscope 0.1.0 → 0.1.2

package/docs/monitoring.md

@@ -0,0 +1,112 @@
+# Monitoring and alerts
+
+This guide explains what Rigscope collects, how long it keeps data, and how
+access and alerts work. For setup, see [Installation](installation.md).
+
+## Collection and sharing
+
+Each collector samples its machine once and sends readings independently to up
+to eight hubs. Collection continues when no browser is open. Each hub owns its
+users, permissions, history and alerts.
+
+Managed enrollment shares CPU, memory, swap, GPU, disk and network readings,
+plus on-demand process details. Network addresses and full process arguments
+start disabled. Sharing is set separately for each hub; see the
+[configuration reference](configuration.md#sharing-settings).
+
+Missing or inaccessible readings stay unavailable. GPU support depends on the
+driver and readable counters; see [Hardware](installation.md#hardware) and
+[Sensor readings](bottlenecks.md).
+
+## History and delivery
+
+The hub acknowledges samples after committing them to SQLite. A collector keeps
+unacknowledged samples in a persistent queue for each hub and retries after a
+disconnect. The hub removes duplicates using the collector, run and sequence IDs.
+Replayed samples fill History without replacing current live readings or opening
+alerts for past threshold crossings.
+
+Default retention:
+
+| Resolution | Kept for |
+| --- | --- |
+| Raw samples | 48 hours |
+| One-minute summaries | 30 days |
+| One-hour summaries | 400 days |
+
+Summaries retain minimum and maximum values. History queries limit the number
+of points, series, scanned rows and time spent. The UI marks partial results.
+
+Hover over a History or Overview recent-usage chart to inspect a recorded time
+and its bucket averages. The time marker follows across charts in the same view;
+per-core heatmaps show the bucket peak at that time. Missing readings stay
+unavailable. With a chart focused, use Left/Right arrows to move between readings,
+Home/End to jump to the first/last reading, and Escape to dismiss the marker.
+
+### Storage limits
+
+Each collector queue keeps up to ten minutes or 64 MiB of queued JSON per hub,
+with a 256 MiB total JSON limit. Physical SQLite files, including WAL files,
+have separate limits of 80 MiB per hub and 640 MiB across eight hubs.
+
+The hub's default database limit is 2 GiB, including its WAL file and all tables.
+Retention protects the newest hour of telemetry. If old-data cleanup cannot
+free enough space, the hub stops acknowledging new samples and collectors retry.
+Existing oversized databases report pressure; startup does not run a destructive
+cleanup.
+
+These are application limits, not filesystem quotas. A transaction can briefly
+grow the WAL before checkpointing. Blocked checkpoints stop further ingestion
+instead of allowing continued growth. Keep disk space for backups and sidecar
+files. See [Storage pressure recovery](operations.md#storage-budgets-and-pressure-recovery).
+
+## Live readings and process details
+
+A receipt becomes stale after `max(15 seconds, 3 × expected delivery interval)`.
+If the cadence is unknown, Rigscope uses 15 seconds and discloses that fallback.
+Connection status, unavailable readings and clock skew are reported separately.
+Rigscope does not synchronize the browser and hub clocks.
+
+Process details are scanned while viewers request them. Viewers share one
+collector scan. Pausing, hiding or closing the view releases the extra work;
+expired viewer requests and ended sessions do the same. Process details are
+never persisted.
+
+Process CPU uses 100% per logical core. Machine CPU is normalized across all
+logical cores. See [Exports](exports.md) for CSV and JSON behavior.
+
+## Users and administration
+
+Use **Administration** to manage users, machine access, names, tags, collector
+revocation and enrollment grants. Administration requires an administrator role
+and the appropriate machine access. Access to all machines alone does not make
+a user an administrator.
+
+User changes revoke existing browser sessions. Collector credential changes
+close connections that no longer have access. User records and collector
+credentials persist in SQLite. Browser sessions stay in memory, expire after
+12 hours and end when the hub restarts.
+
+## Alerts and webhooks
+
+Alerts support above/below thresholds, durations, unavailable readings, missing
+receipts, maintenance and recovery. Value, unavailable and stale-receipt rules
+apply to the machines tracked by that hub. Replayed samples do not open past
+incidents.
+
+In **Alerts**, configure one HTTPS webhook channel. Its URL, headers and HMAC
+secret are write-only. Loopback HTTP is allowed for controlled test sinks.
+Delivery may be repeated: receivers should deduplicate the `Idempotency-Key`.
+Rigscope makes up to six attempts, with a five-second deadline per request and
+bounded persistent queues.
+
+## Scale and limits
+
+There is no fixed machine-count limit. The default limit is 100 concurrent
+collector connections. [Scale validation](scale-validation.md) records synthetic
+workloads of up to 75 machines on one host; those results are not a capacity
+guarantee.
+
+Rigscope does not kill processes, change their priorities, control remote
+services, federate hubs or provide a clustered hub. See the
+[hardware notes](installation.md#hardware) for physical testing and sensor limits.

package/docs/operations.md

@@ -1,80 +1,107 @@
-# Production operations
+# Operations
 
-This guide covers the supported single-host, two-process deployment:
+For an npm installation, use the managed commands below. The
+[manifest archive workflow](#low-level-manifest-archive-workflow) is for operators
+building and deploying from source.
 
-- `dist/hub.cjs` — Node 26 custom server: listener, WebSocket collectors,
-  browser SSE, authentication and the production Next.js handler from `.next`.
-- `dist/collector.cjs` — outbound-only collector with durable per-destination
-  outboxes.
-- `dist/ops.cjs` — inspect, migrate, backup, restore, package, install,
-  activate, rollback and bounded synthetic measurement.
-
-There is no Next.js standalone output and no `next start`. The hub runs the
-custom server from the installed release directory; the production UI must be
-present in `.next`.
-
-All paths below are examples. Choose a prefix and state directory owned by the
-service user. Paths containing spaces are supported by the generated units.
+| Task | Section |
+| --- | --- |
+| Check, upgrade or roll back an npm installation | [Managed CLI operations](#managed-cli-operations) |
+| Reset a lost password | [Password recovery](#password-recovery) |
+| Change localhost/HTTPS access | [Access changes](#access-changes) |
+| Back up or restore a database | [Database commands](#database-commands) |
+| Plan what to back up | [Consistent backup set](#consistent-backup-set) |
+| Handle a full database or queue | [Storage pressure recovery](#storage-budgets-and-pressure-recovery) |
 
 ## Managed CLI operations
 
-For a per-user installation created by `rigscope install` (default state
-`~/.local/state/rigscope`), the CLI owns the runtime, component units and release pairing:
+These commands apply to an installation created by `rigscope install`. Its
+state directory defaults to `~/.local/state/rigscope`.
+
+### Check health
 
 ```sh
 rigscope status
-rigscope doctor [--json]
-rigscope upgrade [--from PACKAGE] [--timeout SECONDS]
-rigscope rollback [--backup PATH] [--timeout SECONDS]
-rigscope recover-password [--user owner] [--password-file PATH] [--enable]
+rigscope doctor
+```
+
+`status` shows services and the dashboard URL. `doctor` checks configuration,
+runtimes, database integrity, listeners, TLS destinations, disk space and sensor
+access. It follows enabled components: a hub-only installation does not need a
+collector. Checks are read-only and omit credentials. Use `doctor --json` for
+structured output.
+
+### Upgrade
+
+```sh
+rigscope upgrade
 ```
 
-`doctor` is read-only. It checks Node, private configuration and state
-permissions, the installed and previous runtimes, the component units, the database
-and schema, free space, hub port/Unix-socket state, role-appropriate listener
-health, configured destinations with TLS certificate validation, and procfs or
-RAPL readability for a collector. A hub without a collector is valid, and a
-collector without a local hub database is valid. Output contains counts and
-paths, never password hashes, collector tokens or environment values.
-
-`upgrade` stages the package without touching the active runtime, stops the
-running components, takes a consistent pre-upgrade hub database backup when one
-exists, and activates the new runtime. It restarts only components that were
-running before the upgrade and verifies them within `--timeout` seconds (default
-25). Verification follows the installation's enabled components:
-
-- the active package reports the staged version;
-- restarted component services are active;
-- a restarted hub answers `status: "ok"` with the expected version;
-- an existing hub database passes integrity and ledger checks and reaches the
-  staged runtime's supported schema;
-- a restarted collector produces a fresh sampling health marker.
-
-A failed start, migration or health wait restores the previous runtime and the
-paired pre-upgrade database when the old runtime cannot read the new schema. The
-replaced database is preserved first; credentials, collector identities and
-outboxes are not rewritten. `release-state.json` in the state directory records
-which runtime each backup belongs to. An inactive installation is staged and
-activated without starting; live health is then reported as skipped.
-
-`rollback` swaps to `runtime-previous` and validates it with the same bounded
-health and database checks. If the current schema is compatible it keeps the
-database unchanged, including post-upgrade data. If the previous runtime cannot
-read it, rollback restores the paired pre-upgrade backup, refuses when no
-compatible backup is recorded, and prints where the replaced newer database was
-preserved. For installations created before release state existed, pass an
-explicit compatible copy with `--backup PATH`. Rollback never silently deletes
-the newer database and never runs old code against a newer schema.
-
-`recover-password` is offline recovery for the durable managed user store. It
-requires the hub service to be stopped, takes the hub database lease and
-refuses a live or ambiguous lock. Enter the password at the private prompt, or
-pass a private `0600` file with `--password-file`. It updates only that managed
-user's hash in the database, clears `disabled` only with `--enable`, writes a
-bounded `user.password-recovery` audit row and bumps a durable session-epoch
-marker. It never edits `config.json` or `HUB_BROWSER_USERS`, so bootstrap
-password changes do not overwrite managed users. Restarting the hub invalidates
-all in-memory browser sessions; sign in again with the new password.
+Use `--from PACKAGE` for an exact npm version or local archive. The command
+stages the package, stops running components, takes a consistent hub database
+backup and activates the new runtime. It restarts only the components that were
+running, then checks:
+
+- the installed and running versions match the new package;
+- restarted services are active;
+- hub health is OK and its database passes integrity/schema checks;
+- a restarted collector produces a fresh sampling marker.
+
+The health timeout is 25 seconds; change it with `--timeout SECONDS`. A failed
+start, migration or health check restores the previous runtime. It also restores
+the paired database backup if the old runtime cannot read the new schema,
+preserving the replaced database first.
+
+Credentials, collector identities and pending samples are retained.
+`release-state.json` records which backup belongs to each runtime. Upgrading a
+stopped installation leaves it stopped and reports live checks as skipped.
+
+### Roll back
+
+```sh
+rigscope rollback
+```
+
+Rollback uses `runtime-previous` and runs the same health checks. If that runtime
+can read the current database, post-upgrade data stays intact. Otherwise it
+restores the paired pre-upgrade backup and saves the replaced database.
+
+A missing compatible backup stops rollback. Older installations without a
+recorded backup can supply a compatible copy with `--backup PATH`. Never point
+old code at a newer database schema.
+
+## Password recovery
+
+Stop the installation, enter a new password at the private prompt, then restart:
+
+```sh
+rigscope stop
+rigscope recover-password --user owner
+rigscope start
+```
+
+Recovery requires the hub to be stopped and refuses a live or uncertain database
+lock. For unattended recovery, use `--password-file` with an owned `0600` file.
+It updates the existing user's stored hash. `--enable` also clears that user's
+disabled flag; recovery never creates a user or grants administrator access.
+
+The operation records an audit event and invalidates sessions. It keeps
+configuration, history and collector credentials. Sign in again after restart;
+editing bootstrap hashes in `config.json` or `HUB_BROWSER_USERS` does not reset a
+managed account.
+
+## Access changes
+
+Follow the commands in [Remote access](installation.md#access-from-another-computer).
+`configure` requires a stopped hub and, when changing its local collector's
+destination, a stopped collector.
+
+It validates the HTTPS origin and private socket, backs up the old settings, and
+updates the matching local collector destination. It moves that destination's
+whole queue directory, including SQLite WAL files. Credentials and other hubs
+stay unchanged. An existing target queue stops the conversion to avoid merging
+or discarding pending records. Accounts and the hub history database are retained.
+Rollback targets must support the saved configuration and database schema.
 
 ## Low-level manifest archive workflow
 
@@ -84,6 +111,14 @@ used by `rigscope install` and `rigscope upgrade`. For ordinary installations,
 follow [installation](installation.md) and the managed commands above. The npm
 package excludes source helper scripts and does not require users to build it.
 
+The built deployment runs `dist/hub.cjs` for the dashboard and collector
+connections, and `dist/collector.cjs` for collection. `dist/ops.cjs` provides the
+database and archive tools below. The hub serves the production Next.js output
+from `.next`; it does not use `next start` or standalone output.
+
+Choose a prefix and state directory owned by the service user. All paths below
+are examples; generated units support paths containing spaces.
+
 ### Build and package
 
 ```sh
@@ -406,7 +441,7 @@ a hub back below the schema its database already uses; `rollback` enforces that
 boundary, and collector rollback keeps its private state and outboxes. Process
 detail remains request-scoped and is never persisted or included in the backup
 set. Separate machines and hubs are upgraded independently. System installations
-use the component controls described in [installation](installation.md#system-component-lifecycle).
+use the component controls in [System installation](system-installation.md#system-component-lifecycle).
 
 On upgrade from the previous collector endpoint, saved `/api/collector/v1/connect`
 URLs normalize to `/api/collector/connect` without changing identity, credentials,

package/docs/releasing.md

@@ -1,18 +1,21 @@
 # Releasing Rigscope
 
-The first public version is `0.1.0`. Source and issue links target
-`https://github.com/Triet1402/rigscope`. Create that repository and push the
-release source before npm publication. The package includes MIT licensing,
+The first public version, `0.1.0`, is published on
+[npm](https://www.npmjs.com/package/rigscope). Source and issues are on
+[GitHub](https://github.com/Triet1402/rigscope). The package includes MIT licensing,
 dependency notices, the security policy and [release notes](../CHANGELOG.md).
 
+The commands below use `0.1.0` as an example. For a new release, substitute its
+new version everywhere; do not attempt to republish `0.1.0`. Documentation edits
+can be committed and pushed to GitHub independently. The published archive and
+its bundled documentation stay unchanged; ship updated package documentation
+with the next version. Keep each release tag on the revision used to build it.
+
 ## Recommended release order
 
-1. Finish the version, public docs, release notes and repository metadata. Commit
-   those changes. Keep the installation examples in the package ready for the
-   version being published.
+1. Update the version, docs, release notes and repository metadata. Commit them.
 2. Push the source to GitHub and run the manual CI workflow on that revision.
-   This repository deliberately has no push, pull-request or scheduled CI
-   triggers. A local result is not a GitHub Actions result.
+   CI runs manually; pushes and pull requests do not trigger it.
 3. Build one npm archive from the final revision. Retain it, its SHA-256 checksum
    and verification results. Test and publish that archive without rebuilding it.
 4. Run the publish dry run, check the npm account, and publish the verified archive.
@@ -111,9 +114,8 @@ npm publish ./out/release/rigscope-0.1.0.tgz --access public --tag latest \
   --registry=https://registry.npmjs.org --dry-run
 ```
 
-The dry run does not publish, reserve the name, or prove that registry
-authorization and two-factor authentication will succeed. The repository
-metadata does not establish npm package ownership.
+The dry run checks the package without publishing. It does not verify permission
+to publish or complete two-factor authentication.
 
 When releasing, this command actually publishes the verified archive publicly:
 
@@ -129,8 +131,8 @@ check. Uninstall that test installation afterward. Use the same isolation
 settings for all test commands so the check does not select a real installation.
 
 The public quick start uses `npx --yes rigscope@latest install --boot` and the
-installed `rigscope upgrade`. Archive commands remain available before npm
-publication and for explicit version selection. Users do not need a global npm
+installed `rigscope upgrade`. Archive commands remain available for downloaded
+releases and testing before publication. Users do not need a global npm
 installation of Rigscope.
 
 GitHub Actions trusted publishing can automate later releases. Configure it for
@@ -146,7 +148,7 @@ It is a separate format from the npm `.tgz`. The low-level installer validates
 size/hash and required artifacts before activation. Production dependencies must
 be installed in the staged release; activation never silently runs npm or service
 commands. See [operations](operations.md#low-level-manifest-archive-workflow) and
-[system installation](installation.md#system-installation).
+[system installation](system-installation.md).
 
 Configuration, database schema and wire-protocol revisions are internal
 compatibility checks, independent of the application package version.

package/docs/scale-validation.md

@@ -9,6 +9,22 @@ concurrency listed for each case. They do not prove that a larger fleet,
 different hardware, slower storage or a different browser workload will behave
 the same way.
 
+## Later capacity search
+
+The [November 4 capacity stress test](stress-2026-11-04/README.md) extends these
+measurements with an exponential/binary search and two-minute confirmations:
+
+- 500 collectors plus 64 live browser sessions passed at one sample and one
+  History poll per second per participant.
+- Ingestion without live viewers passed at 1,050 collectors; 1,125 exceeded the
+  experiment's 500 ms acknowledgment p95 target.
+- Full-fleet streaming passed with 529 representative machines, but 530 exceeded
+  the live feed's 1 MiB buffer guard and disconnected the viewer.
+
+That report, its measurement JSON and chart are included in the npm package.
+These results describe a bounded synthetic workload, not long-term capacity.
+The five-case suite below remains the earlier sustained measurement.
+
 ## What was missing
 
 The earlier `ops measure` command was a bounded synthetic burst capped at 50
@@ -86,12 +102,13 @@ phase boundary.
 
 ### Bounds
 
-- machines: 1–1,000,
+- machines: 1–8,192 (the total-record bound below still applies),
 - measured phase: up to 1 hour,
 - sample cadence: at least 100 ms (the protocol rate limit),
 - records per live batch: 1–32,
-- viewers: 1–50 authenticated sessions,
-- SSE clients: 1–64,
+- viewers: 1–200 authenticated sessions,
+- SSE attempts: 0–65; the hub still admits at most 64 (zero isolates ingestion,
+  and the 65th attempt measures admission rejection),
 - replay: at most 128 records per reconnect,
 - estimated live plus replay records: at most 500,000 per case,
 - database: 512 MiB main+WAL by default (`--database-budget-mb 0` disables the
@@ -201,6 +218,10 @@ payloads, storage and viewer counts.
 
 ## Reproduction
 
+For the later resource-guarded capacity search, including the distinction between
+collector throughput and full-fleet stream limits, see
+[the November 4 stress test](stress-2026-11-04/README.md).
+
 ```sh
 # full five-case suite (about five minutes)
 npx tsx scripts/scale-harness.ts --suite \
@@ -213,6 +234,11 @@ npx tsx scripts/scale-harness.ts --case fleet-75m \
 
 # retain the throwaway database and configuration for inspection
 npx tsx scripts/scale-harness.ts --case small-5m --keep
+
+# Keep stream parsing independent of collector sample generation
+npx tsx scripts/scale-harness.ts --case fleet-75m \
+  --machines 500 --viewers 64 --sse-clients 64 --sse-workers 4 \
+  --duration-ms 60000 --json-out /tmp/rigscope-scale-mixed.json
 ```
 
 The short `tests/scale-harness.test.ts` smoke runs the same child with two
@@ -224,6 +250,11 @@ test; it proves the harness paths work in CI.
 - The hub child and all load generators share one host. The hub event loop uses
   one thread; the load processes can run on other cores. This is
   not a distributed-load test and the generator competes with the hub.
+  `--sse-workers N` separates stream readers into up to eight processes. With
+  workers enabled, a one-second client warmup precedes the measured window;
+  without workers, collector generation and stream parsing share an event loop,
+  which can under-drive large mixed workloads. Report configuration records
+  the worker count and client warmup. The original five results used no workers.
 - History queries are synchronous SQLite work on the hub event loop.
   "Concurrent viewers" means overlapping authenticated HTTP requests, not
   parallel SQLite readers.