rigscope 0.1.8 → 0.2.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (94) hide show
  1. package/.next/BUILD_ID +1 -1
  2. package/.next/app-path-routes-manifest.json +1 -1
  3. package/.next/build-manifest.json +2 -2
  4. package/.next/server/app/_global-error/page_client-reference-manifest.js +1 -1
  5. package/.next/server/app/_global-error.html +1 -1
  6. package/.next/server/app/_global-error.rsc +1 -1
  7. package/.next/server/app/_global-error.segments/_full.segment.rsc +1 -1
  8. package/.next/server/app/_global-error.segments/_global-error/__PAGE__.segment.rsc +1 -1
  9. package/.next/server/app/_global-error.segments/_tree.segment.rsc +1 -1
  10. package/.next/server/app/_not-found/page_client-reference-manifest.js +1 -1
  11. package/.next/server/app/_not-found.html +1 -1
  12. package/.next/server/app/_not-found.rsc +3 -3
  13. package/.next/server/app/_not-found.segments/_full.segment.rsc +3 -3
  14. package/.next/server/app/_not-found.segments/_not-found/__PAGE__.segment.rsc +3 -3
  15. package/.next/server/app/_not-found.segments/_tree.segment.rsc +2 -2
  16. package/.next/server/app/administration/page_client-reference-manifest.js +1 -1
  17. package/.next/server/app/administration.html +1 -1
  18. package/.next/server/app/administration.rsc +3 -3
  19. package/.next/server/app/administration.segments/_full.segment.rsc +3 -3
  20. package/.next/server/app/administration.segments/_tree.segment.rsc +2 -2
  21. package/.next/server/app/administration.segments/administration/__PAGE__.segment.rsc +3 -3
  22. package/.next/server/app/alerts/page_client-reference-manifest.js +1 -1
  23. package/.next/server/app/alerts.html +1 -1
  24. package/.next/server/app/alerts.rsc +3 -3
  25. package/.next/server/app/alerts.segments/_full.segment.rsc +3 -3
  26. package/.next/server/app/alerts.segments/_tree.segment.rsc +2 -2
  27. package/.next/server/app/alerts.segments/alerts/__PAGE__.segment.rsc +3 -3
  28. package/.next/server/app/connections/page.js +1 -1
  29. package/.next/server/app/connections/page_client-reference-manifest.js +1 -1
  30. package/.next/server/app/connections.html +1 -1
  31. package/.next/server/app/connections.rsc +4 -4
  32. package/.next/server/app/connections.segments/_full.segment.rsc +4 -4
  33. package/.next/server/app/connections.segments/_tree.segment.rsc +2 -2
  34. package/.next/server/app/connections.segments/connections/__PAGE__.segment.rsc +4 -4
  35. package/.next/server/app/history/page.js +2 -2
  36. package/.next/server/app/history/page_client-reference-manifest.js +1 -1
  37. package/.next/server/app/history.html +1 -1
  38. package/.next/server/app/history.rsc +6 -6
  39. package/.next/server/app/history.segments/_full.segment.rsc +6 -6
  40. package/.next/server/app/history.segments/_tree.segment.rsc +2 -2
  41. package/.next/server/app/history.segments/history/__PAGE__.segment.rsc +6 -6
  42. package/.next/server/app/index.html +1 -1
  43. package/.next/server/app/index.rsc +3 -3
  44. package/.next/server/app/index.segments/__PAGE__.segment.rsc +3 -3
  45. package/.next/server/app/index.segments/_full.segment.rsc +3 -3
  46. package/.next/server/app/index.segments/_tree.segment.rsc +2 -2
  47. package/.next/server/app/login/page_client-reference-manifest.js +1 -1
  48. package/.next/server/app/login.html +1 -1
  49. package/.next/server/app/login.rsc +3 -3
  50. package/.next/server/app/login.segments/_full.segment.rsc +3 -3
  51. package/.next/server/app/login.segments/_tree.segment.rsc +2 -2
  52. package/.next/server/app/login.segments/login/__PAGE__.segment.rsc +3 -3
  53. package/.next/server/app/machines/[machineId]/page_client-reference-manifest.js +1 -1
  54. package/.next/server/app/page_client-reference-manifest.js +1 -1
  55. package/.next/server/app/processes/page_client-reference-manifest.js +1 -1
  56. package/.next/server/app/processes.html +1 -1
  57. package/.next/server/app/processes.rsc +3 -3
  58. package/.next/server/app/processes.segments/_full.segment.rsc +3 -3
  59. package/.next/server/app/processes.segments/_tree.segment.rsc +2 -2
  60. package/.next/server/app/processes.segments/processes/__PAGE__.segment.rsc +3 -3
  61. package/.next/server/app-paths-manifest.json +1 -1
  62. package/.next/server/chunks/161.js +1 -1
  63. package/.next/server/chunks/168.js +1 -1
  64. package/.next/server/middleware-build-manifest.js +1 -1
  65. package/.next/server/pages/404.html +1 -1
  66. package/.next/server/pages/500.html +1 -1
  67. package/.next/static/chunks/408-351fe4a1e9a2d556.js +1 -0
  68. package/.next/static/chunks/939-92930fde48061e20.js +1 -0
  69. package/.next/static/chunks/app/connections/page-7d18b901f2338f20.js +1 -0
  70. package/.next/static/chunks/app/history/page-adc25194b30c689d.js +3 -0
  71. package/.next/static/css/{f5d3162842dcac31.css → 2abef6d49c1c2970.css} +1 -1
  72. package/ARCHITECTURE.md +43 -15
  73. package/CHANGELOG.md +62 -0
  74. package/README.md +10 -3
  75. package/dist/cli.cjs +868 -511
  76. package/dist/collector.cjs +603 -120
  77. package/dist/hub.cjs +3429 -1451
  78. package/dist/ops.cjs +1178 -888
  79. package/dist/system.cjs +1322 -363
  80. package/docs/configuration.md +34 -1
  81. package/docs/installation.md +12 -6
  82. package/docs/monitoring.md +60 -17
  83. package/docs/operations.md +78 -22
  84. package/docs/system-installation.md +11 -4
  85. package/docs/telemetry-storage.md +258 -0
  86. package/package.json +3 -1
  87. package/packaging/collector.env.example +6 -0
  88. package/packaging/system/rigscope-collector.service.example +3 -0
  89. package/.next/static/chunks/408-9fdba9e4e956ef08.js +0 -1
  90. package/.next/static/chunks/939-514e554087419c71.js +0 -1
  91. package/.next/static/chunks/app/connections/page-75abd16fc495fcbd.js +0 -1
  92. package/.next/static/chunks/app/history/page-a25ba8fab7bd7cb9.js +0 -3
  93. /package/.next/static/{ip43TIgmaByEBn2GFtvfO → JbaLjudQSxrMUD_VHPrnD}/_buildManifest.js +0 -0
  94. /package/.next/static/{ip43TIgmaByEBn2GFtvfO → JbaLjudQSxrMUD_VHPrnD}/_ssgManifest.js +0 -0
@@ -17,7 +17,21 @@ configuration files at `0600`; do not commit them.
17
17
  | --- | --- |
18
18
  | `HUB_HOST`, `HUB_PORT` | Local listener: `127.0.0.1:8081`. Only literal loopback addresses are accepted. Port `0` selects an unused port for tests. |
19
19
  | `HUB_PUBLIC_ORIGIN`, `HUB_SOCKET_PATH` | Remote access: set both the exact HTTPS origin and an absolute Unix socket path in a private directory. The hub then has no TCP listener. |
20
- | `HUB_DATABASE_PATH` | SQLite database for telemetry, users, alerts and the collector registry. Defaults beside `HUB_STATE_FILE`, or under `$XDG_STATE_HOME/rigscope/hub/` (falling back to `~/.local/state`). |
20
+ | `HUB_DATABASE_PATH` | SQLite database for control state (users, alerts and the collector registry) and, only when enabled, telemetry history. Defaults beside `HUB_STATE_FILE`, or under `$XDG_STATE_HOME/rigscope/hub/` (falling back to `~/.local/state`). |
21
+ | `HUB_TELEMETRY_MODE` | `memory` (default) keeps bounded history in RAM and writes no routine telemetry rows. `persistent` opts into compact SQLite summaries. See [telemetry storage](telemetry-storage.md). |
22
+ | `HUB_HISTORY_MEMORY_BYTES` | Global RAM budget for memory history. Default `134217728` (128 MiB). |
23
+ | `HUB_TELEMETRY_FLUSH_MS` | Persistent-mode write interval. Default and maximum `5000` ms (5 seconds); allowed 1000–5000. The bound keeps one commit inside the 64-batch collector window at the fastest supported 100 ms cadence and well below the collector's 30 s ack timeout. |
24
+ | `HUB_RETENTION_RAW_MS` | Persistent raw-mode retention. Default 2 days. Persistent summary mode does not write raw rows. |
25
+ | `HUB_RETENTION_10S_MS` | Ten-second summary retention in persistent mode. Default 2 hours. |
26
+ | `HUB_RETENTION_1M_MS` | One-minute rollup retention. Default 30 days. |
27
+ | `HUB_RETENTION_1H_MS` | One-hour rollup retention. Default 400 days. |
28
+ | `HUB_RECEIPT_RETENTION_MS` | Deduplication receipt lifetime, independent of chart retention. Default 7 days. |
29
+ | `HUB_GAP_RETENTION_MS` | Explicit collector gap retention. Default 400 days. |
30
+ | `HUB_RECORDINGS_DIR` | Private directory for explicit recordings. Default `recordings/` beside the hub database (0700 directory, 0600 files). |
31
+ | `HUB_RECORDING_MAX_DURATION_MS` | Maximum recording duration. Default 1 hour; maximum 24 hours. |
32
+ | `HUB_RECORDING_MAX_BYTES` | Maximum recording file size. Default 64 MiB; maximum 1 GiB. |
33
+ | `HUB_RECORDING_BACKLOG_MS` | Recent RAM history a new recording may include. Default 10 minutes. |
34
+ | `HUB_RECORDING_MAX_ACTIVE` | Concurrent active recordings. Default 4; maximum 32. |
21
35
  | `HUB_BROWSER_USERS` | First-run users as JSON: `{username,passwordHash,machineIds,admin?}`. `machineIds` is `"*"` or a list; `[]` allows no machines. |
22
36
  | `HUB_ADMIN_USERS` | Comma-separated usernames to make administrators during first import. If omitted, an existing bootstrap account named `owner` becomes an administrator. Other users need an explicit `admin:true`. |
23
37
  | `HUB_COLLECTORS` | First-run collector credentials as JSON: `{collectorId,machineId,token}`. The limit applies to JSON bytes, not a fixed machine count. |
@@ -25,6 +39,24 @@ configuration files at `0600`; do not commit them.
25
39
  | `HUB_STATE_FILE` | Private enrollment file to import from an existing installation; also enables enrollment. Keep it with the deployment configuration for recovery. |
26
40
  | `HUB_ENROLLMENT_GRANTS` | Initial grants as JSON: `{token,expiresAt}`. Expiry must be within 15 minutes. Used grants stay consumed after import and restart. |
27
41
 
42
+ ### Telemetry history mode
43
+
44
+ The default `HUB_TELEMETRY_MODE=memory` keeps recent charts in a bounded RAM
45
+ budget and deliberately loses telemetry history when the hub restarts. Users,
46
+ credentials, enrollment, settings and alert records remain in SQLite.
47
+
48
+ Set `HUB_TELEMETRY_MODE=persistent` to keep compact ten-second summaries, with
49
+ one transaction roughly every `HUB_TELEMETRY_FLUSH_MS` (five seconds by
50
+ default). Collectors still acknowledge only after that transaction commits, so
51
+ the flush interval cannot exceed the collector's 64-batch in-flight window at
52
+ its advertised cadence; the hub rejects values above five seconds. See
53
+ [telemetry storage](telemetry-storage.md) for aggregation, deduplication,
54
+ recording and reset details.
55
+
56
+ Switching an existing installation to memory mode does not automatically delete
57
+ old telemetry rows. They are ignored, but can be reclaimed explicitly with
58
+ `ops reset-telemetry` after stopping the hub (see [operations](operations.md)).
59
+
28
60
  ### First-run users and credentials
29
61
 
30
62
  The database imports bootstrap users and collector credentials once. Later
@@ -44,6 +76,7 @@ hashes; it reads passwords from stdin. Keep hashes private.
44
76
  | `COLLECTOR_INTERVAL_MS` | Sampling interval: `1000` ms by default, minimum `100` ms. GPU polling runs at most once per second. |
45
77
  | `COLLECTOR_HUBS` | Explicit destinations as JSON: `{url,token,sharing?,intervalMs?}`, up to eight. Otherwise the collector uses saved enrollment. |
46
78
  | `COLLECTOR_HOSTNAME` | Display metadata; defaults to the OS hostname in real mode. |
79
+ | `COLLECTOR_QUEUE_MODE` | Where unacknowledged samples wait: `memory` by default (RAM, no queue written to disk, lost on collector restart) or `durable` (a SQLite outbox per destination that survives a collector restart, at one database write per sample). Bounds are identical: 10 minutes and 64 MiB per destination. |
47
80
 
48
81
  ### Sharing settings
49
82
 
@@ -27,7 +27,7 @@ install or source checkout is needed. The launcher and services use the installe
27
27
  runtime, so removing the npx cache does not break them.
28
28
 
29
29
  Releases are on [npm](https://www.npmjs.com/package/rigscope). `@latest` selects
30
- the current stable release; replace it with `@0.1.8` to select that exact version.
30
+ the current stable release; replace it with `@0.2.0` to select that exact version.
31
31
 
32
32
  ### Installation options
33
33
 
@@ -53,7 +53,7 @@ For a downloaded package or one built from source, run from the directory
53
53
  containing the archive, or give its absolute path:
54
54
 
55
55
  ```sh
56
- npx --yes --package ./rigscope-0.1.8.tgz rigscope install --boot
56
+ npx --yes --package ./rigscope-0.2.0.tgz rigscope install --boot
57
57
  ```
58
58
 
59
59
  The same installation options apply. npm still needs registry access or cached
@@ -156,7 +156,9 @@ tailscale serve --bg --tls-terminated-tcp=8081 \
156
156
  ```
157
157
 
158
158
  This keeps accounts, history and collector credentials. It updates the local
159
- collector's destination and moves its pending samples. Rigscope asks to stop
159
+ collector's destination and moves its pending samples, which is only relevant in
160
+ `COLLECTOR_QUEUE_MODE=durable`: the default RAM queue drains with the collector.
161
+ Rigscope asks to stop
160
162
  running components during conversion and resumes them afterward. See [Access changes](operations.md#access-changes)
161
163
  for backup and queue checks.
162
164
 
@@ -290,12 +292,16 @@ npx --yes rigscope@latest install --boot
290
292
  ```
291
293
 
292
294
  Both paths stop the owned services and reset the pre-canonical configuration and
293
- history. Future upgrades that stay in the canonical format keep accounts, history,
294
- settings and collector identities.
295
+ history. Upgrades that stay in the canonical format keep accounts, settings and
296
+ collector identities; the first one after the RAM-first release additionally
297
+ completes a one-time transition that discards machine-derived history and
298
+ pre-RAM-first durable collector queues once. Later upgrades preserve durable
299
+ history and queues. See
300
+ [telemetry storage](telemetry-storage.md#the-one-time-machine-data-transition).
295
301
 
296
302
  | Alternative | Command |
297
303
  | --- | --- |
298
- | Select an exact published version | `rigscope upgrade --from rigscope@0.1.8` |
304
+ | Select an exact published version | `rigscope upgrade --from rigscope@0.2.0` |
299
305
  | Install a local archive | `rigscope upgrade --from ./rigscope-VERSION.tgz` |
300
306
  | Return to the retained previous runtime | `rigscope rollback` |
301
307
 
@@ -20,22 +20,61 @@ driver and readable counters; see [Hardware](installation.md#hardware) and
20
20
 
21
21
  ## History and delivery
22
22
 
23
- The hub acknowledges samples after committing them to SQLite. A collector keeps
24
- unacknowledged samples in a persistent queue for each hub and retries after a
25
- disconnect. The hub removes duplicates using the collector, run and sequence IDs.
23
+ History is **RAM-only by default**. Live readings update immediately, recent
24
+ charts come from a bounded memory budget, and telemetry history disappears when
25
+ the hub restarts. Users, credentials, enrollment, settings, alert records,
26
+ machine names/tags and inventory remain persistent.
27
+
28
+ In memory mode the collector's acknowledgement means the hub accepted the sample
29
+ into RAM. The collector queues pending samples in RAM and retries after a
30
+ disconnect, so it gains no disk work from the hub's choice and writes no queue
31
+ of its own. Deduplication
32
+ uses a compact per-run interval set for the hub process lifetime. Because both
33
+ history and receipts disappear together on restart, a replay cannot double-count
34
+ restored chart data.
35
+
36
+ Set `HUB_TELEMETRY_MODE=persistent` to opt into disk history. The hub buffers a
37
+ five-second window, publishes live state immediately, writes one compact
38
+ transaction, and only then acknowledges. Replay deduplication uses durable
39
+ receipts whose lifetime is independent of chart retention (default seven days).
26
40
  Replayed samples fill History without replacing current live readings or opening
27
- alerts for past threshold crossings.
41
+ alerts for past threshold crossings. A graceful shutdown flushes pending
42
+ summaries; after a crash the collector replays anything the hub never
43
+ acknowledged, provided the collector itself kept it: only
44
+ `COLLECTOR_QUEUE_MODE=durable` survives a collector restart.
28
45
 
29
- Default retention:
46
+ Default memory history:
30
47
 
31
48
  | Resolution | Kept for |
32
49
  | --- | --- |
33
- | Raw samples | 48 hours |
34
- | One-minute summaries | 30 days |
35
- | One-hour summaries | 400 days |
36
-
37
- Summaries retain minimum and maximum values. History queries limit the number
38
- of points, series, scanned rows and time spent. The UI marks partial results.
50
+ | 1-second detail | 2 minutes |
51
+ | 10-second summaries | 1 hour |
52
+ | One-minute summaries | 24 hours |
53
+ | One-hour summaries | 7 days |
54
+
55
+ Per-core CPU keeps less detail; machine-wide and capacity metrics keep more.
56
+ Persistent mode writes compact ten-second summaries and folds them into
57
+ one-minute and one-hour tiers; it does not write a raw row per reading. It keeps
58
+ 10-second summaries for 2 hours, one-minute summaries for 30 days and hourly
59
+ summaries for 400 days by default. Both modes retain minimum and maximum values,
60
+ so short spikes remain visible, and both preserve explicit unavailable readings
61
+ and gaps.
62
+
63
+ RAM history is bounded by `HUB_HISTORY_MEMORY_BYTES` (default 128 MiB) across all
64
+ machines. When the budget fills, the oldest fine-grained tiers are dropped first
65
+ and new cardinality beyond the sustainable budget is rejected with visible
66
+ pressure. History queries limit points, series, scanned rows and elapsed time;
67
+ the UI marks partial results.
68
+
69
+ ### Recording
70
+
71
+ Use **History → Recording** to start an explicit capture for the current
72
+ machine and selected metrics. Choose a resolution from 1 second to 1 hour,
73
+ a duration and whether to include recent RAM history. The capture is bounded by
74
+ duration and bytes, survives restart through its private persisted output, and
75
+ can be downloaded by its creator. Recording also works when chart history is
76
+ temporary, which is the intended way to keep an incident snapshot. See
77
+ [telemetry storage](telemetry-storage.md#recording).
39
78
 
40
79
  Hover over a History or Overview recent-usage chart to inspect a recorded time
41
80
  and its bucket averages. The time marker follows across charts in the same view;
@@ -49,11 +88,12 @@ Each collector queue keeps up to ten minutes or 64 MiB of queued JSON per hub,
49
88
  with a 256 MiB total JSON limit. Physical SQLite files, including WAL files,
50
89
  have separate limits of 80 MiB per hub and 640 MiB across eight hubs.
51
90
 
52
- The hub's default database limit is 2 GiB, including its WAL file and all tables.
53
- Retention protects the newest hour of telemetry. If old-data cleanup cannot
54
- free enough space, the hub stops acknowledging new samples and collectors retry.
55
- Existing oversized databases report pressure; startup does not run a destructive
56
- cleanup.
91
+ In persistent mode the hub's default database limit is 2 GiB, including WAL and
92
+ all control tables. Retention protects the newest hour; if old-data cleanup
93
+ cannot free enough space, the hub stops acknowledging new samples and collectors
94
+ retry. Existing oversized databases report pressure; startup does not run a
95
+ destructive cleanup. In memory mode old historical rows may still occupy the
96
+ file until you run `ops reset-telemetry`; they are not read or updated.
57
97
 
58
98
  These are application limits, not filesystem quotas. A transaction can briefly
59
99
  grow the WAL before checkpointing. Blocked checkpoints stop further ingestion
@@ -92,7 +132,10 @@ credentials persist in SQLite. Browser sessions stay in memory, expire after
92
132
  Alerts support above/below thresholds, durations, unavailable readings, missing
93
133
  receipts, maintenance and recovery. Value, unavailable and stale-receipt rules
94
134
  apply to the machines tracked by that hub. Replayed samples do not open past
95
- incidents.
135
+ incidents. Alert rules, incidents and pending notifications stay durable in both
136
+ history modes; with no rules configured, live ingestion does not write per-sample
137
+ alert progress. Enabling rules may write alert state on evaluation, which is
138
+ operational state rather than telemetry history.
96
139
 
97
140
  In **Alerts**, configure one HTTPS webhook channel. Its URL, headers and HMAC
98
141
  secret are write-only. Loopback HTTP is allowed for controlled test sinks.
@@ -10,6 +10,7 @@ building and deploying from source.
10
10
  | Reset a lost password | [Password recovery](#password-recovery) |
11
11
  | Change localhost/HTTPS access | [Access changes](#access-changes) |
12
12
  | Back up or restore a database | [Database commands](#database-commands) |
13
+ | Discard old telemetry history only | [Database commands](#database-commands) |
13
14
  | Plan what to back up | [Consistent backup set](#consistent-backup-set) |
14
15
  | Handle a full database or queue | [Storage pressure recovery](#storage-budgets-and-pressure-recovery) |
15
16
 
@@ -54,9 +55,26 @@ running, then checks:
54
55
  The health timeout is 25 seconds; change it with `--timeout SECONDS`. A failed
55
56
  start, migration or health check restores the previous runtime. It also restores
56
57
  the paired database backup if the old runtime cannot read the new schema,
57
- preserving the replaced database first.
58
-
59
- Credentials, collector identities and pending samples are retained.
58
+ preserving the replaced database first — and additionally whenever the
59
+ one-time machine-data transition described below had already discarded rows,
60
+ even if the schema is compatible, because the loss there is the rows, not the
61
+ schema.
62
+
63
+ The first upgrade after the RAM-first release completes a one-time
64
+ machine-data transition: it discards the previous design's on-disk machine
65
+ history — telemetry rows and summaries, receipts, gaps, latest snapshots,
66
+ inventory and machine-derived alert history — plus pre-RAM-first durable
67
+ collector queues, once, even data previously opted into keeping. Machine
68
+ display names/tags (`machine_meta`), accounts, sessions, enrollment and grants,
69
+ settings, alert rules/jobs/webhook configuration and recordings are never
70
+ discarded. A version-independent `machine-data-transition.json` marker in the
71
+ state directory records completion; every later upgrade preserves durable
72
+ history and queues you opt back into (`COLLECTOR_QUEUE_MODE=durable`, durable
73
+ hub history). The upgrade output reports exactly what was discarded; a cleanup
74
+ that cannot finish is shown as a warning and retried by the next upgrade
75
+ instead of failing an otherwise healthy one. See
76
+ [telemetry storage](telemetry-storage.md#the-one-time-machine-data-transition).
77
+ Credentials and collector identities are always retained.
60
78
  `release-state.json` records which backup belongs to each runtime. Upgrading a
61
79
  stopped installation leaves it stopped and reports live checks as skipped.
62
80
 
@@ -66,7 +84,8 @@ release, stops the hub and collector completely, then removes
66
84
  the old configuration and history and installing the latest release fresh. Back
67
85
  up the state directory first if that history must be kept. There is no migration
68
86
  to the canonical format and no rollback to that old shape. Future upgrades that
69
- stay canonical keep accounts, history, settings and collector identities.
87
+ stay canonical keep accounts, settings and collector identities, and — apart
88
+ from the one-time transition above — durable history and queues.
70
89
 
71
90
  ### Roll back
72
91
 
@@ -191,7 +210,7 @@ The installer creates:
191
210
  hub.env private EnvironmentFile (0600, no secrets printed)
192
211
  collector.env private EnvironmentFile (0600)
193
212
  collector/collector.json collector identity/enrollment (created by collector)
194
- outbox/<sha256(url)[0:16]>/ per-destination collector outboxes
213
+ outbox/<sha256(url)[0:16]>/ per-destination queues, durable mode only
195
214
  backups/ private pre-upgrade/pre-activation database copies
196
215
  install-state.json canonical versions/schema backups, no credentials
197
216
  run/ private runtime directory for a Unix socket
@@ -314,6 +333,9 @@ COLLECTOR_ENROLL_URL=ws://127.0.0.1:8081/api/collector/connect
314
333
  COLLECTOR_ENROLL_TOKEN=<one-use grant>
315
334
  COLLECTOR_MODE=real
316
335
  COLLECTOR_STATE_FILE=/home/operator/.local/state/rigscope/collector/collector.json
336
+ # Optional: keep unacknowledged samples in a per-hub SQLite outbox so they
337
+ # survive a collector restart. Default is RAM (no queue written to disk).
338
+ # COLLECTOR_QUEUE_MODE=durable
317
339
  ```
318
340
 
319
341
  Remove the one-use grant after enrollment and start or restart the collector. The
@@ -369,6 +391,11 @@ node dist/ops.cjs restore \
369
391
  --database /path/telemetry.sqlite \
370
392
  --replace --confirm-stopped
371
393
 
394
+ # Discard only telemetry history, preserving users/enrollment/settings/alerts.
395
+ node dist/ops.cjs reset-telemetry \
396
+ --database /path/telemetry.sqlite \
397
+ --backup-dir /path/private-backups --confirm-stopped
398
+
372
399
  # Offline managed-user recovery; password comes from a private 0600 file.
373
400
  node dist/ops.cjs recover-admin \
374
401
  --database /path/telemetry.sqlite \
@@ -408,18 +435,24 @@ release must never be pointed at a newer schema. `inspect` reports this state.
408
435
 
409
436
  The database backup is not the whole installation. A restorable set is:
410
437
 
411
- 1. Hub SQLite database: `telemetry.sqlite` (includes registry/admin/alerts).
412
- Use `ops backup`; do not copy the main file alone while the hub is running.
438
+ 1. Hub SQLite database: `telemetry.sqlite` (includes registry/admin/alerts and
439
+ any opt-in persistent history). Use `ops backup`; do not copy the main file
440
+ alone while the hub is running.
413
441
  2. Enrollment/config state when configured: `HUB_STATE_FILE` (for
414
442
  example `hub/enrollment.json`, the import source for previously enrolled
415
443
  collectors) and the private hub environment file. These
416
444
  contain credentials: protect and copy them only between private locations.
417
445
  3. Collector identity/enrollment: `collector/collector.json`.
418
- 4. Collector outboxes: `outbox/<destination-hash>/outbox.sqlite` per hub URL,
419
- which may contain unacknowledged telemetry. Replay is at-least-once; the hub
420
- deduplicates by collector/run/sequence.
446
+ 4. Collector queues, `outbox/<destination-hash>/outbox.sqlite` per hub URL, exist
447
+ only under `COLLECTOR_QUEUE_MODE=durable` and may contain unacknowledged
448
+ telemetry. Replay is at-least-once; the hub deduplicates by
449
+ collector/run/sequence. The default RAM queue holds nothing on disk, so there
450
+ is nothing to back up and nothing to restore.
421
451
  5. Optional operational metadata: `install-state.json`, units and the built
422
452
  release. Keep the release matching the database schema for rollback.
453
+ 6. Explicit recordings: the private `HUB_RECORDINGS_DIR` (default
454
+ `<hub database directory>/recordings`) if you want downloadable captures to
455
+ survive a restore. They are user output, not part of the database backup.
423
456
 
424
457
  For a consistent set, prefer stopping both services cleanly, then run the online
425
458
  database backup and copy the other private files. For an online backup, the SQLite
@@ -472,7 +505,10 @@ use the component controls in [System installation](system-installation.md#syste
472
505
  The collector uses only `/api/collector/connect` and the canonical record shape.
473
506
  There is no old-endpoint normalization, outbox migration or old-peer payload
474
507
  conversion. Reset unfinished installations and re-enroll their collectors;
475
- canonical upgrades retain their identities, credentials and pending samples.
508
+ canonical upgrades retain their identities and credentials. Pending samples are
509
+ retained by every upgrade after the one-time machine-data transition, which
510
+ discards pre-RAM-first durable queues once; from then on
511
+ `COLLECTOR_QUEUE_MODE=durable` content survives upgrades.
476
512
 
477
513
  ## Measurement
478
514
 
@@ -486,7 +522,18 @@ node dist/ops.cjs measure --machines 2 --batches 30 --records-per-batch 5 \
486
522
  It writes synthetic records through the real `TelemetryStore` into a
487
523
  temporary private database and reports accepted/rejected/busy counts, ingestion
488
524
  latency, event-loop delay/utilization, RSS, database/WAL size, store queue and
489
- flush counters, and one history-query latency. The output is labelled synthetic.
525
+ flush counters, and one history-query latency.
526
+
527
+ For the history-mode tradeoff, use the repeatable RAM-vs-persistent benchmark.
528
+ It measures process-attributed disk writes, latency, RSS and rows:
529
+
530
+ ```sh
531
+ npm run benchmark:telemetry -- --machines 2 --seconds 120
532
+ npm run benchmark:telemetry -- --machines 10 --seconds 60 --json
533
+ ```
534
+
535
+ Use `--dir` on a real filesystem; tmpfs does not account storage writes in
536
+ `/proc/self/io`. See [telemetry storage](telemetry-storage.md#benchmark). The output is labelled synthetic.
490
537
  Defaults are intentionally small; upper bounds are 1,000 machines, 2,000 batches,
491
538
  32 records per batch and 100,000 records total. It never opens a service socket
492
539
  or contacts a hub. `--keep` retains the temporary directory for inspection. The
@@ -528,20 +575,29 @@ absolute regular file; use the release's `node_modules` installed with
528
575
  ## Storage budgets and pressure recovery
529
576
 
530
577
  Each Collector Hub connection owns policy filtering, cadence, inventory and its
531
- durable outbox submission. On restart, recovered metrics are sanitized with the
532
- current sharing policy before the connection starts; capture-time cadence is not
533
- reapplied to retained records. Endpoint upgrade recovery and aggregate budgets
534
- continue to use the existing outboxes. Hub startup restores stored telemetry,
578
+ delivery queue submission. Under `COLLECTOR_QUEUE_MODE=durable`, recovered metrics
579
+ are sanitized with the current sharing policy before the connection starts;
580
+ capture-time cadence is not reapplied to retained records. Endpoint upgrade
581
+ recovery and aggregate budgets continue to use the existing queues. The default
582
+ RAM queue starts empty, so its bounds are enforced against RAM rather than disk. Hub startup restores stored telemetry,
535
583
  cadence, inventory and metadata through Sample ingestion while retaining a
536
584
  disconnected state until an authenticated Collector reconnects.
537
585
 
538
586
  The default hub main-file plus WAL budget is 2 GiB, shared by every SQLite table.
539
- The main file has a page-count limit; WAL reserves up to 128 MiB inside that budget.
540
- Telemetry writes return busy on pressure or SQLITE_FULL, with no receipt/ack for a
541
- failed transaction. Retention reclaims old raw/one-minute points in bounded chunks
542
- while preserving the newest hour; hourly rollups retain their configured age window.
543
- A full newest-hour working set must fit the deployment's budget. Outboxes retry
544
- while the operator frees space or provisions a larger supported configuration.
587
+ The main file has a page-count limit; WAL reserves up to 128 MiB inside that budget
588
+ and automatic checkpointing waits for a meaningful 32–64 MiB reserve where the
589
+ budget permits. Persistent-mode writes return busy on pressure or SQLITE_FULL,
590
+ with no receipt/ack for a failed transaction. Retention reclaims old raw or
591
+ ten-second points and one-minute rollups in bounded chunks while preserving the
592
+ newest hour; hourly rollups retain their configured age window. A full
593
+ newest-hour working set must fit the deployment's budget. Outboxes retry while
594
+ the operator frees space or provisions a larger supported configuration.
595
+
596
+ Memory mode keeps only a bounded RAM budget (`HUB_HISTORY_MEMORY_BYTES`, default
597
+ 128 MiB). It does not write routine telemetry rows, but existing historical rows
598
+ from an older installation still occupy the database until removed with
599
+ `ops reset-telemetry` while the hub is stopped. That command preserves users,
600
+ credentials, enrollment, settings, alert state, metadata and inventory.
545
601
 
546
602
  Collector defaults are 64 MiB logical JSON and 80 MiB physical main+WAL per hub,
547
603
  256 MiB logical and 640 MiB physical across eight destinations. Physical pressure
@@ -31,7 +31,8 @@ The installed `rigscope system` command and `dist/system.cjs` use the same code.
31
31
  hub/enrollment.json rigscope-hub 0600
32
32
  hub/run/hub.sock private Unix socket (remote mode)
33
33
  collector/state.json rigscope-collector 0600
34
- collector/outbox/ durable collector outboxes
34
+ collector/outbox/ collector queues, durable mode only
35
+ machine-data-transition.json root:root 0600, one-time machine-data transition marker
35
36
  backups/ root:root 0700, pre-upgrade/pre-rollback SQLite copies
36
37
  system-install.json root:root 0600, canonical component state/versions/backups/adoption
37
38
  power-access.json root:root 0600, original counter modes for reversal
@@ -104,8 +105,14 @@ A pre-canonical system installation is reset, not migrated. `system install` and
104
105
  units completely, then remove the old configuration and history and install
105
106
  the canonical layout fresh. Back up `/var/lib/rigscope` first if that history
106
107
  must be kept; there is no rollback to the old format. Later upgrades that stay
107
- canonical keep accounts, history, settings, collector identities and pending
108
- samples, and rollback keeps its paired database backup.
108
+ canonical keep accounts, settings and collector identities and rollback keeps
109
+ its paired database backup. The first such upgrade after the RAM-first release
110
+ also completes the one-time machine-data transition, discarding machine-derived
111
+ history and pre-RAM-first durable collector queues once — even data previously
112
+ opted into keeping — while preserving machine names/tags; a
113
+ `machine-data-transition.json` marker in `/var/lib/rigscope` makes every
114
+ subsequent upgrade preserve durable history and queues. See
115
+ [telemetry storage](telemetry-storage.md#the-one-time-machine-data-transition).
109
116
 
110
117
  - With `--enable`, `--role local` initially enables both components.
111
118
  - With `--enable`, `--role hub` initially enables only the hub.
@@ -131,7 +138,7 @@ identity and RAPL grant. With no destination it tries the colocated hub:
131
138
  cleanly before collection until a destination is saved.
132
139
 
133
140
  Configured remote destinations and credentials are never overwritten. Re-running
134
- `enable collector` leaves its destination, token, identity and outboxes intact.
141
+ `enable collector` leaves its destination, token and identity intact.
135
142
 
136
143
  ### Connecting to an independent hub
137
144