rigscope 0.1.8 → 0.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.next/BUILD_ID +1 -1
- package/.next/app-path-routes-manifest.json +1 -1
- package/.next/build-manifest.json +2 -2
- package/.next/server/app/_global-error/page_client-reference-manifest.js +1 -1
- package/.next/server/app/_global-error.html +1 -1
- package/.next/server/app/_global-error.rsc +1 -1
- package/.next/server/app/_global-error.segments/_full.segment.rsc +1 -1
- package/.next/server/app/_global-error.segments/_global-error/__PAGE__.segment.rsc +1 -1
- package/.next/server/app/_global-error.segments/_tree.segment.rsc +1 -1
- package/.next/server/app/_not-found/page_client-reference-manifest.js +1 -1
- package/.next/server/app/_not-found.html +1 -1
- package/.next/server/app/_not-found.rsc +3 -3
- package/.next/server/app/_not-found.segments/_full.segment.rsc +3 -3
- package/.next/server/app/_not-found.segments/_not-found/__PAGE__.segment.rsc +3 -3
- package/.next/server/app/_not-found.segments/_tree.segment.rsc +2 -2
- package/.next/server/app/administration/page_client-reference-manifest.js +1 -1
- package/.next/server/app/administration.html +1 -1
- package/.next/server/app/administration.rsc +3 -3
- package/.next/server/app/administration.segments/_full.segment.rsc +3 -3
- package/.next/server/app/administration.segments/_tree.segment.rsc +2 -2
- package/.next/server/app/administration.segments/administration/__PAGE__.segment.rsc +3 -3
- package/.next/server/app/alerts/page_client-reference-manifest.js +1 -1
- package/.next/server/app/alerts.html +1 -1
- package/.next/server/app/alerts.rsc +3 -3
- package/.next/server/app/alerts.segments/_full.segment.rsc +3 -3
- package/.next/server/app/alerts.segments/_tree.segment.rsc +2 -2
- package/.next/server/app/alerts.segments/alerts/__PAGE__.segment.rsc +3 -3
- package/.next/server/app/connections/page.js +1 -1
- package/.next/server/app/connections/page_client-reference-manifest.js +1 -1
- package/.next/server/app/connections.html +1 -1
- package/.next/server/app/connections.rsc +4 -4
- package/.next/server/app/connections.segments/_full.segment.rsc +4 -4
- package/.next/server/app/connections.segments/_tree.segment.rsc +2 -2
- package/.next/server/app/connections.segments/connections/__PAGE__.segment.rsc +4 -4
- package/.next/server/app/history/page.js +2 -2
- package/.next/server/app/history/page_client-reference-manifest.js +1 -1
- package/.next/server/app/history.html +1 -1
- package/.next/server/app/history.rsc +6 -6
- package/.next/server/app/history.segments/_full.segment.rsc +6 -6
- package/.next/server/app/history.segments/_tree.segment.rsc +2 -2
- package/.next/server/app/history.segments/history/__PAGE__.segment.rsc +6 -6
- package/.next/server/app/index.html +1 -1
- package/.next/server/app/index.rsc +3 -3
- package/.next/server/app/index.segments/__PAGE__.segment.rsc +3 -3
- package/.next/server/app/index.segments/_full.segment.rsc +3 -3
- package/.next/server/app/index.segments/_tree.segment.rsc +2 -2
- package/.next/server/app/login/page_client-reference-manifest.js +1 -1
- package/.next/server/app/login.html +1 -1
- package/.next/server/app/login.rsc +3 -3
- package/.next/server/app/login.segments/_full.segment.rsc +3 -3
- package/.next/server/app/login.segments/_tree.segment.rsc +2 -2
- package/.next/server/app/login.segments/login/__PAGE__.segment.rsc +3 -3
- package/.next/server/app/machines/[machineId]/page_client-reference-manifest.js +1 -1
- package/.next/server/app/page_client-reference-manifest.js +1 -1
- package/.next/server/app/processes/page_client-reference-manifest.js +1 -1
- package/.next/server/app/processes.html +1 -1
- package/.next/server/app/processes.rsc +3 -3
- package/.next/server/app/processes.segments/_full.segment.rsc +3 -3
- package/.next/server/app/processes.segments/_tree.segment.rsc +2 -2
- package/.next/server/app/processes.segments/processes/__PAGE__.segment.rsc +3 -3
- package/.next/server/app-paths-manifest.json +1 -1
- package/.next/server/chunks/161.js +1 -1
- package/.next/server/chunks/168.js +1 -1
- package/.next/server/middleware-build-manifest.js +1 -1
- package/.next/server/pages/404.html +1 -1
- package/.next/server/pages/500.html +1 -1
- package/.next/static/chunks/408-351fe4a1e9a2d556.js +1 -0
- package/.next/static/chunks/939-92930fde48061e20.js +1 -0
- package/.next/static/chunks/app/connections/page-7d18b901f2338f20.js +1 -0
- package/.next/static/chunks/app/history/page-adc25194b30c689d.js +3 -0
- package/.next/static/css/{f5d3162842dcac31.css → 2abef6d49c1c2970.css} +1 -1
- package/ARCHITECTURE.md +43 -15
- package/CHANGELOG.md +62 -0
- package/README.md +10 -3
- package/dist/cli.cjs +868 -511
- package/dist/collector.cjs +603 -120
- package/dist/hub.cjs +3429 -1451
- package/dist/ops.cjs +1178 -888
- package/dist/system.cjs +1322 -363
- package/docs/configuration.md +34 -1
- package/docs/installation.md +12 -6
- package/docs/monitoring.md +60 -17
- package/docs/operations.md +78 -22
- package/docs/system-installation.md +11 -4
- package/docs/telemetry-storage.md +258 -0
- package/package.json +3 -1
- package/packaging/collector.env.example +6 -0
- package/packaging/system/rigscope-collector.service.example +3 -0
- package/.next/static/chunks/408-9fdba9e4e956ef08.js +0 -1
- package/.next/static/chunks/939-514e554087419c71.js +0 -1
- package/.next/static/chunks/app/connections/page-75abd16fc495fcbd.js +0 -1
- package/.next/static/chunks/app/history/page-a25ba8fab7bd7cb9.js +0 -3
- /package/.next/static/{ip43TIgmaByEBn2GFtvfO → JbaLjudQSxrMUD_VHPrnD}/_buildManifest.js +0 -0
- /package/.next/static/{ip43TIgmaByEBn2GFtvfO → JbaLjudQSxrMUD_VHPrnD}/_ssgManifest.js +0 -0
package/docs/configuration.md
CHANGED
|
@@ -17,7 +17,21 @@ configuration files at `0600`; do not commit them.
|
|
|
17
17
|
| --- | --- |
|
|
18
18
|
| `HUB_HOST`, `HUB_PORT` | Local listener: `127.0.0.1:8081`. Only literal loopback addresses are accepted. Port `0` selects an unused port for tests. |
|
|
19
19
|
| `HUB_PUBLIC_ORIGIN`, `HUB_SOCKET_PATH` | Remote access: set both the exact HTTPS origin and an absolute Unix socket path in a private directory. The hub then has no TCP listener. |
|
|
20
|
-
| `HUB_DATABASE_PATH` | SQLite database for
|
|
20
|
+
| `HUB_DATABASE_PATH` | SQLite database for control state (users, alerts and the collector registry) and, only when enabled, telemetry history. Defaults beside `HUB_STATE_FILE`, or under `$XDG_STATE_HOME/rigscope/hub/` (falling back to `~/.local/state`). |
|
|
21
|
+
| `HUB_TELEMETRY_MODE` | `memory` (default) keeps bounded history in RAM and writes no routine telemetry rows. `persistent` opts into compact SQLite summaries. See [telemetry storage](telemetry-storage.md). |
|
|
22
|
+
| `HUB_HISTORY_MEMORY_BYTES` | Global RAM budget for memory history. Default `134217728` (128 MiB). |
|
|
23
|
+
| `HUB_TELEMETRY_FLUSH_MS` | Persistent-mode write interval. Default and maximum `5000` ms (5 seconds); allowed 1000–5000. The bound keeps one commit inside the 64-batch collector window at the fastest supported 100 ms cadence and well below the collector's 30 s ack timeout. |
|
|
24
|
+
| `HUB_RETENTION_RAW_MS` | Persistent raw-mode retention. Default 2 days. Persistent summary mode does not write raw rows. |
|
|
25
|
+
| `HUB_RETENTION_10S_MS` | Ten-second summary retention in persistent mode. Default 2 hours. |
|
|
26
|
+
| `HUB_RETENTION_1M_MS` | One-minute rollup retention. Default 30 days. |
|
|
27
|
+
| `HUB_RETENTION_1H_MS` | One-hour rollup retention. Default 400 days. |
|
|
28
|
+
| `HUB_RECEIPT_RETENTION_MS` | Deduplication receipt lifetime, independent of chart retention. Default 7 days. |
|
|
29
|
+
| `HUB_GAP_RETENTION_MS` | Explicit collector gap retention. Default 400 days. |
|
|
30
|
+
| `HUB_RECORDINGS_DIR` | Private directory for explicit recordings. Default `recordings/` beside the hub database (0700 directory, 0600 files). |
|
|
31
|
+
| `HUB_RECORDING_MAX_DURATION_MS` | Maximum recording duration. Default 1 hour; maximum 24 hours. |
|
|
32
|
+
| `HUB_RECORDING_MAX_BYTES` | Maximum recording file size. Default 64 MiB; maximum 1 GiB. |
|
|
33
|
+
| `HUB_RECORDING_BACKLOG_MS` | Recent RAM history a new recording may include. Default 10 minutes. |
|
|
34
|
+
| `HUB_RECORDING_MAX_ACTIVE` | Concurrent active recordings. Default 4; maximum 32. |
|
|
21
35
|
| `HUB_BROWSER_USERS` | First-run users as JSON: `{username,passwordHash,machineIds,admin?}`. `machineIds` is `"*"` or a list; `[]` allows no machines. |
|
|
22
36
|
| `HUB_ADMIN_USERS` | Comma-separated usernames to make administrators during first import. If omitted, an existing bootstrap account named `owner` becomes an administrator. Other users need an explicit `admin:true`. |
|
|
23
37
|
| `HUB_COLLECTORS` | First-run collector credentials as JSON: `{collectorId,machineId,token}`. The limit applies to JSON bytes, not a fixed machine count. |
|
|
@@ -25,6 +39,24 @@ configuration files at `0600`; do not commit them.
|
|
|
25
39
|
| `HUB_STATE_FILE` | Private enrollment file to import from an existing installation; also enables enrollment. Keep it with the deployment configuration for recovery. |
|
|
26
40
|
| `HUB_ENROLLMENT_GRANTS` | Initial grants as JSON: `{token,expiresAt}`. Expiry must be within 15 minutes. Used grants stay consumed after import and restart. |
|
|
27
41
|
|
|
42
|
+
### Telemetry history mode
|
|
43
|
+
|
|
44
|
+
The default `HUB_TELEMETRY_MODE=memory` keeps recent charts in a bounded RAM
|
|
45
|
+
budget and deliberately loses telemetry history when the hub restarts. Users,
|
|
46
|
+
credentials, enrollment, settings and alert records remain in SQLite.
|
|
47
|
+
|
|
48
|
+
Set `HUB_TELEMETRY_MODE=persistent` to keep compact ten-second summaries, with
|
|
49
|
+
one transaction roughly every `HUB_TELEMETRY_FLUSH_MS` (five seconds by
|
|
50
|
+
default). Collectors still acknowledge only after that transaction commits, so
|
|
51
|
+
the flush interval cannot exceed the collector's 64-batch in-flight window at
|
|
52
|
+
its advertised cadence; the hub rejects values above five seconds. See
|
|
53
|
+
[telemetry storage](telemetry-storage.md) for aggregation, deduplication,
|
|
54
|
+
recording and reset details.
|
|
55
|
+
|
|
56
|
+
Switching an existing installation to memory mode does not automatically delete
|
|
57
|
+
old telemetry rows. They are ignored, but can be reclaimed explicitly with
|
|
58
|
+
`ops reset-telemetry` after stopping the hub (see [operations](operations.md)).
|
|
59
|
+
|
|
28
60
|
### First-run users and credentials
|
|
29
61
|
|
|
30
62
|
The database imports bootstrap users and collector credentials once. Later
|
|
@@ -44,6 +76,7 @@ hashes; it reads passwords from stdin. Keep hashes private.
|
|
|
44
76
|
| `COLLECTOR_INTERVAL_MS` | Sampling interval: `1000` ms by default, minimum `100` ms. GPU polling runs at most once per second. |
|
|
45
77
|
| `COLLECTOR_HUBS` | Explicit destinations as JSON: `{url,token,sharing?,intervalMs?}`, up to eight. Otherwise the collector uses saved enrollment. |
|
|
46
78
|
| `COLLECTOR_HOSTNAME` | Display metadata; defaults to the OS hostname in real mode. |
|
|
79
|
+
| `COLLECTOR_QUEUE_MODE` | Where unacknowledged samples wait: `memory` by default (RAM, no queue written to disk, lost on collector restart) or `durable` (a SQLite outbox per destination that survives a collector restart, at one database write per sample). Bounds are identical: 10 minutes and 64 MiB per destination. |
|
|
47
80
|
|
|
48
81
|
### Sharing settings
|
|
49
82
|
|
package/docs/installation.md
CHANGED
|
@@ -27,7 +27,7 @@ install or source checkout is needed. The launcher and services use the installe
|
|
|
27
27
|
runtime, so removing the npx cache does not break them.
|
|
28
28
|
|
|
29
29
|
Releases are on [npm](https://www.npmjs.com/package/rigscope). `@latest` selects
|
|
30
|
-
the current stable release; replace it with `@0.
|
|
30
|
+
the current stable release; replace it with `@0.2.0` to select that exact version.
|
|
31
31
|
|
|
32
32
|
### Installation options
|
|
33
33
|
|
|
@@ -53,7 +53,7 @@ For a downloaded package or one built from source, run from the directory
|
|
|
53
53
|
containing the archive, or give its absolute path:
|
|
54
54
|
|
|
55
55
|
```sh
|
|
56
|
-
npx --yes --package ./rigscope-0.
|
|
56
|
+
npx --yes --package ./rigscope-0.2.0.tgz rigscope install --boot
|
|
57
57
|
```
|
|
58
58
|
|
|
59
59
|
The same installation options apply. npm still needs registry access or cached
|
|
@@ -156,7 +156,9 @@ tailscale serve --bg --tls-terminated-tcp=8081 \
|
|
|
156
156
|
```
|
|
157
157
|
|
|
158
158
|
This keeps accounts, history and collector credentials. It updates the local
|
|
159
|
-
collector's destination and moves its pending samples
|
|
159
|
+
collector's destination and moves its pending samples, which is only relevant in
|
|
160
|
+
`COLLECTOR_QUEUE_MODE=durable`: the default RAM queue drains with the collector.
|
|
161
|
+
Rigscope asks to stop
|
|
160
162
|
running components during conversion and resumes them afterward. See [Access changes](operations.md#access-changes)
|
|
161
163
|
for backup and queue checks.
|
|
162
164
|
|
|
@@ -290,12 +292,16 @@ npx --yes rigscope@latest install --boot
|
|
|
290
292
|
```
|
|
291
293
|
|
|
292
294
|
Both paths stop the owned services and reset the pre-canonical configuration and
|
|
293
|
-
history.
|
|
294
|
-
|
|
295
|
+
history. Upgrades that stay in the canonical format keep accounts, settings and
|
|
296
|
+
collector identities; the first one after the RAM-first release additionally
|
|
297
|
+
completes a one-time transition that discards machine-derived history and
|
|
298
|
+
pre-RAM-first durable collector queues once. Later upgrades preserve durable
|
|
299
|
+
history and queues. See
|
|
300
|
+
[telemetry storage](telemetry-storage.md#the-one-time-machine-data-transition).
|
|
295
301
|
|
|
296
302
|
| Alternative | Command |
|
|
297
303
|
| --- | --- |
|
|
298
|
-
| Select an exact published version | `rigscope upgrade --from rigscope@0.
|
|
304
|
+
| Select an exact published version | `rigscope upgrade --from rigscope@0.2.0` |
|
|
299
305
|
| Install a local archive | `rigscope upgrade --from ./rigscope-VERSION.tgz` |
|
|
300
306
|
| Return to the retained previous runtime | `rigscope rollback` |
|
|
301
307
|
|
package/docs/monitoring.md
CHANGED
|
@@ -20,22 +20,61 @@ driver and readable counters; see [Hardware](installation.md#hardware) and
|
|
|
20
20
|
|
|
21
21
|
## History and delivery
|
|
22
22
|
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
23
|
+
History is **RAM-only by default**. Live readings update immediately, recent
|
|
24
|
+
charts come from a bounded memory budget, and telemetry history disappears when
|
|
25
|
+
the hub restarts. Users, credentials, enrollment, settings, alert records,
|
|
26
|
+
machine names/tags and inventory remain persistent.
|
|
27
|
+
|
|
28
|
+
In memory mode the collector's acknowledgement means the hub accepted the sample
|
|
29
|
+
into RAM. The collector queues pending samples in RAM and retries after a
|
|
30
|
+
disconnect, so it gains no disk work from the hub's choice and writes no queue
|
|
31
|
+
of its own. Deduplication
|
|
32
|
+
uses a compact per-run interval set for the hub process lifetime. Because both
|
|
33
|
+
history and receipts disappear together on restart, a replay cannot double-count
|
|
34
|
+
restored chart data.
|
|
35
|
+
|
|
36
|
+
Set `HUB_TELEMETRY_MODE=persistent` to opt into disk history. The hub buffers a
|
|
37
|
+
five-second window, publishes live state immediately, writes one compact
|
|
38
|
+
transaction, and only then acknowledges. Replay deduplication uses durable
|
|
39
|
+
receipts whose lifetime is independent of chart retention (default seven days).
|
|
26
40
|
Replayed samples fill History without replacing current live readings or opening
|
|
27
|
-
alerts for past threshold crossings.
|
|
41
|
+
alerts for past threshold crossings. A graceful shutdown flushes pending
|
|
42
|
+
summaries; after a crash the collector replays anything the hub never
|
|
43
|
+
acknowledged, provided the collector itself kept it: only
|
|
44
|
+
`COLLECTOR_QUEUE_MODE=durable` survives a collector restart.
|
|
28
45
|
|
|
29
|
-
Default
|
|
46
|
+
Default memory history:
|
|
30
47
|
|
|
31
48
|
| Resolution | Kept for |
|
|
32
49
|
| --- | --- |
|
|
33
|
-
|
|
|
34
|
-
|
|
|
35
|
-
| One-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
50
|
+
| 1-second detail | 2 minutes |
|
|
51
|
+
| 10-second summaries | 1 hour |
|
|
52
|
+
| One-minute summaries | 24 hours |
|
|
53
|
+
| One-hour summaries | 7 days |
|
|
54
|
+
|
|
55
|
+
Per-core CPU keeps less detail; machine-wide and capacity metrics keep more.
|
|
56
|
+
Persistent mode writes compact ten-second summaries and folds them into
|
|
57
|
+
one-minute and one-hour tiers; it does not write a raw row per reading. It keeps
|
|
58
|
+
10-second summaries for 2 hours, one-minute summaries for 30 days and hourly
|
|
59
|
+
summaries for 400 days by default. Both modes retain minimum and maximum values,
|
|
60
|
+
so short spikes remain visible, and both preserve explicit unavailable readings
|
|
61
|
+
and gaps.
|
|
62
|
+
|
|
63
|
+
RAM history is bounded by `HUB_HISTORY_MEMORY_BYTES` (default 128 MiB) across all
|
|
64
|
+
machines. When the budget fills, the oldest fine-grained tiers are dropped first
|
|
65
|
+
and new cardinality beyond the sustainable budget is rejected with visible
|
|
66
|
+
pressure. History queries limit points, series, scanned rows and elapsed time;
|
|
67
|
+
the UI marks partial results.
|
|
68
|
+
|
|
69
|
+
### Recording
|
|
70
|
+
|
|
71
|
+
Use **History → Recording** to start an explicit capture for the current
|
|
72
|
+
machine and selected metrics. Choose a resolution from 1 second to 1 hour,
|
|
73
|
+
a duration and whether to include recent RAM history. The capture is bounded by
|
|
74
|
+
duration and bytes, survives restart through its private persisted output, and
|
|
75
|
+
can be downloaded by its creator. Recording also works when chart history is
|
|
76
|
+
temporary, which is the intended way to keep an incident snapshot. See
|
|
77
|
+
[telemetry storage](telemetry-storage.md#recording).
|
|
39
78
|
|
|
40
79
|
Hover over a History or Overview recent-usage chart to inspect a recorded time
|
|
41
80
|
and its bucket averages. The time marker follows across charts in the same view;
|
|
@@ -49,11 +88,12 @@ Each collector queue keeps up to ten minutes or 64 MiB of queued JSON per hub,
|
|
|
49
88
|
with a 256 MiB total JSON limit. Physical SQLite files, including WAL files,
|
|
50
89
|
have separate limits of 80 MiB per hub and 640 MiB across eight hubs.
|
|
51
90
|
|
|
52
|
-
|
|
53
|
-
Retention protects the newest hour
|
|
54
|
-
free enough space, the hub stops acknowledging new samples and collectors
|
|
55
|
-
Existing oversized databases report pressure; startup does not run a
|
|
56
|
-
cleanup.
|
|
91
|
+
In persistent mode the hub's default database limit is 2 GiB, including WAL and
|
|
92
|
+
all control tables. Retention protects the newest hour; if old-data cleanup
|
|
93
|
+
cannot free enough space, the hub stops acknowledging new samples and collectors
|
|
94
|
+
retry. Existing oversized databases report pressure; startup does not run a
|
|
95
|
+
destructive cleanup. In memory mode old historical rows may still occupy the
|
|
96
|
+
file until you run `ops reset-telemetry`; they are not read or updated.
|
|
57
97
|
|
|
58
98
|
These are application limits, not filesystem quotas. A transaction can briefly
|
|
59
99
|
grow the WAL before checkpointing. Blocked checkpoints stop further ingestion
|
|
@@ -92,7 +132,10 @@ credentials persist in SQLite. Browser sessions stay in memory, expire after
|
|
|
92
132
|
Alerts support above/below thresholds, durations, unavailable readings, missing
|
|
93
133
|
receipts, maintenance and recovery. Value, unavailable and stale-receipt rules
|
|
94
134
|
apply to the machines tracked by that hub. Replayed samples do not open past
|
|
95
|
-
incidents.
|
|
135
|
+
incidents. Alert rules, incidents and pending notifications stay durable in both
|
|
136
|
+
history modes; with no rules configured, live ingestion does not write per-sample
|
|
137
|
+
alert progress. Enabling rules may write alert state on evaluation, which is
|
|
138
|
+
operational state rather than telemetry history.
|
|
96
139
|
|
|
97
140
|
In **Alerts**, configure one HTTPS webhook channel. Its URL, headers and HMAC
|
|
98
141
|
secret are write-only. Loopback HTTP is allowed for controlled test sinks.
|
package/docs/operations.md
CHANGED
|
@@ -10,6 +10,7 @@ building and deploying from source.
|
|
|
10
10
|
| Reset a lost password | [Password recovery](#password-recovery) |
|
|
11
11
|
| Change localhost/HTTPS access | [Access changes](#access-changes) |
|
|
12
12
|
| Back up or restore a database | [Database commands](#database-commands) |
|
|
13
|
+
| Discard old telemetry history only | [Database commands](#database-commands) |
|
|
13
14
|
| Plan what to back up | [Consistent backup set](#consistent-backup-set) |
|
|
14
15
|
| Handle a full database or queue | [Storage pressure recovery](#storage-budgets-and-pressure-recovery) |
|
|
15
16
|
|
|
@@ -54,9 +55,26 @@ running, then checks:
|
|
|
54
55
|
The health timeout is 25 seconds; change it with `--timeout SECONDS`. A failed
|
|
55
56
|
start, migration or health check restores the previous runtime. It also restores
|
|
56
57
|
the paired database backup if the old runtime cannot read the new schema,
|
|
57
|
-
preserving the replaced database first
|
|
58
|
-
|
|
59
|
-
|
|
58
|
+
preserving the replaced database first — and additionally whenever the
|
|
59
|
+
one-time machine-data transition described below had already discarded rows,
|
|
60
|
+
even if the schema is compatible, because the loss there is the rows, not the
|
|
61
|
+
schema.
|
|
62
|
+
|
|
63
|
+
The first upgrade after the RAM-first release completes a one-time
|
|
64
|
+
machine-data transition: it discards the previous design's on-disk machine
|
|
65
|
+
history — telemetry rows and summaries, receipts, gaps, latest snapshots,
|
|
66
|
+
inventory and machine-derived alert history — plus pre-RAM-first durable
|
|
67
|
+
collector queues, once, even data previously opted into keeping. Machine
|
|
68
|
+
display names/tags (`machine_meta`), accounts, sessions, enrollment and grants,
|
|
69
|
+
settings, alert rules/jobs/webhook configuration and recordings are never
|
|
70
|
+
discarded. A version-independent `machine-data-transition.json` marker in the
|
|
71
|
+
state directory records completion; every later upgrade preserves durable
|
|
72
|
+
history and queues you opt back into (`COLLECTOR_QUEUE_MODE=durable`, durable
|
|
73
|
+
hub history). The upgrade output reports exactly what was discarded; a cleanup
|
|
74
|
+
that cannot finish is shown as a warning and retried by the next upgrade
|
|
75
|
+
instead of failing an otherwise healthy one. See
|
|
76
|
+
[telemetry storage](telemetry-storage.md#the-one-time-machine-data-transition).
|
|
77
|
+
Credentials and collector identities are always retained.
|
|
60
78
|
`release-state.json` records which backup belongs to each runtime. Upgrading a
|
|
61
79
|
stopped installation leaves it stopped and reports live checks as skipped.
|
|
62
80
|
|
|
@@ -66,7 +84,8 @@ release, stops the hub and collector completely, then removes
|
|
|
66
84
|
the old configuration and history and installing the latest release fresh. Back
|
|
67
85
|
up the state directory first if that history must be kept. There is no migration
|
|
68
86
|
to the canonical format and no rollback to that old shape. Future upgrades that
|
|
69
|
-
stay canonical keep accounts,
|
|
87
|
+
stay canonical keep accounts, settings and collector identities, and — apart
|
|
88
|
+
from the one-time transition above — durable history and queues.
|
|
70
89
|
|
|
71
90
|
### Roll back
|
|
72
91
|
|
|
@@ -191,7 +210,7 @@ The installer creates:
|
|
|
191
210
|
hub.env private EnvironmentFile (0600, no secrets printed)
|
|
192
211
|
collector.env private EnvironmentFile (0600)
|
|
193
212
|
collector/collector.json collector identity/enrollment (created by collector)
|
|
194
|
-
outbox/<sha256(url)[0:16]>/ per-destination
|
|
213
|
+
outbox/<sha256(url)[0:16]>/ per-destination queues, durable mode only
|
|
195
214
|
backups/ private pre-upgrade/pre-activation database copies
|
|
196
215
|
install-state.json canonical versions/schema backups, no credentials
|
|
197
216
|
run/ private runtime directory for a Unix socket
|
|
@@ -314,6 +333,9 @@ COLLECTOR_ENROLL_URL=ws://127.0.0.1:8081/api/collector/connect
|
|
|
314
333
|
COLLECTOR_ENROLL_TOKEN=<one-use grant>
|
|
315
334
|
COLLECTOR_MODE=real
|
|
316
335
|
COLLECTOR_STATE_FILE=/home/operator/.local/state/rigscope/collector/collector.json
|
|
336
|
+
# Optional: keep unacknowledged samples in a per-hub SQLite outbox so they
|
|
337
|
+
# survive a collector restart. Default is RAM (no queue written to disk).
|
|
338
|
+
# COLLECTOR_QUEUE_MODE=durable
|
|
317
339
|
```
|
|
318
340
|
|
|
319
341
|
Remove the one-use grant after enrollment and start or restart the collector. The
|
|
@@ -369,6 +391,11 @@ node dist/ops.cjs restore \
|
|
|
369
391
|
--database /path/telemetry.sqlite \
|
|
370
392
|
--replace --confirm-stopped
|
|
371
393
|
|
|
394
|
+
# Discard only telemetry history, preserving users/enrollment/settings/alerts.
|
|
395
|
+
node dist/ops.cjs reset-telemetry \
|
|
396
|
+
--database /path/telemetry.sqlite \
|
|
397
|
+
--backup-dir /path/private-backups --confirm-stopped
|
|
398
|
+
|
|
372
399
|
# Offline managed-user recovery; password comes from a private 0600 file.
|
|
373
400
|
node dist/ops.cjs recover-admin \
|
|
374
401
|
--database /path/telemetry.sqlite \
|
|
@@ -408,18 +435,24 @@ release must never be pointed at a newer schema. `inspect` reports this state.
|
|
|
408
435
|
|
|
409
436
|
The database backup is not the whole installation. A restorable set is:
|
|
410
437
|
|
|
411
|
-
1. Hub SQLite database: `telemetry.sqlite` (includes registry/admin/alerts
|
|
412
|
-
Use `ops backup`; do not copy the main file
|
|
438
|
+
1. Hub SQLite database: `telemetry.sqlite` (includes registry/admin/alerts and
|
|
439
|
+
any opt-in persistent history). Use `ops backup`; do not copy the main file
|
|
440
|
+
alone while the hub is running.
|
|
413
441
|
2. Enrollment/config state when configured: `HUB_STATE_FILE` (for
|
|
414
442
|
example `hub/enrollment.json`, the import source for previously enrolled
|
|
415
443
|
collectors) and the private hub environment file. These
|
|
416
444
|
contain credentials: protect and copy them only between private locations.
|
|
417
445
|
3. Collector identity/enrollment: `collector/collector.json`.
|
|
418
|
-
4. Collector
|
|
419
|
-
|
|
420
|
-
deduplicates by
|
|
446
|
+
4. Collector queues, `outbox/<destination-hash>/outbox.sqlite` per hub URL, exist
|
|
447
|
+
only under `COLLECTOR_QUEUE_MODE=durable` and may contain unacknowledged
|
|
448
|
+
telemetry. Replay is at-least-once; the hub deduplicates by
|
|
449
|
+
collector/run/sequence. The default RAM queue holds nothing on disk, so there
|
|
450
|
+
is nothing to back up and nothing to restore.
|
|
421
451
|
5. Optional operational metadata: `install-state.json`, units and the built
|
|
422
452
|
release. Keep the release matching the database schema for rollback.
|
|
453
|
+
6. Explicit recordings: the private `HUB_RECORDINGS_DIR` (default
|
|
454
|
+
`<hub database directory>/recordings`) if you want downloadable captures to
|
|
455
|
+
survive a restore. They are user output, not part of the database backup.
|
|
423
456
|
|
|
424
457
|
For a consistent set, prefer stopping both services cleanly, then run the online
|
|
425
458
|
database backup and copy the other private files. For an online backup, the SQLite
|
|
@@ -472,7 +505,10 @@ use the component controls in [System installation](system-installation.md#syste
|
|
|
472
505
|
The collector uses only `/api/collector/connect` and the canonical record shape.
|
|
473
506
|
There is no old-endpoint normalization, outbox migration or old-peer payload
|
|
474
507
|
conversion. Reset unfinished installations and re-enroll their collectors;
|
|
475
|
-
canonical upgrades retain their identities
|
|
508
|
+
canonical upgrades retain their identities and credentials. Pending samples are
|
|
509
|
+
retained by every upgrade after the one-time machine-data transition, which
|
|
510
|
+
discards pre-RAM-first durable queues once; from then on
|
|
511
|
+
`COLLECTOR_QUEUE_MODE=durable` content survives upgrades.
|
|
476
512
|
|
|
477
513
|
## Measurement
|
|
478
514
|
|
|
@@ -486,7 +522,18 @@ node dist/ops.cjs measure --machines 2 --batches 30 --records-per-batch 5 \
|
|
|
486
522
|
It writes synthetic records through the real `TelemetryStore` into a
|
|
487
523
|
temporary private database and reports accepted/rejected/busy counts, ingestion
|
|
488
524
|
latency, event-loop delay/utilization, RSS, database/WAL size, store queue and
|
|
489
|
-
flush counters, and one history-query latency.
|
|
525
|
+
flush counters, and one history-query latency.
|
|
526
|
+
|
|
527
|
+
For the history-mode tradeoff, use the repeatable RAM-vs-persistent benchmark.
|
|
528
|
+
It measures process-attributed disk writes, latency, RSS and rows:
|
|
529
|
+
|
|
530
|
+
```sh
|
|
531
|
+
npm run benchmark:telemetry -- --machines 2 --seconds 120
|
|
532
|
+
npm run benchmark:telemetry -- --machines 10 --seconds 60 --json
|
|
533
|
+
```
|
|
534
|
+
|
|
535
|
+
Use `--dir` on a real filesystem; tmpfs does not account storage writes in
|
|
536
|
+
`/proc/self/io`. See [telemetry storage](telemetry-storage.md#benchmark). The output is labelled synthetic.
|
|
490
537
|
Defaults are intentionally small; upper bounds are 1,000 machines, 2,000 batches,
|
|
491
538
|
32 records per batch and 100,000 records total. It never opens a service socket
|
|
492
539
|
or contacts a hub. `--keep` retains the temporary directory for inspection. The
|
|
@@ -528,20 +575,29 @@ absolute regular file; use the release's `node_modules` installed with
|
|
|
528
575
|
## Storage budgets and pressure recovery
|
|
529
576
|
|
|
530
577
|
Each Collector Hub connection owns policy filtering, cadence, inventory and its
|
|
531
|
-
|
|
532
|
-
current sharing policy before the connection starts;
|
|
533
|
-
reapplied to retained records. Endpoint upgrade
|
|
534
|
-
continue to use the existing
|
|
578
|
+
delivery queue submission. Under `COLLECTOR_QUEUE_MODE=durable`, recovered metrics
|
|
579
|
+
are sanitized with the current sharing policy before the connection starts;
|
|
580
|
+
capture-time cadence is not reapplied to retained records. Endpoint upgrade
|
|
581
|
+
recovery and aggregate budgets continue to use the existing queues. The default
|
|
582
|
+
RAM queue starts empty, so its bounds are enforced against RAM rather than disk. Hub startup restores stored telemetry,
|
|
535
583
|
cadence, inventory and metadata through Sample ingestion while retaining a
|
|
536
584
|
disconnected state until an authenticated Collector reconnects.
|
|
537
585
|
|
|
538
586
|
The default hub main-file plus WAL budget is 2 GiB, shared by every SQLite table.
|
|
539
|
-
The main file has a page-count limit; WAL reserves up to 128 MiB inside that budget
|
|
540
|
-
|
|
541
|
-
|
|
542
|
-
|
|
543
|
-
|
|
544
|
-
|
|
587
|
+
The main file has a page-count limit; WAL reserves up to 128 MiB inside that budget
|
|
588
|
+
and automatic checkpointing waits for a meaningful 32–64 MiB reserve where the
|
|
589
|
+
budget permits. Persistent-mode writes return busy on pressure or SQLITE_FULL,
|
|
590
|
+
with no receipt/ack for a failed transaction. Retention reclaims old raw or
|
|
591
|
+
ten-second points and one-minute rollups in bounded chunks while preserving the
|
|
592
|
+
newest hour; hourly rollups retain their configured age window. A full
|
|
593
|
+
newest-hour working set must fit the deployment's budget. Outboxes retry while
|
|
594
|
+
the operator frees space or provisions a larger supported configuration.
|
|
595
|
+
|
|
596
|
+
Memory mode keeps only a bounded RAM budget (`HUB_HISTORY_MEMORY_BYTES`, default
|
|
597
|
+
128 MiB). It does not write routine telemetry rows, but existing historical rows
|
|
598
|
+
from an older installation still occupy the database until removed with
|
|
599
|
+
`ops reset-telemetry` while the hub is stopped. That command preserves users,
|
|
600
|
+
credentials, enrollment, settings, alert state, metadata and inventory.
|
|
545
601
|
|
|
546
602
|
Collector defaults are 64 MiB logical JSON and 80 MiB physical main+WAL per hub,
|
|
547
603
|
256 MiB logical and 640 MiB physical across eight destinations. Physical pressure
|
|
@@ -31,7 +31,8 @@ The installed `rigscope system` command and `dist/system.cjs` use the same code.
|
|
|
31
31
|
hub/enrollment.json rigscope-hub 0600
|
|
32
32
|
hub/run/hub.sock private Unix socket (remote mode)
|
|
33
33
|
collector/state.json rigscope-collector 0600
|
|
34
|
-
collector/outbox/ durable
|
|
34
|
+
collector/outbox/ collector queues, durable mode only
|
|
35
|
+
machine-data-transition.json root:root 0600, one-time machine-data transition marker
|
|
35
36
|
backups/ root:root 0700, pre-upgrade/pre-rollback SQLite copies
|
|
36
37
|
system-install.json root:root 0600, canonical component state/versions/backups/adoption
|
|
37
38
|
power-access.json root:root 0600, original counter modes for reversal
|
|
@@ -104,8 +105,14 @@ A pre-canonical system installation is reset, not migrated. `system install` and
|
|
|
104
105
|
units completely, then remove the old configuration and history and install
|
|
105
106
|
the canonical layout fresh. Back up `/var/lib/rigscope` first if that history
|
|
106
107
|
must be kept; there is no rollback to the old format. Later upgrades that stay
|
|
107
|
-
canonical keep accounts,
|
|
108
|
-
|
|
108
|
+
canonical keep accounts, settings and collector identities and rollback keeps
|
|
109
|
+
its paired database backup. The first such upgrade after the RAM-first release
|
|
110
|
+
also completes the one-time machine-data transition, discarding machine-derived
|
|
111
|
+
history and pre-RAM-first durable collector queues once — even data previously
|
|
112
|
+
opted into keeping — while preserving machine names/tags; a
|
|
113
|
+
`machine-data-transition.json` marker in `/var/lib/rigscope` makes every
|
|
114
|
+
subsequent upgrade preserve durable history and queues. See
|
|
115
|
+
[telemetry storage](telemetry-storage.md#the-one-time-machine-data-transition).
|
|
109
116
|
|
|
110
117
|
- With `--enable`, `--role local` initially enables both components.
|
|
111
118
|
- With `--enable`, `--role hub` initially enables only the hub.
|
|
@@ -131,7 +138,7 @@ identity and RAPL grant. With no destination it tries the colocated hub:
|
|
|
131
138
|
cleanly before collection until a destination is saved.
|
|
132
139
|
|
|
133
140
|
Configured remote destinations and credentials are never overwritten. Re-running
|
|
134
|
-
`enable collector` leaves its destination, token
|
|
141
|
+
`enable collector` leaves its destination, token and identity intact.
|
|
135
142
|
|
|
136
143
|
### Connecting to an independent hub
|
|
137
144
|
|