@ouro.bot/cli 0.1.0-alpha.741 → 0.1.0-alpha.743
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/changelog.json +42 -0
- package/deploy/unraid/Dockerfile +7 -2
- package/deploy/unraid/README.txt +717 -106
- package/deploy/unraid/migrate-sanctuary-bundle.mjs +21 -0
- package/deploy/unraid/ouro-events/bootstrap-spool.sh +92 -0
- package/deploy/unraid/ouro-events/emit-event.mjs +310 -0
- package/deploy/unraid/ouro-events/emit-usenet-event.sh +54 -0
- package/deploy/unraid/ouro-events/install-usenet-guard.sh +434 -0
- package/deploy/unraid/ouro-events/usenet-health.sh +178 -0
- package/deploy/unraid/sanctuary-acceptance-contract.json +4 -3
- package/deploy/unraid/sanctuary-unit16-host-broker.mjs +5 -1
- package/deploy/unraid/sanctuary.ouro/agent.json +2 -2
- package/deploy/unraid/sanctuary.ouro/bundle-meta.json +2 -2
- package/deploy/unraid/sanctuary.ouro/habits/sanctuary-health.md +1 -1
- package/deploy/unraid/sanctuary.ouro/provider-readiness.json +3 -10
- package/deploy/unraid/sanctuary.ouro/psyche/ASPIRATIONS.md +3 -1
- package/deploy/unraid/sanctuary.ouro/psyche/IDENTITY.md +3 -1
- package/deploy/unraid/sanctuary.ouro/psyche/LORE.md +4 -0
- package/deploy/unraid/sanctuary.ouro/psyche/SOUL.md +6 -2
- package/deploy/unraid/sanctuary.ouro/psyche/TACIT.md +4 -2
- package/deploy/unraid/sanctuary.ouro/state/policy/steward.json +7 -0
- package/deploy/unraid/sanctuary.ouro/tool-profiles.json +19 -17
- package/deploy/unraid/sanctuary.xml +12 -9
- package/dist/arc/cares.js +171 -37
- package/dist/arc/obligations.js +38 -0
- package/dist/heart/awaiting/await-alert.js +4 -2
- package/dist/heart/awaiting/await-expiry.js +39 -4
- package/dist/heart/awaiting/await-parser.js +8 -0
- package/dist/heart/context-loss-gauntlet.js +31 -2
- package/dist/heart/context-loss-sentinel.js +2 -0
- package/dist/heart/core.js +37 -7
- package/dist/heart/cross-chat-delivery.js +1 -0
- package/dist/heart/daemon/container-healthcheck.js +3 -1
- package/dist/heart/daemon/container-runtime.js +13 -0
- package/dist/heart/daemon/container-spec-auditor-main.js +11 -7
- package/dist/heart/daemon/container-spec-auditor.js +38 -11
- package/dist/heart/daemon/daemon-entry.js +19 -5
- package/dist/heart/daemon/daemon.js +154 -22
- package/dist/heart/daemon/sanctuary-acceptance-adapter.js +151 -87
- package/dist/heart/daemon/sanctuary-acceptance-harness.js +26 -29
- package/dist/heart/daemon/sanctuary-acceptance-scenarios.js +42 -51
- package/dist/heart/daemon/sanctuary-bundle-migration.js +398 -0
- package/dist/heart/daemon/sanctuary-scheduler-origin.js +11 -5
- package/dist/heart/daemon/supercronic-supervisor.js +16 -1
- package/dist/heart/external-events/router.js +1025 -43
- package/dist/heart/provider-models.js +10 -3
- package/dist/heart/session-events.js +23 -5
- package/dist/heart/steward-policy.js +302 -0
- package/dist/heart/tool-approval.js +3 -1
- package/dist/heart/turn-context.js +1 -0
- package/dist/mind/context.js +8 -3
- package/dist/mind/pending.js +25 -0
- package/dist/mind/prompt.js +49 -1
- package/dist/repertoire/guardrails.js +2 -1
- package/dist/repertoire/relationship-authorization.js +194 -0
- package/dist/repertoire/tools-awaiting.js +229 -26
- package/dist/repertoire/tools-base.js +15 -0
- package/dist/repertoire/tools-continuity.js +259 -4
- package/dist/repertoire/tools-notes.js +29 -3
- package/dist/repertoire/tools-session.js +219 -4
- package/dist/repertoire/tools-steward-policy.js +88 -0
- package/dist/repertoire/tools-telegram-contacts.js +37 -0
- package/dist/repertoire/tools-unraid.js +113 -6
- package/dist/repertoire/tools.js +104 -9
- package/dist/repertoire/unraid-restart.js +86 -4
- package/dist/senses/private-runtime-worker.js +36 -3
- package/dist/senses/private-runtime.js +310 -63
- package/dist/senses/sanctuary-health-acceptance-probe.js +8 -6
- package/dist/senses/sanctuary-health-runner.js +51 -96
- package/dist/senses/sanctuary-health.js +54 -93
- package/dist/senses/sanctuary-runtime.js +36 -2
- package/dist/senses/sanctuary-sab.js +80 -0
- package/dist/senses/shared-turn.js +64 -9
- package/dist/senses/telegram-admission.js +684 -0
- package/dist/senses/telegram-approval-runtime.js +46 -6
- package/dist/senses/telegram-client.js +236 -72
- package/dist/senses/telegram-effect-adapter.js +787 -0
- package/dist/senses/telegram-entry.js +4 -1
- package/dist/senses/telegram.js +728 -48
- package/npm-shrinkwrap.json +6 -6
- package/package.json +2 -2
package/deploy/unraid/README.txt
CHANGED
|
@@ -2,8 +2,9 @@ Mendelow Cloud Butler operator runbook
|
|
|
2
2
|
|
|
3
3
|
The production container is ouro-butler. It runs as UID/GID 10001, publishes no
|
|
4
4
|
ports, uses host networking only so its loopback-only Unraid GraphQL client can
|
|
5
|
-
reach 127.0.0.1, mounts
|
|
6
|
-
the
|
|
5
|
+
reach 127.0.0.1, mounts the runtime and sanctuary.ouro bundle read-write plus
|
|
6
|
+
the privileged event spool and SAB configuration read-only, and uses restart
|
|
7
|
+
policy unless-stopped.
|
|
7
8
|
|
|
8
9
|
Start/stop:
|
|
9
10
|
docker start ouro-butler
|
|
@@ -14,15 +15,25 @@ Status and health:
|
|
|
14
15
|
docker logs --tail 200 ouro-butler
|
|
15
16
|
|
|
16
17
|
Effective-spec audit helper:
|
|
17
|
-
Run Update and Restore command sequences from a root shell with `set -eu`.
|
|
18
|
-
Before
|
|
18
|
+
Run Update, Backup, and Restore command sequences from a root shell with `set -eu`.
|
|
19
|
+
Before any sequence, define this helper. It captures one actual container
|
|
19
20
|
inspect and its reviewed image inspect without printing either, invokes the
|
|
20
|
-
packaged auditor from
|
|
21
|
-
or shell exit:
|
|
21
|
+
packaged auditor from the explicit validated runner image argument, and
|
|
22
|
+
removes its mode-0600 inputs on success or shell exit:
|
|
22
23
|
audit_effective() {
|
|
23
24
|
(
|
|
24
25
|
AUDIT_CONTAINER=$1
|
|
25
26
|
AUDIT_EXPECTED_IMAGE=$2
|
|
27
|
+
AUDIT_RUNNER_IMAGE_ID=$3
|
|
28
|
+
AUDIT_MOUNT_CONTRACT=${4-canonical}
|
|
29
|
+
validate_exact_image_id "$AUDIT_EXPECTED_IMAGE" || return $?
|
|
30
|
+
validate_exact_image_id "$AUDIT_RUNNER_IMAGE_ID" || return $?
|
|
31
|
+
test "$AUDIT_RUNNER_IMAGE_ID" != sha256:681449ad47a2621705cd339b481e6339236b31dc65e195b1cf5025d0f2191d7d || return $?
|
|
32
|
+
case "$AUDIT_MOUNT_CONTRACT" in
|
|
33
|
+
canonical) set -- ;;
|
|
34
|
+
legacy-alpha742) set -- --mount-contract legacy-alpha742 ;;
|
|
35
|
+
*) return 1 ;;
|
|
36
|
+
esac
|
|
26
37
|
INSPECT_DIR=$(mktemp -d /mnt/user/appdata/ouro-butler/staging/inspect.XXXXXX) || return $?
|
|
27
38
|
trap 'rm -f -- "$INSPECT_DIR/container.json" "$INSPECT_DIR/image.json"; rmdir -- "$INSPECT_DIR"' EXIT || return $?
|
|
28
39
|
chmod 0700 "$INSPECT_DIR" || return $?
|
|
@@ -34,7 +45,7 @@ Effective-spec audit helper:
|
|
|
34
45
|
--entrypoint /opt/ouro/deploy/unraid/audit-container-spec.sh \
|
|
35
46
|
--mount "type=bind,src=$INSPECT_DIR/container.json,dst=/audit/container.json,readonly" \
|
|
36
47
|
--mount "type=bind,src=$INSPECT_DIR/image.json,dst=/audit/image.json,readonly" \
|
|
37
|
-
"$
|
|
48
|
+
"$AUDIT_RUNNER_IMAGE_ID" --inspect /audit/container.json --image-inspect /audit/image.json --expected-image "$AUDIT_EXPECTED_IMAGE" "$@" || return $?
|
|
38
49
|
rm -f -- "$INSPECT_DIR/container.json" "$INSPECT_DIR/image.json" || return $?
|
|
39
50
|
rmdir -- "$INSPECT_DIR" || return $?
|
|
40
51
|
trap - EXIT || return $?
|
|
@@ -72,18 +83,23 @@ Effective-spec audit helper:
|
|
|
72
83
|
http://127.0.0.1/plugins/dynamix.docker.manager/include/UpdateConfig.php || return $?
|
|
73
84
|
)
|
|
74
85
|
}
|
|
75
|
-
|
|
86
|
+
butler_autostart_counts() {
|
|
76
87
|
(
|
|
77
|
-
EXPECTED_AUTOSTART_COUNTS=$1
|
|
78
88
|
AUTOSTART_FILE=/var/lib/docker/unraid-autostart
|
|
79
89
|
test -f "$AUTOSTART_FILE" || return $?
|
|
80
|
-
|
|
90
|
+
awk '
|
|
81
91
|
$1 == "ouro-butler" { production++ }
|
|
82
92
|
$1 == "ouro-butler-staging" { staging++ }
|
|
83
93
|
$1 == "ouro-butler-rollback" { rollback++ }
|
|
84
94
|
$1 == "ouro-butler-legacy-evidence" { legacy++ }
|
|
85
95
|
END { printf "%d %d %d %d", production + 0, staging + 0, rollback + 0, legacy + 0 }
|
|
86
|
-
' "$AUTOSTART_FILE"
|
|
96
|
+
' "$AUTOSTART_FILE"
|
|
97
|
+
)
|
|
98
|
+
}
|
|
99
|
+
verify_butler_autostart() {
|
|
100
|
+
(
|
|
101
|
+
EXPECTED_AUTOSTART_COUNTS=$1
|
|
102
|
+
AUTOSTART_COUNTS=$(butler_autostart_counts) || return $?
|
|
87
103
|
test "$AUTOSTART_COUNTS" = "$EXPECTED_AUTOSTART_COUNTS" || return $?
|
|
88
104
|
)
|
|
89
105
|
}
|
|
@@ -189,6 +205,21 @@ Effective-spec audit helper:
|
|
|
189
205
|
esac
|
|
190
206
|
assert_only_running_butler ouro-butler || return $?
|
|
191
207
|
}
|
|
208
|
+
assert_legacy_alpha742_source() {
|
|
209
|
+
EXPECTED_SOURCE_IMAGE_ID=$1
|
|
210
|
+
AUDIT_RUNNER_IMAGE_ID=$2
|
|
211
|
+
test "$EXPECTED_SOURCE_IMAGE_ID" = sha256:681449ad47a2621705cd339b481e6339236b31dc65e195b1cf5025d0f2191d7d || return $?
|
|
212
|
+
audit_effective ouro-butler "$EXPECTED_SOURCE_IMAGE_ID" "$AUDIT_RUNNER_IMAGE_ID" legacy-alpha742
|
|
213
|
+
}
|
|
214
|
+
assert_update_source() {
|
|
215
|
+
EXPECTED_SOURCE_IMAGE_ID=$1
|
|
216
|
+
AUDIT_RUNNER_IMAGE_ID=$2
|
|
217
|
+
if test "$EXPECTED_SOURCE_IMAGE_ID" = sha256:681449ad47a2621705cd339b481e6339236b31dc65e195b1cf5025d0f2191d7d; then
|
|
218
|
+
assert_legacy_alpha742_source "$EXPECTED_SOURCE_IMAGE_ID" "$AUDIT_RUNNER_IMAGE_ID"
|
|
219
|
+
else
|
|
220
|
+
audit_effective ouro-butler "$EXPECTED_SOURCE_IMAGE_ID" "$AUDIT_RUNNER_IMAGE_ID"
|
|
221
|
+
fi
|
|
222
|
+
}
|
|
192
223
|
validate_sanctuary_roots() {
|
|
193
224
|
(
|
|
194
225
|
VALIDATE_RUNTIME_ROOT=$1
|
|
@@ -197,6 +228,8 @@ Effective-spec audit helper:
|
|
|
197
228
|
test -d "$VALIDATE_AGENT_ROOT" || return $?
|
|
198
229
|
VALIDATE_BAD_SHAPE=$(find "$VALIDATE_RUNTIME_ROOT" "$VALIDATE_AGENT_ROOT" -xdev \( -type l -o \( ! -type d -a ! -type f \) \) -print -quit) || return $?
|
|
199
230
|
test -z "$VALIDATE_BAD_SHAPE" || return $?
|
|
231
|
+
VALIDATE_UNEXPECTED_SOCKET=$(find "$VALIDATE_RUNTIME_ROOT" "$VALIDATE_AGENT_ROOT" -xdev -type s -print -quit) || return $?
|
|
232
|
+
test -z "$VALIDATE_UNEXPECTED_SOCKET" || return $?
|
|
200
233
|
for VALIDATE_REQUIRED_FILE in \
|
|
201
234
|
agent.json bundle-meta.json provider-readiness.json tool-profiles.json \
|
|
202
235
|
psyche/SOUL.md habits/sanctuary-health.md; do
|
|
@@ -207,18 +240,142 @@ Effective-spec audit helper:
|
|
|
207
240
|
test "$(printf '%s\n' "$VALIDATE_UNLOCK_FILES" | awk 'NF { count++ } END { print count + 0 }')" = 1 || return $?
|
|
208
241
|
test ! -e "$VALIDATE_RUNTIME_ROOT/container-credentials.json" || return $?
|
|
209
242
|
test ! -e "$VALIDATE_RUNTIME_ROOT/container-credentials.json.consuming" || return $?
|
|
243
|
+
test ! -S "$VALIDATE_AGENT_ROOT/state/acceptance/telegram-control.sock" || return $?
|
|
210
244
|
VALIDATE_WRONG_OWNER=$(find "$VALIDATE_RUNTIME_ROOT" "$VALIDATE_AGENT_ROOT" -xdev \( ! -user 10001 -o ! -group 10001 \) -print -quit) || return $?
|
|
211
245
|
test -z "$VALIDATE_WRONG_OWNER" || return $?
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
246
|
+
VALIDATE_WRONG_RUNTIME_DIR_MODE=$(find "$VALIDATE_RUNTIME_ROOT" -xdev -type d ! -perm 0700 \
|
|
247
|
+
! \( -perm 0755 \( \
|
|
248
|
+
-path "$VALIDATE_RUNTIME_ROOT/scheduler" -o -path "$VALIDATE_RUNTIME_ROOT/scheduler/*" -o \
|
|
249
|
+
-path "$VALIDATE_RUNTIME_ROOT/daemon/logs" -o -path "$VALIDATE_RUNTIME_ROOT/daemon/logs/*" \
|
|
250
|
+
\) \) -print -quit) || return $?
|
|
251
|
+
test -z "$VALIDATE_WRONG_RUNTIME_DIR_MODE" || return $?
|
|
252
|
+
VALIDATE_WRONG_AGENT_DIR_MODE=$(find "$VALIDATE_AGENT_ROOT" -xdev -type d ! -perm 0700 \
|
|
253
|
+
! \( -perm 0755 \( \
|
|
254
|
+
-path "$VALIDATE_AGENT_ROOT/arc/flight-recorder" -o -path "$VALIDATE_AGENT_ROOT/arc/flight-recorder/*" -o \
|
|
255
|
+
-path "$VALIDATE_AGENT_ROOT/state/health" -o -path "$VALIDATE_AGENT_ROOT/state/health/*" -o \
|
|
256
|
+
-path "$VALIDATE_AGENT_ROOT/state/logs" -o -path "$VALIDATE_AGENT_ROOT/state/logs/*" -o \
|
|
257
|
+
-path "$VALIDATE_AGENT_ROOT/state/habits" -o -path "$VALIDATE_AGENT_ROOT/state/habits/*" \
|
|
258
|
+
\) \) -print -quit) || return $?
|
|
259
|
+
test -z "$VALIDATE_WRONG_AGENT_DIR_MODE" || return $?
|
|
260
|
+
VALIDATE_WRONG_RUNTIME_FILE_MODE=$(find "$VALIDATE_RUNTIME_ROOT" -xdev -type f ! -perm 0600 \
|
|
261
|
+
! \( -perm 0644 \( \
|
|
262
|
+
-path "$VALIDATE_RUNTIME_ROOT/daemon/logs/*" -o \
|
|
263
|
+
-path "$VALIDATE_RUNTIME_ROOT/scheduler/*" -o \
|
|
264
|
+
-path "$VALIDATE_RUNTIME_ROOT/pulse.json" -o \
|
|
265
|
+
-path "$VALIDATE_RUNTIME_ROOT/pulse-delivered.json" -o \
|
|
266
|
+
-path "$VALIDATE_RUNTIME_ROOT/daemon.pids" -o \
|
|
267
|
+
-path "$VALIDATE_RUNTIME_ROOT/daemon-health.json" \
|
|
268
|
+
\) \) -print -quit) || return $?
|
|
269
|
+
test -z "$VALIDATE_WRONG_RUNTIME_FILE_MODE" || return $?
|
|
270
|
+
VALIDATE_WRONG_AGENT_FILE_MODE=$(find "$VALIDATE_AGENT_ROOT" -xdev -type f ! -perm 0600 \
|
|
271
|
+
! \( -perm 0644 \( \
|
|
272
|
+
-path "$VALIDATE_AGENT_ROOT/arc/flight-recorder/*" -o \
|
|
273
|
+
-path "$VALIDATE_AGENT_ROOT/state/health/*" -o \
|
|
274
|
+
-path "$VALIDATE_AGENT_ROOT/state/logs/*" -o \
|
|
275
|
+
-path "$VALIDATE_AGENT_ROOT/state/habits/*" \
|
|
276
|
+
\) \) -print -quit) || return $?
|
|
277
|
+
test -z "$VALIDATE_WRONG_AGENT_FILE_MODE" || return $?
|
|
278
|
+
)
|
|
279
|
+
}
|
|
280
|
+
verify_sanctuary_snapshot_provenance() {
|
|
281
|
+
(
|
|
282
|
+
SNAPSHOT_ROOT=$1
|
|
283
|
+
EXPECTED_SNAPSHOT_IMAGE_ID=$2
|
|
284
|
+
SNAPSHOT_PROVENANCE_ROOT=$SNAPSHOT_ROOT/provenance
|
|
285
|
+
for SNAPSHOT_TOP_LEVEL_DIRECTORY in runtime agent host provenance; do
|
|
286
|
+
test -d "$SNAPSHOT_ROOT/$SNAPSHOT_TOP_LEVEL_DIRECTORY" || return $?
|
|
287
|
+
test ! -L "$SNAPSHOT_ROOT/$SNAPSHOT_TOP_LEVEL_DIRECTORY" || return 1
|
|
288
|
+
done
|
|
289
|
+
test "$(find "$SNAPSHOT_ROOT" -xdev -mindepth 1 -maxdepth 1 -print | wc -l)" -eq 4 || return $?
|
|
290
|
+
test -d "$SNAPSHOT_PROVENANCE_ROOT" || return $?
|
|
291
|
+
test ! -L "$SNAPSHOT_PROVENANCE_ROOT" || return 1
|
|
292
|
+
test "$(stat -c '%u:%g:%a' "$SNAPSHOT_PROVENANCE_ROOT")" = 0:0:700 || return $?
|
|
293
|
+
SNAPSHOT_BAD_PROVENANCE_ENTRY=$(find "$SNAPSHOT_PROVENANCE_ROOT" -xdev -mindepth 1 ! -type f -print -quit) || return $?
|
|
294
|
+
test -z "$SNAPSHOT_BAD_PROVENANCE_ENTRY" || return $?
|
|
295
|
+
for SNAPSHOT_PROVENANCE_FILE in image-id container-inspect.json image-inspect.json package-version manifest.sha256; do
|
|
296
|
+
test -f "$SNAPSHOT_PROVENANCE_ROOT/$SNAPSHOT_PROVENANCE_FILE" || return $?
|
|
297
|
+
test ! -L "$SNAPSHOT_PROVENANCE_ROOT/$SNAPSHOT_PROVENANCE_FILE" || return 1
|
|
298
|
+
test "$(stat -c '%u:%g:%a' "$SNAPSHOT_PROVENANCE_ROOT/$SNAPSHOT_PROVENANCE_FILE")" = 0:0:600 || return $?
|
|
299
|
+
done
|
|
300
|
+
test "$(find "$SNAPSHOT_PROVENANCE_ROOT" -xdev -mindepth 1 -maxdepth 1 -type f | wc -l)" -eq 5 || return $?
|
|
301
|
+
test "$(wc -l <"$SNAPSHOT_PROVENANCE_ROOT/image-id")" -eq 1 || return $?
|
|
302
|
+
IFS= read -r SNAPSHOT_IMAGE_ID <"$SNAPSHOT_PROVENANCE_ROOT/image-id" || return $?
|
|
303
|
+
test "$SNAPSHOT_IMAGE_ID" = "$EXPECTED_SNAPSHOT_IMAGE_ID" || return $?
|
|
304
|
+
case "$SNAPSHOT_IMAGE_ID" in sha256:????????????????????????????????????????????????????????????????) ;; *) return 1 ;; esac
|
|
305
|
+
case "${SNAPSHOT_IMAGE_ID#sha256:}" in *[!0-9a-f]*) return 1 ;; esac
|
|
306
|
+
/usr/local/bin/node -e '
|
|
307
|
+
const fs = require("node:fs");
|
|
308
|
+
const [containerPath, imagePath, imageId, hostPath] = process.argv.slice(1);
|
|
309
|
+
const container = JSON.parse(fs.readFileSync(containerPath, "utf8"));
|
|
310
|
+
const image = JSON.parse(fs.readFileSync(imagePath, "utf8"));
|
|
311
|
+
if (!Array.isArray(container) || container.length !== 1 || !container[0] || container[0].Image !== imageId) process.exit(1);
|
|
312
|
+
if (!Array.isArray(image) || image.length !== 1 || !image[0] || image[0].Id !== imageId) process.exit(1);
|
|
313
|
+
if (image[0].Config?.Labels?.["org.opencontainers.image.source"] !== "https://github.com/ourostack/ouroboros") process.exit(1);
|
|
314
|
+
const expected = ["custom/usenet_health.sh", "custom/ouro-events/bootstrap-spool.sh", "custom/ouro-events/emit-event.mjs", "custom/ouro-events/emit-usenet-event.sh", "custom/ouro-events/install-usenet-guard.sh", "go.butler-lines", "crontab.butler-lines", "global-state"];
|
|
315
|
+
const expectedUid = 0;
|
|
316
|
+
const expectedGid = 0;
|
|
317
|
+
const hostRoot = fs.lstatSync(hostPath);
|
|
318
|
+
if (!hostRoot.isDirectory() || hostRoot.isSymbolicLink() || hostRoot.uid !== expectedUid || hostRoot.gid !== expectedGid || (hostRoot.mode & 0o777) !== 0o700) process.exit(1);
|
|
319
|
+
const rows = fs.readFileSync(`${hostPath}/inventory`, "utf8").trim().split("\n").map(line => line.split("\t"));
|
|
320
|
+
if (rows.length !== expected.length || rows.some((row, index) => row.length !== 2 || row[1] !== expected[index] || !["present", "absent"].includes(row[0]))) process.exit(1);
|
|
321
|
+
const present = rows.filter(row => row[0] === "present").map(row => row[1]);
|
|
322
|
+
if (!["go.butler-lines", "crontab.butler-lines", "global-state"].every(relative => present.includes(relative))) process.exit(1);
|
|
323
|
+
const globalRows = fs.readFileSync(`${hostPath}/global-state`, "utf8").trim().split("\n").map(line => line.split("\t"));
|
|
324
|
+
if (globalRows.length !== 2 || globalRows[0][0] !== "go" || globalRows[1][0] !== "crontab") process.exit(1);
|
|
325
|
+
for (const [name, state, digest, count] of globalRows) {
|
|
326
|
+
if (!["go", "crontab"].includes(name) || !["present", "absent"].includes(state) || !/^(?:[0-9a-f]{64}|-)$/.test(digest) || !/^(?:0|[1-9][0-9]*)$/.test(count)) process.exit(1);
|
|
327
|
+
if ((state === "absent") !== (digest === "-")) process.exit(1);
|
|
328
|
+
const fragmentLines = fs.readFileSync(`${hostPath}/${name}.butler-lines`, "utf8").split("\n").filter(Boolean);
|
|
329
|
+
if (fragmentLines.length !== Number(count)) process.exit(1);
|
|
330
|
+
}
|
|
331
|
+
for (const [state, relative] of rows) {
|
|
332
|
+
if (fs.existsSync(`${hostPath}/${relative}`) !== (state === "present")) process.exit(1);
|
|
333
|
+
}
|
|
334
|
+
const files = [];
|
|
335
|
+
const directories = [];
|
|
336
|
+
const walk = (directory, prefix = "") => {
|
|
337
|
+
for (const entry of fs.readdirSync(directory, { withFileTypes: true })) {
|
|
338
|
+
const relative = prefix ? `${prefix}/${entry.name}` : entry.name;
|
|
339
|
+
const metadata = fs.lstatSync(`${directory}/${entry.name}`);
|
|
340
|
+
if (metadata.uid !== expectedUid || metadata.gid !== expectedGid) process.exit(1);
|
|
341
|
+
if (entry.isDirectory() && (metadata.mode & 0o777) === 0o700) { directories.push(relative); walk(`${directory}/${entry.name}`, relative); }
|
|
342
|
+
else if (entry.isFile() && (metadata.mode & 0o777) === 0o600) files.push(relative);
|
|
343
|
+
else process.exit(1);
|
|
344
|
+
}
|
|
345
|
+
};
|
|
346
|
+
walk(hostPath);
|
|
347
|
+
if (JSON.stringify(directories.sort()) !== JSON.stringify(["custom", "custom/ouro-events"])) process.exit(1);
|
|
348
|
+
if (JSON.stringify(files.sort()) !== JSON.stringify(["inventory", ...present].sort())) process.exit(1);
|
|
349
|
+
' "$SNAPSHOT_PROVENANCE_ROOT/container-inspect.json" "$SNAPSHOT_PROVENANCE_ROOT/image-inspect.json" "$SNAPSHOT_IMAGE_ID" "$SNAPSHOT_ROOT/host" || return $?
|
|
350
|
+
test "$(wc -l <"$SNAPSHOT_PROVENANCE_ROOT/package-version")" -eq 1 || return $?
|
|
351
|
+
grep -Eq '^[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?$' "$SNAPSHOT_PROVENANCE_ROOT/package-version" || return $?
|
|
352
|
+
test "$(stat -c '%u:%g:%a' "$SNAPSHOT_ROOT/host")" = 0:0:700 || return $?
|
|
353
|
+
SNAPSHOT_BAD_HOST_ENTRY=$(find "$SNAPSHOT_ROOT/host" -xdev \( \( -type d ! -perm 0700 \) -o \( -type f ! -perm 0600 \) -o \( ! -type d -a ! -type f \) \) -print -quit) || return $?
|
|
354
|
+
test -z "$SNAPSHOT_BAD_HOST_ENTRY" || return $?
|
|
355
|
+
test -f "$SNAPSHOT_ROOT/host/inventory" || return $?
|
|
356
|
+
test ! -e "$SNAPSHOT_ROOT/host/notify.conf" || return 1
|
|
357
|
+
test ! -e "$SNAPSHOT_ROOT/host/sabnzbd.ini" || return 1
|
|
358
|
+
SNAPSHOT_VERIFY_ROOT=$(mktemp -d /tmp/ouro-snapshot-verify.XXXXXX) || return $?
|
|
359
|
+
trap 'rm -f -- "$SNAPSHOT_VERIFY_ROOT/current-files" "$SNAPSHOT_VERIFY_ROOT/manifest-files"; rmdir -- "$SNAPSHOT_VERIFY_ROOT"' EXIT || return $?
|
|
360
|
+
(
|
|
361
|
+
cd "$SNAPSHOT_ROOT" || return $?
|
|
362
|
+
find runtime agent host provenance -xdev -type f ! -path provenance/manifest.sha256 -print | LC_ALL=C sort >"$SNAPSHOT_VERIFY_ROOT/current-files" || return $?
|
|
363
|
+
grep -E '^[0-9a-f]{64} [^/].+$' provenance/manifest.sha256 >/dev/null || return $?
|
|
364
|
+
! grep -Ev '^[0-9a-f]{64} [^/].+$' provenance/manifest.sha256 >/dev/null || return 1
|
|
365
|
+
sed 's/^[0-9a-f]\{64\} //' provenance/manifest.sha256 | LC_ALL=C sort >"$SNAPSHOT_VERIFY_ROOT/manifest-files" || return $?
|
|
366
|
+
cmp -s "$SNAPSHOT_VERIFY_ROOT/current-files" "$SNAPSHOT_VERIFY_ROOT/manifest-files" || return $?
|
|
367
|
+
sha256sum -c -- provenance/manifest.sha256 >/dev/null || return $?
|
|
368
|
+
) || return $?
|
|
369
|
+
rm -f -- "$SNAPSHOT_VERIFY_ROOT/current-files" "$SNAPSHOT_VERIFY_ROOT/manifest-files" || return $?
|
|
370
|
+
rmdir -- "$SNAPSHOT_VERIFY_ROOT" || return $?
|
|
371
|
+
trap - EXIT || return $?
|
|
216
372
|
)
|
|
217
373
|
}
|
|
218
374
|
assert_restore_preflight() {
|
|
219
375
|
(
|
|
220
376
|
test -n "${BACKUP_ROOT-}" || return 1
|
|
221
377
|
test -n "${IMAGE_ID-}" || return 1
|
|
378
|
+
test -n "${AUDIT_RUNNER_IMAGE_ID-}" || return 1
|
|
222
379
|
case "$BACKUP_ROOT" in
|
|
223
380
|
/*) ;;
|
|
224
381
|
*) return 1 ;;
|
|
@@ -228,8 +385,11 @@ Effective-spec audit helper:
|
|
|
228
385
|
test "$RESTORE_BACKUP_ROOT" = "$BACKUP_ROOT" || return $?
|
|
229
386
|
test -d "$BACKUP_ROOT/runtime/.ouro-cli" || return $?
|
|
230
387
|
test -d "$BACKUP_ROOT/agent/sanctuary.ouro" || return $?
|
|
388
|
+
verify_sanctuary_snapshot_provenance "$BACKUP_ROOT" "$IMAGE_ID" || return $?
|
|
231
389
|
validate_sanctuary_roots "$BACKUP_ROOT/runtime/.ouro-cli" "$BACKUP_ROOT/agent/sanctuary.ouro" || return $?
|
|
232
390
|
validate_exact_image_id "$IMAGE_ID" || return $?
|
|
391
|
+
validate_exact_image_id "$AUDIT_RUNNER_IMAGE_ID" || return $?
|
|
392
|
+
test "$AUDIT_RUNNER_IMAGE_ID" != sha256:681449ad47a2621705cd339b481e6339236b31dc65e195b1cf5025d0f2191d7d || return $?
|
|
233
393
|
RESTORE_CONTAINER_NAMES=$(docker container ls -a --format '{{.Names}}') || return $?
|
|
234
394
|
RESTORE_NAME_COUNTS=$(printf '%s\n' "$RESTORE_CONTAINER_NAMES" | awk '
|
|
235
395
|
$0 == "ouro-butler" { production++ }
|
|
@@ -255,6 +415,7 @@ Effective-spec audit helper:
|
|
|
255
415
|
validate_exact_image_id "$RESTORE_LEGACY_IMAGE_ID" || return $?
|
|
256
416
|
fi
|
|
257
417
|
assert_only_running_butler ouro-butler || return $?
|
|
418
|
+
assert_update_source "$RESTORE_PRODUCTION_IMAGE_ID" "$AUDIT_RUNNER_IMAGE_ID" || return $?
|
|
258
419
|
)
|
|
259
420
|
}
|
|
260
421
|
ensure_sanctuary_machine_identity() {
|
|
@@ -280,6 +441,141 @@ Effective-spec audit helper:
|
|
|
280
441
|
chmod 0600 "$MACHINE_PATH" || return $?
|
|
281
442
|
sync -f "$MACHINE_PATH" || return $?
|
|
282
443
|
}
|
|
444
|
+
migrate_sanctuary_package_managed_bundle() {
|
|
445
|
+
MIGRATE_IMAGE_ID=$1
|
|
446
|
+
MIGRATE_OPERATION=$2
|
|
447
|
+
MIGRATE_ROLLBACK_IMAGE_ID=${3-}
|
|
448
|
+
validate_exact_image_id "$MIGRATE_IMAGE_ID" || return $?
|
|
449
|
+
case "$MIGRATE_OPERATION" in migrate|rollback|commit|status) ;; *) return 1 ;; esac
|
|
450
|
+
if test "$MIGRATE_OPERATION" = migrate; then
|
|
451
|
+
validate_exact_image_id "$MIGRATE_ROLLBACK_IMAGE_ID" || return $?
|
|
452
|
+
test "$MIGRATE_IMAGE_ID" != "$MIGRATE_ROLLBACK_IMAGE_ID" || return 1
|
|
453
|
+
set -- --rollback-image-id "$MIGRATE_ROLLBACK_IMAGE_ID" --target-image-id "$MIGRATE_IMAGE_ID"
|
|
454
|
+
else
|
|
455
|
+
set --
|
|
456
|
+
fi
|
|
457
|
+
docker run --rm --pull=never --network=none --read-only --user 10001:10001 \
|
|
458
|
+
--entrypoint /usr/local/bin/node \
|
|
459
|
+
--mount "type=bind,src=/mnt/user/appdata/ouro-butler/agent/sanctuary.ouro,dst=/home/ouro/AgentBundles/sanctuary.ouro" \
|
|
460
|
+
"$MIGRATE_IMAGE_ID" /opt/ouro/deploy/unraid/migrate-sanctuary-bundle.mjs \
|
|
461
|
+
--package-root /opt/ouro/deploy/unraid/sanctuary.ouro \
|
|
462
|
+
--agent-root /home/ouro/AgentBundles/sanctuary.ouro \
|
|
463
|
+
--operation "$MIGRATE_OPERATION" "$@" || return $?
|
|
464
|
+
}
|
|
465
|
+
recover_pending_sanctuary_bundle_migration() {
|
|
466
|
+
RECOVERY_IMAGE_ID=$1
|
|
467
|
+
RECOVERY_AGENT_ROOT=/mnt/user/appdata/ouro-butler/agent/sanctuary.ouro
|
|
468
|
+
RECOVERY_RECORD=$RECOVERY_AGENT_ROOT/.sanctuary-package-managed-rollback.json
|
|
469
|
+
RECOVERY_COMMITTING_RECORD=$RECOVERY_RECORD.committing
|
|
470
|
+
RECOVERY_RECORD_COUNT=0
|
|
471
|
+
for RECOVERY_CANDIDATE in "$RECOVERY_RECORD" "$RECOVERY_COMMITTING_RECORD"; do
|
|
472
|
+
test ! -L "$RECOVERY_CANDIDATE" || return 1
|
|
473
|
+
if test -e "$RECOVERY_CANDIDATE"; then
|
|
474
|
+
test -f "$RECOVERY_CANDIDATE" || return 1
|
|
475
|
+
RECOVERY_RECORD_COUNT=$((RECOVERY_RECORD_COUNT + 1))
|
|
476
|
+
fi
|
|
477
|
+
done
|
|
478
|
+
test "$RECOVERY_RECORD_COUNT" -le 1 || return 1
|
|
479
|
+
test "$RECOVERY_RECORD_COUNT" -eq 1 || return 0
|
|
480
|
+
RECOVERY_STATUS=$(migrate_sanctuary_package_managed_bundle "$RECOVERY_IMAGE_ID" status) || return $?
|
|
481
|
+
RECOVERY_IDENTITIES=$(printf '%s' "$RECOVERY_STATUS" | /usr/local/bin/node -e '
|
|
482
|
+
let input = "";
|
|
483
|
+
process.stdin.setEncoding("utf8");
|
|
484
|
+
process.stdin.on("data", chunk => { input += chunk; });
|
|
485
|
+
process.stdin.on("end", () => {
|
|
486
|
+
const value = JSON.parse(input);
|
|
487
|
+
if (!value || !["rollback", "committing"].includes(value.state)
|
|
488
|
+
|| typeof value.rollbackImageId !== "string" || typeof value.targetImageId !== "string") process.exit(1);
|
|
489
|
+
process.stdout.write(`${value.state} ${value.rollbackImageId} ${value.targetImageId}`);
|
|
490
|
+
});
|
|
491
|
+
') || return $?
|
|
492
|
+
set -- $RECOVERY_IDENTITIES
|
|
493
|
+
test "$#" -eq 3 || return 1
|
|
494
|
+
RECOVERY_STATE=$1
|
|
495
|
+
RECOVERY_ROLLBACK_IMAGE_ID=$2
|
|
496
|
+
RECOVERY_TARGET_IMAGE_ID=$3
|
|
497
|
+
validate_exact_image_id "$RECOVERY_ROLLBACK_IMAGE_ID" || return $?
|
|
498
|
+
validate_exact_image_id "$RECOVERY_TARGET_IMAGE_ID" || return $?
|
|
499
|
+
test "$RECOVERY_TARGET_IMAGE_ID" != "$RECOVERY_ROLLBACK_IMAGE_ID" || return 1
|
|
500
|
+
disable_butler_autostart || return $?
|
|
501
|
+
if test "$RECOVERY_STATE" = committing; then
|
|
502
|
+
! docker container inspect ouro-butler-staging >/dev/null 2>&1 || return 1
|
|
503
|
+
RECOVERY_PRODUCTION_IMAGE_ID=$(docker inspect --format '{{.Image}}' ouro-butler) || return $?
|
|
504
|
+
test "$(docker inspect --format '{{.State.Running}}' ouro-butler)" = true || return 1
|
|
505
|
+
if test "$RECOVERY_PRODUCTION_IMAGE_ID" = "$RECOVERY_TARGET_IMAGE_ID"; then
|
|
506
|
+
RECOVERY_CURRENT_ROLLBACK_IMAGE_ID=$(docker inspect --format '{{.Image}}' ouro-butler-rollback) || return $?
|
|
507
|
+
test "$RECOVERY_CURRENT_ROLLBACK_IMAGE_ID" = "$RECOVERY_ROLLBACK_IMAGE_ID" || return 1
|
|
508
|
+
test "$(docker inspect --format '{{.State.Running}}' ouro-butler-rollback)" = false || return 1
|
|
509
|
+
elif test "$RECOVERY_PRODUCTION_IMAGE_ID" = "$RECOVERY_ROLLBACK_IMAGE_ID"; then
|
|
510
|
+
! docker container inspect ouro-butler-rollback >/dev/null 2>&1 || return 1
|
|
511
|
+
else
|
|
512
|
+
return 1
|
|
513
|
+
fi
|
|
514
|
+
assert_only_running_butler ouro-butler || return $?
|
|
515
|
+
assert_update_source "$RECOVERY_PRODUCTION_IMAGE_ID" "$AUDIT_RUNNER_IMAGE_ID" || return $?
|
|
516
|
+
wait_butler_ready ouro-butler || return $?
|
|
517
|
+
enable_butler_autostart || return $?
|
|
518
|
+
migrate_sanctuary_package_managed_bundle "$RECOVERY_IMAGE_ID" commit || return $?
|
|
519
|
+
return 0
|
|
520
|
+
fi
|
|
521
|
+
if docker container inspect ouro-butler-rollback >/dev/null 2>&1; then
|
|
522
|
+
RECOVERY_CURRENT_ROLLBACK_IMAGE_ID=$(docker inspect --format '{{.Image}}' ouro-butler-rollback) || return $?
|
|
523
|
+
test "$RECOVERY_CURRENT_ROLLBACK_IMAGE_ID" = "$RECOVERY_ROLLBACK_IMAGE_ID" || return 1
|
|
524
|
+
test "$(docker inspect --format '{{.State.Running}}' ouro-butler-rollback)" = false || return 1
|
|
525
|
+
if docker container inspect ouro-butler-staging >/dev/null 2>&1; then
|
|
526
|
+
RECOVERY_STAGING_IMAGE_ID=$(docker inspect --format '{{.Image}}' ouro-butler-staging) || return $?
|
|
527
|
+
test "$RECOVERY_STAGING_IMAGE_ID" = "$RECOVERY_TARGET_IMAGE_ID" || return 1
|
|
528
|
+
docker stop ouro-butler-staging >/dev/null 2>&1 || true
|
|
529
|
+
docker rm --force ouro-butler-staging || return $?
|
|
530
|
+
fi
|
|
531
|
+
if docker container inspect ouro-butler >/dev/null 2>&1; then
|
|
532
|
+
RECOVERY_PRODUCTION_IMAGE_ID=$(docker inspect --format '{{.Image}}' ouro-butler) || return $?
|
|
533
|
+
test "$RECOVERY_PRODUCTION_IMAGE_ID" = "$RECOVERY_TARGET_IMAGE_ID" || return 1
|
|
534
|
+
docker stop ouro-butler >/dev/null 2>&1 || true
|
|
535
|
+
docker rm --force ouro-butler || return $?
|
|
536
|
+
fi
|
|
537
|
+
migrate_sanctuary_package_managed_bundle "$RECOVERY_IMAGE_ID" rollback || return $?
|
|
538
|
+
docker rename ouro-butler-rollback ouro-butler || return $?
|
|
539
|
+
elif docker container inspect ouro-butler >/dev/null 2>&1; then
|
|
540
|
+
! docker container inspect ouro-butler-staging >/dev/null 2>&1 || return 1
|
|
541
|
+
RECOVERY_PRODUCTION_IMAGE_ID=$(docker inspect --format '{{.Image}}' ouro-butler) || return $?
|
|
542
|
+
test "$RECOVERY_PRODUCTION_IMAGE_ID" = "$RECOVERY_ROLLBACK_IMAGE_ID" || return 1
|
|
543
|
+
! docker container inspect ouro-butler-rollback >/dev/null 2>&1 || return 1
|
|
544
|
+
else
|
|
545
|
+
return 1
|
|
546
|
+
fi
|
|
547
|
+
RECOVERY_PRODUCTION_IMAGE_ID=$(docker inspect --format '{{.Image}}' ouro-butler) || return $?
|
|
548
|
+
test "$RECOVERY_PRODUCTION_IMAGE_ID" = "$RECOVERY_ROLLBACK_IMAGE_ID" || return 1
|
|
549
|
+
assert_update_source "$RECOVERY_ROLLBACK_IMAGE_ID" "$AUDIT_RUNNER_IMAGE_ID" || return $?
|
|
550
|
+
docker start ouro-butler || return $?
|
|
551
|
+
assert_only_running_butler ouro-butler || return $?
|
|
552
|
+
wait_butler_ready ouro-butler || return $?
|
|
553
|
+
enable_butler_autostart || return $?
|
|
554
|
+
migrate_sanctuary_package_managed_bundle "$RECOVERY_IMAGE_ID" commit || return $?
|
|
555
|
+
}
|
|
556
|
+
rollback_sanctuary_bundle_if_pending() {
|
|
557
|
+
OPTIONAL_ROLLBACK_IMAGE_ID=$1
|
|
558
|
+
OPTIONAL_ROLLBACK_RECORD=/mnt/user/appdata/ouro-butler/agent/sanctuary.ouro/.sanctuary-package-managed-rollback.json
|
|
559
|
+
OPTIONAL_COMMITTING_RECORD=$OPTIONAL_ROLLBACK_RECORD.committing
|
|
560
|
+
test ! -L "$OPTIONAL_ROLLBACK_RECORD" || return 1
|
|
561
|
+
test ! -L "$OPTIONAL_COMMITTING_RECORD" || return 1
|
|
562
|
+
test ! -e "$OPTIONAL_COMMITTING_RECORD" || return 1
|
|
563
|
+
if test -e "$OPTIONAL_ROLLBACK_RECORD"; then
|
|
564
|
+
test -f "$OPTIONAL_ROLLBACK_RECORD" || return 1
|
|
565
|
+
migrate_sanctuary_package_managed_bundle "$OPTIONAL_ROLLBACK_IMAGE_ID" rollback || return $?
|
|
566
|
+
fi
|
|
567
|
+
}
|
|
568
|
+
# Package installation and migration never grant restart authority. The
|
|
569
|
+
# migration is the pre-activation assertion: it fails before live mutation
|
|
570
|
+
# if packaged steward.json carries any routine grant. It preserves an
|
|
571
|
+
# existing live steward policy and its audit receipts byte for byte. On a
|
|
572
|
+
# new installation, start the healthy private runtime, then use
|
|
573
|
+
# an authenticated owner Telegram turn to grant each exact restart target
|
|
574
|
+
# through steward_policy_manage. That existing tool binds the grant to the
|
|
575
|
+
# verified owner relationship and durable session event. Do not seed grants
|
|
576
|
+
# in the packaged steward.json, copy an old session-event ID, or edit the
|
|
577
|
+
# live policy/audit files directly. A missing grant remains fail-closed until
|
|
578
|
+
# that explicit owner ceremony succeeds.
|
|
283
579
|
prepare_canonical_sanctuary_roots() {
|
|
284
580
|
PREPARE_IMAGE_ID=$1
|
|
285
581
|
validate_exact_image_id "$PREPARE_IMAGE_ID" || return $?
|
|
@@ -490,8 +786,10 @@ Effective-spec audit helper:
|
|
|
490
786
|
&& docker create --name ouro-butler-staging --network host --restart unless-stopped --user 10001:10001 \
|
|
491
787
|
--mount "type=bind,src=/mnt/user/appdata/ouro-butler/runtime/.ouro-cli,dst=/home/ouro/.ouro-cli" \
|
|
492
788
|
--mount "type=bind,src=/mnt/user/appdata/ouro-butler/agent/sanctuary.ouro,dst=/home/ouro/AgentBundles/sanctuary.ouro" \
|
|
789
|
+
--mount "type=bind,src=/boot/config/custom/ouro-events/spool,dst=/run/ouro-events,readonly" \
|
|
790
|
+
--mount "type=bind,src=/mnt/user/appdata/sabnzbd/sabnzbd.ini,dst=/run/sanctuary/sabnzbd.ini,readonly" \
|
|
493
791
|
"$IMAGE_ID" \
|
|
494
|
-
&& audit_effective ouro-butler-staging "$IMAGE_ID" \
|
|
792
|
+
&& audit_effective ouro-butler-staging "$IMAGE_ID" "$IMAGE_ID" \
|
|
495
793
|
&& docker start ouro-butler-staging \
|
|
496
794
|
&& assert_only_running_butler ouro-butler-staging \
|
|
497
795
|
&& wait_butler_ready ouro-butler-staging \
|
|
@@ -501,8 +799,10 @@ Effective-spec audit helper:
|
|
|
501
799
|
&& docker create --name ouro-butler --network host --restart unless-stopped --user 10001:10001 \
|
|
502
800
|
--mount "type=bind,src=/mnt/user/appdata/ouro-butler/runtime/.ouro-cli,dst=/home/ouro/.ouro-cli" \
|
|
503
801
|
--mount "type=bind,src=/mnt/user/appdata/ouro-butler/agent/sanctuary.ouro,dst=/home/ouro/AgentBundles/sanctuary.ouro" \
|
|
802
|
+
--mount "type=bind,src=/boot/config/custom/ouro-events/spool,dst=/run/ouro-events,readonly" \
|
|
803
|
+
--mount "type=bind,src=/mnt/user/appdata/sabnzbd/sabnzbd.ini,dst=/run/sanctuary/sabnzbd.ini,readonly" \
|
|
504
804
|
"$IMAGE_ID" \
|
|
505
|
-
&& audit_effective ouro-butler "$IMAGE_ID" \
|
|
805
|
+
&& audit_effective ouro-butler "$IMAGE_ID" "$IMAGE_ID" \
|
|
506
806
|
&& docker start ouro-butler \
|
|
507
807
|
&& assert_only_running_butler ouro-butler \
|
|
508
808
|
&& wait_butler_ready ouro-butler \
|
|
@@ -713,18 +1013,16 @@ local unlock: available
|
|
|
713
1013
|
machineIdMigration: {
|
|
714
1014
|
sourceMachineId: process.argv[1],
|
|
715
1015
|
targetMachineId: process.argv[2],
|
|
716
|
-
discardProviderCredentialRecords: { providers: ["minimax"] },
|
|
717
1016
|
},
|
|
718
1017
|
}).then(
|
|
719
1018
|
() => process.exit(0),
|
|
720
1019
|
() => { process.stderr.write("credential bootstrap failed\n"); process.exit(1) },
|
|
721
1020
|
)
|
|
722
1021
|
' "$BOOTSTRAP_SOURCE_MACHINE_ID" "$BOOTSTRAP_TARGET_MACHINE_ID" || return $?
|
|
723
|
-
# The path-locked Sanctuary migration
|
|
724
|
-
#
|
|
1022
|
+
# The path-locked Sanctuary migration imports the legacy MiniMax provider
|
|
1023
|
+
# record into the agent vault before strict validation and persistence.
|
|
725
1024
|
# The original legacy envelope remains byte-for-byte unchanged and is the
|
|
726
|
-
# sole digest authority
|
|
727
|
-
# only by the separate hidden-input authentication commands below.
|
|
1025
|
+
# sole digest authority.
|
|
728
1026
|
! docker container inspect ouro-butler-credential-bootstrap >/dev/null 2>&1 || return 1
|
|
729
1027
|
test ! -e "$BOOTSTRAP_CREDENTIAL_SOURCE" || return $?
|
|
730
1028
|
test ! -e "$BOOTSTRAP_CREDENTIAL_CLAIM" || return $?
|
|
@@ -778,54 +1076,29 @@ local unlock: available
|
|
|
778
1076
|
const policy = JSON.parse(fs.readFileSync(`${root}/provider-readiness.json`, "utf8"));
|
|
779
1077
|
const expectedPolicy = { version: 1, selectionPolicy: "explicit-same-lane-only" };
|
|
780
1078
|
const expected = new Map([
|
|
781
|
-
["
|
|
782
|
-
["openai-compatible-gemini", { provider: "openai-compatible-gemini", model: "gemini-3.6-flash", vaultItem: "providers/openai-compatible-gemini", baseUrl: "https://generativelanguage.googleapis.com/v1beta/openai/" }],
|
|
1079
|
+
["minimax", { provider: "minimax", model: "MiniMax-M3", vaultItem: "providers/minimax" }],
|
|
783
1080
|
]);
|
|
784
|
-
const exactLane = value => value && value.provider === "
|
|
1081
|
+
const exactLane = value => value && value.provider === "minimax" && value.model === "MiniMax-M3";
|
|
785
1082
|
if (!exactLane(agent.humanFacing) || !exactLane(agent.agentFacing)
|
|
786
1083
|
|| policy.version !== expectedPolicy.version || policy.selectionPolicy !== expectedPolicy.selectionPolicy
|
|
787
1084
|
|| !Array.isArray(policy.providers) || policy.providers.length !== expected.size) throw new Error();
|
|
788
1085
|
for (const entry of policy.providers) {
|
|
789
1086
|
const wanted = expected.get(entry.provider);
|
|
790
|
-
if (!wanted || entry.model !== wanted.model || entry.vaultItem !== wanted.vaultItem
|
|
1087
|
+
if (!wanted || entry.model !== wanted.model || entry.vaultItem !== wanted.vaultItem) throw new Error();
|
|
791
1088
|
expected.delete(entry.provider);
|
|
792
1089
|
}
|
|
793
1090
|
if (expected.size !== 0) throw new Error();
|
|
794
1091
|
const { refreshProviderCredentialPool } = require("/opt/ouro/dist/heart/provider-credentials.js");
|
|
795
|
-
const { pingProvider } = require("/opt/ouro/dist/heart/provider-ping.js");
|
|
796
1092
|
const refreshed = await refreshProviderCredentialPool("sanctuary", {
|
|
797
|
-
providers: ["
|
|
1093
|
+
providers: ["minimax"],
|
|
798
1094
|
skipCache: true,
|
|
799
1095
|
});
|
|
800
1096
|
if (!refreshed.ok) throw new Error();
|
|
801
|
-
const
|
|
802
|
-
const
|
|
803
|
-
const exactRecord = (record, provider, baseUrl) => record && record.provider === provider
|
|
1097
|
+
const minimax = refreshed.pool.providers["minimax"];
|
|
1098
|
+
const exactRecord = (record, provider) => record && record.provider === provider
|
|
804
1099
|
&& typeof record.revision === "string" && record.revision.length > 0
|
|
805
|
-
&& typeof record.credentials?.apiKey === "string" && record.credentials.apiKey.trim().length > 0
|
|
806
|
-
|
|
807
|
-
if (!exactRecord(glm, "openai-compatible", "https://api.z.ai/api/paas/v4/")
|
|
808
|
-
|| !exactRecord(gemini, "openai-compatible-gemini", "https://generativelanguage.googleapis.com/v1beta/openai/")
|
|
809
|
-
|| glm.credentials.apiKey === gemini.credentials.apiKey
|
|
810
|
-
|| new Set([glm.revision, gemini.revision]).size !== 2) throw new Error();
|
|
811
|
-
const probes = [
|
|
812
|
-
{ provider: "openai-compatible", model: "glm-5.2", credentialRevision: glm.revision, record: glm },
|
|
813
|
-
{ provider: "openai-compatible", model: "glm-5.2", credentialRevision: glm.revision, record: glm },
|
|
814
|
-
{ provider: "openai-compatible-gemini", model: "gemini-3.6-flash", credentialRevision: gemini.revision, record: gemini },
|
|
815
|
-
];
|
|
816
|
-
const results = await Promise.all(probes.map(({ provider, model, record }) => pingProvider(
|
|
817
|
-
provider,
|
|
818
|
-
{ ...record.credentials, ...record.config },
|
|
819
|
-
{ model, timeoutMs: 10000, attemptPolicy: { baseDelayMs: 0 } },
|
|
820
|
-
)));
|
|
821
|
-
for (let index = 0; index < results.length; index += 1) {
|
|
822
|
-
const result = results[index];
|
|
823
|
-
const probe = probes[index];
|
|
824
|
-
if (!result.ok || !Array.isArray(result.attempts) || result.attempts.length < 1 || result.attempts.length > 3
|
|
825
|
-
|| !result.attempts.every(attempt => attempt.provider === probe.provider && attempt.model === probe.model
|
|
826
|
-
&& attempt.operation === "ping" && typeof attempt.ok === "boolean")
|
|
827
|
-
|| result.attempts.at(-1)?.ok !== true) throw new Error();
|
|
828
|
-
}
|
|
1100
|
+
&& typeof record.credentials?.apiKey === "string" && record.credentials.apiKey.trim().length > 0;
|
|
1101
|
+
if (!exactRecord(minimax, "minimax")) throw new Error();
|
|
829
1102
|
process.stdout.write("Sanctuary provider readiness verified.\n");
|
|
830
1103
|
} catch {
|
|
831
1104
|
process.stderr.write("Sanctuary provider readiness verification failed.\n");
|
|
@@ -1036,17 +1309,93 @@ Update:
|
|
|
1036
1309
|
IMAGE_ID=$(docker image inspect --format '{{.Id}}' "$IMAGE_TAG")
|
|
1037
1310
|
printf '%s\n' "$IMAGE_ID" | grep -Eq '^sha256:[0-9a-f]{64}$'
|
|
1038
1311
|
docker image inspect "$IMAGE_ID" >/dev/null
|
|
1312
|
+
AUDIT_RUNNER_IMAGE_ID=$IMAGE_ID
|
|
1313
|
+
validate_exact_image_id "$AUDIT_RUNNER_IMAGE_ID"
|
|
1314
|
+
Before stopping, renaming, or creating any Butler container, extract the packaged event, template, and runtime-policy assets from that exact image ID. Install the event assets transactionally, bind the staged template to the exact local image ID, and audit both staged deployment files. Do not copy these files from a checkout or another image:
|
|
1315
|
+
EVENT_ASSET_STAGE=$(mktemp -d /mnt/user/appdata/ouro-butler/staging/ouro-events.XXXXXX)
|
|
1316
|
+
chmod 0700 "$EVENT_ASSET_STAGE"
|
|
1317
|
+
EVENT_SCRIPT_STAGE="$EVENT_ASSET_STAGE/ouro-events"
|
|
1318
|
+
mkdir "$EVENT_SCRIPT_STAGE"
|
|
1319
|
+
STAGED_TEMPLATE="$EVENT_ASSET_STAGE/sanctuary.xml"
|
|
1320
|
+
STAGED_RUNTIME_POLICY="$EVENT_ASSET_STAGE/container-runtime.json"
|
|
1321
|
+
EVENT_ASSET_CONTAINER=
|
|
1322
|
+
cleanup_event_asset_stage() {
|
|
1323
|
+
if test -n "$EVENT_ASSET_CONTAINER"; then
|
|
1324
|
+
docker rm --force "$EVENT_ASSET_CONTAINER" >/dev/null 2>&1 || true
|
|
1325
|
+
fi
|
|
1326
|
+
rm -rf -- "$EVENT_ASSET_STAGE"
|
|
1327
|
+
}
|
|
1328
|
+
trap cleanup_event_asset_stage EXIT
|
|
1329
|
+
EVENT_ASSET_CONTAINER=$(docker create --pull=never --network none --read-only --entrypoint /bin/false "$IMAGE_ID")
|
|
1330
|
+
test "$(docker inspect --format '{{.Image}}' "$EVENT_ASSET_CONTAINER")" = "$IMAGE_ID"
|
|
1331
|
+
docker cp "$EVENT_ASSET_CONTAINER:/opt/ouro/deploy/unraid/ouro-events/." "$EVENT_SCRIPT_STAGE/"
|
|
1332
|
+
docker cp "$EVENT_ASSET_CONTAINER:/opt/ouro/deploy/unraid/sanctuary.xml" "$STAGED_TEMPLATE"
|
|
1333
|
+
docker cp "$EVENT_ASSET_CONTAINER:/opt/ouro/deploy/unraid/container-runtime.json" "$STAGED_RUNTIME_POLICY"
|
|
1334
|
+
docker rm "$EVENT_ASSET_CONTAINER"
|
|
1335
|
+
EVENT_ASSET_CONTAINER=
|
|
1336
|
+
EXPECTED_RELEASE_ASSETS=$(printf '%s\n' container-runtime.json ouro-events sanctuary.xml)
|
|
1337
|
+
ACTUAL_RELEASE_ASSETS=$(find "$EVENT_ASSET_STAGE" -mindepth 1 -maxdepth 1 -exec basename {} \; | LC_ALL=C sort)
|
|
1338
|
+
test "$ACTUAL_RELEASE_ASSETS" = "$EXPECTED_RELEASE_ASSETS"
|
|
1339
|
+
test -d "$EVENT_SCRIPT_STAGE" && test ! -L "$EVENT_SCRIPT_STAGE"
|
|
1340
|
+
test -f "$STAGED_TEMPLATE" && test ! -L "$STAGED_TEMPLATE"
|
|
1341
|
+
test -f "$STAGED_RUNTIME_POLICY" && test ! -L "$STAGED_RUNTIME_POLICY"
|
|
1342
|
+
EXPECTED_EVENT_ASSETS=$(printf '%s\n' bootstrap-spool.sh emit-event.mjs emit-usenet-event.sh install-usenet-guard.sh usenet-health.sh)
|
|
1343
|
+
ACTUAL_EVENT_ASSETS=$(find "$EVENT_SCRIPT_STAGE" -mindepth 1 -maxdepth 1 -type f -exec basename {} \; | LC_ALL=C sort)
|
|
1344
|
+
test "$ACTUAL_EVENT_ASSETS" = "$EXPECTED_EVENT_ASSETS"
|
|
1345
|
+
test -z "$(find "$EVENT_SCRIPT_STAGE" -mindepth 1 -maxdepth 1 \( -type l -o ! -type f \) -print -quit)"
|
|
1346
|
+
/bin/bash "$EVENT_SCRIPT_STAGE/install-usenet-guard.sh" --source-root "$EVENT_SCRIPT_STAGE"
|
|
1347
|
+
/bin/bash /boot/config/custom/ouro-events/install-usenet-guard.sh --boot --install-root /boot/config/custom
|
|
1348
|
+
verify_installed_usenet_guard() {
|
|
1349
|
+
test "$(findmnt -n -o FSTYPE --target /boot/config/custom)" = vfat
|
|
1350
|
+
test "$(stat -c '%u:%g:%a' /boot/config/custom/usenet_health.sh)" = 0:0:600
|
|
1351
|
+
test "$(stat -c '%u:%g:%a' /boot/config/custom/ouro-events/bootstrap-spool.sh)" = 0:0:600
|
|
1352
|
+
test "$(stat -c '%u:%g:%a' /boot/config/custom/ouro-events/emit-event.mjs)" = 0:0:600
|
|
1353
|
+
test "$(stat -c '%u:%g:%a' /boot/config/custom/ouro-events/emit-usenet-event.sh)" = 0:0:600
|
|
1354
|
+
test "$(stat -c '%u:%g:%a' /boot/config/custom/ouro-events/install-usenet-guard.sh)" = 0:0:600
|
|
1355
|
+
cmp -s "$EVENT_SCRIPT_STAGE/usenet-health.sh" /boot/config/custom/usenet_health.sh
|
|
1356
|
+
cmp -s "$EVENT_SCRIPT_STAGE/bootstrap-spool.sh" /boot/config/custom/ouro-events/bootstrap-spool.sh
|
|
1357
|
+
cmp -s "$EVENT_SCRIPT_STAGE/emit-event.mjs" /boot/config/custom/ouro-events/emit-event.mjs
|
|
1358
|
+
cmp -s "$EVENT_SCRIPT_STAGE/emit-usenet-event.sh" /boot/config/custom/ouro-events/emit-usenet-event.sh
|
|
1359
|
+
cmp -s "$EVENT_SCRIPT_STAGE/install-usenet-guard.sh" /boot/config/custom/ouro-events/install-usenet-guard.sh
|
|
1360
|
+
test "$(grep -Fxc '/bin/bash /boot/config/custom/ouro-events/install-usenet-guard.sh --boot --install-root /boot/config/custom' /boot/config/go)" = 1
|
|
1361
|
+
test "$(grep -Fxc '/boot/config/custom/ouro-events/bootstrap-spool.sh --mount' /boot/config/go || true)" = 0
|
|
1362
|
+
INSTALLED_GUARD_CRON=$(crontab -l)
|
|
1363
|
+
test "$(printf '%s\n' "$INSTALLED_GUARD_CRON" | grep -Fxc '*/15 * * * * /bin/bash /boot/config/custom/usenet_health.sh # ouro:usenet-health')" = 1
|
|
1364
|
+
INSTALLED_LEGACY_GUARD_COUNT=$(printf '%s\n' "$INSTALLED_GUARD_CRON" | grep -Fxc '*/15 * * * * /bin/bash /boot/config/custom/usenet_health.sh' || true)
|
|
1365
|
+
test "$INSTALLED_LEGACY_GUARD_COUNT" = 0
|
|
1366
|
+
test "$(printf '%s\n' "$INSTALLED_GUARD_CRON" | grep -Fc '/bin/bash /boot/config/custom/usenet_health.sh')" = 1
|
|
1367
|
+
test "$(findmnt -n -o FSTYPE --target /boot/config/custom/ouro-events/spool)" = tmpfs
|
|
1368
|
+
test "$(stat -c '%u:%g:%a' /boot/config/custom/ouro-events/spool)" = 0:0:755
|
|
1369
|
+
INSTALLED_SPOOL_OPTIONS=$(findmnt -n -o OPTIONS --target /boot/config/custom/ouro-events/spool)
|
|
1370
|
+
for INSTALLED_SPOOL_OPTION in nodev nosuid noexec; do
|
|
1371
|
+
case ",$INSTALLED_SPOOL_OPTIONS," in *",$INSTALLED_SPOOL_OPTION,"*) ;; *) return 1 ;; esac
|
|
1372
|
+
done
|
|
1373
|
+
INSTALLED_SPOOL_SIZE=$(findmnt -bn -o SIZE --target /boot/config/custom/ouro-events/spool)
|
|
1374
|
+
test "$INSTALLED_SPOOL_SIZE" -le 4194304
|
|
1375
|
+
}
|
|
1376
|
+
verify_installed_usenet_guard
|
|
1377
|
+
/usr/local/bin/node -e '
|
|
1378
|
+
const fs = require("node:fs");
|
|
1379
|
+
const [templatePath, imageId] = process.argv.slice(1);
|
|
1380
|
+
if (!/^sha256:[0-9a-f]{64}$/.test(imageId)) throw new Error("invalid exact image ID");
|
|
1381
|
+
const source = fs.readFileSync(templatePath, "utf8");
|
|
1382
|
+
const repositories = [...source.matchAll(/<Repository>[^<]*<\/Repository>/g)];
|
|
1383
|
+
if (repositories.length !== 1) throw new Error("template must contain exactly one Repository element");
|
|
1384
|
+
const staged = source.replace(repositories[0][0], `<Repository>${imageId}</Repository>`);
|
|
1385
|
+
if ((staged.match(/<Repository>[^<]*<\/Repository>/g) ?? []).length !== 1
|
|
1386
|
+
|| !staged.includes(`<Repository>${imageId}</Repository>`)) throw new Error("exact Repository staging failed");
|
|
1387
|
+
fs.writeFileSync(templatePath, staged);
|
|
1388
|
+
' "$STAGED_TEMPLATE" "$IMAGE_ID"
|
|
1389
|
+
If extraction, transactional installation, boot activation, or verification fails, abort the update before any Butler mutation. The installer restores the previous assets, go file, and cron on transactional failure; production stays running. If boot activation or verification fails after the transaction commits, leave production untouched, repair or rerun this exact-image installation, and do not continue. Container rollback begins only after the later production stop succeeds.
|
|
1039
1390
|
Initial install/adoption is a separate terminal path for the verified live
|
|
1040
1391
|
legacy state: no production or rollback, exactly one running (possibly
|
|
1041
1392
|
unhealthy) ouro-butler-staging, and no legacy-evidence container. After the
|
|
1042
1393
|
helper definitions above are loaded and IMAGE_ID is resolved, run this exact
|
|
1043
|
-
sequence.
|
|
1044
|
-
|
|
1394
|
+
sequence. MiniMax credentials are imported from the byte-verified legacy
|
|
1395
|
+
bootstrap envelope; do not place credentials in arguments, shell variables, or
|
|
1045
1396
|
history.
|
|
1046
1397
|
Sanctuary legacy adoption commands:
|
|
1047
1398
|
prepare_sanctuary_legacy_adoption "$IMAGE_ID"
|
|
1048
|
-
authenticate_sanctuary_provider "$IMAGE_ID" openai-compatible
|
|
1049
|
-
authenticate_sanctuary_provider "$IMAGE_ID" openai-compatible-gemini
|
|
1050
1399
|
verify_sanctuary_provider_readiness "$IMAGE_ID"
|
|
1051
1400
|
install_from_legacy_staging
|
|
1052
1401
|
These commands are one terminal path; after install succeeds, stop and do not
|
|
@@ -1068,14 +1417,19 @@ Update:
|
|
|
1068
1417
|
Failure removes only partial target-image staging/production containers,
|
|
1069
1418
|
quarantines the exact legacy container without deletion or restart, leaves
|
|
1070
1419
|
autostart disabled, and propagates the original status.
|
|
1071
|
-
For
|
|
1072
|
-
|
|
1420
|
+
For this cutover, preflight either the exact known alpha.742 two-mount source
|
|
1421
|
+
or an already-canonical source before any autostart or container mutation.
|
|
1422
|
+
Production must be the only running Butler poller;
|
|
1073
1423
|
staging must be absent; rollback may be absent or one stopped container with
|
|
1074
1424
|
the exact production image. A stopped legacy-evidence container is preserved.
|
|
1075
1425
|
Disable every Butler name in Unraid's array-autostart file and verify that
|
|
1076
1426
|
result before stopping production. First resolve and validate the exact image
|
|
1077
1427
|
ID of the known-good production container while it is still running, so a
|
|
1078
1428
|
lookup failure cannot strand a renamed container:
|
|
1429
|
+
recover_pending_sanctuary_bundle_migration "$IMAGE_ID"
|
|
1430
|
+
/bin/bash /boot/config/custom/ouro-events/bootstrap-spool.sh --mount
|
|
1431
|
+
test "$(findmnt -n -o FSTYPE --target /boot/config/custom/ouro-events/spool)" = tmpfs
|
|
1432
|
+
test "$(stat -c '%u:%g:%a' /boot/config/custom/ouro-events/spool)" = 0:0:755
|
|
1079
1433
|
ROLLBACK_IMAGE_ID=$(docker inspect --format '{{.Image}}' ouro-butler)
|
|
1080
1434
|
validate_exact_image_id "$ROLLBACK_IMAGE_ID"
|
|
1081
1435
|
if assert_update_topology "$ROLLBACK_IMAGE_ID"; then
|
|
@@ -1084,20 +1438,18 @@ Update:
|
|
|
1084
1438
|
UPDATE_PREFLIGHT_STATUS=$?
|
|
1085
1439
|
(exit "$UPDATE_PREFLIGHT_STATUS")
|
|
1086
1440
|
fi
|
|
1087
|
-
|
|
1088
|
-
|
|
1089
|
-
|
|
1090
|
-
|
|
1091
|
-
|
|
1092
|
-
Replace sha256:REPLACE_WITH_EXACT_LOCAL_IMAGE_ID in the staged template at
|
|
1093
|
-
$STAGED_TEMPLATE with exactly $IMAGE_ID. Before docker create, run the
|
|
1094
|
-
packaged auditor from that exact image ID with both staged inputs mounted
|
|
1095
|
-
read-only:
|
|
1441
|
+
assert_update_source "$ROLLBACK_IMAGE_ID" "$AUDIT_RUNNER_IMAGE_ID"
|
|
1442
|
+
The earlier exact-image asset stage has already replaced the template's sole
|
|
1443
|
+
Repository element with $IMAGE_ID. After the topology and source gates pass,
|
|
1444
|
+
audit the staged template and runtime policy with the packaged auditor, then
|
|
1445
|
+
remove the private stage before any Butler mutation:
|
|
1096
1446
|
docker run --rm --pull=never --network=none \
|
|
1097
1447
|
--entrypoint /opt/ouro/deploy/unraid/audit-container-spec.sh \
|
|
1098
1448
|
--mount "type=bind,src=$STAGED_TEMPLATE,dst=/audit/sanctuary.xml,readonly" \
|
|
1099
1449
|
--mount "type=bind,src=$STAGED_RUNTIME_POLICY,dst=/audit/container-runtime.json,readonly" \
|
|
1100
1450
|
"$IMAGE_ID" --template /audit/sanctuary.xml --runtime-policy /audit/container-runtime.json --expected-image "$IMAGE_ID"
|
|
1451
|
+
cleanup_event_asset_stage
|
|
1452
|
+
trap - EXIT
|
|
1101
1453
|
Guard the atomic autostart disable separately. If it fails, production has not
|
|
1102
1454
|
been touched and the captured status is propagated:
|
|
1103
1455
|
if disable_butler_autostart; then
|
|
@@ -1125,11 +1477,37 @@ ouro-butler-rollback
|
|
|
1125
1477
|
esac
|
|
1126
1478
|
}
|
|
1127
1479
|
Stop production, remove only a stopped stale rollback, rename the known-good
|
|
1128
|
-
container,
|
|
1480
|
+
container, verify it remains stopped, and apply the exact target image's
|
|
1481
|
+
package-managed bundle migration in one explicit preparation guard.
|
|
1482
|
+
Package-managed files are exactly `provider-readiness.json`,
|
|
1483
|
+
`tool-profiles.json`, `habits/sanctuary-health.md`, and the five canonical
|
|
1484
|
+
files under `psyche/`. The migration also merges the three bundle-meta version
|
|
1485
|
+
fields. It never installs or mutates steward policy authority. It preserves
|
|
1486
|
+
agent.json, all steward policy and audit bytes, relationships, sessions, and
|
|
1487
|
+
every other state path. Repeating it is a no-op.
|
|
1488
|
+
Before its first managed write, the migrator atomically fsyncs one mode-0600
|
|
1489
|
+
rollback record at `.sanctuary-package-managed-rollback.json`. That transient
|
|
1490
|
+
record contains a verified SHA-256 digest over the exact prior bytes, modes,
|
|
1491
|
+
and parent existence for those
|
|
1492
|
+
files plus bundle-meta, bound to distinct exact rollback and target image IDs.
|
|
1493
|
+
It is the migration transaction record, not a second bundle authority. Normal
|
|
1494
|
+
managed-file failure restores it immediately; every
|
|
1495
|
+
later container rollback restores it before the old container restarts, while
|
|
1496
|
+
successful production readiness and autostart commit it. A killed updater is
|
|
1497
|
+
reconciled from the same record before update topology preflight: partial new
|
|
1498
|
+
containers are removed, interrupted atomic-write stages are discarded, the
|
|
1499
|
+
exact prior bundle is restored idempotently, and only the image ID recorded in
|
|
1500
|
+
the transaction may resume as production. A malformed record, wrong image,
|
|
1501
|
+
or restore failure leaves autostart disabled and fails closed. Once target
|
|
1502
|
+
readiness and autostart have passed, commit first durably renames the record;
|
|
1503
|
+
a kill then keeps target production in place and the next run validates its
|
|
1504
|
+
exact topology/readiness before finishing commit. Migration failure then enters the same exact container
|
|
1505
|
+
rollback arm before staging starts:
|
|
1129
1506
|
if docker stop ouro-butler \
|
|
1130
1507
|
&& remove_stopped_rollback_if_present "$ROLLBACK_IMAGE_ID" \
|
|
1131
1508
|
&& docker rename ouro-butler ouro-butler-rollback \
|
|
1132
|
-
&& test "$(docker inspect --format '{{.State.Running}}' ouro-butler-rollback)" = false
|
|
1509
|
+
&& test "$(docker inspect --format '{{.State.Running}}' ouro-butler-rollback)" = false \
|
|
1510
|
+
&& migrate_sanctuary_package_managed_bundle "$IMAGE_ID" migrate "$ROLLBACK_IMAGE_ID"; then
|
|
1133
1511
|
:
|
|
1134
1512
|
else
|
|
1135
1513
|
PRODUCTION_PREPARATION_STATUS=$?
|
|
@@ -1141,26 +1519,30 @@ ouro-butler-rollback
|
|
|
1141
1519
|
docker rm --force ouro-butler-rollback >/dev/null 2>&1 || true
|
|
1142
1520
|
fi
|
|
1143
1521
|
! docker container inspect ouro-butler-rollback >/dev/null 2>&1
|
|
1144
|
-
|
|
1522
|
+
rollback_sanctuary_bundle_if_pending "$IMAGE_ID"
|
|
1523
|
+
assert_update_source "$ROLLBACK_IMAGE_ID" "$AUDIT_RUNNER_IMAGE_ID"
|
|
1145
1524
|
docker start ouro-butler
|
|
1146
1525
|
assert_only_running_butler ouro-butler
|
|
1147
1526
|
wait_butler_ready ouro-butler
|
|
1148
1527
|
enable_butler_autostart
|
|
1528
|
+
migrate_sanctuary_package_managed_bundle "$IMAGE_ID" commit
|
|
1149
1529
|
elif docker container inspect ouro-butler-rollback >/dev/null 2>&1; then
|
|
1150
1530
|
docker stop ouro-butler-rollback >/dev/null 2>&1 || true
|
|
1151
1531
|
test "$(docker inspect --format '{{.Image}}' ouro-butler-rollback)" = "$ROLLBACK_IMAGE_ID"
|
|
1532
|
+
rollback_sanctuary_bundle_if_pending "$IMAGE_ID"
|
|
1152
1533
|
docker rename ouro-butler-rollback ouro-butler
|
|
1153
|
-
|
|
1534
|
+
assert_update_source "$ROLLBACK_IMAGE_ID" "$AUDIT_RUNNER_IMAGE_ID"
|
|
1154
1535
|
docker start ouro-butler
|
|
1155
1536
|
assert_only_running_butler ouro-butler
|
|
1156
1537
|
wait_butler_ready ouro-butler
|
|
1157
1538
|
enable_butler_autostart
|
|
1539
|
+
migrate_sanctuary_package_managed_bundle "$IMAGE_ID" commit
|
|
1158
1540
|
fi
|
|
1159
1541
|
(exit "$PRODUCTION_PREPARATION_STATUS")
|
|
1160
1542
|
fi
|
|
1161
1543
|
Preparation failure therefore either restores the still-named exact production
|
|
1162
1544
|
after removing any stale rollback, or renames the exact stopped rollback back;
|
|
1163
|
-
both recoveries
|
|
1545
|
+
both recoveries revalidate, start, bounded-wait, atomically restore production-only
|
|
1164
1546
|
autostart, and propagate the original failure. If neither exact container can
|
|
1165
1547
|
be found, the failure propagates with Butler autostart disabled.
|
|
1166
1548
|
Put the entire post-rename staging phase in one explicit conditional so
|
|
@@ -1172,8 +1554,10 @@ ouro-butler-rollback
|
|
|
1172
1554
|
if docker create --name ouro-butler-staging --network host --restart unless-stopped --user 10001:10001 \
|
|
1173
1555
|
--mount "type=bind,src=/mnt/user/appdata/ouro-butler/runtime/.ouro-cli,dst=/home/ouro/.ouro-cli" \
|
|
1174
1556
|
--mount "type=bind,src=/mnt/user/appdata/ouro-butler/agent/sanctuary.ouro,dst=/home/ouro/AgentBundles/sanctuary.ouro" \
|
|
1557
|
+
--mount "type=bind,src=/boot/config/custom/ouro-events/spool,dst=/run/ouro-events,readonly" \
|
|
1558
|
+
--mount "type=bind,src=/mnt/user/appdata/sabnzbd/sabnzbd.ini,dst=/run/sanctuary/sabnzbd.ini,readonly" \
|
|
1175
1559
|
"$IMAGE_ID" \
|
|
1176
|
-
&& audit_effective ouro-butler-staging "$IMAGE_ID" \
|
|
1560
|
+
&& audit_effective ouro-butler-staging "$IMAGE_ID" "$AUDIT_RUNNER_IMAGE_ID" \
|
|
1177
1561
|
&& docker start ouro-butler-staging \
|
|
1178
1562
|
&& assert_only_running_butler ouro-butler-staging \
|
|
1179
1563
|
&& wait_butler_ready ouro-butler-staging \
|
|
@@ -1191,17 +1575,19 @@ ouro-butler-rollback
|
|
|
1191
1575
|
! docker container inspect ouro-butler-staging >/dev/null 2>&1
|
|
1192
1576
|
CURRENT_ROLLBACK_IMAGE_ID=$(docker inspect --format '{{.Image}}' ouro-butler-rollback)
|
|
1193
1577
|
test "$CURRENT_ROLLBACK_IMAGE_ID" = "$ROLLBACK_IMAGE_ID"
|
|
1578
|
+
migrate_sanctuary_package_managed_bundle "$IMAGE_ID" rollback
|
|
1194
1579
|
docker rename ouro-butler-rollback ouro-butler
|
|
1195
|
-
|
|
1580
|
+
assert_update_source "$ROLLBACK_IMAGE_ID" "$AUDIT_RUNNER_IMAGE_ID"
|
|
1196
1581
|
docker start ouro-butler
|
|
1197
1582
|
assert_only_running_butler ouro-butler
|
|
1198
1583
|
wait_butler_ready ouro-butler
|
|
1199
1584
|
enable_butler_autostart
|
|
1585
|
+
migrate_sanctuary_package_managed_bundle "$IMAGE_ID" commit
|
|
1200
1586
|
(exit "$STAGING_ACTIVATION_STATUS")
|
|
1201
1587
|
fi
|
|
1202
1588
|
The failure arm safely handles staging that was never created, remains
|
|
1203
1589
|
stopped, is running, or exited: it force-removes any partial staging state,
|
|
1204
|
-
verifies the name is absent, restores and
|
|
1590
|
+
verifies the name is absent, restores and revalidates the old production against
|
|
1205
1591
|
its exact pre-recorded image ID, starts and bounded-waits it, atomically
|
|
1206
1592
|
restores production-only autostart, and propagates the original failure.
|
|
1207
1593
|
At no point may production and staging run together against the same Telegram token.
|
|
@@ -1216,8 +1602,10 @@ ouro-butler-rollback
|
|
|
1216
1602
|
if docker create --name ouro-butler --network host --restart unless-stopped --user 10001:10001 \
|
|
1217
1603
|
--mount "type=bind,src=/mnt/user/appdata/ouro-butler/runtime/.ouro-cli,dst=/home/ouro/.ouro-cli" \
|
|
1218
1604
|
--mount "type=bind,src=/mnt/user/appdata/ouro-butler/agent/sanctuary.ouro,dst=/home/ouro/AgentBundles/sanctuary.ouro" \
|
|
1605
|
+
--mount "type=bind,src=/boot/config/custom/ouro-events/spool,dst=/run/ouro-events,readonly" \
|
|
1606
|
+
--mount "type=bind,src=/mnt/user/appdata/sabnzbd/sabnzbd.ini,dst=/run/sanctuary/sabnzbd.ini,readonly" \
|
|
1219
1607
|
"$IMAGE_ID" \
|
|
1220
|
-
&& audit_effective ouro-butler "$IMAGE_ID" \
|
|
1608
|
+
&& audit_effective ouro-butler "$IMAGE_ID" "$AUDIT_RUNNER_IMAGE_ID" \
|
|
1221
1609
|
&& docker start ouro-butler \
|
|
1222
1610
|
&& assert_only_running_butler ouro-butler \
|
|
1223
1611
|
&& test "$(docker inspect --format '{{.State.Running}}' ouro-butler-rollback)" = false \
|
|
@@ -1235,42 +1623,259 @@ ouro-butler-rollback
|
|
|
1235
1623
|
! docker container inspect ouro-butler >/dev/null 2>&1
|
|
1236
1624
|
CURRENT_ROLLBACK_IMAGE_ID=$(docker inspect --format '{{.Image}}' ouro-butler-rollback)
|
|
1237
1625
|
test "$CURRENT_ROLLBACK_IMAGE_ID" = "$ROLLBACK_IMAGE_ID"
|
|
1626
|
+
migrate_sanctuary_package_managed_bundle "$IMAGE_ID" rollback
|
|
1238
1627
|
docker rename ouro-butler-rollback ouro-butler
|
|
1239
|
-
|
|
1628
|
+
assert_update_source "$ROLLBACK_IMAGE_ID" "$AUDIT_RUNNER_IMAGE_ID"
|
|
1240
1629
|
docker start ouro-butler
|
|
1241
1630
|
assert_only_running_butler ouro-butler
|
|
1242
1631
|
wait_butler_ready ouro-butler
|
|
1243
1632
|
enable_butler_autostart
|
|
1633
|
+
migrate_sanctuary_package_managed_bundle "$IMAGE_ID" commit
|
|
1244
1634
|
(exit "$PRODUCTION_ACTIVATION_STATUS")
|
|
1245
1635
|
fi
|
|
1636
|
+
migrate_sanctuary_package_managed_bundle "$IMAGE_ID" commit
|
|
1246
1637
|
Keep ouro-butler-rollback stopped until the new production container is proven
|
|
1247
1638
|
or the explicit rollback arm restores it.
|
|
1248
1639
|
Never create a container from the mutable lookup tag.
|
|
1249
1640
|
|
|
1250
1641
|
Backup:
|
|
1251
|
-
|
|
1642
|
+
Set BACKUP_ROOT to a new absolute snapshot path on the destination filesystem.
|
|
1643
|
+
Set AUDIT_RUNNER_IMAGE_TAG to the reviewed new image containing the
|
|
1644
|
+
legacy-aware auditor; the alpha.742 source image cannot be the runner.
|
|
1645
|
+
Capture and validate the exact source image before stopping ouro-butler, then
|
|
1646
|
+
build a temporary snapshot with root-only provenance and atomically rename it into place:
|
|
1647
|
+
test -n "${BACKUP_ROOT-}"
|
|
1648
|
+
case "$BACKUP_ROOT" in /*) ;; *) exit 1 ;; esac
|
|
1649
|
+
test "$BACKUP_ROOT" != /
|
|
1650
|
+
test ! -e "$BACKUP_ROOT"
|
|
1651
|
+
BACKUP_PARENT=$(dirname -- "$BACKUP_ROOT")
|
|
1652
|
+
test "$(cd -- "$BACKUP_PARENT" && pwd -P)" = "$BACKUP_PARENT"
|
|
1653
|
+
BACKUP_TMP=$BACKUP_ROOT.tmp.$$
|
|
1654
|
+
test ! -e "$BACKUP_TMP"
|
|
1655
|
+
AUDIT_RUNNER_IMAGE_TAG=ouro-butler:<new-version>
|
|
1656
|
+
AUDIT_RUNNER_IMAGE_ID=$(docker image inspect --format '{{.Id}}' "$AUDIT_RUNNER_IMAGE_TAG")
|
|
1657
|
+
validate_exact_image_id "$AUDIT_RUNNER_IMAGE_ID"
|
|
1658
|
+
test "$AUDIT_RUNNER_IMAGE_ID" != sha256:681449ad47a2621705cd339b481e6339236b31dc65e195b1cf5025d0f2191d7d
|
|
1659
|
+
BACKUP_IMAGE_ID=$(docker inspect --format '{{.Image}}' ouro-butler)
|
|
1660
|
+
validate_exact_image_id "$BACKUP_IMAGE_ID"
|
|
1661
|
+
assert_update_source "$BACKUP_IMAGE_ID" "$AUDIT_RUNNER_IMAGE_ID"
|
|
1662
|
+
assert_only_running_butler ouro-butler
|
|
1663
|
+
wait_butler_ready ouro-butler
|
|
1664
|
+
BACKUP_AUTOSTART_COUNTS=$(butler_autostart_counts)
|
|
1665
|
+
case "$BACKUP_AUTOSTART_COUNTS" in "0 0 0 0"|"1 0 0 0") ;; *) exit 1 ;; esac
|
|
1666
|
+
backup_host_file() {
|
|
1667
|
+
BACKUP_HOST_SOURCE=$1
|
|
1668
|
+
BACKUP_HOST_RELATIVE=$2
|
|
1669
|
+
case "$BACKUP_HOST_RELATIVE" in
|
|
1670
|
+
custom/usenet_health.sh|custom/ouro-events/bootstrap-spool.sh|custom/ouro-events/emit-event.mjs|custom/ouro-events/emit-usenet-event.sh|custom/ouro-events/install-usenet-guard.sh) ;;
|
|
1671
|
+
*) return 1 ;;
|
|
1672
|
+
esac
|
|
1673
|
+
if test -e "$BACKUP_HOST_SOURCE" || test -L "$BACKUP_HOST_SOURCE"; then
|
|
1674
|
+
test -f "$BACKUP_HOST_SOURCE" && test ! -L "$BACKUP_HOST_SOURCE" || return 1
|
|
1675
|
+
! host_file_contains_inline_credential "$BACKUP_HOST_SOURCE" || return 1
|
|
1676
|
+
install -D -m 0600 -o 0 -g 0 "$BACKUP_HOST_SOURCE" "$BACKUP_TMP/host/$BACKUP_HOST_RELATIVE" || return $?
|
|
1677
|
+
printf 'present\t%s\n' "$BACKUP_HOST_RELATIVE" >>"$BACKUP_TMP/host/inventory" || return $?
|
|
1678
|
+
else
|
|
1679
|
+
printf 'absent\t%s\n' "$BACKUP_HOST_RELATIVE" >>"$BACKUP_TMP/host/inventory" || return $?
|
|
1680
|
+
fi
|
|
1681
|
+
}
|
|
1682
|
+
host_file_contains_inline_credential() {
|
|
1683
|
+
LC_ALL=C awk '
|
|
1684
|
+
BEGIN { IGNORECASE = 1; found = 0 }
|
|
1685
|
+
/[0-9]{6,}:[A-Za-z0-9_-]{20,}/ { found = 1 }
|
|
1686
|
+
/Authorization:[[:space:]]*(Bearer|Basic)[[:space:]]+["\047]?[A-Za-z0-9][A-Za-z0-9._~+\/=:-]{11,}/ { found = 1 }
|
|
1687
|
+
/--(token|api-key|api_key|secret|password)(=|[[:space:]]+)["\047]?[A-Za-z0-9][A-Za-z0-9._~+\/=:-]{11,}/ { found = 1 }
|
|
1688
|
+
/[A-Za-z][A-Za-z0-9+.-]*:\/\/[^\/@[:space:]]+:[^\/@[:space:]]+@/ { found = 1 }
|
|
1689
|
+
/-----BEGIN ([A-Z0-9]+ )*PRIVATE KEY-----/ { found = 1 }
|
|
1690
|
+
/(token|secret|password|pass|api[_-]?key)[A-Za-z0-9_-]*["\047[:space:]]*[:=][[:space:]]*["\047]?[A-Za-z0-9][A-Za-z0-9._~+\/=:-]{11,}/ { found = 1 }
|
|
1691
|
+
END { exit found ? 0 : 1 }
|
|
1692
|
+
' "$1"
|
|
1693
|
+
}
|
|
1694
|
+
snapshot_butler_go_fragments() {
|
|
1695
|
+
BACKUP_GO_SOURCE=$1
|
|
1696
|
+
BACKUP_GO_TARGET=$2
|
|
1697
|
+
: >"$BACKUP_GO_TARGET" || return $?
|
|
1698
|
+
while IFS= read -r BACKUP_GO_LINE || test -n "$BACKUP_GO_LINE"; do
|
|
1699
|
+
case "$BACKUP_GO_LINE" in
|
|
1700
|
+
"/boot/config/custom/ouro-events/bootstrap-spool.sh --mount"|"/bin/bash /boot/config/custom/ouro-events/bootstrap-spool.sh --mount"|"/bin/bash /boot/config/custom/ouro-events/install-usenet-guard.sh --boot"|"/bin/bash /boot/config/custom/ouro-events/install-usenet-guard.sh --boot --install-root /boot/config/custom")
|
|
1701
|
+
printf '%s\n' "$BACKUP_GO_LINE" >>"$BACKUP_GO_TARGET" || return $?
|
|
1702
|
+
;;
|
|
1703
|
+
esac
|
|
1704
|
+
done <"$BACKUP_GO_SOURCE"
|
|
1705
|
+
}
|
|
1706
|
+
snapshot_butler_cron_fragments() {
|
|
1707
|
+
BACKUP_CRON_INPUT=$1
|
|
1708
|
+
BACKUP_CRON_TARGET=$2
|
|
1709
|
+
: >"$BACKUP_CRON_TARGET" || return $?
|
|
1710
|
+
while IFS= read -r BACKUP_CRON_LINE || test -n "$BACKUP_CRON_LINE"; do
|
|
1711
|
+
case "$BACKUP_CRON_LINE" in
|
|
1712
|
+
"*/15 * * * * /bin/bash /boot/config/custom/usenet_health.sh"|"*/15 * * * * /bin/bash /boot/config/custom/usenet_health.sh # ouro:usenet-health")
|
|
1713
|
+
printf '%s\n' "$BACKUP_CRON_LINE" >>"$BACKUP_CRON_TARGET" || return $?
|
|
1714
|
+
;;
|
|
1715
|
+
esac
|
|
1716
|
+
done <"$BACKUP_CRON_INPUT"
|
|
1717
|
+
}
|
|
1718
|
+
create_private_cron_capture() {
|
|
1719
|
+
BACKUP_CRON_ROOT=$(mktemp -d /tmp/ouro-backup-cron.XXXXXX) || return $?
|
|
1720
|
+
BACKUP_CRON_SOURCE=$BACKUP_CRON_ROOT/stdout
|
|
1721
|
+
BACKUP_CRON_ERROR=$BACKUP_CRON_ROOT/stderr
|
|
1722
|
+
test -d "$BACKUP_CRON_ROOT" && test ! -L "$BACKUP_CRON_ROOT" || return 1
|
|
1723
|
+
test "$(stat -c '%u:%a' "$BACKUP_CRON_ROOT" 2>/dev/null || stat -f '%u:%Lp' "$BACKUP_CRON_ROOT")" = "$(id -u):700" || return 1
|
|
1724
|
+
install -m 0600 /dev/null "$BACKUP_CRON_SOURCE" || return $?
|
|
1725
|
+
install -m 0600 /dev/null "$BACKUP_CRON_ERROR" || return $?
|
|
1726
|
+
test -f "$BACKUP_CRON_SOURCE" && test ! -L "$BACKUP_CRON_SOURCE" || return 1
|
|
1727
|
+
test -f "$BACKUP_CRON_ERROR" && test ! -L "$BACKUP_CRON_ERROR" || return 1
|
|
1728
|
+
test "$(stat -c '%u:%a' "$BACKUP_CRON_SOURCE" 2>/dev/null || stat -f '%u:%Lp' "$BACKUP_CRON_SOURCE")" = "$(id -u):600" || return 1
|
|
1729
|
+
test "$(stat -c '%u:%a' "$BACKUP_CRON_ERROR" 2>/dev/null || stat -f '%u:%Lp' "$BACKUP_CRON_ERROR")" = "$(id -u):600" || return 1
|
|
1730
|
+
}
|
|
1731
|
+
cleanup_private_cron_capture() {
|
|
1732
|
+
rm -f -- "$BACKUP_CRON_SOURCE" "$BACKUP_CRON_ERROR" || return $?
|
|
1733
|
+
rmdir -- "$BACKUP_CRON_ROOT"
|
|
1734
|
+
}
|
|
1735
|
+
snapshot_butler_host_fragments() {
|
|
1736
|
+
create_private_cron_capture || return $?
|
|
1737
|
+
trap 'cleanup_private_cron_capture' RETURN
|
|
1738
|
+
if test -f /boot/config/go && test ! -L /boot/config/go; then
|
|
1739
|
+
BACKUP_GO_STATE=present
|
|
1740
|
+
BACKUP_GO_DIGEST=$(sha256sum /boot/config/go | awk '{print $1}') || return $?
|
|
1741
|
+
snapshot_butler_go_fragments /boot/config/go "$BACKUP_TMP/host/go.butler-lines" || return $?
|
|
1742
|
+
elif test -e /boot/config/go || test -L /boot/config/go; then
|
|
1743
|
+
return 1
|
|
1744
|
+
else
|
|
1745
|
+
BACKUP_GO_STATE=absent
|
|
1746
|
+
BACKUP_GO_DIGEST=-
|
|
1747
|
+
: >"$BACKUP_TMP/host/go.butler-lines" || return $?
|
|
1748
|
+
fi
|
|
1749
|
+
BACKUP_CRON_USER=$(id -un) || return $?
|
|
1750
|
+
if crontab -l >"$BACKUP_CRON_SOURCE" 2>"$BACKUP_CRON_ERROR"; then
|
|
1751
|
+
BACKUP_CRON_STATE=present
|
|
1752
|
+
BACKUP_CRON_DIGEST=$(sha256sum "$BACKUP_CRON_SOURCE" | awk '{print $1}') || return $?
|
|
1753
|
+
else
|
|
1754
|
+
BACKUP_CRON_STATUS=$?
|
|
1755
|
+
if test "$BACKUP_CRON_STATUS" -eq 1 && grep -Fxq "no crontab for $BACKUP_CRON_USER" "$BACKUP_CRON_ERROR"; then
|
|
1756
|
+
BACKUP_CRON_STATE=absent
|
|
1757
|
+
BACKUP_CRON_DIGEST=-
|
|
1758
|
+
: >"$BACKUP_CRON_SOURCE" || return $?
|
|
1759
|
+
else
|
|
1760
|
+
return "$BACKUP_CRON_STATUS"
|
|
1761
|
+
fi
|
|
1762
|
+
fi
|
|
1763
|
+
rm -f -- "$BACKUP_CRON_ERROR" || return $?
|
|
1764
|
+
snapshot_butler_cron_fragments "$BACKUP_CRON_SOURCE" "$BACKUP_TMP/host/crontab.butler-lines" || return $?
|
|
1765
|
+
BACKUP_GO_COUNT=$(awk 'END { print NR + 0 }' "$BACKUP_TMP/host/go.butler-lines") || return $?
|
|
1766
|
+
BACKUP_CRON_COUNT=$(awk 'END { print NR + 0 }' "$BACKUP_TMP/host/crontab.butler-lines") || return $?
|
|
1767
|
+
printf 'go\t%s\t%s\t%s\ncrontab\t%s\t%s\t%s\n' "$BACKUP_GO_STATE" "$BACKUP_GO_DIGEST" "$BACKUP_GO_COUNT" "$BACKUP_CRON_STATE" "$BACKUP_CRON_DIGEST" "$BACKUP_CRON_COUNT" >"$BACKUP_TMP/host/global-state" || return $?
|
|
1768
|
+
chown 0:0 "$BACKUP_TMP/host/go.butler-lines" "$BACKUP_TMP/host/crontab.butler-lines" "$BACKUP_TMP/host/global-state" || return $?
|
|
1769
|
+
chmod 0600 "$BACKUP_TMP/host/go.butler-lines" "$BACKUP_TMP/host/crontab.butler-lines" "$BACKUP_TMP/host/global-state" || return $?
|
|
1770
|
+
printf 'present\t%s\n' go.butler-lines crontab.butler-lines global-state >>"$BACKUP_TMP/host/inventory" || return $?
|
|
1771
|
+
cleanup_private_cron_capture || return $?
|
|
1772
|
+
trap - RETURN
|
|
1773
|
+
}
|
|
1774
|
+
Snapshot both of these directories together:
|
|
1252
1775
|
/mnt/user/appdata/ouro-butler/runtime/.ouro-cli
|
|
1253
1776
|
/mnt/user/appdata/ouro-butler/agent/sanctuary.ouro
|
|
1254
1777
|
A routine backup must not contain container-credentials.json or
|
|
1255
1778
|
container-credentials.json.consuming. If either exists, credential migration
|
|
1256
1779
|
is pending or failed and must be reconciled before taking the backup.
|
|
1780
|
+
The Telegram control socket is live process state, not durable data. Exclude
|
|
1781
|
+
exactly that socket while preserving every regular file byte-for-byte, then
|
|
1782
|
+
require it to be absent from the stopped backup:
|
|
1783
|
+
BACKUP_OPERATION_STATUS=0
|
|
1784
|
+
if docker stop ouro-butler \
|
|
1785
|
+
&& test "$(docker inspect --format '{{.State.Running}}' ouro-butler)" = false \
|
|
1786
|
+
&& install -d -m 0700 -o 0 -g 0 "$BACKUP_TMP" "$BACKUP_TMP/runtime" "$BACKUP_TMP/agent" "$BACKUP_TMP/host" "$BACKUP_TMP/provenance" \
|
|
1787
|
+
&& install -d -m 0700 -o 0 -g 0 "$BACKUP_TMP/host/custom/ouro-events" \
|
|
1788
|
+
&& install -m 0600 -o 0 -g 0 /dev/null "$BACKUP_TMP/host/inventory" \
|
|
1789
|
+
&& backup_host_file /boot/config/custom/usenet_health.sh custom/usenet_health.sh \
|
|
1790
|
+
&& backup_host_file /boot/config/custom/ouro-events/bootstrap-spool.sh custom/ouro-events/bootstrap-spool.sh \
|
|
1791
|
+
&& backup_host_file /boot/config/custom/ouro-events/emit-event.mjs custom/ouro-events/emit-event.mjs \
|
|
1792
|
+
&& backup_host_file /boot/config/custom/ouro-events/emit-usenet-event.sh custom/ouro-events/emit-usenet-event.sh \
|
|
1793
|
+
&& backup_host_file /boot/config/custom/ouro-events/install-usenet-guard.sh custom/ouro-events/install-usenet-guard.sh \
|
|
1794
|
+
&& snapshot_butler_host_fragments \
|
|
1795
|
+
&& test ! -e "$BACKUP_TMP/host/notify.conf" \
|
|
1796
|
+
&& test ! -e "$BACKUP_TMP/host/sabnzbd.ini" \
|
|
1797
|
+
&& rsync -a --exclude='/state/acceptance/telegram-control.sock' /mnt/user/appdata/ouro-butler/runtime/.ouro-cli "$BACKUP_TMP/runtime/" \
|
|
1798
|
+
&& rsync -a --exclude='/state/acceptance/telegram-control.sock' /mnt/user/appdata/ouro-butler/agent/sanctuary.ouro "$BACKUP_TMP/agent/" \
|
|
1799
|
+
&& test ! -S "$BACKUP_TMP/agent/sanctuary.ouro/state/acceptance/telegram-control.sock" \
|
|
1800
|
+
&& validate_sanctuary_roots "$BACKUP_TMP/runtime/.ouro-cli" "$BACKUP_TMP/agent/sanctuary.ouro" \
|
|
1801
|
+
&& docker container inspect ouro-butler >"$BACKUP_TMP/provenance/container-inspect.json" \
|
|
1802
|
+
&& docker image inspect "$BACKUP_IMAGE_ID" >"$BACKUP_TMP/provenance/image-inspect.json" \
|
|
1803
|
+
&& docker run --rm --pull=never --network=none --read-only --entrypoint /usr/local/bin/node "$BACKUP_IMAGE_ID" -p 'require("/opt/ouro/package.json").version' >"$BACKUP_TMP/provenance/package-version" \
|
|
1804
|
+
&& printf '%s\n' "$BACKUP_IMAGE_ID" >"$BACKUP_TMP/provenance/image-id" \
|
|
1805
|
+
&& chown 0:0 "$BACKUP_TMP/provenance/container-inspect.json" "$BACKUP_TMP/provenance/image-inspect.json" "$BACKUP_TMP/provenance/package-version" "$BACKUP_TMP/provenance/image-id" \
|
|
1806
|
+
&& chmod 0600 "$BACKUP_TMP/provenance/container-inspect.json" "$BACKUP_TMP/provenance/image-inspect.json" "$BACKUP_TMP/provenance/package-version" "$BACKUP_TMP/provenance/image-id" \
|
|
1807
|
+
&& (cd "$BACKUP_TMP" && find runtime agent host provenance -xdev -type f ! -path provenance/manifest.sha256 -exec sha256sum -- {} + | LC_ALL=C sort -k2 >provenance/manifest.sha256) \
|
|
1808
|
+
&& chown 0:0 "$BACKUP_TMP/provenance/manifest.sha256" \
|
|
1809
|
+
&& chmod 0600 "$BACKUP_TMP/provenance/manifest.sha256" \
|
|
1810
|
+
&& verify_sanctuary_snapshot_provenance "$BACKUP_TMP" "$BACKUP_IMAGE_ID" \
|
|
1811
|
+
&& sync -f "$BACKUP_TMP/provenance/manifest.sha256" \
|
|
1812
|
+
&& mv -- "$BACKUP_TMP" "$BACKUP_ROOT" \
|
|
1813
|
+
&& sync -f "$BACKUP_PARENT"; then
|
|
1814
|
+
:
|
|
1815
|
+
else
|
|
1816
|
+
BACKUP_OPERATION_STATUS=$?
|
|
1817
|
+
fi
|
|
1818
|
+
if assert_update_source "$BACKUP_IMAGE_ID" "$AUDIT_RUNNER_IMAGE_ID" \
|
|
1819
|
+
&& docker start ouro-butler \
|
|
1820
|
+
&& assert_only_running_butler ouro-butler \
|
|
1821
|
+
&& wait_butler_ready ouro-butler \
|
|
1822
|
+
&& verify_butler_autostart "$BACKUP_AUTOSTART_COUNTS"; then
|
|
1823
|
+
:
|
|
1824
|
+
else
|
|
1825
|
+
BACKUP_RECOVERY_STATUS=$?
|
|
1826
|
+
if test -d "$BACKUP_ROOT"; then
|
|
1827
|
+
printf '%s\n' "CRITICAL: production recovery failed after backup; completed snapshot remains intact at $BACKUP_ROOT" >&2
|
|
1828
|
+
else
|
|
1829
|
+
printf '%s\n' "CRITICAL: production recovery failed and no completed snapshot was published; inspect $BACKUP_TMP" >&2
|
|
1830
|
+
fi
|
|
1831
|
+
(exit "$BACKUP_RECOVERY_STATUS")
|
|
1832
|
+
fi
|
|
1833
|
+
if test "$BACKUP_OPERATION_STATUS" -ne 0; then
|
|
1834
|
+
if test -d "$BACKUP_ROOT"; then
|
|
1835
|
+
printf '%s\n' "Backup publication was not durably confirmed; production was recovered; completed snapshot remains intact at $BACKUP_ROOT" >&2
|
|
1836
|
+
else
|
|
1837
|
+
printf '%s\n' "Backup failed before atomic publication; production was recovered; inspect $BACKUP_TMP" >&2
|
|
1838
|
+
fi
|
|
1839
|
+
(exit "$BACKUP_OPERATION_STATUS")
|
|
1840
|
+
fi
|
|
1257
1841
|
|
|
1258
1842
|
Restore:
|
|
1259
1843
|
Set BACKUP_ROOT to the exact verified snapshot containing `runtime/.ouro-cli`
|
|
1260
|
-
and `agent/sanctuary.ouro`,
|
|
1844
|
+
and `agent/sanctuary.ouro`, set IMAGE_ID to its recorded local image ID, and
|
|
1845
|
+
set AUDIT_RUNNER_IMAGE_ID to the exact reviewed new image containing the
|
|
1846
|
+
legacy-aware auditor. The runner is independent from the restored image.
|
|
1847
|
+
AUDIT_RUNNER_IMAGE_TAG=ouro-butler:<new-version>
|
|
1848
|
+
AUDIT_RUNNER_IMAGE_ID=$(docker image inspect --format '{{.Id}}' "$AUDIT_RUNNER_IMAGE_TAG")
|
|
1261
1849
|
Before any autostart, root, or container mutation, run the nounset-safe input,
|
|
1262
1850
|
backup-root, image, and topology preflight. It requires a nonempty canonical
|
|
1263
1851
|
absolute BACKUP_ROOT (not /), both exact required directories, an exact local
|
|
1264
1852
|
sha256 image ID, canonical production as the only running Butler poller, no
|
|
1265
|
-
staging or rollback, and at most one exact stopped legacy-evidence container
|
|
1853
|
+
staging or rollback, and at most one exact stopped legacy-evidence container.
|
|
1854
|
+
It also executes the reviewed runner against the live source container before
|
|
1855
|
+
any autostart, root, or container mutation:
|
|
1266
1856
|
if assert_restore_preflight; then
|
|
1267
1857
|
:
|
|
1268
1858
|
else
|
|
1269
1859
|
RESTORE_PREFLIGHT_STATUS=$?
|
|
1270
1860
|
(exit "$RESTORE_PREFLIGHT_STATUS")
|
|
1271
1861
|
fi
|
|
1862
|
+
The verified host snapshot is restored only by this explicit Restore path.
|
|
1863
|
+
Ordinary update failure continues to use the installer's short-lived
|
|
1864
|
+
transaction rollback. Extract the installer from the reviewed runner image,
|
|
1865
|
+
then let that fixed allowlist transaction restore exact presence/absence for
|
|
1866
|
+
the five guard assets and reapply only the captured Butler-owned go/crontab
|
|
1867
|
+
fragments. Unrelated current go/crontab lines are never stored in the durable
|
|
1868
|
+
snapshot and remain untouched by restore. The transaction deactivates the
|
|
1869
|
+
current owned cron before swapping files and restores the entire pre-restore
|
|
1870
|
+
host state if any swap or final crontab activation fails:
|
|
1871
|
+
HOST_RESTORE_INSTALLER=$(mktemp /tmp/ouro-usenet-host-restore.XXXXXX)
|
|
1872
|
+
docker run --rm --pull=never --network=none --entrypoint /bin/cat "$AUDIT_RUNNER_IMAGE_ID" /opt/ouro/deploy/unraid/ouro-events/install-usenet-guard.sh >"$HOST_RESTORE_INSTALLER"
|
|
1873
|
+
chmod 0500 "$HOST_RESTORE_INSTALLER"
|
|
1272
1874
|
Guard the atomic autostart disable before stopping or removing any Butler
|
|
1273
1875
|
container. Failure propagates while the existing production remains untouched:
|
|
1876
|
+
/bin/bash /boot/config/custom/ouro-events/bootstrap-spool.sh --mount
|
|
1877
|
+
test "$(findmnt -n -o FSTYPE --target /boot/config/custom/ouro-events/spool)" = tmpfs
|
|
1878
|
+
test "$(stat -c '%u:%g:%a' /boot/config/custom/ouro-events/spool)" = 0:0:755
|
|
1274
1879
|
if disable_butler_autostart; then
|
|
1275
1880
|
:
|
|
1276
1881
|
else
|
|
@@ -1295,12 +1900,16 @@ Restore:
|
|
|
1295
1900
|
&& rsync -a --delete "$BACKUP_ROOT/agent/sanctuary.ouro/" /mnt/user/appdata/ouro-butler/agent/sanctuary.ouro/ \
|
|
1296
1901
|
&& chown -R 10001:10001 /mnt/user/appdata/ouro-butler/runtime/.ouro-cli /mnt/user/appdata/ouro-butler/agent/sanctuary.ouro \
|
|
1297
1902
|
&& validate_sanctuary_roots /mnt/user/appdata/ouro-butler/runtime/.ouro-cli /mnt/user/appdata/ouro-butler/agent/sanctuary.ouro \
|
|
1903
|
+
&& /bin/bash "$HOST_RESTORE_INSTALLER" --restore-root "$BACKUP_ROOT/host" \
|
|
1904
|
+
&& rm -f "$HOST_RESTORE_INSTALLER" \
|
|
1298
1905
|
&& docker image inspect "$IMAGE_ID" >/dev/null \
|
|
1299
1906
|
&& docker create --name ouro-butler --network host --restart unless-stopped --user 10001:10001 \
|
|
1300
1907
|
--mount "type=bind,src=/mnt/user/appdata/ouro-butler/runtime/.ouro-cli,dst=/home/ouro/.ouro-cli" \
|
|
1301
1908
|
--mount "type=bind,src=/mnt/user/appdata/ouro-butler/agent/sanctuary.ouro,dst=/home/ouro/AgentBundles/sanctuary.ouro" \
|
|
1909
|
+
--mount "type=bind,src=/boot/config/custom/ouro-events/spool,dst=/run/ouro-events,readonly" \
|
|
1910
|
+
--mount "type=bind,src=/mnt/user/appdata/sabnzbd/sabnzbd.ini,dst=/run/sanctuary/sabnzbd.ini,readonly" \
|
|
1302
1911
|
"$IMAGE_ID" \
|
|
1303
|
-
&& audit_effective ouro-butler "$IMAGE_ID" \
|
|
1912
|
+
&& audit_effective ouro-butler "$IMAGE_ID" "$AUDIT_RUNNER_IMAGE_ID" \
|
|
1304
1913
|
&& docker start ouro-butler \
|
|
1305
1914
|
&& assert_only_running_butler ouro-butler \
|
|
1306
1915
|
&& wait_butler_ready ouro-butler \
|
|
@@ -1315,6 +1924,7 @@ Restore:
|
|
|
1315
1924
|
docker rm --force ouro-butler >/dev/null 2>&1 || true
|
|
1316
1925
|
fi
|
|
1317
1926
|
! docker container inspect ouro-butler >/dev/null 2>&1
|
|
1927
|
+
rm -f "$HOST_RESTORE_INSTALLER"
|
|
1318
1928
|
(exit "$RESTORE_ACTIVATION_STATUS")
|
|
1319
1929
|
fi
|
|
1320
1930
|
A restore has no retained old container or pre-restore data root to roll back
|
|
@@ -1427,6 +2037,7 @@ Packaged Unit 16 acceptance execution:
|
|
|
1427
2037
|
Never hand-author a Unit 16 config. Materialize it from the exact image, then
|
|
1428
2038
|
execute it in the order below. Before every execution the launcher regenerates
|
|
1429
2039
|
the config from the packaged fixed contract and requires byte-for-byte equality.
|
|
2040
|
+
Unit 16d-2 stops at the pre-model quarantine boundary: use a genuinely distinct private Telegram sender, confirm the fixed acknowledgement and owner admission card, and do not approve the contact during this scenario. The production-identical allow-to-one-turn continuation is covered by the Telegram admission integration suite when a second live account is unavailable. Unit 16h is acceptance-only: it exercises the delivery path against isolated state, restores exact health and cron bytes, and does not activate a production daily digest.
|
|
1430
2041
|
The cursor snapshot is deliberately materialized and executed twice around the
|
|
1431
2042
|
live scenario. Telegram bootstrap reads the new bot token from host file
|
|
1432
2043
|
descriptor 3; callback injection reads the reviewed saved callback-update JSON
|
|
@@ -1441,7 +2052,7 @@ Packaged Unit 16 acceptance execution:
|
|
|
1441
2052
|
printf 'Telegram bot token: ' >&2
|
|
1442
2053
|
IFS= read -r -s UNIT16_BOT_TOKEN || return $?
|
|
1443
2054
|
printf '\n' >&2
|
|
1444
|
-
if "$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" telegram-bootstrap telegram-bootstrap.json \
|
|
2055
|
+
if "$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" --profile final telegram-bootstrap telegram-bootstrap.json \
|
|
1445
2056
|
3< <(printf '%s\n' "$UNIT16_BOT_TOKEN"); then
|
|
1446
2057
|
UNIT16_BOT_STATUS=0
|
|
1447
2058
|
else
|
|
@@ -1514,40 +2125,40 @@ Packaged Unit 16 acceptance execution:
|
|
|
1514
2125
|
test "$(stat -c '%d:%i' "$UNIT16_CALLBACK_FILE")" = "$(stat -Lc '%d:%i' /proc/self/fd/3)" || exit 1
|
|
1515
2126
|
UNIT16_CALLBACK_VALIDATED=yes
|
|
1516
2127
|
test "$(stat -Lc '%F %u:%g %a' /proc/self/fd/3)" = "regular file 0:0 600" || exit 1
|
|
1517
|
-
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" callback-inject callback-inject.json 3<&3
|
|
2128
|
+
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" --profile final callback-inject callback-inject.json 3<&3
|
|
1518
2129
|
)
|
|
1519
2130
|
}
|
|
1520
|
-
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" materialize telegram-bootstrap
|
|
2131
|
+
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" --profile final materialize telegram-bootstrap
|
|
1521
2132
|
run_unit16_telegram_bootstrap
|
|
1522
|
-
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" materialize cursor-snapshot before
|
|
1523
|
-
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" cursor-snapshot cursor-snapshot.json
|
|
2133
|
+
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" --profile final materialize cursor-snapshot before
|
|
2134
|
+
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" --profile final cursor-snapshot cursor-snapshot.json
|
|
1524
2135
|
# Perform the live scenario whose cursor movement is being measured.
|
|
1525
|
-
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" materialize cursor-snapshot after
|
|
1526
|
-
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" cursor-snapshot cursor-snapshot.json
|
|
1527
|
-
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" materialize cursor-delta
|
|
1528
|
-
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" cursor-delta cursor-delta.json
|
|
1529
|
-
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" materialize callback-inject
|
|
2136
|
+
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" --profile final materialize cursor-snapshot after
|
|
2137
|
+
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" --profile final cursor-snapshot cursor-snapshot.json
|
|
2138
|
+
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" --profile final materialize cursor-delta
|
|
2139
|
+
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" --profile final cursor-delta cursor-delta.json
|
|
2140
|
+
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" --profile final materialize callback-inject
|
|
1530
2141
|
run_unit16_callback_inject
|
|
1531
|
-
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" materialize unraid-key-rotate
|
|
1532
|
-
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" unraid-key-rotate unraid-key-rotate.json
|
|
1533
|
-
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" materialize reboot-request
|
|
1534
|
-
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" reboot-request reboot-request.json
|
|
2142
|
+
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" --profile final materialize unraid-key-rotate
|
|
2143
|
+
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" --profile final unraid-key-rotate unraid-key-rotate.json
|
|
2144
|
+
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" --profile final materialize reboot-request
|
|
2145
|
+
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" --profile final reboot-request reboot-request.json
|
|
1535
2146
|
# The preceding command captures and seals preflight evidence before it asks
|
|
1536
2147
|
# the broker to stage the reboot, then seals the requested-phase evidence and
|
|
1537
2148
|
# fsyncs the requested checkpoint before invoking the host reboot. Reconnect
|
|
1538
2149
|
# only after Unraid and Docker are ready.
|
|
1539
|
-
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" materialize reboot-resume
|
|
1540
|
-
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" reboot-resume reboot-resume.json
|
|
1541
|
-
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" materialize evidence-snapshot
|
|
1542
|
-
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" evidence-snapshot evidence-snapshot.json
|
|
2150
|
+
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" --profile final materialize reboot-resume
|
|
2151
|
+
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" --profile final reboot-resume reboot-resume.json
|
|
2152
|
+
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" --profile final materialize evidence-snapshot
|
|
2153
|
+
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" --profile final evidence-snapshot evidence-snapshot.json
|
|
1543
2154
|
# reboot-resume seals the changed boot identity and recovery milestones.
|
|
1544
2155
|
# evidence-snapshot then runs the remaining scenarios sequentially with a
|
|
1545
2156
|
# 72-minute aggregate bound; each completed scenario records provenance at
|
|
1546
2157
|
# capture time and always clears its public gate, private marker, and receipt.
|
|
1547
|
-
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" materialize evidence-bundle-index
|
|
1548
|
-
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" evidence-bundle-index evidence-bundle-index.json
|
|
1549
|
-
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" materialize evidence-bundle-verify
|
|
1550
|
-
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" evidence-bundle-verify evidence-bundle-verify.json
|
|
2158
|
+
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" --profile final materialize evidence-bundle-index
|
|
2159
|
+
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" --profile final evidence-bundle-index evidence-bundle-index.json
|
|
2160
|
+
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" --profile final materialize evidence-bundle-verify
|
|
2161
|
+
"$UNIT16_ROOT/sanctuary-unit16-run.sh" "$IMAGE_ID" --profile final evidence-bundle-verify evidence-bundle-verify.json
|
|
1551
2162
|
Each execution revalidates the immutable image and the production container's
|
|
1552
2163
|
exact image ID. The main one-shot gets a read-only root filesystem, one writable
|
|
1553
2164
|
evidence mount, narrowly selected runtime/bundle modes, and individual read-only
|