agcoord 0.7.0__tar.gz → 0.7.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {agcoord-0.7.0 → agcoord-0.7.2}/CHANGELOG.md +56 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/Cargo.lock +1 -1
- {agcoord-0.7.0 → agcoord-0.7.2}/Cargo.toml +1 -1
- {agcoord-0.7.0 → agcoord-0.7.2}/PKG-INFO +1 -1
- {agcoord-0.7.0 → agcoord-0.7.2}/conformance/manifest-v3.json +60 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/docs/concepts.md +4 -3
- {agcoord-0.7.0 → agcoord-0.7.2}/docs/coordinator.md +46 -7
- {agcoord-0.7.0 → agcoord-0.7.2}/docs/native_broker.md +1 -1
- {agcoord-0.7.0 → agcoord-0.7.2}/docs/native_host.md +88 -12
- {agcoord-0.7.0 → agcoord-0.7.2}/docs/releasing.md +6 -2
- {agcoord-0.7.0 → agcoord-0.7.2}/docs/troubleshooting.md +15 -4
- {agcoord-0.7.0 → agcoord-0.7.2}/native/agcoord-broker/src/broker.rs +124 -53
- {agcoord-0.7.0 → agcoord-0.7.2}/native/agcoord-broker/src/cgroup.rs +44 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/native/agcoord-broker/src/host.rs +26 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/native/agcoord-broker/src/main.rs +52 -0
- agcoord-0.7.2/native/agcoord-broker/src/recovery.rs +471 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/native/agcoord-broker/src/store.rs +205 -68
- agcoord-0.7.2/native/agcoord-broker/tests/host_recovery.rs +550 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/native/agcoord-broker/tests/scheduler.rs +453 -14
- {agcoord-0.7.0 → agcoord-0.7.2}/scripts/check-conformance +1 -1
- {agcoord-0.7.0 → agcoord-0.7.2}/scripts/install-native-host +25 -6
- agcoord-0.7.2/scripts/test-native-host-recovery +249 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord/__init__.py +1 -1
- {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord/cli.py +124 -6
- {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord/config.py +25 -2
- {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord/native_host.py +363 -28
- agcoord-0.7.2/src/agcoord/native_host_pin.json +5 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord/queue.py +42 -8
- {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord.egg-info/PKG-INFO +1 -1
- {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord.egg-info/SOURCES.txt +5 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/tests/conftest.py +6 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/tests/test_cli.py +97 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/tests/test_conformance_gate.py +9 -3
- agcoord-0.7.2/tests/test_host_config.py +201 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/tests/test_native_client.py +7 -0
- agcoord-0.7.2/tests/test_native_host_recovery.py +402 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/tests/test_native_host_upgrade.py +9 -1
- {agcoord-0.7.0 → agcoord-0.7.2}/tests/test_release_candidate.py +4 -1
- {agcoord-0.7.0 → agcoord-0.7.2}/tests/test_user_broker_install.py +5 -1
- agcoord-0.7.0/src/agcoord/native_host_pin.json +0 -5
- {agcoord-0.7.0 → agcoord-0.7.2}/AGENTS.md +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/LICENSE +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/MANIFEST.in +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/README.md +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/docs/agents.md +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/docs/assets/README.md +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/docs/assets/agcoord-gourd-mascot.png +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/docs/changelog.md +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/docs/comparison.md +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/docs/conformance.md +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/docs/contributing.md +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/docs/guardrails.md +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/docs/index.md +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/docs/native_migration.md +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/docs/overview.md +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/docs/quickstart.md +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/docs/session_handoff_format.md +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/docs/why.md +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/native/THIRD_PARTY_LICENSES.tsv +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/native/agcoord-broker/Cargo.toml +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/native/agcoord-broker/build.rs +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/native/agcoord-broker/src/error.rs +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/native/agcoord-broker/src/platform.rs +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/native/agcoord-broker/src/project_quota.rs +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/native/agcoord-broker/src/resources.rs +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/native/agcoord-broker/src/worker.rs +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/native/agcoord-broker/tests/client_compatibility.rs +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/native/agcoord-broker/tests/host_deployment.rs +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/native/agcoord-broker/tests/identity.rs +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/packaging/apparmor/usr.libexec.agcoord.agcoord-broker +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/packaging/systemd/agcoord-broker.service +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/pyproject.toml +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/rust-toolchain.toml +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/scripts/audit-native-broker +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/scripts/build-native-broker +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/scripts/build-native-host-package +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/scripts/check-native-host-package +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/scripts/check-native-licenses +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/scripts/check-native-reproducible +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/scripts/native-source-id +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/scripts/test-native-host-enforcement +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/scripts/verify-release-candidate +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/setup.cfg +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord/__main__.py +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord/avoid.py +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord/frame.py +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord/github.py +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord/github_release.py +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord/land.py +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord/merge.py +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord/native_client.py +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord/py.typed +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord/pytest_xdist.py +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord/resources.py +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord/tui.py +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord.egg-info/dependency_links.txt +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord.egg-info/entry_points.txt +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord.egg-info/requires.txt +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord.egg-info/top_level.txt +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/tests/test_avoid.py +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/tests/test_distribution.py +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/tests/test_github_adapter.py +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/tests/test_leases.py +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/tests/test_merge.py +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/tests/test_native_host_assets.py +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/tests/test_native_host_download.py +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/tests/test_pytest_xdist.py +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/tests/test_queue.py +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/tests/test_resource_bindings.py +0 -0
- {agcoord-0.7.0 → agcoord-0.7.2}/tests/test_tui.py +0 -0
|
@@ -3,6 +3,62 @@
|
|
|
3
3
|
All notable user-facing changes to AGCoord are recorded here. Versions follow semantic
|
|
4
4
|
versioning; dates use ISO 8601.
|
|
5
5
|
|
|
6
|
+
## Unreleased
|
|
7
|
+
|
|
8
|
+
## 0.7.2 — 2026-10-01
|
|
9
|
+
|
|
10
|
+
- Add `agc host recover PACKAGE` and `agc host recover --download` for a dead managed owner
|
|
11
|
+
that cannot drain. Recovery always verifies a pinned replacement; a new recovery refuses live
|
|
12
|
+
or ambiguous workers, and preserves queued IDs and history behind a durable guard. Only the bound
|
|
13
|
+
`cpu=1` enforcement proof can run before successful completion reopens ordinary work. Dead
|
|
14
|
+
running rows become `interrupted`; bounded failures park under the same recovery identity.
|
|
15
|
+
Three proof attempts persist across retries, which require the same replacement and helper.
|
|
16
|
+
A running replacement under the exact matching guard can continue its existing proof without
|
|
17
|
+
reactivation; exact-ID completion retries return the validated durable result after a lost reply
|
|
18
|
+
(#242).
|
|
19
|
+
- Recover when systemd recreates the managed broker's delegated cgroup root after every worker
|
|
20
|
+
recorded against the old root is gone. The replacement now abandons only its private recovery
|
|
21
|
+
metadata, records those dead rows `interrupted`, and admits later work against the new root;
|
|
22
|
+
a live worker or any other cgroup identity mismatch still fails closed (#235).
|
|
23
|
+
- A fresh host install now declares the machine's memory as well as its CPUs, and enforces both
|
|
24
|
+
when the host allows it. The configuration written when a state directory has none derives
|
|
25
|
+
`memory` capacity from `MemTotal` minus a reserve (the larger of 4 GiB or an eighth of RAM) and
|
|
26
|
+
adds the required cgroup-v2 `memory` binding only when that controller is delegated to the
|
|
27
|
+
broker's slice, so an install can no longer write a configuration the broker refuses at
|
|
28
|
+
startup. A state directory that already holds a configuration keeps it (#226).
|
|
29
|
+
- Add `agc host config`, which prints the configuration this host would be given and writes it
|
|
30
|
+
with `--write`, refusing to replace an existing file without `--force`. `--cpu`, `--jobs`,
|
|
31
|
+
`--memory`, and `--reserve` override what is derived, `--cgroup-root` selects the slice whose
|
|
32
|
+
delegated controllers decide the bindings, and `--tmpfs` with `--tmpfs-inodes` adds an
|
|
33
|
+
enforced scratch policy that stays opt-in because it is bounded by the memory capacity (#226).
|
|
34
|
+
|
|
35
|
+
## 0.7.1 — 2026-09-11
|
|
36
|
+
|
|
37
|
+
- Fix the native broker and its waiting clients spending a host's CPU in proportion to the
|
|
38
|
+
finished history in the spool. Every 25 ms scheduling tick and every `status` poll decoded
|
|
39
|
+
every terminal row ever recorded, so on a host with a few thousand finished gates the idle
|
|
40
|
+
broker held a whole core and each waiting `agc run` or `agc land` spent about a fifth of a
|
|
41
|
+
CPU-second per poll, leaving an admitted row a fraction of the CPU it claimed. Scheduling,
|
|
42
|
+
observation, and `status` now read only queued and running rows. A following client also
|
|
43
|
+
backs off from a tenth of a second to one second while its job is queued, and a `--json`
|
|
44
|
+
wait does the same for the job's whole life (#217).
|
|
45
|
+
- Stopping the native broker, including `systemctl --user stop` and `restart`, now drains instead
|
|
46
|
+
of cancelling in-flight work. The broker stops admitting, lets running jobs finish, and exits;
|
|
47
|
+
queued jobs wait for the next owner. A job still running when the new `stop_grace`
|
|
48
|
+
configuration elapses (600 seconds by default; `0` interrupts at once), or when a second stop
|
|
49
|
+
signal arrives, is ended and recorded `interrupted` with exit status 125 and failure reason
|
|
50
|
+
`broker-stopped`, and a following client says it claims no verdict and should be submitted
|
|
51
|
+
again. Previously such a job was recorded `cancelled` with exit 130, indistinguishable from a
|
|
52
|
+
caller's own cancellation (#218).
|
|
53
|
+
- Fix a client or a gate intermittently refused with `broker-schema-invalid: coordinator
|
|
54
|
+
database has no readable protocol metadata` while the spool was only busy. The broker's
|
|
55
|
+
own short-lived connections were often the spool's last, and closing the last connection to a
|
|
56
|
+
WAL database checkpoints and deletes the WAL under an exclusive lock that every opening reader
|
|
57
|
+
waits behind; on a slow disk that outlasted a short `database_timeout`, and the protocol read
|
|
58
|
+
reported the busy lock as a schema fault. A live broker now holds one connection for as long
|
|
59
|
+
as it owns the spool, and a busy protocol read is refused as the retryable
|
|
60
|
+
`broker-database-busy` (#223).
|
|
61
|
+
|
|
6
62
|
## 0.7.0 — 2026-09-06
|
|
7
63
|
|
|
8
64
|
- `agc land --reuse-full` lets one passed `agc full` receipt stand in for the land gate instead
|
|
@@ -176,6 +176,27 @@
|
|
|
176
176
|
]
|
|
177
177
|
}
|
|
178
178
|
},
|
|
179
|
+
{
|
|
180
|
+
"id": "contention.owner-holds-the-spool-open",
|
|
181
|
+
"domain": "contention",
|
|
182
|
+
"description": "A live owner keeps one spool connection open so readers never wait behind a last-connection WAL checkpoint, and a busy protocol read is refused as retryable contention rather than a schema fault.",
|
|
183
|
+
"tests": {
|
|
184
|
+
"rust_native": [
|
|
185
|
+
"scheduler::a_serving_broker_never_closes_the_last_spool_connection",
|
|
186
|
+
"scheduler::a_busy_protocol_read_is_refused_as_retryable_contention"
|
|
187
|
+
]
|
|
188
|
+
}
|
|
189
|
+
},
|
|
190
|
+
{
|
|
191
|
+
"id": "contention.history-independent-polling",
|
|
192
|
+
"domain": "contention",
|
|
193
|
+
"description": "Scheduling ticks and status polls read only live rows, so the broker and its waiting clients spend no CPU in proportion to terminal history.",
|
|
194
|
+
"tests": {
|
|
195
|
+
"rust_native": [
|
|
196
|
+
"scheduler::scheduling_ticks_and_status_polls_do_not_pay_for_terminal_history"
|
|
197
|
+
]
|
|
198
|
+
}
|
|
199
|
+
},
|
|
179
200
|
{
|
|
180
201
|
"id": "contention.durable-maintenance-drain",
|
|
181
202
|
"domain": "contention",
|
|
@@ -197,6 +218,19 @@
|
|
|
197
218
|
]
|
|
198
219
|
}
|
|
199
220
|
},
|
|
221
|
+
{
|
|
222
|
+
"id": "cancellation.graceful-stop-drains",
|
|
223
|
+
"domain": "cancellation",
|
|
224
|
+
"description": "A graceful stop admits nothing new and lets running jobs finish; after the stop grace or a second stop signal it interrupts them as broker-stopped instead of cancelling.",
|
|
225
|
+
"tests": {
|
|
226
|
+
"rust_native": [
|
|
227
|
+
"scheduler::graceful_stop_lets_running_work_finish_and_leaves_queued_work_for_the_next_owner",
|
|
228
|
+
"scheduler::graceful_stop_interrupts_work_that_outlives_the_stop_grace",
|
|
229
|
+
"scheduler::a_second_stop_signal_interrupts_running_work_without_waiting_for_the_grace",
|
|
230
|
+
"scheduler::graceful_stop_retries_a_contended_interruption_commit"
|
|
231
|
+
]
|
|
232
|
+
}
|
|
233
|
+
},
|
|
200
234
|
{
|
|
201
235
|
"id": "recovery.adopt-without-duplicate",
|
|
202
236
|
"domain": "recovery",
|
|
@@ -207,6 +241,17 @@
|
|
|
207
241
|
]
|
|
208
242
|
}
|
|
209
243
|
},
|
|
244
|
+
{
|
|
245
|
+
"id": "recovery.recreated-cgroup-root",
|
|
246
|
+
"domain": "recovery",
|
|
247
|
+
"description": "A replacement owner abandons a reused cgroup root only for dead recorded workers and refuses while a matching worker is alive.",
|
|
248
|
+
"tests": {
|
|
249
|
+
"rust_native": [
|
|
250
|
+
"scheduler::replacement_recovers_a_dead_worker_after_the_cgroup_root_is_recreated",
|
|
251
|
+
"scheduler::replacement_refuses_a_recreated_cgroup_root_while_its_worker_is_live"
|
|
252
|
+
]
|
|
253
|
+
}
|
|
254
|
+
},
|
|
210
255
|
{
|
|
211
256
|
"id": "crash.database-transactions",
|
|
212
257
|
"domain": "crash_database",
|
|
@@ -328,6 +373,21 @@
|
|
|
328
373
|
"scheduler::required_cgroup_binding_refuses_before_user_code_when_delegation_is_unavailable"
|
|
329
374
|
]
|
|
330
375
|
}
|
|
376
|
+
},
|
|
377
|
+
{
|
|
378
|
+
"id": "recovery.guarded-managed-host",
|
|
379
|
+
"domain": "recovery",
|
|
380
|
+
"description": "A dead managed owner can be replaced while queued identities and history remain guarded; live owners or workers refuse recovery and ordinary work stays blocked across restarts.",
|
|
381
|
+
"tests": {
|
|
382
|
+
"rust_native": [
|
|
383
|
+
"host_recovery::recovery_preserves_queued_work_and_guards_admission_across_restarts",
|
|
384
|
+
"host_recovery::recovery_refuses_a_live_owner_and_a_live_retained_worker",
|
|
385
|
+
"host_recovery::a_successful_unenforced_proof_cannot_complete_or_resume_recovery",
|
|
386
|
+
"host_recovery::mismatched_recovery_identity_and_changed_helper_preserve_the_guard",
|
|
387
|
+
"host_recovery::failed_proofs_retry_with_the_same_identity_and_park_at_the_attempt_bound",
|
|
388
|
+
"host_recovery::parking_cancels_only_the_active_proof_and_allows_a_bounded_retry"
|
|
389
|
+
]
|
|
390
|
+
}
|
|
331
391
|
}
|
|
332
392
|
],
|
|
333
393
|
"intentional_differences": [
|
|
@@ -127,9 +127,10 @@ Exact rules: [durable job shape](coordinator.md#durable-job-shape).
|
|
|
127
127
|
|
|
128
128
|
Every job is `queued` until the broker admits it, `running` while its command lives, and then
|
|
129
129
|
exactly one of four terminal states. `passed` and `failed` carry the command's verdict.
|
|
130
|
-
`cancelled` means someone asked
|
|
131
|
-
|
|
132
|
-
it could report,
|
|
130
|
+
`cancelled` means someone asked with `agc cancel`; the whole process group is gone before the
|
|
131
|
+
row becomes terminal. `interrupted` means the coordinator claims no verdict: the worker vanished
|
|
132
|
+
before it could report (`worker-result-lost`), or a stopping broker ended the job after its
|
|
133
|
+
stop grace (`broker-stopped`). A graceful broker stop first lets running jobs finish. Terminal rows are immutable
|
|
133
134
|
history until `agc clear` removes them while the queue is idle.
|
|
134
135
|
|
|
135
136
|
A `land` moves through named phases inside `running`. *Preflight* checks the pull request,
|
|
@@ -41,8 +41,15 @@ the recorded process identity before classifying a live job; it never reruns an
|
|
|
41
41
|
spawned command merely because the old broker disappeared. An exception that ends the broker
|
|
42
42
|
does not request cancellation or signal live process groups: ownership is released so a
|
|
43
43
|
replacement can adopt each worker whose recorded PID and process-start token still match.
|
|
44
|
-
|
|
45
|
-
|
|
44
|
+
A graceful stop — SIGTERM or SIGINT, which is what `systemctl --user stop` and `restart` send —
|
|
45
|
+
drains instead of cancelling. The owner stops admitting work, leaves queued rows for the next
|
|
46
|
+
owner, keeps observing running jobs and granting their child CPU leases, and releases ownership
|
|
47
|
+
once none is running. If jobs are still running when `stop_grace` elapses, or when a second stop
|
|
48
|
+
signal arrives, the owner ends each one that is not publishing as a complete process group and
|
|
49
|
+
records it `interrupted` with exit status 125 and failure reason `broker-stopped`, so its agent
|
|
50
|
+
can tell an operator stop from a verdict and submit it again. A publishing land or retained
|
|
51
|
+
merge is never interrupted; the owner waits for its authoritative result. A caller's own
|
|
52
|
+
`agc cancel` during a stop still records `cancelled`.
|
|
46
53
|
|
|
47
54
|
A land worker reports its final overall status durably while its row is still running. After
|
|
48
55
|
an unclean owner loss, a replacement preserves the live worker's lane and resources, never
|
|
@@ -50,6 +57,12 @@ reruns its gate, waits for the exact process group to disappear, reclaims scratc
|
|
|
50
57
|
exposes the reported passed or typed-failed result. If the worker disappears before it can
|
|
51
58
|
report a result, `interrupted` is the only safe terminal classification.
|
|
52
59
|
|
|
60
|
+
If systemd recreated the broker service's delegated cgroup root after every recorded worker
|
|
61
|
+
using the old root disappeared, the replacement abandons only its private recovery metadata
|
|
62
|
+
for that old root, records the dead rows `interrupted`, and prepares later work under the new
|
|
63
|
+
root. It never removes or signals anything in the new root. A matching live worker, or any
|
|
64
|
+
other identity mismatch, still refuses recovery rather than treating a reused path as owned.
|
|
65
|
+
|
|
53
66
|
## Durable maintenance drain
|
|
54
67
|
|
|
55
68
|
Maintenance closes the submission race before waiting for an idle spool:
|
|
@@ -105,7 +118,7 @@ the entire machine:
|
|
|
105
118
|
|
|
106
119
|
One JSON file, `config.json` in the state directory, configures the broker that owns that
|
|
107
120
|
directory. It holds at most `capacities`, `bindings`, `cgroup_root`, `cgroup_io`,
|
|
108
|
-
`database_timeout`, `land_gate_reuse_max_age`, and `native_broker`; invalid JSON, a top-level
|
|
121
|
+
`database_timeout`, `land_gate_reuse_max_age`, `stop_grace`, and `native_broker`; invalid JSON, a top-level
|
|
109
122
|
value that is not an object,
|
|
110
123
|
an unknown key, a section that is not an object, or an empty `cgroup_root` is refused when the
|
|
111
124
|
client or broker loads its configuration. When present, `cgroup_io` contains exactly one
|
|
@@ -113,10 +126,23 @@ nonempty `paths` list of unique absolute strings. `database_timeout` is a positi
|
|
|
113
126
|
of seconds no greater than `2147483.647` (SQLite's millisecond limit) and defaults to `10`.
|
|
114
127
|
`land_gate_reuse_max_age` is a finite number of seconds from `0` to `2147483.647`, defaults to
|
|
115
128
|
`3600`, and bounds how long a passed `full` receipt may stand in for a land gate; `0` refuses
|
|
116
|
-
every reuse on that host even when a caller asks for one.
|
|
129
|
+
every reuse on that host even when a caller asks for one. `stop_grace` is a finite number of
|
|
130
|
+
seconds from `0` to `2147483.647`, defaults to `600`, and bounds how long a gracefully stopping
|
|
131
|
+
broker lets running jobs finish before it interrupts them; `0` interrupts them at once. An
|
|
117
132
|
absent file uses `/usr/libexec/agcoord/agcoord-broker`, requires its release trust policy, and
|
|
118
133
|
defaults capacity to `jobs=2`.
|
|
119
134
|
|
|
135
|
+
`agc host config` prints the configuration this host would be given, and writes it with
|
|
136
|
+
`--write`; it refuses to replace an existing file without `--force`, because a live owner keeps
|
|
137
|
+
the configuration it acquired the spool with. The same derivation writes the configuration an
|
|
138
|
+
install creates when a state directory has none: `cpu` and `jobs` from the available
|
|
139
|
+
CPU-affinity count, `memory` from `MemTotal` minus a reserve, and a required cgroup-v2 binding
|
|
140
|
+
for each of those kinds whose controller is delegated to the broker's slice, so a derived
|
|
141
|
+
configuration is never one the broker refuses at startup. `--cpu`, `--jobs`, `--memory`, and
|
|
142
|
+
`--reserve` override what is derived, `--cgroup-root` selects the slice to inspect, and
|
|
143
|
+
`--tmpfs` with `--tmpfs-inodes` adds the scratch policy, which stays opt-in because its bytes
|
|
144
|
+
are charged against the same memory capacity.
|
|
145
|
+
|
|
120
146
|
```json
|
|
121
147
|
{
|
|
122
148
|
"capacities": {"jobs": 4, "cpu": 8, "browser": 1},
|
|
@@ -157,7 +183,12 @@ wait without changing existing connections.
|
|
|
157
183
|
|
|
158
184
|
Schema setup places every current-protocol spool in SQLite WAL journal mode, including an
|
|
159
185
|
existing spool the next time a compatible client or broker opens it. Readers therefore do not
|
|
160
|
-
block behind an ordinary writer.
|
|
186
|
+
block behind an ordinary writer. A live owner also keeps one connection open for as long as
|
|
187
|
+
it owns the spool. Closing the last connection to a WAL database checkpoints and removes the
|
|
188
|
+
WAL under an exclusive lock that every opening reader must wait behind, so an owner whose own
|
|
189
|
+
short-lived connections were the last would stall clients on a slow disk. The configured
|
|
190
|
+
timeout bounds each remaining lock wait, and a lock that outlasts it is reported as the
|
|
191
|
+
retryable `broker-database-busy`, never as a schema fault;
|
|
161
192
|
transient busy or locked results in the broker pump and idle health check are retried, and a
|
|
162
193
|
contended best-effort activity heartbeat never changes an already successful public operation
|
|
163
194
|
into an apparent failure.
|
|
@@ -963,8 +994,9 @@ target update.
|
|
|
963
994
|
|
|
964
995
|
A land request is cancellable while queued, preflighting, or gating. Once its durable phase
|
|
965
996
|
is `publishing`, cancellation is refused because killing a client during an authenticated
|
|
966
|
-
atomic mutation would leave the outcome indeterminate.
|
|
967
|
-
|
|
997
|
+
atomic mutation would leave the outcome indeterminate. A graceful broker stop lets every phase
|
|
998
|
+
finish; once `stop_grace` elapses it interrupts earlier phases as `broker-stopped` but still
|
|
999
|
+
waits for publishing and records its authoritative result.
|
|
968
1000
|
|
|
969
1001
|
### Reusing a full receipt as the land gate
|
|
970
1002
|
|
|
@@ -1072,6 +1104,13 @@ for the configured `database_timeout` rather than aborting on the first busy loc
|
|
|
1072
1104
|
broker committing a publication or checkpointing its WAL does not end a client that is only
|
|
1073
1105
|
watching its own row.
|
|
1074
1106
|
|
|
1107
|
+
Watching costs the host little while work waits. A following client polls a queued job's row
|
|
1108
|
+
at an interval that doubles from a tenth of a second to one second, then streams the admitted
|
|
1109
|
+
job's log every tenth of a second; a `--json` wait, which streams nothing, backs off the same
|
|
1110
|
+
way for the job's whole life. The broker's scheduling tick and each `status` read only queued
|
|
1111
|
+
and running rows, so neither grows with the terminal history that accumulates until
|
|
1112
|
+
`agc clear`.
|
|
1113
|
+
|
|
1075
1114
|
The native owner authenticates a worker with its PID, Linux start token, and requirement that
|
|
1076
1115
|
the PID lead its recorded process group. Replacement recovery adopts only that exact live
|
|
1077
1116
|
identity. If the PID has been reused, its token changed, or it belongs to another group, the run
|
|
@@ -142,7 +142,7 @@ owner record is:
|
|
|
142
142
|
pid=12345
|
|
143
143
|
protocol=5
|
|
144
144
|
implementation=rust-native
|
|
145
|
-
version=0.7.
|
|
145
|
+
version=0.7.1
|
|
146
146
|
build=sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
|
|
147
147
|
capacities={"cpu":8,"jobs":2}
|
|
148
148
|
resource_bindings={}
|
|
@@ -63,14 +63,20 @@ Create the state directory as the broker user with mode `0700` and `config.json`
|
|
|
63
63
|
`app.slice`, and `DelegateSubgroup=supervisor` make the configured service root deterministic
|
|
64
64
|
and keep the broker process out of the inner node where it enables controllers.
|
|
65
65
|
|
|
66
|
-
On a fresh `agc host install`, an absent configuration is
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
66
|
+
On a fresh `agc host install`, an absent configuration is derived from the host: `cpu` and `jobs`
|
|
67
|
+
both take the invoking process's available CPU-affinity count, `memory` takes `MemTotal` minus a
|
|
68
|
+
reserve of the larger of 4 GiB or an eighth of RAM, and each of those kinds receives the required
|
|
69
|
+
cgroup-v2 binding shown above only when its controller is delegated to the service's slice. A
|
|
70
|
+
kind whose controller is missing keeps its capacity and stays admission-only, so the derived
|
|
71
|
+
configuration is never one the broker refuses at startup. Run `agc host config` first to see
|
|
72
|
+
exactly what this host would be given, and `agc host config --write` to install it before the
|
|
73
|
+
first `agc host install`. The scheduler greedily packs each job's complete declared resource
|
|
74
|
+
vector against those totals; it does not divide the machine into a fixed number of equal CPU
|
|
75
|
+
partitions. `jobs` remains a ceiling for submissions that omit other claims. To add RAM-disk,
|
|
76
|
+
persistent-disk, I/O, or project-defined capacities, or to choose your own memory reserve or a
|
|
77
|
+
tmpfs scratch policy, write the owner-only configuration before first install, with
|
|
78
|
+
`agc host config --tmpfs SIZE --write` or by hand; the command validates and preserves an
|
|
79
|
+
existing safe configuration instead of overwriting it.
|
|
74
80
|
|
|
75
81
|
On every managed start, the broker fails before acquiring the spool unless all of these match:
|
|
76
82
|
|
|
@@ -378,17 +384,87 @@ a nondefault, unsafe, fresh, or pre-protocol-5 spool. A failure before activatio
|
|
|
378
384
|
host unchanged; a failure after stop reports the exact drain ID and leaves the service stopped
|
|
379
385
|
and coordinator drained unless the resume step had already succeeded.
|
|
380
386
|
|
|
381
|
-
For manual upgrade or recovery, run `stage` while the
|
|
382
|
-
the staged package digest, run `agc drain`, retain
|
|
383
|
-
|
|
387
|
+
For manual upgrade or recovery where the outgoing broker can drain, run `stage` while the
|
|
388
|
+
current service remains available. Inspect the staged package digest, run `agc drain`, retain
|
|
389
|
+
its exact ID, and stop the user service after the receipt says `drained`. Run
|
|
390
|
+
`activate --drain-id ID`, `daemon-reload`, any required `migrate`,
|
|
384
391
|
exact-ID `resume`, and `start` in that order. An explicitly chosen cancellation policy may shorten
|
|
385
392
|
the drain, but cancellation never replaces its durable submission guard. Never replace the live
|
|
386
393
|
binary and ask systemd to restart while work remains. After start, inspect `systemctl --user
|
|
387
394
|
status agcoord-broker.service`, `agc list`, and rerun the enforced-host proof.
|
|
388
395
|
|
|
396
|
+
### Guarded recovery when the outgoing broker cannot drain
|
|
397
|
+
|
|
398
|
+
The normal drain flow depends on a functioning outgoing broker. It cannot produce a completed
|
|
399
|
+
drain when that broker is dead or cannot start. Use the separate managed recovery operation
|
|
400
|
+
with the exact matching client and a pinned replacement:
|
|
401
|
+
|
|
402
|
+
```bash
|
|
403
|
+
agc host recover /path/to/native-host-bundle/agcoord-native-host-x86_64-linux.tar.gz
|
|
404
|
+
# Or fetch the matching release bundle:
|
|
405
|
+
agc host recover --download
|
|
406
|
+
```
|
|
407
|
+
|
|
408
|
+
Recovery always requires a broker pin, including a bundle supplied by path. Starting a new
|
|
409
|
+
recovery refuses a live owner, live or ambiguous recorded workers, unavailable identity
|
|
410
|
+
evidence, and an invalid package. It never substitutes a forced activation or deletion of the
|
|
411
|
+
spool. Normal `agc host upgrade` and its drain flow remain unchanged.
|
|
412
|
+
|
|
413
|
+
The command stages the verified replacement and stops the failed service. The staged native
|
|
414
|
+
holder takes the exclusive owner/maintenance lock, proves every recorded worker identity is
|
|
415
|
+
gone, and establishes a durable `recovery-` ID followed by 12 hexadecimal digits. It holds the
|
|
416
|
+
lock through activation and installed-identity verification. The restarted replacement
|
|
417
|
+
settles dead running rows as `interrupted`, without an execution verdict. Queued job IDs and
|
|
418
|
+
their work, terminal history, and the spool remain intact; recovery never reruns an interrupted
|
|
419
|
+
or terminal command.
|
|
420
|
+
|
|
421
|
+
The durable guard blocks new submissions and ordinary queued-job admission through activation
|
|
422
|
+
and restart. Recovery admits only the exact bound enforcement helper, with `cpu=1` and
|
|
423
|
+
`jobs=1`, while ordinary work stays blocked. Completion requires that proof to pass with exit
|
|
424
|
+
status zero, a receipt with requested and applied CPU both 1 and observed peak CPU at least 1,
|
|
425
|
+
the unchanged helper digest, and no running rows. Only successful completion clears the guard.
|
|
426
|
+
A process exit, service restart, normal `agc drain`, or `agc resume` cannot clear it.
|
|
427
|
+
|
|
428
|
+
If the verified replacement is already running under the same active recovery guard, rerunning
|
|
429
|
+
`agc host recover` continues that guard instead of stopping or reactivating the service. The
|
|
430
|
+
replacement identity and helper path and digest must match exactly. An ordinary live owner or
|
|
431
|
+
a mismatched guard still refuses. Continuation reuses a queued, running, or passed proof while
|
|
432
|
+
that recovery is verifying; it does not submit a duplicate or spend another proof attempt.
|
|
433
|
+
|
|
434
|
+
The native hold, status, proof, completion, and park operations retain the exact recovery ID,
|
|
435
|
+
replacement identity, helper path and digest, proof job ID, attempt count, phase, and incident.
|
|
436
|
+
Each host phase and enforcement proof is bounded to 120 seconds; owner acquisition and dead-row
|
|
437
|
+
settlement waits are each bounded to 30 seconds. At most three proof attempts are allowed
|
|
438
|
+
across retries. Retrying must use the same exact verified replacement package and helper path;
|
|
439
|
+
it continues the same recovery identity and never resets the attempt budget. Exhausted proof
|
|
440
|
+
attempts require operator repair, with no reset bypass.
|
|
441
|
+
|
|
442
|
+
Before completion, a failed or interrupted operation parks safely with the guard intact and
|
|
443
|
+
nonsecret evidence of the failed phase. If the service was touched, failure stops it or
|
|
444
|
+
explicitly reports that the stop is unconfirmed; an earlier staging failure leaves it unchanged.
|
|
445
|
+
The command does not silently resume work or claim an unproved host is ready. Preserve the
|
|
446
|
+
recovery ID and address the reported incident before retrying the same package.
|
|
447
|
+
|
|
448
|
+
Completion retries up to three times with the exact recovery ID. If completion committed but
|
|
449
|
+
its reply was lost, the native operation validates the retained completed identity and passed
|
|
450
|
+
CPU proof, then returns the same open receipt without another proof or recovery operation.
|
|
451
|
+
If completion still cannot be confirmed, the error says to inspect the durable receipt; it
|
|
452
|
+
does not claim the guard remains when completion may already have cleared it.
|
|
453
|
+
|
|
454
|
+
### Normal restart and rollback
|
|
455
|
+
|
|
389
456
|
`Restart=on-failure` recovers an unexpected broker exit without an idle shutdown. The durable
|
|
390
457
|
spool remains the authority: the replacement adopts only identity-verified live workers and
|
|
391
|
-
never reruns their command. A deliberate `systemctl stop` does not restart.
|
|
458
|
+
never reruns their command. A deliberate `systemctl stop` does not restart.
|
|
459
|
+
|
|
460
|
+
`systemctl --user stop` and `restart` are graceful: the broker stops admitting, lets running jobs
|
|
461
|
+
finish, and exits, so the command returns when the last running job ends or `stop_grace` (600
|
|
462
|
+
seconds unless configured) elapses and the broker interrupts the rest as `broker-stopped`. Queued
|
|
463
|
+
jobs wait for the restarted owner. The unit's `TimeoutStopSec=infinity` keeps systemd from
|
|
464
|
+
killing work during that wait. To interrupt running jobs at once, send the broker alone a second
|
|
465
|
+
stop signal with `systemctl --user kill --kill-whom=main --signal=SIGTERM agcoord-broker.service`;
|
|
466
|
+
a plain `systemctl kill` signals every process in the unit, including the jobs, and bypasses
|
|
467
|
+
the broker's record of why they ended. If staging is
|
|
392
468
|
interrupted, no live file changed; repeat it. If activation is interrupted, leave the service
|
|
393
469
|
stopped and repeat activation for the same selected package, which revalidates every final
|
|
394
470
|
file before start. Never delete or replace the state directory as host-package recovery.
|
|
@@ -154,8 +154,12 @@ interface and must be smoke-tested from the built wheel.
|
|
|
154
154
|
`0755` on only the fixed broker and three fixed helper names before running this verifier;
|
|
155
155
|
content sidecars and the host archive's internal modes remain independently checked.
|
|
156
156
|
7. Install the host bundle on the supported Ubuntu configuration through the staged runbook.
|
|
157
|
-
For an existing spool, retain the exact durable drain
|
|
158
|
-
ID and resume only after owner-locked maintenance
|
|
157
|
+
For an existing spool with a functioning outgoing owner, retain the exact durable drain
|
|
158
|
+
receipt, require activation to match its ID, and resume only after owner-locked maintenance
|
|
159
|
+
completes. If the outgoing owner is dead or cannot start, use the
|
|
160
|
+
[guarded managed recovery path](native_host.md#guarded-recovery-when-the-outgoing-broker-cannot-drain)
|
|
161
|
+
with the exact pinned replacement. Retain its recovery ID and proof receipt; ordinary work
|
|
162
|
+
stays blocked until verified completion, and normal drain/resume cannot clear that guard.
|
|
159
163
|
Keep `kernel.apparmor_restrict_unprivileged_userns=1`, start the ordinary unprivileged managed
|
|
160
164
|
service, and retain the shipped `cpu=1` receipt proving its AppArmor transition, cgroup
|
|
161
165
|
namespace root, exact CPU control, and durable applied/peak evidence. A spool left below
|
|
@@ -37,7 +37,8 @@ remain the authority for the behavior behind each entry, and the
|
|
|
37
37
|
| `resource '<name>' has no configured machine capacity` | The broker's `config.json` declares no such capacity. | Claim only configured names, or add the capacity to `config.json` and restart the broker. |
|
|
38
38
|
| `resource '<name>' requests N, above capacity M` | The claim can never be admitted on this machine. | Lower the claim; `agc --json list` prints `capacities`. |
|
|
39
39
|
| `a coordinated job cannot submit another coordinated job; invoke it directly from the checkout` | Nested submission from inside an admitted job. | Run the tool directly; a job may only call `agc show "$AGCOORD_RUN_ID"` and `agc verify-admission`. |
|
|
40
|
-
| `broker-draining` (`agcoord-maintenance-draining` for a direct SQLite writer) | Maintenance closed submissions
|
|
40
|
+
| `broker-draining` (`agcoord-maintenance-draining` for a direct SQLite writer) | Maintenance closed submissions. Normal drain finishes admitted work; guarded recovery admits only its bound proof. | Inspect the maintenance state. For `recovering`, follow the managed recovery receipt; ordinary submissions stay blocked until proof succeeds. `list`, `show`, `log`, the TUI, and `cancel` still work. |
|
|
41
|
+
| `host-recovery-required` from drain or resume | A guarded host recovery is active; ordinary drain and resume cannot clear its guard. | Preserve the recovery ID and follow [managed recovery](native_host.md#guarded-recovery-when-the-outgoing-broker-cannot-drain); retry the same verified package after repair. |
|
|
41
42
|
| `broker-drained` | The spool finished draining and has no owner; nothing autostarts. | `agc resume <drain-id>` with the ID the drain printed, or finish the maintenance that needed it. |
|
|
42
43
|
| `Gate queue: <id> waiting at position N for branch …` | Queued, not refused: capacity or a barrier is holding it. | `blocked_by` in `agc --json show <id>` names the jobs ahead; a `land` in the same lane, or a job from the same worktree, waits for that land. |
|
|
43
44
|
| `Gate queue: lost contact with the coordinator while following <id>: …` and exit status 75 | The streaming client could not reach the coordinator for five seconds after acceptance. The job itself continues. | `agc show <id>` for the verdict and `agc log <id> --follow` to keep watching. Never read 75 as a refusal or a red gate. |
|
|
@@ -59,8 +60,9 @@ These arrive as a failed row. `failure_reason` is stable; the log has the detail
|
|
|
59
60
|
| `publish-failed` | The forge rejected the atomic update; nothing moved. | Check `gh auth status`, branch protection (a required hosted merge queue rejects the update), and the pull request; resubmit. |
|
|
60
61
|
| `memory-oom` | The job exceeded its declared memory and was ended as one process group. | Raise the `memory` claim or reduce the job's footprint; the receipt's `peak` says how far over it went. |
|
|
61
62
|
| `resource-enforcement-failed` | A `required` binding could not be applied before user code; the launcher exited 125. | Read the receipt's events for the backend's code; fix the host or make the binding `best-effort`. |
|
|
62
|
-
| status `interrupted` | The
|
|
63
|
-
| status `
|
|
63
|
+
| status `interrupted`, reason `broker-stopped` | The broker was stopped and the job outlived `stop_grace`, or a second stop signal arrived; no verdict is claimed. | Submit again. Publication is never interrupted. |
|
|
64
|
+
| status `interrupted`, reason `worker-result-lost` | The worker vanished before it could report; no verdict is claimed. | Read the log; unless it ends with `LANDED`, submit again. |
|
|
65
|
+
| status `cancelled` | `agc cancel` asked for it. | Resubmit when appropriate. Publication is never cancelled. |
|
|
64
66
|
|
|
65
67
|
## Maintenance
|
|
66
68
|
|
|
@@ -84,6 +86,7 @@ These arrive as a failed row. `failure_reason` is stable; the log has the detail
|
|
|
84
86
|
| `broker-not-running` | A command needed a live owner and found none. | Unmanaged: any `agc list` starts one. Managed: `systemctl --user status agcoord-broker.service`. |
|
|
85
87
|
| `broker-database-busy` | SQLite stayed locked longer than `database_timeout` (10 s by default). | Look for a stuck writer; raise `database_timeout` in `config.json` if the disk is slow. |
|
|
86
88
|
| `broker-config-invalid` | `config.json` has an unknown key, a non-object section, an invalid binding, or an empty `cgroup_root`. | Fix the file against the [configuration contract](coordinator.md#repository-lanes-and-resources). |
|
|
89
|
+
| `broker-row-invalid` naming `root-reused` | A release before this fix found that systemd recreated the delegated cgroup root after a broker loss and could not settle a dead worker. | Upgrade AGCoord. The replacement now abandons only old private recovery metadata after proving every recorded worker is gone; a live worker or another identity mismatch still refuses. |
|
|
87
90
|
| `broker-state-invalid`, `broker-schema-invalid`, `broker-owner-metadata-invalid`, `broker-owner-lock-unavailable` | The state directory or its files are not what the broker expects. | Do not repair by hand. Check ownership and modes (`0700` directory, `0600` files), then the [native broker contract](native_broker.md#stable-refusal-envelope). |
|
|
88
91
|
| `native broker identity is incompatible with this client` | The selected executable is not a protocol-5 AGCoord broker of a supported line. | Point `native_broker.path` at the installed broker, or reinstall it. |
|
|
89
92
|
|
|
@@ -96,7 +99,7 @@ These arrive as a failed row. `failure_reason` is stable; the log has the detail
|
|
|
96
99
|
| `native-host-version-mismatch` | The bundle is not the installed client's version. | Install the matching client first; `--download` then fetches the right bundle. |
|
|
97
100
|
| `native-host-bundle-invalid` | A missing, oversized, symlinked, or wrongly owned file in the bundle. | Download the bundle again, or check the directory's modes (`0700`). |
|
|
98
101
|
| `native-host-pin-mismatch` | The bundle's broker is not the one this client was released against. | Fetch the release that matches the client. Never override the pin. |
|
|
99
|
-
| `native-host-unpinned-client` | A development client carries no pin, so it cannot verify a download. | Install a release client, pass a bundle path you verified yourself, or supply `--broker-sha256`. |
|
|
102
|
+
| `native-host-unpinned-client` | A development client carries no pin, so it cannot verify a download. | Install a release client, pass a bundle path you verified yourself, or supply `--broker-sha256`. Recovery always requires a pin, even for a local bundle. |
|
|
100
103
|
| `native-host-pin-conflict` | `--broker-sha256` disagrees with the shipped pin. | Drop the option; a supplied digest never replaces a pin. |
|
|
101
104
|
| `native-host-bundle-source-conflict`, `native-host-bundle-source-missing` | Both or neither of a bundle path and `--download` (or `--user`). | Pass exactly one source. |
|
|
102
105
|
| `native-host-download-insecure-url`, `native-host-download-invalid-repository`, `native-host-download-unknown-adapter`, `native-host-download-digest-mismatch` | The download source or its bytes are not acceptable. | Use an `https` base URL, an `owner/name` repository, the `github` adapter; a transport digest mismatch means a corrupted or substituted download. |
|
|
@@ -104,7 +107,15 @@ These arrive as a failed row. `failure_reason` is stable; the log has the detail
|
|
|
104
107
|
| `native-host-install-authorization-failed`, `native-host-upgrade-authorization-failed` | `sudo` could not authorize the privileged step. | Run the command in a terminal that can answer sudo; nothing was changed. |
|
|
105
108
|
| `…-stage-failed`, `…-activation-failed`, `…-reload-failed`, `…-start-failed`, `…-stop-failed` | The named phase failed; the receipt says what was rolled back. | Fix the cause the phase names and rerun the same command. |
|
|
106
109
|
| `native-host-install-incomplete`, `native-host-upgrade-incomplete` | Activation happened but verification or the enforcement proof failed; an unproved fresh service is disabled again. | Read the receipt's inner error, fix it, rerun. An upgrade's drain ID is retained; `agc resume` it only when the host is proved. |
|
|
110
|
+
| `agc host recover PACKAGE` or `agc host recover --download` | The outgoing managed broker is dead or cannot start, so normal drain cannot finish. Recovery stages a pinned replacement and retains queued work behind its own durable guard. | Use the exact matching client and verified replacement. A new recovery refuses live work; a running matching replacement may continue its exact guard and proof. |
|
|
111
|
+
| `native-host-user-live-broker` from recovery | A broker owns the spool without a matching active recovery identity/helper; recovery refuses to replace it. | Use the normal drain/upgrade flow for a working owner. A verified replacement under the exact matching guard may continue without stop or reactivation; retry the same package/helper. |
|
|
112
|
+
| `host-recovery-worker-live`, `host-recovery-owner-invalid`, `host-recovery-invalid` | Worker absence, spool ownership, recovery metadata, or the bound replacement/helper cannot be proven. | Resolve the stated cause without deleting the spool or changing the recovery identity. Retry the same exact verified package and helper path. |
|
|
113
|
+
| `host-recovery-settling` | Dead running rows have not yet settled; the enforcement proof cannot be admitted. | The command waits up to 30 seconds. If it parks, inspect the reported phase and repair before retrying. Ordinary queued work remains blocked. |
|
|
114
|
+
| `host-recovery-proof-failed`, `host-recovery-proof-timeout` | The bound proof did not produce a passed enforced CPU receipt within its 120-second bound. | Repair the enforcement failure and retry the same package/helper. Only a successful applied/peak receipt permits completion. |
|
|
115
|
+
| `host-recovery-exhausted` | Three proof attempts have already been used across retries. | Operator repair is required. Do not delete or reset the guard, identity, or attempt budget to bypass exhaustion. |
|
|
116
|
+
| `native-host-recovery-incomplete` | A bounded recovery phase failed. Before completion, any established guard remains; a completion error instead reports whether its durable result could be confirmed. Service state is reported as unchanged, stopped, or stop unconfirmed. | Preserve the recovery ID and incident evidence. Inspect the durable completion receipt if the reply was lost; exact-ID completion retries return the same validated open receipt after a committed completion. Otherwise repair and retry the same package/helper; drain/resume cannot clear an active recovery guard. |
|
|
107
117
|
| `native-host-upgrade-drain-invalid` | The drain receipt the upgrade needs is missing or malformed. | Rerun `agc drain`, then the upgrade. |
|
|
118
|
+
| `systemctl --user stop` or `restart` does not return | A graceful stop waits for running jobs, up to `stop_grace` (600 seconds by default). | Wait, or interrupt them now with `systemctl --user kill --kill-whom=main --signal=SIGTERM agcoord-broker.service`; they end `interrupted` (`broker-stopped`). Never use a plain `systemctl kill`, which signals the jobs too. |
|
|
108
119
|
|
|
109
120
|
## Enforced-host probe failures
|
|
110
121
|
|