agcoord 0.7.0__tar.gz → 0.7.2__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (110) hide show
  1. {agcoord-0.7.0 → agcoord-0.7.2}/CHANGELOG.md +56 -0
  2. {agcoord-0.7.0 → agcoord-0.7.2}/Cargo.lock +1 -1
  3. {agcoord-0.7.0 → agcoord-0.7.2}/Cargo.toml +1 -1
  4. {agcoord-0.7.0 → agcoord-0.7.2}/PKG-INFO +1 -1
  5. {agcoord-0.7.0 → agcoord-0.7.2}/conformance/manifest-v3.json +60 -0
  6. {agcoord-0.7.0 → agcoord-0.7.2}/docs/concepts.md +4 -3
  7. {agcoord-0.7.0 → agcoord-0.7.2}/docs/coordinator.md +46 -7
  8. {agcoord-0.7.0 → agcoord-0.7.2}/docs/native_broker.md +1 -1
  9. {agcoord-0.7.0 → agcoord-0.7.2}/docs/native_host.md +88 -12
  10. {agcoord-0.7.0 → agcoord-0.7.2}/docs/releasing.md +6 -2
  11. {agcoord-0.7.0 → agcoord-0.7.2}/docs/troubleshooting.md +15 -4
  12. {agcoord-0.7.0 → agcoord-0.7.2}/native/agcoord-broker/src/broker.rs +124 -53
  13. {agcoord-0.7.0 → agcoord-0.7.2}/native/agcoord-broker/src/cgroup.rs +44 -0
  14. {agcoord-0.7.0 → agcoord-0.7.2}/native/agcoord-broker/src/host.rs +26 -0
  15. {agcoord-0.7.0 → agcoord-0.7.2}/native/agcoord-broker/src/main.rs +52 -0
  16. agcoord-0.7.2/native/agcoord-broker/src/recovery.rs +471 -0
  17. {agcoord-0.7.0 → agcoord-0.7.2}/native/agcoord-broker/src/store.rs +205 -68
  18. agcoord-0.7.2/native/agcoord-broker/tests/host_recovery.rs +550 -0
  19. {agcoord-0.7.0 → agcoord-0.7.2}/native/agcoord-broker/tests/scheduler.rs +453 -14
  20. {agcoord-0.7.0 → agcoord-0.7.2}/scripts/check-conformance +1 -1
  21. {agcoord-0.7.0 → agcoord-0.7.2}/scripts/install-native-host +25 -6
  22. agcoord-0.7.2/scripts/test-native-host-recovery +249 -0
  23. {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord/__init__.py +1 -1
  24. {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord/cli.py +124 -6
  25. {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord/config.py +25 -2
  26. {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord/native_host.py +363 -28
  27. agcoord-0.7.2/src/agcoord/native_host_pin.json +5 -0
  28. {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord/queue.py +42 -8
  29. {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord.egg-info/PKG-INFO +1 -1
  30. {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord.egg-info/SOURCES.txt +5 -0
  31. {agcoord-0.7.0 → agcoord-0.7.2}/tests/conftest.py +6 -0
  32. {agcoord-0.7.0 → agcoord-0.7.2}/tests/test_cli.py +97 -0
  33. {agcoord-0.7.0 → agcoord-0.7.2}/tests/test_conformance_gate.py +9 -3
  34. agcoord-0.7.2/tests/test_host_config.py +201 -0
  35. {agcoord-0.7.0 → agcoord-0.7.2}/tests/test_native_client.py +7 -0
  36. agcoord-0.7.2/tests/test_native_host_recovery.py +402 -0
  37. {agcoord-0.7.0 → agcoord-0.7.2}/tests/test_native_host_upgrade.py +9 -1
  38. {agcoord-0.7.0 → agcoord-0.7.2}/tests/test_release_candidate.py +4 -1
  39. {agcoord-0.7.0 → agcoord-0.7.2}/tests/test_user_broker_install.py +5 -1
  40. agcoord-0.7.0/src/agcoord/native_host_pin.json +0 -5
  41. {agcoord-0.7.0 → agcoord-0.7.2}/AGENTS.md +0 -0
  42. {agcoord-0.7.0 → agcoord-0.7.2}/LICENSE +0 -0
  43. {agcoord-0.7.0 → agcoord-0.7.2}/MANIFEST.in +0 -0
  44. {agcoord-0.7.0 → agcoord-0.7.2}/README.md +0 -0
  45. {agcoord-0.7.0 → agcoord-0.7.2}/docs/agents.md +0 -0
  46. {agcoord-0.7.0 → agcoord-0.7.2}/docs/assets/README.md +0 -0
  47. {agcoord-0.7.0 → agcoord-0.7.2}/docs/assets/agcoord-gourd-mascot.png +0 -0
  48. {agcoord-0.7.0 → agcoord-0.7.2}/docs/changelog.md +0 -0
  49. {agcoord-0.7.0 → agcoord-0.7.2}/docs/comparison.md +0 -0
  50. {agcoord-0.7.0 → agcoord-0.7.2}/docs/conformance.md +0 -0
  51. {agcoord-0.7.0 → agcoord-0.7.2}/docs/contributing.md +0 -0
  52. {agcoord-0.7.0 → agcoord-0.7.2}/docs/guardrails.md +0 -0
  53. {agcoord-0.7.0 → agcoord-0.7.2}/docs/index.md +0 -0
  54. {agcoord-0.7.0 → agcoord-0.7.2}/docs/native_migration.md +0 -0
  55. {agcoord-0.7.0 → agcoord-0.7.2}/docs/overview.md +0 -0
  56. {agcoord-0.7.0 → agcoord-0.7.2}/docs/quickstart.md +0 -0
  57. {agcoord-0.7.0 → agcoord-0.7.2}/docs/session_handoff_format.md +0 -0
  58. {agcoord-0.7.0 → agcoord-0.7.2}/docs/why.md +0 -0
  59. {agcoord-0.7.0 → agcoord-0.7.2}/native/THIRD_PARTY_LICENSES.tsv +0 -0
  60. {agcoord-0.7.0 → agcoord-0.7.2}/native/agcoord-broker/Cargo.toml +0 -0
  61. {agcoord-0.7.0 → agcoord-0.7.2}/native/agcoord-broker/build.rs +0 -0
  62. {agcoord-0.7.0 → agcoord-0.7.2}/native/agcoord-broker/src/error.rs +0 -0
  63. {agcoord-0.7.0 → agcoord-0.7.2}/native/agcoord-broker/src/platform.rs +0 -0
  64. {agcoord-0.7.0 → agcoord-0.7.2}/native/agcoord-broker/src/project_quota.rs +0 -0
  65. {agcoord-0.7.0 → agcoord-0.7.2}/native/agcoord-broker/src/resources.rs +0 -0
  66. {agcoord-0.7.0 → agcoord-0.7.2}/native/agcoord-broker/src/worker.rs +0 -0
  67. {agcoord-0.7.0 → agcoord-0.7.2}/native/agcoord-broker/tests/client_compatibility.rs +0 -0
  68. {agcoord-0.7.0 → agcoord-0.7.2}/native/agcoord-broker/tests/host_deployment.rs +0 -0
  69. {agcoord-0.7.0 → agcoord-0.7.2}/native/agcoord-broker/tests/identity.rs +0 -0
  70. {agcoord-0.7.0 → agcoord-0.7.2}/packaging/apparmor/usr.libexec.agcoord.agcoord-broker +0 -0
  71. {agcoord-0.7.0 → agcoord-0.7.2}/packaging/systemd/agcoord-broker.service +0 -0
  72. {agcoord-0.7.0 → agcoord-0.7.2}/pyproject.toml +0 -0
  73. {agcoord-0.7.0 → agcoord-0.7.2}/rust-toolchain.toml +0 -0
  74. {agcoord-0.7.0 → agcoord-0.7.2}/scripts/audit-native-broker +0 -0
  75. {agcoord-0.7.0 → agcoord-0.7.2}/scripts/build-native-broker +0 -0
  76. {agcoord-0.7.0 → agcoord-0.7.2}/scripts/build-native-host-package +0 -0
  77. {agcoord-0.7.0 → agcoord-0.7.2}/scripts/check-native-host-package +0 -0
  78. {agcoord-0.7.0 → agcoord-0.7.2}/scripts/check-native-licenses +0 -0
  79. {agcoord-0.7.0 → agcoord-0.7.2}/scripts/check-native-reproducible +0 -0
  80. {agcoord-0.7.0 → agcoord-0.7.2}/scripts/native-source-id +0 -0
  81. {agcoord-0.7.0 → agcoord-0.7.2}/scripts/test-native-host-enforcement +0 -0
  82. {agcoord-0.7.0 → agcoord-0.7.2}/scripts/verify-release-candidate +0 -0
  83. {agcoord-0.7.0 → agcoord-0.7.2}/setup.cfg +0 -0
  84. {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord/__main__.py +0 -0
  85. {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord/avoid.py +0 -0
  86. {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord/frame.py +0 -0
  87. {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord/github.py +0 -0
  88. {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord/github_release.py +0 -0
  89. {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord/land.py +0 -0
  90. {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord/merge.py +0 -0
  91. {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord/native_client.py +0 -0
  92. {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord/py.typed +0 -0
  93. {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord/pytest_xdist.py +0 -0
  94. {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord/resources.py +0 -0
  95. {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord/tui.py +0 -0
  96. {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord.egg-info/dependency_links.txt +0 -0
  97. {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord.egg-info/entry_points.txt +0 -0
  98. {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord.egg-info/requires.txt +0 -0
  99. {agcoord-0.7.0 → agcoord-0.7.2}/src/agcoord.egg-info/top_level.txt +0 -0
  100. {agcoord-0.7.0 → agcoord-0.7.2}/tests/test_avoid.py +0 -0
  101. {agcoord-0.7.0 → agcoord-0.7.2}/tests/test_distribution.py +0 -0
  102. {agcoord-0.7.0 → agcoord-0.7.2}/tests/test_github_adapter.py +0 -0
  103. {agcoord-0.7.0 → agcoord-0.7.2}/tests/test_leases.py +0 -0
  104. {agcoord-0.7.0 → agcoord-0.7.2}/tests/test_merge.py +0 -0
  105. {agcoord-0.7.0 → agcoord-0.7.2}/tests/test_native_host_assets.py +0 -0
  106. {agcoord-0.7.0 → agcoord-0.7.2}/tests/test_native_host_download.py +0 -0
  107. {agcoord-0.7.0 → agcoord-0.7.2}/tests/test_pytest_xdist.py +0 -0
  108. {agcoord-0.7.0 → agcoord-0.7.2}/tests/test_queue.py +0 -0
  109. {agcoord-0.7.0 → agcoord-0.7.2}/tests/test_resource_bindings.py +0 -0
  110. {agcoord-0.7.0 → agcoord-0.7.2}/tests/test_tui.py +0 -0
@@ -3,6 +3,62 @@
3
3
  All notable user-facing changes to AGCoord are recorded here. Versions follow semantic
4
4
  versioning; dates use ISO 8601.
5
5
 
6
+ ## Unreleased
7
+
8
+ ## 0.7.2 — 2026-10-01
9
+
10
+ - Add `agc host recover PACKAGE` and `agc host recover --download` for a dead managed owner
11
+ that cannot drain. Recovery always verifies a pinned replacement; a new recovery refuses live
12
+ or ambiguous workers, and preserves queued IDs and history behind a durable guard. Only the bound
13
+ `cpu=1` enforcement proof can run before successful completion reopens ordinary work. Dead
14
+ running rows become `interrupted`; bounded failures park under the same recovery identity.
15
+ Three proof attempts persist across retries, which require the same replacement and helper.
16
+ A running replacement under the exact matching guard can continue its existing proof without
17
+ reactivation; exact-ID completion retries return the validated durable result after a lost reply
18
+ (#242).
19
+ - Recover when systemd recreates the managed broker's delegated cgroup root after every worker
20
+ recorded against the old root is gone. The replacement now abandons only its private recovery
21
+ metadata, records those dead rows `interrupted`, and admits later work against the new root;
22
+ a live worker or any other cgroup identity mismatch still fails closed (#235).
23
+ - A fresh host install now declares the machine's memory as well as its CPUs, and enforces both
24
+ when the host allows it. The configuration written when a state directory has none derives
25
+ `memory` capacity from `MemTotal` minus a reserve (the larger of 4 GiB or an eighth of RAM) and
26
+ adds the required cgroup-v2 `memory` binding only when that controller is delegated to the
27
+ broker's slice, so an install can no longer write a configuration the broker refuses at
28
+ startup. A state directory that already holds a configuration keeps it (#226).
29
+ - Add `agc host config`, which prints the configuration this host would be given and writes it
30
+ with `--write`, refusing to replace an existing file without `--force`. `--cpu`, `--jobs`,
31
+ `--memory`, and `--reserve` override what is derived, `--cgroup-root` selects the slice whose
32
+ delegated controllers decide the bindings, and `--tmpfs` with `--tmpfs-inodes` adds an
33
+ enforced scratch policy that stays opt-in because it is bounded by the memory capacity (#226).
34
+
35
+ ## 0.7.1 — 2026-09-11
36
+
37
+ - Fix the native broker and its waiting clients spending a host's CPU in proportion to the
38
+ finished history in the spool. Every 25 ms scheduling tick and every `status` poll decoded
39
+ every terminal row ever recorded, so on a host with a few thousand finished gates the idle
40
+ broker held a whole core and each waiting `agc run` or `agc land` spent about a fifth of a
41
+ CPU-second per poll, leaving an admitted row a fraction of the CPU it claimed. Scheduling,
42
+ observation, and `status` now read only queued and running rows. A following client also
43
+ backs off from a tenth of a second to one second while its job is queued, and a `--json`
44
+ wait does the same for the job's whole life (#217).
45
+ - Stopping the native broker, including `systemctl --user stop` and `restart`, now drains instead
46
+ of cancelling in-flight work. The broker stops admitting, lets running jobs finish, and exits;
47
+ queued jobs wait for the next owner. A job still running when the new `stop_grace`
48
+ configuration elapses (600 seconds by default; `0` interrupts at once), or when a second stop
49
+ signal arrives, is ended and recorded `interrupted` with exit status 125 and failure reason
50
+ `broker-stopped`, and a following client says it claims no verdict and should be submitted
51
+ again. Previously such a job was recorded `cancelled` with exit 130, indistinguishable from a
52
+ caller's own cancellation (#218).
53
+ - Fix a client or a gate intermittently refused with `broker-schema-invalid: coordinator
54
+ database has no readable protocol metadata` while the spool was only busy. The broker's
55
+ own short-lived connections were often the spool's last, and closing the last connection to a
56
+ WAL database checkpoints and deletes the WAL under an exclusive lock that every opening reader
57
+ waits behind; on a slow disk that outlasted a short `database_timeout`, and the protocol read
58
+ reported the busy lock as a schema fault. A live broker now holds one connection for as long
59
+ as it owns the spool, and a busy protocol read is refused as the retryable
60
+ `broker-database-busy` (#223).
61
+
6
62
  ## 0.7.0 — 2026-09-06
7
63
 
8
64
  - `agc land --reuse-full` lets one passed `agc full` receipt stand in for the land gate instead
@@ -4,7 +4,7 @@ version = 4
4
4
 
5
5
  [[package]]
6
6
  name = "agcoord-broker"
7
- version = "0.7.0"
7
+ version = "0.7.2"
8
8
  dependencies = [
9
9
  "libc",
10
10
  "rusqlite",
@@ -3,7 +3,7 @@ members = ["native/agcoord-broker"]
3
3
  resolver = "2"
4
4
 
5
5
  [workspace.package]
6
- version = "0.7.0"
6
+ version = "0.7.2"
7
7
  edition = "2024"
8
8
  rust-version = "1.94.1"
9
9
  license = "MIT"
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: agcoord
3
- Version: 0.7.0
3
+ Version: 0.7.2
4
4
  Summary: Local CI and merge queue for coding agents that share one machine
5
5
  License-Expression: MIT
6
6
  Project-URL: Homepage, https://github.com/xu-hao/agcoord
@@ -176,6 +176,27 @@
176
176
  ]
177
177
  }
178
178
  },
179
+ {
180
+ "id": "contention.owner-holds-the-spool-open",
181
+ "domain": "contention",
182
+ "description": "A live owner keeps one spool connection open so readers never wait behind a last-connection WAL checkpoint, and a busy protocol read is refused as retryable contention rather than a schema fault.",
183
+ "tests": {
184
+ "rust_native": [
185
+ "scheduler::a_serving_broker_never_closes_the_last_spool_connection",
186
+ "scheduler::a_busy_protocol_read_is_refused_as_retryable_contention"
187
+ ]
188
+ }
189
+ },
190
+ {
191
+ "id": "contention.history-independent-polling",
192
+ "domain": "contention",
193
+ "description": "Scheduling ticks and status polls read only live rows, so the broker and its waiting clients spend no CPU in proportion to terminal history.",
194
+ "tests": {
195
+ "rust_native": [
196
+ "scheduler::scheduling_ticks_and_status_polls_do_not_pay_for_terminal_history"
197
+ ]
198
+ }
199
+ },
179
200
  {
180
201
  "id": "contention.durable-maintenance-drain",
181
202
  "domain": "contention",
@@ -197,6 +218,19 @@
197
218
  ]
198
219
  }
199
220
  },
221
+ {
222
+ "id": "cancellation.graceful-stop-drains",
223
+ "domain": "cancellation",
224
+ "description": "A graceful stop admits nothing new and lets running jobs finish; after the stop grace or a second stop signal it interrupts them as broker-stopped instead of cancelling.",
225
+ "tests": {
226
+ "rust_native": [
227
+ "scheduler::graceful_stop_lets_running_work_finish_and_leaves_queued_work_for_the_next_owner",
228
+ "scheduler::graceful_stop_interrupts_work_that_outlives_the_stop_grace",
229
+ "scheduler::a_second_stop_signal_interrupts_running_work_without_waiting_for_the_grace",
230
+ "scheduler::graceful_stop_retries_a_contended_interruption_commit"
231
+ ]
232
+ }
233
+ },
200
234
  {
201
235
  "id": "recovery.adopt-without-duplicate",
202
236
  "domain": "recovery",
@@ -207,6 +241,17 @@
207
241
  ]
208
242
  }
209
243
  },
244
+ {
245
+ "id": "recovery.recreated-cgroup-root",
246
+ "domain": "recovery",
247
+ "description": "A replacement owner abandons a reused cgroup root only for dead recorded workers and refuses while a matching worker is alive.",
248
+ "tests": {
249
+ "rust_native": [
250
+ "scheduler::replacement_recovers_a_dead_worker_after_the_cgroup_root_is_recreated",
251
+ "scheduler::replacement_refuses_a_recreated_cgroup_root_while_its_worker_is_live"
252
+ ]
253
+ }
254
+ },
210
255
  {
211
256
  "id": "crash.database-transactions",
212
257
  "domain": "crash_database",
@@ -328,6 +373,21 @@
328
373
  "scheduler::required_cgroup_binding_refuses_before_user_code_when_delegation_is_unavailable"
329
374
  ]
330
375
  }
376
+ },
377
+ {
378
+ "id": "recovery.guarded-managed-host",
379
+ "domain": "recovery",
380
+ "description": "A dead managed owner can be replaced while queued identities and history remain guarded; live owners or workers refuse recovery and ordinary work stays blocked across restarts.",
381
+ "tests": {
382
+ "rust_native": [
383
+ "host_recovery::recovery_preserves_queued_work_and_guards_admission_across_restarts",
384
+ "host_recovery::recovery_refuses_a_live_owner_and_a_live_retained_worker",
385
+ "host_recovery::a_successful_unenforced_proof_cannot_complete_or_resume_recovery",
386
+ "host_recovery::mismatched_recovery_identity_and_changed_helper_preserve_the_guard",
387
+ "host_recovery::failed_proofs_retry_with_the_same_identity_and_park_at_the_attempt_bound",
388
+ "host_recovery::parking_cancels_only_the_active_proof_and_allows_a_bounded_retry"
389
+ ]
390
+ }
331
391
  }
332
392
  ],
333
393
  "intentional_differences": [
@@ -127,9 +127,10 @@ Exact rules: [durable job shape](coordinator.md#durable-job-shape).
127
127
 
128
128
  Every job is `queued` until the broker admits it, `running` while its command lives, and then
129
129
  exactly one of four terminal states. `passed` and `failed` carry the command's verdict.
130
- `cancelled` means someone asked, with `agc cancel` or a graceful broker stop; the whole process
131
- group is gone before the row becomes terminal. `interrupted` means the worker vanished before
132
- it could report, and the coordinator claims no verdict for it. Terminal rows are immutable
130
+ `cancelled` means someone asked with `agc cancel`; the whole process group is gone before the
131
+ row becomes terminal. `interrupted` means the coordinator claims no verdict: the worker vanished
132
+ before it could report (`worker-result-lost`), or a stopping broker ended the job after its
133
+ stop grace (`broker-stopped`). A graceful broker stop first lets running jobs finish. Terminal rows are immutable
133
134
  history until `agc clear` removes them while the queue is idle.
134
135
 
135
136
  A `land` moves through named phases inside `running`. *Preflight* checks the pull request,
@@ -41,8 +41,15 @@ the recorded process identity before classifying a live job; it never reruns an
41
41
  spawned command merely because the old broker disappeared. An exception that ends the broker
42
42
  does not request cancellation or signal live process groups: ownership is released so a
43
43
  replacement can adopt each worker whose recorded PID and process-start token still match.
44
- An explicit broker close remains a graceful cancellation boundary and reaps safe workers
45
- before releasing ownership.
44
+ A graceful stop — SIGTERM or SIGINT, which is what `systemctl --user stop` and `restart` send —
45
+ drains instead of cancelling. The owner stops admitting work, leaves queued rows for the next
46
+ owner, keeps observing running jobs and granting their child CPU leases, and releases ownership
47
+ once none is running. If jobs are still running when `stop_grace` elapses, or when a second stop
48
+ signal arrives, the owner ends each one that is not publishing as a complete process group and
49
+ records it `interrupted` with exit status 125 and failure reason `broker-stopped`, so its agent
50
+ can tell an operator stop from a verdict and submit it again. A publishing land or retained
51
+ merge is never interrupted; the owner waits for its authoritative result. A caller's own
52
+ `agc cancel` during a stop still records `cancelled`.
46
53
 
47
54
  A land worker reports its final overall status durably while its row is still running. After
48
55
  an unclean owner loss, a replacement preserves the live worker's lane and resources, never
@@ -50,6 +57,12 @@ reruns its gate, waits for the exact process group to disappear, reclaims scratc
50
57
  exposes the reported passed or typed-failed result. If the worker disappears before it can
51
58
  report a result, `interrupted` is the only safe terminal classification.
52
59
 
60
+ If systemd recreated the broker service's delegated cgroup root after every recorded worker
61
+ using the old root disappeared, the replacement abandons only its private recovery metadata
62
+ for that old root, records the dead rows `interrupted`, and prepares later work under the new
63
+ root. It never removes or signals anything in the new root. A matching live worker, or any
64
+ other identity mismatch, still refuses recovery rather than treating a reused path as owned.
65
+
53
66
  ## Durable maintenance drain
54
67
 
55
68
  Maintenance closes the submission race before waiting for an idle spool:
@@ -105,7 +118,7 @@ the entire machine:
105
118
 
106
119
  One JSON file, `config.json` in the state directory, configures the broker that owns that
107
120
  directory. It holds at most `capacities`, `bindings`, `cgroup_root`, `cgroup_io`,
108
- `database_timeout`, `land_gate_reuse_max_age`, and `native_broker`; invalid JSON, a top-level
121
+ `database_timeout`, `land_gate_reuse_max_age`, `stop_grace`, and `native_broker`; invalid JSON, a top-level
109
122
  value that is not an object,
110
123
  an unknown key, a section that is not an object, or an empty `cgroup_root` is refused when the
111
124
  client or broker loads its configuration. When present, `cgroup_io` contains exactly one
@@ -113,10 +126,23 @@ nonempty `paths` list of unique absolute strings. `database_timeout` is a positi
113
126
  of seconds no greater than `2147483.647` (SQLite's millisecond limit) and defaults to `10`.
114
127
  `land_gate_reuse_max_age` is a finite number of seconds from `0` to `2147483.647`, defaults to
115
128
  `3600`, and bounds how long a passed `full` receipt may stand in for a land gate; `0` refuses
116
- every reuse on that host even when a caller asks for one. An
129
+ every reuse on that host even when a caller asks for one. `stop_grace` is a finite number of
130
+ seconds from `0` to `2147483.647`, defaults to `600`, and bounds how long a gracefully stopping
131
+ broker lets running jobs finish before it interrupts them; `0` interrupts them at once. An
117
132
  absent file uses `/usr/libexec/agcoord/agcoord-broker`, requires its release trust policy, and
118
133
  defaults capacity to `jobs=2`.
119
134
 
135
+ `agc host config` prints the configuration this host would be given, and writes it with
136
+ `--write`; it refuses to replace an existing file without `--force`, because a live owner keeps
137
+ the configuration it acquired the spool with. The same derivation writes the configuration an
138
+ install creates when a state directory has none: `cpu` and `jobs` from the available
139
+ CPU-affinity count, `memory` from `MemTotal` minus a reserve, and a required cgroup-v2 binding
140
+ for each of those kinds whose controller is delegated to the broker's slice, so a derived
141
+ configuration is never one the broker refuses at startup. `--cpu`, `--jobs`, `--memory`, and
142
+ `--reserve` override what is derived, `--cgroup-root` selects the slice to inspect, and
143
+ `--tmpfs` with `--tmpfs-inodes` adds the scratch policy, which stays opt-in because its bytes
144
+ are charged against the same memory capacity.
145
+
120
146
  ```json
121
147
  {
122
148
  "capacities": {"jobs": 4, "cpu": 8, "browser": 1},
@@ -157,7 +183,12 @@ wait without changing existing connections.
157
183
 
158
184
  Schema setup places every current-protocol spool in SQLite WAL journal mode, including an
159
185
  existing spool the next time a compatible client or broker opens it. Readers therefore do not
160
- block behind an ordinary writer. The configured timeout bounds each remaining lock wait;
186
+ block behind an ordinary writer. A live owner also keeps one connection open for as long as
187
+ it owns the spool. Closing the last connection to a WAL database checkpoints and removes the
188
+ WAL under an exclusive lock that every opening reader must wait behind, so an owner whose own
189
+ short-lived connections were the last would stall clients on a slow disk. The configured
190
+ timeout bounds each remaining lock wait, and a lock that outlasts it is reported as the
191
+ retryable `broker-database-busy`, never as a schema fault;
161
192
  transient busy or locked results in the broker pump and idle health check are retried, and a
162
193
  contended best-effort activity heartbeat never changes an already successful public operation
163
194
  into an apparent failure.
@@ -963,8 +994,9 @@ target update.
963
994
 
964
995
  A land request is cancellable while queued, preflighting, or gating. Once its durable phase
965
996
  is `publishing`, cancellation is refused because killing a client during an authenticated
966
- atomic mutation would leave the outcome indeterminate. Graceful broker stop cancels safe
967
- earlier phases but waits for publishing and records its authoritative result.
997
+ atomic mutation would leave the outcome indeterminate. A graceful broker stop lets every phase
998
+ finish; once `stop_grace` elapses it interrupts earlier phases as `broker-stopped` but still
999
+ waits for publishing and records its authoritative result.
968
1000
 
969
1001
  ### Reusing a full receipt as the land gate
970
1002
 
@@ -1072,6 +1104,13 @@ for the configured `database_timeout` rather than aborting on the first busy loc
1072
1104
  broker committing a publication or checkpointing its WAL does not end a client that is only
1073
1105
  watching its own row.
1074
1106
 
1107
+ Watching costs the host little while work waits. A following client polls a queued job's row
1108
+ at an interval that doubles from a tenth of a second to one second, then streams the admitted
1109
+ job's log every tenth of a second; a `--json` wait, which streams nothing, backs off the same
1110
+ way for the job's whole life. The broker's scheduling tick and each `status` read only queued
1111
+ and running rows, so neither grows with the terminal history that accumulates until
1112
+ `agc clear`.
1113
+
1075
1114
  The native owner authenticates a worker with its PID, Linux start token, and requirement that
1076
1115
  the PID lead its recorded process group. Replacement recovery adopts only that exact live
1077
1116
  identity. If the PID has been reused, its token changed, or it belongs to another group, the run
@@ -142,7 +142,7 @@ owner record is:
142
142
  pid=12345
143
143
  protocol=5
144
144
  implementation=rust-native
145
- version=0.7.0
145
+ version=0.7.1
146
146
  build=sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
147
147
  capacities={"cpu":8,"jobs":2}
148
148
  resource_bindings={}
@@ -63,14 +63,20 @@ Create the state directory as the broker user with mode `0700` and `config.json`
63
63
  `app.slice`, and `DelegateSubgroup=supervisor` make the configured service root deterministic
64
64
  and keep the broker process out of the inner node where it enables controllers.
65
65
 
66
- On a fresh `agc host install`, an absent configuration is created with `cpu` and `jobs` both set
67
- to the invoking process's available CPU-affinity count, plus the required cgroup-v2
68
- `cpu/logical-cpu` binding shown above. The scheduler greedily packs each job's complete declared
69
- resource vector against those totals; it does not divide the machine into a fixed number of
70
- equal CPU partitions. `jobs` remains a ceiling for submissions that omit other claims. To add
71
- memory, RAM-disk, persistent-disk, I/O, or project-defined capacities, create the owner-only
72
- configuration before first install; the command validates and preserves an existing safe
73
- configuration instead of overwriting it.
66
+ On a fresh `agc host install`, an absent configuration is derived from the host: `cpu` and `jobs`
67
+ both take the invoking process's available CPU-affinity count, `memory` takes `MemTotal` minus a
68
+ reserve of the larger of 4 GiB or an eighth of RAM, and each of those kinds receives the required
69
+ cgroup-v2 binding shown above only when its controller is delegated to the service's slice. A
70
+ kind whose controller is missing keeps its capacity and stays admission-only, so the derived
71
+ configuration is never one the broker refuses at startup. Run `agc host config` first to see
72
+ exactly what this host would be given, and `agc host config --write` to install it before the
73
+ first `agc host install`. The scheduler greedily packs each job's complete declared resource
74
+ vector against those totals; it does not divide the machine into a fixed number of equal CPU
75
+ partitions. `jobs` remains a ceiling for submissions that omit other claims. To add RAM-disk,
76
+ persistent-disk, I/O, or project-defined capacities, or to choose your own memory reserve or a
77
+ tmpfs scratch policy, write the owner-only configuration before first install, with
78
+ `agc host config --tmpfs SIZE --write` or by hand; the command validates and preserves an
79
+ existing safe configuration instead of overwriting it.
74
80
 
75
81
  On every managed start, the broker fails before acquiring the spool unless all of these match:
76
82
 
@@ -378,17 +384,87 @@ a nondefault, unsafe, fresh, or pre-protocol-5 spool. A failure before activatio
378
384
  host unchanged; a failure after stop reports the exact drain ID and leaves the service stopped
379
385
  and coordinator drained unless the resume step had already succeeded.
380
386
 
381
- For manual upgrade or recovery, run `stage` while the current service remains available. Inspect
382
- the staged package digest, run `agc drain`, retain its exact ID, and stop the user service after
383
- the receipt says `drained`. Run `activate --drain-id ID`, `daemon-reload`, any required `migrate`,
387
+ For manual upgrade or recovery where the outgoing broker can drain, run `stage` while the
388
+ current service remains available. Inspect the staged package digest, run `agc drain`, retain
389
+ its exact ID, and stop the user service after the receipt says `drained`. Run
390
+ `activate --drain-id ID`, `daemon-reload`, any required `migrate`,
384
391
  exact-ID `resume`, and `start` in that order. An explicitly chosen cancellation policy may shorten
385
392
  the drain, but cancellation never replaces its durable submission guard. Never replace the live
386
393
  binary and ask systemd to restart while work remains. After start, inspect `systemctl --user
387
394
  status agcoord-broker.service`, `agc list`, and rerun the enforced-host proof.
388
395
 
396
+ ### Guarded recovery when the outgoing broker cannot drain
397
+
398
+ The normal drain flow depends on a functioning outgoing broker. It cannot produce a completed
399
+ drain when that broker is dead or cannot start. Use the separate managed recovery operation
400
+ with the exact matching client and a pinned replacement:
401
+
402
+ ```bash
403
+ agc host recover /path/to/native-host-bundle/agcoord-native-host-x86_64-linux.tar.gz
404
+ # Or fetch the matching release bundle:
405
+ agc host recover --download
406
+ ```
407
+
408
+ Recovery always requires a broker pin, including a bundle supplied by path. Starting a new
409
+ recovery refuses a live owner, live or ambiguous recorded workers, unavailable identity
410
+ evidence, and an invalid package. It never substitutes a forced activation or deletion of the
411
+ spool. Normal `agc host upgrade` and its drain flow remain unchanged.
412
+
413
+ The command stages the verified replacement and stops the failed service. The staged native
414
+ holder takes the exclusive owner/maintenance lock, proves every recorded worker identity is
415
+ gone, and establishes a durable `recovery-` ID followed by 12 hexadecimal digits. It holds the
416
+ lock through activation and installed-identity verification. The restarted replacement
417
+ settles dead running rows as `interrupted`, without an execution verdict. Queued job IDs and
418
+ their work, terminal history, and the spool remain intact; recovery never reruns an interrupted
419
+ or terminal command.
420
+
421
+ The durable guard blocks new submissions and ordinary queued-job admission through activation
422
+ and restart. Recovery admits only the exact bound enforcement helper, with `cpu=1` and
423
+ `jobs=1`, while ordinary work stays blocked. Completion requires that proof to pass with exit
424
+ status zero, a receipt with requested and applied CPU both 1 and observed peak CPU at least 1,
425
+ the unchanged helper digest, and no running rows. Only successful completion clears the guard.
426
+ A process exit, service restart, normal `agc drain`, or `agc resume` cannot clear it.
427
+
428
+ If the verified replacement is already running under the same active recovery guard, rerunning
429
+ `agc host recover` continues that guard instead of stopping or reactivating the service. The
430
+ replacement identity and helper path and digest must match exactly. An ordinary live owner or
431
+ a mismatched guard still refuses. Continuation reuses a queued, running, or passed proof while
432
+ that recovery is verifying; it does not submit a duplicate or spend another proof attempt.
433
+
434
+ The native hold, status, proof, completion, and park operations retain the exact recovery ID,
435
+ replacement identity, helper path and digest, proof job ID, attempt count, phase, and incident.
436
+ Each host phase and enforcement proof is bounded to 120 seconds; owner acquisition and dead-row
437
+ settlement waits are each bounded to 30 seconds. At most three proof attempts are allowed
438
+ across retries. Retrying must use the same exact verified replacement package and helper path;
439
+ it continues the same recovery identity and never resets the attempt budget. Exhausted proof
440
+ attempts require operator repair, with no reset bypass.
441
+
442
+ Before completion, a failed or interrupted operation parks safely with the guard intact and
443
+ nonsecret evidence of the failed phase. If the service was touched, failure stops it or
444
+ explicitly reports that the stop is unconfirmed; an earlier staging failure leaves it unchanged.
445
+ The command does not silently resume work or claim an unproved host is ready. Preserve the
446
+ recovery ID and address the reported incident before retrying the same package.
447
+
448
+ Completion retries up to three times with the exact recovery ID. If completion committed but
449
+ its reply was lost, the native operation validates the retained completed identity and passed
450
+ CPU proof, then returns the same open receipt without another proof or recovery operation.
451
+ If completion still cannot be confirmed, the error says to inspect the durable receipt; it
452
+ does not claim the guard remains when completion may already have cleared it.
453
+
454
+ ### Normal restart and rollback
455
+
389
456
  `Restart=on-failure` recovers an unexpected broker exit without an idle shutdown. The durable
390
457
  spool remains the authority: the replacement adopts only identity-verified live workers and
391
- never reruns their command. A deliberate `systemctl stop` does not restart. If staging is
458
+ never reruns their command. A deliberate `systemctl stop` does not restart.
459
+
460
+ `systemctl --user stop` and `restart` are graceful: the broker stops admitting, lets running jobs
461
+ finish, and exits, so the command returns when the last running job ends or `stop_grace` (600
462
+ seconds unless configured) elapses and the broker interrupts the rest as `broker-stopped`. Queued
463
+ jobs wait for the restarted owner. The unit's `TimeoutStopSec=infinity` keeps systemd from
464
+ killing work during that wait. To interrupt running jobs at once, send the broker alone a second
465
+ stop signal with `systemctl --user kill --kill-whom=main --signal=SIGTERM agcoord-broker.service`;
466
+ a plain `systemctl kill` signals every process in the unit, including the jobs, and bypasses
467
+ the broker's record of why they ended. If staging is
392
468
  interrupted, no live file changed; repeat it. If activation is interrupted, leave the service
393
469
  stopped and repeat activation for the same selected package, which revalidates every final
394
470
  file before start. Never delete or replace the state directory as host-package recovery.
@@ -154,8 +154,12 @@ interface and must be smoke-tested from the built wheel.
154
154
  `0755` on only the fixed broker and three fixed helper names before running this verifier;
155
155
  content sidecars and the host archive's internal modes remain independently checked.
156
156
  7. Install the host bundle on the supported Ubuntu configuration through the staged runbook.
157
- For an existing spool, retain the exact durable drain receipt, require activation to match its
158
- ID and resume only after owner-locked maintenance completes.
157
+ For an existing spool with a functioning outgoing owner, retain the exact durable drain
158
+ receipt, require activation to match its ID, and resume only after owner-locked maintenance
159
+ completes. If the outgoing owner is dead or cannot start, use the
160
+ [guarded managed recovery path](native_host.md#guarded-recovery-when-the-outgoing-broker-cannot-drain)
161
+ with the exact pinned replacement. Retain its recovery ID and proof receipt; ordinary work
162
+ stays blocked until verified completion, and normal drain/resume cannot clear that guard.
159
163
  Keep `kernel.apparmor_restrict_unprivileged_userns=1`, start the ordinary unprivileged managed
160
164
  service, and retain the shipped `cpu=1` receipt proving its AppArmor transition, cgroup
161
165
  namespace root, exact CPU control, and durable applied/peak evidence. A spool left below
@@ -37,7 +37,8 @@ remain the authority for the behavior behind each entry, and the
37
37
  | `resource '<name>' has no configured machine capacity` | The broker's `config.json` declares no such capacity. | Claim only configured names, or add the capacity to `config.json` and restart the broker. |
38
38
  | `resource '<name>' requests N, above capacity M` | The claim can never be admitted on this machine. | Lower the claim; `agc --json list` prints `capacities`. |
39
39
  | `a coordinated job cannot submit another coordinated job; invoke it directly from the checkout` | Nested submission from inside an admitted job. | Run the tool directly; a job may only call `agc show "$AGCOORD_RUN_ID"` and `agc verify-admission`. |
40
- | `broker-draining` (`agcoord-maintenance-draining` for a direct SQLite writer) | Maintenance closed submissions; admitted work is finishing. | Wait. `list`, `show`, `log`, the TUI, and `cancel` still work. |
40
+ | `broker-draining` (`agcoord-maintenance-draining` for a direct SQLite writer) | Maintenance closed submissions. Normal drain finishes admitted work; guarded recovery admits only its bound proof. | Inspect the maintenance state. For `recovering`, follow the managed recovery receipt; ordinary submissions stay blocked until proof succeeds. `list`, `show`, `log`, the TUI, and `cancel` still work. |
41
+ | `host-recovery-required` from drain or resume | A guarded host recovery is active; ordinary drain and resume cannot clear its guard. | Preserve the recovery ID and follow [managed recovery](native_host.md#guarded-recovery-when-the-outgoing-broker-cannot-drain); retry the same verified package after repair. |
41
42
  | `broker-drained` | The spool finished draining and has no owner; nothing autostarts. | `agc resume <drain-id>` with the ID the drain printed, or finish the maintenance that needed it. |
42
43
  | `Gate queue: <id> waiting at position N for branch …` | Queued, not refused: capacity or a barrier is holding it. | `blocked_by` in `agc --json show <id>` names the jobs ahead; a `land` in the same lane, or a job from the same worktree, waits for that land. |
43
44
  | `Gate queue: lost contact with the coordinator while following <id>: …` and exit status 75 | The streaming client could not reach the coordinator for five seconds after acceptance. The job itself continues. | `agc show <id>` for the verdict and `agc log <id> --follow` to keep watching. Never read 75 as a refusal or a red gate. |
@@ -59,8 +60,9 @@ These arrive as a failed row. `failure_reason` is stable; the log has the detail
59
60
  | `publish-failed` | The forge rejected the atomic update; nothing moved. | Check `gh auth status`, branch protection (a required hosted merge queue rejects the update), and the pull request; resubmit. |
60
61
  | `memory-oom` | The job exceeded its declared memory and was ended as one process group. | Raise the `memory` claim or reduce the job's footprint; the receipt's `peak` says how far over it went. |
61
62
  | `resource-enforcement-failed` | A `required` binding could not be applied before user code; the launcher exited 125. | Read the receipt's events for the backend's code; fix the host or make the binding `best-effort`. |
62
- | status `interrupted` | The worker vanished before it could report; no verdict is claimed. | Read the log; unless it ends with `LANDED`, submit again. |
63
- | status `cancelled` | `agc cancel`, or a graceful broker stop reaped a queued or gating job. | Resubmit when appropriate. Publication is never cancelled. |
63
+ | status `interrupted`, reason `broker-stopped` | The broker was stopped and the job outlived `stop_grace`, or a second stop signal arrived; no verdict is claimed. | Submit again. Publication is never interrupted. |
64
+ | status `interrupted`, reason `worker-result-lost` | The worker vanished before it could report; no verdict is claimed. | Read the log; unless it ends with `LANDED`, submit again. |
65
+ | status `cancelled` | `agc cancel` asked for it. | Resubmit when appropriate. Publication is never cancelled. |
64
66
 
65
67
  ## Maintenance
66
68
 
@@ -84,6 +86,7 @@ These arrive as a failed row. `failure_reason` is stable; the log has the detail
84
86
  | `broker-not-running` | A command needed a live owner and found none. | Unmanaged: any `agc list` starts one. Managed: `systemctl --user status agcoord-broker.service`. |
85
87
  | `broker-database-busy` | SQLite stayed locked longer than `database_timeout` (10 s by default). | Look for a stuck writer; raise `database_timeout` in `config.json` if the disk is slow. |
86
88
  | `broker-config-invalid` | `config.json` has an unknown key, a non-object section, an invalid binding, or an empty `cgroup_root`. | Fix the file against the [configuration contract](coordinator.md#repository-lanes-and-resources). |
89
+ | `broker-row-invalid` naming `root-reused` | A release before this fix found that systemd recreated the delegated cgroup root after a broker loss and could not settle a dead worker. | Upgrade AGCoord. The replacement now abandons only old private recovery metadata after proving every recorded worker is gone; a live worker or another identity mismatch still refuses. |
87
90
  | `broker-state-invalid`, `broker-schema-invalid`, `broker-owner-metadata-invalid`, `broker-owner-lock-unavailable` | The state directory or its files are not what the broker expects. | Do not repair by hand. Check ownership and modes (`0700` directory, `0600` files), then the [native broker contract](native_broker.md#stable-refusal-envelope). |
88
91
  | `native broker identity is incompatible with this client` | The selected executable is not a protocol-5 AGCoord broker of a supported line. | Point `native_broker.path` at the installed broker, or reinstall it. |
89
92
 
@@ -96,7 +99,7 @@ These arrive as a failed row. `failure_reason` is stable; the log has the detail
96
99
  | `native-host-version-mismatch` | The bundle is not the installed client's version. | Install the matching client first; `--download` then fetches the right bundle. |
97
100
  | `native-host-bundle-invalid` | A missing, oversized, symlinked, or wrongly owned file in the bundle. | Download the bundle again, or check the directory's modes (`0700`). |
98
101
  | `native-host-pin-mismatch` | The bundle's broker is not the one this client was released against. | Fetch the release that matches the client. Never override the pin. |
99
- | `native-host-unpinned-client` | A development client carries no pin, so it cannot verify a download. | Install a release client, pass a bundle path you verified yourself, or supply `--broker-sha256`. |
102
+ | `native-host-unpinned-client` | A development client carries no pin, so it cannot verify a download. | Install a release client, pass a bundle path you verified yourself, or supply `--broker-sha256`. Recovery always requires a pin, even for a local bundle. |
100
103
  | `native-host-pin-conflict` | `--broker-sha256` disagrees with the shipped pin. | Drop the option; a supplied digest never replaces a pin. |
101
104
  | `native-host-bundle-source-conflict`, `native-host-bundle-source-missing` | Both or neither of a bundle path and `--download` (or `--user`). | Pass exactly one source. |
102
105
  | `native-host-download-insecure-url`, `native-host-download-invalid-repository`, `native-host-download-unknown-adapter`, `native-host-download-digest-mismatch` | The download source or its bytes are not acceptable. | Use an `https` base URL, an `owner/name` repository, the `github` adapter; a transport digest mismatch means a corrupted or substituted download. |
@@ -104,7 +107,15 @@ These arrive as a failed row. `failure_reason` is stable; the log has the detail
104
107
  | `native-host-install-authorization-failed`, `native-host-upgrade-authorization-failed` | `sudo` could not authorize the privileged step. | Run the command in a terminal that can answer sudo; nothing was changed. |
105
108
  | `…-stage-failed`, `…-activation-failed`, `…-reload-failed`, `…-start-failed`, `…-stop-failed` | The named phase failed; the receipt says what was rolled back. | Fix the cause the phase names and rerun the same command. |
106
109
  | `native-host-install-incomplete`, `native-host-upgrade-incomplete` | Activation happened but verification or the enforcement proof failed; an unproved fresh service is disabled again. | Read the receipt's inner error, fix it, rerun. An upgrade's drain ID is retained; `agc resume` it only when the host is proved. |
110
+ | `agc host recover PACKAGE` or `agc host recover --download` | The outgoing managed broker is dead or cannot start, so normal drain cannot finish. Recovery stages a pinned replacement and retains queued work behind its own durable guard. | Use the exact matching client and verified replacement. A new recovery refuses live work; a running matching replacement may continue its exact guard and proof. |
111
+ | `native-host-user-live-broker` from recovery | A broker owns the spool without a matching active recovery identity/helper; recovery refuses to replace it. | Use the normal drain/upgrade flow for a working owner. A verified replacement under the exact matching guard may continue without stop or reactivation; retry the same package/helper. |
112
+ | `host-recovery-worker-live`, `host-recovery-owner-invalid`, `host-recovery-invalid` | Worker absence, spool ownership, recovery metadata, or the bound replacement/helper cannot be proven. | Resolve the stated cause without deleting the spool or changing the recovery identity. Retry the same exact verified package and helper path. |
113
+ | `host-recovery-settling` | Dead running rows have not yet settled; the enforcement proof cannot be admitted. | The command waits up to 30 seconds. If it parks, inspect the reported phase and repair before retrying. Ordinary queued work remains blocked. |
114
+ | `host-recovery-proof-failed`, `host-recovery-proof-timeout` | The bound proof did not produce a passed enforced CPU receipt within its 120-second bound. | Repair the enforcement failure and retry the same package/helper. Only a successful applied/peak receipt permits completion. |
115
+ | `host-recovery-exhausted` | Three proof attempts have already been used across retries. | Operator repair is required. Do not delete or reset the guard, identity, or attempt budget to bypass exhaustion. |
116
+ | `native-host-recovery-incomplete` | A bounded recovery phase failed. Before completion, any established guard remains; a completion error instead reports whether its durable result could be confirmed. Service state is reported as unchanged, stopped, or stop unconfirmed. | Preserve the recovery ID and incident evidence. Inspect the durable completion receipt if the reply was lost; exact-ID completion retries return the same validated open receipt after a committed completion. Otherwise repair and retry the same package/helper; drain/resume cannot clear an active recovery guard. |
107
117
  | `native-host-upgrade-drain-invalid` | The drain receipt the upgrade needs is missing or malformed. | Rerun `agc drain`, then the upgrade. |
118
+ | `systemctl --user stop` or `restart` does not return | A graceful stop waits for running jobs, up to `stop_grace` (600 seconds by default). | Wait, or interrupt them now with `systemctl --user kill --kill-whom=main --signal=SIGTERM agcoord-broker.service`; they end `interrupted` (`broker-stopped`). Never use a plain `systemctl kill`, which signals the jobs too. |
108
119
 
109
120
  ## Enforced-host probe failures
110
121