agcoord 0.7.2__tar.gz → 0.7.4__tar.gz

{agcoord-0.7.2 → agcoord-0.7.4}/CHANGELOG.md

@@ -3,7 +3,45 @@
 All notable user-facing changes to AGCoord are recorded here. Versions follow semantic
 versioning; dates use ISO 8601.
 
-## Unreleased
+## 0.7.4 — 2026-10-21
+
+- Make the pinned native broker digest a property of the source and a declared toolchain. The
+  C compiler and musl versions that build the bundled SQLite are now declared in
+  `native/build-toolchain.json`, included in the native source identity, and recorded in broker
+  provenance. `build-native-broker` refuses an undeclared C toolchain, and both
+  `check-native-reproducible` and the release-candidate verifier refuse a broker built with one.
+  Native CI and tag builds run on a fixed `ubuntu-24.04` runner with exact musl packages instead
+  of `ubuntu-latest`, whose image change had produced different broker bytes from identical
+  source (#250).
+
+- Let admitted jobs build unprivileged Bubblewrap sandboxes to isolate their own test
+  processes, while every other submitted program keeps the user-namespace denial. The packaged
+  AppArmor policy enters a permissive `agcoord-admitted` anchor before `no_new_privs`; the
+  anchor's exec rule then stacks a restricting `agcoord-job` profile at `execve` (denying
+  capabilities and user-namespace creation), so submitted code runs as
+  `agcoord-admitted//&agcoord-job`. Stacking onto the retained anchor is always permitted under
+  `no_new_privs`, and the transition fails closed — the command cannot start — if `agcoord-job`
+  is not loaded. A static rule routes the fixed system executable `/usr/bin/bwrap` to
+  `agcoord-admitted//&agcoord-bwrap`, which may create a user namespace and use in-namespace
+  capabilities to assemble a sandbox, and routes everything it launches to the sticky, capability-
+  and namespace-denied `agcoord-bwrap-child`. The widening needs no per-job flag, operator
+  allowlist, or executable digest, so an ordinary Bubblewrap package upgrade keeps working; a
+  program of that name reached through another mount namespace is named under a disconnected
+  prefix and never matches. Activation refuses unless all six domains are loaded in enforce mode.
+  The observable admitted label an admitted job reports changes from `agcoord-admitted (enforce)`
+  to `agcoord-admitted//&agcoord-job (enforce)`. Bubblewrap stays an optional host package: the
+  enforced-host probe exercises the sandbox widening when `/usr/bin/bwrap` is installed and
+  reports that it was not exercised otherwise.
+
+## 0.7.3 — 2026-10-02
+
+- Recover a normal drain stranded behind a dead or unstartable managed broker without losing
+  accepted work or maintenance intent. Host repair preserves the original drain ID and
+  metadata, keeps submissions continuously guarded, and restores the drain after the bound
+  replacement-host proof. Accepted queued jobs finish normally; new submissions remain blocked
+  until explicit resume with the original drain ID after maintenance. Completion retries retain
+  that restoration result and audit instead of always opening the spool; malformed metadata
+  and live or ambiguous workers refuse, and proof attempts are never reset (#244).
 
 ## 0.7.2 — 2026-10-01
 

{agcoord-0.7.2 → agcoord-0.7.4}/Cargo.lock

@@ -4,7 +4,7 @@ version = 4
 
 [[package]]
 name = "agcoord-broker"
-version = "0.7.2"
+version = "0.7.4"
 dependencies = [
  "libc",
  "rusqlite",

{agcoord-0.7.2 → agcoord-0.7.4}/Cargo.toml

@@ -3,7 +3,7 @@ members = ["native/agcoord-broker"]
 resolver = "2"
 
 [workspace.package]
-version = "0.7.2"
+version = "0.7.4"
 edition = "2024"
 rust-version = "1.94.1"
 license = "MIT"

{agcoord-0.7.2 → agcoord-0.7.4}/MANIFEST.in

@@ -7,6 +7,7 @@ include rust-toolchain.toml
 recursive-include conformance *.json
 recursive-include docs *.md *.png
 recursive-include native *.rs *.toml *.tsv
+include native/build-toolchain.json
 recursive-include packaging *
 include scripts/*native*
 include scripts/check-conformance

{agcoord-0.7.2 → agcoord-0.7.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: agcoord
-Version: 0.7.2
+Version: 0.7.4
 Summary: Local CI and merge queue for coding agents that share one machine
 License-Expression: MIT
 Project-URL: Homepage, https://github.com/xu-hao/agcoord

{agcoord-0.7.2 → agcoord-0.7.4}/conformance/manifest-v3.json

@@ -385,7 +385,10 @@
           "host_recovery::a_successful_unenforced_proof_cannot_complete_or_resume_recovery",
           "host_recovery::mismatched_recovery_identity_and_changed_helper_preserve_the_guard",
           "host_recovery::failed_proofs_retry_with_the_same_identity_and_park_at_the_attempt_bound",
-          "host_recovery::parking_cancels_only_the_active_proof_and_allows_a_bounded_retry"
+          "host_recovery::parking_cancels_only_the_active_proof_and_allows_a_bounded_retry",
+          "host_recovery::stranded_normal_drain_can_enter_recovery_without_releasing_accepted_work",
+          "host_recovery::verified_recovery_restores_original_drain_and_replays_without_an_owner",
+          "host_recovery::malformed_original_drain_cannot_be_read_or_reused_as_a_recovery_guard"
         ]
       }
     }

{agcoord-0.7.2 → agcoord-0.7.4}/docs/coordinator.md

@@ -99,6 +99,20 @@ in one transaction and returns the spool to `open`; a wrong or stale identifier
 submissions. Migration and rollback preserve the current marker across the protocol boundary
 and never resurrect a marker already removed by a successful resume.
 
+A normal drain stranded behind a dead or unstartable managed owner may be repaired through
+[guarded managed host recovery](native_host.md#guarded-recovery-when-the-outgoing-broker-cannot-drain).
+The original drain ID, state, reason, and start time remain durable maintenance
+intent. Submission guards stay continuous while the temporary recovery guard admits only its
+bound host proof. After verification, recovery atomically restores the original drain;
+accepted queued work finishes normally, and the owner yields after the last accepted row
+becomes terminal. New submissions remain refused until explicit `resume` with the exact
+original drain ID after the intended maintenance succeeds. Repair and proof do not authorize that resume.
+Recovery completion and its retry receipt retain the original drain audit rather than always
+reporting `open`. Invalid maintenance metadata or live/ambiguous worker identities refuse;
+retries never reset proof attempts. Repeating a stranded drain alone cannot make an
+unstartable broker progress, and raw spool edits or cancellation of accepted work are not a
+repair path.
+
 ## Repository lanes and resources
 
 Each submission belongs to a stable repository lane and records its resolved worktree. A

{agcoord-0.7.2 → agcoord-0.7.4}/docs/native_broker.md

@@ -251,6 +251,17 @@ start only while the marker is `draining` and live rows remain; it adopts or saf
 those rows and completes the handoff. A `drained` marker prevents activity writes and ordinary
 autostart until exact-ID `resume` holds the owner lock and removes every guard.
 
+Guarded managed-host recovery may temporarily suspend a stranded normal drain's accepted-job
+admission to verify a pinned replacement, without relaxing its submission guards. It retains
+the original drain ID, state, reason, and start time in its durable recovery audit.
+Successful host proof restores that drain atomically instead of opening submissions: accepted
+queued rows finish normally and the owner yields when the drain completes. Only explicit
+`resume` with the exact original drain ID after maintenance reopens submissions. Completion retries preserve
+the same restoration result and never reset proof attempts or perform an implicit resume.
+Malformed maintenance metadata and live or ambiguous worker identities refuse repair. The
+[host recovery contract](native_host.md#guarded-recovery-when-the-outgoing-broker-cannot-drain)
+defines the verification-only admission, bounded failure, and parking rules.
+
 ### Implemented worker boundary
 
 The native owner now forks its launcher path directly without an internal command-line mode.
@@ -264,7 +275,11 @@ Between the releases, the child resets inherited broker signal handlers and comp
 and scratch setup. When it inherited the managed `agcoord-broker` AppArmor domain, it makes the
 one-way transition into `agcoord-admitted` and verifies that domain before clearing effective,
 permitted, inheritable, and ambient capabilities, setting `no_new_privs`, and verifying those
-values from `/proc/self/status`. It closes every inherited descriptor except standard I/O and the
+values from `/proc/self/status`. The `agcoord-admitted` anchor's exec rule then confines the
+submitted command: its `execve` transitions, at exec, to the stack `agcoord-admitted//&agcoord-job`
+(`agcoord-job` is the restriction that denies namespace creation and capabilities). Stacking onto
+the retained anchor is always allowed under `no_new_privs`, and the exec fails closed if
+`agcoord-job` is not loaded. It closes every inherited descriptor except standard I/O and the
 two private channels, reports the token-bound setup result, waits for final release, closes those
 channels, verifies the final descriptor set, and calls `execve`. Under an applied tmpfs policy it
 instead forks once more, supervises the command from those namespaces while sampling the mount for
@@ -434,14 +449,25 @@ this resource boundary does not prevent the command from explicitly naming paths
 The setup-only `agcoord-broker` AppArmor profile attaches to the immutable root-owned executable,
 not to a user-editable service directive. Its public command parser exposes no worker mode or
 arbitrary setup-domain exec path. The authenticated worker makes a one-way transition into
-`agcoord-admitted` before setting `no_new_privs` and before reporting successful setup. Arbitrary
-interpreters inherit that domain. When admitted work invokes the broker, AppArmor stacks
-`agcoord-broker-client` onto the admitted domain; adding confinement is compatible with
-`no_new_privs` and cannot regain setup permission. Both restricted profiles deny user-namespace
-creation and changing back to the setup domain. Every domain is compiled in explicit enforce
-mode rather than Ubuntu 24.04's unconfined `default_allow` mode. A failure to verify the loaded
-setup profile, service cgroup, global restriction, or backend namespace probe makes required work
-unavailable.
+`agcoord-admitted` and sets `no_new_privs`; `agcoord-admitted` is therefore the kernel's recorded
+anchor, which every later transition retains. The anchor's `/** px -> agcoord-admitted//&agcoord-job`
+exec rule confines submitted code at `execve`: the command runs as `agcoord-admitted//&agcoord-job`.
+Stacking onto the retained anchor is always allowed under `no_new_privs` (it only removes
+permission), and the exec fails closed if `agcoord-job` is not loaded; the anchor alone permits
+user-namespace creation, but submitted code always runs with `agcoord-job` stacked, which denies
+it. When admitted work invokes the broker, AppArmor substitutes `agcoord-broker-client` for
+`agcoord-job` on the anchor; adding confinement is compatible with `no_new_privs` and cannot regain
+setup permission. The one deliberate widening is the fixed system Bubblewrap
+executable `/usr/bin/bwrap`, which transitions to `agcoord-admitted//&agcoord-bwrap` and may build
+an unprivileged sandbox; everything it launches runs as
+`agcoord-admitted//&agcoord-bwrap-child//&agcoord-job` (AppArmor's sorted rendering of the stack),
+which is capability- and user-namespace-denied and sticky across further execs, so no other
+submitted code gains namespace permission. Bubblewrap is an optional host package; without
+`/usr/bin/bwrap` these domains are inert. Every one of the six domains is compiled in explicit enforce mode rather than
+Ubuntu 24.04's unconfined `default_allow` mode. A failure to verify the loaded setup profile, the
+service cgroup, the global restriction, or the backend namespace probe makes required work
+unavailable; a missing `agcoord-job` profile makes the submitted command's exec transition fail
+closed so the command cannot start.
 The complete package and transition contract is in [the native host runbook](native_host.md).
 
 ## Compatibility and migration

{agcoord-0.7.2 → agcoord-0.7.4}/docs/native_host.md

@@ -318,7 +318,9 @@ machine to the managed host later needs an empty default spool: drain the user s
 Run the shipped probe as an ordinary coordinated job with exactly one CPU unit; do not invoke
 it from a nested coordinator. It proves the admitted AppArmor transition, user-namespace
 denial, cgroup namespace root, exact `cpu.max`, release owner identity, and the restricted
-profile reached when admitted work invokes the broker. It also calls the installed Python
+profile reached when admitted work invokes the broker. When `/usr/bin/bwrap` is installed it also
+proves that Bubblewrap can build an unprivileged sandbox whose processes keep the
+capability- and user-namespace-denied restriction. It also calls the installed Python
 client's operation-specific callback by showing the exact admitted run while it is live; this
 guards the complete `agc`-to-native path used by land reporting and pytest-xdist child leases.
 The final assertion proves that the durable public receipt records the enforced allocation:
@@ -341,14 +343,45 @@ the quota when short parallel bursts are rounded upward.
 The setup-only `agcoord-broker` profile attaches only to the fixed root-owned executable; it is
 not selected by the user-editable systemd unit. The public binary has no internal-worker or
 arbitrary setup-domain exec command. The broker's authenticated in-process worker makes a one-way
-transition into `agcoord-admitted`, verifies it, and only then clears capabilities and sets
-`no_new_privs` before release. Arbitrary interpreter execution inherits that profile. When
-admitted work invokes the fixed broker, AppArmor stacks `agcoord-broker-client` onto the existing
-admitted confinement; this adds restrictions without requesting a replacement domain after
-`no_new_privs`. Both restricted profiles deny user-namespace creation and changing back to setup.
-All three profiles use explicit enforce mode and broad enumerated host permissions;
-`default_allow` is not accepted because Ubuntu 24.04 implements it as an unconfined profile that
-does not apply these denials.
+transition into `agcoord-admitted`, verifies it, clears capabilities, and sets `no_new_privs`
+before release. The `agcoord-admitted` anchor is the domain the kernel records at `no_new_privs`,
+so every later transition must retain it. The anchor never runs submitted code as itself: its
+`/** px -> agcoord-admitted//&agcoord-job` exec rule means every program it runs transitions, at
+`execve`, to the stack `agcoord-admitted//&agcoord-job`. Stacking a profile onto the retained
+anchor is always permitted under `no_new_privs` (it only removes permission), and the transition
+fails closed — the command cannot start — if `agcoord-job` is not loaded. `agcoord-job` is the
+step that confines submitted code: the anchor alone permits user-namespace creation and
+capabilities, but submitted code always runs as `agcoord-admitted//&agcoord-job`, and `agcoord-job`
+— like the inner profile of every other stack — denies both. When admitted work invokes the fixed
+broker, AppArmor substitutes `agcoord-broker-client` for `agcoord-job` on the anchor; this keeps
+the anchor without requesting a replacement domain after `no_new_privs`.
+
+One exec is widened on purpose: when admitted work runs the fixed system Bubblewrap executable
+`/usr/bin/bwrap`, AppArmor transitions it to `agcoord-admitted//&agcoord-bwrap`, which may create a
+user namespace and use capabilities inside that namespace to assemble and pivot into an
+unprivileged sandbox. This is a static rule in the packaged policy, not a per-job flag, operator
+allowlist, or executable digest, so an ordinary distribution upgrade of Bubblewrap keeps working.
+Everything `/usr/bin/bwrap` launches transitions to the stack of `agcoord-admitted`,
+`agcoord-job`, and `agcoord-bwrap-child`, which denies capabilities and user-namespace creation and
+stays confined across every further exec, so the sandbox builder cannot hand its namespace
+permission to arbitrary code — the pattern of Ubuntu's `bwrap-userns-restrict` profile. AppArmor
+renders a stacked label with its profiles in sorted order, so a sandboxed process reports
+`agcoord-admitted//&agcoord-bwrap-child//&agcoord-job (enforce)`. Only the fixed `/usr/bin/bwrap`
+path is widened: a program of that name reached through another mount namespace is named under a
+distinct disconnected prefix and never matches the rule, and submitted code that is not
+`/usr/bin/bwrap` still cannot create a user namespace at all. The `/usr/bin/bwrap` transition keeps
+the environment (matching the upstream profile), so the sandbox sees the job's variables such as
+`TMPDIR` unless Bubblewrap is told otherwise. All six profiles use explicit enforce mode and broad
+enumerated host permissions; `default_allow` is not accepted because Ubuntu 24.04 implements it as
+an unconfined profile that does not apply these denials. Activation fails closed unless every one of
+the six domains is loaded in enforce mode.
+
+Bubblewrap itself is an optional host package. AGCoord neither requires nor installs it; without
+`/usr/bin/bwrap` the `agcoord-bwrap` domains are inert and every other boundary is unchanged.
+Install it (`sudo apt-get install bubblewrap`) only on hosts whose admitted jobs build sandboxes.
+The enforced-host probe exercises the widening when `/usr/bin/bwrap` is present and prints
+`bubblewrap sandbox widening verified`; otherwise it prints that the widening was not exercised
+and still proves the user-namespace denial.
 
 Within the admitted user namespace, `stat(2)` reports the host-root-owned installed binary with
 the overflow UID because host root is intentionally absent from the one-entry identity map. The
@@ -410,6 +443,12 @@ recovery refuses a live owner, live or ambiguous recorded workers, unavailable i
 evidence, and an invalid package. It never substitutes a forced activation or deletion of the
 spool. Normal `agc host upgrade` and its drain flow remain unchanged.
 
+An existing normal drain is maintenance intent, not permission to reopen submissions. Recovery
+preserves its original drain ID, state, reason, and start time as durable audit
+metadata within the shared protocol-5 recovery contract. Its submission guards remain continuous
+while the recovery guard temporarily blocks ordinary accepted-job admission for host verification. Malformed original maintenance metadata
+refuses recovery; neither a retry nor repair may replace its identity or reset proof attempts.
+
 The command stages the verified replacement and stops the failed service. The staged native
 holder takes the exclusive owner/maintenance lock, proves every recorded worker identity is
 gone, and establishes a durable `recovery-` ID followed by 12 hexadecimal digits. It holds the
@@ -422,8 +461,15 @@ The durable guard blocks new submissions and ordinary queued-job admission throu
 and restart. Recovery admits only the exact bound enforcement helper, with `cpu=1` and
 `jobs=1`, while ordinary work stays blocked. Completion requires that proof to pass with exit
 status zero, a receipt with requested and applied CPU both 1 and observed peak CPU at least 1,
-the unchanged helper digest, and no running rows. Only successful completion clears the guard.
-A process exit, service restart, normal `agc drain`, or `agc resume` cannot clear it.
+the unchanged helper digest, and no running rows. Only successful completion ends the
+verification-only guard. If no normal drain existed, completion returns the spool to `open`.
+If recovery inherited a normal drain, completion atomically restores that original drain and
+its submission guards. Accepted queued jobs then finish through their normal state machines;
+proven-dead running rows remain interrupted. The owner reaches `drained` and yields when no
+accepted work remains. New submissions stay blocked until explicit `agc resume` with the exact
+original drain ID after the intended maintenance succeeds. Host proof never performs that
+resume. A process exit, service restart, normal `agc drain`, or `agc resume` cannot clear an
+active recovery guard.
 
 If the verified replacement is already running under the same active recovery guard, rerunning
 `agc host recover` continues that guard instead of stopping or reactivating the service. The
@@ -432,7 +478,8 @@ a mismatched guard still refuses. Continuation reuses a queued, running, or pass
 that recovery is verifying; it does not submit a duplicate or spend another proof attempt.
 
 The native hold, status, proof, completion, and park operations retain the exact recovery ID,
-replacement identity, helper path and digest, proof job ID, attempt count, phase, and incident.
+replacement identity, helper path and digest, proof job ID, attempt count, phase, incident,
+and any original drain metadata. The completed recovery audit retains that original drain.
 Each host phase and enforcement proof is bounded to 120 seconds; owner acquisition and dead-row
 settlement waits are each bounded to 30 seconds. At most three proof attempts are allowed
 across retries. Retrying must use the same exact verified replacement package and helper path;
@@ -447,9 +494,23 @@ recovery ID and address the reported incident before retrying the same package.
 
 Completion retries up to three times with the exact recovery ID. If completion committed but
 its reply was lost, the native operation validates the retained completed identity and passed
-CPU proof, then returns the same open receipt without another proof or recovery operation.
-If completion still cannot be confirmed, the error says to inspect the durable receipt; it
-does not claim the guard remains when completion may already have cleared it.
+CPU proof, then returns the same completion receipt without another proof or recovery
+operation. The completion receipt carries `state`, `recovery_id`, `proof_run_id`, and `drain_id`.
+Without an inherited drain, its state is `open` and `drain_id` is null. With an inherited drain,
+it retains that original ID and reports `draining` if accepted work remains, or `drained` if
+none remains. Completion retries return that recorded result and never implicitly resume it.
+The public host result reports service `verified` for a retained drain, since the broker may
+finish the accepted work and yield before the command returns; `active` applies to recovery
+that opens the spool.
+If completion still cannot be confirmed, inspect the durable recovery and original drain
+receipts. Do not assume submissions reopened or that the verification-only guard remains.
+
+A stranded drain has an original marker plus accepted queued or running rows, while the
+outgoing broker is dead or cannot start. Repeating the normal drain returns the same marker
+but cannot make those rows progress. Earlier recovery refused this state with `normal
+maintenance is active; recovery cannot replace its identity`. Use the verified recovery path
+above; retain both IDs and the original maintenance metadata. Raw database edits, premature
+resume, or cancelling accepted work are not repair workarounds.
 
 ### Normal restart and rollback
 

{agcoord-0.7.2 → agcoord-0.7.4}/docs/releasing.md

@@ -23,10 +23,19 @@ has one target: `x86_64-unknown-linux-musl`. Other architectures remain unsuppor
 have a native CI runner and the same kernel conformance coverage.
 
 Build inputs are pinned by `rust-toolchain.toml`, `Cargo.lock`, exact direct dependency
-requirements, and the source digest produced by `scripts/native-source-id`. The artifact embeds
-that digest as its build identity and reports protocol, implementation, target, and bundled
-SQLite version through `identity --json`. Install Rust 1.94.1 with the declared musl target and
-Ubuntu's `musl-tools`, then run:
+requirements, `native/build-toolchain.json`, and the source digest produced by
+`scripts/native-source-id`. The C compiler and musl headers compile the bundled SQLite, so they
+are build inputs too: `native/build-toolchain.json` declares the exact C compiler version line and
+musl version, and it is part of the source digest. `scripts/native-toolchain describe CC` reports
+the toolchain a compiler provides. `build-native-broker` refuses an undeclared C toolchain, and both
+`check-native-reproducible` and the release-candidate verifier refuse a broker whose provenance
+records one, so a different compiler or musl can never silently change a pinned digest. Changing the
+C toolchain is a deliberate source change: update the declaration, which gives a new build
+identity, and re-pin. The artifact embeds the source digest as its build identity and reports
+protocol, implementation, target, and bundled SQLite version through `identity --json`. The
+declared toolchain is Ubuntu 24.04's: install Rust 1.94.1 with the declared musl target and
+`musl=1.2.4-2 musl-dev=1.2.4-2 musl-tools=1.2.4-2` on Ubuntu 24.04, as the native CI and tag
+jobs do on their fixed `ubuntu-24.04` runner, then run:
 
 ```bash
 ./scripts/build-native-broker
@@ -40,7 +49,8 @@ Ubuntu's `musl-tools`, then run:
 ```
 
 The build emits the executable, a SHA-256 sidecar, and JSON provenance recording the artifact
-and source digests, Rust and Cargo versions, C compiler, target, protocol, and source epoch. The
+and source digests, Rust and Cargo versions, C compiler and musl versions, target, protocol, and
+source epoch. The
 audit requires a static or static-PIE ELF, no program interpreter, no `DT_NEEDED` entry, a valid
 checksum, and an exact release identity from an empty environment. Bundled SQLite is part of
 that ELF. The dependency graph must exactly match `native/THIRD_PARTY_LICENSES.tsv`, use only
@@ -48,10 +58,16 @@ the crates.io registry or workspace source, and contain only the reviewed licens
 
 Reproducibility CI builds two fresh checkout copies with separate Cargo target directories,
 path remapping, disabled incremental compilation, a fixed source epoch, and the same recorded
-toolchains; their executable bytes must match. This proves reproducibility for the declared
-runner and compiler inputs. It does not claim byte identity across different C compilers or
-host toolchain builds. `AGCOORD_MUSL_CC=cc` is an explicit local compatibility escape hatch for
-an audited x86_64 compiler when `musl-gcc` cannot be installed; release CI never uses it.
+toolchains; their executable bytes must match. The build remaps both the checkout and the Cargo
+home (where dependency sources live), so the artifact does not depend on where either sits on
+the builder, and the check refuses an artifact that still embeds the builder's Cargo home,
+checkout, or temporary build path. This proves reproducibility for the declared compiler inputs:
+the same Rust toolchain, `musl` package, and C compiler produce the same bytes on a maintainer
+host as on the release runner. It does not claim byte identity across different C compilers or
+host toolchain builds; the declaration makes that limit explicit rather than leaving it to the
+runner image. `AGCOORD_MUSL_CC=cc` is an explicit local compatibility escape hatch for an audited
+x86_64 compiler when `musl-gcc` cannot be installed: such a build records its actual toolchain and
+is never accepted by the reproducibility check or the release verifier; release CI never uses it.
 
 The host-package build emits a deterministic root-owned tar archive, package checker, staged
 installer, enforced-host probe, and their SHA-256 sidecars under `dist/host/`. CI retains these
@@ -160,6 +176,9 @@ interface and must be smoke-tested from the built wheel.
    [guarded managed recovery path](native_host.md#guarded-recovery-when-the-outgoing-broker-cannot-drain)
    with the exact pinned replacement. Retain its recovery ID and proof receipt; ordinary work
    stays blocked until verified completion, and normal drain/resume cannot clear that guard.
+   If recovery inherited a normal drain, completion restores its original ID and metadata;
+   accepted work finishes under that drain while new submissions remain blocked. Retain both
+   receipts and resume the exact original drain only after its intended maintenance succeeds.
    Keep `kernel.apparmor_restrict_unprivileged_userns=1`, start the ordinary unprivileged managed
    service, and retain the shipped `cpu=1` receipt proving its AppArmor transition, cgroup
    namespace root, exact CPU control, and durable applied/peak evidence. A spool left below

{agcoord-0.7.2 → agcoord-0.7.4}/docs/troubleshooting.md

@@ -70,7 +70,9 @@ These arrive as a failed row. `failure_reason` is stable; the log has the detail
 | --- | --- | --- |
 | `agc drain --reason …` | Installs a durable guard, waits for admitted work, then the owner yields. The receipt's `drain_id` is the only key that reopens the spool. | Save the ID. `--no-wait` returns after installing the guard. Repeating `drain` returns the original receipt. |
 | `agc resume <id>` refused with `broker-drain-id-mismatch` or `broker-drain-id-invalid` | Not the exact `drain-…` identifier the drain printed. | Use the receipt's ID; `agc list` prints it in its header while drained. |
-| `broker-drain-live-work` | Resume was asked while rows are still queued or running. | Wait for the rows to end, or cancel them. |
+| `broker-drain-live-work` | Resume was asked while rows are still queued or running. | Let accepted work finish. If the outgoing broker cannot start, use guarded host recovery preserving the original drain; cancelling accepted work is not a repair workaround. |
+| Normal drain marker with queued/running rows, outgoing broker dead or unable to start; earlier recovery says `normal maintenance is active; recovery cannot replace its identity` | The drain is stranded: repeating it preserves its marker but cannot progress accepted work. | Use [guarded host recovery](native_host.md#guarded-recovery-when-the-outgoing-broker-cannot-drain) with the matching client and pinned replacement. Preserve original drain metadata and both IDs; do not raw-edit the spool, prematurely resume, or cancel accepted work to bypass repair. |
+| Host recovery completes with retained normal maintenance | Host proof succeeded and restored the original drain as `draining` or `drained`; submissions stay blocked. Public service status `verified` permits the owner to finish accepted work and yield. | Keep the original drain and recovery receipts. Complete intended maintenance, wait for the drain to yield, then explicitly resume its exact original ID. Host completion does not authorize an automatic resume. |
 | `broker-resume-owner-live` | A broker still holds the ownership lock. | Let it exit; on a managed host, stop the service first. |
 | `broker-not-draining` | Nothing to resume. | No action. |
 | `agc avoid` refused with `avoid-sha-invalid` | The commit must be the full 40-character SHA. | Pass the full SHA; `--list` shows the stored set and `--remove SHA` deletes one. |
@@ -111,9 +113,9 @@ These arrive as a failed row. `failure_reason` is stable; the log has the detail
 | `native-host-user-live-broker` from recovery | A broker owns the spool without a matching active recovery identity/helper; recovery refuses to replace it. | Use the normal drain/upgrade flow for a working owner. A verified replacement under the exact matching guard may continue without stop or reactivation; retry the same package/helper. |
 | `host-recovery-worker-live`, `host-recovery-owner-invalid`, `host-recovery-invalid` | Worker absence, spool ownership, recovery metadata, or the bound replacement/helper cannot be proven. | Resolve the stated cause without deleting the spool or changing the recovery identity. Retry the same exact verified package and helper path. |
 | `host-recovery-settling` | Dead running rows have not yet settled; the enforcement proof cannot be admitted. | The command waits up to 30 seconds. If it parks, inspect the reported phase and repair before retrying. Ordinary queued work remains blocked. |
-| `host-recovery-proof-failed`, `host-recovery-proof-timeout` | The bound proof did not produce a passed enforced CPU receipt within its 120-second bound. | Repair the enforcement failure and retry the same package/helper. Only a successful applied/peak receipt permits completion. |
+| `host-recovery-proof-failed`, `host-recovery-proof-timeout` | The bound proof did not produce a passed enforced CPU receipt within its 120-second bound. | Repair the enforcement failure and retry the same package/helper. Only a successful applied/peak receipt permits completion; an inherited normal drain is restored, not resumed. |
 | `host-recovery-exhausted` | Three proof attempts have already been used across retries. | Operator repair is required. Do not delete or reset the guard, identity, or attempt budget to bypass exhaustion. |
-| `native-host-recovery-incomplete` | A bounded recovery phase failed. Before completion, any established guard remains; a completion error instead reports whether its durable result could be confirmed. Service state is reported as unchanged, stopped, or stop unconfirmed. | Preserve the recovery ID and incident evidence. Inspect the durable completion receipt if the reply was lost; exact-ID completion retries return the same validated open receipt after a committed completion. Otherwise repair and retry the same package/helper; drain/resume cannot clear an active recovery guard. |
+| `native-host-recovery-incomplete` | A bounded recovery phase failed. Before completion, any established guard remains; a completion error instead reports whether its durable result could be confirmed. Service state is reported as unchanged, stopped, or stop unconfirmed. | Preserve the recovery ID and incident evidence. Inspect the durable completion receipt if the reply was lost; exact-ID completion retries return the same validated result after a committed completion, including retained original drain metadata rather than always `open`. Otherwise repair and retry the same package/helper; drain/resume cannot clear an active recovery guard. |
 | `native-host-upgrade-drain-invalid` | The drain receipt the upgrade needs is missing or malformed. | Rerun `agc drain`, then the upgrade. |
 | `systemctl --user stop` or `restart` does not return | A graceful stop waits for running jobs, up to `stop_grace` (600 seconds by default). | Wait, or interrupt them now with `systemctl --user kill --kill-whom=main --signal=SIGTERM agcoord-broker.service`; they end `interrupted` (`broker-stopped`). Never use a plain `systemctl kill`, which signals the jobs too. |
 
@@ -127,6 +129,7 @@ check; `journalctl -k` distinguishes a kernel refusal from an AppArmor denial.
 | --- | --- | --- |
 | `host-apparmor-profile-mismatch` | The service did not enter the enforced `agcoord-broker` profile. | `aa-status`; reload the profile from `/etc/apparmor.d`; `systemctl --user restart agcoord-broker.service`. |
 | `host-apparmor-restriction-disabled` | `kernel.apparmor_restrict_unprivileged_userns` is not 1. | Restore the sysctl; the boundary depends on it. |
+| A submitted command exits with an exec/permission error and never runs | The `agcoord-admitted` exec rule stacks `agcoord-job` at `execve`; if `agcoord-job` (or another of the six domains) is not loaded, that transition fails closed so the command cannot start. | Confirm all six domains (`agcoord-admitted`, `agcoord-job`, `agcoord-bwrap`, `agcoord-bwrap-child`, `agcoord-broker`, `agcoord-broker-client`) are loaded in enforce mode (`aa-status`); reload `/etc/apparmor.d/usr.libexec.agcoord.agcoord-broker` and restart the service. A partial or stale policy load is the usual cause. |
 | `host-cgroup-controllers-unavailable`, `host-cgroup-mount-invalid`, `host-service-cgroup-mismatch` | CPU, memory, or PID controllers are not delegated, the mount is not read-write `nsdelegate` cgroup2, or the service's cgroup is not the configured root. | `cat /sys/fs/cgroup/user.slice/user-<uid>.slice/user@<uid>.service/cgroup.subtree_control`; compare `cgroup_root` in `config.json` with the unit's `Delegate=yes` and `DelegateSubgroup=supervisor`. |
 | `host-executable-digest-mismatch`, `host-executable-identity-mismatch`, `host-executable-invalid`, `host-executable-path-mismatch` | The installed file changed, is not root-owned, or the configuration does not select it with release trust and `managed_service: true`. | Rerun `agc host install --download` or `agc host upgrade --download`; check `native_broker` in `config.json`. |
 | `host-drain-required` and the other `host-drain-*` codes | Activation on an existing spool needs the exact drain receipt. | `agc drain`, keep the ID, and let the upgrade pass it. |