@parity/product-lab 0.0.1-placeholder → 0.1.0

package/adapters/environment/engine/README.md

@@ -0,0 +1,123 @@
+# @product-lab/env-engine
+
+Environment provider for [previewnet-engine](https://github.com/paritytech/previewnet-engine)
+(Zombienet) full forks, with Engine's own session record and the routing hosts
+use to reach an Engine network. Imported from the original lab.
+
+| Module | Purpose |
+| --- | --- |
+| `session/session.ts` | `EngineSession` (schema 1: People, Asset Hub, Bulletin, identity backend, dashboard, spawn provenance), `validateEngineSession`, `readEngineSession`, `engineRoutes(session)` (the domain's `HostNetworkRoutes`: every chain with its genesis and the identity backend, validated loopback, distinct genesis, network and namespace), `appConfig`, `localUrl`. Refactor step 7 part b removed `LabSession` (the union with the manual-seal schema 2, now env-relayless's `ManualSealSession`), `validateSession`, `checkSession` and `NETWORKS` (the domain's `HOST_NETWORKS`) |
+| `session/readiness.ts` | `attachEngine(rpc, …)`, `checkEngineSession(rpc, …)`, `observeChain(rpc, …)`, `rpc(rpc, …)`: ready means advancing finality, not an open port; since 3 October 2026 (fork cut 1) only on the chains `ReadinessOptions.finality` names (the lab's `requiredChains`): the others must answer and are read once, each chain's `readiness` (`finality` or `answered`) is in the session, and a resume's stall verdict counts only the finality chains. Since backlog 141 (5 October 2026) a start's checks (an owned first boot, a saved network's resume) pass `ReadinessOptions.firstBest`: a chain is ready only once its finalized head passes the best head the start first read, because after every node start a fork's parachains finalize the blocks they had and then pause while the validators prepare their PVFs (`tests/first-best-readiness.test.ts`, with the old rule as its negative control). `checkEngineSession` (every reconnect and resume) refuses a restarted Engine (spawn stamp), another genesis or endpoint, and since Phase 6 another runtime (`:code` hash or spec) than the session recorded. `checkAfterStateChange(check, warn, checks = STATE_CHANGE_CHECKS)` (backlog 93, 4 October 2026): an environment's own network check after an `@isolated` save or restore; restored validators restart their approval voting and the relay's finality can stand still for about one check's 90 s, so while every failure is a standing finalized head (`ChainsNotReadyError.standing()`) it checks again, up to 3 times, and a chain that does not answer or any other failure fails at once (`tests/check-after-state-change.test.ts`). Every JSON-RPC call goes through the injected `JsonRpcClient` and only to loopback endpoints (refused before anything is sent); the dashboard's HTTP reads still use `fetch` |
+| `engine/live-network.ts`, `engine/fork-snapshot.ts` | `publishedNetworkEndpoints(engine, network)` (backlog A2, 3 October 2026): a live network's public endpoint per chain from the published fork bundle's manifest (`chains.*.endpoint`), else from Engine's network file (`liveEndpoints`), with the runtimes the bundle was bitten at; each endpoint is marked `public: true` and nothing connects, so a caller still opts in to every connection (`publicEndpoint: { reason }`); neither source naming one throws, naming the network. `forkSnapshotFrom`/`readForkSnapshot`: the bundle's bite time, source, blocks and runtimes by chain name; Web3 Storage is para 1503 or, on previewnet, 1600. `tests/engine-networks.test.ts`, `tests/fork-snapshot.test.ts` |
+| `session/engine-networks.ts` | `ENGINE_NETWORKS`: what attaching expects of each network Engine knows, per network (backlog A1, 3 October 2026): the para id of People, Asset Hub and Bulletin and People's spec name. `paseo-next-v2` and `previewnet` hold the same values (1502, 1500, 1501; `next-people-paseo`); previewnet's Web3 Storage (1600) is not routed. `tests/engine-networks.test.ts` checks them against Engine's own network files (`tests/fixtures/engine-networks/`, from the tested Engine ref) and both published bundles' manifests |
+| `session/runtime-dependency.ts` | `RuntimeDependency<Session>` lifecycle contract (default `EngineSession`) and `AttachedRuntimeDependency(rpc, session)` for an Engine session |
+| `session/network-lock.ts` | `acquireScenarioLock(locks, …)`: one mutating owner per network generation, on the injected `LockManager` (People endpoint + genesis + Engine spawn stamp or node peer id); dead owners are reported with their evidence and reclaimed only with `PRODUCT_LAB_RECLAIM_STALE_LOCK=1` Since 3 October 2026 (backlog 17h) `networkLockPath(session, directory)` and `reclaimLockPredatedByRestore(locks, session, savedAt, directory)`: after a snapshot restore, the restored network's lock is reclaimed only when its holder is gone (the lock manager's `stale`), it took the lock after the snapshot's `savedAt`, and the lock records exactly that network's identity; otherwise it stays for the operator (`tests/network-lock.test.ts`, each condition with its negative control) |
+| `engine/owned-engine.ts` | `OwnedEngineDependency(config, reports, system, options)`: a fresh owned Engine per reset. `system` (`EngineSystem`) is the injected `ProcessRunner` (or an open `OwnedProcesses` registry), `CommandRunner`, `PortAllocator`, `JsonRpcClient`, `EnvironmentVariables` and `Logger`, and optionally a `LockManager` (`requireEngineSystem(deps)` names what a lab left out). Engine's tree is recorded in the registry and stopped with `ENGINE_STOP` (SIGINT, 20 s, then SIGKILL, 5 s); setup jobs are awaited through `commands(id)`. Refactor step 7 deleted its own spawn, process-group signalling, `ps` scans and port probe. **Preparatory commands** (bite, bundle fetch, `fork toml`, `genesis-toml`) run through the same registry, asynchronously, via `runEnginePrepare`. Each is a recorded tree (`<id>-prepare-<n>`, `PREPARE_STOP`), with its own log, a pointer line in the Engine log, and `stdoutTo` as the port's `stdout`. It is bounded by its `timeoutMs` and by the start's signal. However it ends, its tree is stopped, and the registry must report no recorded member alive before data is removed; a recorded survivor is `PrepareNotStopped`, and the data is kept. That is not proof the whole tree is gone: `NodeOwnedProcesses` records every process in the tree's group at the command's exit (its group anchor), but a descendant that left the group (`setsid`) before the registry saw it is never recorded, and is not covered. The setup-job wait takes the start's signal too, and the signal is checked again before the fork setup; `start()` while a stop is under way is refused. `start(signal)` is cancelled by that signal or by `stop()`, which joins it; the readiness wait is cancellable too. After a cancellation nothing of Engine starts. A preparatory tree that did not stop makes `stop()` fail, never report a clean stop, and a later `stop()` retries it |
+| `engine/engine-config.ts`, `engine/engine-prepare.ts`, `engine/engine-setup-jobs.ts` | The owned Engine's phases around `OwnedEngineDependency`, split out of `owned-engine.ts`, which still exports their public names: the configuration (`engineConfiguration`, `EngineSystem`, `requireEngineSystem`, `ENGINE_STOP`); the preparatory commands and stopping their trees (`runEnginePrepare`, `PREPARE_STOP`, `PrepareNotStopped`); the setup-job wait after readiness (`engineServices`: Engine's long-running processes, every other custom process is a one-shot job the start waits for; since 3 October 2026 it names previewnet's Web3 Storage provider `storage-provider-node`, para 1600, which the start had waited on until its bound). engine.json `pinProducts` (default `false`): the lab passes `PRODUCT_SYNC=0`, so Engine does not spawn its best-effort DotNS product import (`pin-bulletin-products`, a one-shot job); `true` passes `PRODUCT_SYNC=1`, and a topology's own `engine.env` still decides. Off because previewnet's network file turns the import on, and its fork had imported 50 of 637 products from previewnet.substrate.dev after about 10 minutes (3 October 2026); next-v2 never imports. `tests/setup-jobs.test.ts` |
+| `engine/fork-isolation.ts` | Fork isolation, on by default (engine.json `isolate`, `expose`): `--no-mdns` on every node including a collator's relay side, loopback-only listeners, RPC and Prometheus unless `expose`, fresh node keys (0600 key files) and renamed bootnodes for a persisted network, cleared authority-discovery caches, and an IPFS repository with its own identity and no public network. No per-fork chain spec `forkId`: collators then resolve no validator. At a first boot zombie-cli starts every node on `0.0.0.0` with `--prometheus-external`; unless `expose`, the isolated TOML launches relay nodes through a product-lab launcher named like the binary (`<data>.isolation/relay/<binary>`, `nodeWrapperScript`) and collators through `collator.sh`, which rewrite both sides to loopback (a bare relay-side `--port` becomes a loopback `--listen-addr`), and `inspectIsolation` refuses a first boot whose node listens beyond loopback without one (`nonLoopbackListeners`). zombie-cli's own port reservations (`*:<port>` until it spawns each node, about 30 s) are upstream behaviour and not changed Old workspaces migrate with `persistentNetwork(..., { isolate: true })` |
+| `engine/node-isolation.ts`, `engine/fork-toml.ts`, `engine/persisted-isolation.ts`, `engine/ipfs-isolation.ts` | Fork isolation by phase, split out of `fork-isolation.ts`, which keeps the check of what ran (`inspectIsolation`, `nonLoopbackListeners`) and still exports every public name: what the phases share (the node-argument rewrite, `zombieNodes`, `chainSpecFiles`); a first boot (the isolated fork TOML and its launchers); a resume (fresh node keys, renamed bootnodes, cleared caches); IPFS. Since backlog 190 (8 October 2026) two engine.json settings shape a parachain collator's own arguments, at a first boot (the fork TOML's collator lists, by section) and on **every start** of a saved network (`persistentNetwork`'s start loop, so a plan saved before them gets them too): `txpoolLog` (`debug` by default, `trace`) adds `-ltxpool=<level>` (`withNodeLogTargets`, `nodeLogTargets`) unless its `-l` lists set `txpool`, so a transaction a client hears of only as "Extrinsic marked as invalid" (polkadot-sdk#8497) shows in the collator's log (debug: counts per block and path; trace: each transaction by hash with its validity, about 46 MB an hour on a busy collator); `bulletinAuthoring` (`slot-based` by default, or `lookahead`, Engine's choice) gives Bulletin's collator (the one with `--ipfs-server`) `--authoring=slot-based` (`withBulletinAuthoring`), as Engine runs Asset Hub and People: Bulletin's lookahead collator forks its own chain (300 reorganizations in 6 h); on a light fork's A/B (`tools/verify/live/engine/bulletin-authoring.ts`, 10 minutes each) slot-based took forked collations from 19 to 0 and retracted best blocks from 20 to 3. It reduces reorgs but does not end them, nor the failures they cause: in an hour with stores 15 depth-1 reorgs remained, and each store signed at a retracted block (14 of 535) was marked invalid. `SavedForkSettings` carries both into a saved fork's engine.json. Relay validators and a collator's relay side get neither (`tests/fork-isolation.test.ts`, `tests/network-supervision.test.ts`) |
+| `engine/persistent-network.ts` | With `options.versions` (Phase 6) a new network is checked against the resolution and records it in `versions.json` (0600); an existing one resumes only with the combination it recorded (`VersionedWorkspace.resume`): a downgrade, a version change, or versions known on one side only are refused before any port check or process, and nothing is reset. Resume a persisted Engine network through the injected system (`{ processes, files, ports, rpc, environment, logger }`: the workspace's `OwnedProcesses`, a `FileSystem`, a `PortAllocator`, a `JsonRpcClient`, `EnvironmentVariables`, a `Logger`); the bootstrap Engine is recorded in the workspace registry as `engine-bootstrap`; a first boot is cancelled by the start's signal (Engine's `start(signal)`: a preparatory command such as the bundle download is stopped and no network is recorded; `tools/verify/tests/engine-owned-start.test.ts`); `network.json` version 2 is isolated, version 1 resumes with a warning until migrated. Every `start()` completes the saved plan (`completePlan`: eth-rpc on `ETH_RPC_PORT` from `<data>/engine-ports.json`, which a first boot writes, or engine.json `ports`; with engine.json `nodePorts` every port inside it): a plan saved by an older version is rewritten on its next resume, only with its old and new ports free (a taken one is refused, nothing is killed), the previous plan kept as `network.previous.json`. eth-rpc is optional: if it exits or does not answer `eth_chainId` and `eth_blockNumber` within 60 s the network still starts and `network.ethRpc` says why (`EthRpcStatus`; `probeEthRpc`); a first boot with `nodePorts` gives zombie-cli every node port (`assignForkTomlPorts`: `prometheus_port`, `p2p_port`, a collator's `full_node_p2p_port` and `full_node_prometheus_port`) and eth-rpc its port. With engine.json `template` the first boot saves the template (`engine/network-template.ts`). `start()` first checks the saved data (`requirePersistedData`, `missingPersistedData`, `hasChainDatabase`): every node's own `--base-path` must hold a chain database (RocksDB `CURRENT` naming a live manifest, or ParityDB metadata; a collator's relay side keeps none), the identity backend's Postgres cluster its `PG_VERSION`, and IPFS its repository. Anything missing is refused before any port check or process, named: a node started without its database would begin again from the fork's genesis (or resync from peers) instead of resuming. Once its ports are free it restates what another network's boot from the same Engine checkout rewrote: its data directories in `ports.local.env` (`restateEngineLocalEnv`) and its ports in the checkout's `config/ports.env` (`restateEnginePorts`, `engineCheckout`). A taken port is named (`portNames`); the readiness wait shows its last answer and fails early on an exited service or a dashboard refusing its port (`readyTimeoutMs`) |
+| `engine/plan-ports.ts` | Every port of a persisted network inside the workspace's own ports, and eth-rpc in the saved plan. `containPlanPorts(plan, range, named, taken?)`: each port a node listens on (`--prometheus-port`, `--rpc-port`, `--port`, `--listen-addr`, both sides of a collator's `--`) that is neither config/ports.env's (`named`) nor the plan's own is moved into engine.json `nodePorts`, in process order, deterministic and idempotent, around the `taken` ports (backlog 237: the persistent network passes the range's ports busy right now, probed only when a port moves, so an outgoing connection holding one of the OS's ephemeral range is gone around; the first boot's `planForkTomlPorts` does the same with `busyPorts`); `--bootnodes`/`--reserved-nodes` and chain spec `bootNodes` (`remapBootNodePorts`, `remapSpecBootNodePorts`) follow. `withEthRpc`/`ethRpcProcess`: Engine's eth-rpc (`scripts/eth-rpc.sh`'s `--node-rpc-url` Asset Hub and `--base-path <data>/eth-rpc-db`) plus `--rpc-port <ETH_RPC_PORT>` (Engine's launcher passes none, so it listened on 8545 while its dashboard probed ETH_RPC_PORT) and `--no-prometheus`, loopback, before the dashboard; or the reason it cannot run (no ETH_RPC_PORT known, no binary). `planPorts`, `listenedPorts` |
+| `engine/fork-topology.ts` | How an owned fork is laid out (backlog 144, 5 October 2026): engine.json `forkTopology`, `faithful` (the default, Engine's own: 6 relay validators, Asset Hub on 3 cores) or `light` (4 validators, Asset Hub on 1 core, one PVF worker of each kind per validator). Engine publishes bundles of its own layout only, so a light fork lives in a workspace of its own (`forkTopologyWorkspace`: `<workspace>.light`, Engine's binaries cloned in, its IPFS repository not), where `applyForkLayoutDescriptor` merges the descriptor overlay `patches/previewnet-engine/light-topology.json` (`relay.validators` 4; shipped as `LIGHT_TOPOLOGY_OVERLAY` in `setup/patch-catalogue.ts`, since the package carries no `patches/`) into `networks/<network>.json` before each start, `engineRun` adds `--cores asset-hub=1` to `ppn start` and bites the bundle with it when there is none (never downloads it), and the isolated TOML appends the PVF limits to the relay chain's `default_args` (`relayArgs`). Needs isolation (refused without it). Seeds are kept per layout (`networkSeedDirectory`, `networkSeedProblems`); `savedForkUpdates({ published: false })` refuses `--refresh` and fetches no newer published state. The checkout and Engine's own workspace are never written. Backlog 184: `light` also lays out previewnet (the same overlay), and `LIGHT_BUNDLE_SOURCE` (off: `bite`) is the one setting that makes a light fork's first start download Engine's own light bundle (`ppn fork fetch-bundle` with `PPN_FORK_BUNDLE_ASSET`) once Engine publishes one; `publishesLayout` says which. `tests/fork-topology.test.ts` |
+| `engine/turn-relay.ts` | Engine v0.7.0's TURN/STUN relay (eturnal, beside the identity backend on any network with People) as the lab runs it (2 October 2026). `TURN_PORT_OFFSETS`/`turnPorts(base)`: each lab block names its own `TURN_*` ports (349, 478, 479, UDP relay range 200-239), so two networks never share Engine's defaults (3478, 49152-49407). eturnal publishes no macOS build (Homebrew's processone tap builds it; Linux: `ppn fetch` downloads it): `chooseTurnRelay`, called by `OwnedEngineDependency` before a first boot, keeps a workspace copy of each descriptor with `services.turn: false` while eturnal is missing (`applyTurnRelayChoice`; Engine reads the workspace's `networks/` first) and warns once (`turnRelayOffNotice`, naming `ETURNAL_BREW_INSTALL`); peers on the machine still connect over host candidates. With eturnal, a second relay on the machine is refused (`otherTurnRelays`, `refuseSecondTurnRelay`): eturnal's fixed Erlang node name `eturnal@127.0.0.1` allows one (its `ERL_DIST_PORT`, which skips EPMD, is unverified). A resumed network keeps the relay state of its first boot. Exported from `.`. Tests: `tests/turn-relay.test.ts` |
+| `engine/network-template.ts` | engine.json `template: true`: `saveNetworkTemplate` keeps a clean copy of a new network right after its first boot (Engine stopped, before the first resume) in `<workspace>/template/` (its data folder, network.json, versions.json, the session, `template.json`); `restoreNetworkTemplate` (a reset) clones it back as a new generation: a new spawn stamp in `<data>/spawn.json` and the session (the session check and the network lock compare it) and a new run id; its chain state is the first boot's, so nothing created since exists on it. `cloneTree`: `cp -c -R` (APFS copy-on-write) on macOS, `cp --reflink=auto -a` elsewhere, else a plain copy with a warning. The template serves its own workspace's resets only: every saved path names `<workspace>/network` |
+| `engine/network-snapshots.ts` | Fixtures: `networkSnapshots(control)`, the `EnvironmentSnapshots` of a persistent network: `save` records each chain's finalized head, stops the network, copies `<workspace>/network` through the `TreeCloner` port (an APFS clone on macOS) with network.json, versions.json and the session, and resumes; `restore` refuses another kind, another lineage (`networkLineage`: the template's first boot, else the session's run; a network forked again is another lineage) or another workspace (saved paths name it), moves the live network aside, copies the snapshot in and resumes it, putting the moved one back when that fails. Since 30 September 2026 (backlog 1a) `control.files` names more plain files of the workspace to save beside the network (a consumer's own manifests, for example a workspace, an identity pool and its actors) under `files/`; restore puts each back, and one absent when saved stays absent. `networkSnapshotsUnsupported(owned, trees)` says why an attached network offers none. Since 30 September 2026 `networkSnapshots({ upgrade })` offers `EnvironmentSnapshots.upgrade` (a lab that owns the fork passes `savedForkUpdates(...).upgradeToLive`: `fixture refresh` upgrades a restored fixture in place); without it the snapshots say `upgradeUnsupported`. An Engine topology run (`engineEnvironment`) is disposable and reports `snapshotsUnsupported` Since 3 October 2026 (backlog 17h) `save` records `detail.savedAt` once the network has stopped, and with `control.locks` (`{ manager, directory?, log? }`) a restore reclaims the restored network's stale lock that a killed run took after that moment (`reclaimLockPredatedByRestore`), logging `Reclaimed the network lock … the restored state predates it (saved …)` |
+| `engine/network-adoption.ts` | Fixture sharing (30 September 2026): `planNetworkAdoption` reads an imported network snapshot and this workspace's running plan and says what adopting it moves: the saved workspace directory and prepared Engine tooling directory to this checkout's (another tooling hash, another Engine, is refused) and, when the port block differs by one offset, every port of the old block wherever a port is named (plan arguments and environment, session, `engine-ports.json`, isolation and zombienet records, chain spec boot nodes, IPFS addresses; `rewriteText`); it refuses, naming each, another port layout, content-addressed IPFS blocks naming an old loopback port, and any other small file that still names the old workspace or ports. `applyNetworkAdoption` rewrites the snapshot copy; `networkSnapshots` `restore(…, { adopt: true })` plans before anything stops, applies it to its own snapshot copy, keeps the brought lineage in `network/adopted-lineage.json` (`readAdoptedLineage`, read first by `networkLineage`; a template reset replaces the folder and forgets it) and returns what moved. `save` records Engine's bite (`fork-snapshot.json`) and ports as `detail.provenance`. Since backlog 242 the workspace's own files a snapshot carries under `files/` (the environment names them: `networkSnapshots({ files })`) are rewritten too, so an environment's records with their ports and directories move with it. `tests/network-adoption.test.ts`; a dry run on a real saved fork: `tools/verify/live/engine/fixture-adoption.ts` |
+| `engine/finality-stall.ts` | A saved network whose finality never advances after a resume (backlog 76). Seen live: relay validators that restart together can each lose the approval check of a candidate in their first second (`Session params unavailable (no active leaf?)`, upstream behaviour), and when too few are left to approve it (four of six lost) the relay block is never approved and nothing finalizes until the validators restart. `persistentNetwork.start()` recovers within a bound: it reads the validators' logs every second (`lostApprovalChecks`, `unapprovable`: the verdict arrives about 12 s into the start) and also calls a stall from the symptom alone (every chain answers, no finalized head advanced for `stallAfterMs`, default 2 minutes); it keeps what every process logged in `<workspace>/stall-evidence/<time>-start-<n>/` with `stall.json` (`keepStallEvidence`; `history.jsonl` lists every stall and outcome), stops the saved processes and starts them again, at most `stallRecovery.restarts` times (default 2). A recovered stall is reported: the `network-ready` phase says `finality advancing after 1 restart (finality had stalled: …)`, the logger warns, and `network.recovery.stalls` lists each stall (`NetworkStartRecovery`). One that is not recovered throws `FinalityStalledError` (`attempts`, `stalls`, each with its evidence directory), with the processes stopped and the data kept. `tests/finality-stall.test.ts` (the control: a network that never finalizes fails after 3 starts); the live loop is `tools/verify/live/engine/resume-loop.ts` Backlog 143 (5 October 2026): chains that stand while the relay's finalized head advances (read from a relay validator's RPC, `relayEndpoint` in `engine/drain.ts`) are waiting for the validators to prepare the parachains' PVFs, so they get no standing verdict and no restart, said in the start's progress; the start's deadline still bounds them, and a relay that stands too, or cannot be read, gives the verdict as before (`tests/finality-stall.test.ts`, with a frozen relay as its control). |
+| `engine/chain-clock.ts` | A resumed network's chains reach the wall clock before it is ready (backlog 106). A saved chain keeps the time of its last block until it builds a new one, and readiness waits for finality only on the lab's required chains, so a chain that only has to answer (Notes: Asset Hub) could still be at the save's time. That time is signed with: the truAPI CLI host timestamps its dotNS registration with Asset Hub's `Timestamp.Now` at the best block, and the identity backend refuses one more than 600 s old (seen 5 October 2026: gate batch 21's resume of a fork saved hours earlier failed onboarding with `dotns.signedAt is older than the intake freshness bound (600s)`). `persistentNetwork.start()` reads every chain's `Timestamp.Now` and best block after readiness (`waitForChainClocks`, `chainTime`) and waits until each is within 2 minutes (`CHAIN_CLOCK_MAX_LAG_MS`) of the wall clock, the `network-ready` phase naming what it waits for; one still behind after `CHAIN_CLOCK_TIMEOUT_MS` (2 minutes) is not building blocks, and the start fails with `ChainClockBehindError` (`behind`: each chain, its lag and best block), naming why and the way out (restart the network, or a fresh fork). The logger gives each chain's time, its lag at the first read and how long the wait took (`onRead`). `chainClock: false` skips it. `tests/chain-clock.test.ts` (the fake dashboard's `chainLagMs` sets each chain's time) |
+| `engine/approval-pin.ts` | Relay finality pinned at the approval threshold after a session change (backlog 93, 239). A start whose first block changes the session builds two gossip topologies before any peer connects; polkadot-sdk's `SessionGridTopologies::update_authority_ids` gives the peers to the first only, so some approval votes reach nobody until approval-distribution's level-1 aggression (`l1_threshold` 16), and every block then finalizes about 16 blocks late for the rest of the start. `ApprovalPinWatch` and the constants `APPROVAL_PIN_GAP` (14), `APPROVAL_PIN_HOLD_MS` (30 s), `APPROVAL_PIN_RESTARTS` (1): once the relay has finalized past the best head of the start's first read (so a backlog the start began with, as after the pin's own restart without a drain, is finalized first), a best head 14 or more ahead of the finalized one for 30 s restarts the network once (`stallRecovery.pinned`), recorded in `recovery.stalls` as `pinned` ("finality pinned at the approval threshold after a session change; restarted once", with the heads and the gap), its logs kept like a stall's; a second pin fails the start (`FinalityPinnedError`). Watched while the start waits for readiness and, if the relay is at the threshold when it is ready, until it falls below or holds |
+| `engine/node-temp-dir.ts` | A relay validator's TMPDIR fits its PVF sockets (backlog 93). A validator binds its PVF workers' Unix sockets directly in its TMPDIR (`pvf-host-prepare-<10>`); a path of `sun_path`'s size or more is refused (`path must be shorter than SUN_LEN`: 104 bytes on macOS and the BSDs, 108 on Linux, `SUN_PATH_BYTES`), so no candidate is validated, no approval completes and the relay's finality stops at the last block before the first unapproved candidate. Seen 4 October 2026: a fork started with TMPDIR under a lane's `_artifacts` folder (107-byte socket paths) stalled at relay #1,310,664 on a fresh boot and on every resume. `nodeTempDir(network, inherited)`: when the inherited TMPDIR is too long (`pvfSocketsFit`), the network's processes get `TMPDIR=/tmp/pl-pvf-<12 hex of the network directory>` (0700), on the first boot (`OwnedEngineDependency`: Engine, zombie-cli and so every node) and on each resume (`persistentNetwork`: every saved process), with a warning naming the byte counts; it is removed on stop, and the next start of the same network removes one a crashed run left. A TMPDIR that fits changes nothing. `tests/node-temp-dir.test.ts` (both sides of the limit on macOS and Linux), `tests/finality-stall.test.ts` (a resume), `tools/verify/tests/engine-owned-start.test.ts` (a first boot); each fails with the override removed |
+| `engine/drain.ts` | The drain before a saved network stops (backlog 17d): the stall above comes from relay blocks left unfinalized at the stop, whose candidates every validator checks again at the next start. `persistentNetwork.stop()` (and so a fixture save) first stops the collators (`drainPlan`: the processes whose arguments carry `--collator`; the relay at a validator's `--rpc-port`), then waits until no parachain head (`Paras.Heads`, read by the `PARAS_HEADS_PREFIX` keys) has changed for `DRAIN_QUIET_BLOCKS` (3) relay blocks and the block of the last change is finalized (with asynchronous backing a candidate backed before the stop is still included a few blocks later; a fixed two-block margin was too short), bounded by `DRAIN_TIMEOUT_MS` (120 s), then stops the rest (`drainCollators`, `DrainResult`, `network.drain`; a snapshot record keeps it under `detail.drain`). A relay that does not finalize in time or does not answer is stopped anyway with a warning, and the stall recovery still covers the next resume, as it covers a network stopped by a crash or a kill. `drain: false` stops everything at once (the loop's control, `DRAIN=0`). A restore's stop does not drain (`network.stop({ drain: false })`, lane speed, 7 October 2026): the snapshot's data replaces the stopped data, so no resume reads what a drain would settle (27-38 s on a full local fork); only a failed restore resumes that data again, and the stall recovery covers it then (`tests/network-snapshots.test.ts`, `tests/network-supervision.test.ts`). `tests/drain.test.ts`; live: `tools/verify/live/engine/resume-loop.ts` counts each resume's lost approval checks |
+| `engine/fork-setup.ts` | What a fork needs after Engine's setup jobs, checked on the chain instead of trusted. `grantDotnsAttestationAllowance` (DotNS), and since 30 September 2026 `ensurePeopleLiteAttestationAllowance`: Engine's job `increase-people-lite-attestation-allowance` submits once and can lose a nonce race against `grant-invites` (`Invalid: Stale`), leaving the allowance at 0, which showed later as a failed onboarding. The start reads `PeopleLite.AttestationAllowance` of the verifier account with the `dot` CLI; when it is short it grants it through sudo (at most 3 submissions, read back each time) and says so (`setupReport.forkSetup.peopleLiteAttestationAllowance`: `grantedBy` `engine` or `lab`, a warning, the setup phase's detail). A grant that does not land throws `ForkSetupError`, which ends the start at once instead of being retried until Engine's deadline. `tests/fork-setup.test.ts` |
+| `engine/network-seed.ts` | A first boot shared on one machine (backlog 77): `publishNetworkSeed(workspace, target)` copies a workspace's template and its launchers (`network.isolation/`) to `<shared>/fork-seeds/<network>-<tooling>-<platform>-<arch>/` (`networkSeedDirectory`) with `seed.json`. With engine.json `seed` naming that directory, a new workspace adopts it instead of forking (`adoptNetworkSeed`: clone, move every saved path to this workspace through `planNetworkAdoption`, check that each launcher's binary exists), then restores it as a new generation, as a reset does. Refused, with each reason in the `network-seed` phase, and the ordinary first boot runs: another network or mode, another platform, another prepared Engine (tooling hash), another port layout (`networkSeedProblems`, `NetworkSeedRefused`). Since 3 October 2026 (fork cut 4) a seed fits another port block of the same layout (one offset for every Engine port): the adoption plan moves the saved plan, records and chain specs, every zombienet TOML beside the launchers gets its ports moved by `moveForkTomlPorts` (port keys, loopback addresses, multiaddresses, port arguments; a port of the old block it cannot move refuses the seed), and the collator and relay node launchers are written again by `writeIsolatedForkToml` for this workspace's `nodePorts` window and Engine ports, never text-rewritten. Not used with `options.versions`. A stopped first boot (a data folder without `network.json`, which a boot, a restore and a seed write last) is removed with a warning before the seed or a new boot (backlog 210; before, the boot refused it until a retry), unless that boot may still run: the registry's `engine-bootstrap` alive or Engine's dashboard port taken refuses the start by name and removes nothing. A seeded network shares its first boot's chain state and node keys with every other copy: for local tests only. `tests/network-seed.test.ts` |
+| `engine/network-reset.ts` | `resetPersistentNetwork(directory, { stop, sessionFile, commands })` (29 September 2026): stops the saved network, removes its data and network.json, and restores the first boot's template as a new generation (`method: "template"`, with `stopMs` and the copy); a workspace without a template gets `method: "fork-again"` with the reason, and the caller's next `persistentNetwork` forks again (the old path). Notes' `paseo-copy` reset uses it. The persistent network's `stop()` (`persistent-network.ts`) now stops every saved process at once (`stopTogether`; on Notes' fork 3.8 s and 4.2 s against 4.5 s one after another), each with its own stop policy, every failure reported together (`tests/stop-together.test.ts`) |
+| `engine/runtime-upgrade.ts` | An in-place runtime upgrade of one chain of a running network (relay, People, Asset Hub, Bulletin): `upgradeRuntime(request)` runs Engine's own `ppn upgrade <chain> <wasm> --ws <loopback> --enact-timeout <min>` (with `--allow-same-spec` for the same spec or `withoutChecks`; the caller decides whether Engine's authorization path applies to its network) as a recorded tree of the network's registry through `runEnginePrepare`, writes its code through the injected `FileSystem.writePrivateBytes`, and judges the result on the chain: `observeRuntime(rpc, ws)` before and after (finalized `:code` hash, spec, block, and `System.LastRuntimeUpgrade`, decoded by `decodeLastRuntimeUpgrade`, which says the new runtime's migrations ran). While Engine waits for enactment, the chain is read every 5 s: an authorization dropped with nothing scheduled and the code unchanged (`System.RejectedInvalidAuthorizedUpgrade`: another chain's runtime, a lower spec), which Engine's tool does not notice until its enact timeout, stops the wait at once as a failure. Outcomes: `already-running` (nothing submitted), `upgraded` (the chain runs exactly the code, `migrated` true or false), `failed` (with `unchanged`: the chain still runs its old code). `recordRuntimeUpgrade(files, …)` rewrites a persistent network's saved session (`network.json` and the session file) on purpose, since `checkEngineSession` refuses any runtime change it was not told about, and appends `runtime-upgrades.json` |
+| `engine/fork-bundle.ts` | A fork bundle is reused only once verified complete. `verifyForkBundle(dir)`: `manifest.json` parses, every snapshot it names is there at the size it records (`snapshotBytes`, written by every bite since Engine v0.8.0; else one snapshot per chain) and decompresses to its end (3.5 s for the 642 MB Paseo bundle). `markForkBundle` writes `.product-lab-complete` (the manifest's sha256, each snapshot's size); `markedComplete` holds while both match. `forkBundleState(bundle)`: complete, absent, or incomplete; an unmarked bundle (fetched before this check, or copied in) is verified once and marked. A start's fetch or bite writes into `<bundle>.incoming` (cleared first; `EngineCommand.installs`), and `installIncomingBundle` verifies it, marks it and renames it into place after the command exits 0, so a fetch or bite stopped part way never leaves a bundle that is reused, and a failed one leaves the bundle in place unchanged. `acquireForkBundleLock` (`<bundle>.lock`, through the injected `LockManager`) is held from the bundle's check to the end of the preparatory commands: a second start from one workspace waits, then finds it complete. A start without isolation, where Engine's own `ppn start` fetches a missing bundle, is not covered |
+| `engine/published-bundle.ts` | Engine's published fork bundles (the rolling `PPN_REPO@PPN_BITE_TAG` pre-release of the checkout's `config/versions.env`, `engineReleaseSettings`): `publishedBundle({ files, checkout, network, fetch })` (bounded by `timeoutMs`, default 180 000 ms, with the caller's optional `signal`) reads the asset record from GitHub's API and the bundle's `manifest.json` by streaming at most the first 48 MiB of the compressed tarball (an HTTP range, stopping once `manifest.json` is read), so what is published is known without downloading it; the caller passes `fetch` (the public-endpoint opt-in) and the `FileSystem` system port for the checkout settings and staged bundles. `startPrefetch` downloads it with Engine's own `ppn fork fetch-bundle` into `fork-bundle-<network>.next.partial` at background priority (`backgroundPriority`: macOS `taskpolicy -c utility`, else `nice -n 19`) as a recorded process of the caller's registry; `finishPrefetch` verifies it complete and marks it (`engine/fork-bundle.ts`; an incomplete download is removed), then `completePrefetch` stages it (`.next`, `stagedBundle`, which offers only a verified one); `installStagedBundle` makes it the bundle new forks start from, keeping the old one until `discardPrevious` or `restorePrevious`. `currentBundle`, `bittenAfter`, `prefetchActive`, `bundleLocations` |
+| `engine/fork-freshness.ts`, `engine/fork-refresh.ts`, `engine/fork-updates.ts` | Freshness, `--update-runtime` and `--refresh` for any saved Engine fork (fixture sharing, 30 September 2026, backlog 17c). They were lifted from an example, which now wraps them. State: `forkState` reads what a saved fork runs: its saved session, `runtimes.json` and its bite. `liveEndpoints` reads the public RPCs from Engine's network file; a relative one (previewnet's `relay/alice`, `people`) resolves under the file's `bite.source` as `wss://` (`resolveNetworkRpc`, as Engine's bite does). Freshness line: `freshness` is pure: it gives the age, the blocks behind live, and each chain whose spec or code changed, ending with the update command. The live read is injected as `LiveRuntimeReader`; the composition root's is `readLiveRuntimes`. `--update-runtime`: `updateForkRuntimes` applies each changed live runtime in place through `upgradeRuntime`. The code comes from an injected `LiveCodeFetcher`, checked against its hash. It then records the upgrade and waits `settleBlocks`. It refuses without a live read or a sudo key. Other pieces: `recordForkRuntimes`. `prefetchNewerState` fetches a newer published state in the background and never rejects. `prepareRefresh` sets the old fork aside for the newest published state; a lab record is carried across with `carry`. `commit` or `rollback` ends it. For a lab's start: `savedForkUpdates(options)` combines these into three calls. Before the start it runs `--refresh` when asked, and runs the live read and line beside the start. `started(target)` runs `--update-runtime` before anything connects, records the runtimes, commits a refresh and prefetches. `failed()` rolls a refresh back. `liveRpc` (3 October 2026): the lab's own RPC per chain for the live reads, over the network file's. `upgradeToLive(target, signal)` serves `networkSnapshots({ upgrade })` for `fixture refresh`. `tests/fork-freshness.test.ts` |
+| `engine/owned-environment.ts` | `ownedEngineEnvironment`: owned Engine + consumer app sessions as one resettable environment; its local `engineSessionEnvironment` forwards optional app surfaces across reset and closes the current sessions and Engine once |
+| `engine/network-plan.ts`, `engine/persisted-data.ts`, `engine/eth-rpc-probe.ts` | The persistent network's phases around `persistentNetwork`, split out of `persistent-network.ts`, which still exports their public names: the saved plan (`NetworkPlan`, `resumePlan`, `isolatedPlan`, `completePlan`, `planFacts`, `ENGINE_PORTS_FILE`); the resume's preconditions (`hasChainDatabase`, `missingPersistedData`, `requirePersistedData`, `restateEngineLocalEnv`, `engineCheckout`, `restateEnginePorts`, `portNames`, `requireFreePorts`; and, in `engine/saved-tooling.ts`, `missingSavedTooling` and `requireSavedTooling`: every command, what the lab's launchers in `<network>.isolation` exec, and the Engine checkout are on this machine); eth-rpc's health after a start (`probeEthRpc`, `EthRpcStatus`) |
+| `engine/checkout-lock.ts` | Exclusive per checkout, enforced: fresh starts and saved resumes require the injected `LockManager` and hold the canonical checkout’s `config/ports.env.lock` from before start through completed stop, including Engine’s `finishGroup`. Another live network is refused immediately, naming the shared-checkout sweep gap and the fix: a separate cache or checkout. The lock records the registry of the holder's Engine and service trees. A holder that is gone (PID plus start time) is recovered and reported only once every member of every registry it recorded is gone too: Engine's trees run detached and can outlive their lab, so a live member refuses the recovery. A member of the workspace's own registry (`<workspace>/processes`) names the workspace, whose next start stops it; a member of a per-start registry (an owned Engine's reports, which no later start reads) names that registry and the live members' commands to stop. A record that names no registry, or an empty list, is refused (an older lab, or a holder that recorded none: a lifetime without a registry records no `registries` field). A saved fork's survivors step (`stopSavedForkSurvivors`) reads its engine.json through `namedCheckout`, so a damaged one names no checkout instead of failing the step, as for `persistentNetwork`. Live and unknown owners are never broken. `stopWorkspaceSurvivors` (and `persistentNetwork`'s `survivors: "stop"`, a workspace's first open in a new lab process) stops what an interrupted owner left in the workspace's own registry only while it holds the lock of every checkout the workspace names; `stop()` stops the saved network's trees under the lock and leaves the registry's other trees (hosts, downloads) to their owners. The historical `acquireCheckoutPorts`, `checkoutPortsLock` and `CheckoutLockTimeout` names remain; `waitMs` no longer enables waiting. |
+| `engine/engine-start-phases.ts` | The start's phases through the progress port (`OwnedEngineDependency` option `phase`, `persistentNetwork` option `phase`; the Engine topology passes `EnvironmentStartContext.phase`): `EngineStartPhases`, `ENGINE_PHASES`, `zombienetNodes`, `readServiceHealth`. Network state (cached when Engine's bundle test passes, active while the preparatory `fork fetch-bundle` or `bite` runs, done on its exit 0), node binaries (Engine's fetch lines: `artifacts present` is cached, `✓` lines are counted, `Done!` is done), booting the network (zombienet's `should be running now` lines against the isolated TOML's relay validators and collators; done only on `network is up`), Engine services (the dashboard's `/api/health` for the `/api/network` services, with those down named; done when the dashboard answers with this network's spawn stamp), setup jobs (n of m finished, then the fork setup). `persistentNetwork` adds saving a first boot, the saved processes started (n of m) and the chains answering (`checkEngineSession`). Engine's downloads read each response whole before writing it (its `lib/github.ts`), so they are reported without a byte count or percentage |
+| `engine/engine-health.ts` | Lab-side node supervision for a network's whole lifetime (backlog item 70). `engineNodeInventory(zombieJson, plan?)` / `readEngineNodeInventory(file, plan?)`: every node zombie.json lists (the relay's nodes and every parachain's collators, whatever their ids; the fixed 1500/1501/1502 check is gone), on a resume each as its saved process `network-<name>` on the metrics port its saved arguments give; a missing or malformed inventory, a duplicate name, a non-loopback endpoint, a node the plan does not start or one with `--no-prometheus` throws `EngineNodeInventoryRefused`. `superviseEngineNodes(endpoints, { intervalMs = 5000, timeoutMs = 20000, misses = 2, liveness, signal })`: every node answers `process_start_time_seconds` first (as zombie-cli's `metric` verifier reads it), then every interval a node the registry reports gone (`liveness`) or whose start time changed is lost at once, and one whose metrics fail (refused, timed out, HTTP error, no metric) `misses` checks in a row is lost: `EngineNodeLost` names each node and why (`Engine node collator-2000 is not listening (connection refused at …: killed or exited) on 2 checks in a row while the network ran`). `stop()` cancels the check in progress and joins it; the lab's own stop is never a loss. engine.json `nodeSupervision` (default `true`): `OwnedEngineDependency` starts Engine with `ppn start --node-verifier none` (`NODE_VERIFIER_OPTION`, `nodeVerifierArgs`; stock Engine v0.11.0 passes it to zombie-cli) and supervises from readiness until stop, racing its setup jobs and fork setup against a loss (`onLost`, `lost`, `monitorNodes`; `ppn start` exiting is a loss too); `false` passes no option, so zombie-cli keeps its `metric` default, and supervises nothing. `persistentNetwork` (option `nodeSupervision`, else engine.json's, default on) supervises every resume from readiness until `stop()` (`network.supervision`, `network.onLost`), refusing an unsupervisable inventory before any process starts. The Engine topology environment reports a loss through `RunningEnvironment.onLost`, and the runtime fails the active run with it. `tests/engine-health.test.ts`, `tests/network-supervision.test.ts`; on real processes `tools/verify/tests/engine-owned-start.test.ts` |
+| `evidence/*` | Progress diagnostics, log preservation |
+| `errors.ts` | Typed failures callers branch on instead of the message: `EndpointUnansweredError` (the dashboard or identity backend did not answer: `url`, and `systemCode`, the socket's `ECONNREFUSED`/`ECONNRESET` from the cause chain, `systemErrorCode`); `PortOccupiedError` (`port`, `owner`: `requireFreePorts`); `IncompleteNetworkError` (`directory`, `missing`: which process lacks which data, `MissingPersistedData`, also from `missingPersistedParts(plan)`); `MissingEngineToolingError` (`directory`, `missing`: `MissingToolingPath`, each process and path; `savedCheckout`, `currentCheckout`): a saved network whose Engine tooling is gone, its cache deleted or moved or another `PRODUCT_LAB_CACHE_DIR` set, refused before any port check or process with the way out (that cache again, or a new saved network, its identities discarded). Since 30 September 2026 the readiness check's failures are typed too: `FinalityNotAdvancingError` (one chain), `ChainsNotReadyError` (`standing()` gives each chain's finalized number when every chain answered and none advanced), `FinalityStalledError` (the recovery gave up: `attempts`, `stalls`), `FinalityPinnedError` (backlog 239: relay finality pinned at the approval threshold again after its one restart; `stalls`) and `ForkSetupError`. The wait for a resumed network takes a dashboard that runs but refuses its own port from the type and code (`refusedUnder`). `errorMessage` appends a cause's message unless the message already quotes it: that text comparison is formatting only and stays counted in the architecture baseline |
+| `topology/engine-plan.ts` | `engineLaunchPlan`: a resolved relay-backed `EngineTopology` as `ppn` commands (fork, `--fresh-bite`, `--upgrade`, `--runtime`, `--binary`, genesis subsets through `genesis-toml --parachains` then `start --toml`) and `config/ports.env` keys |
+| `topology/engine-topology.ts` | `resolveEngineTopology`, `engineOptionsSchema`, `engineProfiles` (full-fork, genesis-subset), `engineFlagOptions`, `ENGINE_FLAGS` (`--engine-arg`, `--engine-env`, `--engine-toml`), `ENGINE_KIND`, `DEFAULT_ENGINE_PORTS`: the adapter's own topology options, defaults, flags and issues (refactor step 5) |
+| `topology/engine-environment-factory.ts` | `EngineEnvironmentFactory`: the adapter's `TopologyAdapterFactory` (kind `engine`, section `engine`), constructed by the composition root when a lab selects `defineLab({ environmentAdapters: { engine: { engineConfig } } })` (`composition/src/adapter-catalogue.ts`), or registered directly with `defineLab({ environmentFactories: [new EngineEnvironmentFactory({ engineConfig })] })` |
+| `setup/prepare-engine.ts`, `setup/engine-build.ts`, `setup/patches.ts`, `setup/patch-catalogue.ts` | Stock Engine v0.11.0 preparation and shipped release pin; `preparedEngine` checks the pin, stock record, checkout HEAD, built CLI and dot CLI. Retired patched records and requests are refused. Descriptor overlays remain digest-checked through the manifest. |
+| `setup/setup-command.ts` | The command every Engine refusal names, for where this adapter runs (`engineSetupCommand(environment?)`): `bun run setup` in product-lab's checkout, `npx --no-install product-lab fetch --environment <id>` in a product, which has no `bun run setup` (`insideProductLabCheckout`, this adapter's own copy of the CLI's check) |
+| `saved-fork/saved-fork-tooling.ts` | The Engine checkout a saved fork starts from, `<cacheDir>/engine` (16 October 2026): `savedForkEngine` (the start's check, `as-prepared`, its refusal naming `SavedForkSettings.setupCommand`, default `engineSetupCommand`), and `savedForkTooling`, the fork's `EnvironmentProvider.tooling` (id `engine:<cacheDir>/engine`, shared by every saved fork with that cache; kind `engine`): `check` is `savedForkEngine`'s, `prepare` runs `prepareEngine` there for the fork's network with the request's source (`repository`, `ref`, `patched: false` for stock, also the default; `patched: true` is refused) and returns the check of exactly what it asked for. `product-lab fetch` prepares it and `product-lab doctor` reports it in a product. A saved fork started from a stock build writes `enginePatch: "none"` in its own engine.json |
+| `testing` subpath | Fake Engine dashboard and chain RPCs for contract tests (`codeHashOffset`, `specVersion` change the served runtime) |
+| `versions/engine-versions.ts` | Phase 6: `engineVersionPlan(topology, resolution, { snapshot, now })` (what Engine honours: runtimes read from the network it forks; refused as unsupported: released or git runtimes, resolved binaries, a runtime override on the same chain, a network runtime on genesis, a published bundle that `SnapshotFreshness` finds stale or on another runtime), `engineVersionCheck(session, plan, snapshot)` (each chain's observed `:code` hash, spec, genesis and bite block against the resolution), `VersionedEngineDependency` (checks every start and reset, stops and throws `EngineVersionMismatch` on other code), `engineVersionsRecord`, `workspaceVersionPlan` (persistent networks). Engine cannot fork at a chosen block: a `deployed` resolution is honoured by runtime, and the bite blocks are recorded beside the resolved block |
+
+The engine's running environment (`engineEnvironment`, a `RunningEnvironment`
+from `@product-lab/ports/environment`) exposes `routes` (`engineRoutes` of the
+current session, recomputed after a reset) and `processes()` (the Engine tree
+its registry records). It offers no `time` control (backlog 16): `timeUnsupported` is `ENGINE_TIME_UNSUPPORTED`,
+exported for consumers' own Engine environments (Notes' `paseo-copy`, another example's `local`): its validators and
+collators author on the wall clock, so no block is built on request and no chain's clock can be moved.
+The manual-seal session of the relay-less fast lane is
+[`@product-lab/env-relayless`](../relayless)'s own since refactor step 7 part b;
+an example that uses both composes them in its own session.
+
+Local state defaults to `.product-lab/` in the consumer's directory (tooling,
+`engine.json`, `session.json`, network locks). Commands, run from the
+consumer's root (they never touch a developer's own Engine clone):
+
+```sh
+node --import tsx <product-lab>/composition/src/engine-setup.ts --dry-run   # show paths only
+node --import tsx <product-lab>/composition/src/engine-setup.ts             # clone and build stock Engine
+node --import tsx <product-lab>/composition/src/engine-setup.ts --engine-repository <url|path> --engine-ref <ref> --no-patch   # stock Engine from another repository
+node --import tsx <product-lab>/tools/verify/src/engine-network.ts                       # start an owned local fork
+```
+
+The saved forks' Engine (`paseo-copy` and `previewnet-copy`) lives in the shared cache,
+`<PRODUCT_LAB_CACHE_DIR or ~/.cache/product-lab>/engine`. `bun run setup` in this repository and
+`npx --no-install product-lab fetch --environment paseo-copy` in an installed product both prepare stock
+Engine v0.11.0 (`e11792b234ca0c39a5fc1d47af491f82fc8e2670`) with no Engine flags required.
+`--no-patch` and `--no-engine-patch` remain compatible aliases for that default. `--engine-repository` and
+`--engine-ref` can prepare another explicit source; `preparedEngine` checks the tested pin unless its caller
+explicitly supplies a ref. A saved fork's normal start checks the tested pin.
+
+The build record and generated engine.json say `patch: "none"` and `enginePatch: "none"`. Before accepting the
+prepared checkout, the adapter checks its immutable ref, empty patch digest, checkout HEAD, built CLI and pinned
+dot CLI. A stale or legacy patched record is refused with the setup command. The retired patch API also refuses.
+Stock Engine carries the neutral `--node-verifier` option (#43), so the lab continues passing `none` only while
+supervising every node. The light layout's four-validator descriptor overlay stays necessary; Engine's new light
+specs are for light clients and do not reduce the fork's nodes.
+
+`engine-network.ts` moved to `tools/verify` in refactor step 7: it wires the Node system adapters, which this adapter never imports.
+Blocking setup commands have explicit limits: 30 s for local Git checks and key tools,
+2 minutes to resolve a remote ref, and 30 minutes for a setup build command. A copy-on-write
+template clone is limited to 10 minutes before falling back to the ordinary file copy.
+
+Existence probes (`missingPersistedData`'s files, `applyIpfsPorts`, `usableForkBundle`, the start log's poll) answer
+"absent" only for a path that does not exist; one that exists but cannot be read is reported as that error, not
+taken as missing (`tests/probe-failures.test.ts`).
+
+Several forks on one machine need separate caches or prepared Engine checkouts, plus distinct ports. Stock Engine v0.11.0 has a known shared-checkout sweep gap: its `finishGroup` cleanup matches service commands by checkout path, even when they belong to another workspace. The lab enforces checkout exclusivity for the whole network lifetime, including shutdown. A second start on that checkout is refused immediately. Saved networks may reuse a checkout sequentially; each resume restates its own ports (`restateEnginePorts`). PID and process start time distinguish stale lock owners from reused PIDs, and a stale lock is recovered (with a warning) only when every Engine and service process its holder recorded is gone; otherwise it stays, naming the workspace to stop. Restarting that workspace stops what its interrupted run left while holding the lock. A live or unknown owner remains locked. This is a checkout-exclusivity guarantee, not a general ownership-safety claim.
+
+Version resolutions (Phase 6): `engineEnvironment` takes `EnvironmentStartContext.versions`, refuses what `engineVersionPlan` refuses before anything starts, checks the fork after every start and reset (`VersionedEngineDependency`), adds the plan's notes to the environment's notices, and records `extra.versions` (the resolution, each runtime's row, what did not apply) in the run manifest and `versions` in the provenance. `EngineEnvironmentSettings.maxSnapshotAgeMs` (default 24 h) bounds a published bundle's age for a `deployed` resolution.
+
+System ports (refactor step 7): `EngineEnvironmentFactory.create(topology, deps)` passes the lab's system ports (the composition root's Node ports, plus any the lab gives as `defineLab({ system })`) to the environment, which requires `processes`, `commands`, `ports`, `rpc`, `environment`, `logger` and `runManifests` when it starts (not when it is created) and refuses with the missing names. Fork setup signs the DotNS attestation allowance with three bounded `dot` commands through `commands`, using the start's abort signal. The run manifest (`<reports>/run-manifest-<run id>.json`) is built by the domain's `createRunManifest` with an adapter-owned timestamp (which redacts sensitive customizations) and written through the injected `RunManifestStore` (`writeTo`); the adapter's own writer is deleted.
+
+## Package
+
+| | |
+| --- | --- |
+| Nx project | `env-engine` |
+| Tags | `scope:provider`, `type:adapter` |
+| Exports | `.` (`src/index.ts`: session, readiness, network lock, owned Engine, fork isolation, persistent network (including its data guard, `completePlan`, `probeEthRpc`), plan ports and eth-rpc (`containPlanPorts`, `withEthRpc`), network templates (`saveNetworkTemplate`, `restoreNetworkTemplate`, `cloneTree`), stall recovery (`lostApprovalChecks`, `unapprovable`, `keepStallEvidence`, `FinalityStalledError`, `NetworkStartRecovery`, `StallRecoveryOptions`), a relay pinned at the approval threshold (`ApprovalPinWatch`, `relayHeads`, `describeApprovalPin`, `APPROVAL_PIN_GAP`, `APPROVAL_PIN_HOLD_MS`, `APPROVAL_PIN_RESTARTS`, `FinalityPinnedError`), chain time after a resume (`waitForChainClocks`, `chainTime`, `ChainClockBehindError`, `ChainLag`, `CHAIN_CLOCK_MAX_LAG_MS`, `CHAIN_CLOCK_TIMEOUT_MS`, `TIMESTAMP_NOW_KEY`), the drain before a stop (`drainPlan`, `drainCollators`, `DrainResult`, `DRAIN_TIMEOUT_MS`, `DRAIN_QUIET_BLOCKS`, `PARAS_HEADS_PREFIX`), the fork setup checks (`ensurePeopleLiteAttestationAllowance`, `ForkSetupError`), network seeds (`publishNetworkSeed`, `adoptNetworkSeed`, `networkSeedDirectory`, `readNetworkSeed`, `NetworkSeedRefused`), runtime upgrade (`upgradeRuntime`, `observeRuntime`, `recordRuntimeUpgrade`), published bundles (`publishedBundle`, `startPrefetch`, `installStagedBundle`), fork freshness (`forkState`, `freshness`, `updateForkRuntimes`, `prepareRefresh`, `prefetchNewerState`, `savedForkUpdates`), setup (`prepareEngine`, `preparedEngine`, `EngineNotPrepared`, `engineBuildIssue`; since 16 October 2026 `EngineSource`, `engineRepository`, `defaultEngineRepository`, `ENGINE_PATCHES`, `EnginePatch`, `PreparedEnginePatch`, `recordedPatch`, `engineSetupCommand`, `CHECKOUT_ENGINE_SETUP`), saved forks (the user's decision 5: `savedForkEnvironment`, `SavedForkSettings`, `SAVED_FORK_LAYOUTS`, `savedForkFor`, `savedForkPorts`, `savedForkBlock`, `SAVED_FORK_STARTUP_MS`, and their Engine: `savedForkTooling`, `savedForkEngine`, `savedForkEngineState`, `savedForkSetupCommand`, `engineBuildWords`; `src/saved-fork/saved-fork.ts`, moved from Notes' owned fork; composition's `paseoCopy()` builds them; `tests/saved-fork.test.ts`), `engineLaunchPlan`, `resolveEngineTopology`, `EngineEnvironmentFactory`, `engineEnvironment`), `./session` (`src/session/session.ts`), `./testing` (`src/testing/engine-dashboard.ts`) |
+| Implements | `TopologyAdapterFactory` (`EngineEnvironmentFactory`) and `TopologyEnvironment` (`engineEnvironment`, whose `start` returns a `RunningEnvironment` with `routes` and `processes()`), both `@product-lab/ports/environment` (step 7 part b; `ownedEngineEnvironment` is a legacy `LabEnvironment`); takes `ProcessRunner`/`OwnedProcesses`, `CommandRunner`, `PortAllocator`, `JsonRpcClient`, `EnvironmentVariables`, `Logger`, `FileSystem` and `LockManager` from `@product-lab/ports/system` and `RunManifestStore` from `@product-lab/ports/storage` |
+| Depends on | `@product-lab/domain`, `@product-lab/ports`, `zod`. The shared topology options schema is its own copy (`src/topology/topology-options-schema.ts`); the environment contract kit's `malformed-throws` case keeps it the same as the other topology adapters' |
+| Used by | the composition root (`composition/src/adapter-catalogue.ts`), `examples/notes` and another example, `tools/verify`. No adapter imports it: `host-truapi-cli` since step 7 part a, `env-relayless` since step 7 part b (it imported the `LabSession` contract) |
+
+Contract kit: `tests/contract.test.ts` runs the [environment contract kit](../../../contracts/environment/README.md), declaring `topology`, `profiles`, `flags`, `workspaces` and `sensitiveOptions`. The lifecycle cases (`lifecycle`, `reset`, `routes`, and `sensitiveOptions` on a running network) start a real Engine network through the Node system ports, which this package's tests never import: they moved to `tools/verify/tests/environment-contract-live.test.ts` in step 7 and run only with `PRODUCT_LAB_KIT_LIVE=1` and `PRODUCT_LAB_KIT_ENGINE_CONFIG` (not run by the gates).
+
+Tests: `bunx nx run env-engine:test` (fakes for Engine, processes, JSON-RPC and the lock; no fork; `tests/persistent-data.test.ts`
+covers the data guard on temporary directories; `tests/shared-checkout.test.ts` two networks resuming on one Engine checkout,
+a taken port named, and the readiness wait's reasons). With the Node
+system adapters: the lock on disk, `tools/verify/tests/scenario-lock.test.ts`; an owned Engine started, awaited and
+stopped against a stand-in checkout, and a stock build passing the pre-start check while legacy patched configurations are refused, `tools/verify/tests/engine-owned-start.test.ts`; an Engine session routing the
+truAPI CLI host, `tools/verify/tests/host-routes.test.ts`.

package/adapters/environment/engine/engine.example.json

@@ -0,0 +1,9 @@
+{
+  "checkout": "/absolute/path/to/patched-previewnet-engine",
+  "workspace": "/absolute/path/to/prepared-engine-workspace",
+  "network": "paseo-next-v2",
+  "mode": "fork",
+  "dashboard": "http://127.0.0.1:8090",
+  "extraPath": [],
+  "startupTimeoutMs": 600000
+}

package/adapters/environment/engine/src/engine/approval-pin.d.ts

@@ -0,0 +1,106 @@
+import type { JsonRpcClient } from "@parity/product-lab/ports/system";
+/**
+ * How far the relay's best head may run ahead of its finalized head before finality is called pinned: polkadot-sdk's
+ * approval-distribution `AggressionConfig::default().l1_threshold` (16), less two blocks, as the gap read between two
+ * blocks stands at 14 to 17 while pinned (row 93, row 231). A normal start runs 2 to 5 behind.
+ */
+export declare const APPROVAL_PIN_GAP = 14;
+/** How long the gap must stay at `APPROVAL_PIN_GAP` or more, once relay finality first moved, before the start restarts. */
+export declare const APPROVAL_PIN_HOLD_MS = 30000;
+/** How often a start restarts for a pinned relay: once; a second pin fails the start (`FinalityPinnedError`). */
+export declare const APPROVAL_PIN_RESTARTS = 1;
+/** The relay's two heads, read together. */
+export interface RelayHeads {
+    readonly best: bigint;
+    readonly finalized: bigint;
+    /** When they were read (ms since the epoch). */
+    readonly at: number;
+}
+/** A pinned relay: its heads when the hold was reached, the gap, and since when it stood at the threshold or more. */
+export interface ApprovalPin {
+    readonly best: bigint;
+    readonly finalized: bigint;
+    readonly gap: bigint;
+    readonly heldMs: number;
+}
+/**
+ * What the watch says after a read: `pinned` (restart), `watching` (at the threshold, not yet held), `pending` (at the
+ * threshold while blocks the start began with are still being finalized), or `clear`.
+ */
+export type PinState = {
+    readonly state: "pinned";
+    readonly pin: ApprovalPin;
+} | {
+    readonly state: "watching";
+    readonly gap: bigint;
+    readonly heldMs: number;
+} | {
+    readonly state: "pending";
+    readonly gap: bigint;
+    readonly firstBest: bigint;
+} | {
+    readonly state: "clear";
+    readonly gap?: bigint;
+};
+/**
+ * The rule, fed one read at a time: nothing counts until the relay's finalized head passes the best head of the first
+ * read, so blocks the start began with are finalized first (a start's first minutes, before the validators finalize at
+ * all, are the standing verdict's). That matters after the pin's own restart: it stops the network without a drain, and
+ * the second start begins 16 blocks behind, finalizing one block per block until that backlog is through, the gap
+ * standing at 16 meanwhile (live 8 October 2026, `239L-2044` start 3). After that, a gap of `gap` or more for `holdMs`
+ * without a read below it is a pin.
+ */
+export declare class ApprovalPinWatch {
+    private readonly gap;
+    private readonly holdMs;
+    private firstBest;
+    private passed;
+    private since;
+    constructor(gap?: number, holdMs?: number);
+    observe(heads: RelayHeads): PinState;
+}
+/** The relay's best and finalized heads at a validator's RPC. */
+export declare function relayHeads(client: JsonRpcClient, ws: string, now?: () => number): Promise<RelayHeads>;
+/** The start's record of a pin, with the numbers: what `recovery.stalls`, the log and the evidence say. */
+export declare function describeApprovalPin(pin: ApprovalPin, restarted: boolean): string;
+/** The pin rule's bounds for one start. */
+export interface PinPolicy {
+    readonly gap: number;
+    readonly holdMs: number;
+    /** How often the relay is read. */
+    readonly pollMs: number;
+}
+/** What a start does on a pin: the verdict kind persistent-network.ts restarts or fails on, with the numbers. */
+export interface PinVerdict {
+    readonly kind: "pinned";
+    readonly reason: string;
+    readonly detail: Record<string, unknown>;
+}
+/** One start's pin watch: the relay read at most once per `pollMs`, into the rule. */
+export declare class RelayPinSampler {
+    private readonly client;
+    private readonly ws;
+    readonly policy: PinPolicy;
+    /** Restarts already made for a pin in this `start()` (decides the verdict's wording). */
+    private readonly restartsSoFar;
+    readonly watch: ApprovalPinWatch;
+    private readAt;
+    private last;
+    /** Why the last read failed, when it did. */
+    lastError: string | undefined;
+    constructor(client: JsonRpcClient, ws: string, policy: PinPolicy, 
+    /** Restarts already made for a pin in this `start()` (decides the verdict's wording). */
+    restartsSoFar: number);
+    /** The rule after a read (`force`: read now, else at most once per `pollMs`); undefined while the relay cannot be read. */
+    read(force?: boolean): Promise<PinState | undefined>;
+    /** The verdict when `state` is a pin. */
+    verdict(state: PinState | undefined): PinVerdict | undefined;
+    /**
+     * After the chains are ready: while the relay stands at the threshold it is read until it falls below (undefined, no
+     * restart) or holds (the verdict); a relay below the threshold costs one read and no wait. While it is still
+     * finalizing the blocks the start began with (`pending`), it is read for at most `pendingMs`, then the start goes on.
+     */
+    settle(pendingMs: number, watching: (state: Extract<PinState, {
+        state: "watching" | "pending";
+    }>) => void, signal?: AbortSignal): Promise<PinVerdict | undefined>;
+}

package/adapters/environment/engine/src/engine/approval-pin.js

@@ -0,0 +1,98 @@
+// Copyright 2026 Parity Technologies (UK) Ltd.
+// SPDX-License-Identifier: Apache-2.0
+import { setTimeout as delay } from "node:timers/promises";
+import { bestNumber, finalizedNumber } from "../session/readiness.js";
+export const APPROVAL_PIN_GAP = 14;
+export const APPROVAL_PIN_HOLD_MS = 30_000;
+export const APPROVAL_PIN_RESTARTS = 1;
+export class ApprovalPinWatch {
+    gap;
+    holdMs;
+    firstBest;
+    passed = false;
+    since;
+    constructor(gap = APPROVAL_PIN_GAP, holdMs = APPROVAL_PIN_HOLD_MS) {
+        this.gap = gap;
+        this.holdMs = holdMs;
+    }
+    observe(heads) {
+        this.firstBest ??= heads.best;
+        if (heads.finalized > this.firstBest)
+            this.passed = true;
+        const gap = heads.best - heads.finalized;
+        if (gap < BigInt(this.gap)) {
+            this.since = undefined;
+            return { state: "clear", gap };
+        }
+        if (!this.passed)
+            return { state: "pending", gap, firstBest: this.firstBest };
+        this.since ??= heads.at;
+        const heldMs = heads.at - this.since;
+        return heldMs >= this.holdMs ? { state: "pinned", pin: { best: heads.best, finalized: heads.finalized, gap, heldMs } } : { state: "watching", gap, heldMs };
+    }
+}
+export async function relayHeads(client, ws, now = Date.now) {
+    const [best, finalized] = await Promise.all([bestNumber(client, ws), finalizedNumber(client, ws)]);
+    return { best: BigInt(best), finalized, at: now() };
+}
+export function describeApprovalPin(pin, restarted) {
+    const numbers = `relay best #${pin.best}, finalized #${pin.finalized}: ${pin.gap} blocks behind for ${Math.round(pin.heldMs / 1000)} s, at or past ${APPROVAL_PIN_GAP} (approval-distribution's level-1 threshold is 16)`;
+    return restarted
+        ? `finality pinned at the approval threshold after a session change; restarted once (${numbers})`
+        : `finality pinned at the approval threshold after a session change, again after one restart (${numbers})`;
+}
+export class RelayPinSampler {
+    client;
+    ws;
+    policy;
+    restartsSoFar;
+    watch;
+    readAt = 0;
+    last;
+    lastError;
+    constructor(client, ws, policy, 
+    restartsSoFar) {
+        this.client = client;
+        this.ws = ws;
+        this.policy = policy;
+        this.restartsSoFar = restartsSoFar;
+        this.watch = new ApprovalPinWatch(policy.gap, policy.holdMs);
+    }
+    async read(force = false) {
+        if (!force && Date.now() - this.readAt < this.policy.pollMs)
+            return this.last;
+        this.readAt = Date.now();
+        try {
+            this.last = this.watch.observe(await relayHeads(this.client, this.ws));
+            this.lastError = undefined;
+        }
+        catch (error) {
+            this.lastError = error instanceof Error ? error.message : String(error);
+            this.last = undefined;
+        }
+        return this.last;
+    }
+    verdict(state) {
+        if (state?.state !== "pinned")
+            return undefined;
+        const { pin } = state;
+        return {
+            kind: "pinned",
+            reason: describeApprovalPin(pin, this.restartsSoFar < APPROVAL_PIN_RESTARTS),
+            detail: { relay: { best: pin.best.toString(), finalized: pin.finalized.toString(), gap: pin.gap.toString(), heldMs: pin.heldMs } },
+        };
+    }
+    async settle(pendingMs, watching, signal) {
+        const until = Date.now() + pendingMs;
+        for (;;) {
+            signal?.throwIfAborted();
+            const state = await this.read(true);
+            if (state?.state === "pending" && Date.now() >= until)
+                return undefined;
+            if (state?.state !== "watching" && state?.state !== "pending")
+                return this.verdict(state);
+            watching(state);
+            await delay(this.policy.pollMs);
+        }
+    }
+}

package/adapters/environment/engine/src/engine/chain-clock.d.ts

@@ -0,0 +1,55 @@
+import type { JsonRpcClient } from "@parity/product-lab/ports/system";
+/** `Timestamp.Now`'s storage key: twox128("Timestamp") ++ twox128("Now"). */
+export declare const TIMESTAMP_NOW_KEY = "0xf0c365c3cf59d671eb72da0e7a4113c49f1f0515f462cdcf84e0f1d6045dfcbb";
+/** How far behind the wall clock a chain's time may be: well inside the identity backend's 600 s. */
+export declare const CHAIN_CLOCK_MAX_LAG_MS = 120000;
+/** How long a start waits for every chain's time to catch up. */
+export declare const CHAIN_CLOCK_TIMEOUT_MS = 120000;
+/** One chain whose time is checked. */
+export interface ClockedChain {
+    readonly id: string;
+    readonly name: string;
+    readonly ws: string;
+}
+/** A chain's time and best block, as read once. */
+export interface ChainTime {
+    readonly chain: ClockedChain;
+    /** `Timestamp.Now` at the best block (ms since the epoch). */
+    readonly at: number;
+    readonly best: number;
+}
+/** A chain's time and how far it is behind the wall clock when read (negative: ahead). */
+export type ChainLag = ChainTime & {
+    readonly lagMs: number;
+};
+/** Every chain that has not caught up, with how far behind it is. */
+export declare class ChainClockBehindError extends Error {
+    readonly behind: readonly ChainLag[];
+    readonly waitedMs: number;
+    constructor(behind: readonly ChainLag[], waitedMs: number);
+}
+/** `chain`'s `Timestamp.Now` and best block number, read at its best block. */
+export declare function chainTime(client: JsonRpcClient, chain: ClockedChain, signal?: AbortSignal): Promise<ChainTime>;
+/** A SCALE `u64` (little-endian, 8 bytes) given as hex. */
+export declare function decodeMoment(hex: string, name: string): number;
+export interface ChainClockOptions {
+    maxLagMs?: number;
+    timeoutMs?: number;
+    pollMs?: number;
+    /** The wall clock (ms since the epoch). */
+    now?: () => number;
+    wait?: (ms: number) => Promise<void>;
+    signal?: AbortSignal;
+    /** Told what is still behind while the wait lasts. */
+    waiting?: (behind: readonly ChainLag[]) => void;
+    /** Told every chain's time and lag at each read (the first read is how far behind the resume found them). */
+    onRead?: (times: readonly ChainLag[]) => void;
+}
+/**
+ * Wait until every chain's time is within `maxLagMs` of the wall clock (a chain ahead of it is fine: its node's clock
+ * may run a little fast). Returns the times read last; throws `ChainClockBehindError` when one has not caught up by
+ * `timeoutMs`. A chain that does not answer fails as its read does.
+ */
+export declare function waitForChainClocks(client: JsonRpcClient, chains: readonly ClockedChain[], options?: ChainClockOptions): Promise<readonly ChainTime[]>;
+/** A duration as people read it: `45 s`, `12 min`, `3 h 14 min`. */
+export declare function span(ms: number): string;

package/adapters/environment/engine/src/engine/chain-clock.js

@@ -0,0 +1,66 @@
+// Copyright 2026 Parity Technologies (UK) Ltd.
+// SPDX-License-Identifier: Apache-2.0
+import { rpc } from "../session/readiness.js";
+export const TIMESTAMP_NOW_KEY = "0xf0c365c3cf59d671eb72da0e7a4113c49f1f0515f462cdcf84e0f1d6045dfcbb";
+export const CHAIN_CLOCK_MAX_LAG_MS = 120_000;
+export const CHAIN_CLOCK_TIMEOUT_MS = 120_000;
+export class ChainClockBehindError extends Error {
+    behind;
+    waitedMs;
+    constructor(behind, waitedMs) {
+        super(`${behind.map((chain) => `${chain.chain.name}'s time is ${span(chain.lagMs)} behind this machine's clock (best block #${chain.best})`).join("; ")} ` +
+            `after waiting ${span(waitedMs)}: ${behind.length === 1 ? "it is" : "they are"} not building blocks on the saved chain. ` +
+            "Signatures timestamped with chain time are refused: the truAPI CLI host signs its dotNS registration with Asset Hub time, " +
+            "and the identity backend refuses one more than 600 s old. Restart the network, or start a fresh fork.");
+        this.behind = behind;
+        this.waitedMs = waitedMs;
+        this.name = "ChainClockBehindError";
+    }
+}
+export async function chainTime(client, chain, signal) {
+    const [value, header] = await Promise.all([
+        rpc(client, chain.ws, "state_getStorage", [TIMESTAMP_NOW_KEY], signal),
+        rpc(client, chain.ws, "chain_getHeader", [], signal),
+    ]);
+    return { chain, at: decodeMoment(value, chain.name), best: Number.parseInt(header.number, 16) };
+}
+export function decodeMoment(hex, name) {
+    const bytes = /^0x([0-9a-f]{16})$/i.exec(hex)?.[1];
+    if (!bytes)
+        throw new Error(`${name}'s Timestamp.Now is not a u64: ${hex}`);
+    let value = 0n;
+    for (let index = 7; index >= 0; index--)
+        value = (value << 8n) | BigInt(Number.parseInt(bytes.slice(index * 2, index * 2 + 2), 16));
+    return Number(value);
+}
+export async function waitForChainClocks(client, chains, options = {}) {
+    const now = options.now ?? Date.now;
+    const wait = options.wait ?? ((ms) => new Promise((resolve) => setTimeout(resolve, ms)));
+    const maxLagMs = options.maxLagMs ?? CHAIN_CLOCK_MAX_LAG_MS;
+    const timeoutMs = options.timeoutMs ?? CHAIN_CLOCK_TIMEOUT_MS;
+    const started = now();
+    for (;;) {
+        options.signal?.throwIfAborted();
+        const times = await Promise.all(chains.map((chain) => chainTime(client, chain, options.signal)));
+        const read = now();
+        const lags = times.map((time) => ({ ...time, lagMs: read - time.at }));
+        options.onRead?.(lags);
+        const behind = lags.filter((time) => time.lagMs > maxLagMs);
+        if (!behind.length)
+            return times;
+        if (read - started >= timeoutMs)
+            throw new ChainClockBehindError(behind, read - started);
+        options.waiting?.(behind);
+        await wait(options.pollMs ?? 2000);
+    }
+}
+export function span(ms) {
+    const seconds = Math.round(ms / 1000);
+    if (seconds < 120)
+        return `${seconds} s`;
+    const minutes = Math.round(seconds / 60);
+    if (minutes < 120)
+        return `${minutes} min`;
+    const hours = Math.floor(minutes / 60);
+    return `${hours} h ${minutes % 60} min`;
+}

package/adapters/environment/engine/src/engine/checkout-lock.d.ts

@@ -0,0 +1,52 @@
+import type { FileSystem, HeldLock, LockManager, LockOwner, Logger, OwnedProcesses, ProcessRunner } from "@parity/product-lab/ports/system";
+/** The lock beside the files it guards: `<checkout>/config/ports.env.lock`. */
+export declare const checkoutPortsLock: (checkout: string) => string;
+export declare class CheckoutLockTimeout extends Error {
+    readonly lock: string;
+    readonly waitedMs: number;
+    readonly owner?: LockOwner | undefined;
+    constructor(lock: string, waitedMs: number, owner?: LockOwner | undefined);
+}
+export interface CheckoutLockOptions {
+    /** Recorded in the lock's owner record (who holds it). */
+    network: string;
+    /** This lifetime's registry, recorded in the lock: every Engine and service tree it starts is recorded there. */
+    registry?: string;
+    /** Reads a dead holder's recorded registries (their status only; nothing is signalled). */
+    inspect?: Pick<ProcessRunner, "open">;
+    /** The caller's own open registry: read instead of opening its directory a second time. */
+    own?: OwnedProcesses;
+    /**
+     * A dead holder's members still alive: `refuse` (the default) names the workspace to stop. `own`: members alive only
+     * in `own` (this workspace's interrupted owner) do not block recovery; the caller stops them while it holds the lock.
+     */
+    survivors?: "refuse" | "own";
+    /** Retained for compatibility; checkout contention now refuses immediately, without waiting. */
+    waitMs: number;
+    signal?: AbortSignal;
+    /** Called once when another network holds the checkout, before refusing. */
+    waiting?: (owner: LockOwner | undefined) => void;
+    /** Reports a verified stale owner whose lock was recovered. */
+    recovered?: (owner: LockOwner) => void;
+}
+/**
+ * Exclusive per checkout, enforced from before start until stop has joined Engine's finishGroup. A lock whose holder
+ * is gone is recovered only when the holder is fingerprinted and every member of every registry it recorded is gone
+ * too: Engine's trees run detached, so a dead lab can leave a live Engine whose cleanup sweeps the checkout later.
+ */
+export declare function acquireCheckoutPorts(locks: LockManager | undefined, checkout: string | undefined, options: CheckoutLockOptions): Promise<HeldLock | undefined>;
+/**
+ * What an interrupted owner left running in a workspace's registry (PersistentNetworkOptions.survivors), stopped while
+ * this lab holds the lock of every Engine checkout the workspace names: stopping a leftover `ppn` runs Engine's cleanup,
+ * which sweeps its checkout's services, so it never runs while another workspace may be live there. The lock of a
+ * dead holder is recovered only when its live members are this registry's. A workspace that names no Engine checkout
+ * (an attached network) holds no Engine tree of the lab's, and its leftovers are stopped as they are.
+ */
+export declare function stopWorkspaceSurvivors(directory: string, system: {
+    processes: OwnedProcesses;
+    registries?: Pick<ProcessRunner, "open">;
+    locks?: LockManager | undefined;
+    logger: Pick<Logger, "warn">;
+}, named: (string | undefined)[], signal?: AbortSignal): Promise<void>;
+/** The checkout engine.json names, if it can be read (readEngineConfiguration reports an unreadable one). */
+export declare function namedCheckout(files: Pick<FileSystem, "readJson">, engineConfig: string | undefined): string | undefined;