@elaraai/e3-rack 0.0.0-stage → 1.0.94

package/LICENSE.md

@@ -0,0 +1,54 @@
+# Business Source License 1.1
+
+Copyright (c) 2025 Elara AI Pty Ltd
+
+## License
+
+**Licensor:** Elara AI Pty Ltd
+
+**Licensed Work:** @elaraai/e3-rack
+
+**Change Date:** Four years from the date of each release
+
+**Change License:** AGPL-3.0
+
+## Terms
+
+The Licensed Work is provided under the terms of the Business Source License 1.1 as detailed below.
+
+### Grant of Rights
+
+The Licensor grants you the right to copy, modify, create derivative works, redistribute, and make non-production use of the Licensed Work.
+
+### Production Use Limitation
+
+**"Production Use"** means any use by or on behalf of a for-profit entity, other than for evaluation, testing, or development purposes.
+
+Production Use requires a separate commercial license from the Licensor.
+
+### Change Date
+
+On the Change Date (four years after each release), the Licensed Work will be made available under the Change License (AGPL-3.0).
+
+### No Warranty
+
+THE LICENSED WORK IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND. THE LICENSOR DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
+
+### Package-Specific Licenses
+
+Individual packages within this repository may have different licensing terms. See the LICENSE.md file in each package directory for details.
+
+## Commercial Licensing
+
+To obtain a commercial license for Production Use, contact:
+
+**Email:** support@elara.ai
+**Website:** https://elaraai.com
+
+## Governing Law
+
+This license is governed by the laws of New South Wales, Australia.
+
+---
+
+*Elara AI Pty Ltd*

package/README.md

@@ -1,3 +1,410 @@
-# Temporary Holding Version
+# @elaraai/e3-rack
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+Delegate eligible task bodies from a local e3 repository to a rack agent. The
+local process still plans the graph, checks caches and owns the repository.
+One hub per operating-system user coordinates the rack fleet across projects.
+
+## Connect a rack
+
+A rack reaches this machine's hub over HTTPS. The hub serves its rack listener
+with its own certificate, made the first time the listener serves TLS, and every
+rack pins that certificate by its SHA-256 fingerprint. No certificate authority
+or DNS name is involved, so the rack can reach the hub through any relay, and a
+relay or tunnel in between forwards bytes it cannot read.
+
+The usual path is an SSH reverse tunnel. The hub listens on the loopback only;
+this machine keeps a tunnel open to a relay that the rack can reach, for
+example `ssh -N -R 7401:127.0.0.1:7331 relay`, and nothing connects into this
+machine directly:
+
+```sh
+e3 rack enroll --listen 127.0.0.1:7331 --allow 127.0.0.1 \
+  --advertise-url https://relay.example:7401 --label workshop
+```
+
+On a private network the rack can connect to the hub directly instead:
+
+```sh
+e3 rack enroll --listen 10.0.0.5:7331 --label workshop
+```
+
+This starts the hub and prints the **source origin**, the hub's **certificate
+SHA-256**, the matching e3-rack release and a short-lived single-use enrollment
+token. The source origin is how the rack reaches this machine: the
+`--advertise-url`, or the listener's own address when none is set. It is not a
+repository URL or the rack server's address; `localhost` on the rack would point
+back at the worker itself. Keep the hub and any tunnel available while work runs.
+
+`--advertise-url` names the origin racks reach the hub at: a relay's, a
+tunnel's or a TLS terminator's. The hub builds every rack and data URL on it.
+Without it, data URLs use the origin each rack request named, which is right
+only when the rack connects to the listener directly. A listener on every
+address (`0.0.0.0` or `::`) has no address to hand out, so enrollment needs
+`--advertise-url` there.
+
+The certificate is kept in the hub's private state (`hub-tls.beast2`, beside
+its other state) until it is removed. With the hub stopped, removing that file
+makes a new certificate on the next start, which every worker must then pin again.
+
+Plain HTTP (`--no-tls`) is for a TLS terminator in front of the hub, as in
+`e3 rack config --no-tls --listen 127.0.0.1:7331 --allow 127.0.0.1 --advertise-url https://terminator.example`.
+The hub refuses a plain-HTTP listener on a wildcard or public address, or
+without `--allow`: anything on the path of plain HTTP reads every rack token,
+data URL, input, output and log. A VPN whose hub decrypts and re-encrypts
+traffic is not encrypted end to end. `--allow <ip,...>` admits only the named
+addresses (through a local tunnel or terminator, `127.0.0.1`); it does not
+encrypt anything.
+
+Use the matching, verified companion release from e3-cloud **inside an already
+prepared worker VM**, at the release `e3 rack enroll` printed. Replace the public
+allocation path below with the one the host provisioner supplies; do not write
+allocation JSON yourself:
+
+```sh
+sudo /opt/e3-rack/releases/<release>/e3-rack configure \
+  --allocation '/var/lib/e3-rack/<agent-uuid>/allocation.json'
+sudo /opt/e3-rack/releases/<release>/e3-rack enroll \
+  --allocation '/var/lib/e3-rack/<agent-uuid>/allocation.json'
+```
+
+`configure` asks for the source origin and, for an HTTPS origin, the
+certificate SHA-256 the hub printed (none for a publicly trusted certificate,
+such as a cloud deployment's), then the worker label (`workshop` here), runtime
+settings and optional private-registry access. `enroll` asks for the one-time
+token. Both use hidden terminal prompts. The worker stores origins, pins,
+identities, settings and registry credentials together in encrypted storage; no
+token or source address is included in the printed shell commands. The
+enrollment token is intentionally displayed by this local minting command: do
+not copy its output into logs or a plaintext file.
+
+A plain-HTTP source origin needs `--allow-vpn-http` on both worker commands, and
+the worker's host must authorize that exact destination. Use it only on a path
+encrypted end to end, such as an SSH tunnel.
+
+The local command waits for a **new registration**, including one with no boot,
+tiers or capacity yet; `--no-wait` returns after printing the handoff. A label
+filters the observed registrations, so compare the result with the worker's
+enrollment result. Enrollment does not prove service activation or readiness.
+`e3 rack list` shows `enrolled-not-started` before a first heartbeat.
+
+Once the worker has been activated and explicitly resumed, its heartbeat
+publishes concurrent capacity as free plus occupied slots. Busy workers remain
+eligible; fenced, draining or unhealthy workers publish zero capacity and show
+`unavailable`. Existing work can still finish during drain. Only heartbeats
+refresh readiness: lease polling cannot keep an old capacity report alive.
+After 45 seconds without a heartbeat the worker stops counting as available.
+Source boot approval, when required, remains a separate gate from worker resume:
+`e3 rack config --require-approval` quarantines a rack after each reboot until
+`e3 rack approve <rack>` admits the boot it reports; enrollment admits the first.
+
+Independent agents belong to separate worker VMs, each with its own identity,
+encrypted configuration and host-enforced resource allocation. Whole GPUs are
+assigned through the host lifecycle, independently of this enrollment command.
+One worker can connect to up to 32 mutually trusted e3 sources. To add another,
+first drain the worker and stop its services and socket activation. Inside it:
+
+```sh
+sudo /opt/e3-rack/releases/<release>/e3-rack configure \
+  --allocation '/var/lib/e3-rack/<agent-uuid>/allocation.json' --add-source
+sudo /opt/e3-rack/releases/<release>/e3-rack enroll \
+  --allocation '/var/lib/e3-rack/<agent-uuid>/allocation.json' --source '<new-source-uuid>'
+```
+
+`configure --add-source` asks you to confirm that these sources may share the
+agent's trust boundary, then privately asks for the new origin. It prints the
+source UUID for `enroll`. Obtain that source's token with `e3 rack enroll` on
+the additional local e3 machine, or from the chosen cloud deployment. Omit the
+HTTP opt-in for HTTPS. The host must approve the additional endpoint. Existing
+identities and registry access are preserved, every source must be enrolled
+before startup, and the next boot remains fenced.
+
+Connections have separate clients, credentials, caches and completion journals.
+One worker-wide task/GPU limit bounds actual execution across them; per-source
+capacity reports must not be added together as physical reservations. All
+connections share the worker's lifecycle. They also share its process and
+credential trust boundary: a worker is the trust boundary. Put sources on one
+worker only when they trust each other, such as one team's developer hubs.
+Each client environment needs a worker of its own, and a cloud deployment and
+developer hubs belong on separate workers. Independent agents have no shared
+e3 task scheduler.
+
+The hub keeps no task data: leases and data URLs live in its memory, and every
+object passes through it from the repository's session. What a worker keeps is
+the worker's own policy. The hub checks each object a rack reads against its
+lease's closure only, so an agent that fetches every object again for each task
+and keeps nothing once the task ends works with it unchanged.
+
+The companion now provides `e3-rack source list --allocation <path>` inside the
+worker. It lists source UUIDs and enrollment state; `--show-origins` displays
+origins only on a terminal. To disconnect or re-enroll, drain and stop the worker,
+revoke the selected rack at its source, then use `source remove` or `source reset`
+with `--allocation <path> --source <UUID> --acknowledge-revocation`. Confirm the
+UUID at the hidden prompt. Reset prints a **new** UUID. Use it with `enroll
+--source`, supplying a fresh token obtained from the e3 source. Reset preserves
+the origin and keeps old caches/pending results out of the new identity.
+The flag acknowledges prior revocation; it does
+not revoke remotely. Other sources and retained data are preserved. Reconcile
+host endpoint policy before activation; the next boot remains fenced.
+
+Private registry access has its own worker-local commands:
+
+```sh
+sudo /opt/e3-rack/releases/<release>/e3-rack registry status \
+  --allocation '/var/lib/e3-rack/<agent-uuid>/allocation.json'
+# After a verified drain and stopping the worker services/sockets:
+sudo /opt/e3-rack/releases/<release>/e3-rack registry configure \
+  --allocation '/var/lib/e3-rack/<agent-uuid>/allocation.json'
+```
+
+Status can run while active and shows only whether credentials are stored; it
+does not test GHCR access. Configure privately prompts for username/read-only
+token without re-entering worker settings. Use `registry clear --allocation
+<path>` to remove stored access, confirming the installation UUID privately.
+Changes require the same release, the writer lock and no unresolved enrollment
+or service transition. They preserve current source identities/rotations, settings,
+journals and cached images/task data. Credentials stay in the encrypted bundle;
+no Docker login file is created. A later service activation receives the change,
+with the next boot still fenced. Neither replacing nor clearing revokes a token
+remotely. After interruption, repeat configure with the intended values or clear
+to ensure absence; repeating the current values or an absent clear makes no write.
+
+The companion also provides worker-local `reconcile-services`: after a completed
+drain and stopped services, it applies the host's next allocation generation and
+resource limits. The previous service-plan hash and exact drain/operation/boot
+binding protect the transition. Interrupted updates keep activation guarded and
+retry with the same operation; ciphertext and journals remain unchanged. This
+supports prepared GPU/VFIO assignments at the same executable release; the host
+still performs and verifies the physical move, and network quarantine remains.
+
+For a compatible upgrade or downgrade, the target verified executable's
+`switch-release` uses another completed drain, the previous plan hash and a new
+allocation generation, with unchanged resource/device budgets. Both binaries
+must report the same managed-state contract. It changes the runtime version in
+the **current** encrypted configuration, preserving all identities, registry
+access and journals. Interrupted commits stay guarded and retry with the same
+operation; downgrades never restore saved credentials or state. Keep both
+four-file releases available through receipt recovery. Services stay stopped
+and quarantined until separately activated and resumed.
+
+If the selected release never resumed, its exact recorded predecessor's
+`recover-release` uses the completed release receipt, current plan hash and a
+new operation at another generation increase. It requires either the original
+drained journal or a valid fenced journal with no operation at the failed
+generation, and verifies both releases against the recorded fingerprints. Any
+resume attempt, including one from a prior boot, prevents this recovery. Current
+credentials and journals are preserved; services remain stopped and quarantined.
+A release that resumed needs a fresh drain and normal `switch-release`; general
+journal repair and recovery after an undrainable resumed release remain open.
+
+The named host manager, source aliases, incompatible-state migrations,
+service/network activation and hardware acceptance remain
+[cloud #253](https://github.com/elaraai/e3-cloud/issues/253) work. Enrollment alone
+is not an end-to-end deployment. See the companion's
+[worker configuration and service contract](https://github.com/elaraai/e3-cloud/blob/72c5c629e1708421171d3c86b8d7cb470f46802e/packages/e3-rack-agent/credentials/README.md).
+
+## Select work
+
+```sh
+e3 rack policy . set 'train_*' rack --size large
+e3 dataflow run . dev --rack
+e3 watch src/main.ts . dev --start --rack
+```
+
+Policies live in `<repo>/rack/policy.beast2`. Rules match the entire task name
+with `*` and `?`; the first matching rule wins. By default, unmatched tasks
+stay local. Rules and placement do not change task or cache hashes.
+
+```sh
+e3 rack policy . show
+e3 rack policy . enable                  # opt in without a flag on each run
+e3 rack policy . mode auto               # route eligible unmatched tasks too
+e3 rack policy . set 'ingest_*' local
+e3 rack policy . defaults --no-spill --claim-timeout 30
+e3 dataflow run . dev --no-rack           # override the saved opt-in
+e3 dataflow run . dev --rack-only         # wait when compatible racks are busy
+e3 rack policy . unset 'train_*'
+e3 rack policy . reset
+```
+
+Defaults are disabled, opt-in selection, `large`, a 1440-minute timeout,
+capacity spill enabled, a 30-second claim recheck and no local retry of task
+failures. `--rack-only` disables capacity spill; infrastructure failures can
+still fall back locally. It does not force ineligible tasks onto a rack.
+Rack flags apply to local dataflows and watch with `--start`, and are refused
+for remote repository URLs. Calls, mutations and intake remain local.
+
+Only stock runner configurations can delegate. Custom commands, environments,
+platform extensions and unscannable programs stay local. Programs that access
+host files, environment variables or network resources also stay local by
+default: their dependencies may exist only on this machine. A task rule's
+`--allow-host-access` deliberately overrides that guard, for a rack that
+provides those resources itself. The companion agent's task VMs have no network
+device, host files or injected environment, so such a task fails there.
+
+## Execution and recovery
+
+Current sources include optional first-claim queue timings in grant response
+headers. The companion agent records admission, input/image preparation, VM and
+guest startup milestones on its own monotonic clock. Its
+[latency benchmark and p50/p95 report](https://github.com/elaraai/e3-cloud/blob/998d879ac6c199e7f300bf9b9531293b4e8f5dc0/scripts/rack/LATENCY.md)
+separate cached CPU, GPU and first-use observations, including failures and
+missing data. Guest execution entry precedes runtime imports and CUDA startup;
+source queue time is separate and cannot be added as exact end-to-end latency.
+This adds no persisted or BEAST2 wire field and requires no new credentials.
+
+After a cache miss, the local planner offers an eligible task or split-task
+unit to the hub. A compatible healthy agent claims it, downloads the task and
+its complete input object graph, executes in a VM, and streams logs. It uploads
+each output object by hash and size, then commits its upload receipt. The local
+session verifies the complete output graph before recording success. Planning,
+splitting and merging decisions remain local; map and merge bodies can both
+run remotely, including when the map outputs were already cached.
+
+Local work and fallback share the run's existing `-j` and `--memory` budget.
+Remote bodies consume rack slots without reserving a local runner slot. Local
+orchestrator width includes compatible rack capacity.
+
+Concurrent sessions of the same canonical repository and e3 release can share
+one lease. Cancelling one subscriber leaves the others running; cancelling
+the last stops the lease. An interrupted remote attempt and its local retry
+have distinct execution records. The task's timeout and the run's own
+cancellation are not retried locally; an attempt the rack stopped on its own
+is. Task failures are returned unless the policy explicitly enables
+`--retry-task-failures-locally`; infrastructure failures fall back locally.
+
+Ctrl-C in `e3 dataflow run` requests cancellation of that run, including its
+delegated work. A current companion agent keeps an outbound cancellation
+request open while work is active. The source checks the watched leases every
+250 ms and answers as soon as one stops; the agent aborts that task and destroys
+its VM before waiting for log delivery. Other tasks and sources keep running.
+The normal health heartbeat remains 15 seconds. Against an older source without
+this endpoint, the new agent uses a one-second heartbeat while busy.
+
+The companion also cancels input/layer downloads, output uploads, storage requests
+and upload retry waits. It stops parallel transfers and waits for their cleanup
+before deleting staging or releasing the invocation slot. A finished guest's VM
+is destroyed before waiting for output delivery. Each object/layer HTTP attempt
+has a 120-second deadline, including its body; cancellation interrupts it immediately.
+Cancellation also reaches first-use image builds, environment composition,
+scratch-disk preparation and VM startup. A shared image build continues for its
+remaining leases; the last cancelled waiter stops it and waits for cleanup.
+Cancelled boots retain GPU ownership until process exit is confirmed, and
+uncertain teardown requires recovery. Filesystem operations and final publication
+already in progress still finish safely before cleanup; a process stuck in kernel I/O can
+delay that. See the [companion cancellation boundaries](https://github.com/elaraai/e3-cloud/blob/72c5c629e1708421171d3c86b8d7cb470f46802e/packages/e3-rack-agent/README.md#cancelling-delegated-work).
+
+This targets cancellation delivery within about a second on a healthy connection;
+network failures and VM/GPU teardown can take longer. The CLI's cancellation
+message is not confirmation that the remote VM has exited. Killing the CLI
+without allowing its signal handler to run relies on lost-session cleanup.
+
+```sh
+e3 rack status                         # does not start an absent hub
+e3 rack list
+e3 task logs . dev.train_model --follow
+e3 rack remove workshop                # revoke this rack's credential
+e3 rack config --require-approval      # quarantine each reboot until approved
+e3 rack approve workshop               # admit the boot this rack last reported
+e3 rack stop                           # stop the shared hub
+```
+
+Hub state lives under `~/.e3/rack` (override with `E3_RACK_HOME`). The rack
+listener is off until configured. A running hub and client must have compatible
+control schemas (currently protocol 2 exactly); incompatible clients receive
+an explicit diagnostic. Agents must report a bundled e3 version at least as
+new as the project's. Stop and restart the hub after an incompatible upgrade.
+
+A hub that `e3 dataflow run` starts exits after 15 idle minutes, while a run
+waits up to 10 seconds for an enrolled rack to report healthy and fixes its
+parallelism as it starts; with no rack enrolled, or a policy that selects no
+task for a rack, it starts at once. An agent reconnecting after the hub exits
+backs off for up to 30 seconds. For a hub racks can always reach, run
+`e3 rack hub` as a service, or set `e3 rack config --idle-exit 0`.
+
+A rack reads and writes each object through a data URL that lasts 15 minutes,
+and a transfer may take all of it; an object is at most 5 GiB. Over a slow
+link, a large object can outlast its URL, and the work then falls back locally.
+
+## Local API server
+
+Install matching `@elaraai/e3-api-server` and `@elaraai/e3-rack`, then opt in:
+
+```sh
+e3-api-server --repos ./repos --rack -j 4 --memory 8G
+```
+
+Each API dataflow run gets its own rack session using the server's existing
+shared budget. Repository rules still select tasks; explicit server opt-in
+overrides the policy's `enabled` default. A programmatic server can use:
+
+```ts
+import { createServer } from '@elaraai/e3-api-server';
+import { createDataflowRunnerFactory } from '@elaraai/e3-rack';
+
+const server = await createServer({
+  reposDir: './repos',
+  dataflowRunner: createDataflowRunnerFactory(),
+});
+await server.start();
+```
+
+The server can run without the optional rack package. Its portable handlers
+accept a runner factory; they do not import Node-only hub code.
+
+## Compatibility and verification boundaries
+
+If `rack status` reports a listener error, confirm the configured address is
+present on this machine and the port is free; a listener the hub refuses to
+serve says why. Prefer a stable source origin, such as a relay's address. A
+changed source origin needs a deliberate managed source transition;
+re-running worker `configure` preserves an enrolled source's origin and identity.
+If a rack is incompatible, its executable's bundled e3 must be upgraded along
+with its runtime images. Managed `configure` deliberately refuses a release
+change: use the companion's guarded `switch-release` workflow above. Releases
+without its managed-state compatibility contract are refused. Do not switch its executable symlink or erase its service plan and
+journals to bypass this guard. If a task stays local, inspect `rack policy show`,
+the task's runner/platforms and host-access dependencies; `--rack-only` cannot
+override eligibility.
+
+Public package entries are `@elaraai/e3-rack` (local hub/client/routing/runner
+APIs), `@elaraai/e3-rack/protocol` (wire types),
+`@elaraai/e3-rack/server` (shared control-plane mechanisms without local hub startup),
+`@elaraai/e3-rack/testing` (test agent and contracts), and
+`@elaraai/e3-rack/hub-main` (hub process entry point).
+
+Hosts adopt shared routes, dispatch, identity and store interfaces through the
+server entry; this avoids importing the local hub's filesystem/runner adapters.
+They implement `RackRegistry.compareAndSetRack`:
+replace exactly the observed registration, atomically, and refuse a missing or
+changed record. `updateRack` retries a patch against the latest value. This
+preserves revocation, boot approval and drain state across concurrent requests;
+a process-local coordinator alone cannot protect a distributed control plane.
+Run `rackRegistryContract` from the testing entry against each backend. The file
+registry retains its single-writer hub lock and heartbeat coalescing. This changes
+no persisted or wire schema.
+
+The shared routes retain a fixed 250 ms claim cadence by default. Cloud hosts
+can set `pollMaxMs` to back off an idle poll (×1.5 per empty check), and
+`policyRecheckMs` to reduce idle policy reads. For example, the cloud adapter
+uses a one-second maximum and five-second idle policy refresh. Every actual
+claim rereads registration and policy, including drain, revocation and boot
+approval; cached idle permission cannot authorize work. Cancellation watches
+keep their separate 250 ms cadence. Object reads, target signing and environment
+layer signing run at most eight operations concurrently. Output declarations
+are reserved first, and every started operation settles before the lease fence
+is released, including after a failure. Commits stay sequential for distributed
+backends whose transactions condition on the same lease row.
+
+This implementation speaks e3-cloud's v2 task/unit leases with per-object
+uploads. The protocol golden fixtures come from e3-cloud's rack protocol, and
+the cancellation-watch fixtures match the companion agent
+([e3-cloud #254](https://github.com/elaraai/e3-cloud/pull/254)). The test rack
+agent executes real e3 runners and exercises the transfer path, over TLS with a
+pinned certificate; those tests do not establish KVM or hardware acceptance for
+the deployed agent. A companion agent must bundle e3 at least as new as this
+release, and pin the hub's certificate; an older bundle falls back locally.
+
+See [the epic reconciliation](../../design/e3-rack-reconciliation.md) for the
+changes required by current cloud and workspace APIs, wire migration rules,
+and the remaining companion work. Custom environments, remote function calls,
+automatic reconnect after hub restart and embedded UI integration are deferred.

package/dist/src/client/client.spec.d.ts

@@ -0,0 +1,6 @@
+/**
+ * Copyright (c) 2025 Elara AI Pty Ltd
+ * Licensed under BSL 1.1. See LICENSE for details.
+ */
+export {};
+//# sourceMappingURL=client.spec.d.ts.map

package/dist/src/client/client.spec.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"client.spec.d.ts","sourceRoot":"","sources":["../../../src/client/client.spec.ts"],"names":[],"mappings":"AAAA;;;GAGG"}

package/dist/src/client/client.spec.js

@@ -0,0 +1,150 @@
+/**
+ * Copyright (c) 2025 Elara AI Pty Ltd
+ * Licensed under BSL 1.1. See LICENSE for details.
+ */
+import { afterEach, it } from 'node:test';
+import assert from 'node:assert/strict';
+import { mkdtemp, readFile, readdir, rm } from 'node:fs/promises';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { spawn } from 'node:child_process';
+import { setTimeout as delay } from 'node:timers/promises';
+import { Hono } from 'hono';
+import { some, variant } from '@elaraai/east';
+import { LocalStorage, repoInit, uuidv7 } from '@elaraai/e3-core';
+import { sendSuccess } from '../server/beast2.js';
+import { connectHub, RackHubOutdatedError, RackHubUnavailableError } from './connect.js';
+import { HubClient } from './hub-client.js';
+import { RackSession } from './session.js';
+import { closeSocketServer, serveOnSocket } from './local-http.js';
+import { EmptyType, HubHelloType } from '../protocol/control.js';
+import { HUB_PROTOCOL } from '../version.js';
+import { hubSocketPath } from '../paths.js';
+import { readHubLock } from '../hub/lock.js';
+import { TestRackAgent } from '../testing/test-rack-agent.js';
+const cleanup = [];
+afterEach(async () => { for (const close of cleanup.splice(0).reverse())
+    await close(); });
+async function home() {
+    const path = await mkdtemp(join(tmpdir(), 'e3c-'));
+    cleanup.push(() => rm(path, { recursive: true, force: true }));
+    cleanup.push(async () => {
+        await new HubClient(path).stop().catch(() => { });
+        for (let i = 0; i < 100 && await readHubLock(path) !== null; i++)
+            await delay(20);
+        const lock = await readHubLock(path);
+        if (lock !== null)
+            try {
+                process.kill(Number(lock.pid), 'SIGKILL');
+            }
+            catch { /* already gone */ }
+    });
+    return path;
+}
+async function fixture() {
+    const path = await home();
+    const client = await connectHub({ home: path });
+    await client.setConfig({ ...await client.getConfig(), listen: some({ host: '127.0.0.1', port: 0n }) });
+    const mint = await client.enroll();
+    const agent = await TestRackAgent.enroll(mint.apiUrl, mint.enrollmentToken, { certificateSha256: mint.certificateSha256.type === 'some' ? mint.certificateSha256.value : undefined });
+    cleanup.push(() => agent.close());
+    const repo = join(path, 'repo');
+    assert(repoInit(repo).success);
+    const storage = new LocalStorage();
+    const session = await RackSession.open(client, { storage, repoPath: repo, workspace: 'dev', label: 'client test' });
+    cleanup.push(() => session.close());
+    const request = { tier: 'node', computeSize: 'large', eventJson: JSON.stringify({ version: 2, workspace: 'dev', taskName: 'task', closure: { hashes: [] },
+            runnerEvent: { mode: 'task', taskHash: 'a'.repeat(64), repo: 'replaced', inputHashes: [], launchId: uuidv7(), timeoutMs: 30000 } }) };
+    return { path, client, agent, repo, storage, session, request };
+}
+it('cold-starts the published hub entry, logs startup and shares it across three processes', async () => {
+    const path = await home();
+    const module = new URL('./connect.js', import.meta.url).href;
+    const script = `import { connectHub } from ${JSON.stringify(module)};
+    const hub = await connectHub({ home: process.argv[1] });
+    console.log(Number((await hub.hello()).pid));`;
+    const run = () => new Promise((resolve, reject) => {
+        const child = spawn(process.execPath, ['--input-type=module', '-e', script, path], { stdio: ['ignore', 'pipe', 'pipe'], windowsHide: true });
+        let out = '';
+        let error = '';
+        child.stdout.on('data', (data) => { out += data.toString(); });
+        child.stderr.on('data', (data) => { error += data.toString(); });
+        child.once('error', reject);
+        child.once('exit', (code) => { if (code !== 0)
+            reject(new Error(error));
+        else
+            resolve(Number(out.trim())); });
+    });
+    const pids = await Promise.all([run(), run(), run()]);
+    assert.equal(new Set(pids).size, 1);
+    const client = await connectHub({ home: path, spawn: false });
+    assert.equal(Number((await client.status()).hello.pid), pids[0]);
+    assert.match(await readFile(join(path, 'hub.log'), 'utf8'), /rack hub started/);
+    assert.equal(Number((await readHubLock(path)).pid), pids[0]);
+});
+it('drains an older protocol, refuses a newer one and leaves its sessions alone', async () => {
+    const path = await home();
+    let protocol = BigInt(HUB_PROTOCOL) - 1n;
+    let drains = 0;
+    const app = new Hono();
+    app.post('/v1/hello', () => sendSuccess(HubHelloType, { version: '1.0.0', protocol, pid: BigInt(process.pid), draining: false }));
+    app.post('/v1/drain', () => { drains++; return sendSuccess(EmptyType, {}); });
+    const server = await serveOnSocket(app, hubSocketPath(path));
+    cleanup.push(() => closeSocketServer(server, hubSocketPath(path)));
+    await assert.rejects(connectHub({ home: path }), (err) => err instanceof RackHubOutdatedError && /older.*exit once idle/.test(err.message));
+    assert.equal(drains, 1);
+    protocol = BigInt(HUB_PROTOCOL) + 1n;
+    await assert.rejects(connectHub({ home: path }), (err) => err instanceof RackHubOutdatedError && /newer/.test(err.message));
+    assert.equal(drains, 1);
+});
+it('delivers claim, log tee and terminal outcome in order, then removes its session data socket', async () => {
+    const f = await fixture();
+    const handle = await f.session.createLease(f.request);
+    const grant = await f.agent.claim();
+    assert(grant);
+    const claimed = await handle.next();
+    assert.equal(claimed.kind, 'claimed');
+    assert(handle.attempt);
+    const logs = [];
+    f.session.onLog(handle.attempt.executionId, (_stream, data) => { logs.push(data); });
+    await f.agent.logs(grant, [{ stream: variant('stdout', null), data: 'first\n' }, { stream: variant('stdout', null), data: 'second\n' }]);
+    assert.deepEqual(logs, ['first\n', 'second\n']);
+    await f.agent.complete(grant, { taskName: 'task', status: 'failed', exitCode: 7, error: 'expected failure' });
+    const completed = await handle.next();
+    assert.equal(completed.kind, 'completed');
+    if (completed.kind === 'completed')
+        assert.equal(completed.result.exitCode, 7);
+    await f.session.close();
+    await f.session.close();
+    assert.equal((await f.client.status()).sessions.length, 0);
+    assert.equal((await readdir(f.path)).filter((name) => name.startsWith('s-') && name.endsWith('.sock')).length, 0);
+});
+it('unblocks every lease promptly after the real hub is killed and refuses reconnecting the old session', async () => {
+    const f = await fixture();
+    const handle = await f.session.createLease(f.request);
+    const hello = await f.client.hello();
+    const waiting = handle.next();
+    const started = Date.now();
+    process.kill(Number(hello.pid), 'SIGKILL');
+    const update = await waiting;
+    assert.equal(update.kind, 'lost');
+    assert(Date.now() - started < 3000);
+    assert.equal(f.session.hubLost, true);
+    await assert.rejects(f.session.createLease(f.request), RackHubUnavailableError);
+    const replacement = await connectHub({ home: f.path });
+    assert.notEqual(Number((await replacement.hello()).pid), Number(hello.pid));
+    assert.equal((await replacement.status()).sessions.length, 0);
+});
+it('keeps another handle subscribed when identical work is requested twice in one session', async () => {
+    const f = await fixture();
+    const [first, second] = await Promise.all([f.session.createLease(f.request), f.session.createLease(f.request)]);
+    assert.equal(first.leaseId, second.leaseId);
+    await first.cancel();
+    assert.equal((await first.next()).kind, 'cancelled');
+    const grant = await f.agent.claim();
+    assert(grant);
+    assert.equal((await second.next()).kind, 'claimed');
+    await f.agent.complete(grant, { taskName: 'task', status: 'failed' });
+    assert.equal((await second.next()).kind, 'completed');
+});
+//# sourceMappingURL=client.spec.js.map

package/dist/src/client/client.spec.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"client.spec.js","sourceRoot":"","sources":["../../../src/client/client.spec.ts"],"names":[],"mappings":"AAAA;;;GAGG;AAEH,OAAO,EAAE,SAAS,EAAE,EAAE,EAAE,MAAM,WAAW,CAAC;AAC1C,OAAO,MAAM,MAAM,oBAAoB,CAAC;AACxC,OAAO,EAAE,OAAO,EAAE,QAAQ,EAAE,OAAO,EAAE,EAAE,EAAE,MAAM,kBAAkB,CAAC;AAClE,OAAO,EAAE,MAAM,EAAE,MAAM,SAAS,CAAC;AACjC,OAAO,EAAE,IAAI,EAAE,MAAM,WAAW,CAAC;AACjC,OAAO,EAAE,KAAK,EAAE,MAAM,oBAAoB,CAAC;AAC3C,OAAO,EAAE,UAAU,IAAI,KAAK,EAAE,MAAM,sBAAsB,CAAC;AAC3D,OAAO,EAAE,IAAI,EAAE,MAAM,MAAM,CAAC;AAC5B,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,MAAM,eAAe,CAAC;AAC9C,OAAO,EAAE,YAAY,EAAE,QAAQ,EAAE,MAAM,EAAE,MAAM,kBAAkB,CAAC;AAClE,OAAO,EAAE,WAAW,EAAE,MAAM,qBAAqB,CAAC;AAClD,OAAO,EAAE,UAAU,EAAE,oBAAoB,EAAE,uBAAuB,EAAE,MAAM,cAAc,CAAC;AACzF,OAAO,EAAE,SAAS,EAAE,MAAM,iBAAiB,CAAC;AAC5C,OAAO,EAAE,WAAW,EAAE,MAAM,cAAc,CAAC;AAC3C,OAAO,EAAE,iBAAiB,EAAE,aAAa,EAAE,MAAM,iBAAiB,CAAC;AACnE,OAAO,EAAE,SAAS,EAAE,YAAY,EAAE,MAAM,wBAAwB,CAAC;AACjE,OAAO,EAAE,YAAY,EAAE,MAAM,eAAe,CAAC;AAC7C,OAAO,EAAE,aAAa,EAAE,MAAM,aAAa,CAAC;AAC5C,OAAO,EAAE,WAAW,EAAE,MAAM,gBAAgB,CAAC;AAC7C,OAAO,EAAE,aAAa,EAAE,MAAM,+BAA+B,CAAC;AAE9D,MAAM,OAAO,GAA+B,EAAE,CAAC;AAC/C,SAAS,CAAC,KAAK,IAAI,EAAE,GAAG,KAAK,MAAM,KAAK,IAAI,OAAO,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,OAAO,EAAE;IAAE,MAAM,KAAK,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC;AAC3F,KAAK,UAAU,IAAI;IACjB,MAAM,IAAI,GAAG,MAAM,OAAO,CAAC,IAAI,CAAC,MAAM,EAAE,EAAE,MAAM,CAAC,CAAC,CAAC;IACnD,OAAO,CAAC,IAAI,CAAC,GAAG,EAAE,CAAC,EAAE,CAAC,IAAI,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC;IAC/D,OAAO,CAAC,IAAI,CAAC,KAAK,IAAI,EAAE;QACtB,MAAM,IAAI,SAAS,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,CAAC,KAAK,CAAC,GAAG,EAAE,GAAE,CAAC,CAAC,CAAC;QACjD,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,GAAG,IAAI,MAAM,WAAW,CAAC,IAAI,CAAC,KAAK,IAAI,EAAE,CAAC,EAAE;YAAE,MAAM,KAAK,CAAC,EAAE,CAAC,CAAC;QAClF,MAAM,IAAI,GAAG,MAAM,WAAW,CAAC,IAAI,CAAC,CAAC;QACrC,IAAI,IAAI,KAAK,IAAI;YAAE,IAAI,CAAC;gBAAC,OAAO,CAAC,IAAI,CAAC,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,SAAS,CAAC,CAAC;YAAC,CAAC;YAAC,MAAM,CAAC,CAAC,kBAAkB,CAAC,CAAC;IACpG,CAAC,CAAC,CAAC;IACH,OAAO,IAAI,CAAC;AACd,CAAC;AACD,KAAK,UAAU,OAAO;IACpB,MAAM,IAAI,GAAG,MAAM,IAAI,EAAE,CAAC;IAC1B,MAAM,MAAM,GAAG,MAAM,UAAU,CAAC,EAAE,IAAI,EAAE,IAAI,EAAE,CAAC,CAAC;IAChD,MAAM,MAAM,CAAC,SAAS,CAAC,EAAE,GAAG,MAAM,MAAM,CAAC,SAAS,EAAE,EAAE,MAAM,EAAE,IAAI,CAAC,EAAE,IAAI,EAAE,WAAW,EAAE,IAAI,EAAE,EAAE,EAAE,CAAC,EAAE,CAAC,CAAC;IACvG,MAAM,IAAI,GAAG,MAAM,MAAM,CAAC,MAAM,EAAE,CAAC;IACnC,MAAM,KAAK,GAAG,MAAM,aAAa,CAAC,MAAM,CAAC,IAAI,CAAC,MAAM,EAAE,IAAI,CAAC,eAAe,EAAE,EAAE,iBAAiB,EAAE,IAAI,CAAC,iBAAiB,CAAC,IAAI,KAAK,MAAM,CAAC,CAAC,CAAC,IAAI,CAAC,iBAAiB,CAAC,KAAK,CAAC,CAAC,CAAC,SAAS,EAAE,CAAC,CAAC;IACtL,OAAO,CAAC,IAAI,CAAC,GAAG,EAAE,CAAC,KAAK,CAAC,KAAK,EAAE,CAAC,CAAC;IAClC,MAAM,IAAI,GAAG,IAAI,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC;IAAC,MAAM,CAAC,QAAQ,CAAC,IAAI,CAAC,CAAC,OAAO,CAAC,CAAC;IAChE,MAAM,OAAO,GAAG,IAAI,YAAY,EAAE,CAAC;IACnC,MAAM,OAAO,GAAG,MAAM,WAAW,CAAC,IAAI,CAAC,MAAM,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE,IAAI,EAAE,SAAS,EAAE,KAAK,EAAE,KAAK,EAAE,aAAa,EAAE,CAAC,CAAC;IACpH,OAAO,CAAC,IAAI,CAAC,GAAG,EAAE,CAAC,OAAO,CAAC,KAAK,EAAE,CAAC,CAAC;IACpC,MAAM,OAAO,GAAG,EAAE,IAAI,EAAE,MAAM,EAAE,WAAW,EAAE,OAAO,EAAE,SAAS,EAAE,IAAI,CAAC,SAAS,CAAC,EAAE,OAAO,EAAE,CAAC,EAAE,SAAS,EAAE,KAAK,EAAE,QAAQ,EAAE,MAAM,EAAE,OAAO,EAAE,EAAE,MAAM,EAAE,EAAE,EAAE;YACvJ,WAAW,EAAE,EAAE,IAAI,EAAE,MAAM,EAAE,QAAQ,EAAE,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC,EAAE,IAAI,EAAE,UAAU,EAAE,WAAW,EAAE,EAAE,EAAE,QAAQ,EAAE,MAAM,EAAE,EAAE,SAAS,EAAE,KAAK,EAAE,EAAE,CAAC,EAAE,CAAC;IACxI,OAAO,EAAE,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,IAAI,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,CAAC;AAClE,CAAC;AAED,EAAE,CAAC,wFAAwF,EAAE,KAAK,IAAI,EAAE;IACtG,MAAM,IAAI,GAAG,MAAM,IAAI,EAAE,CAAC;IAC1B,MAAM,MAAM,GAAG,IAAI,GAAG,CAAC,cAAc,EAAE,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,IAAI,CAAC;IAC7D,MAAM,MAAM,GAAG,8BAA8B,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC;;kDAEnB,CAAC;IACjD,MAAM,GAAG,GAAG,GAAG,EAAE,CAAC,IAAI,OAAO,CAAS,CAAC,OAAO,EAAE,MAAM,EAAE,EAAE;QACxD,MAAM,KAAK,GAAG,KAAK,CAAC,OAAO,CAAC,QAAQ,EAAE,CAAC,qBAAqB,EAAE,IAAI,EAAE,MAAM,EAAE,IAAI,CAAC,EAAE,EAAE,KAAK,EAAE,CAAC,QAAQ,EAAE,MAAM,EAAE,MAAM,CAAC,EAAE,WAAW,EAAE,IAAI,EAAE,CAAC,CAAC;QAC7I,IAAI,GAAG,GAAG,EAAE,CAAC;QAAC,IAAI,KAAK,GAAG,EAAE,CAAC;QAC7B,KAAK,CAAC,MAAM,CAAC,EAAE,CAAC,MAAM,EAAE,CAAC,IAAY,EAAE,EAAE,GAAG,GAAG,IAAI,IAAI,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC;QACvE,KAAK,CAAC,MAAM,CAAC,EAAE,CAAC,MAAM,EAAE,CAAC,IAAY,EAAE,EAAE,GAAG,KAAK,IAAI,IAAI,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC;QACzE,KAAK,CAAC,IAAI,CAAC,OAAO,EAAE,MAAM,CAAC,CAAC;QAC5B,KAAK,CAAC,IAAI,CAAC,MAAM,EAAE,CAAC,IAAI,EAAE,EAAE,GAAG,IAAI,IAAI,KAAK,CAAC;YAAE,MAAM,CAAC,IAAI,KAAK,CAAC,KAAK,CAAC,CAAC,CAAC;;YAAM,OAAO,CAAC,MAAM,CAAC,GAAG,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IAChH,CAAC,CAAC,CAAC;IACH,MAAM,IAAI,GAAG,MAAM,OAAO,CAAC,GAAG,CAAC,CAAC,GAAG,EAAE,EAAE,GAAG,EAAE,EAAE,GAAG,EAAE,CAAC,CAAC,CAAC;IACtD,MAAM,CAAC,KAAK,CAAC,IAAI,GAAG,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC;IACpC,MAAM,MAAM,GAAG,MAAM,UAAU,CAAC,EAAE,IAAI,EAAE,IAAI,EAAE,KAAK,EAAE,KAAK,EAAE,CAAC,CAAC;IAC9D,MAAM,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,MAAM,MAAM,CAAC,MAAM,EAAE,CAAC,CAAC,KAAK,CAAC,GAAG,CAAC,EAAE,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC;IACjE,MAAM,CAAC,KAAK,CAAC,MAAM,QAAQ,CAAC,IAAI,CAAC,IAAI,EAAE,SAAS,CAAC,EAAE,MAAM,CAAC,EAAE,kBAAkB,CAAC,CAAC;IAChF,MAAM,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,MAAM,WAAW,CAAC,IAAI,CAAC,CAAE,CAAC,GAAG,CAAC,EAAE,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC;AAChE,CAAC,CAAC,CAAC;AAEH,EAAE,CAAC,6EAA6E,EAAE,KAAK,IAAI,EAAE;IAC3F,MAAM,IAAI,GAAG,MAAM,IAAI,EAAE,CAAC;IAC1B,IAAI,QAAQ,GAAG,MAAM,CAAC,YAAY,CAAC,GAAG,EAAE,CAAC;IAAC,IAAI,MAAM,GAAG,CAAC,CAAC;IACzD,MAAM,GAAG,GAAG,IAAI,IAAI,EAAE,CAAC;IACvB,GAAG,CAAC,IAAI,CAAC,WAAW,EAAE,GAAG,EAAE,CAAC,WAAW,CAAC,YAAY,EAAE,EAAE,OAAO,EAAE,OAAO,EAAE,QAAQ,EAAE,GAAG,EAAE,MAAM,CAAC,OAAO,CAAC,GAAG,CAAC,EAAE,QAAQ,EAAE,KAAK,EAAE,CAAC,CAAC,CAAC;IAClI,GAAG,CAAC,IAAI,CAAC,WAAW,EAAE,GAAG,EAAE,GAAG,MAAM,EAAE,CAAC,CAAC,OAAO,WAAW,CAAC,SAAS,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IAC9E,MAAM,MAAM,GAAG,MAAM,aAAa,CAAC,GAAG,EAAE,aAAa,CAAC,IAAI,CAAC,CAAC,CAAC;IAC7D,OAAO,CAAC,IAAI,CAAC,GAAG,EAAE,CAAC,iBAAiB,CAAC,MAAM,EAAE,aAAa,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IACnE,MAAM,MAAM,CAAC,OAAO,CAAC,UAAU,CAAC,EAAE,IAAI,EAAE,IAAI,EAAE,CAAC,EAAE,CAAC,GAAY,EAAE,EAAE,CAAC,GAAG,YAAY,oBAAoB,IAAI,uBAAuB,CAAC,IAAI,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC;IACrJ,MAAM,CAAC,KAAK,CAAC,MAAM,EAAE,CAAC,CAAC,CAAC;IACxB,QAAQ,GAAG,MAAM,CAAC,YAAY,CAAC,GAAG,EAAE,CAAC;IACrC,MAAM,MAAM,CAAC,OAAO,CAAC,UAAU,CAAC,EAAE,IAAI,EAAE,IAAI,EAAE,CAAC,EAAE,CAAC,GAAY,EAAE,EAAE,CAAC,GAAG,YAAY,oBAAoB,IAAI,OAAO,CAAC,IAAI,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC;IACrI,MAAM,CAAC,KAAK,CAAC,MAAM,EAAE,CAAC,CAAC,CAAC;AAC1B,CAAC,CAAC,CAAC;AAEH,EAAE,CAAC,6FAA6F,EAAE,KAAK,IAAI,EAAE;IAC3G,MAAM,CAAC,GAAG,MAAM,OAAO,EAAE,CAAC;IAC1B,MAAM,MAAM,GAAG,MAAM,CAAC,CAAC,OAAO,CAAC,WAAW,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC;IACtD,MAAM,KAAK,GAAG,MAAM,CAAC,CAAC,KAAK,CAAC,KAAK,EAAE,CAAC;IAAC,MAAM,CAAC,KAAK,CAAC,CAAC;IACnD,MAAM,OAAO,GAAG,MAAM,MAAM,CAAC,IAAI,EAAE,CAAC;IAAC,MAAM,CAAC,KAAK,CAAC,OAAO,CAAC,IAAI,EAAE,SAAS,CAAC,CAAC;IAC3E,MAAM,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC;IACvB,MAAM,IAAI,GAAa,EAAE,CAAC;IAC1B,CAAC,CAAC,OAAO,CAAC,KAAK,CAAC,MAAM,CAAC,OAAO,CAAC,WAAW,EAAE,CAAC,OAAO,EAAE,IAAI,EAAE,EAAE,GAAG,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IACrF,MAAM,CAAC,CAAC,KAAK,CAAC,IAAI,CAAC,KAAK,EAAE,CAAC,EAAE,MAAM,EAAE,OAAO,CAAC,QAAQ,EAAE,IAAI,CAAC,EAAE,IAAI,EAAE,SAAS,EAAE,EAAE,EAAE,MAAM,EAAE,OAAO,CAAC,QAAQ,EAAE,IAAI,CAAC,EAAE,IAAI,EAAE,UAAU,EAAE,CAAC,CAAC,CAAC;IACzI,MAAM,CAAC,SAAS,CAAC,IAAI,EAAE,CAAC,SAAS,EAAE,UAAU,CAAC,CAAC,CAAC;IAChD,MAAM,CAAC,CAAC,KAAK,CAAC,QAAQ,CAAC,KAAK,EAAE,EAAE,QAAQ,EAAE,MAAM,EAAE,MAAM,EAAE,QAAQ,EAAE,QAAQ,EAAE,CAAC,EAAE,KAAK,EAAE,kBAAkB,EAAE,CAAC,CAAC;IAC9G,MAAM,SAAS,GAAG,MAAM,MAAM,CAAC,IAAI,EAAE,CAAC;IAAC,MAAM,CAAC,KAAK,CAAC,SAAS,CAAC,IAAI,EAAE,WAAW,CAAC,CAAC;IACjF,IAAI,SAAS,CAAC,IAAI,KAAK,WAAW;QAAE,MAAM,CAAC,KAAK,CAAC,SAAS,CAAC,MAAM,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC;IAC/E,MAAM,CAAC,CAAC,OAAO,CAAC,KAAK,EAAE,CAAC;IAAC,MAAM,CAAC,CAAC,OAAO,CAAC,KAAK,EAAE,CAAC;IACjD,MAAM,CAAC,KAAK,CAAC,CAAC,MAAM,CAAC,CAAC,MAAM,CAAC,MAAM,EAAE,CAAC,CAAC,QAAQ,CAAC,MAAM,EAAE,CAAC,CAAC,CAAC;IAC3D,MAAM,CAAC,KAAK,CAAC,CAAC,MAAM,OAAO,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,UAAU,CAAC,IAAI,CAAC,IAAI,IAAI,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,EAAE,CAAC,CAAC,CAAC;AACpH,CAAC,CAAC,CAAC;AAEH,EAAE,CAAC,qGAAqG,EAAE,KAAK,IAAI,EAAE;IACnH,MAAM,CAAC,GAAG,MAAM,OAAO,EAAE,CAAC;IAC1B,MAAM,MAAM,GAAG,MAAM,CAAC,CAAC,OAAO,CAAC,WAAW,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC;IACtD,MAAM,KAAK,GAAG,MAAM,CAAC,CAAC,MAAM,CAAC,KAAK,EAAE,CAAC;IACrC,MAAM,OAAO,GAAG,MAAM,CAAC,IAAI,EAAE,CAAC;IAC9B,MAAM,OAAO,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;IAC3B,OAAO,CAAC,IAAI,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,CAAC,EAAE,SAAS,CAAC,CAAC;IAC3C,MAAM,MAAM,GAAG,MAAM,OAAO,CAAC;IAC7B,MAAM,CAAC,KAAK,CAAC,MAAM,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC;IAAC,MAAM,CAAC,IAAI,CAAC,GAAG,EAAE,GAAG,OAAO,GAAG,IAAI,CAAC,CAAC;IACvE,MAAM,CAAC,KAAK,CAAC,CAAC,CAAC,OAAO,CAAC,OAAO,EAAE,IAAI,CAAC,CAAC;IACtC,MAAM,MAAM,CAAC,OAAO,CAAC,CAAC,CAAC,OAAO,CAAC,WAAW,CAAC,CAAC,CAAC,OAAO,CAAC,EAAE,uBAAuB,CAAC,CAAC;IAChF,MAAM,WAAW,GAAG,MAAM,UAAU,CAAC,EAAE,IAAI,EAAE,CAAC,CAAC,IAAI,EAAE,CAAC,CAAC;IACvD,MAAM,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAC,MAAM,WAAW,CAAC,KAAK,EAAE,CAAC,CAAC,GAAG,CAAC,EAAE,MAAM,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC;IAC5E,MAAM,CAAC,KAAK,CAAC,CAAC,MAAM,WAAW,CAAC,MAAM,EAAE,CAAC,CAAC,QAAQ,CAAC,MAAM,EAAE,CAAC,CAAC,CAAC;AAChE,CAAC,CAAC,CAAC;AAEH,EAAE,CAAC,uFAAuF,EAAE,KAAK,IAAI,EAAE;IACrG,MAAM,CAAC,GAAG,MAAM,OAAO,EAAE,CAAC;IAC1B,MAAM,CAAC,KAAK,EAAE,MAAM,CAAC,GAAG,MAAM,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,WAAW,CAAC,CAAC,CAAC,OAAO,CAAC,EAAE,CAAC,CAAC,OAAO,CAAC,WAAW,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC;IAChH,MAAM,CAAC,KAAK,CAAC,KAAK,CAAC,OAAO,EAAE,MAAM,CAAC,OAAO,CAAC,CAAC;IAC5C,MAAM,KAAK,CAAC,MAAM,EAAE,CAAC;IAAC,MAAM,CAAC,KAAK,CAAC,CAAC,MAAM,KAAK,CAAC,IAAI,EAAE,CAAC,CAAC,IAAI,EAAE,WAAW,CAAC,CAAC;IAC3E,MAAM,KAAK,GAAG,MAAM,CAAC,CAAC,KAAK,CAAC,KAAK,EAAE,CAAC;IAAC,MAAM,CAAC,KAAK,CAAC,CAAC;IACnD,MAAM,CAAC,KAAK,CAAC,CAAC,MAAM,MAAM,CAAC,IAAI,EAAE,CAAC,CAAC,IAAI,EAAE,SAAS,CAAC,CAAC;IACpD,MAAM,CAAC,CAAC,KAAK,CAAC,QAAQ,CAAC,KAAK,EAAE,EAAE,QAAQ,EAAE,MAAM,EAAE,MAAM,EAAE,QAAQ,EAAE,CAAC,CAAC;IACtE,MAAM,CAAC,KAAK,CAAC,CAAC,MAAM,MAAM,CAAC,IAAI,EAAE,CAAC,CAAC,IAAI,EAAE,WAAW,CAAC,CAAC;AACxD,CAAC,CAAC,CAAC"}