@simon-says-build/harbormaster 0.6.1 → 0.8.0

package/DESIGN.md

@@ -99,10 +99,38 @@ A lease is reclaimed when ANY of:
    daemon-booted suites and reserved (caller-booted) blocks. A single failed connect
    check on a loaded machine must never kill a healthy suite.
 
-Teardown always `killpg`s the whole process group so the emulator's **JVM children are
-reaped** — a v1 bug leaked Firestore/Storage JVMs because only the node parent was
-killed (verify this in v2). The daemon owns lifecycle end-to-end; callers use
-`acquire`/`release`, never `pkill`.
+Teardown frees the lease's **resources**, not just the process the daemon spawned —
+because the spawned process's group is provably not enough. v2 shipped assuming
+"`killpg` reaps the JVM children"; it doesn't: firebase-tools spawns its downloadable
+emulators and functions workers with `detached: true`, i.e. **their own process
+groups**, which `killpg` on the suite's group can structurally never reach. That
+false assumption leaked 51 Firestore JVMs and 284 functions workers on one machine
+before it was caught (2026-07-29). Teardown in v0.7 is therefore layered:
+
+1. `killpg` the recorded group (TERM → wait on **group** emptiness, not parent
+   death → KILL), guarded against PID reuse by the spawn-time start time.
+2. **Port sweep**: anything still LISTENing on any port of the lease's block is
+   killed (TERM → KILL). The block *is* the lease — this catches detached JVMs,
+   the caller's own Metro on the leased port, and anything else that squatted.
+3. Verify the block's ports are actually free, and log every action taken —
+   a teardown that leaves a listener behind says so instead of pretending.
+
+Beyond per-lease teardown, the daemon reconciles **both directions**: the ledger
+against the world (as before), and the world against the ledger — at startup and
+every `FBPORTS_SWEEP_INTERVAL` (300s) it evicts listeners on pool ports no lease
+covers, firebase-tools runtimes re-parented to pid 1 (the detached-children
+signature: `functionsEmulatorRuntime`, `cloud-firestore-emulator*.jar`, …), and
+`run-*`/`alloc-*` simulators with no lease. `harbormaster doctor [--fix]` runs the
+same sweep on demand. The daemon owns lifecycle end-to-end; callers use
+`acquire`/`release`, never `pkill` — and the sweep kills only by port ownership,
+pid-1 ancestry with our signatures, or our own sim naming scheme, never by name
+alone.
+
+The ledger itself is written atomically (temp + rename) and a corrupt ledger is
+quarantined loudly instead of silently zeroed — with the world sweep adopting
+whatever a lost ledger orphaned. Suites are registered in the ledger at spawn
+time (state `booting`), not after the readiness wait, so a daemon crash mid-boot
+can no longer strand a running suite outside the ledger.
 
 ## Owning PID is the caller, not the CLI
 
@@ -121,6 +149,7 @@ POST /heartbeat {leaseId, ttl?}                             -> {ok}          # e
 GET  /status                                               -> {leases:[...]}
 GET  /health
 POST /shutdown
+POST /doctor    {fix?}                                      -> {findings, fixed, leases}
 ```
 
 CLI: `acquire` / `up` (alias) / `down` (release your lease) / `release <id>` / `status`
@@ -141,6 +170,9 @@ eventarc=+8 database=+9` (kept in sync with the run driver's derivation).
 ## Open items before this goes live
 
 - Validate `acquire` boots + owner-death reap + PID-reuse guard on a quiet machine.
-- Confirm `killpg` reaps JVM children (fix the v1 leak if not).
+- ~~Confirm `killpg` reaps JVM children (fix the v1 leak if not).~~ Confirmed NOT
+  reaped (detached groups); fixed in v0.7 by the port sweep + world sweep, with
+  regression coverage in `test/e2e.sh` (release-path and crash-path both assert
+  the detached firestore JVM dies and the ports free).
 - Migrate the agent-qa run driver to pass `owner_pid` to `/allocate`.
 - Migrate the app.config / scrub-script / CLAUDE.md consumers to `acquire --env`.

package/README.md

@@ -1,10 +1,5 @@
 # harbormaster
 
-> Formerly `firebase-ports`. The scope outgrew the name: leases now cover Metro/Expo
-> ports and private iOS simulators too, not just Firebase emulators. The npm package
-> still installs a `firebase-ports` bin alias, and the daemon state dir is unchanged,
-> so existing consumers keep working.
-
 A localhost daemon that leases each **caller** its own isolated port block — a
 Firebase emulator sandbox with fresh data, a Metro port, a private simulator —
 and reclaims it when the caller's process exits.

package/harbormaster

@@ -1,19 +1,23 @@
 #!/usr/bin/env python3
 """harbormaster — caller-owned port leases: Firebase emulator sandboxes, Metro
-ports, private simulators. (Formerly firebase-ports.)
+ports, private simulators.
 
 A localhost daemon hands each CALLER its own emulator sandbox (a pool port block +
 fresh data) and reclaims it when the caller's process exits. No registry, no shared
 per-project instances — two callers that both want "an impulse emulator" get two
 independent suites, so they never collide on ports or data.
 
-  acquire [--cwd DIR] [--owner PID] [--only a,b] [--ttl N] [--metro] [--sim] [--json|--env]
+  acquire [--cwd DIR] [--owner PID] [--only a,b] [--ttl N] [--metro] [--sim]
+          [--fn-env KEY=VAL ...] [--json|--env]
         Boot a private sandbox for the project in DIR (default: nearest firebase.json
         above $PWD). Owned by --owner (default: the calling process's parent), reaped
         when that process exits. Prints the ports (use --env to eval into your shell).
         --metro also reserves a Metro/Expo port (METRO_PORT). --sim also creates+boots
         a private iOS simulator (DETOX_UDID/DETOX_SIM_NAME) and implies --metro; the
-        sim is deleted when the lease is reaped. Tune with --sim-device / --sim-runtime
+        sim is deleted when the lease is reaped. --fn-env KEY=VAL (repeatable)
+        injects env into the spawned emulator process — the functions runtime
+        inherits it (e.g. LLM_CACHE_DIR for the e2e LLM fixture cache).
+        Tune with --sim-device / --sim-runtime
         (or FBPORTS_SIM_DEVICE / FBPORTS_SIM_RUNTIME). This is the e2e dev-env: one
         lease = emulators + metro port + simulator, all torn down when the owner exits.
   up ...                 Alias for acquire (interactive dev; owner = your shell).
@@ -50,6 +54,11 @@ independent suites, so they never collide on ports or data.
         its process running, its simulator still present, and (for booted suites)
         its ports open. The reuse gate for long-lived sessions:
             harbormaster verify "$FBPORTS_LEASE_ID" || <re-lease>
+  doctor [--fix]         Everything alive in the world that the ledger doesn't
+        account for: pool-port squatters, orphaned firebase runtimes (detached
+        JVMs / functions workers re-parented to pid 1), stray run-*/alloc-* sims.
+        --fix evicts them; exits 1 when findings remain unfixed. The daemon runs
+        the same sweep itself at startup and every FBPORTS_SWEEP_INTERVAL (300s).
   serve                  Run the daemon (auto-starts on first command).
 
 NOTE: firebase.json ports are optional — the daemon injects (and overrides) a
@@ -71,7 +80,7 @@ import uuid
 from datetime import datetime, timezone
 from pathlib import Path
 
-VERSION = "0.6.1"
+VERSION = "0.8.0"
 
 # Overridable for tests / parallel installs; default is the shared user daemon.
 # The dir keeps the old "firebase-ports" name: pre-rename binaries still installed
@@ -80,7 +89,13 @@ STATE_DIR = Path(os.environ.get("FBPORTS_STATE_DIR",
                                 str(Path.home() / ".local" / "state" / "firebase-ports")))
 LEDGER = STATE_DIR / "ledger.json"
 BROKER_HOST, BROKER_PORT = "127.0.0.1", int(os.environ.get("FBPORTS_PORT", "4999"))
-POOL_BASE, BLOCK, POOL_BLOCKS = 11000, 20, 200          # blocks of 20 ports, 11000-14999
+# Blocks of 20 ports, 11000-14999 by default. Overridable so an isolated test
+# daemon (FBPORTS_PORT + FBPORTS_STATE_DIR) can also use a DISJOINT pool — its
+# world sweep evicts unledgered pool listeners, which on the shared pool would
+# mean evicting the real daemon's leases.
+POOL_BASE = int(os.environ.get("FBPORTS_POOL_BASE", "11000"))
+POOL_BLOCKS = int(os.environ.get("FBPORTS_POOL_BLOCKS", "200"))
+BLOCK = 20
 BLOCK_OFFSETS = {"auth": 0, "firestore": 1, "functions": 2, "storage": 3, "ui": 4,
                  "hub": 5, "logging": 6, "pubsub": 7, "eventarc": 8, "database": 9,
                  # Firestore's UI websocket is a SECOND port it needs (CLI default 9150).
@@ -100,6 +115,18 @@ REAP_INTERVAL = int(os.environ.get("FBPORTS_REAP_INTERVAL", "10"))
 # is healthy — sweep it whole rather than leave a half-alive session.
 SIM_CHECK_INTERVAL = int(os.environ.get("FBPORTS_SIM_CHECK_INTERVAL", "60"))
 BOOT_BACKSTOP = 600          # only for a genuinely wedged boot; readiness is liveness-driven
+# How often the daemon sweeps the WORLD for resources that escaped the ledger:
+# listeners on pool ports with no lease, orphaned firebase runtimes, stray sims.
+SWEEP_INTERVAL = int(os.environ.get("FBPORTS_SWEEP_INTERVAL", "300"))
+# Process-name signatures of firebase-tools' own children. firebase-tools spawns
+# these `detached: true` — their OWN process groups — so killpg on the suite can
+# structurally never reach them (the v1 "killpg fixes it" assumption was wrong:
+# it fixed the storage-rules JVM, which stays in-group, and missed the firestore
+# JVM and every functions worker). Once their node parent dies they re-parent to
+# pid 1; that ancestry is the sweep's orphan test.
+SWEEP_SIGNATURES = ("functionsEmulatorRuntime", "cloud-firestore-emulator",
+                    "cloud-storage-rules-runtime", "pubsub-emulator",
+                    "cloud-datastore-emulator", "firebase-database-emulator")
 
 
 def _now():
@@ -299,12 +326,91 @@ def _sim_udids():
 
 
 def _sim_teardown(udid):
-    """Shut down + delete a leased simulator (best effort) so devices never leak."""
+    """Shut down + delete a leased simulator. Returns True when the device is
+    confirmed gone. A failed delete is NOT fatal — the world sweep retries any
+    harbormaster-named device that has no lease — but it must be visible."""
     for args in (["shutdown", udid], ["delete", udid]):
         try:
             subprocess.run(["xcrun", "simctl", *args], capture_output=True, text=True, timeout=60)
         except (subprocess.SubprocessError, OSError):
             pass
+    udids = _sim_udids()
+    return udids is None or udid not in udids
+
+
+def _sim_inventory():
+    """{udid: name} for every simulator, or None when indeterminate."""
+    try:
+        r = subprocess.run(["xcrun", "simctl", "list", "devices", "-j"],
+                           capture_output=True, text=True, timeout=15)
+        if r.returncode != 0:
+            return None
+        devices = json.loads(r.stdout).get("devices", {})
+        return {d["udid"]: d.get("name", "")
+                for lst in devices.values() for d in lst if d.get("udid")}
+    except (subprocess.SubprocessError, OSError, json.JSONDecodeError,
+            AttributeError, TypeError):
+        return None
+
+
+def _group_alive(pgid):
+    """Any member of the process group left? EPERM counts as alive."""
+    try:
+        os.killpg(int(pgid), 0)
+        return True
+    except PermissionError:
+        return True
+    except (OSError, ProcessLookupError, ValueError, TypeError):
+        return False
+
+
+def _listeners():
+    """{port: {pids}} for this user's TCP LISTEN sockets — ONE lsof call, so
+    teardown and the world sweep can check whole port ranges cheaply."""
+    out = {}
+    try:
+        r = subprocess.run(["lsof", "-nP", "-iTCP", "-sTCP:LISTEN", "-Fpn"],
+                           capture_output=True, text=True, timeout=15)
+        pid = None
+        for line in r.stdout.splitlines():
+            if line[:1] == "p":
+                try:
+                    pid = int(line[1:])
+                except ValueError:
+                    pid = None
+            elif line[:1] == "n" and pid:
+                try:
+                    port = int(line.rsplit(":", 1)[1])
+                except (ValueError, IndexError):
+                    continue
+                out.setdefault(port, set()).add(pid)
+    except (subprocess.SubprocessError, OSError):
+        pass
+    return out
+
+
+def _kill_wave(pids, wait=5.0):
+    """SIGTERM the pids, wait, SIGKILL survivors. Returns the pids signalled.
+    Never signals ourselves or anything system-side (pid <= 1)."""
+    victims = sorted({int(p) for p in pids} - {os.getpid()})
+    victims = [p for p in victims if p > 1]
+    if not victims:
+        return []
+    for p in victims:
+        try:
+            os.kill(p, signal.SIGTERM)
+        except (OSError, ProcessLookupError):
+            pass
+    deadline = time.time() + wait
+    while time.time() < deadline and any(_pid_alive(p) for p in victims):
+        time.sleep(0.1)
+    for p in victims:
+        if _pid_alive(p):
+            try:
+                os.kill(p, signal.SIGKILL)
+            except (OSError, ProcessLookupError):
+                pass
+    return victims
 
 
 # ---- the daemon ----------------------------------------------------------
@@ -315,17 +421,33 @@ class Broker:
         self.leases = {}                  # leaseId -> caller-owned lease
         self._load()
         self._reconcile()
+        # _reconcile just swept; without this the reaper's throttle (epoch 0)
+        # would fire a second "periodic" sweep on its first tick, making
+        # FBPORTS_SWEEP_INTERVAL meaningless for the first interval.
+        self._swept = time.time()
 
     def _load(self):
         if LEDGER.exists():
             try:
                 self.leases = json.loads(LEDGER.read_text()).get("leases", {})
-            except (json.JSONDecodeError, OSError):
-                pass
+            except (json.JSONDecodeError, OSError) as e:
+                # A silently-dropped ledger orphans every live suite/sim/port at
+                # once. Quarantine it loudly; the world sweep adopts the fallout.
+                quarantine = LEDGER.with_suffix(f".corrupt-{int(time.time())}")
+                try:
+                    LEDGER.rename(quarantine)
+                except OSError:
+                    pass
+                print(f"ledger unreadable ({e}); quarantined to {quarantine} — "
+                      f"world sweep will reap anything it was tracking", flush=True)
 
     def _save(self):
         STATE_DIR.mkdir(parents=True, exist_ok=True)
-        LEDGER.write_text(json.dumps({"leases": self.leases}, indent=2))
+        # Atomic: a crash mid-write must never leave a half-written ledger
+        # (see _load — a corrupt ledger orphans everything at once).
+        tmp = LEDGER.with_suffix(".tmp")
+        tmp.write_text(json.dumps({"leases": self.leases}, indent=2))
+        tmp.replace(LEDGER)
 
     def _reconcile(self):
         for lid in list(self.leases):
@@ -334,6 +456,10 @@ class Broker:
                 print(f"reaping {lid}: {reason}", flush=True)
                 self._cleanup(self.leases.pop(lid))
         self._save()
+        # Ledger-vs-world is only half the job: anything that escaped the
+        # ledger (daemon crash mid-boot, corrupt ledger, detached children of
+        # past suites) is invisible to the loop above and immortal without this.
+        self._sweep_world(apply=True)
 
     def _dead(self, l, now):
         return bool(self._dead_reason(l, now))
@@ -375,6 +501,15 @@ class Broker:
             if l.get("down_since") and now - l["down_since"] >= RECLAIM_GRACE:
                 return "reserved block down past grace"
             return None
+        if l.get("state") == "booting":
+            # Provisional record written at spawn time so a daemon crash can't
+            # orphan a mid-boot suite. Ports aren't expected open yet — only a
+            # dead process or a blown boot window reaps it.
+            if not _child_alive(l.get("pid")):
+                return f"suite pid {l.get('pid')} died during boot"
+            if now - l.get("booting_since", now) > BOOT_BACKSTOP + 60:
+                return "boot window exceeded"
+            return None
         # Booted suite: the process is the strong liveness signal — reap a dead
         # process immediately. Ports are a weak signal (a single 0.5s connect can
         # fail on a loaded machine while the suite is fine), so closed ports only
@@ -398,7 +533,7 @@ class Broker:
                 return base
         raise RuntimeError("emulator port pool exhausted")
 
-    def _boot(self, cwd, block, only, project_id, label):
+    def _boot(self, cwd, block, only, project_id, label, on_spawn=None, extra_env=None):
         all_ports = _block_ports(block)
         tmp = _write_injected_config(cwd, all_ports)
         # Readiness/liveness track ONLY the data services actually started (respect
@@ -413,8 +548,20 @@ class Broker:
             cmd += ["--only", ",".join(only)]
         if project_id:
             cmd += ["--project", project_id]
+        # Caller-supplied env (acquire --fn-env KEY=VAL) rides on top of the
+        # daemon's env — the functions emulator runtime inherits it, which is
+        # the only way a CALLER can set per-suite env (e.g. LLM_CACHE_DIR)
+        # when the daemon, not the caller, spawns the emulator.
+        spawn_env = None
+        if extra_env:
+            spawn_env = {**os.environ, **{str(k): str(v) for k, v in dict(extra_env).items()}}
         with open(log, "w") as lf:
-            proc = subprocess.Popen(cmd, cwd=cwd, stdout=lf, stderr=subprocess.STDOUT, start_new_session=True)
+            proc = subprocess.Popen(cmd, cwd=cwd, stdout=lf, stderr=subprocess.STDOUT, start_new_session=True, env=spawn_env)
+        if on_spawn:
+            # Register the suite BEFORE the (up to 600s) readiness wait: a
+            # daemon crash inside this window used to leave a fully-booted,
+            # session-detached, ledger-less suite that nothing could ever find.
+            on_spawn(proc.pid, tmp, str(log))
         # Liveness-driven: keep waiting while the process is alive (a slow JVM is still
         # booting, not stuck); succeed when the ports open; fail if it exits; the
         # backstop only guards a genuinely wedged boot.
@@ -433,36 +580,121 @@ class Broker:
             time.sleep(1)
 
     def _cleanup(self, rec):
-        """Coordinated teardown — killpg the whole group so JVM children die too. Never pkill."""
+        """Teardown frees the lease's RESOURCES — process group, ports, sim —
+        and verifies, loudly. Killing only the process we spawned is not
+        enough: firebase-tools detaches its JVMs and functions workers into
+        their own groups (killpg can't reach them), and the caller's Metro on
+        the leased port was never ours at all. The port block IS the lease, so
+        after the group kill anything still LISTENing on the block dies too,
+        and the block is re-checked before we forget the lease. Never pkill —
+        every kill here is by recorded pid or by a port the lease owns."""
+        lid = rec.get("leaseId") or "?"
+        actions = []
         pid = rec.get("pid")
-        # _child_alive, not _pid_alive: a daemon-spawned process that already
-        # exited lingers as our zombie child, which passes kill-0 — treating it
-        # as alive would stall this loop through a futile TERM+KILL cycle.
-        if _child_alive(pid):
+        # PID-reuse guard (same contract as owners): if the pid's start time no
+        # longer matches what we recorded at spawn, it's a recycled pid — do
+        # not signal a stranger. Old ledger records lack pid_start; for those
+        # the group kill proceeds as before.
+        recorded_start = rec.get("pid_start")
+        if pid and recorded_start:
+            observed = _proc_start(pid)
+            if observed and observed != recorded_start:
+                actions.append(f"pid {pid} recycled — not signalled")
+                pid = None
+        pgid = rec.get("pgid")
+        if pgid is None and pid and _pid_alive(pid):
+            try:
+                pgid = os.getpgid(int(pid))
+            except (OSError, ProcessLookupError):
+                pgid = None
+        def reap_zombie():
+            # Our exited child lingers as a zombie until waited on — and a
+            # zombie still counts as a group member, so _group_alive would
+            # read "survived" forever without this.
+            if pid:
+                try:
+                    os.waitpid(int(pid), os.WNOHANG)
+                except (ChildProcessError, OSError, ValueError):
+                    pass
+
+        if pgid and int(pgid) > 1:
+            # Escalation waits on the GROUP, not the parent: the old loop broke
+            # the moment node exited on SIGTERM, so SIGKILL never reached slow
+            # or stubborn group members.
             for sig in (signal.SIGTERM, signal.SIGKILL):
+                reap_zombie()
+                if not _group_alive(pgid):
+                    break
                 try:
-                    os.killpg(os.getpgid(int(pid)), sig)
+                    os.killpg(int(pgid), sig)
                 except (OSError, ProcessLookupError):
                     break
                 for _ in range(50):
-                    if not _child_alive(pid):
+                    reap_zombie()
+                    if not _group_alive(pgid):
                         break
                     time.sleep(0.1)
-                if not _child_alive(pid):
-                    break
+            reap_zombie()
+            actions.append(f"group {pgid} " +
+                           ("SURVIVED kill" if _group_alive(pgid) else "killed"))
+        else:
+            reap_zombie()
+        # Port sweep: every port the lease covers. rec["ports"] catches legacy
+        # records; the block range catches everything else (metro, websocket).
+        ports = {v for v in (rec.get("ports") or {}).values() if isinstance(v, int)}
+        if rec.get("block"):
+            ports |= {rec["block"] + off for off in range(BLOCK)}
+        if ports:
+            listening = _listeners()
+            squatters = {p for port in ports for p in listening.get(port, ())}
+            killed = _kill_wave(squatters)
+            if killed:
+                actions.append(f"port sweep killed {killed}")
+            still = sorted(port for port, pids in _listeners().items()
+                           if port in ports and pids)
+            if still:
+                actions.append(f"ports STILL HELD after sweep: {still}")
         tmp = rec.get("tmp_config")
         if tmp and rec.get("cwd") and (Path(rec["cwd"]) / tmp).exists():
             (Path(rec["cwd"]) / tmp).unlink()
         if rec.get("sim_udid"):
-            _sim_teardown(rec["sim_udid"])
+            gone = _sim_teardown(rec["sim_udid"])
+            actions.append(f"sim {rec['sim_udid'][:8]} " +
+                           ("deleted" if gone else "DELETE FAILED (sweep will retry)"))
+        if actions:
+            print(f"teardown {lid}: " + "; ".join(actions), flush=True)
 
     def acquire(self, cwd, owner_pid, owner_start=None, only=None, ttl=DEFAULT_TTL,
-                metro=False, sim=False, sim_device=None, sim_runtime=None):
+                metro=False, sim=False, sim_device=None, sim_runtime=None, env=None):
         with self.lock:
             base = self._free_block()
             lid = f"run:{uuid.uuid4().hex[:12]}"
             project_id = _firebaserc_project(cwd)
-            pid, tmp, log, ports, check = self._boot(cwd, base, only, project_id, lid.replace(":", "-"))
+
+            def on_spawn(spawn_pid, spawn_tmp, spawn_log):
+                # Provisional record, saved before the readiness wait: a daemon
+                # crash mid-boot must leave a findable lease, not a ghost suite.
+                self.leases[lid] = {"leaseId": lid, "kind": "acquired",
+                                    "state": "booting", "booting_since": time.time(),
+                                    "cwd": cwd, "block": base, "ports": {},
+                                    "pid": spawn_pid, "pgid": spawn_pid,
+                                    "pid_start": _proc_start(spawn_pid),
+                                    "owner_pid": owner_pid, "owner_start": owner_start,
+                                    "tmp_config": spawn_tmp, "log": spawn_log,
+                                    "project_id": project_id,
+                                    "started": _now(), "expires": time.time() + ttl}
+                self._save()
+
+            try:
+                pid, tmp, log, ports, check = self._boot(
+                    cwd, base, only, project_id, lid.replace(":", "-"), on_spawn,
+                    extra_env=env)
+            except Exception:
+                rec = self.leases.pop(lid, None)
+                if rec:
+                    self._cleanup(rec)
+                    self._save()
+                raise
             # A leased simulator implies a metro port (the e2e dev-env wants both).
             # The metro port lives in the lease's own 20-port block, so it's
             # collision-free by construction and needs no health check.
@@ -476,10 +708,13 @@ class Broker:
                                                 sim_runtime or DEFAULT_SIM_RUNTIME, sim_name)
                 except Exception:
                     # Don't leak the emulators we just booted if the sim fails.
-                    self._cleanup({"pid": pid, "cwd": cwd, "tmp_config": tmp})
+                    self._cleanup(self.leases.pop(lid, None) or
+                                  {"pid": pid, "cwd": cwd, "tmp_config": tmp, "block": base})
+                    self._save()
                     raise
             self.leases[lid] = {"leaseId": lid, "kind": "acquired", "cwd": cwd, "block": base,
                                 "ports": ports, "check_ports": check, "pid": pid,
+                                "pgid": pid, "pid_start": _proc_start(pid),
                                 "owner_pid": owner_pid, "owner_start": owner_start,
                                 "tmp_config": tmp, "log": log, "project_id": project_id,
                                 "sim_udid": sim_udid, "sim_name": sim_name,
@@ -541,6 +776,7 @@ class Broker:
                   f"{f' spawned pid {pid}' if pid else ''}", flush=True)
             self.leases[lid] = {"leaseId": lid, "kind": "reserved", "cwd": spawn_cwd,
                                 "block": base, "ports": ports, "pid": pid,
+                                "pgid": pid, "pid_start": _proc_start(pid) if pid else None,
                                 "owner_pid": owner_pid,
                                 "owner_start": owner_start, "tmp_config": None,
                                 "log": log, "spawn": spawn,
@@ -571,6 +807,7 @@ class Broker:
             for lid in list(self.leases):
                 l = self.leases[lid]
                 if l.get("owner_pid") == owner_pid and (cwd is None or l.get("cwd") == cwd):
+                    print(f"released {lid} (owner {owner_pid} asked)", flush=True)
                     self._cleanup(self.leases.pop(lid))
                     n += 1
             self._save()
@@ -597,10 +834,17 @@ class Broker:
         with self.lock:
             now = time.time()
             for lid in list(self.leases):
-                if self._dead(self.leases[lid], now):
+                reason = self._dead_reason(self.leases[lid], now)
+                if reason:
+                    # The routine reap path used to be silent — weeks of leaked
+                    # JVMs left zero trace. Every reap states its reason.
+                    print(f"reaping {lid}: {reason}", flush=True)
                     self._cleanup(self.leases.pop(lid))
             self._reap_deleted_sims(now)
             self._save()
+            if now - getattr(self, "_swept", 0) >= SWEEP_INTERVAL:
+                self._swept = now
+                self._sweep_world(apply=True)
 
     def _reap_deleted_sims(self, now):
         """Reap leases whose simulator was deleted externally (Xcode cleanup,
@@ -649,6 +893,80 @@ class Broker:
                     reasons.append("ports down: " + ",".join(sorted(down)))
             return {"ok": not reasons, "reasons": reasons}
 
+    def _sweep_world(self, apply):
+        """The other half of reconciliation: find resources that exist in the
+        WORLD but not in the ledger, and (apply=True) evict them. Three classes,
+        each one an observed leak:
+
+        - listeners on pool ports no lease covers (caller-run Metros, suites
+          that outlived a lost ledger entry);
+        - firebase-tools runtimes re-parented to pid 1 (detached JVMs and
+          functions workers whose suite died — killpg can never reach these);
+        - harbormaster-named simulators (run-*/alloc-*) with no lease.
+
+        Scoped tightly on purpose: only pool ports, only this user's processes,
+        only pid-1 ancestry with firebase-tools signatures, only our own sim
+        naming scheme. Returns findings for `doctor`."""
+        findings = []
+        lease_pids = {int(l["pid"]) for l in self.leases.values() if l.get("pid")}
+        leased_ports = set()
+        for l in self.leases.values():
+            if l.get("block"):
+                leased_ports |= {l["block"] + off for off in range(BLOCK)}
+        pool_end = POOL_BASE + POOL_BLOCKS * BLOCK
+        for port, pids in _listeners().items():
+            if POOL_BASE <= port < pool_end and port not in leased_ports:
+                for pid in pids:
+                    if pid not in lease_pids:
+                        findings.append({"kind": "listener", "pid": pid, "port": port,
+                                         "cmd": _proc_field(pid, "command=")[:200]})
+        try:
+            r = subprocess.run(["ps", "-xo", "pid=,ppid=,command="],
+                               capture_output=True, text=True, timeout=10)
+            for line in r.stdout.splitlines():
+                parts = line.split(None, 2)
+                if len(parts) < 3 or parts[1] != "1":
+                    continue
+                pid, cmdline = int(parts[0]), parts[2]
+                if pid in lease_pids or pid == os.getpid():
+                    continue
+                if (any(sig in cmdline for sig in SWEEP_SIGNATURES)
+                        or ".firebase.brokered-" in cmdline):
+                    findings.append({"kind": "orphan", "pid": pid,
+                                     "cmd": cmdline[:200]})
+        except (subprocess.SubprocessError, OSError, ValueError):
+            pass
+        inv = _sim_inventory()
+        if inv is not None:
+            leased_udids = {l.get("sim_udid") for l in self.leases.values()}
+            for udid, name in inv.items():
+                if ((name.startswith("run-") or name.startswith("alloc-"))
+                        and udid not in leased_udids):
+                    findings.append({"kind": "sim", "udid": udid, "name": name})
+        if apply and findings:
+            pids = {f["pid"] for f in findings if f["kind"] in ("listener", "orphan")}
+            if pids:
+                killed = _kill_wave(pids)
+                print(f"world sweep killed {killed}", flush=True)
+            for f in findings:
+                if f["kind"] == "sim":
+                    gone = _sim_teardown(f["udid"])
+                    print(f"world sweep sim {f['name']} ({f['udid'][:8]}): "
+                          + ("deleted" if gone else "DELETE FAILED"), flush=True)
+                else:
+                    print(f"world sweep {f['kind']}: pid {f['pid']}"
+                          + (f" port {f['port']}" if f.get("port") else "")
+                          + f" — {f['cmd']}", flush=True)
+        return findings
+
+    def doctor(self, fix=False):
+        """Report (and with fix=True, evict) everything the sweep can see —
+        the command that would have made 51 orphaned JVMs visible on day one."""
+        with self.lock:
+            findings = self._sweep_world(apply=fix)
+            return {"findings": findings, "fixed": bool(fix and findings),
+                    "leases": len(self.leases)}
+
 
 class _Handler(http.server.BaseHTTPRequestHandler):
     broker = None
@@ -685,7 +1003,8 @@ class _Handler(http.server.BaseHTTPRequestHandler):
                 return self._send(200, b.acquire(d["cwd"], d.get("owner_pid"), d.get("owner_start"),
                                                  d.get("only"), d.get("ttl", DEFAULT_TTL),
                                                  d.get("metro", False), d.get("sim", False),
-                                                 d.get("sim_device"), d.get("sim_runtime")))
+                                                 d.get("sim_device"), d.get("sim_runtime"),
+                                                 d.get("env")))
             if self.path == "/allocate":
                 return self._send(200, b.allocate(d.get("owner_pid"), d.get("owner_start"),
                                                   d.get("ttl", DEFAULT_TTL), d.get("label"),
@@ -700,6 +1019,8 @@ class _Handler(http.server.BaseHTTPRequestHandler):
                 return self._send(200, b.heartbeat(d.get("leaseId"), d.get("ttl", DEFAULT_TTL)))
             if self.path == "/verify":
                 return self._send(200, b.verify(d.get("leaseId")))
+            if self.path == "/doctor":
+                return self._send(200, b.doctor(d.get("fix", False)))
             if self.path == "/shutdown":
                 threading.Thread(target=self.server.shutdown, daemon=True).start()
                 return self._send(200, {"ok": True})
@@ -844,6 +1165,11 @@ def _print(out, fmt):
         if sim:
             print(f"export DETOX_UDID={sim['udid']}")          # Detox targets this simulator by UDID
             print(f"export DETOX_SIM_NAME={sim['name']}")
+        # The lease id, so a long-running caller can `harbormaster release` a
+        # suite it has decided is dead and re-acquire, instead of leaking it
+        # until the reaper notices.
+        if out.get("leaseId"):
+            print(f"export FBPORTS_LEASE_ID={out['leaseId']}")
     else:
         print(f"{out.get('leaseId', '')}  ({pid})".strip())
         for svc, port in ports.items():
@@ -860,7 +1186,13 @@ def _parse(args):
         if a.startswith("--"):
             if "=" in a:
                 k, v = a[2:].split("=", 1)
-                opts[k] = v
+                if k == "fn-env":
+                    opts.setdefault("fn-env", []).append(v)
+                else:
+                    opts[k] = v
+            elif a[2:] == "fn-env" and i + 1 < len(args):
+                opts.setdefault("fn-env", []).append(args[i + 1])
+                i += 1
             elif a[2:] in ("cwd", "only", "ttl", "owner", "label", "sim-device", "sim-runtime", "spawn") and i + 1 < len(args):
                 opts[a[2:]] = args[i + 1]
                 i += 1
@@ -900,7 +1232,8 @@ def main():
                                              "ttl": int(opts.get("ttl", DEFAULT_TTL)),
                                              "metro": "metro" in flags, "sim": "sim" in flags,
                                              "sim_device": opts.get("sim-device"),
-                                             "sim_runtime": opts.get("sim-runtime")}), "acquire"), fmt)
+                                             "sim_runtime": opts.get("sim-runtime"),
+                                             "env": dict(kv.split("=", 1) for kv in opts.get("fn-env", []))}), "acquire"), fmt)
     elif cmd == "down":
         if _up():
             out = _call("POST", "/release-owned", {"owner_pid": os.getppid(), "cwd": _resolve_cwd(opts)})[1]
@@ -995,6 +1328,27 @@ def main():
         else:
             print("; ".join(out.get("reasons") or [out.get("error", "not ok")]))
             sys.exit(1)
+    elif cmd == "doctor":
+        # doctor [--fix] — everything alive in the WORLD that the ledger does
+        # not account for: pool-port squatters, orphaned firebase runtimes,
+        # stray harbormaster sims. --fix evicts them. Exit 1 when findings
+        # remain unfixed, so health checks can gate on it.
+        _ensure()
+        out = ok(_call("POST", "/doctor", {"fix": "fix" in flags}), "doctor")
+        findings = out.get("findings", [])
+        if not findings:
+            print(f"clean — {out.get('leases', 0)} lease(s), no strays.")
+            return
+        for f in findings:
+            if f["kind"] == "sim":
+                print(f"  stray sim    {f['name']}  {f['udid']}")
+            else:
+                where = f" port {f['port']}" if f.get("port") else ""
+                print(f"  {f['kind']:<12} pid {f['pid']}{where}  {f['cmd']}")
+        print(f"{len(findings)} finding(s) " +
+              ("evicted." if out.get("fixed") else "— run `harbormaster doctor --fix` to evict."))
+        if not out.get("fixed"):
+            sys.exit(1)
     elif cmd in ("status", "leases"):
         if not _up():
             print("daemon not running.")
@@ -1016,7 +1370,7 @@ def main():
                   f"{l.get('label') or ''}  {l.get('cwd') or ''}".rstrip())
     else:
         print(f"Unknown command: {cmd}", file=sys.stderr)
-        print("Commands: acquire, up, down, release, allocate, verify, status, leases, serve", file=sys.stderr)
+        print("Commands: acquire, up, down, release, allocate, verify, doctor, status, leases, serve", file=sys.stderr)
         sys.exit(1)
 
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@simon-says-build/harbormaster",
-  "version": "0.6.1",
+  "version": "0.8.0",
   "description": "A localhost daemon that leases each caller its own port block — isolated Firebase emulator sandboxes, Metro/Expo ports, private iOS simulators — reaped when the caller exits. No port collisions, no shared instances, no pkill.",
   "bin": {
     "harbormaster": "harbormaster",