pi-proxy-guard 0.6.0 → 0.6.1

package/README.md

@@ -4,7 +4,7 @@ Auto-recover [pi](https://github.com/earendil-works/pi) sessions when the proxy
 
 When all provider retries fail (`Retry failed after N attempts` / `Stream ended without finish_reason`), pi pauses the session. This extension:
 
-1. **Checks connectivity** (`curl https://www.google.com/generate_204`, inheriting the same `https_proxy` env pi uses).
+1. **Checks connectivity**, two tiers: the watchdog uses a cheap ping (`generate_204` through the same `https_proxy` env pi uses); at settle/recheck time it does a **deep probe** — API host reachability (`ctx.model.baseUrl`, any HTTP response counts) **plus a sustained-transfer check** (`STREAM_MIN_BYTES` from `STREAM_URL`) — the signal a 0-byte ping can't see: whether the chain can actually hold a streaming connection. 'Down' requires `PROBE_ATTEMPTS` consecutive failures.
 2. **Auto-continues** the run if the proxy is fine (transient stream cut).
 3. **Repairs the proxy** if it's down — runs a macOS Shortcuts shortcut, falls back to `shadowrocket://` URL schemes — then continues once connectivity is verified.
 4. **Watchdog**: if pi is still paused, it keeps watching; once the proxy is verified healthy again it sends `continue` itself. It also repairs the proxy while pi is mid-retry, so many incidents never reach the pause at all.
@@ -38,18 +38,27 @@ pi install ~/Developer/pi-proxy-guard
 
 Restart pi. `/proxyguard check|restart|status` verifies it's live; logs go to `~/.pi/agent/proxy-guard.log`.
 
-## Setup: the Shortcuts shortcut
+## Setup: repairing Shadowrocket on macOS
 
-Default name: **Reconnect Shadowrocket** (set `PI_PROXY_GUARD_SHORTCUT` to rename). Build it in macOS Shortcuts — the user already has `Disconnect VPN` / `Connect VPN` shortcuts whose actions can be reused:
+Zero-setup (verified on the Apple-Silicon iOS-app runtime): the extension runs
 
-1. `Disconnect VPN` (or Shadowrocket "Stop" action)
-2. Wait 2s
-3. `Connect VPN` (or Shadowrocket "Start" action)
-4. Wait 5s
+```sh
+open -g "shadowrocket://disconnect?autoclose=true"   # tunnel drops
+sleep 3
+open -g "shadowrocket://connect?autoclose=true"     # fresh tunnel
+```
+
+`-g` keeps Shadowrocket in the background; `autoclose=true` lets it quit itself. If you have **always-on** enabled, the tunnel may auto-reconnect before the explicit connect — either way you end up on a fresh tunnel, which is the point (the dead stream is already dead; repair makes the *next* request land on a rebuilt chain).
+
+### Shortcut path (optional)
+
+`PI_PROXY_GUARD_SHORTCUT` (default name **Reconnect Shadowrocket**) runs `shortcuts run <name>` first, scheme fallback after. **Verified caveat**: on macOS, `shortcuts run` does *not* fire iOS-app SiriKit intents — a shortcut wrapping Shadowrocket's `StopVPNIntent`/`StartVPNIntent` is a silent no-op via CLI (it only fires from the Shortcuts.app GUI). So on macOS the shortcut path only helps if it wraps shell-able actions (e.g. `Run Shell Script` driving another client). Set `PI_PROXY_GUARD_SHORTCUT=""` to skip it entirely.
+
+Node switching is intentionally left to the proxy client (chain-proxy friendly). To make the *repair* more effective, you can chain actions into the URL scheme too — e.g. `shadowrocket://select?s=<node>` before `connect` selects a different entry node.
 
-Or, zero-setup: it falls back to `open shadowrocket://stop` → `shadowrocket://start` automatically (works when Shadowrocket registers its URL scheme — on macOS this needs the Apple Silicon iOS app).
+### Verification note
 
-Node switching is intentionally left to the proxy client (chain-proxy friendly). To make the *repair* more effective, build the node switch into the shortcut itself — e.g. select a different entry node inside Shadowrocket before reconnecting.
+`scutil --nc show/list` reports the NE manager's *intent* state and lags reality by several seconds — during a real disconnect it still says `Connected`. The extension therefore verifies repair with real HTTP probes through the proxy, never scutil.
 
 ## Tuning pi's own retries
 
@@ -64,9 +73,15 @@ Node switching is intentionally left to the proxy client (chain-proxy friendly).
 | Var | Default | Meaning |
 |---|---|---|
 | `PI_PROXY_GUARD` | `1` | `0` disables |
-| `PI_PROXY_GUARD_URL` | `https://www.google.com/generate_204` | health-check target; for tighter signal use your provider's API URL |
+| `PI_PROXY_GUARD_URL` | `https://www.google.com/generate_204` | cheap ping (watchdog ticks) |
+| `PI_PROXY_GUARD_API_URL` | `ctx.model.baseUrl` | deep-probe target; defaults to the active model's API host — any HTTP response counts as reachable |
+| `PI_PROXY_GUARD_STREAM_URL` | `https://speed.cloudflare.com/__down?bytes=65536` | sustained-transfer probe used in deep checks |
+| `PI_PROXY_GUARD_STREAM_MIN_BYTES` | `60000` | min bytes the transfer must deliver; `0` disables it |
+| `PI_PROXY_GUARD_VPN_SERVICE` | _unset_ | `scutil --nc` service name — tunnel-down short-circuits straight to repair |
 | `PI_PROXY_GUARD_PROXY` | _(env)_ | explicit `-x` proxy for the check |
-| `PI_PROXY_GUARD_TIMEOUT_MS` | `10000` | check timeout |
+| `PI_PROXY_GUARD_TIMEOUT_MS` | `20000` | per-attempt check timeout |
+| `PI_PROXY_GUARD_PROBE_ATTEMPTS` | `2` | consecutive failures required to declare "down" — tolerates latency spikes without mis-firing a repair |
+| `PI_PROXY_GUARD_PROBE_GAP_MS` | `1500` | gap between probe attempts |
 | `PI_PROXY_GUARD_SHORTCUT` | `Reconnect Shadowrocket` | `""` disables |
 | `PI_PROXY_GUARD_SCHEME` | `1` | `0` disables `shadowrocket://` fallback |
 | `PI_PROXY_GUARD_ESCALATE_AFTER` | `2` | consecutive errors → repair despite OK ping |

package/extensions/proxy-guard.ts

@@ -32,13 +32,32 @@
  * Env knobs (defaults):
  *   PI_PROXY_GUARD=0                     disable
  *   PI_PROXY_GUARD_URL                   https://www.google.com/generate_204
+ *                                        (cheap ping used by the watchdog)
+ *   PI_PROXY_GUARD_API_URL               (unset = use ctx.model.baseUrl)
+ *   PI_PROXY_GUARD_STREAM_URL            https://speed.cloudflare.com/__down?bytes=65536
+ *   PI_PROXY_GUARD_STREAM_MIN_BYTES      60000  (0 disables transfer probe)
+ *   PI_PROXY_GUARD_VPN_SERVICE           ""    (scutil --nc fast-path name)
  *   PI_PROXY_GUARD_PROXY                 (unset = inherit env proxy vars)
- *   PI_PROXY_GUARD_TIMEOUT_MS            10000
+ *   PI_PROXY_GUARD_TIMEOUT_MS            20000
+ *   PI_PROXY_GUARD_PROBE_ATTEMPTS        2   (down requires N consecutive fails)
+ *   PI_PROXY_GUARD_PROBE_GAP_MS          1500 (gap between probe attempts)
  *   PI_PROXY_GUARD_SHORTCUT              "Reconnect Shadowrocket" ("" disables)
  *   PI_PROXY_GUARD_SHORTCUT_TIMEOUT_MS   120000
+ *                                          NOTE: `shortcuts run` on macOS does NOT
+ *                                          fire iOS-app SiriKit intents (verified):
+ *                                          a shortcut wrapping Shadowrocket's
+ *                                          StopVPNIntent is a no-op via CLI, works
+ *                                          only from Shortcuts.app GUI. Only useful
+ *                                          for shortcuts with shell-able actions.
  *   PI_PROXY_GUARD_SCHEME                "1" (shadowrocket:// fallback)
- *   PI_PROXY_GUARD_SCHEME_STOP           shadowrocket://stop
- *   PI_PROXY_GUARD_SCHEME_START          shadowrocket://start
+ *   PI_PROXY_GUARD_SCHEME_STOP           shadowrocket://disconnect?autoclose=true
+ *   PI_PROXY_GUARD_SCHEME_START          shadowrocket://connect?autoclose=true
+ *                                          Verified working on macOS (iOS-app
+ *                                          runtime): `open -g` delivers the URL
+ *                                          without foregrounding; tunnel drops and
+ *                                          comes back. If Shadowrocket "always-on"
+ *                                          is enabled it may auto-reconnect before
+ *                                          our explicit connect — harmless either way.
  *   PI_PROXY_GUARD_ESCALATE_AFTER        2   (consecutive errors w/ OK check)
  *   PI_PROXY_GUARD_PAUSED_REPAIRS        3   (active repair cap while paused)
  *   PI_PROXY_GUARD_REPAIR_COOLDOWN_MS    90000
@@ -68,13 +87,21 @@ import { setTimeout as sleep } from "node:timers/promises";
 
 const DISABLED = process.env.PI_PROXY_GUARD === "0";
 const CHECK_URL = process.env.PI_PROXY_GUARD_URL ?? "https://www.google.com/generate_204";
+const API_URL_OVERRIDE = process.env.PI_PROXY_GUARD_API_URL;
+const STREAM_URL = process.env.PI_PROXY_GUARD_STREAM_URL ?? "https://speed.cloudflare.com/__down?bytes=65536";
+const STREAM_MIN_BYTES = Number(process.env.PI_PROXY_GUARD_STREAM_MIN_BYTES ?? 60_000);
+const VPN_SERVICE = process.env.PI_PROXY_GUARD_VPN_SERVICE ?? "";
 const CHECK_PROXY = process.env.PI_PROXY_GUARD_PROXY;
-const CHECK_TIMEOUT_MS = Number(process.env.PI_PROXY_GUARD_TIMEOUT_MS ?? 10_000);
+const CHECK_TIMEOUT_MS = Number(process.env.PI_PROXY_GUARD_TIMEOUT_MS ?? 20_000);
+const PROBE_ATTEMPTS = Number(process.env.PI_PROXY_GUARD_PROBE_ATTEMPTS ?? 2);
+const PROBE_GAP_MS = Number(process.env.PI_PROXY_GUARD_PROBE_GAP_MS ?? 1_500);
 const SHORTCUT = process.env.PI_PROXY_GUARD_SHORTCUT ?? "Reconnect Shadowrocket";
 const SHORTCUT_TIMEOUT_MS = Number(process.env.PI_PROXY_GUARD_SHORTCUT_TIMEOUT_MS ?? 120_000);
 const USE_SCHEME = process.env.PI_PROXY_GUARD_SCHEME !== "0";
-const SCHEME_STOP = process.env.PI_PROXY_GUARD_SCHEME_STOP ?? "shadowrocket://stop";
-const SCHEME_START = process.env.PI_PROXY_GUARD_SCHEME_START ?? "shadowrocket://start";
+// Verified against Shadowrocket's docs: connect/disconnect (not start/stop).
+// autoclose=true lets the app quit itself after handling the action.
+const SCHEME_STOP = process.env.PI_PROXY_GUARD_SCHEME_STOP ?? "shadowrocket://disconnect?autoclose=true";
+const SCHEME_START = process.env.PI_PROXY_GUARD_SCHEME_START ?? "shadowrocket://connect?autoclose=true";
 const ESCALATE_AFTER = Number(process.env.PI_PROXY_GUARD_ESCALATE_AFTER ?? 2);
 const PAUSED_REPAIR_MAX = Number(process.env.PI_PROXY_GUARD_PAUSED_REPAIRS ?? 3);
 const REPAIR_COOLDOWN_MS = Number(process.env.PI_PROXY_GUARD_REPAIR_COOLDOWN_MS ?? 90_000);
@@ -128,27 +155,68 @@ interface CheckResult {
 	error?: string;
 }
 
-function checkProxy(): Promise<CheckResult> {
+function execCmd(cmd: string, args: string[], timeoutMs: number): Promise<{ ok: boolean; output?: string; error?: string }> {
+	return new Promise((resolve) => {
+		execFile(cmd, args, { timeout: timeoutMs }, (err, stdout, stderr) => {
+			resolve(err
+				? { ok: false, output: stdout, error: ((stderr || "") + err.message).split("\n")[0].trim() }
+				: { ok: true, output: stdout });
+		});
+	});
+}
+
+/** Reachability probe: any HTTP response (even 404/401/502) means the
+ *  transport path to the host is alive — that's what we measure, not
+ *  whether the endpoint "succeeds". */
+function httpProbe(url: string): Promise<CheckResult> {
 	return new Promise((resolve) => {
 		const args = ["-sS", "-o", "/dev/null", "-w", "%{http_code}", "--max-time", String(Math.ceil(CHECK_TIMEOUT_MS / 1000))];
 		if (CHECK_PROXY) args.push("-x", CHECK_PROXY);
-		args.push(CHECK_URL);
+		args.push(url);
 		execFile("curl", args, { timeout: CHECK_TIMEOUT_MS + 2_000 }, (err, stdout) => {
 			const status = Number.parseInt((stdout ?? "").trim(), 10);
-			if (!err && status >= 200 && status < 400) resolve({ ok: true, status });
+			if (!err && status > 0) resolve({ ok: true, status });
 			else resolve({ ok: false, status: Number.isFinite(status) ? status : undefined, error: (err?.message ?? "").split("\n")[0] });
 		});
 	});
 }
 
-function execCmd(cmd: string, args: string[], timeoutMs: number): Promise<{ ok: boolean; error?: string }> {
+/** Sustained-transfer probe: downloads STREAM_URL and requires the body to
+ *  actually arrive (>= STREAM_MIN_BYTES). A half-dead chain that passes tiny
+ *  pings but cuts long-lived streams gets caught here. */
+function streamProbe(url: string): Promise<CheckResult> {
 	return new Promise((resolve) => {
-		execFile(cmd, args, { timeout: timeoutMs }, (err, _stdout, stderr) => {
-			resolve(err ? { ok: false, error: ((stderr || "") + err.message).split("\n")[0].trim() } : { ok: true });
+		const args = ["-sS", "-o", "/dev/null", "-w", "%{http_code} %{size_download}", "--max-time", String(Math.ceil(CHECK_TIMEOUT_MS / 1000))];
+		if (CHECK_PROXY) args.push("-x", CHECK_PROXY);
+		args.push(url);
+		execFile("curl", args, { timeout: CHECK_TIMEOUT_MS + 2_000 }, (err, stdout) => {
+			const [statusRaw, bytesRaw] = (stdout ?? "").trim().split(/\s+/);
+			const status = Number.parseInt(statusRaw, 10);
+			const bytes = Number.parseInt(bytesRaw, 10);
+			if (!err && status > 0 && bytes >= STREAM_MIN_BYTES) resolve({ ok: true, status });
+			else
+				resolve({
+					ok: false,
+					status: Number.isFinite(status) ? status : undefined,
+					error: err ? err.message.split("\n")[0] : `stream cut: got ${bytes}B < ${STREAM_MIN_BYTES}B`,
+				});
 		});
 	});
 }
 
+/** Optional fast path: the VPN tunnel state itself, zero network traffic. */
+async function tunnelUp(): Promise<boolean | undefined> {
+	if (!VPN_SERVICE) return undefined;
+	const r = await execCmd("scutil", ["--nc", "show", VPN_SERVICE], 5_000);
+	if (!r.ok || r.output === undefined) return undefined;
+	return /\bConnected\b/.test(r.output) ? true : /\bDisconnected\b/.test(r.output) ? false : undefined;
+}
+
+/** Back-compat shim: reachability via the configured CHECK_URL. */
+function checkProxy(): Promise<CheckResult> {
+	return httpProbe(CHECK_URL);
+}
+
 export default function (pi: ExtensionAPI) {
 	if (DISABLED) return;
 
@@ -237,14 +305,39 @@ export default function (pi: ExtensionAPI) {
 		}
 	}
 
-	/** checkProxy + episode bookkeeping: an OK probe resets the repair counter. */
-	async function probe(): Promise<CheckResult> {
-		const check = await checkProxy();
-		if (check.ok) {
-			failedRepairs = 0;
-			gaveUpNotified = false;
+	/** checkProxy + episode bookkeeping: an OK probe resets the repair counter.
+	 *  A single failure never means "down" on a wobbly chain proxy — we
+	 *  require PROBE_ATTEMPTS consecutive failures before declaring dead.
+	 *  This prevents latency spikes from triggering needless VPN bounces
+	 *  (which would kill the very streams we're trying to protect).
+	 *
+	 *  deep=true (settle/recheck): probes the actual API host (ctx.model.baseUrl
+	 *  or PI_PROXY_GUARD_API_URL) and adds a sustained-transfer check — the
+	 *  "can it hold an SSE stream" signal a 0-byte ping can't see.
+	 *  deep=false (watchdog): cheap generate_204 ping only. */
+	async function probe(ctx?: ExtensionContext, deep = false): Promise<CheckResult> {
+		const vpn = await tunnelUp();
+		if (vpn === false) return { ok: false, error: `VPN tunnel "${VPN_SERVICE}" disconnected` };
+
+		const target = (deep ? (API_URL_OVERRIDE ?? ctx?.model?.baseUrl) : undefined) ?? CHECK_URL;
+		let last: CheckResult = { ok: false };
+		for (let attempt = 1; attempt <= PROBE_ATTEMPTS; attempt++) {
+			last = await httpProbe(target);
+			if (last.ok && deep && STREAM_MIN_BYTES > 0) {
+				const stream = await streamProbe(STREAM_URL);
+				if (!stream.ok) last = stream;
+			}
+			if (last.ok) {
+				failedRepairs = 0;
+				gaveUpNotified = false;
+				return last;
+			}
+			if (attempt < PROBE_ATTEMPTS) {
+				log(`probe attempt ${attempt}/${PROBE_ATTEMPTS} failed (${last.error ?? `HTTP ${last.status}`}); retrying`);
+				await sleep(PROBE_GAP_MS);
+			}
 		}
-		return check;
+		return last;
 	}
 
 	/** True while we may still launch an active repair this down-episode. */
@@ -262,7 +355,7 @@ export default function (pi: ExtensionAPI) {
 		try {
 			const recheck = async (): Promise<boolean> => {
 				await sleep(RECHECK_DELAY_MS);
-				const check = await checkProxy();
+				const check = await probe(ctx, true); // deep: verify real capacity before unpausing
 				if (!check.ok) log(`still down: ${check.error ?? `HTTP ${check.status}`}`);
 				return check.ok;
 			};
@@ -279,13 +372,14 @@ export default function (pi: ExtensionAPI) {
 			const viaScheme = async (): Promise<boolean> => {
 				if (!USE_SCHEME) return false;
 				notify(ctx, "Bouncing Shadowrocket via URL scheme…", "warning");
-				const stop = await execCmd("open", [SCHEME_STOP], 15_000);
+				// `open -g`: background — do NOT activate/foreground Shadowrocket.
+				const stop = await execCmd("open", ["-g", SCHEME_STOP], 15_000);
 				if (!stop.ok) {
 					log(`scheme stop failed: ${stop.error}`);
 					return false;
 				}
 				await sleep(3_000);
-				const start = await execCmd("open", [SCHEME_START], 15_000);
+				const start = await execCmd("open", ["-g", SCHEME_START], 15_000);
 				if (!start.ok) {
 					log(`scheme start failed: ${start.error}`);
 					return false;
@@ -315,10 +409,9 @@ export default function (pi: ExtensionAPI) {
 					// Only spend effort when pi actually needs the connection:
 					// a live run (internal retries benefit) or a paused-by-us session.
 					if (!runActive && !pausedByUs) return;
-					if (budgetLeft() <= 0) return;
 					ticking = true;
 					try {
-						const check = await probe();
+						const check = await probe(sessionCtx, false); // cheap ping for the periodic watch
 						if (check.ok) {
 							// Recovered while paused: resume.
 							if (pausedByUs && !runActive) {
@@ -341,6 +434,8 @@ export default function (pi: ExtensionAPI) {
 							}
 							return;
 						}
+						// Repairing the network is free — only resumes/continues spend
+						// provider requests, so the budget gates those, not the repair.
 						if (await repairProxy(sessionCtx)) {
 							if (pausedByUs && !runActive && budgetLeft() > 0) {
 								chargeBudget();
@@ -410,13 +505,18 @@ export default function (pi: ExtensionAPI) {
 				`settle error #${consecutiveErrors}; canContinue=${event.context.canContinue}; ` +
 					`budget=${budgetLeft()}/${MAX_REPAIRS}; pausedByUs=${pausedByUs}`,
 			);
-			if (budgetLeft() <= 0) {
+			const check = await probe(ctx, true); // deep: API host + sustained transfer
+			const canContinue = budgetLeft() > 0;
+			if (!canContinue) {
+				// Budget is about *provider requests*, not connectivity: still
+				// repair a dead proxy so the user's manual take-over lands on a
+				// working network — then stay paused without continuing.
+				if (!check.ok) await repairProxy(ctx);
 				notify(ctx, `Auto-continue budget exhausted (${MAX_REPAIRS}/${Math.round(WINDOW_MS / 60_000)}min). Staying paused — take over manually.`, "warning");
 				systemNotify("auto-continue budget exhausted — session paused; check proxy or take over");
+				pausedByUs = true;
 				return;
 			}
-
-			const check = await probe();
 			const halfDead = check.ok && consecutiveErrors >= ESCALATE_AFTER;
 
 			if (!check.ok && !mayRepair()) {
@@ -431,7 +531,7 @@ export default function (pi: ExtensionAPI) {
 				if (!(await repairProxy(ctx))) {
 					// On the half-dead path the pre-repair check was OK: don't trust the
 					// failed repair to mean we're offline — verify once more.
-					const post = await probe();
+					const post = await probe(ctx, true);
 					if (!post.ok) {
 						pausedByUs = true;
 						notify(ctx, "Proxy still down after repair attempts. Staying paused; watchdog will auto-resume once it's back.", "error");
@@ -463,10 +563,10 @@ export default function (pi: ExtensionAPI) {
 		handler: async (args, ctx) => {
 			const sub = (args ?? "").trim().split(/\s+/)[0] || "check";
 			if (sub === "check") {
-				const check = await checkProxy();
+				const check = await probe(ctx, true);
 				notify(
 					ctx,
-					check.ok ? `Proxy OK (HTTP ${check.status}, ${CHECK_URL})` : `Proxy DOWN (${check.error ?? `HTTP ${check.status}`}, ${CHECK_URL})`,
+					check.ok ? `Proxy OK (HTTP ${check.status})` : `Proxy DOWN (${check.error ?? `HTTP ${check.status}`})`,
 					check.ok ? "info" : "error",
 				);
 			} else if (sub === "restart") {
@@ -475,7 +575,8 @@ export default function (pi: ExtensionAPI) {
 			} else if (sub === "status") {
 				notify(
 					ctx,
-					`url=${CHECK_URL} proxy=${CHECK_PROXY ?? "(env)"} shortcut="${SHORTCUT || "off"}" scheme=${USE_SCHEME ? "on" : "off"} ` +
+					`ping=${CHECK_URL} api=${API_URL_OVERRIDE ?? ctx.model?.baseUrl ?? "(none)"} stream=${STREAM_MIN_BYTES > 0 ? STREAM_URL : "off"} vpn=${VPN_SERVICE || "off"} ` +
+						`proxy=${CHECK_PROXY ?? "(env)"} shortcut="${SHORTCUT || "off"}" scheme=${USE_SCHEME ? "on" : "off"} ` +
 						`escalateAfter=${ESCALATE_AFTER} budget=${budgetLeft()}/${MAX_REPAIRS}/${Math.round(WINDOW_MS / 60_000)}min ` +
 						`watchdog=${WATCHDOG_MS}ms pausedByUs=${pausedByUs} consecErrs=${consecutiveErrors} failedRepairs=${failedRepairs}/${PAUSED_REPAIR_MAX} log=${LOG_FILE || "off"}`,
 				);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-proxy-guard",
-  "version": "0.6.0",
+  "version": "0.6.1",
   "description": "Auto-recover pi sessions when the proxy drops: connectivity check, Shadowrocket restart/node-switch via Shortcuts or URL scheme, watchdog auto-resume.",
   "license": "MIT",
   "repository": {