pi-unsloth-webtools 0.9.0 → 0.9.1

package/README.md

@@ -172,9 +172,10 @@ third-party rendering service.
   still gets its chance.
 - Lightpanda identifies itself honestly and refuses to impersonate a browser user agent, so hard
   anti-bot walls are returned as failures (the direct error, or the incomplete-content note).
-- Binary resolution: `lightpanda` on `PATH`, then `PI_LIGHTPANDA_BIN`, then
-  `webRender.lightpandaPath`. Prebuilt binaries exist for Linux (glibc; musl needs a source
-  build) and macOS, plus Docker images; Windows needs WSL2.
+- Binary resolution: `webRender.lightpandaPath`, then `PI_LIGHTPANDA_BIN`, then the launcher
+  installed by `scripts/install-lightpanda.sh`, then `lightpanda` on `PATH`. Prebuilt binaries
+  exist for Linux (glibc; musl needs a source build) and macOS, plus Docker images; Windows
+  needs WSL2.
 - Version matters: 1.0.0 renders JavaScript-heavy pages that the 0.2.x line cannot — measured on
   the same machine, IMDb went from a 76-byte empty document to 21k characters, dribbble from 32
   characters to 18k, and a Medium article from a challenge page to real text. Linux builds from
@@ -186,7 +187,9 @@ third-party rendering service.
 - `bash scripts/install-lightpanda.sh` installs the newest stable release. When the system glibc
   predates what the binary needs, it downloads Debian's `libc6` for the current stable suite,
   extracts it into the install directory, and writes a launcher shim — so 1.0.0 runs on a
-  glibc 2.36 host without touching the system libraries. The script prints the path to use.
+  glibc 2.36 host without touching the system libraries. It also writes
+  `webRender.lightpandaPath` into the global settings, so the extension picks the binary up with
+  no further setup; `--no-configure` skips that and `--print-path` prints the launcher path for CI.
 - Disable the tier with `webRender.lightpandaEnabled: false`; `web_fetch` then stops after the
   network attempts and reports the original error.
 
@@ -309,7 +312,7 @@ Optional settings in `~/.pi/agent/settings.json` or `.pi/settings.json` (project
 | `unslothWebTools.allowLocalFiles` / `webFetch.allowLocalFiles` | `true` | Opt out to refuse local files in `web_fetch` (`file://` URLs, absolute, `~/`, or `./` paths); when enabled, PDFs are extracted and HTML converted |
 | `webFetch.transport` / `unslothWebTools.transport` | `tls-first` | Fetch transport order: `tls-first` (default), `direct-first`, or `off` to disable the browser-fingerprint transport entirely |
 | `webRender.lightpandaEnabled` / `unslothWebTools.lightpandaEnabled` | `true` | Opt out to disable local Lightpanda rendering |
-| `webRender.lightpandaPath` / `unslothWebTools.lightpandaPath` | `lightpanda` on `PATH` (`PI_LIGHTPANDA_BIN` fallback) | Path to the Lightpanda binary used for local rendering |
+| `webRender.lightpandaPath` / `unslothWebTools.lightpandaPath` | launcher installed by `scripts/install-lightpanda.sh`, else `lightpanda` on `PATH` | Path to the Lightpanda binary used for local rendering |
 | `webRender.lightpandaCommand` / `unslothWebTools.lightpandaCommand` | none | Command prefix that launches the renderer, for WSL (`["wsl.exe","-e","<path>"]`) or containers; overrides `lightpandaPath`. The fetch flags are appended to it |
 
 Environment overrides: `PI_UNSLOTH_CACHE_DIR` changes the fetch cache directory, `PI_UNSLOTH_WEBTOOLS_STATS` opts into append-only sweep stats JSONL, `PI_CODING_AGENT_DIR` / `PI_AGENT_DIR` change the global settings directory, and `PI_LIGHTPANDA_BIN` points at the local renderer binary. Cache entries live 1 hour and stale copies are served only after a network failure. SOCKS5 proxies named by `HTTPS_PROXY`, `HTTP_PROXY`, or `ALL_PROXY` are honored on every fetch (`NO_PROXY` exclusions apply).
@@ -354,30 +357,9 @@ npm test
 npm run test:unit
 npm run test:smoke
 bash scripts/install-lightpanda.sh
-npm run compare:fetch
-npm run compare:browsers
 npm run camoufox:warmup
-npm run stealth:matrix
-/tmp/pyenv/bin/python scripts/stealth-python.py
 ```
 
-`npm run compare:fetch` runs a live head-to-head of the direct fetch, the TLS-impersonation retry,
-local Lightpanda rendering over a target list. It accepts URLs as arguments and `--no-lightpanda`
-to drop the render tier. `npm run compare:browsers` adds a Camoufox column;
-install it separately (`npm i camoufox-js playwright-core && npx camoufox-js fetch`, plus GTK3
-libraries on Linux) and use `--seconds=N` to bound how long it waits out a JS challenge.
-`npm run stealth:matrix` compares stealth-browser options against one walled page (plus
-`bot.sannysoft.com` detection rows and a plain-page sanity check): raw CDP to a system Chromium
-(`PI_CHROMIUM_BIN` to point at it), Playwright with its bundled Chromium, Patchright, and Camoufox.
-Every browser dependency is loaded through a guarded dynamic import, so nothing is added to
-`package.json`; install whichever rows you want to measure. `--attempts=N` and `--seconds=N` bound
-the walled-page attempts, and passing row names runs a subset (`raw-cdp`, `playwright`, `patchright`,
-`camoufox`).
-
-`scripts/stealth-python.py` is the same idea for the Python-side options (nodriver, CloakBrowser,
-DrissionPage, cloudscraper, curl_cffi) against the same walled page; it needs a venv with those
-packages installed and is not wired into any npm script. Measured findings are in its module docstring.
-
 `npm run camoufox:warmup` measures what a warm Camoufox costs and buys: launch time, idle CPU and
 RSS, per-fetch latency with the browser already running, and whether a persistent profile
 (`user_data_dir`, pinned fingerprint) lets a Cloudflare clearance survive a restart. Flags:

package/lightpanda.ts

@@ -1,4 +1,7 @@
 import { spawn as spawnProcess } from "node:child_process";
+import { existsSync } from "node:fs";
+import { homedir } from "node:os";
+import { join } from "node:path";
 import { visibleChars } from "./html-to-md.ts";
 import {
   checkUrlAccess,
@@ -201,11 +204,31 @@ function withProvenance(prefixed: string, url: string): string {
   return [...lines.slice(0, index), ...header, ...lines.slice(index)].join("\n");
 }
 
+function installedLauncher(): string | null {
+  let dataHome = process.env.XDG_DATA_HOME?.trim();
+  if (!dataHome) {
+    try {
+      const home = homedir();
+      if (!home) return null;
+      dataHome = join(home, ".local", "share");
+    } catch {
+      return null;
+    }
+  }
+  const dir = join(dataHome, "pi-unsloth-webtools", "lightpanda");
+  for (const name of ["lightpanda-run", "lightpanda"]) {
+    const candidate = join(dir, name);
+    if (existsSync(candidate)) return candidate;
+  }
+  return null;
+}
+
 export function lightpandaBinary(options: LightpandaRenderOptions = {}): string {
   return (
     options.binaryPath ??
     options.settings?.binaryPath ??
     process.env.PI_LIGHTPANDA_BIN?.trim() ??
+    installedLauncher() ??
     DEFAULT_BINARY
   );
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-unsloth-webtools",
-  "version": "0.9.0",
+  "version": "0.9.1",
   "type": "module",
   "description": "Pi extension: web_search and web_fetch tools that began as a port of the Unsloth Studio codebase and now diverge from it (multi-engine search, opt-in SSRF guard, HTML-to-Markdown extraction)",
   "main": "index.ts",
@@ -59,10 +59,7 @@
     "test:unit": "vitest run --exclude **/smoke.test.ts",
     "test:smoke": "vitest run test/smoke.test.ts",
     "test:lightpanda": "vitest run test/lightpanda-live.test.ts",
-    "compare:fetch": "node scripts/fetch-compare.ts",
-    "compare:browsers": "node scripts/compare-browsers.ts",
     "camoufox:warmup": "node scripts/camoufox-warmup.ts",
-    "stealth:matrix": "node scripts/stealth-matrix.ts",
     "typecheck": "tsc --noEmit",
     "check:package": "node -e \"const fs=require('fs');const pkg=require('./package.json');const listed=new Set(pkg.files);const bad=[];for(const f of pkg.files){if(!fs.existsSync(f))bad.push('missing file: '+f)}for(const f of fs.readdirSync('.').filter(f=>f.endsWith('.ts'))){if(!listed.has(f))bad.push('unlisted source: '+f)}if(bad.length){console.error(bad.join('\\n'));process.exit(1)}\"",
     "prepublishOnly": "npm run typecheck && npm run lint && npm run test:unit && npm run check:package && npm run check:publint",

package/scripts/install-lightpanda.sh

@@ -10,6 +10,7 @@ VERSION=""
 FORCE=0
 ALLOW_SHIM=1
 PRINT_PATH=0
+NO_CONFIGURE=0
 
 usage() {
   cat <<'EOF'
@@ -23,12 +24,15 @@ Usage: bash scripts/install-lightpanda.sh [options]
   --force        Reinstall even when a working binary is already present
   --no-shim      Fail instead of bundling a newer libc when the system glibc is too old
   --print-path   Print only the launcher path (for scripts and CI)
+  --no-configure Skip writing webRender.lightpandaPath into the global settings
   -h, --help     Show this message
 
 On Linux, if the release needs a newer glibc than the system provides, the script
 downloads Debian's libc6 for the current stable suite, extracts it into the install
-directory, and writes a launcher shim that runs the binary with that libc. Point
-`webRender.lightpandaPath` (settings.json) or `PI_LIGHTPANDA_BIN` at the printed path.
+directory, and writes a launcher shim that runs the binary with that libc.
+The script also writes `webRender.lightpandaPath` into the global settings so the
+extension picks the binary up with no further setup; `--no-configure` skips that,
+and `--print-path` prints only the launcher path for scripts and CI.
 EOF
 }
 
@@ -39,6 +43,7 @@ for arg in "$@"; do
     --force) FORCE=1 ;;
     --no-shim) ALLOW_SHIM=0 ;;
     --print-path) PRINT_PATH=1 ;;
+    --no-configure) NO_CONFIGURE=1 ;;
     -h|--help) usage; exit 0 ;;
     *) echo "unknown option: $arg" >&2; usage >&2; exit 2 ;;
   esac
@@ -100,11 +105,96 @@ mkdir -p "$DIR"
 binary="$DIR/lightpanda"
 shim="$DIR/lightpanda-run"
 
+settings_file_for_agent() {
+  local agent_dir="${PI_CODING_AGENT_DIR:-${PI_AGENT_DIR:-$HOME/.pi/agent}}"
+  case "$agent_dir" in
+    "~") agent_dir="$HOME" ;;
+    "~/"*) agent_dir="$HOME/${agent_dir#\~/}" ;;
+  esac
+  printf '%s\n' "$agent_dir/settings.json"
+}
+
+configure_renderer() {
+  local binary_path="$1"
+  local settings_file
+  local status
+  settings_file="$(settings_file_for_agent)"
+  if [ "$NO_CONFIGURE" = 1 ]; then
+    echo "settings:   skipped (--no-configure); set PI_LIGHTPANDA_BIN or webRender.lightpandaPath to $binary_path"
+    return
+  fi
+  if ! command -v node >/dev/null 2>&1; then
+    echo "settings:   node not found; set PI_LIGHTPANDA_BIN=$binary_path or webRender.lightpandaPath yourself" >&2
+    return
+  fi
+  status="$(SETTINGS_FILE="$settings_file" LIGHTPANDA_PATH="$binary_path" node --input-type=module <<'NODE'
+import { existsSync, mkdirSync, readFileSync, renameSync, rmSync, statSync, writeFileSync } from "node:fs";
+import { dirname } from "node:path";
+
+const settingsFile = process.env.SETTINGS_FILE;
+const binaryPath = process.env.LIGHTPANDA_PATH;
+function fail(message) {
+  console.error(message);
+  process.exit(3);
+}
+
+let settings = {};
+let mode;
+if (existsSync(settingsFile)) {
+  try {
+    settings = JSON.parse(readFileSync(settingsFile, "utf8"));
+  } catch {
+    fail(`settings:   keeping ${settingsFile}: not valid JSON`);
+  }
+  if (!settings || typeof settings !== "object" || Array.isArray(settings)) {
+    fail(`settings:   keeping ${settingsFile}: not a JSON object`);
+  }
+  mode = statSync(settingsFile).mode & 0o777;
+}
+
+const sections = [settings.unslothWebTools, settings.webRender].filter(
+  (section) => section && typeof section === "object" && !Array.isArray(section),
+);
+const alreadyConfigured = sections.some(
+  (section) =>
+    (typeof section.lightpandaPath === "string" && section.lightpandaPath.length > 0) ||
+    (Array.isArray(section.lightpandaCommand) && section.lightpandaCommand.length > 0),
+);
+if (alreadyConfigured) {
+  console.log("kept");
+  process.exit(0);
+}
+
+if (settings.webRender !== undefined && (!settings.webRender || typeof settings.webRender !== "object" || Array.isArray(settings.webRender))) {
+  fail(`settings:   keeping ${settingsFile}: webRender is not a JSON object`);
+}
+
+settings.webRender = { ...(settings.webRender ?? {}), lightpandaPath: binaryPath };
+const temporary = `${settingsFile}.tmp.${process.pid}`;
+try {
+  mkdirSync(dirname(settingsFile), { recursive: true });
+  writeFileSync(temporary, JSON.stringify(settings, null, 2) + "\n", mode === undefined ? {} : { mode });
+  renameSync(temporary, settingsFile);
+} catch (error) {
+  rmSync(temporary, { force: true });
+  fail(`settings:   keeping ${settingsFile}: ${error instanceof Error ? error.message : String(error)}`);
+}
+console.log("configured");
+NODE
+)" || status=""
+  case "$status" in
+    configured) echo "settings:   ${settings_file} now sets webRender.lightpandaPath" ;;
+    kept) echo "settings:   ${settings_file} already configures a lightpanda path or command; left unchanged" ;;
+    *) echo "settings:   could not update ${settings_file} automatically" >&2 ;;
+  esac
+}
+
 report_path() {
   if [ "$PRINT_PATH" = 1 ]; then
     printf '%s\n' "$1"
   else
     echo "already installed and working: $1" >&2
+    configure_renderer "$1"
   fi
   exit 0
 }
@@ -200,7 +290,5 @@ fi
 echo
 echo "Lightpanda $installed_version installed"
 echo "  run this:          $run_path"
-echo "  configure one of:"
-echo "    export PI_LIGHTPANDA_BIN=$run_path"
-echo "    settings.json:   { \"webRender\": { \"lightpandaPath\": \"$run_path\" } }"
 echo "  verify:            $run_path fetch --dump markdown --wait-until networkidle https://example.com/"
+configure_renderer "$run_path"