alvin-bot 4.8.8 → 4.9.0

package/skills/browse/SKILL.md

@@ -1,136 +1,161 @@
 ---
 name: Browser Automation
-description: Interactive browser control — navigate, click, fill forms, screenshot, test web apps
-triggers: browse, browser, test webapp, test app, test website, screenshot page, interact with, click on, fill form, visual test, qa test, check page, open page, test my app, browse to, open url, puppeteer, playwright, browser automation, test die seite, teste die app, schau dir an, öffne die seite, teste mal, visual check, check the ui, check the page
+description: 3-tier browser control — stealth scraping, CDP with persistent cookies, visual oversight. Navigate, screenshot, extract text, interact with logged-in pages.
+triggers: browse, browser, test webapp, test app, test website, screenshot page, interact with, click on, fill form, visual test, qa test, check page, open page, test my app, browse to, open url, puppeteer, playwright, browser automation, linkedin, stepstone, indeed, scrape, fetch page, crawl, teste die seite, teste die app, schau dir an, öffne die seite, teste mal, visual check, check the ui, check the page, webseite öffnen, seite abrufen
 priority: 8
 category: automation
 ---
 
-# Browser Automation — Playwright Interactive
+# Browser Automation — 3-Tier Router
 
-## Browser Strategies
+Du hast drei Browser-Strategien plus WebFetch. **Wähle die billigste passende Stufe** und eskaliere nur wenn nötig.
 
-Alvin Bot auto-selects the best browser approach:
+## Entscheidungsregel (in dieser Reihenfolge)
 
-| Strategy | When | How |
-|----------|------|-----|
-| **CLI** (default) | Simple screenshots, text extraction, PDF | Headless Playwright, one-shot |
-| **HTTP Gateway** | Interactive browsing, form-filling, QA testing | Persistent browser server on port 3800 |
-| **CDP** | Attach to user's Chrome (with login state) | Chrome DevTools Protocol via CDP_URL |
+| Task | Tool | Warum |
+|------|------|-------|
+| Einzelne öffentliche Seite, nur Text | WebFetch oder `curl` | Am schnellsten, keine Browser-Engine |
+| Öffentliche Seite mit JS / Cloudflare | **Tier 1 Stealth** | Headless + Fingerprint-Masking |
+| Login-pflichtige Seite (LinkedIn, Gmail, …) | **Tier 2 CDP** | Echtes Chrome, persistente Cookies |
+| Komplexer Multi-Step-Flow, User soll zusehen | **Tier 3 Extension** | Visuelle Kontrolle |
 
-The gateway starts automatically when needed and shuts down after 5 min idle.
-For CDP: Launch Chrome with `--remote-debugging-port=9222` and set `CDP_URL=http://localhost:9222`.
+**NIEMALS** `scripts/browse-server.cjs` nutzen — existiert nicht mehr. **NIEMALS** nacktes `node -e "const {chromium}…"` für externe Seiten — wird sofort geblockt.
 
 ---
 
-You have a persistent Playwright browser server that gives you **eyes** and **hands** to interact with web pages. You can navigate, see screenshots, read the accessibility tree, click buttons, fill forms, and test running web apps.
+## Tier 0 — WebFetch / curl (schnellster Pfad)
 
-## Quick Start
+Für statische Seiten oder APIs, die keine JS-Rendering brauchen:
 
 ```bash
-# 1. Ensure server is running (auto-shuts down after 5 min idle)
-curl -s http://127.0.0.1:3800/health 2>/dev/null | grep -q '"ok":true' || \
-  (BOT_DIR=$(node -e "console.log(require('path').resolve(require.resolve('alvin-bot/package.json'), '..'))" 2>/dev/null || echo ".") && cd "$BOT_DIR" && node scripts/browse-server.cjs &) && sleep 3
+# Direkter curl
+curl -sL -H "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36" \
+  "https://www.michaelpage.de/jobs/it-director" | htmlq -t "h1, .job-title"
 
-# 2. Navigate to a page
-curl -s "http://127.0.0.1:3800/navigate?url=https://example.com" | jq
+# Oder das WebFetch-Tool, wenn verfügbar (interpretiert Inhalt direkt)
+```
 
-# 3. Take a screenshot (view it with Read tool)
-SHOT=$(curl -s "http://127.0.0.1:3800/screenshot" | jq -r '.path')
-# Then use Read tool on $SHOT to see the image
+Wenn das einen 403/Captcha gibt → eskaliere auf Tier 1.
 
-# 4. Get interactive elements
-curl -s "http://127.0.0.1:3800/tree" | jq '.tree[]' -r
+---
 
-# 5. Click something
-curl -s "http://127.0.0.1:3800/click?ref=e5" | jq
-```
+## Tier 1 — Playwright Stealth (headless, schnell, maskiert)
+
+**Router-Script:** `~/.claude/hub/SCRIPTS/browser.sh`
+
+```bash
+# Seite laden, JSON-Metadata zurück (title, url, html_length)
+~/.claude/hub/SCRIPTS/browser.sh stealth "https://www.stepstone.de/jobs/it-delivery"
 
-## All Routes
-
-| Route | Params | What it does |
-|-------|--------|-------------|
-| `/navigate` | `url` | Open a URL, returns title + accessibility tree |
-| `/screenshot` | `full=true` (optional) | Take screenshot, returns file path |
-| `/tree` | `limit=N` (optional) | Get all interactive elements with @eN refs |
-| `/click` | `ref=eN` | Click element by ref |
-| `/fill` | `ref=eN`, `value=text` | Fill input field |
-| `/type` | `ref=eN`, `text=chars` | Type character by character (for special inputs) |
-| `/press` | `key=Enter`, `ref=eN` (opt) | Press keyboard key |
-| `/select` | `ref=eN`, `value=opt` | Select dropdown option |
-| `/hover` | `ref=eN` | Hover over element |
-| `/scroll` | `direction=down/up/top/bottom`, `amount=600` | Scroll page |
-| `/eval` | `js=expression` | Run JavaScript on page |
-| `/wait` | `ms=2000` or `selector=.class` | Wait for time or element |
-| `/viewport` | `device=mobile/tablet` or `width=W&height=H` | Change viewport |
-| `/cookies` | `set=[{...}]` (optional) | Get or set cookies |
-| `/back` | — | Browser back |
-| `/forward` | — | Browser forward |
-| `/reload` | — | Reload page |
-| `/network` | `limit=20` | Recent network requests |
-| `/info` | — | Current page info |
-| `/close` | — | Close browser + shutdown server |
-| `/health` | — | Server status check |
-
-## Element Refs (@eN)
-
-The accessibility tree assigns **refs** like `@e1`, `@e2`, `@e3` to every interactive element (links, buttons, inputs, etc.). Use these refs for all interactions — they're more robust than CSS selectors.
-
-Example tree:
+# Mit Screenshot (PNG in ~/.claude/hub/BROWSER/screenshots/)
+~/.claude/hub/SCRIPTS/browser.sh stealth "https://example.com" --screenshot=page.png
 ```
-@e1 <a href="/"> "Home"
-@e2 <a href="/dashboard"> "Dashboard"
-@e3 <input type="email" name="email" placeholder="Enter email">
-@e4 <input type="password" name="password" placeholder="Password">
-@e5 <button> "Sign In"
-@e6 <a href="/forgot"> "Forgot password?"
+
+**Was du bekommst:** JSON mit `{title, url, html_length, screenshot}`. Der volle HTML liegt nicht in stdout — zum Parsen den `stealth.js` direkt als Modul importieren oder `/tmp/`-File lesen.
+
+**Wann blockt das:** reCAPTCHA v3, aggressive Cloudflare, Login-Walls.
+
+**Konkrete funktionierende Targets (Stand 2026):**
+- StepStone (alle Job-Suchen) ✅
+- Michael Page ✅
+- Hays ✅
+- Öffentliche Blog-Posts, News-Sites ✅
+- LinkedIn (ohne Login) ❌ → Tier 2
+- Indeed / Glassdoor ❌ (403 Scraping-Block) → nur über E-Mail-Alerts
+
+---
+
+## Tier 2 — Chrome CDP (persistent Profile, echte Cookies)
+
+Echtes Chrome mit Profil unter `~/.claude/hub/BROWSER/profile/`. Login-Cookies für LinkedIn/Gmail/etc. bleiben über Sessions erhalten.
+
+```bash
+# Einmal starten (checkt ob schon läuft)
+~/.claude/hub/SCRIPTS/browser.sh cdp start headless   # headless — für Cron/Daemon
+~/.claude/hub/SCRIPTS/browser.sh cdp start headful    # sichtbar — wenn User zusehen soll
+
+# Navigieren
+~/.claude/hub/SCRIPTS/browser.sh cdp goto "https://www.linkedin.com/jobs/search/?keywords=IT+Director"
+
+# Screenshot
+~/.claude/hub/SCRIPTS/browser.sh cdp shot "https://www.linkedin.com/feed/" linkedin_feed.png
+
+# Tabs auflisten
+~/.claude/hub/SCRIPTS/browser.sh cdp tabs
+
+# Stoppen (meistens nicht nötig, Chrome läuft persistent)
+~/.claude/hub/SCRIPTS/browser.sh cdp stop
 ```
 
-To login:
+**Login-Setup (einmalig):** Falls LinkedIn ausgeloggt ist, Ali per Telegram fragen:
+> "Bitte einmal in Chrome (Hub-Profil) bei LinkedIn einloggen. Cookies bleiben dann dauerhaft erhalten."
+
+Starten mit `cdp start headful` und Chrome öffnet sichtbar → Ali loggt ein → ab dann bleiben Cookies im Profil.
+
+**Wie teste ich ob eingeloggt:** nach `cdp goto` die URL prüfen — wenn `/authwall` oder `/login` im Pfad steht, bist du ausgeloggt.
+
+---
+
+## Tier 3 — Claude-in-Chrome Extension (visuelle Kontrolle)
+
+Nur in interaktiven CLI-Sessions, nicht im Cron/Daemon.
+
 ```bash
-curl -s "http://127.0.0.1:3800/fill?ref=e3&value=user@example.com"
-curl -s "http://127.0.0.1:3800/fill?ref=e4&value=mypassword"
-curl -s "http://127.0.0.1:3800/click?ref=e5"
+# Check ob Extension verbunden
+~/.claude/hub/SCRIPTS/browser.sh ext check
+
+# Dann MCP-Tools über ToolSearch laden:
+#   mcp__claude-in-chrome__tabs_context_mcp
+#   mcp__claude-in-chrome__navigate
+#   mcp__claude-in-chrome__computer
 ```
 
-## Standard Workflow: Test a Web App
+**Wann nutzen:** Drag&Drop, komplexe UI, User soll live zusehen und eingreifen können.
 
-1. **Start** the browse server if not running
-2. **Navigate** to the app URL
-3. **Screenshot** → view with Read tool to see current state
-4. **Tree** → see all interactive elements
-5. **Interact** (click, fill, press) using @eN refs
-6. **Screenshot** again to verify the result
-7. **Repeat** for each test step
-8. **Report** findings to the user
-9. **Close** when done
+---
 
-## Mobile Testing
+## Eskalations-Regel (PFLICHT)
+
+```
+Öffentliche Text-Seite → Tier 0 (WebFetch/curl)
+  ↓ 403/Cloudflare/leerer HTML?
+Tier 1 (stealth) → browser.sh stealth <url>
+  ↓ Captcha/Login-Wall?
+Tier 2 (CDP) → cdp start headless/headful + cdp goto <url>
+  ↓ Cookies fehlen?
+Ali fragen: "Bitte einmal in Chrome bei X einloggen, dann kann ich weitermachen."
+```
+
+**NIEMALS aufgeben mit "Browser funktioniert nicht"** — es gibt immer einen nächsten Schritt. Lieber ehrlich melden "Tier 1 blockt mit Captcha, versuche Tier 2" als "Failed to load".
+
+## Status-Checks
 
 ```bash
-# Switch to mobile viewport
-curl -s "http://127.0.0.1:3800/viewport?device=mobile"
-curl -s "http://127.0.0.1:3800/screenshot" | jq -r '.path'
-# Switch back to desktop
-curl -s "http://127.0.0.1:3800/viewport?width=1280&height=720"
+# Übersicht aller Tiers + Health
+~/.claude/hub/SCRIPTS/browser.sh status
+
+# Ist CDP Chrome gerade auf Port 9222?
+curl -s http://127.0.0.1:9222/json/version | head -c 200
 ```
 
-## Auth / Cookie Injection
+## Screenshot-Ausgabe ansehen
+
+Screenshots werden gespeichert unter `~/.claude/hub/BROWSER/screenshots/` (relativ) oder dem absoluten Pfad, den du angibst. Read-Tool auf den Pfad zeigt dir das Bild direkt an.
+
+## Interaktive Ops (Klicken, Formular füllen)
+
+Für einfache Fälle: `cdp eval` mit JavaScript, das in der Seite ausgeführt wird:
 
-For pages that need authentication:
 ```bash
-# Set cookies manually
-curl -s 'http://127.0.0.1:3800/cookies?set=[{"name":"session","value":"abc123","domain":"example.com","path":"/"}]'
-# Then navigate to the authenticated page
-curl -s "http://127.0.0.1:3800/navigate?url=https://example.com/dashboard"
+~/.claude/hub/SCRIPTS/browser.sh cdp eval "https://example.com/login" \
+  "document.querySelector('#username').value='test'; document.querySelector('#password').value='pw'; document.querySelector('form').submit();"
 ```
 
-## Important Notes
+Für komplexere Flows (sequentielles Klicken nach DOM-Updates) → Tier 3 (Extension) nutzen.
+
+## Wichtige Notes
 
-- **Server auto-shuts down** after 5 min idle — restart if needed
-- **One page at a time** — navigation replaces the current page
-- **Screenshots** are saved to `/tmp/alvin-bot/browse/` — view with Read tool
-- **127.0.0.1 only** — not accessible from outside
-- **URL-encode** values with special chars: `value=hello%20world`
-- **Refs reset** on every navigation/click — always get fresh /tree after page changes
-- For **local dev servers**: use `http://localhost:PORT` as the URL
+- **CDP-Profil-Konflikt:** Chrome kann `~/.claude/hub/BROWSER/profile/` nicht doppelt öffnen. Wenn Ali es lokal auf hatte, Port 9222 checken und `cdp stop` + `cdp start` machen.
+- **Headless vs Headful:** Im Cron/Daemon (launchd) IMMER `headless` — sonst scheitert Chrome an fehlendem Display.
+- **Nach Seiten-Navigation** (`cdp goto`) neue Tabs legt Playwright standardmäßig an — reuseTab ist nicht exponiert. Das ist OK für einzelne Scrapes, kann aber zu Tab-Explosion führen. `cdp stop` & Neustart räumt auf.
+- **Persistenz:** Cookies, LocalStorage, IndexedDB, alles in `~/.claude/hub/BROWSER/profile/`. Komplett persistiert zwischen Bot-Restarts.

package/test/browser-webfetch.test.ts

@@ -0,0 +1,121 @@
+/**
+ * Fix #11 (minimal) — webfetch Tier 0 for browser-manager.
+ *
+ * Background: the current browser fallback chain is
+ *   gateway → cdp → hub-stealth → cli
+ * Every tier spawns playwright (or talks to a CDP-controlled Chrome),
+ * which is slow and occasionally impossible under load. Many scraping
+ * tasks only need plain HTTP — an RSS feed, a JSON API, an OG meta-
+ * tag sniff. For those, Node's native `fetch` is 100× faster and
+ * doesn't need a browser at all.
+ *
+ * Contract: `webfetchNavigate(url)` returns `{ title, url }` for a
+ * successful GET, or throws a distinct `WebfetchFailed` error that the
+ * cascade can catch and fall through to the next tier. Title is the
+ * first `<title>` tag content; if none, the URL is returned.
+ *
+ * Keep it small — this is a Tier 0 helper, not a full scraper.
+ */
+import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
+import {
+  webfetchNavigate,
+  WebfetchFailed,
+  parseTitle,
+} from "../src/services/browser-webfetch.js";
+
+describe("parseTitle (Fix #11)", () => {
+  it("extracts a simple <title>", () => {
+    expect(parseTitle("<html><head><title>Hello World</title></head></html>")).toBe("Hello World");
+  });
+
+  it("handles whitespace and newlines", () => {
+    expect(parseTitle("<title>\n  Multi  line  \n</title>")).toBe("Multi line");
+  });
+
+  it("returns empty string when there's no title", () => {
+    expect(parseTitle("<html><body>no title</body></html>")).toBe("");
+  });
+
+  it("decodes basic HTML entities", () => {
+    expect(parseTitle("<title>A &amp; B</title>")).toBe("A & B");
+    expect(parseTitle("<title>&quot;quoted&quot;</title>")).toBe('"quoted"');
+  });
+
+  it("is case-insensitive for the tag name", () => {
+    expect(parseTitle("<HEAD><TITLE>Foo</TITLE></HEAD>")).toBe("Foo");
+  });
+});
+
+describe("webfetchNavigate (Fix #11)", () => {
+  let originalFetch: typeof fetch;
+
+  beforeEach(() => {
+    originalFetch = globalThis.fetch;
+  });
+  afterEach(() => {
+    globalThis.fetch = originalFetch;
+  });
+
+  it("returns title + url on a 200 response", async () => {
+    globalThis.fetch = vi.fn(async () =>
+      new Response(
+        "<html><head><title>GitHub · alvbln/alvin-bot</title></head></html>",
+        { status: 200, headers: { "content-type": "text/html" } },
+      ),
+    ) as unknown as typeof fetch;
+
+    const result = await webfetchNavigate("https://github.com/alvbln/alvin-bot");
+    expect(result.title).toBe("GitHub · alvbln/alvin-bot");
+    expect(result.url).toBe("https://github.com/alvbln/alvin-bot");
+  });
+
+  it("throws WebfetchFailed with the HTTP status on 4xx/5xx", async () => {
+    globalThis.fetch = vi.fn(async () =>
+      new Response("blocked", { status: 403 }),
+    ) as unknown as typeof fetch;
+
+    await expect(webfetchNavigate("https://example.com")).rejects.toThrow(WebfetchFailed);
+    try {
+      await webfetchNavigate("https://example.com");
+    } catch (err) {
+      expect((err as WebfetchFailed).status).toBe(403);
+    }
+  });
+
+  it("throws WebfetchFailed when the response is not HTML and forceHtml=true", async () => {
+    globalThis.fetch = vi.fn(async () =>
+      new Response('{"json":true}', {
+        status: 200,
+        headers: { "content-type": "application/json" },
+      }),
+    ) as unknown as typeof fetch;
+
+    await expect(
+      webfetchNavigate("https://api.example.com/data", { forceHtml: true }),
+    ).rejects.toThrow(WebfetchFailed);
+  });
+
+  it("accepts non-HTML responses when forceHtml is false (default)", async () => {
+    globalThis.fetch = vi.fn(async () =>
+      new Response("plain text", {
+        status: 200,
+        headers: { "content-type": "text/plain" },
+      }),
+    ) as unknown as typeof fetch;
+
+    const result = await webfetchNavigate("https://example.com/raw");
+    // No <title> in plain text → falls back to URL as display title
+    expect(result.url).toBe("https://example.com/raw");
+    expect(result.title).toBe("https://example.com/raw");
+  });
+
+  it("wraps network errors in WebfetchFailed so the cascade can catch a single type", async () => {
+    globalThis.fetch = vi.fn(async () => {
+      throw new Error("getaddrinfo ENOTFOUND nonexistent.invalid");
+    }) as unknown as typeof fetch;
+
+    await expect(
+      webfetchNavigate("https://nonexistent.invalid/"),
+    ).rejects.toThrow(WebfetchFailed);
+  });
+});

package/test/console-timestamps.test.ts

@@ -0,0 +1,98 @@
+/**
+ * Fix #10 — console output must carry ISO timestamps so out.log / err.log
+ * are actually debuggable. Also: silence libsignal's "Closing session"
+ * SessionEntry dumps which were pushing tens of KB per day into the logs
+ * and making forensic work painful.
+ *
+ * Contract: `installConsoleFormatter(console)` wraps console.log /
+ * console.warn / console.error so every line is prefixed with the
+ * current ISO timestamp (zero-padded, UTC), and certain noise patterns
+ * (libsignal session dumps) are dropped entirely.
+ *
+ * The wrapper is idempotent — calling it twice is a no-op.
+ */
+import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
+import {
+  installConsoleFormatter,
+  uninstallConsoleFormatter,
+  isNoisyLine,
+} from "../src/util/console-formatter.js";
+
+describe("installConsoleFormatter (Fix #10)", () => {
+  let stdoutWrites: string[];
+  let stderrWrites: string[];
+  let origStdout: typeof process.stdout.write;
+  let origStderr: typeof process.stderr.write;
+
+  beforeEach(() => {
+    stdoutWrites = [];
+    stderrWrites = [];
+    origStdout = process.stdout.write.bind(process.stdout);
+    origStderr = process.stderr.write.bind(process.stderr);
+    process.stdout.write = ((chunk: unknown) => {
+      stdoutWrites.push(String(chunk));
+      return true;
+    }) as typeof process.stdout.write;
+    process.stderr.write = ((chunk: unknown) => {
+      stderrWrites.push(String(chunk));
+      return true;
+    }) as typeof process.stderr.write;
+  });
+
+  afterEach(() => {
+    uninstallConsoleFormatter();
+    process.stdout.write = origStdout;
+    process.stderr.write = origStderr;
+  });
+
+  it("prefixes console.log output with an ISO timestamp", () => {
+    installConsoleFormatter();
+    console.log("hello world");
+    const line = stdoutWrites.join("");
+    // ISO format like 2026-04-11T14:00:00.000Z
+    expect(line).toMatch(/^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z\s+hello world/);
+  });
+
+  it("prefixes console.error output with an ISO timestamp", () => {
+    installConsoleFormatter();
+    console.error("boom");
+    const line = stderrWrites.join("");
+    expect(line).toMatch(/^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z\s+boom/);
+  });
+
+  it("is idempotent — second install call does not double-prefix", () => {
+    installConsoleFormatter();
+    installConsoleFormatter();
+    console.log("once");
+    const line = stdoutWrites.join("");
+    // Exactly one ISO timestamp, not two
+    const matches = line.match(/\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z/g);
+    expect(matches).not.toBeNull();
+    expect(matches!.length).toBe(1);
+  });
+});
+
+describe("isNoisyLine (Fix #10)", () => {
+  it("treats libsignal session dumps as noise", () => {
+    const dump = `Closing session: SessionEntry {
+  _chains: {
+    'BUQxzJlwgVTCxCL5C4rTbZP/7a0ciMPnyo47Pwr4flJt': { chainKey: [Object], chainType: 1, messageKeys: {} }
+  },
+  registrationId: 1446528770`;
+    expect(isNoisyLine(dump)).toBe(true);
+  });
+
+  it("treats the one-line 'Closing open session' as noise", () => {
+    expect(isNoisyLine("Closing open session in favor of incoming prekey bundle")).toBe(true);
+  });
+
+  it("treats the repetitive claude native binary banner as noise", () => {
+    expect(isNoisyLine("[claude] Native binary: /Users/foo/.local/share/claude/versions/2.1.101")).toBe(true);
+  });
+
+  it("does NOT silence normal log output", () => {
+    expect(isNoisyLine("⏰ Cron scheduler started (30s interval)")).toBe(false);
+    expect(isNoisyLine("[watchdog] started — beacon every 30s")).toBe(false);
+    expect(isNoisyLine("Cron: Running job \"Daily Job Alert\"")).toBe(false);
+  });
+});