crawlforge-mcp-server 6.4.0 → 6.6.0

package/src/skills/agent-skills/crawlforge-browser-sessions/SKILL.md

@@ -0,0 +1,178 @@
+---
+name: crawlforge-browser-sessions
+description: "Keeps one browser page alive across several tool calls with CrawlForge's browser_session tool. Use when the user needs to log in and then read pages behind that login, work through a multi-step flow (search, filter, paginate, fill a wizard), or look at a page before deciding what to click. The loop is: open a session on a URL, snapshot it to list the interactive elements with stable refs (@e1, @e2), act on those refs, read the content, close. Unlike scrape_with_actions, which is one-shot and closes its browser when the call returns, a session survives between calls, so one login is paid for once and a wrong selector costs one call instead of a whole chain."
+metadata:
+  version: 5.6.6
+  source: crawlforge-mcp-server
+---
+
+# CrawlForge Browser Sessions
+
+Drive a real browser across several calls. `browser_session` keeps one page —
+its cookies, its login, whatever it has already clicked — alive between calls, so
+you can look at the page, act, and look again, instead of guessing a whole chain
+of CSS selectors for a page you have never seen.
+
+## When to use
+
+- **Log in once, then read several pages behind the login** → session.
+- **You do not know what is on the page yet** — observe, then act → session.
+- **A flow that spans more than one tool call** (wizard, filters, pagination,
+  multi-step form) → session.
+- A **known, fixed** action chain on one page → `scrape_with_actions` instead:
+  one call, 5 credits, browser closed for you (see crawlforge-batch-automation).
+- A page that renders **without interaction** → `scrape` (2 credits, see
+  crawlforge-web-scraping).
+
+## The loop
+
+`open` → `snapshot` → `act` on the refs → (`snapshot` again after a navigation)
+→ `read` → `close`.
+
+```json
+{ "tool": "browser_session", "params": { "operation": "open", "url": "https://app.example.com/login" } }
+```
+
+Returns a `sessionId` every later call passes as `session_id`, plus `expiresAt`
+and `idleExpiresAt`.
+
+```json
+{ "tool": "browser_session", "params": { "operation": "snapshot", "session_id": "<id>" } }
+```
+
+Returns an accessibility tree with a stable ref on every interactive element:
+
+```
+[document] "Sign in"
+  @e1 [textbox] "Email"
+  @e2 [textbox] "Password"
+  @e3 [button] "Sign in"
+```
+
+Act on those refs — a ref goes in `selector`, exactly where a CSS selector would:
+
+```json
+{
+  "tool": "browser_session",
+  "params": {
+    "operation": "act",
+    "session_id": "<id>",
+    "actions": [
+      { "type": "type", "selector": "@e1", "text": "user@example.com" },
+      { "type": "type", "selector": "@e2", "text": "hunter2" },
+      { "type": "click", "selector": "@e3" },
+      { "type": "wait", "duration": 2000 }
+    ]
+  }
+}
+```
+
+`actions` is the same array `scrape_with_actions` takes: `wait`, `click`,
+`type`, `press`, `scroll`, `select`, `hover`, `navigate`, `screenshot`,
+`snapshot`, `executeJavaScript`. 1–20 per call; `continue_on_error: true` keeps
+going past a failed one.
+
+```json
+{ "tool": "browser_session", "params": { "operation": "read", "session_id": "<id>", "formats": ["markdown"] } }
+{ "tool": "browser_session", "params": { "operation": "close", "session_id": "<id>" } }
+```
+
+`read` extracts the **live** DOM — post-login, post-click — so it sees what a
+fresh scrape of the same URL would not. `formats`: `markdown`, `html`, `text`,
+`json`.
+
+Other operations: `screenshot` (`full_page`, `format`, `quality`, `selector` —
+a ref works there too; the image is stored as a
+`crawlforge://screenshot/{actionId}` resource) and `list` (your open sessions).
+
+## Refs go stale on navigation
+
+`@e1` is only valid for the page the snapshot was taken on. Any navigation —
+clicking a link, submitting a form, an in-session `navigate` action — invalidates
+every ref. A stale ref does not silently hit the wrong element; it fails and
+tells you to take a new snapshot. **After anything that navigates, snapshot
+again** before using refs.
+
+`snapshot` is also available as an action type inside `scrape_with_actions`, for
+the one-shot case where you want to observe and act within a single call.
+
+## Worked example — log in, then read the dashboard
+
+```json
+{"tool":"browser_session","params":{"operation":"open","url":"https://app.example.com/login"}}
+→ { "sessionId": "8f2c…", "expiresAt": …, "idleExpiresAt": … }
+
+{"tool":"browser_session","params":{"operation":"snapshot","session_id":"8f2c…"}}
+→ tree with @e1 [textbox] "Email", @e2 [textbox] "Password", @e3 [button] "Sign in"
+
+{"tool":"browser_session","params":{"operation":"act","session_id":"8f2c…","actions":[
+  {"type":"type","selector":"@e1","text":"user@example.com"},
+  {"type":"type","selector":"@e2","text":"hunter2"},
+  {"type":"click","selector":"@e3"},
+  {"type":"wait","duration":2000}]}}
+→ success, url now https://app.example.com/dashboard
+
+{"tool":"browser_session","params":{"operation":"snapshot","session_id":"8f2c…"}}
+→ fresh refs for the dashboard (the login refs are gone — the page navigated)
+
+{"tool":"browser_session","params":{"operation":"read","session_id":"8f2c…","formats":["markdown"]}}
+→ the dashboard as markdown, logged in
+
+{"tool":"browser_session","params":{"operation":"close","session_id":"8f2c…"}}
+```
+
+Total: 3 + 1 + 1 + 1 + 2 + 1 = **9 credits**, and the login happened once. The
+same flow as repeated `scrape_with_actions` calls costs 5 per call and logs in
+again every time.
+
+## browser_session (cost: 3 to open, then 1–2 per call)
+
+| Operation | Credits |
+|-----------|---------|
+| `open` | 3 |
+| `read` | 2 |
+| `snapshot` | 1 |
+| `act` | 1 |
+| `screenshot` | 1 |
+| `close` | 1 |
+| `list` | 1 |
+
+The published flat rate is 3 — the ceiling, charged when an operation is not one
+of the above. Nothing here is free.
+
+## Limits
+
+- **Two clocks.** A session dies 600s after it opens (`ttl`, 30–3600) or 300s
+  after its last use (`activity_ttl`, 10–3600), whichever comes first. Close
+  sessions when you are done rather than leaving them to time out.
+- **Concurrent sessions.** One account may hold **1** session at a time over
+  CrawlForge's hosted REST API; a local (stdio) or self-hosted install allows
+  **3** per API key, and the process itself caps the total. A session over the
+  cap is refused, never queued — `close` one first.
+- **Sessions live in the server process** that opened them. They do not survive
+  a server restart, and they are not shared between installs.
+- **`executeJavaScript` is refused** in a session on a remotely-served instance
+  (the script would run in a browser on the server, not on your machine). Use
+  `click` / `type` / `select` / `press`, or run the server locally over stdio.
+- **Every in-session navigation is re-gated**: SSRF checks, the host blocklist
+  and robots.txt run again on each hop, not just at `open`.
+- `stealth: true` on `open` runs the session in the anti-bot browser (see
+  crawlforge-stealth-browsing); `viewport` sets the window size.
+
+## CLI
+
+```bash
+crawlforge browser https://example.com                      # open, snapshot, close — see the refs
+crawlforge browser https://app.example.com --steps flow.json --read
+```
+
+One invocation is one session: the CLI opens, snapshots, runs the steps in
+`flow.json` (each `{"operation": …}` with the session supplied), optionally
+reads, and closes. A session cannot span two CLI invocations — the page lives in
+the process. Use the MCP tool when you need a session to outlive the call.
+
+## Cost note
+
+An `open` + `snapshot` + `act` + `read` + `close` round trip is 8 credits, and
+each extra look is 1. Reach for `scrape_with_actions` (5) when you can write the
+whole interaction down in advance, and for a session when you cannot.

package/src/skills/agent-skills/crawlforge-getting-started/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: crawlforge-getting-started
-description: "Orientation and tool-selection guide for the CrawlForge MCP server's 30 web tools. Use when the user is getting started with CrawlForge, asks which CrawlForge tool to use, how to set up the API key, how skills or the CLI work, what a tool costs in credits, or when one tool fails and a fallback is needed. Routes requests to the right specialized skill (web scraping, deep research, stealth, structured extraction, change tracking, batch automation), and explains MCP-tools-vs-CLI, the Ollama-first LLM fallback chain, and per-tool credit costs."
+description: "Orientation and tool-selection guide for the CrawlForge MCP server's 31 web tools. Use when the user is getting started with CrawlForge, asks which CrawlForge tool to use, how to set up the API key, how skills or the CLI work, what a tool costs in credits, or when one tool fails and a fallback is needed. Routes requests to the right specialized skill (web scraping, deep research, stealth, structured extraction, change tracking, batch automation, browser sessions), and explains MCP-tools-vs-CLI, the Ollama-first LLM fallback chain, and per-tool credit costs."
 metadata:
   version: 5.6.6
   source: crawlforge-mcp-server
@@ -8,7 +8,7 @@ metadata:
 
 # CrawlForge: Getting Started
 
-CrawlForge is an MCP server with **30 tools** for web scraping, crawling,
+CrawlForge is an MCP server with **31 tools** for web scraping, crawling,
 extraction, research, change tracking, and AI-compliance. This skill orients you
 and routes each request to the right specialized skill.
 
@@ -51,8 +51,9 @@ stored at `~/.crawlforge/config.json`.
 | Extract JSON/fields, parse a PDF, summarize, analyze sentiment | **crawlforge-structured-extraction** |
 | Watch a page for changes / monitor pricing | **crawlforge-change-tracking** |
 | Scrape many URLs, run browser actions, generate llms.txt | **crawlforge-batch-automation** |
+| Keep a browser open across calls: log in then read, click through a multi-step flow | **crawlforge-browser-sessions** |
 
-## The 30 tools at a glance
+## The 31 tools at a glance
 
 - **Basic (6):** fetch_url, extract_text, extract_links, extract_metadata, scrape_structured, read_result
 - **Unified (1):** scrape (multi-format single fetch)
@@ -62,12 +63,13 @@ stored at `~/.crawlforge/config.json`.
 - **Batch & automation (4):** batch_scrape, get_batch_results, scrape_with_actions, generate_llms_txt
 - **Stealth & locale (2):** stealth_mode, localization
 - **Templates & tracking (2):** scrape_template, track_changes
+- **Browser sessions (1):** browser_session (one page kept alive across calls)
 
 ## MCP tools vs CLI
 
 - **MCP tools** — call inline within an AI assistant session (Claude Code,
   Cursor, etc.). This is the default in chat.
-- **CLI** (`crawlforge <command>`) — for scripts, CI, and pipelines. 15 tool
+- **CLI** (`crawlforge <command>`) — for scripts, CI, and pipelines. 16 tool
   commands + 2 skill commands. See [cli](references/cli.md).
 
 Both hit the same backend and consume the same credits.
@@ -94,6 +96,7 @@ Do not suggest adding API keys — local Ollama is the intended zero-cost defaul
 | No template for a known site | `scrape_structured` → `extract_structured` → `extract_with_llm` |
 | LLM extraction unavailable (no Ollama/keys) | `scrape_structured` with CSS selectors |
 | Single page too slow / many pages | `batch_scrape` (async + webhook) |
+| A `scrape_with_actions` chain keeps breaking on guessed selectors, or the flow needs more than one call | `browser_session` — snapshot for refs, then act (crawlforge-browser-sessions) |
 | Wrong region / currency shown | `localization` |
 | Need a big report but cost is high | lower `maxUrls` on `deep_research` |
 | Result came back `truncated: true` with a `result_handle` | `read_result` (search, slice, lines, json_path) — never fetch the page again |

package/src/skills/agent-skills/crawlforge-getting-started/references/cli.md

@@ -22,7 +22,7 @@ crawlforge init                            # detect key, install skills, merge M
 | `--timeout <ms>` | Request timeout (default 30000). |
 | `--version` / `--help` | Version / help. |
 
-## Tool commands (15)
+## Tool commands (16)
 
 | Command | Maps to | Example |
 |---------|---------|---------|
@@ -37,6 +37,7 @@ crawlforge init                            # detect key, install skills, merge M
 | `stealth <url>` | stealth_mode | `crawlforge stealth <url> --engine camoufox --screenshot` |
 | `batch <file>` | batch_scrape | `crawlforge batch urls.txt --format markdown --concurrency 10` |
 | `actions <url>` | scrape_with_actions | `crawlforge actions <url> --script flow.json --screenshot` |
+| `browser <url>` | browser_session | `crawlforge browser <url> --steps flow.json --read` |
 | `localize <url>` | localization | `crawlforge localize <url> --locale fr-FR --country FR` |
 | `llmstxt <url>` | generate_llms_txt | `crawlforge llmstxt <url> --include-full` |
 | `template <id> <target>` | scrape_template | `crawlforge template github-repo https://github.com/owner/repo` |
@@ -68,4 +69,8 @@ crawlforge scrape https://example.com --quiet && echo ok  # exit code only
 | `CRAWLFORGE_STEALTH_ENGINE` | Force `playwright` or `camoufox`. |
 | `CRAWLFORGE_BROWSER_BACKEND` | `local` or `browserbase`. |
 
+`crawlforge browser` runs a whole session (open, snapshot, steps, close) in one
+invocation — a session cannot span two CLI processes. Use the `browser_session`
+MCP tool when a session has to outlive the call that opened it.
+
 Exit code 0 = success, 1 = error. `crawlforge <command> --help` for per-command help.

package/src/skills/agent-skills/crawlforge-getting-started/references/credits.md

@@ -36,6 +36,7 @@ metered; there is no free tier. Tools marked "scales" cost more as work grows.
 | `analyze_content` | Sentiment / entities / keywords. |
 | `extract_structured` | Schema-driven (LLM + CSS fallback). |
 | `extract_with_llm` | NL-prompt extraction. |
+| `browser_session` | Ceiling, charged for `open`. **Priced per operation:** `open` 3, `read` 2, `snapshot` / `act` / `screenshot` / `close` / `list` 1 each. |
 
 ## 4 credits
 
@@ -75,6 +76,9 @@ metered; there is no free tier. Tools marked "scales" cost more as work grows.
 - Cap dynamic tools: `deep_research`/`agent` via `maxUrls`, `crawl_deep` via
   `max_pages`, `batch_scrape` via the URL list size.
 - `get_batch_results` (1) is cheap — submit a batch once, page through results.
+- A `browser_session` pays the 3-credit `open` once; every look after that is 1.
+  Use it when a flow spans several calls, and `scrape_with_actions` (5, one call)
+  when the whole interaction can be written down in advance.
 - A result over `max_inline_chars` costs nothing extra; read the stored copy with
   `read_result` (1) instead of fetching again.
 - On a site that blocks, one `scrape` with `escalate:true` costs at most 7 and

package/src/skills/installer.js

@@ -254,7 +254,7 @@ export async function uninstall({ target = 'all', cwd = process.cwd(), homeDir =
 const HOOK_MARKER = 'CrawlForge skill';
 const HOOK_COMMAND =
   "echo 'Consider whether a CrawlForge skill applies: web scraping, deep research, " +
-  "stealth browsing, structured extraction, change tracking, or batch automation.'";
+  "stealth browsing, structured extraction, change tracking, batch automation, or browser sessions.'";
 
 /**
  * Add a UserPromptSubmit forced-eval reminder to ~/.claude/settings.json.