browser4-cli 0.1.22 → 0.1.23

package/README.md

@@ -143,7 +143,7 @@ The tables below mirror the commands surfaced by the global `browser4-cli help`
 | `check <ref>` | Check a checkbox or radio button |
 | `uncheck <ref>` | Uncheck a checkbox or radio button |
 | `drag <startRef> <endRef>` | Drag and drop between two elements |
-| `snapshot` | Capture accessibility snapshot. Supports `--filename=<path>` to save to file, `--boxes` for bounding boxes, `-i`/`--interactive` for interactive-only, `-u`/`--urls` for link URLs, `-c`/`--compact` for compact output, `-d`/`--depth <n>` for depth limiting, and `-s`/`--selector <sel>` for CSS scoping. |
+| `snapshot` | Capture accessibility snapshot. Supports `--filename=<path>` to save to file, `--boxes` for bounding boxes, `-i`/`--interactive` for interactive-only, `-u`/`--urls` for link URLs, `-c`/`--compact` for compact output, `-d`/`--depth <n>` for depth limiting, `-s`/`--selector <sel>` for CSS scoping, and `--raw` for raw output without page info. |
 | `eval <expression> [ref]` | Evaluate JavaScript on the page or a target element. Use `--file=<path>` to read the expression from a file. |
 | `get <mode> <selector> [name]` | Extract data from a page element. Modes: `text`, `html`, `box`, `styles`, `property`, `attr`. `name` is required for `property` and `attr`. |
 | `scroll <direction> <pixels>` | Scroll the page. Direction: `up`, `down`, `left`, or `right`. |
@@ -280,6 +280,7 @@ Query `browser4-cli help <command>` for the exact syntax when you need them.
 | `swarm query <url>` | Run an X-SQL query against a loaded webpage |
 | `swarm status <id>` | Check the status of a scrape or query job |
 | `swarm result <id>` | Get the result of a completed job |
+| `crawl <url>` | Crawl a website from a seed URL, following links up to a configurable depth |
 
 ## Agent task workflow (`agent <subcommand>`)
 
@@ -381,6 +382,8 @@ coordinates multiple browser contexts in the Browser4 backend.
 | `swarm query <url>` | Run X-SQL queries against loaded pages | `POST /api/swarm/query` |
 | `swarm status <id>` | Poll job status | `GET /api/swarm/{id}/status` |
 | `swarm result <id>` | Fetch completed job result | `GET /api/swarm/{id}/result` |
+| `crawl <url>` | Start a website crawl | `POST /api/crawl` |
+| (poll) | Track crawl progress | `GET /api/crawl/{id}/result` |
 
 ### URL scraping with `swarm submit`
 
@@ -438,6 +441,57 @@ browser4-cli swarm query --sql @query.sql --seed-file=./urls.txt --refresh
 - Prefix the `--sql` value with `@` to read from a file (e.g. `--sql @query.sql`).
 - All commands return a task ID; use `swarm status` / `swarm result` to track progress.
 
+## Crawl (`crawl`)
+
+Recursive website crawling from a seed URL.
+
+### Command overview
+
+| Command | Purpose | Backend endpoint |
+|---|---|---|
+| `crawl <url>` | Crawl a website from a seed URL | `POST /api/crawl` |
+| (poll) | Track crawl progress | `GET /api/crawl/{id}/result` |
+
+### Basic crawling
+
+```shell
+# depth=1: extract all links from homepage, load each linked page
+browser4-cli crawl "https://example.com" --out-link-selector "a[href]"
+
+# depth=2: follow links two levels deep, filter by pattern
+browser4-cli crawl "https://shop.example.com" \
+  --depth 2 \
+  --out-link-selector "a.product-link" \
+  --out-link-pattern "/product/" \
+  --top-links 10
+
+# with LoadOptions passthrough
+browser4-cli crawl "https://example.com" -ol "a[href]" -a "-refresh -nMaxRetry 5"
+```
+
+### Key flags
+
+| Flag | Default | Description |
+|---|---|---|
+| `-d`, `--depth` | `1` | Maximum crawl depth |
+| `-ol`, `--out-link-selector` | — | CSS selector to extract links from each page |
+| `-olp`, `--out-link-pattern` | `.+` | Regex pattern to filter extracted links |
+| `-tl`, `--top-links` | `20` | Maximum links to extract per page |
+| `-a`, `--args` | — | Additional LoadOptions passthrough |
+| `--refresh` | — | Force a fresh fetch, ignoring cache |
+| `--parse` | — | Parse each page immediately after fetching |
+| `--expires` | — | Cache expiration duration |
+| `--store-content` | — | Persist page content to storage |
+| `--page-load-timeout` | — | Maximum time to wait for page load |
+| `--readonly` | — | Non-destructive mode |
+
+### Notes
+
+- All LoadOptions flags available via `--args` passthrough (e.g. `-a "-nMaxRetry 5 -lazyFlush"`).
+- Depth=1 reuses `PulsarSession.submitForOutPages`; depth>1 uses BFS continuous crawl with visited-URL dedup.
+- CLI timeout: 600s default, configurable via `BROWSER4_CLI_CRAWL_TIMEOUT_SECS`.
+- Duplicate URLs are skipped (normalized: lowercase, no trailing slash, no query string).
+
 ## DOM Snapshot commands (`domsnapshot <subcommand>`)
 
 The `domsnapshot` commands capture and query a **static DOM snapshot** of the
@@ -462,6 +516,37 @@ browser4-cli domsnapshot export --file=page.html
 | `domsnapshot get <field> [selector] [name]` | Extract elements from the static DOM snapshot |
 | `domsnapshot query [url] --sql=<query>` | Run X-SQL against the DOM snapshot via the scrape API |
 | `domsnapshot export [--file=<path>]` | Save full snapshot HTML content to a local file |
+| `domsnapshot grep [OPTIONS] <pattern>` | Search snapshot HTML with regex and grep-style output |
+
+### `domsnapshot grep` flags
+
+Line numbers are shown by default (unlike GNU grep). Use `--no-line-number` to suppress them.
+
+| Flag | Description |
+|---|---|
+| `-i` | Case-insensitive matching |
+| `-A N`, `-B N`, `-C N` | Context lines after / before / around each match |
+| `-v` | Invert match (select non-matching lines) |
+| `-c` | Print only count of matching lines |
+| `-l` | Print only whether matches exist (prints "domsnapshot" if matches found; use with `\| grep -q domsnapshot` for CI pass/fail) |
+| `-F` | Treat pattern as a literal string |
+| `-w` | Match whole words only |
+| `--no-line-number` | Suppress line numbers in output |
+| `--selector <CSS>` | Scope search to a CSS selector |
+
+```shell
+# Basic search
+browser4-cli domsnapshot grep -i error
+
+# With context and literal match
+browser4-cli domsnapshot grep -F -C 2 "404 Not Found"
+
+# Count matches
+browser4-cli domsnapshot grep -c "div"
+
+# Search within a specific element
+browser4-cli domsnapshot grep --selector main "Submit"
+```
 
 ### `domsnapshot get` fields
 
@@ -609,6 +694,9 @@ browser4-cli snapshot -i -c -d 5
 # Scope snapshot to a specific CSS selector
 browser4-cli snapshot -s "#main-content"
 
+# Raw output — no page info, just the snapshot content (useful for piping)
+browser4-cli snapshot --raw | grep "button"
+
 # Capture a static DOM snapshot and extract data
 browser4-cli domsnapshot
 browser4-cli domsnapshot get text "#productTitle"

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "browser4-cli",
-  "version": "0.1.22",
+  "version": "0.1.23",
   "description": "Browser automation CLI for AI agents",
   "type": "module",
   "files": [