browser4-cli 4.13.11 → 4.14.0-rc.2

package/README.md

@@ -38,6 +38,7 @@ English | [简体中文](README.zh.md) | [中国镜像](https://gitee.com/platon
   - [🚀 Build from Source](#-build-from-source)
   - [Architecture](#architecture)
   - [📦 Modules Overview](#-modules-overview)
+  - [🧩 Programming-Agent Kernel (browser4-coding)](#-programming-agent-kernel-browser4-coding)
   - [🧪 Test Fixture Server (MockSite)](#-test-fixture-server-mocksite)
   - [🤝 Support & Community](#-support--community)
   - [📜 Documentation](#-documentation)
@@ -51,11 +52,12 @@ English | [简体中文](README.zh.md) | [中国镜像](https://gitee.com/platon
 
 ### ✨ Key Capabilities
 
-* 🤖 **Agent Browser** — Enable AI agents to browse, interact, and automate real-world websites.
-* 🧠 **ML-Powered Extraction** — Learn page structures and extract structured data without LLM token costs.
-* ⚡ **High-Performance Runtime** — Coroutine-safe architecture supporting 100k–200k complex page visits per machine per day.
-* 🧬 **Hybrid Intelligence** — Combine LLM, ML, X-SQL, and selectors for robust extraction and experience reuse.
-* 📦 **Enterprise-Scale Automation** — Swarm crawling, CDP-native control, batch jobs, stateful sessions, plugins, extensions, and more.
+* 🤖 **Agent Browser** — AI agents and humans drive real browsers via a Rust CLI, MCP, and an agentic backend: navigate, click, fill, snapshot, batch, and loop.
+* 🧬 **Zero-Token Extraction** — X-SQL + CSS selectors for deterministic extraction from live pages or stored HTML snapshots; WebMiner ML clustering turns HTML corpora into spreadsheet and report views with no LLM tokens.
+* 🧠 **Hybrid Intelligence** — Combine LLM extraction, ML clustering, X-SQL, and a progressive experience store that reuses learned selectors and blockers.
+* ⚡ **High-Performance Runtime** — Coroutine-safe, CDP-native engine designed for 100k–200k complex page visits per machine per day via swarm/crawl scale-out.
+* 📦 **Enterprise-Scale Automation** — Swarm crawling, batch/loop jobs, stateful sessions, plugins, runtime skills, browser extension, and MCP-over-HTTP.
+* 🛠️ **Programming-Agent Kernel** — 50+ `coding.*` tools (sandboxed shell/fs, scaffolding, validation, self-development) for agents building Browser4 artifacts — or Browser4 itself.
 
 ## Quick Start
 
@@ -273,7 +275,7 @@ export DEEPSEEK_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxx
 
 | Command | Description |
 |---|---|
-| `open [url]` | Open a browser session or reconnect to an existing one. **Headless by default.** Supports `--headed` (visible window), `--headless`, `--profile <path>`, `--profile-mode <DEFAULT\|SYSTEM_DEFAULT\|SEQUENTIAL\|TEMPORARY>`, `--interact-level <FASTEST\|FAST\|DEFAULT>`. |
+| `open [url]` | Open a browser session or reconnect to an existing one. **Headless by default.** Supports `--headed` (visible window), `--headless`, `--profile <path>`, `--profile-mode <DEFAULT\|SYSTEM_DEFAULT\|SEQUENTIAL\|TEMPORARY>`, `--interact-level <FASTEST\|FAST\|DEFAULT>`. **Note:** `SYSTEM_DEFAULT` is deprecated and unsupported on Chrome ≥ 143 — use `attach` + `state-save`/`state-load` to reuse system browser state (see [browser-state-import.md](skills/browser4-cli/references/browser-state-import.md)). |
 | `attach` | Attach to an existing browser via CDP or the Browser4 extension. Supports `--cdp <url\|port\|channel>` and remote endpoint options. |
 | `close` | Close the active browser session. |
 | `list` | List browser sessions with their status and next-open behavior. Supports `--all`. |
@@ -281,10 +283,11 @@ export DEEPSEEK_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxx
 | `close-all` | Close all sessions without stopping the backend. |
 | `kill-all` | Force-stop the backend and Browser4-managed browser processes. |
 | `stop` | Gracefully stop the Browser4 server. |
-| `status` | Show server version, port, and health. |
+| `status` | Show server version, port, health, and the web status panel URL (`http://<server>:8182/status`). |
 | `doctor` | Run diagnostics: build info, LLM status, stale daemon cleanup, optional repair. Supports `--verbose` and `--fix`. |
 | `doctor log [name]` | List, view, tail, or grep backend log files. Supports `--tail`, grep-style flags, and `doctor log <name> grep <pattern>`. |
 | `doctor metrics [filter]` | List, filter, or grep backend metrics. Supports `doctor metrics grep <pattern>`. |
+| `doctor status [--section <name>] [--verbose]` | Print the aggregated status panel report in the terminal: summary layer by default, full detail with `--verbose`, one report with `--section` (health, build, runtime, llm, sessions, pulsar-sessions, swarm, url-pool, browsers, drivers, privacy, plugins, skills, metrics, logs), machine-readable JSON with `--json`. |
 | `delete-data` | Delete session data. |
 | `install` | Install the Browser4 runtime bundle. Supports `--tag <version>` and `--force`. |
 | `upgrade` | Upgrade the CLI/runtime bundle. Supports `--tag <version>` and `--force`. |
@@ -296,8 +299,31 @@ browser4-cli attach --cdp chrome
 browser4-cli doctor --verbose
 browser4-cli doctor log server.log --tail
 browser4-cli doctor metrics grep request
+browser4-cli doctor status --section skills --verbose
 ```
 
+**Web status panel:** open `http://127.0.0.1:8182/status` in a browser for a live dashboard
+(health, version, JVM/runtime, LLM config, sessions, **Pulsar sessions** — SDK identity,
+context and main-loop state, **swarm** — swarm session plus task summary, **URL pool** —
+queued/real-time/delay counts per priority cache, browsers & open tabs — per-session
+browser/driver binding and tab counts, with on-demand live tab details via
+`GET /api/system/tabs` — driver pools, plugins: load/enable state and SDK compatibility,
+**skills**: registered skills with origin (classpath/filesystem/programmatic),
+metrics, log files; auto-refreshes, set `?refresh=<ms>` to change the interval). The panel
+is backed by the aggregated `GET /api/system/status` endpoint; the individual endpoints
+(`/api/system/health`, `/api/system/build`, `/api/doctor/llm-status`, `/api/doctor/metrics`,
+`/api/doctor/log-files`, `/api/plugins`, `/api/skills`) remain available. `browser4-cli plugin-list` also
+reports load/enable state and SDK version for every installed plugin; the same reports can be
+read from the terminal with `browser4-cli doctor status`.
+
+**Page screenshots:** open `http://127.0.0.1:8182/pages.html` for a grid of every open page
+across sessions. The active tab of each session is captured automatically (click a screenshot
+to re-capture it); inactive tabs show a placeholder that captures on click. Swarm sessions only
+show placeholders. Screenshots load **asynchronously** — the backend captures in the background
+(`202 Accepted` with `Retry-After` while capturing, cached `image/png` when ready), so the panel
+never blocks on a capture. Backed by `GET /api/pages` and
+`GET /api/pages/{sessionId}/{guid}/screenshot.png` (`?refresh=1` forces a new capture).
+
 #### Navigation
 
 | Command | Description |
@@ -399,6 +425,7 @@ browser4-cli cdp Runtime.evaluate --json '{"expression":"document.title"}'
 | `htmlsnapshot summary` | Generate a compressed Web Page Summary Index (WPSI). |
 | `htmlsnapshot grep <pattern>` | Search stored HTML with grep-style flags. |
 | `htmlsnapshot inspect [selector]` | Discover recurring DOM patterns and selector candidates. Supports `--max`, `--depth`, `--stdin`, `--selector-base64`. |
+| `htmlsnapshot readability [url]` | Extract the main article content with a Readability-style heuristic — no LLM, no tokens. Supports `--text-only` and pagination. |
 
 Important rules:
 
@@ -406,6 +433,7 @@ Important rules:
 - use `htmlsnapshot` when you need repeated DOM extraction
 - `htmlsnapshot query --sql @query.sql` is the recommended way to avoid shell quoting issues
 - for correlated list extraction, prefer `htmlsnapshot query` over repeated `get all`
+- for one-step article extraction (no selectors needed), use `htmlsnapshot readability`
 
 ```bash
 browser4-cli htmlsnapshot
@@ -414,6 +442,7 @@ browser4-cli htmlsnapshot get all text ".result-title" --offset 10 --limit 5
 browser4-cli htmlsnapshot inspect ".s-result-item" --depth 6 --max 20
 browser4-cli htmlsnapshot export --file page.html --clean
 browser4-cli htmlsnapshot query --sql @query.sql
+browser4-cli htmlsnapshot readability --text-only --all
 ```
 
 For deep X-SQL usage, see [skills/browser4-cli/references/htmlsnapshot.md](skills/browser4-cli/references/htmlsnapshot.md) and [skills/browser4-cli/references/x-sql-dom-load-select.md](skills/browser4-cli/references/x-sql-dom-load-select.md).
@@ -468,7 +497,7 @@ These commands require an LLM key.
 | `summarize [instruction]` | Summarize the current page. Supports `--selector`, `--filename`, `--raw`, `--stdout`. |
 | `chat <message>` | Send a plain AI chat request without auto-appended browser context. |
 | `chat-result <id>` | Retrieve the result of an async chat task. |
-| `agent run <task>` | Submit an autonomous browser task and immediately receive a task ID. |
+| `agent run <task>` | Submit an autonomous browser task and immediately receive a task ID. Supports `--wait` (block for the result) and `--wait-timeout <seconds>` (default 600). |
 | `agent status <id>` | Check a running task. |
 | `agent result <id>` | Fetch a completed result. |
 | `agent list` | List tracked agent tasks and their status. |
@@ -617,7 +646,7 @@ The runtime bundle is stored separately in a platform-conventional application-d
 
 ## 🚀 Build from Source
 
-**Prerequisites:** Git, JDK 17+ (21+ recommended), Chrome/Chromium, and PowerShell 7 (Linux/macOS only). For the full prerequisites table, platform-specific tools, and Chrome auto-detection paths, see [Build from Source](docs/build-from-source.md).
+**Prerequisites:** Git, JDK 25+ (Eclipse Temurin), Chrome/Chromium, and PowerShell 7 (Linux/macOS only). For the full prerequisites table, platform-specific tools, and Chrome auto-detection paths, see [Build from Source](docs/build-from-source.md).
 
 1. **Clone the repository**
    ```shell
@@ -676,6 +705,7 @@ browser4-cli (Rust) ──MCP over HTTP──▶ browser4-rest (Kotlin/Spring)
 - **Backend** (`browser4-rest`) — Spring Boot server, dispatches MCP tools to browser drivers
 - **Browser driver** (`browser4-core/browser4-browser`) — wraps Chrome DevTools Protocol
 - **Agent tools** (`browser4-agentic`) — maps MCP tool names to browser automation methods
+- **Programming kernel** (`browser4-coding`) — dependency-light agent toolkit (sandboxed shell/filesystem, scaffolding, validation, self-development tools) — see [below](#-programming-agent-kernel-browser4-coding)
 
 ## 📦 Modules Overview
 
@@ -687,6 +717,7 @@ browser4-cli (Rust) ──MCP over HTTP──▶ browser4-rest (Kotlin/Spring)
 | `browser4-dependencies` | BOM and dependency version alignment                                 |
 | `browser4-tools` | Operational tools and launch helpers                                 |
 | `browser4-agentic` | AI agents, MCP integration, skill registration                       |
+| `browser4-coding` | Programming-agent kernel — sandboxed shell/fs, artifact scaffolding & validation, self-development tools (47 `coding.*` tools) |
 | `browser4-agent-tools` | High-level agent tools: scraping, crawling, stateful page interaction |
 | `browser4-rest` | Spring Boot REST layer & command endpoints                           |
 | `browser4-apps/browser4-standalone` | Product packaging — unified launcher (`target/Browser4.jar`)         |
@@ -697,6 +728,28 @@ browser4-cli (Rust) ──MCP over HTTP──▶ browser4-rest (Kotlin/Spring)
 
 ---
 
+## 🧩 Programming-Agent Kernel (browser4-coding)
+
+`browser4-coding` is the dependency-light programming kernel that lets an AI agent create Browser4 artifacts **and** develop Browser4 itself. It is independent of `browser4-agentic` and `pulsar-common` (only SLF4J + Jackson + coroutines), so it can be reused by non-agent hosts. Heavy backends (LSP servers, kotlin-compiler-embeddable) are probed at runtime and never downloaded by default.
+
+The `coding` domain exposes **47 tools** in four groups:
+
+| Group | Count | Highlights |
+|---|---|---|
+| Shell & filesystem | 28 | sandboxed `coding.shell` (command whitelist), snapshot-based edit primitives with `revert`, `diff` (Myers/Patience), repo-governance protection (`coding.protect`) |
+| Artifact creation & validation | 6 | `scaffold` (plugin/skill/js/script), `scaffoldFlow` (multi-file dev-flow), `scaffoldFromExample` (anti-staleness live templates, directory mode + stem-derived renames), `validate` (incl. `repo-consistency`) |
+| Self-development | 7 | `mvnBuild` (structured diagnostics), `ktSymbols`/`ktReferences`/`ktInheritance` (zero-dep Kotlin analysis), `impact` + `moduleGraph` (live pom graph), `devTask` (AGENTS.md flow + execution), `trapCheck` (CDP pitfalls) |
+| LSP | 4 | on-demand `diagnostics`/`symbols`/`references` for ts/js/py/rs (degrades gracefully when a server is missing) |
+
+**Generic vs project-specific**: the kernel is layered by *mechanism vs data* — diff, sandbox, LSP client, Kotlin analysis, Maven passthrough and the pom-graph scanner are generic and portable; the scaffolds, validators, `ModuleMap`, `CdpTrapCheck` and the governance defaults encode Browser4 conventions and are the layer to rewrite when reusing the kernel elsewhere.
+
+- Full tool reference & workflows: `skills/browser4-coding/SKILL.md`
+- Developing Browser4 itself: `skills/browser4-dev/SKILL.md`
+- Four-artifact comparison examples (real vs scaffold output): `docs-dev/copilot/examples/`
+- Evaluation summary (P1–P7): `docs-dev/copilot/browser4-programming-support-eval.md`
+
+---
+
 ## 🧪 Test Fixture Server (MockSite)
 
 Browser4 includes a lightweight **MockSite** server that serves static HTML pages for testing and demos. Start it from the repository root:

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "browser4-cli",
-  "version": "4.13.11",
+  "version": "4.14.0-rc.2",
   "description": "Browser automation CLI for AI agents",
   "type": "module",
   "files": [
@@ -9,8 +9,9 @@
     "skills"
   ],
   "bin": {
-    "browser4-cli": "bin/browser4-cli.js",
-    "browser4": "bin/browser4-cli.js"
+    "b4": "bin/browser4-cli.js",
+    "browser4": "bin/browser4-cli.js",
+    "browser4-cli": "bin/browser4-cli.js"
   },
   "scripts": {
     "sync:readme": "node scripts/sync-readme.mjs sync",

package/scripts/smoke-test-runtime-bundle.sh

@@ -39,8 +39,20 @@ cleanup() {
         wait "$HTTP_PID" 2>/dev/null || true
     fi
     if [ -n "${TEMP_DIR:-}" ] && [ -d "$TEMP_DIR" ]; then
-        rm -rf "$TEMP_DIR"
+        # On Windows the JVM/Chrome may hold jar and log handles open for a
+        # few seconds after kill-all ("Device or resource busy").  Retry
+        # briefly, and never let a cleanup failure override the script's
+        # PASS/FAIL exit status — the test result is what matters, and CI
+        # runners are ephemeral anyway.
+        local attempt
+        for attempt in 1 2 3 4 5; do
+            if rm -rf "$TEMP_DIR" 2>/dev/null; then
+                break
+            fi
+            sleep 2
+        done
     fi
+    return 0
 }
 trap cleanup EXIT
 
@@ -189,6 +201,13 @@ run_cli() {
     #   BROWSER4_RUNTIME_DIR  → where the installed runtime lives
     #   BROWSER4_CLI_STATE_DIR → where CLI session state is persisted
     #   BROWSER4_SERVER_LOG_DIR → where server startup logs go
+    #   BROWSER4_CLI_FORCE_REMOTE_BUNDLE → skip the "build local bundle from
+    #     the repo checkout" dev path.  Without this, the CLI prefers a
+    #     Maven rebuild of browser4-bundle when run inside a checkout
+    #     (and hard-fails when Maven cannot run, e.g. no/mismatched JDK).
+    #     The smoke test must exercise the *downloaded archive*, not a
+    #     locally rebuilt bundle, so force the installed-runtime path.
+    BROWSER4_CLI_FORCE_REMOTE_BUNDLE=1 \
     BROWSER4_CLI_STATE_DIR="$STATE_DIR" \
     BROWSER4_RUNTIME_DIR="$RUNTIME_DIR" \
     BROWSER4_SERVER_LOG_DIR="$SERVER_LOG_DIR" \