imperium-crawl 2.3.1 → 2.5.1

package/README.md

@@ -1,20 +1,34 @@
 <div align="center">
 
+<img src="assets/hero-banner.png" alt="imperium-crawl — 3-level auto-escalating stealth engine" width="800" />
+
 # imperium-crawl
 
 **The most powerful open-source CLI tool for web scraping, crawling, and data extraction.**
 
-28 tools. Zero API keys required. One `npx` command.
+39 tools. Zero API keys required. One `npx` command.
 
 [![npm version](https://img.shields.io/npm/v/imperium-crawl.svg)](https://www.npmjs.com/package/imperium-crawl)
 [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](./LICENSE)
-[![Tests](https://img.shields.io/badge/tests-466%20passing-brightgreen.svg)]()
+[![Tests](https://img.shields.io/badge/tests-580%20passing-brightgreen.svg)]()
 [![npm downloads](https://img.shields.io/npm/dm/imperium-crawl.svg)](https://www.npmjs.com/package/imperium-crawl)
 
 </div>
 
 ---
 
+## What's new in 2.5.0
+
+Three new tools for document extraction and content monitoring — all zero-API-key, all native:
+
+- **`pdf-extract`** — Pull text, pages, tables, and metadata from any PDF (local or remote) via `pdfjs-dist`. Ideal for regulatory docs, sustainability reports, invoices. Smoke-tested on a 98-page CBAM Guidance PDF (199K chars, confidence 0.99).
+- **`watch`** — Hash-based one-shot change detector for a single URL. Cron-friendly. Fires a webhook on change.
+- **`monitor`** — Multi-URL intelligence digest. Reads a JSON config grouping URLs by topic, emits a markdown digest filtered by minimum change percentage.
+
+See [CHANGELOG.md](./CHANGELOG.md) for the full release notes.
+
+---
+
 ## Quick Start
 
 Get running in 30 seconds.
@@ -31,7 +45,13 @@ npx -y imperium-crawl scrape --url https://example.com
 npm install -g imperium-crawl
 ```
 
-> That's it. 22 of 28 tools work with zero API keys. Add optional keys later to unlock search, AI extraction, and CAPTCHA solving.
+**Install from a local tarball** (e.g. pre-release testing):
+
+```bash
+npm install -g ./imperium-crawl-2.5.0.tgz
+```
+
+> That's it. 33 of 39 tools work with zero API keys. Add optional keys later to unlock search, AI extraction, and CAPTCHA solving.
 
 ---
 
@@ -107,7 +127,7 @@ Scraping 4 URLs (concurrency: 3)...
 ## Why imperium-crawl?
 
 🔓 **Zero API Keys Required**
-22 of 28 tools work out of the box. No accounts, no tokens, no credit cards. Just `npx` and go.
+33 of 39 tools work out of the box. No accounts, no tokens, no credit cards. Just `npx` and go.
 
 🛡️ **3-Level Auto-Escalating Stealth**
 Headers → TLS fingerprinting → headless browser + CAPTCHA solving. Automatically escalates until it gets through.
@@ -115,7 +135,7 @@ Headers → TLS fingerprinting → headless browser + CAPTCHA solving. Automatic
 🧠 **Self-Improving**
 Adaptive learning engine remembers what works per domain. Second visit is 3x faster. The more you use it, the smarter it gets.
 
-🧰 **28 Tools, 2 Modes**
+🧰 **33 Tools, 2 Modes**
 CLI tool or interactive TUI. Scraping, crawling, search, extraction, API discovery, WebSocket monitoring, browser automation, batch processing.
 
 📜 **14 Built-in Recipes**
@@ -131,7 +151,7 @@ Teach it once, run forever. Auto-detect patterns on any page, save as reusable s
 | Feature | **imperium-crawl** | Firecrawl | Crawl4AI | Browserbase | Puppeteer |
 |---------|:------------------:|:---------:|:--------:|:-----------:|:---------:|
 | Price | **Free forever** | $19+/month | Free | $0.01/min | Free |
-| Total tools | **28** | 5 | 2 | 4 | N/A |
+| Total tools | **33** | 5 | 2 | 4 | N/A |
 | Stealth levels | **3 (auto-escalate)** | Cloud-based | 1 | Cloud-based | None |
 | Anti-bot detection | **7 systems** | Partial | Partial | Partial | None |
 | TLS fingerprinting | **JA3/JA4** | No | No | No | No |
@@ -224,7 +244,7 @@ Second visit to cloudflare.com:
 
 ---
 
-## All 28 Tools
+## All 39 Tools
 
 ### 📄 Scraping (no API key needed)
 
@@ -272,7 +292,7 @@ Second visit to cloudflare.com:
 
 | Tool | What It Does |
 |------|-------------|
-| **interact** | Browser automation with 18 action types (click, type, scroll, wait, screenshot, evaluate, select, hover, press, navigate, drag, upload, storage, cookies, pdf, auth_login). Ref targeting via ARIA snapshot, session encryption, action policy, domain filter, network interception, device emulation. |
+| **interact** | Browser automation with 19 action types (click, type, scroll, wait, screenshot, evaluate, select, hover, press, navigate, drag, upload, storage, cookies, pdf, auth_login, refresh). Ref targeting via ARIA snapshot, session encryption, action policy, domain filter, network interception, device emulation. |
 | **snapshot** | ARIA-based page snapshot with interactive element refs. Use refs in interact for precise targeting. Annotated screenshots. |
 
 ### 📱 Social Media (no API key needed)
@@ -299,6 +319,69 @@ Second visit to cloudflare.com:
 | **job_status** | Full results for a specific batch job including per-URL outcomes. |
 | **delete_job** | Clean up completed or failed batch jobs. |
 
+### 🧠 Knowledge Engine (no API key needed)
+
+| Tool | What It Does |
+|------|-------------|
+| **knowledge** | Dump adaptive knowledge engine stats — per-domain success rates, optimal stealth levels, anti-bot detection history, rate limits. Use to debug scraping issues and understand problematic domains. |
+
+### 📄 Documents (no API key needed)
+
+| Tool | What It Does |
+|------|-------------|
+| **pdf_extract** | Extract text, pages, tables, and metadata from a local or remote PDF. Native text-layer strategy via `pdfjs-dist`. OCR + Claude Vision fallbacks deferred to v2.6.0. Use for sustainability reports, invoices, regulatory PDFs. |
+
+```bash
+imperium-crawl pdf-extract --input ./report.pdf --output ./extracted.json
+imperium-crawl pdf-extract --input https://example.com/report.pdf --max-pages 20
+```
+
+### 👀 Change Tracking (no API key needed)
+
+| Tool | What It Does |
+|------|-------------|
+| **watch** | One-shot change detector: scrape a URL, hash its content (readability / markdown / full), compare against the last snapshot, fire a webhook on change. Pair with cron for periodic monitoring. |
+| **monitor** | Portfolio-level change tracker across many URLs grouped by topic. Reads a JSON config, runs `watch` on each URL, emits a markdown digest filtered by minimum change percentage. |
+
+```bash
+# Watch a single URL — run periodically via cron
+imperium-crawl watch --url https://carbonchain.com/pricing \
+  --output-dir ./data/watch \
+  --webhook https://hooks.example.com/on-change
+
+# Monitor many URLs grouped by topic, emit a daily digest
+imperium-crawl monitor --config ./monitor.json --output-dir ./data/monitor
+```
+
+`monitor.json`:
+```json
+{
+  "topics": [
+    {
+      "name": "Competitor pricing",
+      "urls": ["https://carbonchain.com/pricing", "https://spherasolutions.com/cbam"]
+    }
+  ]
+}
+```
+
+### 🔁 Imperium Flows (no API key needed; browser workflows may require Playwright)
+
+| Tool | What It Does |
+|------|-------------|
+| **record_flow** | Record a headed browser workflow as a generic flow family/variant. Stores smart selector metadata and reusable input placeholders. |
+| **run_flow** | Run a saved flow with runtime JSON input, CAPTCHA policy, browser mode, and evidence collection. |
+| **serve_flow** | Expose saved flows through a local HTTP API. Requires bearer auth when bound publicly. |
+| **list_flows** | List project-local and global flow definitions. |
+| **inspect_flow** | Inspect a saved flow JSON definition. |
+| **validate_flow** | Validate a flow schema and report inputs, steps, and storage path. |
+
+```bash
+imperium-crawl record-flow --family generic-search --variant site-a --url https://example.com
+imperium-crawl run-flow generic-search/site-a --input '{"query":"example"}'
+imperium-crawl serve-flow generic-search --port 8787
+```
+
 ---
 
 ## Setup
@@ -366,6 +449,26 @@ imperium-crawl tui
 
 Interactive slash-command terminal with parameter prompts, table rendering, markdown display, and session state. Use `/save` to export results and `/again` to re-run the last command.
 
+### Explore REPL
+
+Interactively explore a site in a headed browser, then save the session as a reusable skill:
+
+```bash
+imperium-crawl explore https://example.com
+```
+
+```
+> navigate https://example.com/login
+> type "#email" "user@example.com"
+> type "#password" "{{env:MY_PASSWORD}}"
+> click "#submit"
+> snapshot
+> save-skill my-login
+✅ Saved skill: my-login (4 actions, 1 parameter detected)
+```
+
+Commands: `navigate`, `click`, `type`, `select`, `hover`, `press`, `scroll`, `wait`, `screenshot`, `snapshot`, `evaluate`, `save-skill`, `history`, `undo`, `status`, `help`, `exit`
+
 ---
 
 ## Skills & Recipes
@@ -389,6 +492,37 @@ run_skill({ name: "tc-ai-news" })
 
 Skills are saved in `~/.imperium-crawl/skills/` as JSON files — human-readable, editable, portable.
 
+### Skill Parameters
+
+Use template variables in skills — resolved at run time:
+
+```bash
+# In skill JSON actions:
+{ "value": "{{input:query}}" }           # passed via --params or prompted
+{ "value": "{{env:SITE_PASSWORD}}" }     # from environment variable
+{ "value": "{{computed:date_today}}" }   # auto-computed (date_today, timestamp, random_string, year, month, day)
+
+# Run with params:
+imperium-crawl run-skill my-search --params '{"query": "machine learning"}'
+```
+
+### Skill Chains
+
+Chain skills together — output of one step becomes input to the next:
+
+```json
+{
+  "type": "chain",
+  "name": "search-and-extract",
+  "steps": [
+    { "skill": "search-results", "output": "search" },
+    { "skill": "extract-details", "input": { "url": "$search.results[0].url" }, "output": "details" }
+  ]
+}
+```
+
+Variable syntax: `$step_name.field.nested[0]` — simple dot-path access, no eval.
+
 ### Built-in Recipes
 
 | Recipe | What It Does |
@@ -435,7 +569,7 @@ Turn any website into an API. No documentation needed.
 
 ## AI Agent Guide
 
-imperium-crawl ships with [`SKILL/`](./SKILL/) — a structured guide that teaches AI agents how to use all 28 tools effectively. Includes proven workflows, decision trees, error recovery, and advanced patterns.
+imperium-crawl ships with [`SKILL/`](./SKILL/) — a structured guide that teaches AI agents how to use all 39 tools effectively. Includes proven workflows, decision trees, error recovery, and advanced patterns.
 
 ### Two Ways to Connect
 
@@ -496,13 +630,14 @@ Every tool tested against production websites with real anti-bot defenses:
 | 📋 **list_jobs** | — | Batch jobs with status and progress |
 | 📊 **job_status** | Batch job | Full per-URL results with timing |
 | 🗑️ **delete_job** | Completed job | Cleaned up job data from disk |
+| 🧠 **knowledge** | Local knowledge file | Per-domain stats: stealth levels, success rates, anti-bot systems detected |
 | 🎬 **youtube** | "web scraping tutorial" | Search results, video details, comments, transcripts — no API key |
 | 💬 **reddit** | r/webscraping | Subreddit posts, comments, search — public JSON API |
 | 📸 **instagram** | @nike profile | Profile details, engagement rate, recent posts — internal API |
 | 📥 **download** | YouTube video, web page images | Auto-detect URL type, download media files — images, video, og:image |
 | 📡 **rss** | Hacker News RSS | Parsed feed items with title, link, date, author, categories |
 
-> **28/28 tools. 34 hidden APIs on Airbnb. Live BTC feed. Zero API keys for scraping.**
+> **39 tools. 34 hidden APIs on Airbnb. Live BTC feed. Reusable browser flows. Zero API keys for scraping.**
 
 ---
 
@@ -535,7 +670,7 @@ cd imperium-crawl
 npm install
 npm run build
 npm run dev         # Watch mode (rebuild on changes)
-npm test            # 466 tests
+npm test            # 546 tests
 npm start           # Start CLI (shows help or TUI)
 ```
 

package/dist/cli-explore.d.ts

@@ -0,0 +1,30 @@
+/**
+ * Explore REPL — interactive browser session with live Playwright.
+ *
+ * Usage: imperium-crawl explore <url>
+ *
+ * Opens a headed (visible) browser window and gives the user a readline REPL
+ * to execute actions interactively. Every successful action is recorded.
+ * At any point, run `save-skill <name>` to export the session as a reusable skill.
+ *
+ * Commands:
+ *   navigate <url>              Navigate to URL
+ *   click <selector>            Click element
+ *   type <selector> <text>      Fill input field
+ *   select <selector> <value>   Select option
+ *   wait [ms]                   Wait N ms (default 1000)
+ *   screenshot [file]           Save screenshot
+ *   snapshot                    Show ARIA tree + refs
+ *   evaluate <script>           Run JS in page
+ *   scroll [up|down] [px]       Scroll page
+ *   hover <selector>            Hover element
+ *   press <key>                 Press keyboard key
+ *   save-skill <name>           Export recording as skill JSON
+ *   status                      Show URL, action count
+ *   history                     List recorded actions
+ *   undo                        Remove last action
+ *   help                        Show command list
+ *   exit / quit                 Close browser and exit
+ */
+export declare function runExplore(startUrl: string, sessionId?: string): Promise<void>;
+//# sourceMappingURL=cli-explore.d.ts.map

package/dist/cli-explore.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"cli-explore.d.ts","sourceRoot":"","sources":["../src/cli-explore.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;GA2BG;AAgGH,wBAAsB,UAAU,CAAC,QAAQ,EAAE,MAAM,EAAE,SAAS,CAAC,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC,CA6RpF"}