imperium-crawl 2.5.3 → 2.6.0

package/README.md

@@ -1,12 +1,12 @@
 <div align="center">
 
-<img src="assets/hero-banner.png" alt="imperium-crawl — 3-level auto-escalating stealth engine" width="800" />
+<img src="assets/hero-banner.png" alt="imperium-crawl — 4-level auto-escalating stealth engine with CamoFox" width="800" />
 
 # imperium-crawl
 
 **The most powerful open-source CLI tool for web scraping, crawling, and data extraction.**
 
-39 tools. Zero API keys required. One `npx` command.
+41 tools. CamoFox C++ anti-detection. Zero API keys required. One `npx` command.
 
 [![npm version](https://img.shields.io/npm/v/imperium-crawl.svg)](https://www.npmjs.com/package/imperium-crawl)
 [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](./LICENSE)
@@ -17,16 +17,17 @@
 
 ---
 
-## What's new in 2.5.1
+## What's new in 2.6.0
 
-**Browser-based image extraction overhaul** — 100% coverage on any website:
+**CamoFox browser engine** — Firefox fork with C++ anti-fingerprinting that bypasses Cloudflare, Google, and most bot detection:
 
-- **Full browser rendering (L3)** for image discovery — JavaScript, lazy-load, shadow DOM, same-origin iframes
-- **7 image sources**: `<img>`, `<picture>`, CSS `background-image`, shadow DOM, JSON-LD, inline scripts, iframes
-- **Precise targeting**: `--selector`, `--index`, `--alt-match`, `--min-width`, `--max-width`
-- **Auto-click** "Load more" / "Gallery" buttons with multilingual keyword matching
-- **Referer injection** fixes 403 errors on image CDN anti-hotlink protection
-- **New `auto_click` action** in `interact` tool for standalone browser automation
+- **C++-level patches** — `navigator.hardwareConcurrency`, WebGL, AudioContext, WebRTC spoofed before JavaScript sees them
+- **Engine abstraction** — Switch between Playwright and CamoFox with a single `engine` flag
+- **Auto-update** — `imperiumcrawl camofox-update` pulls the latest CamoFox release from npm
+- **Engine factory** — `import { resolveEngine } from "imperium-crawl/engines"` for agent-native use
+- **Zero breaking changes** — Same tool API, same response format, same env vars. Just add `engine: "camofox"`.
+- **Codebase reorganized** — CLI in `src/cli/`, core in `src/core/`, tests in 14 category folders.
+- **41 tools total** — Added `camofox_status` and `camofox_update`.
 
 ```bash
 # Download ALL images from any page (100% coverage)
@@ -65,7 +66,7 @@ npm install -g imperium-crawl
 npm install -g ./imperium-crawl-2.5.2.tgz
 ```
 
-> That's it. 33 of 39 tools work with zero API keys. Add optional keys later to unlock search, AI extraction, and CAPTCHA solving.
+> That's it. 33 of 41 tools work with zero API keys. Add optional keys later to unlock search, AI extraction, and CAPTCHA solving.
 
 ---
 
@@ -169,7 +170,7 @@ Scraping 4 URLs (concurrency: 3)...
 ## Why imperium-crawl?
 
 🔓 **Zero API Keys Required**
-33 of 39 tools work out of the box. No accounts, no tokens, no credit cards. Just `npx` and go.
+33 of 41 tools work out of the box. No accounts, no tokens, no credit cards. Just `npx` and go.
 
 🛡️ **3-Level Auto-Escalating Stealth**
 Headers → TLS fingerprinting → headless browser + CAPTCHA solving. Automatically escalates until it gets through.
@@ -612,7 +613,7 @@ Turn any website into an API. No documentation needed.
 
 ## AI Agent Guide
 
-imperium-crawl ships with [`SKILL/`](./SKILL/) — a structured guide that teaches AI agents how to use all 39 tools effectively. Includes proven workflows, decision trees, error recovery, and advanced patterns.
+imperium-crawl ships with [`SKILL/`](./SKILL/) — a structured guide that teaches AI agents how to use all 41 tools effectively. Includes proven workflows, decision trees, error recovery, and advanced patterns.
 
 ### Two Ways to Connect
 
@@ -680,7 +681,7 @@ Every tool tested against production websites with real anti-bot defenses:
 | 📥 **download** | YouTube video, web page images | Auto-detect URL type, download media files — images, video, og:image |
 | 📡 **rss** | Hacker News RSS | Parsed feed items with title, link, date, author, categories |
 
-> **39 tools. 34 hidden APIs on Airbnb. Live BTC feed. Reusable browser flows. Zero API keys for scraping.**
+> **41 tools. 34 hidden APIs on Airbnb. Live BTC feed. Reusable browser flows. Zero API keys for scraping.**
 
 ---
 

package/dist/batch/job-store.js

@@ -1,6 +1,6 @@
 import fs from "node:fs/promises";
 import path from "node:path";
-import { getJobsDir } from "../config.js";
+import { getJobsDir } from "../core/config.js";
 export class JobStore {
     cache = new Map();
     dir;

package/dist/batch/job-store.js.map

@@ -1 +1 @@
-{"version":3,"file":"job-store.js","sourceRoot":"","sources":["../../src/batch/job-store.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,MAAM,kBAAkB,CAAC;AAClC,OAAO,IAAI,MAAM,WAAW,CAAC;AAC7B,OAAO,EAAE,UAAU,EAAE,MAAM,cAAc,CAAC;AAG1C,MAAM,OAAO,QAAQ;IACX,KAAK,GAAG,IAAI,GAAG,EAAoB,CAAC;IACpC,GAAG,CAAS;IAEpB,YAAY,GAAY;QACtB,IAAI,CAAC,GAAG,GAAG,GAAG,IAAI,UAAU,EAAE,CAAC;IACjC,CAAC;IAEO,OAAO,CAAC,EAAU;QACxB,wCAAwC;QACxC,MAAM,IAAI,GAAG,EAAE,CAAC,OAAO,CAAC,kBAAkB,EAAE,GAAG,CAAC,CAAC;QACjD,OAAO,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,GAAG,EAAE,GAAG,IAAI,OAAO,CAAC,CAAC;IAC7C,CAAC;IAED,KAAK,CAAC,IAAI,CAAC,GAAa;QACtB,MAAM,OAAO,GAAa,EAAE,GAAG,GAAG,EAAE,UAAU,EAAE,IAAI,IAAI,EAAE,CAAC,WAAW,EAAE,EAAE,CAAC;QAC3E,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,GAAG,CAAC,EAAE,EAAE,OAAO,CAAC,CAAC;QAEhC,MAAM,EAAE,CAAC,KAAK,CAAC,IAAI,CAAC,GAAG,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAC;QAC9C,MAAM,QAAQ,GAAG,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC;QACtC,MAAM,OAAO,GAAG,QAAQ,GAAG,MAAM,CAAC;QAClC,MAAM,EAAE,CAAC,SAAS,CAAC,OAAO,EAAE,IAAI,CAAC,SAAS,CAAC,OAAO,EAAE,IAAI,EAAE,CAAC,CAAC,EAAE,OAAO,CAAC,CAAC;QACvE,MAAM,EAAE,CAAC,MAAM,CAAC,OAAO,EAAE,QAAQ,CAAC,CAAC;IACrC,CAAC;IAED,KAAK,CAAC,IAAI,CAAC,EAAU;QACnB,IAAI,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,EAAE,CAAC;YAAE,OAAO,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,EAAE,CAAE,CAAC;QAEnD,IAAI,CAAC;YACH,MAAM,IAAI,GAAG,MAAM,EAAE,CAAC,QAAQ,CAAC,IAAI,CAAC,OAAO,CAAC,EAAE,CAAC,EAAE,OAAO,CAAC,CAAC;YAC1D,MAAM,GAAG,GAAG,IAAI,CAAC,KAAK,CAAC,IAAI,CAAa,CAAC;YACzC,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,EAAE,EAAE,GAAG,CAAC,CAAC;YACxB,OAAO,GAAG,CAAC;QACb,CAAC;QAAC,OAAO,GAAY,EAAE,CAAC;YACtB,MAAM,QAAQ,GACZ,GAAG;gBACH,OAAO,GAAG,KAAK,QAAQ;gBACvB,MAAM,IAAI,GAAG;gBACZ,GAA6B,CAAC,IAAI,KAAK,QAAQ,CAAC;YACnD,IAAI,CAAC,QAAQ,EAAE,CAAC;gBACd,OAAO,CAAC,KAAK,CACX,6BAA6B,EAC7B,GAAG,YAAY,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,CACjD,CAAC;YACJ,CAAC;YACD,OAAO,IAAI,CAAC;QACd,CAAC;IACH,CAAC;IAED,KAAK,CAAC,MAAM,CAAC,EAAU;QACrB,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC;QACtB,IAAI,CAAC;YACH,MAAM,EAAE,CAAC,MAAM,CAAC,IAAI,CAAC,OAAO,CAAC,EAAE,CAAC,CAAC,CAAC;QACpC,CAAC;QAAC,MAAM,CAAC;YACP,kCAAkC;QACpC,CAAC;IACH,CAAC;IAED,KAAK,CAAC,IAAI;QACR,IAAI,CAAC;YACH,MAAM,KAAK,GAAG,MAAM,EAAE,CAAC,OAAO,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;YACzC,OAAO,KAAK;iBACT,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,QAAQ,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,QAAQ,CAAC,WAAW,CAAC,CAAC;iBAC9D,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,OAAO,CAAC,SAAS,EAAE,EAAE,CAAC,CAAC,CAAC;QAC1C,CAAC;QAAC,MAAM,CAAC;YACP,OAAO,EAAE,CAAC;QACZ,CAAC;IACH,CAAC;CACF;AAED,kBAAkB;AAElB,IAAI,KAAK,GAAoB,IAAI,CAAC;AAElC,MAAM,UAAU,WAAW;IACzB,IAAI,CAAC,KAAK,EAAE,CAAC;QACX,KAAK,GAAG,IAAI,QAAQ,EAAE,CAAC;IACzB,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC;AAED,oCAAoC;AACpC,MAAM,UAAU,aAAa;IAC3B,KAAK,GAAG,IAAI,CAAC;AACf,CAAC"}
+{"version":3,"file":"job-store.js","sourceRoot":"","sources":["../../src/batch/job-store.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,MAAM,kBAAkB,CAAC;AAClC,OAAO,IAAI,MAAM,WAAW,CAAC;AAC7B,OAAO,EAAE,UAAU,EAAE,MAAM,mBAAmB,CAAC;AAG/C,MAAM,OAAO,QAAQ;IACX,KAAK,GAAG,IAAI,GAAG,EAAoB,CAAC;IACpC,GAAG,CAAS;IAEpB,YAAY,GAAY;QACtB,IAAI,CAAC,GAAG,GAAG,GAAG,IAAI,UAAU,EAAE,CAAC;IACjC,CAAC;IAEO,OAAO,CAAC,EAAU;QACxB,wCAAwC;QACxC,MAAM,IAAI,GAAG,EAAE,CAAC,OAAO,CAAC,kBAAkB,EAAE,GAAG,CAAC,CAAC;QACjD,OAAO,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,GAAG,EAAE,GAAG,IAAI,OAAO,CAAC,CAAC;IAC7C,CAAC;IAED,KAAK,CAAC,IAAI,CAAC,GAAa;QACtB,MAAM,OAAO,GAAa,EAAE,GAAG,GAAG,EAAE,UAAU,EAAE,IAAI,IAAI,EAAE,CAAC,WAAW,EAAE,EAAE,CAAC;QAC3E,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,GAAG,CAAC,EAAE,EAAE,OAAO,CAAC,CAAC;QAEhC,MAAM,EAAE,CAAC,KAAK,CAAC,IAAI,CAAC,GAAG,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAC;QAC9C,MAAM,QAAQ,GAAG,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC;QACtC,MAAM,OAAO,GAAG,QAAQ,GAAG,MAAM,CAAC;QAClC,MAAM,EAAE,CAAC,SAAS,CAAC,OAAO,EAAE,IAAI,CAAC,SAAS,CAAC,OAAO,EAAE,IAAI,EAAE,CAAC,CAAC,EAAE,OAAO,CAAC,CAAC;QACvE,MAAM,EAAE,CAAC,MAAM,CAAC,OAAO,EAAE,QAAQ,CAAC,CAAC;IACrC,CAAC;IAED,KAAK,CAAC,IAAI,CAAC,EAAU;QACnB,IAAI,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,EAAE,CAAC;YAAE,OAAO,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,EAAE,CAAE,CAAC;QAEnD,IAAI,CAAC;YACH,MAAM,IAAI,GAAG,MAAM,EAAE,CAAC,QAAQ,CAAC,IAAI,CAAC,OAAO,CAAC,EAAE,CAAC,EAAE,OAAO,CAAC,CAAC;YAC1D,MAAM,GAAG,GAAG,IAAI,CAAC,KAAK,CAAC,IAAI,CAAa,CAAC;YACzC,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,EAAE,EAAE,GAAG,CAAC,CAAC;YACxB,OAAO,GAAG,CAAC;QACb,CAAC;QAAC,OAAO,GAAY,EAAE,CAAC;YACtB,MAAM,QAAQ,GACZ,GAAG;gBACH,OAAO,GAAG,KAAK,QAAQ;gBACvB,MAAM,IAAI,GAAG;gBACZ,GAA6B,CAAC,IAAI,KAAK,QAAQ,CAAC;YACnD,IAAI,CAAC,QAAQ,EAAE,CAAC;gBACd,OAAO,CAAC,KAAK,CACX,6BAA6B,EAC7B,GAAG,YAAY,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,CACjD,CAAC;YACJ,CAAC;YACD,OAAO,IAAI,CAAC;QACd,CAAC;IACH,CAAC;IAED,KAAK,CAAC,MAAM,CAAC,EAAU;QACrB,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC;QACtB,IAAI,CAAC;YACH,MAAM,EAAE,CAAC,MAAM,CAAC,IAAI,CAAC,OAAO,CAAC,EAAE,CAAC,CAAC,CAAC;QACpC,CAAC;QAAC,MAAM,CAAC;YACP,kCAAkC;QACpC,CAAC;IACH,CAAC;IAED,KAAK,CAAC,IAAI;QACR,IAAI,CAAC;YACH,MAAM,KAAK,GAAG,MAAM,EAAE,CAAC,OAAO,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;YACzC,OAAO,KAAK;iBACT,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,QAAQ,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,QAAQ,CAAC,WAAW,CAAC,CAAC;iBAC9D,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,OAAO,CAAC,SAAS,EAAE,EAAE,CAAC,CAAC,CAAC;QAC1C,CAAC;QAAC,MAAM,CAAC;YACP,OAAO,EAAE,CAAC;QACZ,CAAC;IACH,CAAC;CACF;AAED,kBAAkB;AAElB,IAAI,KAAK,GAAoB,IAAI,CAAC;AAElC,MAAM,UAAU,WAAW;IACzB,IAAI,CAAC,KAAK,EAAE,CAAC;QACX,KAAK,GAAG,IAAI,QAAQ,EAAE,CAAC;IACzB,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC;AAED,oCAAoC;AACpC,MAAM,UAAU,aAAa;IAC3B,KAAK,GAAG,IAAI,CAAC;AACf,CAAC"}

package/dist/brave-api/index.js

@@ -1,4 +1,4 @@
-import { BRAVE_API_BASE } from "../constants.js";
+import { BRAVE_API_BASE } from "../core/constants.js";
 export class BraveApiError extends Error {
     status;
     constructor(status, message) {

package/dist/brave-api/index.js.map

@@ -1 +1 @@
-{"version":3,"file":"index.js","sourceRoot":"","sources":["../../src/brave-api/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,cAAc,EAAE,MAAM,iBAAiB,CAAC;AAEjD,MAAM,OAAO,aAAc,SAAQ,KAAK;IAE7B;IADT,YACS,MAAc,EACrB,OAAe;QAEf,KAAK,CAAC,OAAO,CAAC,CAAC;QAHR,WAAM,GAAN,MAAM,CAAQ;QAIrB,IAAI,CAAC,IAAI,GAAG,eAAe,CAAC;IAC9B,CAAC;CACF;AAED,MAAM,CAAC,KAAK,UAAU,YAAY,CAChC,MAAc,EACd,QAAgB,EAChB,MAAmD;IAEnD,MAAM,GAAG,GAAG,IAAI,GAAG,CAAC,GAAG,cAAc,GAAG,QAAQ,EAAE,CAAC,CAAC;IACpD,KAAK,MAAM,CAAC,GAAG,EAAE,KAAK,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,MAAM,CAAC,EAAE,CAAC;QAClD,IAAI,KAAK,KAAK,SAAS,EAAE,CAAC;YACxB,GAAG,CAAC,YAAY,CAAC,GAAG,CAAC,GAAG,EAAE,MAAM,CAAC,KAAK,CAAC,CAAC,CAAC;QAC3C,CAAC;IACH,CAAC;IAED,MAAM,GAAG,GAAG,MAAM,KAAK,CAAC,GAAG,CAAC,QAAQ,EAAE,EAAE;QACtC,OAAO,EAAE;YACP,MAAM,EAAE,kBAAkB;YAC1B,iBAAiB,EAAE,MAAM;YACzB,sBAAsB,EAAE,MAAM;SAC/B;QACD,MAAM,EAAE,WAAW,CAAC,OAAO,CAAC,MAAM,CAAC;KACpC,CAAC,CAAC;IAEH,IAAI,CAAC,GAAG,CAAC,EAAE,EAAE,CAAC;QACZ,MAAM,IAAI,GAAG,MAAM,GAAG,CAAC,IAAI,EAAE,CAAC,KAAK,CAAC,GAAG,EAAE,CAAC,eAAe,CAAC,CAAC;QAC3D,MAAM,IAAI,aAAa,CAAC,GAAG,CAAC,MAAM,EAAE,aAAa,GAAG,CAAC,MAAM,KAAK,IAAI,EAAE,CAAC,CAAC;IAC1E,CAAC;IAED,OAAO,GAAG,CAAC,IAAI,EAAE,CAAC;AACpB,CAAC"}
+{"version":3,"file":"index.js","sourceRoot":"","sources":["../../src/brave-api/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,cAAc,EAAE,MAAM,sBAAsB,CAAC;AAEtD,MAAM,OAAO,aAAc,SAAQ,KAAK;IAE7B;IADT,YACS,MAAc,EACrB,OAAe;QAEf,KAAK,CAAC,OAAO,CAAC,CAAC;QAHR,WAAM,GAAN,MAAM,CAAQ;QAIrB,IAAI,CAAC,IAAI,GAAG,eAAe,CAAC;IAC9B,CAAC;CACF;AAED,MAAM,CAAC,KAAK,UAAU,YAAY,CAChC,MAAc,EACd,QAAgB,EAChB,MAAmD;IAEnD,MAAM,GAAG,GAAG,IAAI,GAAG,CAAC,GAAG,cAAc,GAAG,QAAQ,EAAE,CAAC,CAAC;IACpD,KAAK,MAAM,CAAC,GAAG,EAAE,KAAK,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,MAAM,CAAC,EAAE,CAAC;QAClD,IAAI,KAAK,KAAK,SAAS,EAAE,CAAC;YACxB,GAAG,CAAC,YAAY,CAAC,GAAG,CAAC,GAAG,EAAE,MAAM,CAAC,KAAK,CAAC,CAAC,CAAC;QAC3C,CAAC;IACH,CAAC;IAED,MAAM,GAAG,GAAG,MAAM,KAAK,CAAC,GAAG,CAAC,QAAQ,EAAE,EAAE;QACtC,OAAO,EAAE;YACP,MAAM,EAAE,kBAAkB;YAC1B,iBAAiB,EAAE,MAAM;YACzB,sBAAsB,EAAE,MAAM;SAC/B;QACD,MAAM,EAAE,WAAW,CAAC,OAAO,CAAC,MAAM,CAAC;KACpC,CAAC,CAAC;IAEH,IAAI,CAAC,GAAG,CAAC,EAAE,EAAE,CAAC;QACZ,MAAM,IAAI,GAAG,MAAM,GAAG,CAAC,IAAI,EAAE,CAAC,KAAK,CAAC,GAAG,EAAE,CAAC,eAAe,CAAC,CAAC;QAC3D,MAAM,IAAI,aAAa,CAAC,GAAG,CAAC,MAAM,EAAE,aAAa,GAAG,CAAC,MAAM,KAAK,IAAI,EAAE,CAAC,CAAC;IAC1E,CAAC;IAED,OAAO,GAAG,CAAC,IAAI,EAAE,CAAC;AACpB,CAAC"}

package/dist/cli/config.d.ts

@@ -0,0 +1,21 @@
+/**
+ * Config file management for imperium-crawl CLI.
+ *
+ * Saves API keys to ~/.imperium-crawl/config.json so users don't
+ * need to set environment variables manually.
+ *
+ * Priority: process.env (system) > config.json
+ * applyCliConfig() fills in env vars from config only if not already set.
+ */
+export declare function getCliConfigPath(): string;
+export declare function loadCliConfig(): Record<string, string>;
+export declare function saveCliConfig(config: Record<string, string>): void;
+/**
+ * Apply config.json values to process.env.
+ * System env vars take priority — config values are only applied
+ * when the key is not already set.
+ *
+ * Call this once at startup, before any tool initialization.
+ */
+export declare function applyCliConfig(): void;
+//# sourceMappingURL=config.d.ts.map

package/dist/cli/config.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"config.d.ts","sourceRoot":"","sources":["../../src/cli/config.ts"],"names":[],"mappings":"AAAA;;;;;;;;GAQG;AASH,wBAAgB,gBAAgB,IAAI,MAAM,CAEzC;AAED,wBAAgB,aAAa,IAAI,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,CAWtD;AAED,wBAAgB,aAAa,CAAC,MAAM,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,GAAG,IAAI,CAIlE;AAED;;;;;;GAMG;AACH,wBAAgB,cAAc,IAAI,IAAI,CAOrC"}

package/dist/cli/config.js

@@ -0,0 +1,51 @@
+/**
+ * Config file management for imperium-crawl CLI.
+ *
+ * Saves API keys to ~/.imperium-crawl/config.json so users don't
+ * need to set environment variables manually.
+ *
+ * Priority: process.env (system) > config.json
+ * applyCliConfig() fills in env vars from config only if not already set.
+ */
+import path from "node:path";
+import os from "node:os";
+import { readFileSync, writeFileSync, mkdirSync } from "node:fs";
+import { SKILLS_DIR_NAME } from "../core/constants.js";
+const CONFIG_FILENAME = "config.json";
+export function getCliConfigPath() {
+    return path.join(os.homedir(), SKILLS_DIR_NAME, CONFIG_FILENAME);
+}
+export function loadCliConfig() {
+    try {
+        const content = readFileSync(getCliConfigPath(), "utf-8");
+        const parsed = JSON.parse(content);
+        if (parsed && typeof parsed === "object" && !Array.isArray(parsed)) {
+            return parsed;
+        }
+    }
+    catch {
+        // File doesn't exist or invalid JSON — return empty config
+    }
+    return {};
+}
+export function saveCliConfig(config) {
+    const configPath = getCliConfigPath();
+    mkdirSync(path.dirname(configPath), { recursive: true });
+    writeFileSync(configPath, JSON.stringify(config, null, 2) + "\n", "utf-8");
+}
+/**
+ * Apply config.json values to process.env.
+ * System env vars take priority — config values are only applied
+ * when the key is not already set.
+ *
+ * Call this once at startup, before any tool initialization.
+ */
+export function applyCliConfig() {
+    const config = loadCliConfig();
+    for (const [key, value] of Object.entries(config)) {
+        if (typeof value === "string" && !process.env[key]) {
+            process.env[key] = value;
+        }
+    }
+}
+//# sourceMappingURL=config.js.map

package/dist/cli/config.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"config.js","sourceRoot":"","sources":["../../src/cli/config.ts"],"names":[],"mappings":"AAAA;;;;;;;;GAQG;AAEH,OAAO,IAAI,MAAM,WAAW,CAAC;AAC7B,OAAO,EAAE,MAAM,SAAS,CAAC;AACzB,OAAO,EAAE,YAAY,EAAE,aAAa,EAAE,SAAS,EAAE,MAAM,SAAS,CAAC;AACjE,OAAO,EAAE,eAAe,EAAE,MAAM,sBAAsB,CAAC;AAEvD,MAAM,eAAe,GAAG,aAAa,CAAC;AAEtC,MAAM,UAAU,gBAAgB;IAC9B,OAAO,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC,OAAO,EAAE,EAAE,eAAe,EAAE,eAAe,CAAC,CAAC;AACnE,CAAC;AAED,MAAM,UAAU,aAAa;IAC3B,IAAI,CAAC;QACH,MAAM,OAAO,GAAG,YAAY,CAAC,gBAAgB,EAAE,EAAE,OAAO,CAAC,CAAC;QAC1D,MAAM,MAAM,GAAY,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC;QAC5C,IAAI,MAAM,IAAI,OAAO,MAAM,KAAK,QAAQ,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,MAAM,CAAC,EAAE,CAAC;YACnE,OAAO,MAAgC,CAAC;QAC1C,CAAC;IACH,CAAC;IAAC,MAAM,CAAC;QACP,2DAA2D;IAC7D,CAAC;IACD,OAAO,EAAE,CAAC;AACZ,CAAC;AAED,MAAM,UAAU,aAAa,CAAC,MAA8B;IAC1D,MAAM,UAAU,GAAG,gBAAgB,EAAE,CAAC;IACtC,SAAS,CAAC,IAAI,CAAC,OAAO,CAAC,UAAU,CAAC,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAC;IACzD,aAAa,CAAC,UAAU,EAAE,IAAI,CAAC,SAAS,CAAC,MAAM,EAAE,IAAI,EAAE,CAAC,CAAC,GAAG,IAAI,EAAE,OAAO,CAAC,CAAC;AAC7E,CAAC;AAED;;;;;;GAMG;AACH,MAAM,UAAU,cAAc;IAC5B,MAAM,MAAM,GAAG,aAAa,EAAE,CAAC;IAC/B,KAAK,MAAM,CAAC,GAAG,EAAE,KAAK,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,MAAM,CAAC,EAAE,CAAC;QAClD,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,EAAE,CAAC;YACnD,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,GAAG,KAAK,CAAC;QAC3B,CAAC;IACH,CAAC;AACH,CAAC"}

package/dist/cli/engine.d.ts

@@ -0,0 +1,17 @@
+/**
+ * CLI engine for imperium-crawl.
+ *
+ * Dynamically maps each tool's Zod schema to Commander options,
+ * runs execute(), then formats output via formatters.ts.
+ *
+ * TTY-aware: spinners + colored output + tables in interactive mode;
+ * clean JSON to stdout when piped (non-TTY / CI / NO_COLOR).
+ *
+ * Lazy loading: only the requested tool's module is imported at startup.
+ * This keeps cold-start time fast even though some tools pull in heavy
+ * deps (cheerio, linkedom, playwright, stealth engine).
+ */
+import { Command } from "commander";
+export declare function buildCli(): Promise<Command>;
+export declare function runCli(): Promise<void>;
+//# sourceMappingURL=engine.d.ts.map

package/dist/cli/engine.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"engine.d.ts","sourceRoot":"","sources":["../../src/cli/engine.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;GAYG;AAEH,OAAO,EAAE,OAAO,EAAU,MAAM,WAAW,CAAC;AA6U5C,wBAAsB,QAAQ,IAAI,OAAO,CAAC,OAAO,CAAC,CAwEjD;AA2ED,wBAAsB,MAAM,IAAI,OAAO,CAAC,IAAI,CAAC,CAG5C"}

package/dist/cli/engine.js

@@ -0,0 +1,440 @@
+/**
+ * CLI engine for imperium-crawl.
+ *
+ * Dynamically maps each tool's Zod schema to Commander options,
+ * runs execute(), then formats output via formatters.ts.
+ *
+ * TTY-aware: spinners + colored output + tables in interactive mode;
+ * clean JSON to stdout when piped (non-TTY / CI / NO_COLOR).
+ *
+ * Lazy loading: only the requested tool's module is imported at startup.
+ * This keeps cold-start time fast even though some tools pull in heavy
+ * deps (cheerio, linkedom, playwright, stealth engine).
+ */
+import { Command, Option } from "commander";
+import { z } from "zod";
+import { writeFileSync } from "fs";
+import { TOOL_MANIFEST } from "../tools/manifest.js";
+import { formatOutput, parseToolOutput, parseImageOutput, } from "../core/formatters.js";
+import { PACKAGE_VERSION } from "../core/constants.js";
+import { isTTY, createSpinner, errorMsg, renderTable, colorUrl, } from "./ui.js";
+/**
+ * Recursively unwrap ZodDefault, ZodOptional, ZodNullable
+ * to get the base type and metadata.
+ */
+function unwrapZod(schema) {
+    let isOptional = false;
+    let hasDefault = false;
+    let defaultValue = undefined;
+    const description = schema.description;
+    let current = schema;
+    for (let i = 0; i < 10; i++) {
+        const typeName = current._def.typeName;
+        if (typeName === "ZodDefault") {
+            hasDefault = true;
+            defaultValue = current._def.defaultValue();
+            current = current._def.innerType;
+        }
+        else if (typeName === "ZodOptional") {
+            isOptional = true;
+            current = current._def.innerType;
+        }
+        else if (typeName === "ZodNullable") {
+            isOptional = true;
+            current = current._def.innerType;
+        }
+        else {
+            break;
+        }
+    }
+    return { base: current, isOptional, hasDefault, defaultValue, description };
+}
+function getZodTypeName(schema) {
+    return schema._def.typeName ?? "";
+}
+// ── Name Converters ──────────────────────────────────────────────────
+/** snake_case → --kebab-case */
+function snakeToKebab(key) {
+    return key.replace(/_/g, "-");
+}
+/** camelCase → snake_case (Commander's camelCase back to Zod's keys) */
+function camelToSnake(key) {
+    return key.replace(/[A-Z]/g, (m) => "_" + m.toLowerCase());
+}
+// ── Zod-to-Commander Option Mapper ───────────────────────────────────
+function addOptionsFromSchema(cmd, schema) {
+    if (getZodTypeName(schema) !== "ZodObject")
+        return;
+    const shape = schema.shape;
+    for (const [key, fieldSchema] of Object.entries(shape)) {
+        const { base, isOptional, hasDefault, defaultValue, description, } = unwrapZod(fieldSchema);
+        const flag = snakeToKebab(key);
+        const desc = description ?? key;
+        const typeName = getZodTypeName(base);
+        const required = !isOptional && !hasDefault;
+        if (typeName === "ZodString") {
+            if (required) {
+                cmd.requiredOption(`--${flag} <value>`, desc);
+            }
+            else {
+                const opt = new Option(`--${flag} <value>`, desc);
+                if (hasDefault)
+                    opt.default(defaultValue);
+                cmd.addOption(opt);
+            }
+        }
+        else if (typeName === "ZodNumber") {
+            const parseNum = (v) => {
+                const n = parseFloat(v);
+                if (isNaN(n))
+                    throw new Error(`Invalid number: ${v}`);
+                return n;
+            };
+            if (required) {
+                cmd.requiredOption(`--${flag} <n>`, desc, parseNum);
+            }
+            else {
+                const opt = new Option(`--${flag} <n>`, desc);
+                opt.argParser(parseNum);
+                if (hasDefault)
+                    opt.default(defaultValue);
+                cmd.addOption(opt);
+            }
+        }
+        else if (typeName === "ZodBoolean") {
+            if (hasDefault && defaultValue === true) {
+                cmd.option(`--no-${flag}`, desc);
+            }
+            else {
+                cmd.option(`--${flag}`, desc, hasDefault ? defaultValue : false);
+            }
+        }
+        else if (typeName === "ZodEnum") {
+            const values = base._def.values;
+            const opt = new Option(`--${flag} <value>`, desc).choices(values);
+            if (hasDefault)
+                opt.default(defaultValue);
+            if (required)
+                opt.makeOptionMandatory(true);
+            cmd.addOption(opt);
+        }
+        else if (typeName === "ZodArray") {
+            const collect = (val, prev) => prev.concat(val);
+            const innerTypeName = getZodTypeName(unwrapZod(base._def.type).base);
+            if (innerTypeName === "ZodEnum") {
+                const values = unwrapZod(base._def.type).base._def
+                    .values;
+                const opt = new Option(`--${flag} <value>`, desc)
+                    .choices(values)
+                    .argParser(collect)
+                    .default(hasDefault ? defaultValue : []);
+                cmd.addOption(opt);
+            }
+            else if (innerTypeName === "ZodObject") {
+                // Parse as single JSON array string (e.g. --actions '[{...},{...}]')
+                const parseJsonArray = (val) => {
+                    try {
+                        const parsed = JSON.parse(val);
+                        if (!Array.isArray(parsed))
+                            throw new Error("Expected JSON array");
+                        return parsed;
+                    }
+                    catch {
+                        throw new Error(`Invalid JSON array for --${flag}: ${val}`);
+                    }
+                };
+                if (required) {
+                    cmd.requiredOption(`--${flag} <json>`, desc, parseJsonArray);
+                }
+                else {
+                    const opt = new Option(`--${flag} <json>`, desc);
+                    opt.argParser(parseJsonArray);
+                    if (hasDefault)
+                        opt.default(defaultValue);
+                    cmd.addOption(opt);
+                }
+            }
+            else {
+                cmd.option(`--${flag} <value>`, desc, collect, (hasDefault ? defaultValue : []));
+            }
+        }
+        else if (typeName === "ZodRecord") {
+            const parseJson = (val) => {
+                try {
+                    return JSON.parse(val);
+                }
+                catch {
+                    throw new Error(`Invalid JSON for --${flag}: ${val}`);
+                }
+            };
+            if (required) {
+                cmd.requiredOption(`--${flag} <json>`, desc, parseJson);
+            }
+            else {
+                const opt = new Option(`--${flag} <json>`, desc);
+                opt.argParser(parseJson);
+                if (hasDefault)
+                    opt.default(defaultValue);
+                cmd.addOption(opt);
+            }
+        }
+        else {
+            cmd.option(`--${flag} <value>`, desc);
+        }
+    }
+}
+// ── Reverse Key Mapper ───────────────────────────────────────────────
+function optsToInput(opts, schema) {
+    if (getZodTypeName(schema) !== "ZodObject")
+        return {};
+    const shape = schema.shape;
+    const schemaKeys = new Set(Object.keys(shape));
+    const input = {};
+    for (const [camelKey, value] of Object.entries(opts)) {
+        if (value === undefined)
+            continue;
+        const snakeKey = camelToSnake(camelKey);
+        if (schemaKeys.has(snakeKey)) {
+            input[snakeKey] = value;
+        }
+        else if (schemaKeys.has(camelKey)) {
+            input[camelKey] = value;
+        }
+    }
+    return input;
+}
+// ── Lazy Tool Loader ─────────────────────────────────────────────────
+/**
+ * Which tool command the user requested (from argv), or null for
+ * global commands (--help, setup, --version).
+ *
+ * Detected before Commander runs so we can load only that tool's module.
+ */
+function getRequestedCmd() {
+    const arg = process.argv[2];
+    if (!arg || arg.startsWith("-"))
+        return null;
+    return arg; // e.g., "scrape", "list-jobs", "setup"
+}
+function positionalKeyForCmd(cmd) {
+    if (["run-flow", "inspect-flow", "validate-flow"].includes(cmd))
+        return "flow";
+    if (cmd === "serve-flow")
+        return "family";
+    return null;
+}
+// ── Table Renderers ──────────────────────────────────────────────────
+function tryRenderTable(toolName, data) {
+    if (!isTTY)
+        return null;
+    if (!data || typeof data !== "object")
+        return null;
+    const obj = data;
+    if (toolName === "list_jobs") {
+        const jobs = Array.isArray(obj.jobs) ? obj.jobs : [];
+        if (jobs.length === 0)
+            return null;
+        const headers = ["Job ID", "Status", "URLs", "Progress"];
+        const rows = jobs.map((j) => {
+            const job = j;
+            const total = Number(job.urls_total ?? 0);
+            const done = Number(job.urls_completed ?? 0);
+            const failed = Number(job.urls_failed ?? 0);
+            const pct = total > 0 ? Math.round(((done + failed) / total) * 100) : 0;
+            return [String(job.id ?? ""), String(job.status ?? ""), `${done + failed}/${total}`, `${pct}%`];
+        });
+        return renderTable(headers, rows);
+    }
+    if (toolName === "list_skills") {
+        const skills = Array.isArray(obj.skills) ? obj.skills : [];
+        if (skills.length === 0)
+            return null;
+        const headers = ["Name", "Type", "URL", "Fields", "Created"];
+        const rows = skills.map((s) => {
+            const skill = s;
+            const fields = Array.isArray(skill.fields) ? skill.fields.join(", ") : "";
+            const created = typeof skill.created_at === "string" ? skill.created_at.split("T")[0] : "";
+            const toolType = String(skill.tool ?? "extract");
+            const typeLabel = skill.builtin ? `${toolType} (built-in)` : toolType;
+            return [String(skill.name ?? ""), typeLabel, colorUrl(String(skill.url ?? "")), fields, created];
+        });
+        return renderTable(headers, rows);
+    }
+    if (toolName === "knowledge") {
+        const domains = Array.isArray(obj.domains) ? obj.domains : [];
+        if (domains.length === 0)
+            return null;
+        const headers = ["Domain", "Level", "Success%", "Reqs", "Rate/RPM", "AntiBot", "Updated"];
+        const rows = domains.map((d) => {
+            const dom = d;
+            return [
+                String(dom.domain ?? ""),
+                `L${dom.optimal_level ?? 1}`,
+                String(dom.success_rate ?? "0%"),
+                String(dom.requests ?? 0),
+                String(dom.rate_limit_rpm ?? 60),
+                String(dom.antibot ?? "none"),
+                String(dom.last_updated ?? ""),
+            ];
+        });
+        return renderTable(headers, rows);
+    }
+    if (["search", "news_search", "image_search", "video_search"].includes(toolName)) {
+        const results = Array.isArray(obj.results) ? obj.results : [];
+        if (results.length === 0)
+            return null;
+        const headers = ["#", "Title", "URL"];
+        const rows = results.slice(0, 20).map((r, i) => {
+            const result = r;
+            const title = String(result.title ?? result.name ?? "").slice(0, 50);
+            const url = colorUrl(String(result.url ?? result.source ?? "").slice(0, 60));
+            return [String(i + 1), title, url];
+        });
+        return renderTable(headers, rows);
+    }
+    return null;
+}
+// ── Contextual Error Messages ────────────────────────────────────────
+function handleToolError(toolName, msg) {
+    errorMsg(msg);
+    const needsKey = msg.includes("API_KEY") ||
+        msg.toLowerCase().includes("api key") ||
+        ["search", "news_search", "image_search", "video_search", "ai_extract"].includes(toolName);
+    if (needsKey) {
+        process.stderr.write("\n  Run \x1b[1mimperiumcrawl setup\x1b[0m to configure API keys.\n\n");
+    }
+}
+// ── Program Builder ──────────────────────────────────────────────────
+export async function buildCli() {
+    const program = new Command();
+    program
+        .name("imperiumcrawl")
+        .description("39-tool web scraping, crawling, search, media download, API discovery, and reusable browser workflow CLI.\nRun without arguments in TTY for interactive TUI.")
+        .version(PACKAGE_VERSION)
+        .addOption(new Option("--output-format <fmt>", "Output format")
+        .choices(["json", "jsonl", "csv", "markdown"])
+        .default("json"))
+        .option("--output <file>", "Write output to file instead of stdout")
+        .option("--pretty", "Pretty-print JSON output", false);
+    // ── Setup wizard ────────────────────────────────────────────────
+    program
+        .command("setup")
+        .description("Configure API keys and save them to ~/.imperium-crawl/config.json")
+        .action(async () => {
+        const { runSetup } = await import("./onboarding.js");
+        await runSetup();
+    });
+    // ── Explore REPL — interactive browser session with recording ──
+    program
+        .command("explore <url>")
+        .description("Open an interactive browser REPL. Navigate, click, type — every action is recorded. Run save-skill inside to export the session as a reusable skill.")
+        .option("--session-id <id>", "Restore and save cookies for this session ID")
+        .action(async (url, opts) => {
+        const { runExplore } = await import("./explore.js");
+        await runExplore(url, opts.sessionId);
+    });
+    // ── Tool commands — lazy loaded ──────────────────────────────────
+    const requestedCmd = getRequestedCmd();
+    for (const { cmd, description } of TOOL_MANIFEST) {
+        const sub = program.command(cmd).description(description);
+        const positionalKey = positionalKeyForCmd(cmd);
+        if (positionalKey === "flow")
+            sub.argument("[flow]", "Flow reference in '<family>/<variant>' form");
+        if (positionalKey === "family")
+            sub.argument("[family]", "Optional flow family to expose");
+        if (cmd === requestedCmd) {
+            // Load the full tool module for this command only.
+            // This is the only heavy import at startup — all others are skipped.
+            const toolModule = await import(`./tools/${cmd}.js`);
+            addOptionsFromSchema(sub, toolModule.schema);
+            sub.action(async (...args) => {
+                const last = args[args.length - 1];
+                const localOpts = typeof last.opts === "function"
+                    ? last.opts()
+                    : last;
+                if (positionalKey && typeof args[0] === "string")
+                    localOpts[positionalKey] = args[0];
+                await runTool(toolModule, localOpts, program);
+            });
+        }
+        else {
+            // For all other commands: register with just name + description.
+            // Options are loaded lazily if the command is somehow invoked.
+            sub.action(async (...args) => {
+                const toolModule = await import(`./tools/${cmd}.js`);
+                // Re-parse options now that we have the schema
+                const rawOpts = sub.opts();
+                if (positionalKey && typeof args[0] === "string")
+                    rawOpts[positionalKey] = args[0];
+                await runTool(toolModule, rawOpts, program);
+            });
+        }
+    }
+    return program;
+}
+async function runTool(tool, localOpts, program) {
+    const globalOpts = program.opts();
+    const input = optsToInput(localOpts, tool.schema);
+    let validated;
+    try {
+        validated = tool.schema.parse(input);
+    }
+    catch (err) {
+        if (err instanceof z.ZodError) {
+            const issues = err.issues
+                .map((i) => `  ${i.path.join(".")}: ${i.message}`)
+                .join("\n");
+            process.stderr.write(`Validation error:\n${issues}\n`);
+            process.exit(1);
+        }
+        throw err;
+    }
+    const spinner = createSpinner(`Running ${tool.name}...`);
+    const start = Date.now();
+    let result;
+    try {
+        result = await tool.execute(validated);
+        spinner.succeed(`Done in ${((Date.now() - start) / 1000).toFixed(1)}s`);
+    }
+    catch (err) {
+        const msg = err instanceof Error ? err.message : String(err);
+        spinner.fail(`Failed: ${msg}`);
+        handleToolError(tool.name, msg);
+        process.exit(1);
+    }
+    // Handle image responses (screenshot tool, interact with return_screenshot)
+    const imageResult = parseImageOutput(result);
+    if (imageResult) {
+        const imgPath = `screenshot-${Date.now()}.png`;
+        writeFileSync(imgPath, Buffer.from(imageResult.data, "base64"));
+        process.stderr.write(`Screenshot saved to ${imgPath}\n`);
+    }
+    const data = parseToolOutput(result);
+    // Image-only response (no text data) — we're done
+    if (imageResult && data === null)
+        return;
+    const table = tryRenderTable(tool.name, data);
+    if (table) {
+        process.stdout.write(table + "\n");
+        return;
+    }
+    const formatOptions = {
+        format: globalOpts.outputFormat,
+        pretty: globalOpts.pretty,
+    };
+    const output = formatOutput(data, formatOptions);
+    const outputFile = globalOpts.output;
+    if (outputFile) {
+        writeFileSync(outputFile, output + "\n", "utf-8");
+        process.stderr.write(`Written to ${outputFile}\n`);
+    }
+    else {
+        process.stdout.write(output + "\n");
+    }
+}
+// ── Entry Point ──────────────────────────────────────────────────────
+export async function runCli() {
+    const program = await buildCli();
+    await program.parseAsync(process.argv);
+}
+//# sourceMappingURL=engine.js.map