edgecrawl 0.3.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Koji Hayashida
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,260 @@
+# edgecrawl
+
+Local AI-powered web scraper. Extract structured JSON from any website using on-device ONNX LLMs. No API keys, no cloud, no Python.
+
+## Features
+
+- **100% Local AI** — Runs Qwen3 ONNX models on your machine via Transformers.js v4 (WebGPU/WASM)
+- **Zero API Keys** — No OpenAI, no Anthropic, no cloud bills. Everything runs on-device
+- **Structured JSON Output** — Define a schema, get clean JSON back
+- **Smart Extraction** — Tries JSON-LD/Open Graph first, falls back to LLM only when needed
+- **CLI + Library** — Use from the command line or import into your Node.js app
+
+## Architecture
+
+```
+Playwright (headless browser)
+  |
+  v
+JSDOM + node-html-markdown -> Clean Markdown
+  |
+  v
+Qwen3 ONNX (Transformers.js v4) -> Structured JSON
+```
+
+## Quick Start
+
+### Install from npm
+
+```bash
+npm install edgecrawl
+npx playwright install chromium
+```
+
+### Install from source
+
+```bash
+git clone https://github.com/couzip/edgecrawl.git
+cd edgecrawl
+npm install
+npx playwright install chromium
+```
+
+Models are downloaded automatically on first run:
+- LLM: Qwen3 ONNX (0.4-2.5 GB depending on preset)
+
+### CLI
+
+```bash
+# Extract structured data from a URL
+edgecrawl extract https://example.com
+
+# With custom schema
+edgecrawl extract https://example.com -s schemas/product.json -o result.json
+
+# Light model on WASM
+edgecrawl extract https://example.com -p light -d wasm
+
+# Convert to Markdown only (no LLM)
+edgecrawl md https://example.com
+```
+
+### Library
+
+```javascript
+import { scrapeAndExtract, cleanup } from "edgecrawl";
+
+const result = await scrapeAndExtract("https://example.com", {
+  preset: "balanced",
+});
+
+console.log(result.extracted);
+await cleanup();
+```
+
+## CLI Commands
+
+### `extract <url>` — Structured extraction
+
+```bash
+# Default (balanced model, WebGPU)
+edgecrawl extract https://example.com
+
+# Light model on WASM
+edgecrawl extract https://example.com -p light -d wasm
+
+# Custom schema + output file
+edgecrawl extract https://example.com -s schemas/product.json -o result.json
+
+# Target a specific section
+edgecrawl extract https://example.com --selector "main article"
+```
+
+### `batch <file>` — Batch processing
+
+```bash
+# Process URL list (one URL per line)
+edgecrawl batch urls.txt -o results.json
+
+# With concurrency control
+edgecrawl batch urls.txt -c 5
+```
+
+### `query <url> <prompt>` — Custom question
+
+```bash
+# Ask a question about page content
+edgecrawl query https://example.com "What are the main products?"
+```
+
+### `md <url>` — Markdown conversion only (no LLM)
+
+```bash
+edgecrawl md https://example.com
+edgecrawl md https://example.com -o page.md --scroll
+```
+
+## CLI Options
+
+### Common Options (extract, batch, query)
+
+| Option | Description | Default |
+|--------|-------------|---------|
+| `-p, --preset <preset>` | Model preset: `light` / `balanced` / `quality` | `balanced` |
+| `-d, --device <device>` | Inference device: `webgpu` / `wasm` | `webgpu` |
+| `-s, --schema <file>` | Custom schema JSON file | built-in default |
+| `-o, --output <file>` | Output file path | stdout |
+| `-t, --max-tokens <n>` | Max input tokens for LLM | `2048` |
+| `--selector <selector>` | CSS selector to narrow target content | - |
+
+### Batch Options
+
+| Option | Description | Default |
+|--------|-------------|---------|
+| `-c, --concurrency <n>` | Concurrent scraping limit | `3` |
+
+### Browser Options (all commands)
+
+| Option | Description | Default |
+|--------|-------------|---------|
+| `--headful` | Show browser window (for debugging) | `false` |
+| `--user-agent <ua>` | Custom User-Agent string | - |
+| `--timeout <ms>` | Page load timeout in milliseconds | `30000` |
+| `--proxy <url>` | Proxy server URL | - |
+| `--cookie <cookie>` | Cookie in `name=value` format (repeatable) | - |
+| `--extra-header <header>` | HTTP header in `Key:Value` format (repeatable) | - |
+| `--viewport <WxH>` | Viewport size | `1280x800` |
+| `--wait-until <event>` | Navigation wait condition: `load` / `domcontentloaded` / `networkidle` | `load` |
+| `--no-block-media` | Disable blocking of images/fonts/media | `false` |
+| `--scroll` | Scroll to bottom (for lazy-loaded content) | `false` |
+| `--wait <selector>` | Wait for CSS selector to appear | - |
+
+## Library Usage
+
+### High-level Pipeline
+
+```javascript
+import {
+  scrapeAndExtract,
+  batchScrapeAndExtract,
+  scrapeAndQuery,
+  cleanup,
+} from "edgecrawl";
+
+// Basic extraction
+const result = await scrapeAndExtract("https://example.com");
+
+// Custom schema
+const product = await scrapeAndExtract("https://shop.example.com/item", {
+  schema: {
+    title: "Product name (string)",
+    price: "Current price (string)",
+    features: "Key features (array of strings)",
+  },
+});
+
+// Batch processing
+const results = await batchScrapeAndExtract(
+  ["https://example.com/1", "https://example.com/2"],
+  { concurrency: 3 }
+);
+
+// Custom query
+const answer = await scrapeAndQuery(
+  "https://example.com",
+  "What are the main products?",
+  { preset: "quality" }
+);
+
+await cleanup();
+```
+
+### Low-level APIs
+
+```javascript
+// Use individual modules
+import { htmlToMarkdown, cleanMarkdown } from "edgecrawl/html2md";
+import { launchBrowser, fetchPage, closeBrowser } from "edgecrawl/scraper";
+import { initLLM, extractStructured } from "edgecrawl/llm";
+
+// HTML to Markdown only
+await launchBrowser();
+const { html } = await fetchPage("https://example.com");
+const { markdown, title } = htmlToMarkdown(html, "https://example.com");
+const cleaned = cleanMarkdown(markdown);
+await closeBrowser();
+
+// Or use the root export
+import { htmlToMarkdown, cleanMarkdown, fetchPage } from "edgecrawl";
+```
+
+## Custom Schemas
+
+Define what data to extract by providing a JSON schema file:
+
+```json
+{
+  "title": "Product name (string)",
+  "price": "Current price (string)",
+  "currency": "Currency code (string)",
+  "description": "Product description (string)",
+  "features": "Key features (array of strings)",
+  "availability": "In stock or out of stock (string)"
+}
+```
+
+```bash
+edgecrawl extract https://shop.example.com/product -s schema.json
+```
+
+See the `schemas/` directory for more examples.
+
+## Model Presets
+
+| Preset | Model | Size | Speed | Quality |
+|--------|-------|------|-------|---------|
+| `light` | Qwen3-0.6B | ~0.4 GB | Fast | Good for simple pages |
+| `balanced` | Qwen3-1.7B | ~1.2 GB | Medium | Best balance (default) |
+| `quality` | Qwen3-4B | ~2.5 GB | Slower | Best accuracy |
+
+All models run locally via ONNX Runtime. First run downloads the model to `.model-cache/`.
+
+## Tech Stack
+
+| Component | Library | Role |
+|-----------|---------|------|
+| Browser | Playwright | Headless scraping |
+| HTML -> Markdown | JSDOM + node-html-markdown | Content cleaning + Markdown conversion |
+| LLM | Transformers.js v4 + Qwen3 ONNX | Local structured extraction |
+| CLI | Commander.js | Command-line interface |
+
+## Requirements
+
+- Node.js >= 20.0.0
+- Chromium (installed via `npx playwright install chromium`)
+- ~1-3 GB disk space for models (downloaded on first run)
+- GPU recommended for WebGPU mode (falls back to WASM/CPU)
+
+## License
+
+MIT

package/index.mjs

@@ -0,0 +1,22 @@
+// index.mjs — edgecrawl package entry point
+
+// High-level pipeline functions
+export {
+  scrapeAndExtract,
+  batchScrapeAndExtract,
+  scrapeAndQuery,
+  cleanup,
+  DEFAULT_SCHEMA,
+} from "./src/pipeline.mjs";
+
+// LLM utilities
+export { initLLM, extractStructured, queryLLM, MODEL_PRESETS } from "./src/llm.mjs";
+
+// HTML-to-Markdown conversion
+export { htmlToMarkdown, cleanMarkdown, truncateForLLM } from "./src/html2md.mjs";
+
+// Structured data extraction (JSON-LD / Open Graph)
+export { tryStructuredExtract } from "./src/structured-extract.mjs";
+
+// Browser / scraping primitives
+export { launchBrowser, fetchPage, fetchPages, closeBrowser } from "./src/scraper.mjs";

package/package.json

@@ -0,0 +1,63 @@
+{
+  "name": "edgecrawl",
+  "version": "0.3.0",
+  "description": "Local AI-powered web scraper. Extract structured JSON from any website using on-device ONNX LLMs. No API keys, no cloud, no Python.",
+  "type": "module",
+  "main": "./index.mjs",
+  "exports": {
+    ".": "./index.mjs",
+    "./scraper": "./src/scraper.mjs",
+    "./html2md": "./src/html2md.mjs",
+    "./llm": "./src/llm.mjs",
+    "./structured-extract": "./src/structured-extract.mjs"
+  },
+  "bin": {
+    "edgecrawl": "./src/cli.mjs"
+  },
+  "files": [
+    "index.mjs",
+    "src/",
+    "schemas/",
+    "LICENSE",
+    "README.md"
+  ],
+  "scripts": {
+    "extract": "node src/cli.mjs extract",
+    "batch": "node src/cli.mjs batch",
+    "query": "node src/cli.mjs query",
+    "md": "node src/cli.mjs md"
+  },
+  "keywords": [
+    "scraper",
+    "web-scraping",
+    "llm",
+    "onnx",
+    "local-ai",
+    "structured-data",
+    "json",
+    "headless",
+    "playwright",
+    "edge-computing",
+    "transformers"
+  ],
+  "author": "Koji Hayashida",
+  "license": "MIT",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/couzip/edgecrawl.git"
+  },
+  "homepage": "https://github.com/couzip/edgecrawl#readme",
+  "bugs": {
+    "url": "https://github.com/couzip/edgecrawl/issues"
+  },
+  "engines": {
+    "node": ">=20.0.0"
+  },
+  "dependencies": {
+    "@huggingface/transformers": "4.0.0-next.3",
+    "commander": "^12.0.0",
+    "jsdom": "^25.0.0",
+    "node-html-markdown": "^2.0.0",
+    "playwright": "^1.48.0"
+  }
+}

package/schemas/article.json

@@ -0,0 +1,15 @@
+{
+  "type": "object",
+  "properties": {
+    "title": { "type": "string", "description": "Article title" },
+    "author": { "type": "string", "description": "Author name" },
+    "published_date": { "type": "string", "description": "Publication date" },
+    "summary": { "type": "string", "description": "Summary of the article (3-5 sentences)" },
+    "tags": {
+      "type": "array",
+      "items": { "type": "string" },
+      "description": "Topics or tags"
+    }
+  },
+  "required": ["title", "summary"]
+}

package/schemas/news.json

@@ -0,0 +1,19 @@
+{
+  "type": "object",
+  "properties": {
+    "articles": {
+      "type": "array",
+      "items": {
+        "type": "object",
+        "properties": {
+          "title": { "type": "string", "description": "Article headline" },
+          "url": { "type": "string", "description": "Article URL" },
+          "summary": { "type": "string", "description": "Brief summary (if available)" },
+          "date": { "type": "string", "description": "Published date (if available)" }
+        },
+        "required": ["title", "url"]
+      }
+    }
+  },
+  "required": ["articles"]
+}

package/schemas/product.json

@@ -0,0 +1,20 @@
+{
+  "type": "object",
+  "properties": {
+    "name": { "type": "string", "description": "Product name" },
+    "brand": { "type": "string", "description": "Brand or manufacturer" },
+    "price": { "type": "number", "description": "Price (numeric)" },
+    "currency": { "type": "string", "description": "Currency code (e.g. USD, EUR, JPY)" },
+    "availability": { "type": "string", "description": "Stock status (in stock, out of stock, etc.)" },
+    "rating": { "type": "number", "description": "Rating score (e.g. 4.5)" },
+    "review_count": { "type": "integer", "description": "Number of reviews" },
+    "description": { "type": "string", "description": "Product description (1-3 sentences)" },
+    "category": { "type": "string", "description": "Product category" },
+    "features": {
+      "type": "array",
+      "items": { "type": "string" },
+      "description": "Key features or specs"
+    }
+  },
+  "required": ["name", "price", "currency"]
+}

package/src/cli.mjs

@@ -0,0 +1,226 @@
+#!/usr/bin/env node
+// src/cli.mjs
+// CLI entry point
+
+import { Command } from "commander";
+import { readFileSync, writeFileSync, existsSync } from "fs";
+
+async function loadPipeline() {
+  return await import("./pipeline.mjs");
+}
+
+const program = new Command();
+
+program
+  .name("edgecrawl")
+  .description("Local AI-powered web scraper — extract structured JSON using on-device ONNX LLMs")
+  .version("0.3.0");
+
+// ---- Repeatable option helper ----
+function collect(val, memo) {
+  memo.push(val);
+  return memo;
+}
+
+// ---- Browser options ----
+function addBrowserOptions(cmd) {
+  return cmd
+    .option("--headful", "Show browser window (for debugging)")
+    .option("--user-agent <ua>", "Custom User-Agent string")
+    .option("--timeout <ms>", "Page load timeout in milliseconds", "30000")
+    .option("--no-block-media", "Disable blocking of images/fonts/media")
+    .option("--viewport <WxH>", "Viewport size (e.g. 1920x1080)", "1280x800")
+    .option("--wait-until <event>", "Navigation wait condition (load/domcontentloaded/networkidle)", "load")
+    .option("--proxy <url>", "Proxy server URL")
+    .option("--cookie <cookie>", "Cookie in name=value format (repeatable)", collect, [])
+    .option("--extra-header <header>", "HTTP header in Key:Value format (repeatable)", collect, [])
+    .option("--scroll", "Scroll to bottom (for lazy-loaded content)")
+    .option("--wait <selector>", "Wait for CSS selector to appear");
+}
+
+// ---- Common options (LLM + browser) ----
+function addCommonOptions(cmd) {
+  cmd = cmd
+    .option("-p, --preset <preset>", "Model preset (light/balanced/quality)", "balanced")
+    .option("-d, --device <device>", "Inference device (webgpu/wasm)", "webgpu")
+    .option("-s, --schema <file>", "Custom schema JSON file")
+    .option("-o, --output <file>", "Output file (default: stdout)")
+    .option("-t, --max-tokens <n>", "Max input tokens for LLM", "2048")
+    .option("--selector <selector>", "CSS selector to narrow target content");
+  return addBrowserOptions(cmd);
+}
+
+function buildBrowserOptions(opts) {
+  return {
+    headless: !opts.headful,
+    ...(opts.proxy && { proxy: opts.proxy }),
+  };
+}
+
+function buildScrapeOptions(opts) {
+  const [vw, vh] = (opts.viewport || "1280x800").split("x").map(Number);
+  return {
+    scrollToBottom: opts.scroll,
+    waitForSelector: opts.wait,
+    userAgent: opts.userAgent,
+    timeout: parseInt(opts.timeout || "30000"),
+    blockMedia: opts.blockMedia,
+    waitUntil: opts.waitUntil,
+    viewportWidth: vw || 1280,
+    viewportHeight: vh || 800,
+    cookies: opts.cookie || [],
+    extraHeaders: opts.extraHeader || [],
+  };
+}
+
+function outputResult(result, opts) {
+  const json = JSON.stringify(result, null, 2);
+  if (opts.output) {
+    writeFileSync(opts.output, json);
+    console.error(`Result saved to ${opts.output}`);
+  } else {
+    console.log(json);
+  }
+}
+
+// ---- Single URL extraction ----
+addCommonOptions(
+  program
+    .command("extract <url>")
+    .description("Extract structured JSON data from a URL")
+).action(async (url, opts) => {
+  try {
+    const { scrapeAndExtract, cleanup } = await loadPipeline();
+    const schema = opts.schema ? JSON.parse(readFileSync(opts.schema, "utf-8")) : undefined;
+
+    console.error(`Scraping: ${url}`);
+    console.error(`Model: ${opts.preset} / Device: ${opts.device}`);
+
+    const result = await scrapeAndExtract(url, {
+      schema,
+      preset: opts.preset,
+      device: opts.device,
+      maxTokens: parseInt(opts.maxTokens),
+      selector: opts.selector,
+      browserOptions: buildBrowserOptions(opts),
+      scrapeOptions: buildScrapeOptions(opts),
+    });
+
+    outputResult(result, opts);
+  } catch (err) {
+    console.error("Error:", err.message);
+    process.exit(1);
+  } finally {
+    const { cleanup } = await loadPipeline();
+    await cleanup();
+  }
+});
+
+// ---- Batch processing ----
+addCommonOptions(
+  program
+    .command("batch <file>")
+    .description("Batch extract from a URL list file (one URL per line)")
+    .option("-c, --concurrency <n>", "Concurrent scraping limit", "3")
+).action(async (file, opts) => {
+  try {
+    const { batchScrapeAndExtract } = await loadPipeline();
+    if (!existsSync(file)) {
+      console.error(`File not found: ${file}`);
+      process.exit(1);
+    }
+
+    const urls = readFileSync(file, "utf-8")
+      .split("\n")
+      .map((l) => l.trim())
+      .filter((l) => l && !l.startsWith("#"));
+
+    console.error(`Processing ${urls.length} URLs...`);
+
+    const schema = opts.schema ? JSON.parse(readFileSync(opts.schema, "utf-8")) : undefined;
+
+    const results = await batchScrapeAndExtract(urls, {
+      schema,
+      preset: opts.preset,
+      device: opts.device,
+      concurrency: parseInt(opts.concurrency),
+      browserOptions: buildBrowserOptions(opts),
+      scrapeOptions: buildScrapeOptions(opts),
+    });
+
+    const outFile = opts.output || "results.json";
+    writeFileSync(outFile, JSON.stringify(results, null, 2));
+    console.error(`Done. ${results.length} results saved to ${outFile}`);
+  } catch (err) {
+    console.error("Error:", err.message);
+    process.exit(1);
+  } finally {
+    const { cleanup } = await loadPipeline();
+    await cleanup();
+  }
+});
+
+// ---- Custom query ----
+addCommonOptions(
+  program
+    .command("query <url> <prompt>")
+    .description("Ask a custom question about a page's content")
+).action(async (url, prompt, opts) => {
+  try {
+    const { scrapeAndQuery, cleanup } = await loadPipeline();
+    console.error(`Scraping: ${url}`);
+    const result = await scrapeAndQuery(url, prompt, {
+      preset: opts.preset,
+      device: opts.device,
+      browserOptions: buildBrowserOptions(opts),
+      scrapeOptions: buildScrapeOptions(opts),
+    });
+
+    outputResult(result, opts);
+  } catch (err) {
+    console.error("Error:", err.message);
+    process.exit(1);
+  } finally {
+    const { cleanup } = await loadPipeline();
+    await cleanup();
+  }
+});
+
+// ---- Markdown conversion only (no LLM) ----
+addBrowserOptions(
+  program
+    .command("md <url>")
+    .description("Convert HTML to Markdown (no LLM)")
+    .option("-o, --output <file>", "Output file")
+    .option("--selector <selector>", "CSS selector to narrow target content")
+).action(async (url, opts) => {
+    try {
+      const { fetchPage, launchBrowser } = await import("./scraper.mjs");
+      const { htmlToMarkdown, cleanMarkdown } = await import("./html2md.mjs");
+
+      await launchBrowser(buildBrowserOptions(opts));
+      const { html, url: finalUrl } = await fetchPage(url, buildScrapeOptions(opts));
+
+      const { markdown, title } = htmlToMarkdown(html, finalUrl, { selector: opts.selector });
+      const cleaned = cleanMarkdown(markdown);
+
+      console.error(`Markdown: ${markdown.length} chars -> cleaned: ${cleaned.length} chars (${Math.round((1 - cleaned.length / markdown.length) * 100)}% reduction)`);
+
+      const output = `# ${title}\n\nSource: ${finalUrl}\n\n---\n\n${cleaned}`;
+
+      if (opts.output) {
+        writeFileSync(opts.output, output);
+        console.error(`Saved to ${opts.output}`);
+      } else {
+        console.log(output);
+      }
+    } catch (err) {
+      console.error("Error:", err.message);
+      process.exit(1);
+    } finally {
+      const { closeBrowser } = await import("./scraper.mjs");
+      await closeBrowser();
+    }
+  });
+
+program.parse();