@robot-resources/scraper 0.1.0

package/README.md

@@ -0,0 +1,228 @@
+[![CI](https://github.com/robot-resources/scraper/actions/workflows/ci.yml/badge.svg)](https://github.com/robot-resources/scraper/actions/workflows/ci.yml)
+[![npm version](https://img.shields.io/npm/v/@robot-resources/scraper)](https://www.npmjs.com/package/@robot-resources/scraper)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/robot-resources/scraper/blob/main/LICENSE)
+[![npm downloads](https://img.shields.io/npm/dm/@robot-resources/scraper)](https://www.npmjs.com/package/@robot-resources/scraper)
+[![codecov](https://codecov.io/gh/robot-resources/scraper/branch/main/graph/badge.svg)](https://codecov.io/gh/robot-resources/scraper)
+[![bundle size](https://img.shields.io/bundlephobia/minzip/@robot-resources/scraper)](https://bundlephobia.com/package/@robot-resources/scraper)
+
+# @robot-resources/scraper
+
+> Context compression for AI agents. Fetch → Extract → Convert pipeline without LLM dependency.
+
+Reduces web page tokens by 70-80% for AI agent consumption. 3-tier fetch with auto-fallback, BFS multi-page crawl, robots.txt compliance.
+
+## Installation
+
+```bash
+npm install @robot-resources/scraper
+```
+
+**Optional peer dependencies** (install only what you need):
+
+```bash
+npm install impit          # Stealth mode — TLS fingerprint impersonation
+npm install playwright     # Render mode — headless browser for JS-rendered pages
+```
+
+## Quick Start
+
+```typescript
+import { scrape } from '@robot-resources/scraper';
+
+const result = await scrape('https://example.com/article');
+
+console.log(result.markdown);     // Compressed content
+console.log(result.tokenCount);   // Estimated tokens
+console.log(result.title);        // Page title
+```
+
+## Fetch Modes
+
+Control how pages are fetched with the `mode` option:
+
+| Mode | How | When to use |
+|------|-----|-------------|
+| `'fast'` | Plain HTTP fetch | Default sites, APIs, docs |
+| `'stealth'` | TLS fingerprint impersonation (impit) | Sites with anti-bot protection |
+| `'render'` | Headless Playwright browser | JS-rendered SPAs, dynamic content |
+| `'auto'` | Fast first, falls back to stealth on 403/challenge | **Default** — best for unknown sites |
+
+```typescript
+// Explicit stealth for a protected site
+const result = await scrape('https://protected-site.com', { mode: 'stealth' });
+
+// Auto mode (default) — tries fast, falls back to stealth if blocked
+const result = await scrape('https://unknown-site.com');
+```
+
+## Crawling Multiple Pages
+
+```typescript
+import { crawl } from '@robot-resources/scraper';
+
+const result = await crawl({
+  url: 'https://docs.example.com',
+  depth: 2,            // Max link depth (default: 2)
+  limit: 20,           // Max pages (default: 50)
+  mode: 'auto',        // Fetch mode per page
+  concurrency: 3,      // Parallel fetches (default: 3)
+  respectRobots: true,  // Obey robots.txt (default: true)
+  include: ['**/docs/**'],   // Only crawl docs paths
+  exclude: ['**/archive/**'], // Skip archive
+});
+
+console.log(`Crawled ${result.totalCrawled} pages in ${result.duration}ms`);
+
+for (const page of result.pages) {
+  console.log(`[depth ${page.depth}] ${page.title}: ${page.tokenCount} tokens`);
+}
+```
+
+The crawler uses BFS link discovery, seeds from sitemap.xml when available, and respects crawl-delay from robots.txt.
+
+## API
+
+### `scrape(url, options?)`
+
+Fetch a URL and return compressed markdown.
+
+```typescript
+const result = await scrape('https://example.com', {
+  mode: 'auto',          // Fetch mode (default: 'auto')
+  timeout: 5000,         // Request timeout ms (default: 10000)
+  maxRetries: 2,         // Retry attempts (default: 3)
+  userAgent: '...',      // Custom user agent
+  respectRobots: false,  // Check robots.txt (default: false)
+});
+```
+
+**Returns:** `ScrapeResult`
+
+```typescript
+interface ScrapeResult {
+  markdown: string;      // Compressed content
+  tokenCount: number;    // Estimated token count
+  title?: string;        // Page title
+  author?: string;       // Author if found
+  siteName?: string;     // Site name if found
+  publishedAt?: string;  // Publish date if found
+  url: string;           // Final URL after redirects
+}
+```
+
+### `crawl(options)`
+
+BFS multi-page crawl from a starting URL.
+
+```typescript
+const result = await crawl({
+  url: 'https://example.com',   // Starting URL (required)
+  depth: 2,                     // Max depth (default: 2)
+  limit: 50,                    // Max pages (default: 50)
+  mode: 'auto',                 // Fetch mode (default: 'auto')
+  include: ['**/blog/**'],      // Include patterns (glob)
+  exclude: ['**/admin/**'],     // Exclude patterns (glob)
+  timeout: 10000,               // Per-page timeout ms
+  concurrency: 3,               // Parallel fetches (default: 3)
+  respectRobots: true,          // Obey robots.txt (default: true)
+});
+```
+
+**Returns:** `CrawlResult`
+
+```typescript
+interface CrawlResult {
+  pages: CrawlPageResult[];  // Scraped pages (extends ScrapeResult + depth)
+  totalDiscovered: number;   // Total URLs found
+  totalCrawled: number;      // Successfully scraped
+  totalSkipped: number;      // Skipped (robots, filter, limit)
+  errors: CrawlError[];      // Per-URL errors
+  duration: number;          // Total ms
+}
+```
+
+### Individual Layers
+
+For advanced usage, use the pipeline layers directly:
+
+```typescript
+import {
+  fetchUrl,
+  fetchStealth,
+  fetchRender,
+  extractContent,
+  convertToMarkdown,
+  estimateTokens,
+} from '@robot-resources/scraper';
+
+// Layer 1: Fetch HTML (choose your tier)
+const fetched = await fetchUrl('https://example.com');
+// or: await fetchStealth(url, options)
+// or: await fetchRender(url, options)
+
+// Layer 2: Extract main content
+const extracted = await extractContent(fetched);
+
+// Layer 3: Convert to markdown
+const converted = await convertToMarkdown(extracted);
+
+// Token estimation
+const htmlTokens = estimateTokens(fetched.html);
+console.log(`Compressed ${htmlTokens} → ${converted.tokenCount} tokens`);
+```
+
+### Robots & Sitemap
+
+```typescript
+import {
+  isAllowedByRobots,
+  getCrawlDelay,
+  getSitemapUrls,
+  parseSitemap,
+} from '@robot-resources/scraper';
+
+const allowed = await isAllowedByRobots('https://example.com/page');
+const delay = await getCrawlDelay('https://example.com');
+const entries = await parseSitemap('https://example.com/sitemap.xml');
+```
+
+### Error Handling
+
+```typescript
+import { scrape, FetchError, ExtractionError } from '@robot-resources/scraper';
+
+try {
+  const result = await scrape(url);
+} catch (error) {
+  if (error instanceof FetchError) {
+    console.log('Fetch failed:', error.statusCode, error.retryable);
+  }
+  if (error instanceof ExtractionError) {
+    console.log('Extraction failed:', error.code);
+  }
+}
+```
+
+## Token Reduction
+
+| Page Type | HTML Tokens | Markdown Tokens | Reduction |
+|-----------|-------------|-----------------|-----------|
+| News article | ~15,000 | ~3,000 | 80% |
+| Documentation | ~12,000 | ~2,500 | 79% |
+| Blog post | ~8,000 | ~1,800 | 77% |
+
+## Requirements
+
+- Node.js 18+
+- ESM or CommonJS
+
+## Related
+
+- [@robot-resources/scraper-mcp](https://npm.im/@robot-resources/scraper-mcp) - MCP server for AI agents
+- [@robot-resources/scraper-tracking](https://npm.im/@robot-resources/scraper-tracking) - Usage tracking
+- [scraper.robotresources.ai](https://scraper.robotresources.ai) - Hosted API
+- [Robot Resources](https://robotresources.ai) - Human Resources, but for your AI agents
+
+## License
+
+MIT

package/bin/setup.js

@@ -0,0 +1,135 @@
+#!/usr/bin/env node
+
+/**
+ * robot-resources-scraper — Setup wizard for @robot-resources/scraper.
+ *
+ * Triggered via `npx @robot-resources/scraper`.
+ * Offers optional GitHub login and shows usage instructions.
+ */
+
+import { readFileSync, writeFileSync, copyFileSync, mkdirSync, existsSync } from "node:fs";
+import { homedir } from "node:os";
+import { join } from "node:path";
+
+// ─── ANSI helpers ────────────────────────────────────────────────────────────
+
+const c = {
+  reset: "\x1b[0m",
+  bold: "\x1b[1m",
+  dim: "\x1b[2m",
+  green: "\x1b[32m",
+  yellow: "\x1b[33m",
+  cyan: "\x1b[36m",
+  blue: "\x1b[34m",
+};
+
+function success(msg) { console.log(`  ${c.green}✓${c.reset} ${msg}`); }
+function step(msg) { console.log(`  ${c.cyan}→${c.reset} ${msg}`); }
+function info(msg) { console.log(`  ${c.dim}${msg}${c.reset}`); }
+function warn(msg) { console.log(`  ${c.yellow}!${c.reset} ${msg}`); }
+
+// ─── Config helpers (inline — no external deps for this bin) ─────────────────
+
+const CONFIG_DIR = join(homedir(), ".robot-resources");
+const CONFIG_FILE = join(CONFIG_DIR, "config.json");
+
+function readConfig() {
+  try { return JSON.parse(readFileSync(CONFIG_FILE, "utf-8")); }
+  catch { return {}; }
+}
+
+// ─── MCP auto-config ─────────────────────────────────────────────────────────
+
+const MCP_KEY = "robot-resources-scraper";
+const MCP_ENTRY = { command: "npx", args: ["-y", "@robot-resources/scraper-mcp"] };
+
+function detectAgents() {
+  const home = homedir();
+  const agents = [
+    {
+      name: "Claude Desktop",
+      configPath: process.platform === "darwin"
+        ? join(home, "Library", "Application Support", "Claude", "claude_desktop_config.json")
+        : join(home, ".config", "Claude", "claude_desktop_config.json"),
+    },
+    { name: "Cursor", configPath: join(home, ".cursor", "mcp.json") },
+  ];
+  return agents.filter((a) => existsSync(a.configPath) || existsSync(join(a.configPath, "..")));
+}
+
+function configureAgentMCP() {
+  const agents = detectAgents();
+  const results = [];
+
+  for (const agent of agents) {
+    try {
+      let config;
+      try { config = JSON.parse(readFileSync(agent.configPath, "utf-8")); }
+      catch { config = {}; }
+
+      config.mcpServers = config.mcpServers || {};
+      if (config.mcpServers[MCP_KEY]) {
+        results.push({ name: agent.name, action: "exists" });
+        continue;
+      }
+
+      // Backup before modifying
+      if (existsSync(agent.configPath)) {
+        copyFileSync(agent.configPath, `${agent.configPath}.bak`);
+      }
+
+      config.mcpServers[MCP_KEY] = MCP_ENTRY;
+      mkdirSync(join(agent.configPath, ".."), { recursive: true });
+      writeFileSync(agent.configPath, JSON.stringify(config, null, 2) + "\n");
+      results.push({ name: agent.name, action: "added" });
+    } catch (err) {
+      results.push({ name: agent.name, action: "error", reason: err.message });
+    }
+  }
+  return results;
+}
+
+// ─── Main ────────────────────────────────────────────────────────────────────
+
+async function main() {
+  console.log(`\n  ${c.blue}${c.bold}██ Robot Resources — Scraper Setup${c.reset}\n`);
+
+  // Step 1: Auth status
+  const config = readConfig();
+  if (config.api_key) {
+    success(`Logged in as ${config.user_name || config.user_email || "unknown"}`);
+  } else {
+    info("Not logged in. Scraper works without login.");
+    info(`To enable telemetry, run: ${c.cyan}npx robot-resources${c.reset}`);
+  }
+
+  // Step 2: MCP auto-config
+  console.log("");
+  step("Configuring MCP in detected agents...");
+
+  const mcpResults = configureAgentMCP();
+  if (mcpResults.length === 0) {
+    info("No supported agents detected (Claude Desktop, Cursor)");
+  } else {
+    for (const r of mcpResults) {
+      if (r.action === "added") success(`${r.name}: scraper MCP configured`);
+      else if (r.action === "exists") success(`${r.name}: already configured`);
+      else warn(`${r.name}: ${r.reason || r.action}`);
+    }
+  }
+
+  // Step 3: Usage
+  console.log(`\n  ${c.blue}${c.bold}── Ready ──${c.reset}\n`);
+  console.log("  Use as a library:");
+  console.log(`  ${c.dim}import { scrape } from '@robot-resources/scraper';${c.reset}`);
+  console.log(`  ${c.dim}const result = await scrape('https://example.com');${c.reset}`);
+  console.log("");
+  console.log("  Use via MCP (already configured above):");
+  console.log(`  ${c.dim}Your agent can call scraper_compress_url(url)${c.reset}`);
+  console.log("");
+}
+
+main().catch((err) => {
+  console.error(`\n  Setup failed: ${err.message}\n`);
+  process.exit(1);
+});