@involvex/bun-crawl 0.0.0-stage → 0.1.0

package/IMPLEMENTATION_PLAN.md

@@ -0,0 +1,452 @@
+# bun-crawl Implementation Plan
+
+## Project Overview
+
+**Name:** `bun-crawl` (CLI binary name)
+**Package Manager:** Bun
+**CLI Framework:** `citty`
+**Language:** TypeScript
+
+---
+
+## Directory Structure
+
+```
+bun-crawl/
+├── src/
+│   ├── index.ts                 # Entry point, exports CLI
+│   ├── cli.ts                   # CLI definition (citty)
+│   ├── core/
+│   │   ├── fetcher.ts           # HTTP fetch wrapper with config
+│   │   ├── cleaner.ts           # linkedom + @mozilla/readability
+│   │   ├── markdown.ts          # turndown configuration
+│   │   └── crawler.ts           # HTMLRewriter-based link extraction & recursion
+│   ├── ai/
+│   │   ├── interface.ts         # AIExtractor interface
+│   │   ├── null-ai.ts           # NoOpAIExtractor (pass-through)
+│   │   └── zen-ai.ts            # Template/Stub for OpenCode Zen (commented)
+│   └── types.ts                 # Shared types/interfaces
+├── package.json
+├── bunfig.toml
+├── tsconfig.json
+└── README.md
+```
+
+---
+
+## Phase 1: Project Setup
+
+### 1.1 Initialize Bun Project
+
+```bash
+bun init -y
+```
+
+### 1.2 Install Dependencies
+
+```bash
+# Production dependencies
+bun add citty turndown @mozilla/readability linkedom
+
+# Dev dependencies
+bun add -d @types/turndown @types/mozilla-readability typescript
+```
+
+### 1.3 Configure `package.json`
+
+```json
+{
+	"name": "bun-crawl",
+	"version": "0.1.0",
+	"type": "module",
+	"main": "src/index.ts",
+	"bin": {
+		"bun-crawl": "./src/cli.ts"
+	},
+	"scripts": {
+		"dev": "bun run src/cli.ts",
+		"build": "bun build --compile --minify ./src/index.ts --outfile bun-crawl",
+		"test": "bun test"
+	},
+	"dependencies": {
+		"citty": "^0.1.6",
+		"turndown": "^7.2.0",
+		"@mozilla/readability": "^0.5.0",
+		"linkedom": "^0.18.0"
+	},
+	"devDependencies": {
+		"@types/turndown": "^5.0.4",
+		"@types/mozilla-readability": "^0.1.0",
+		"typescript": "^5.5.0"
+	}
+}
+```
+
+### 1.4 Configure `tsconfig.json`
+
+```json
+{
+	"compilerOptions": {
+		"target": "ES2022",
+		"module": "ESNext",
+		"moduleResolution": "bundler",
+		"lib": ["ES2022"],
+		"strict": true,
+		"esModuleInterop": true,
+		"skipLibCheck": true,
+		"forceConsistentCasingInFileNames": true,
+		"resolveJsonModule": true,
+		"declaration": true,
+		"outDir": "./dist"
+	},
+	"include": ["src/**/*"],
+	"exclude": ["node_modules", "dist"]
+}
+```
+
+### 1.5 Configure `bunfig.toml`
+
+```toml
+[install]
+cache = true
+
+[run]
+env = { NODE_ENV = "development" }
+```
+
+---
+
+## Phase 2: Core Processing Pipeline
+
+### 2.1 Shared Types (`src/types.ts`)
+
+```typescript
+export interface ScrapeResult {
+	url: string
+	title: string
+	markdown: string
+	html: string
+	metadata: {
+		description?: string
+		author?: string
+		publishedTime?: string
+		[key: string]: unknown
+	}
+}
+
+export interface CrawlOptions {
+	depth: number
+	limit: number
+	sameDomain: boolean
+}
+
+export interface CrawlResult {
+	pages: ScrapeResult[]
+	errors: Array<{url: string; error: string}>
+	stats: {
+		total: number
+		succeeded: number
+		failed: number
+	}
+}
+
+export interface FetcherOptions {
+	timeout?: number
+	userAgent?: string
+	headers?: Record<string, string>
+	followRedirects?: boolean
+}
+```
+
+### 2.2 Fetcher (`src/core/fetcher.ts`)
+
+- Wrapper around native `fetch()`
+- Configurable timeout, user-agent, headers
+- Error handling with descriptive messages
+- Returns raw HTML + response metadata
+
+```typescript
+export class Fetcher {
+	constructor(private options: FetcherOptions = {}) {}
+
+	async fetch(
+		url: string,
+	): Promise<{html: string; url: string; status: number}> {
+		// Implementation with AbortController for timeout
+	}
+}
+```
+
+### 2.3 Cleaner (`src/core/cleaner.ts`)
+
+- Uses `linkedom` to create DOM from HTML
+- Uses `@mozilla/readability` to extract main content
+- Returns cleaned HTML + metadata (title, byline, etc.)
+
+```typescript
+export class Cleaner {
+	clean(
+		html: string,
+		url: string,
+	): {content: string; title: string; metadata: Record<string, unknown>} {
+		// JSDOM via linkedom + Readability
+	}
+}
+```
+
+### 2.4 Markdown Converter (`src/core/markdown.ts`)
+
+- Configures `turndown` with sensible defaults
+- Handles: tables, code blocks (fenced), links, images, headings
+- Options for: heading style, code block style, bullet list marker
+
+```typescript
+export class MarkdownConverter {
+	private turndown: TurndownService
+
+	constructor(options?: TurndownOptions) {}
+
+	convert(html: string): string {
+		return this.turndown.turndown(html)
+	}
+}
+```
+
+### 2.5 Crawler (`src/core/crawler.ts`)
+
+- Uses Bun's native `HTMLRewriter` for streaming link extraction
+- Recursive crawling with depth/limit control
+- Same-domain filtering
+- Deduplication of URLs
+- Concurrent fetching with configurable concurrency
+
+```typescript
+export class Crawler {
+	constructor(
+		private fetcher: Fetcher,
+		private cleaner: Cleaner,
+		private converter: MarkdownConverter,
+		private options: CrawlOptions,
+	) {}
+
+	async crawl(startUrl: string): Promise<CrawlResult> {
+		// BFS with HTMLRewriter for link extraction
+	}
+
+	private extractLinks(html: string, baseUrl: string): string[] {
+		// HTMLRewriter implementation
+	}
+}
+```
+
+---
+
+## Phase 3: AI Extractor Pattern
+
+### 3.1 Interface (`src/ai/interface.ts`)
+
+```typescript
+export interface AIExtractor {
+	extract<T = string>(markdown: string, schemaPrompt: string): Promise<T>
+}
+
+export interface AIExtractorConfig {
+	// Base config for all extractors
+}
+```
+
+### 3.2 NullAIExtractor (`src/ai/null-ai.ts`) - Default
+
+```typescript
+export class NullAIExtractor implements AIExtractor {
+	async extract<T = string>(
+		markdown: string,
+		_schemaPrompt: string,
+	): Promise<T> {
+		return markdown as T // Pass-through
+	}
+}
+```
+
+### 3.3 ZenAIExtractor Template (`src/ai/zen-ai.ts`) - Stub/Template
+
+```typescript
+// TODO: Implement when OpenCode Zen API is available
+// This is a template for future OpenAI-compatible Chat Completions integration
+
+/*
+export interface ZenAIConfig {
+  endpoint: string;        // e.g. "https://api.opencode.ai/v1/chat/completions"
+  apiKey: string;
+  model?: string;          // e.g. "gpt-4o", "claude-3.5-sonnet"
+  temperature?: number;
+  maxTokens?: number;
+}
+
+export class ZenAIExtractor implements AIExtractor {
+  constructor(private config: ZenAIConfig) {}
+  
+  async extract<T = string>(markdown: string, schemaPrompt: string): Promise<T> {
+    const response = await fetch(this.config.endpoint, {
+      method: 'POST',
+      headers: {
+        'Authorization': `Bearer ${this.config.apiKey}`,
+        'Content-Type': 'application/json',
+      },
+      body: JSON.stringify({
+        model: this.config.model || 'gpt-4o',
+        messages: [
+          { role: 'system', content: schemaPrompt },
+          { role: 'user', content: markdown }
+        ],
+        temperature: this.config.temperature ?? 0,
+        max_tokens: this.config.maxTokens ?? 4000,
+      }),
+    });
+    
+    const data = await response.json();
+    return data.choices[0]?.message?.content as T;
+  }
+}
+*/
+
+// Export placeholder for now
+export class ZenAIExtractor {
+	constructor() {
+		throw new Error(
+			'ZenAIExtractor not yet implemented. See src/ai/zen-ai.ts for template.',
+		)
+	}
+}
+```
+
+---
+
+## Phase 4: CLI Interface (`src/cli.ts`)
+
+### Commands Structure
+
+```typescript
+// bun-crawl scrape <url> [options]
+// bun-crawl crawl <url> [options]
+// bun-crawl --help / --version
+```
+
+### Scrape Command Options
+
+| Flag           | Short | Description                          | Default         |
+| -------------- | ----- | ------------------------------------ | --------------- |
+| `--output`     | `-o`  | Output file path (stdout if omitted) | stdout          |
+| `--format`     | `-f`  | Output format: `markdown`, `json`    | markdown        |
+| `--timeout`    | `-t`  | Request timeout in ms                | 30000           |
+| `--user-agent` | `-u`  | Custom User-Agent                    | bun-crawl/0.1.0 |
+
+### Crawl Command Options
+
+| Flag            | Short | Description                         | Default        |
+| --------------- | ----- | ----------------------------------- | -------------- |
+| `--depth`       | `-d`  | Max crawl depth                     | 2              |
+| `--limit`       | `-l`  | Max pages to crawl                  | 20             |
+| `--out-dir`     | `-o`  | Output directory for markdown files | ./crawl-output |
+| `--same-domain` |       | Stay on same domain                 | true           |
+| `--concurrency` | `-c`  | Concurrent requests                 | 5              |
+
+### Global Options
+
+| Flag        | Description                  |
+| ----------- | ---------------------------- |
+| `--json`    | Output machine-readable JSON |
+| `--verbose` | Verbose logging              |
+
+---
+
+## Phase 5: Entry Point (`src/index.ts`)
+
+```typescript
+// Main exports for programmatic use
+export {Fetcher} from './core/fetcher'
+export {Cleaner} from './core/cleaner'
+export {MarkdownConverter} from './core/markdown'
+export {Crawler} from './core/crawler'
+export {NullAIExtractor} from './ai/null-ai'
+export {ZenAIExtractor} from './ai/zen-ai'
+export type {AIExtractor} from './ai/interface'
+export type {
+	ScrapeResult,
+	CrawlResult,
+	CrawlOptions,
+	FetcherOptions,
+} from './types'
+```
+
+---
+
+## Phase 6: Build & Distribution
+
+### Build Script
+
+```json
+"scripts": {
+  "build": "bun build --compile --minify ./src/index.ts --outfile bun-crawl"
+}
+```
+
+### Output
+
+- Single binary: `./bun-crawl` (Linux/macOS) or `./bun-crawl.exe` (Windows)
+- Self-contained, no Node.js/Bun runtime required
+- Cross-compilation via `bun build --compile --target=...`
+
+---
+
+## Implementation Order
+
+1. ✅ **Setup**: `package.json`, `tsconfig.json`, `bunfig.toml`
+2. ✅ **Types**: `src/types.ts`
+3. ✅ **Core**: `fetcher.ts` → `cleaner.ts` → `markdown.ts` → `crawler.ts`
+4. ✅ **AI**: `interface.ts` → `null-ai.ts` → `zen-ai.ts` (stub)
+5. ✅ **CLI**: `cli.ts` with both commands (custom parser replacing citty)
+6. ✅ **Entry**: `index.ts` exports
+7. ✅ **Build**: Test compilation - standalone binary works
+
+---
+
+## Testing Strategy
+
+- Unit tests for each core module
+- Integration test for scrape command
+- Integration test for crawl command
+- Snapshot tests for markdown output
+
+---
+
+## Future Extensions (Post-MVP)
+
+1. **ZenAIExtractor** full implementation
+2. **Playwright fallback** for JS-heavy sites
+3. **Sitemap.xml** discovery
+4. **Robots.txt** compliance
+5. **Rate limiting** per domain
+6. **Cache layer** (file-based or Redis)
+7. **Plugin system** for custom extractors
+
+---
+
+## Clarifications for Review
+
+1. ✅ **CLI Name**: `bun-crawl` (binary & command)
+2. ✅ **Framework**: Custom argument parser (citty's option parsing had issues with subcommands)
+3. ✅ **ZenAIExtractor**: Template/stub only (commented implementation)
+4. ✅ **Default output format for crawl**: Individual `.md` files per page in output directory
+5. ✅ **Concurrency default**: 5 (works well)
+6. ✅ **User-Agent string**: `bun-crawl/0.1.0`
+
+---
+
+## Suggested Next Steps
+
+1. Add unit tests for core modules
+2. Add integration tests for CLI commands
+3. Add README.md with usage examples
+4. Set up CI/CD for automated builds
+5. Consider adding Playwright fallback for JS-heavy sites

package/Plan.md

@@ -0,0 +1,83 @@
+# Bun-Crawl
+
+---
+
+## Architektur & Modul-Design
+
+Die Pipeline folgt dem **Strategy-Pattern**. Der Scraper-Core übergibt das Ergebnis an eine Pipeline, deren letzter Schritt optional die AI-Extraktion ist.
+
+```
+[URL] -> [Fetcher] -> [Readability + Turndown] -> [Markdown Output]
+                                                        │
+                                                        ▼ (Optional / Später)
+                                              [OpenCode Zen API] -> [JSON Output]
+
+```
+
+### Ordnerstruktur
+
+```
+firebun/
+├── src/
+│   ├── cli.ts              # CLI Interface (Commander / Citty)
+│   ├── core/
+│   │   ├── fetcher.ts      # HTTP Requests (optional Playwright Fallback)
+│   │   ├── cleaner.ts      # Linkedom + @mozilla/readability
+│   │   ├── markdown.ts     # Turndown Konvertierung
+│   │   └── crawler.ts      # HTMLRewriter für Rekursion
+│   ├── ai/
+│   │   ├── interface.ts    # AI Extractor Interface
+│   │   └── null-ai.ts      # Standart-Pass-Through (ohne AI)
+│   │   └── zen-ai.ts       # (Später) OpenCode Zen Provider
+│   └── index.ts
+├── package.json
+└── bunfig.toml
+
+```
+
+---
+
+## Implementierungsplan
+
+### Phase 1: Projekt-Setup
+
+- Initalisierung: `bun init`
+- Dependencies: `bun add turndown @mozilla/readability linkedom citty`
+- Dev-Dependencies: `bun add -d @types/turndown @types/mozilla-readability`
+
+### Phase 2: Core Processing Pipeline (Ohne AI)
+
+- **Fetcher (`src/core/fetcher.ts`):** `fetch()`-Wrapper mit konfigurierbaren User-Agents und Timeouts.
+- **Cleaner (`src/core/cleaner.ts`):** DOM-Parsing mit `linkedom` und Content-Extraktion via `@mozilla/readability`.
+- **Markdown (`src/core/markdown.ts`):** Konfiguration von `turndown` zur sauberen Formatierung von Tabellen, Code-Blöcken und Links.
+
+### Phase 3: Crawl-Engine & CLI
+
+- **Link-Parser (`src/core/crawler.ts`):** Nutzung von Buns nativem `HTMLRewriter`, um im HTTP-Stream alle internen Links (`<a href="...">`) zu sammeln und Rekursion (Tiefe/Limit) zu steuern.
+- **CLI-Interface (`src/cli.ts`):** Flag-Parsing (`--depth`, `--out-dir`, `--json`).
+
+### Phase 4: Vorbereitung der OpenCode Zen Schnittstelle
+
+- Erstellen von `src/ai/interface.ts`:
+
+```typescript
+export interface AIExtractor {
+	extract<T>(markdown: string, schemaPrompt: string): Promise<T | string>
+}
+```
+
+- `NullAIExtractor` implementieren, der lediglich das rohe Markdown unverändert zurückgibt.
+- Für die spätere Anbindung muss nur ein `ZenAIExtractor` hinzugefügt werden, der per `fetch()` an den OpenAI-kompatiblen OpenCode Zen Endpoint sendet (`/v1/chat/completions`).
+
+### Phase 5: Compilation
+
+- Script in `package.json`:
+
+```json
+"scripts": {
+  "build": "bun build --compile --minify ./src/index.ts --outfile firebun"
+}
+
+```
+
+---

package/README.md

@@ -1,3 +1,181 @@
-# Temporary Holding Version
+# bun-crawl
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+Open-source Firecrawl alternative built with Bun. Scrape and crawl websites into clean Markdown.
+
+## Features
+
+- **Fast** - Native Bun runtime, no Node.js required
+- **Clean output** - Mozilla Readability + Turndown for article extraction
+- **Recursive crawling** - Depth-limited, same-domain filtering, concurrent requests
+- **Standalone binary** - Single executable, no runtime dependencies
+- **AI-ready** - Extractor interface for LLM-based content processing
+
+## Installation
+
+### Pre-built Binary (Recommended)
+
+Download the latest release for your platform.
+
+### From Source
+
+```bash
+git clone https://github.com/yourusername/bun-crawl
+cd bun-crawl
+bun install
+bun run build
+```
+
+## Usage
+
+### Scrape a Single URL
+
+```bash
+# Output to stdout (markdown)
+bun-crawl scrape https://example.com
+
+# Save to file
+bun-crawl scrape https://example.com -o page.md
+
+# JSON output with metadata
+bun-crawl scrape https://example.com --format json
+bun-crawl scrape https://example.com --json
+
+# Custom timeout and user agent
+bun-crawl scrape https://example.com -t 60000 -u "MyBot/1.0"
+```
+
+### Crawl a Website Recursively
+
+```bash
+# Crawl with defaults (depth 2, limit 20 pages)
+bun-crawl crawl https://example.com
+
+# Custom depth and limit
+bun-crawl crawl https://example.com -d 3 -l 50
+
+# Custom output directory
+bun-crawl crawl https://example.com -O ./my-output
+
+# JSON summary only
+bun-crawl crawl https://example.com --json
+
+# Stay on same domain (default), custom concurrency
+bun-crawl crawl https://example.com --same-domain -c 10
+```
+
+### Options Reference
+
+#### `scrape` Command
+
+| Flag           | Short | Description                          | Default         |
+| -------------- | ----- | ------------------------------------ | --------------- |
+| `--output`     | `-o`  | Output file path (stdout if omitted) | stdout          |
+| `--format`     | `-f`  | Output format: `markdown`, `json`    | markdown        |
+| `--timeout`    | `-t`  | Request timeout in milliseconds      | 30000           |
+| `--user-agent` | `-u`  | Custom User-Agent string             | bun-crawl/0.1.0 |
+| `--json`       |       | Output machine-readable JSON         | false           |
+
+#### `crawl` Command
+
+| Flag            | Short | Description                          | Default         |
+| --------------- | ----- | ------------------------------------ | --------------- |
+| `--depth`       | `-d`  | Maximum crawl depth                  | 2               |
+| `--limit`       | `-l`  | Maximum pages to crawl               | 20              |
+| `--out-dir`     | `-O`  | Output directory for markdown files  | ./crawl-output  |
+| `--same-domain` |       | Stay on the same domain              | true            |
+| `--concurrency` | `-c`  | Concurrent requests                  | 5               |
+| `--timeout`     | `-t`  | Request timeout in milliseconds      | 30000           |
+| `--user-agent`  | `-u`  | Custom User-Agent string             | bun-crawl/0.1.0 |
+| `--json`        |       | Output machine-readable JSON summary | false           |
+
+## Programmatic API
+
+```typescript
+import {
+	Fetcher,
+	Cleaner,
+	MarkdownConverter,
+	Crawler,
+	NullAIExtractor,
+} from 'bun-crawl'
+
+const fetcher = new Fetcher({timeout: 30000, userAgent: 'MyBot/1.0'})
+const cleaner = new Cleaner()
+const converter = new MarkdownConverter()
+const ai = new NullAIExtractor()
+
+// Scrape single page
+const {html, url, status} = await fetcher.fetch('https://example.com')
+const {content, title, metadata} = cleaner.clean(html, url)
+const markdown = await ai.extract(converter.convert(content), '')
+
+// Crawl multiple pages
+const crawler = new Crawler(fetcher, cleaner, converter, {
+	depth: 2,
+	limit: 20,
+	sameDomain: true,
+	concurrency: 5,
+})
+const result = await crawler.crawl('https://example.com')
+```
+
+## AI Extractors
+
+`bun-crawl` uses a pluggable AI extractor interface. The default `NullAIExtractor` passes content through unchanged.
+
+```typescript
+import {AIExtractor} from 'bun-crawl'
+
+class MyAIExtractor implements AIExtractor {
+	async extract<T = string>(
+		markdown: string,
+		schemaPrompt: string,
+	): Promise<T> {
+		// Call your LLM API here
+		return markdown as T
+	}
+}
+```
+
+A `ZenAIExtractor` template is included for future OpenCode Zen integration.
+
+## Building
+
+```bash
+# Development
+bun run dev scrape https://example.com
+
+# Production binary
+bun run build
+# Output: ./bun-crawl (Linux/macOS) or ./bun-crawl.exe (Windows)
+```
+
+### Cross-compilation
+
+```bash
+bun build --compile --target=linux-x64 ./src/cli.ts --outfile bun-crawl-linux
+bun build --compile --target=darwin-x64 ./src/cli.ts --outfile bun-crawl-macos
+bun build --compile --target=windows-x64 ./src/cli.ts --outfile bun-crawl.exe
+```
+
+## Project Structure
+
+```
+src/
+├── cli.ts              # CLI entry point
+├── index.ts            # Public API exports
+├── types.ts            # TypeScript interfaces
+├── core/
+│   ├── fetcher.ts      # HTTP fetch wrapper
+│   ├── cleaner.ts      # Readability content extraction
+│   ├── markdown.ts     # Turndown HTML→Markdown
+│   └── crawler.ts      # Recursive crawler (HTMLRewriter)
+└── ai/
+    ├── interface.ts    # AIExtractor interface
+    ├── null-ai.ts      # Pass-through extractor
+    └── zen-ai.ts       # Template for Zen integration
+```
+
+## License
+
+MIT