@involvex/bun-crawl 0.0.0-stage → 0.1.0

package/.github/FUNDING.yml

@@ -0,0 +1,10 @@
+# These are supported funding model platforms
+
+github: [involvex]
+custom:
+  [
+    https://buymeacoffee.com/involvex,
+    https://paypal.me/involvex,
+    https://opencode.ai/go?ref=XS9FHCZT4C,
+    https://rewards.bing.com/welcome?rh=14525F68&ref=rafsrchae&form=ML2XE3&OCID=ML2XE3&PUBL=RewardsDO&CREA=ML2XE3,
+  ]

package/.prettierignore

@@ -0,0 +1,41 @@
+# AI Agent Directories
+.agent
+.agents
+.kilo
+.claude
+.gemini
+.kilocode
+.opencode
+
+# Build & Dependency Directories
+node_modules
+dist
+build
+coverage
+.next
+.nuxt
+.cache
+
+# Version Control & Config
+.git
+.vscode
+.idea
+
+# Lock Files
+package-lock.json
+yarn.lock
+pnpm-lock.yaml
+bun.lockb
+
+# Minified & Generated Files
+*.min.js
+*.min.css
+*.min.js.map
+*.min.css.map
+
+# System Files
+.DS_Store
+Thumbs.db
+*.log
+*.tgz
+*.tar.gz

package/AGENTS.md

@@ -0,0 +1,357 @@
+# AGENTS.md - Agent Instructions for bun-crawl
+
+This file provides comprehensive instructions for AI agents working on the **bun-crawl** project - an open-source Firecrawl alternative built with Bun for scraping and crawling websites into clean Markdown.
+
+---
+
+## 1. Project Overview
+
+**bun-crawl** is a fast, native Bun-based web scraper and crawler that:
+
+- Uses **Mozilla Readability** for article extraction
+- Uses **Turndown** for HTML → Markdown conversion
+- Supports recursive crawling with depth limiting and same-domain filtering
+- Produces standalone binaries with no runtime dependencies
+- Provides a pluggable AI extractor interface for LLM-based content processing
+
+**Key Technologies:**
+
+- **Runtime:** Bun (v1.3.0+)
+- **Language:** TypeScript (strict mode, ES2022)
+- **CLI Framework:** Citty
+- **HTML Parsing:** linkedom (DOM parser)
+- **Content Extraction:** @mozilla/readability
+- **Markdown Conversion:** turndown
+- **Testing:** Bun's built-in test runner (Jest-compatible)
+- **Linting:** ESLint with TypeScript ESLint
+- **Formatting:** Prettier with @involvex/prettier-config
+
+---
+
+## 2. Useful Commands
+
+### Development
+
+```bash
+# Install dependencies
+bun install
+
+# Run in development mode
+bun run dev scrape https://example.com
+bun run dev crawl https://example.com
+
+# Format code
+bun run format
+
+# Lint (check)
+bun run lint
+
+# Lint (auto-fix)
+bun run lint:fix
+
+# Type check
+bun run typecheck
+
+# Run tests
+bun run test
+```
+
+### Building
+
+```bash
+# Production binary (runs prebuild: format, lint:fix, typecheck)
+bun run build
+# Output: ./dist/bun-crawl (Linux/macOS) or ./dist/bun-crawl.exe (Windows)
+
+# Cross-compilation
+bun build --compile --target=linux-x64 ./src/cli.ts --outfile bun-crawl-linux
+bun build --compile --target=darwin-x64 ./src/cli.ts --outfile bun-crawl-macos
+bun build --compile --target=windows-x64 ./src/cli.ts --outfile bun-crawl.exe
+```
+
+### Testing
+
+```bash
+# Run all tests
+bun test
+
+# Run tests with coverage
+bun test --coverage
+
+# Run specific test file
+bun test src/core/fetcher.test.ts
+
+# Watch mode
+bun test --watch
+```
+
+---
+
+## 3. Project Structure
+
+```
+bun-crawl/
+├── src/
+│   ├── cli.ts              # CLI entry point (Citty-based)
+│   ├── index.ts            # Public API exports
+│   ├── types.ts            # TypeScript interfaces
+│   ├── core/
+│   │   ├── fetcher.ts      # HTTP fetch wrapper with timeout/UA
+│   │   ├── cleaner.ts      # Readability content extraction
+│   │   ├── markdown.ts     # Turndown HTML→Markdown converter
+│   │   └── crawler.ts      # Recursive crawler (HTMLRewriter)
+│   └── ai/
+│       ├── interface.ts    # AIExtractor interface
+│       ├── null-ai.ts      # Pass-through extractor (default)
+│       └── zen-ai.ts       # Template for Zen/OpenCode integration
+├── package.json
+├── tsconfig.json
+├── README.md
+└── AGENTS.md (this file)
+```
+
+---
+
+## 4. Architecture & Core Components
+
+### 4.1 Public API (src/index.ts)
+
+```typescript
+// Core classes
+export {Fetcher} from './core/fetcher'
+export {Cleaner} from './core/cleaner'
+export {MarkdownConverter} from './core/markdown'
+export {Crawler} from './core/crawler'
+
+// AI Extractors
+export {NullAIExtractor} from './ai/null-ai'
+export {ZenAIExtractor} from './ai/zen-ai'
+export type {AIExtractor, AIExtractorConfig} from './ai/interface'
+
+// Types
+export type {
+	CrawlOptions,
+	CrawlResult,
+	FetcherOptions,
+	ScrapeResult,
+} from './types'
+```
+
+### 4.2 Core Classes
+
+| Class               | Purpose                                   | Key Methods                                       |
+| ------------------- | ----------------------------------------- | ------------------------------------------------- |
+| `Fetcher`           | HTTP requests with timeout, UA, redirects | `fetch(url)` → `{html, url, status}`              |
+| `Cleaner`           | Mozilla Readability extraction            | `clean(html, url)` → `{content, title, metadata}` |
+| `MarkdownConverter` | Turndown HTML→Markdown                    | `convert(html)` → `markdown string`               |
+| `Crawler`           | Recursive crawling                        | `crawl(startUrl)` → `CrawlResult`                 |
+
+### 4.3 AI Extractor Interface
+
+```typescript
+interface AIExtractor {
+	extract<T = string>(markdown: string, schemaPrompt: string): Promise<T>
+}
+```
+
+- **NullAIExtractor** - Default, passes content through unchanged
+- **ZenAIExtractor** - Template for OpenCode Zen / LLM integration
+- Custom extractors can be implemented for any LLM API
+
+### 4.4 Data Flow
+
+```
+URL → Fetcher.fetch() → HTML
+    → Cleaner.clean() → {content, title, metadata}
+    → MarkdownConverter.convert() → Markdown
+    → AIExtractor.extract() → Enhanced/Structured Output
+```
+
+---
+
+## 5. Best Practices & Guidelines
+
+### 5.1 Code Style
+
+- **Strict TypeScript:** `strict: true`, `verbatimModuleSyntax: true`
+- **ES Modules:** Use `.js` extensions in imports (Bun requirement)
+- **No `any`:** Use proper types; `unknown` preferred over `any`
+- **Async/await:** Prefer over Promise chains
+- **Error handling:** Always wrap fetch/parse in try-catch; use `Error.cause` for chaining
+
+### 5.2 Testing Guidelines
+
+- Use `bun test` (Jest-compatible API)
+- Test file naming: `*.test.ts` alongside source files
+- Mock external dependencies (fetch, HTML parsing)
+- Test edge cases: empty content, malformed HTML, network errors
+- Aim for >80% coverage on core modules
+
+### 5.3 Performance Considerations
+
+- **Concurrency:** Respect `CrawlOptions.concurrency` (default 5)
+- **Timeouts:** Default 30s; configurable via `FetcherOptions.timeout`
+- **Memory:** Stream large responses when possible; avoid holding full HTML in memory
+- **Binary size:** `--minify` flag in build; avoid unnecessary dependencies
+
+### 5.4 Adding New Features
+
+#### New CLI Command
+
+1. Add command to `cli.ts` in `parseCliArgs()` validation
+2. Create `run<Command>()` function following `runScrape`/`runCrawl` pattern
+3. Add help text in `showHelp()`
+4. Export any new types from `types.ts` and `index.ts`
+
+#### New AI Extractor
+
+1. Implement `AIExtractor` interface in `src/ai/`
+2. Export from `src/ai/index.ts` (create if needed) and `src/index.ts`
+3. Add configuration options to `AIExtractorConfig` if needed
+4. Update CLI to accept extractor selection (future enhancement)
+
+#### New Markdown Rules
+
+Add to `MarkdownConverter.configureRules()` in `markdown.ts`:
+
+```typescript
+this.turndown.addRule('ruleName', {
+	filter: 'tagName', // or function(node) { return condition }
+	replacement: (content, node) => transformedContent,
+})
+```
+
+### 5.5 Security Best Practices
+
+- **Never log sensitive data** (tokens, passwords, full URLs with auth)
+- **Validate URLs** before fetching (protocol allowlist: http/https only)
+- **Respect robots.txt** (future enhancement)
+- **Rate limiting:** Implement delays between requests in crawler
+- **Sanitize output** when rendering HTML in markdown (Turndown handles most)
+
+### 5.6 Cross-Platform Compatibility
+
+- Use `path` module (not string concatenation) for file paths
+- Test on Windows (`bun-crawl.exe`), Linux, macOS
+- Handle line endings correctly (UTF-8, `\n`)
+- Use `dirname()` and `join()` from `path` module
+
+---
+
+## 6. Configuration Files
+
+### tsconfig.json (Key Settings)
+
+```json
+{
+	"compilerOptions": {
+		"target": "ES2022",
+		"module": "ESNext",
+		"moduleResolution": "bundler",
+		"strict": true,
+		"verbatimModuleSyntax": true,
+		"moduleDetection": "force",
+		"types": ["bun"]
+	}
+}
+```
+
+### package.json Scripts
+
+| Script      | Command                           | Purpose              |
+| ----------- | --------------------------------- | -------------------- |
+| `prebuild`  | `format && lint:fix && typecheck` | Pre-build validation |
+| `build`     | `bun build --compile --minify`    | Production binary    |
+| `dev`       | `bun run src/cli.ts`              | Development run      |
+| `format`    | `prettier . --write`              | Code formatting      |
+| `lint`      | `eslint .`                        | Lint check           |
+| `lint:fix`  | `eslint . --fix`                  | Auto-fix lint issues |
+| `test`      | `bun test`                        | Run tests            |
+| `typecheck` | `tsc --noEmit`                    | Type checking        |
+
+---
+
+## 7. Common Tasks & Patterns
+
+### Adding a Test File
+
+```typescript
+// src/core/fetcher.test.ts
+import {describe, it, expect, beforeEach, mock} from 'bun:test'
+import {Fetcher} from './fetcher'
+
+describe('Fetcher', () => {
+	let fetcher: Fetcher
+
+	beforeEach(() => {
+		fetcher = new Fetcher({timeout: 5000})
+	})
+
+	it('should fetch HTML successfully', async () => {
+		// Use mock or test against a local test server
+		const result = await fetcher.fetch('https://example.com')
+		expect(result.status).toBe(200)
+		expect(result.html).toContain('<html')
+	})
+})
+```
+
+### Debugging
+
+```bash
+# Verbose logging
+DEBUG=* bun run dev scrape https://example.com
+
+# Inspect compiled binary
+bun run build && ./dist/bun-crawl --help
+```
+
+### Handling Bun-Specific APIs
+
+- Use `Bun.file()`, `Bun.write()` for file I/O (faster than `fs/promises`)
+- Use `Bun.spawn()` for subprocesses
+- Use `import.meta.resolve()` for module resolution
+- `bun:test` globals: `describe`, `it`, `expect`, `mock`, `spyOn`
+
+---
+
+## 8. Troubleshooting
+
+| Issue                           | Solution                                                     |
+| ------------------------------- | ------------------------------------------------------------ |
+| `TypeScript` errors on `import` | Add `.js` extension to relative imports                      |
+| `fetch` not found               | Ensure `"types": ["bun"]` in tsconfig.json                   |
+| Binary won't run on Linux       | Check glibc compatibility; try `--target=linux-x64-baseline` |
+| Tests timeout                   | Increase timeout in `bun test --timeout=60000`               |
+| HTMLRewriter errors             | Ensure valid HTML; wrap in try-catch                         |
+
+---
+
+## 9. Future Enhancement Areas
+
+- [ ] robots.txt compliance
+- [ ] Rate limiting / politeness delays
+- [ ] Sitemap.xml discovery
+- [ ] JavaScript rendering (headless browser)
+- [ ] Structured data extraction (JSON-LD, microdata)
+- [ ] Proxy support
+- [ ] Cookie/session persistence
+- [ ] Incremental crawling (ETag/Last-Modified)
+- [ ] Distributed crawling (queue-based)
+- [ ] Webhook callbacks for async crawls
+
+---
+
+## 10. References
+
+- **Bun Docs:** https://bun.sh/docs
+- **Mozilla Readability:** https://github.com/mozilla/readability
+- **Turndown:** https://github.com/mixmark-io/turndown
+- **linkedom:** https://github.com/WebReflection/linkedom
+- **Citty:** https://github.com/unjs/citty
+- **TypeScript ESLint:** https://typescript-eslint.io/
+
+---
+
+_Last updated: 2025-09-14_
+_Generated based on project analysis of README.md, package.json, tsconfig.json, and source code_