@ebowwa/markdown-docs-scraper 1.0.0

package/README.md

@@ -0,0 +1,148 @@
+# @ebowwa/markdown-docs-scraper
+
+> Scrape and mirror markdown-based documentation sites
+
+## Features
+
+- 📥 Download full markdown documentation
+- 🔄 Organize into directory structure
+- 📊 Track downloads and failures
+- 🚀 Fast concurrent downloads
+- 🎯 CLI and programmatic API
+
+## Installation
+
+```bash
+bun add @ebowwa/markdown-docs-scraper
+# or
+npm install @ebowwa/markdown-docs-scraper
+```
+
+## CLI Usage
+
+### Quick Start - Anthropic Docs
+
+```bash
+markdown-docs-scraper anthropic -o ./docs
+```
+
+### Scrape Any Site
+
+```bash
+markdown-docs-scraper scrape -u https://docs.example.com -o ./docs
+```
+
+### Discover Available Pages
+
+```bash
+markdown-docs-scraper discover -u https://code.claude.com
+```
+
+### Options
+
+```
+Commands:
+  scrape       Scrape documentation from a URL
+  discover     Discover all available documentation pages
+  anthropic    Quick scrape of Anthropic Claude Code docs
+  help         Display help for command
+
+Options:
+  -u, --url <url>          Base URL of the documentation site
+  -o, --output <dir>       Output directory (default: "./docs")
+  --docs-path <path>       Docs path (default: "/docs/en")
+  -c, --concurrency <num>  Concurrency level (default: "5")
+```
+
+## Programmatic Usage
+
+```typescript
+import { MarkdownDocsScraper } from "@ebowwa/markdown-docs-scraper";
+
+const scraper = new MarkdownDocsScraper({
+  baseUrl: "https://code.claude.com",
+  docsPath: "/docs/en",
+  categories: {
+    "getting-started": ["introduction", "installation", "quick-start"],
+    features: ["inline-edits", "tool-use", "file-operations"],
+  },
+  outputDir: "./docs",
+  concurrency: 5,
+});
+
+const result = await scraper.scrape();
+console.log(`Downloaded: ${result.downloaded.length}`);
+console.log(`Failed: ${result.failed.length}`);
+
+// Save pages to disk
+await scraper.savePages(result.downloaded);
+```
+
+### Convenience Function
+
+```typescript
+import { scrapeMarkdownDocs } from "@ebowwa/markdown-docs-scraper";
+
+const result = await scrapeMarkdownDocs({
+  baseUrl: "https://docs.example.com",
+  outputDir: "./docs",
+});
+```
+
+## API
+
+### `MarkdownDocsScraper`
+
+#### Constructor Options
+
+```typescript
+interface ScraperOptions {
+  baseUrl: string;           // Base URL of the documentation site
+  docsPath?: string;         // Docs path (default: "/docs/en")
+  categories?: Record<string, string[]>;  // Categories and pages
+  outputDir?: string;        // Output directory (default: "./docs")
+  concurrency?: number;      // Concurrent downloads (default: 5)
+  onProgress?: (current: number, total: number) => void;
+}
+```
+
+#### Methods
+
+- `scrape()` - Scrape all configured pages
+- `fetchMarkdown(url)` - Fetch markdown from a URL
+- `downloadPage(category, page)` - Download a single page
+- `savePages(pages)` - Save pages to disk
+- `discoverPages()` - Discover available pages
+
+#### Result
+
+```typescript
+interface ScraperResult {
+  downloaded: DocPage[];     // Successfully downloaded pages
+  failed: Array<{ url: string; error: string }>;
+  duration: number;          // Duration in milliseconds
+}
+```
+
+## Output Format
+
+Each downloaded file includes a header comment:
+
+```markdown
+<!--
+Source: https://code.claude.com/docs/en/introduction.md
+Downloaded: 2026-02-06T00:00:00.000Z
+-->
+
+# Introduction
+
+Original markdown content...
+```
+
+## License
+
+MIT
+
+## Contributing
+
+This package is part of the [codespaces](https://github.com/ebowwa/codespaces) monorepo.