contextmd-cli 1.0.0

package/.github/workflows/publish.yml

@@ -0,0 +1,24 @@
+name: Publish to NPM
+
+on:
+  release:
+    types: [created]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      
+      - uses: actions/setup-node@v4
+        with:
+          node-version: '18.x'
+          registry-url: 'https://registry.npmjs.org'
+
+      - run: npm ci
+      - run: npm run build
+      
+      # This publishes to NPM using the token you added to GitHub Secrets
+      - run: npm publish
+        env:
+          NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Antigravity
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,146 @@
+# 🧠 ContextMD
+
+<p align="center">
+  <strong>Feed your Agents the Context they deserve.</strong>
+</p>
+
+<p align="center">
+  <a href="https://github.com/UditAkhourii/contextmd/actions"><img src="https://img.shields.io/github/actions/workflow/status/UditAkhourii/contextmd/ci.yml?branch=main&style=for-the-badge" alt="CI status"></a>
+  <a href="https://www.npmjs.com/package/contextmd-cli"><img src="https://img.shields.io/npm/v/contextmd-cli?style=for-the-badge&color=blue" alt="NPM Version"></a>
+  <a href="LICENSE"><img src="https://img.shields.io/badge/License-MIT-green.svg?style=for-the-badge" alt="MIT License"></a>
+  <a href="https://typescriptlang.org"><img src="https://img.shields.io/badge/Written_in-TypeScript-3178C6?style=for-the-badge&logo=typescript&logoColor=white" alt="TypeScript"></a>
+</p>
+
+---
+
+**ContextMD** is the ultimate terminal utility for turning complex documentation websites into a single, high-density **AI Context File**.
+
+Modern LLMs and Agents (like Claude 3.5 Sonnet, GPT-4o, or Gemini 1.5 Pro) are powerful, but they struggle to navigate multi-page documentation sites effectively. They get lost in navigation bars, footers, duplicate content, and fragmented pages.
+
+**ContextMD solves this.** It crawls, cleans, and chemically refines entire documentation sites into a single `context.md` file that you can drop directly into your LLM's context window.
+
+## ✨ Features
+
+- **🕷️ Deep Crawling**: Intelligently traverses documentation sites, following links and building a comprehensive map of the content.
+- **🧠 AI-Powered Refinement**: Uses OpenAI's models (configurable) to "read" each page and rewrite it for machine comprehension, stripping fluff and prioritizing logic, API signatures, and examples.
+- **🧹 Noise Reduction**: Automatically detects and separates main content from sidebars, headers, footers, and advertisements.
+- **⚡ High Performance**: Concurrent processing with a beautiful, real-time CLI dashboard.
+- **📄 Single File Output**: Produces a consolidated Markdown file with clear headers and structure, perfect for RAG systems or direct LLM context.
+
+## 🚀 Installation
+
+Ensure you have **Node.js 18+** installed.
+
+### Global Install (Recommended)
+
+```bash
+npm install -g contextmd-cli
+```
+
+### Run via npx (No install required)
+
+```bash
+npx contextmd https://docs.example.com
+```
+
+## 🛠️ Usage
+
+### Quick Start
+
+1.  **Get an OpenAI API Key**: ContextMD uses AI to compress and refine the content.
+2.  **Run the tool**:
+
+```bash
+export OPENAI_API_KEY=sk-proj-...
+contextmd https://docs.turso.tech
+```
+
+This will generate a `context.md` file in your current directory.
+
+### Command Line Options
+
+```bash
+Usage: contextmd [options] <url>
+
+Arguments:
+  url                      Base URL of the documentation to convert
+
+Options:
+  -k, --key <key>          OpenAI API Key (can also be set via OPENAI_API_KEY env var)
+  -o, --output <path>      Output file path (default: "context.md")
+  -l, --limit <number>     Max pages to crawl (default: "100")
+  -h, --help               display help for command
+```
+
+### Examples
+
+**Crawl a specific documentation site with a page limit:**
+
+```bash
+contextmd https://developer.spotify.com/documentation/web-api --limit 50
+```
+
+**Save to a specific location:**
+
+```bash
+contextmd https://stripe.com/docs/api -o ./stripe-context.md
+```
+
+## 🏗️ How It Works
+
+ContextMD operates in a three-stage pipeline:
+
+1.  **The Crawler**:
+    *   Starts at the provided `url`.
+    *   Uses a Breadth-First Search (BFS) algorithm to find internal links.
+    *   Filters out external links, social media, and irrelevant pages.
+    *   Respects the `--limit` flag to prevent infinite loops on massive sites.
+
+2.  **The Processor (The "Brain")**:
+    *   Downloads the raw HTML of each discovered page.
+    *   Uses `turndown` and `cheerio` to convert HTML to Markdown.
+    *   **AI Step**: Sends the raw Markdown to an LLM with a specialized system prompt designed to:
+        *   Summarize verbose sections.
+        *   Preserve code blocks and API schemas exactly.
+        *   Remove marketing fluff.
+        *   Standardize formatting.
+
+3.  **The Compiler**:
+    *   Stitches all processed pages into a single `context.md` file.
+    *   Adds a metadata header and table of contents structure (implicitly via markdown headers).
+
+## 📦 For Developers
+
+Want to build this from source?
+
+1.  **Clone the repo**:
+    ```bash
+    git clone https://github.com/UditAkhourii/contextmd.git
+    cd contextmd
+    ```
+
+2.  **Install dependencies**:
+    ```bash
+    npm install
+    ```
+
+3.  **Build**:
+    ```bash
+    npm run build
+    ```
+
+4.  **Run locally**:
+    ```bash
+    node dist/index.js https://example.com
+    ```
+
+## 🤝 Contributing
+
+We welcome contributions! Please open an issue or submit a PR if you have ideas for:
+- Support for local LLMs (Ollama, etc.)
+- Better crawling heuristics for SPA (Single Page Apps).
+- Output formats (JSON, JSONL for fine-tuning).
+
+## 📄 License
+
+MIT © [Antigravity](https://github.com/UditAkhourii)

package/dist/crawler.js

@@ -0,0 +1,107 @@
+"use strict";
+var __createBinding = (this && this.__createBinding) || (Object.create ? (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    var desc = Object.getOwnPropertyDescriptor(m, k);
+    if (!desc || ("get" in desc ? !m.__esModule : desc.writable || desc.configurable)) {
+      desc = { enumerable: true, get: function() { return m[k]; } };
+    }
+    Object.defineProperty(o, k2, desc);
+}) : (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    o[k2] = m[k];
+}));
+var __setModuleDefault = (this && this.__setModuleDefault) || (Object.create ? (function(o, v) {
+    Object.defineProperty(o, "default", { enumerable: true, value: v });
+}) : function(o, v) {
+    o["default"] = v;
+});
+var __importStar = (this && this.__importStar) || (function () {
+    var ownKeys = function(o) {
+        ownKeys = Object.getOwnPropertyNames || function (o) {
+            var ar = [];
+            for (var k in o) if (Object.prototype.hasOwnProperty.call(o, k)) ar[ar.length] = k;
+            return ar;
+        };
+        return ownKeys(o);
+    };
+    return function (mod) {
+        if (mod && mod.__esModule) return mod;
+        var result = {};
+        if (mod != null) for (var k = ownKeys(mod), i = 0; i < k.length; i++) if (k[i] !== "default") __createBinding(result, mod, k[i]);
+        __setModuleDefault(result, mod);
+        return result;
+    };
+})();
+var __importDefault = (this && this.__importDefault) || function (mod) {
+    return (mod && mod.__esModule) ? mod : { "default": mod };
+};
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.Crawler = void 0;
+const axios_1 = __importDefault(require("axios"));
+const cheerio = __importStar(require("cheerio"));
+const url_1 = require("url");
+class Crawler {
+    constructor(baseUrl) {
+        this.visited = new Set();
+        this.queue = [];
+        this.baseUrl = baseUrl;
+        this.domain = new url_1.URL(baseUrl).hostname;
+        this.queue.push(baseUrl);
+    }
+    normalizeUrl(url, currentUrl) {
+        try {
+            const absolute = new url_1.URL(url, currentUrl);
+            // Only keep http(s)
+            if (!['http:', 'https:'].includes(absolute.protocol))
+                return null;
+            // Stay on domain
+            if (absolute.hostname !== this.domain)
+                return null;
+            // Remove hash
+            absolute.hash = '';
+            return absolute.toString();
+        }
+        catch (e) {
+            return null;
+        }
+    }
+    async crawl(maxPages = 500, onUrlFound) {
+        const pages = [];
+        while (this.queue.length > 0 && pages.length < maxPages) {
+            const url = this.queue.shift();
+            if (this.visited.has(url))
+                continue;
+            this.visited.add(url);
+            if (onUrlFound)
+                onUrlFound(url);
+            try {
+                const { data, headers } = await axios_1.default.get(url, {
+                    headers: { 'User-Agent': 'AgenticDocsConverter/1.0' },
+                    timeout: 10000
+                });
+                const contentType = headers['content-type'] || '';
+                if (!contentType.includes('text/html'))
+                    continue;
+                const $ = cheerio.load(data);
+                const title = $('title').text() || url;
+                // Extract links
+                $('a').each((_, element) => {
+                    const href = $(element).attr('href');
+                    if (href) {
+                        const normalized = this.normalizeUrl(href, url);
+                        if (normalized && !this.visited.has(normalized) && !this.queue.includes(normalized)) {
+                            this.queue.push(normalized);
+                        }
+                    }
+                });
+                pages.push({ url, content: data, title });
+            }
+            catch (error) {
+                // console.error(`Failed to crawl ${url}: ${(error as Error).message}`);
+                // Continue despite errors
+            }
+        }
+        return pages;
+    }
+}
+exports.Crawler = Crawler;

package/dist/index.js

@@ -0,0 +1,103 @@
+#!/usr/bin/env node
+"use strict";
+var __createBinding = (this && this.__createBinding) || (Object.create ? (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    var desc = Object.getOwnPropertyDescriptor(m, k);
+    if (!desc || ("get" in desc ? !m.__esModule : desc.writable || desc.configurable)) {
+      desc = { enumerable: true, get: function() { return m[k]; } };
+    }
+    Object.defineProperty(o, k2, desc);
+}) : (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    o[k2] = m[k];
+}));
+var __setModuleDefault = (this && this.__setModuleDefault) || (Object.create ? (function(o, v) {
+    Object.defineProperty(o, "default", { enumerable: true, value: v });
+}) : function(o, v) {
+    o["default"] = v;
+});
+var __importStar = (this && this.__importStar) || (function () {
+    var ownKeys = function(o) {
+        ownKeys = Object.getOwnPropertyNames || function (o) {
+            var ar = [];
+            for (var k in o) if (Object.prototype.hasOwnProperty.call(o, k)) ar[ar.length] = k;
+            return ar;
+        };
+        return ownKeys(o);
+    };
+    return function (mod) {
+        if (mod && mod.__esModule) return mod;
+        var result = {};
+        if (mod != null) for (var k = ownKeys(mod), i = 0; i < k.length; i++) if (k[i] !== "default") __createBinding(result, mod, k[i]);
+        __setModuleDefault(result, mod);
+        return result;
+    };
+})();
+var __importDefault = (this && this.__importDefault) || function (mod) {
+    return (mod && mod.__esModule) ? mod : { "default": mod };
+};
+Object.defineProperty(exports, "__esModule", { value: true });
+const commander_1 = require("commander");
+const ora_1 = __importDefault(require("ora"));
+const chalk_1 = __importDefault(require("chalk"));
+const fs = __importStar(require("fs/promises"));
+const path = __importStar(require("path"));
+require("dotenv/config"); // Load .env if present
+const crawler_1 = require("./crawler");
+const processor_1 = require("./processor");
+const program = new commander_1.Command();
+program
+    .name('contextmd')
+    .description('Convert any documentation into an Agentic AI ready context file')
+    .argument('<url>', 'Base URL of the documentation to convert')
+    .option('-k, --key <key>', 'OpenAI API Key (or set OPENAI_API_KEY env var)')
+    .option('-o, --output <path>', 'Output file path', 'context.md')
+    .option('-l, --limit <number>', 'Max pages to crawl', '100')
+    .action(async (url, options) => {
+    try {
+        console.log(chalk_1.default.bold.cyan('\n🚀 ContextMD - Agentic AI Context Generator\n'));
+        const apiKey = options.key || process.env.OPENAI_API_KEY;
+        if (!apiKey) {
+            console.error(chalk_1.default.red('❌ Error: OpenAI API Key is required. Provide it via -k flag or OPENAI_API_KEY env var.'));
+            process.exit(1);
+        }
+        const crawler = new crawler_1.Crawler(url);
+        const processor = new processor_1.Processor(apiKey);
+        const spinner = (0, ora_1.default)('Initializing crawler...').start();
+        // 1. Crawl
+        spinner.text = `Crawling ${url}...`;
+        const pages = await crawler.crawl(parseInt(options.limit), (foundUrl) => {
+            spinner.text = `Crawling... Found: ${foundUrl}`;
+        });
+        spinner.succeed(chalk_1.default.green(`Crawling complete! Found ${pages.length} pages.`));
+        // 2. Process
+        const outputContent = [];
+        // Header for context.md
+        outputContent.push(`# Documentation Context\n\nGenerated by ContextMD from ${url} at ${new Date().toISOString()}\n\n---\n\n`);
+        const processSpinner = (0, ora_1.default)('Converting and refining pages with AI...').start();
+        // Process sequentially or in small batches to avoid Rate Limits
+        const batchSize = 5;
+        let processedCount = 0;
+        for (let i = 0; i < pages.length; i += batchSize) {
+            const batch = pages.slice(i, i + batchSize);
+            const results = await Promise.all(batch.map(async (page) => {
+                const result = await processor.processPage(page);
+                return result;
+            }));
+            outputContent.push(...results);
+            processedCount += batch.length;
+            processSpinner.text = `Processing pages... (${Math.min(processedCount, pages.length)}/${pages.length})`;
+        }
+        processSpinner.succeed(chalk_1.default.green('Conversion complete!'));
+        // 3. Write
+        const outputPath = path.resolve(process.cwd(), options.output);
+        await fs.writeFile(outputPath, outputContent.join('\n'));
+        console.log(chalk_1.default.bold.green(`\n✅ Success! Agentic context written to: ${outputPath}`));
+        console.log(chalk_1.default.dim(`\nUsage tip: Give this file to your Agent/LLM to fully understand "${url}".\n`));
+    }
+    catch (error) {
+        console.error(chalk_1.default.red('\n❌ Fatal Error:'), error.message);
+        process.exit(1);
+    }
+});
+program.parse(process.argv);

package/dist/processor.js

@@ -0,0 +1,107 @@
+"use strict";
+var __createBinding = (this && this.__createBinding) || (Object.create ? (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    var desc = Object.getOwnPropertyDescriptor(m, k);
+    if (!desc || ("get" in desc ? !m.__esModule : desc.writable || desc.configurable)) {
+      desc = { enumerable: true, get: function() { return m[k]; } };
+    }
+    Object.defineProperty(o, k2, desc);
+}) : (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    o[k2] = m[k];
+}));
+var __setModuleDefault = (this && this.__setModuleDefault) || (Object.create ? (function(o, v) {
+    Object.defineProperty(o, "default", { enumerable: true, value: v });
+}) : function(o, v) {
+    o["default"] = v;
+});
+var __importStar = (this && this.__importStar) || (function () {
+    var ownKeys = function(o) {
+        ownKeys = Object.getOwnPropertyNames || function (o) {
+            var ar = [];
+            for (var k in o) if (Object.prototype.hasOwnProperty.call(o, k)) ar[ar.length] = k;
+            return ar;
+        };
+        return ownKeys(o);
+    };
+    return function (mod) {
+        if (mod && mod.__esModule) return mod;
+        var result = {};
+        if (mod != null) for (var k = ownKeys(mod), i = 0; i < k.length; i++) if (k[i] !== "default") __createBinding(result, mod, k[i]);
+        __setModuleDefault(result, mod);
+        return result;
+    };
+})();
+var __importDefault = (this && this.__importDefault) || function (mod) {
+    return (mod && mod.__esModule) ? mod : { "default": mod };
+};
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.Processor = void 0;
+const turndown_1 = __importDefault(require("turndown"));
+const openai_1 = require("openai");
+const cheerio = __importStar(require("cheerio"));
+class Processor {
+    constructor(apiKey) {
+        this.openai = new openai_1.OpenAI({ apiKey });
+        this.turndown = new turndown_1.default({
+            headingStyle: 'atx',
+            codeBlockStyle: 'fenced'
+        });
+    }
+    cleanHtml(html) {
+        const $ = cheerio.load(html);
+        // Remove clutter
+        $('script').remove();
+        $('style').remove();
+        $('nav').remove();
+        $('footer').remove();
+        $('iframe').remove();
+        $('noscript').remove();
+        $('[role="navigation"]').remove();
+        $('.nav').remove();
+        $('.footer').remove();
+        $('.sidebar').remove(); // Risk: might remove content if class naming is bad, but usually safe for docs sidebar
+        // Get main content if possible
+        const main = $('main').html() || $('article').html() || $('body').html() || '';
+        return main;
+    }
+    async processPage(page) {
+        // 1. Clean HTML
+        const cleanedHtml = this.cleanHtml(page.content);
+        // 2. Convert to Markdown
+        let markdown = this.turndown.turndown(cleanedHtml);
+        // 3. Enhance with LLM
+        // We use a cheap fast model for basic formatting/cleanup
+        try {
+            const response = await this.openai.chat.completions.create({
+                model: 'gpt-4o-mini', // Cost effective
+                messages: [
+                    {
+                        role: 'system',
+                        content: `You are an expert technical writer optimizing documentation for AI Agents.
+                        Your task is to rewrite the provided documentation Markdown to be:
+                        1. Extremely high-density and concise.
+                        2. Optimized for retrieval (keywords, clear logic).
+                        3. Stripped of conversational filler ("In this tutorial you will...").
+                        4. Strictly preserving ALL code blocks and technical constraints.
+                        5. Formatted with clear headers.
+                        
+                        Input is a raw scrape. Fix broken markdown if any.
+                        Return ONLY the refined markdown.`
+                    },
+                    {
+                        role: 'user',
+                        content: `URL: ${page.url}\nTitle: ${page.title}\n\nContent:\n${markdown}`
+                    }
+                ],
+                temperature: 0.1,
+            });
+            return `## Source: [${page.title}](${page.url})\n\n${response.choices[0].message.content || markdown}\n\n---\n\n`;
+        }
+        catch (e) {
+            // Fallback to raw markdown if OpenAI fails
+            return `## Source: [${page.title}](${page.url})\n\n${markdown}\n\n---\n\n`;
+        }
+    }
+}
+exports.Processor = Processor;

package/package.json

@@ -0,0 +1,39 @@
+{
+  "name": "contextmd-cli",
+  "version": "1.0.0",
+  "description": "The ultimate Agentic AI Context Generator for Documentation.",
+  "bin": {
+    "contextmd": "dist/index.js"
+  },
+  "main": "dist/index.js",
+  "scripts": {
+    "build": "tsc",
+    "start": "node dist/index.js",
+    "test": "echo \"Error: no test specified\" && exit 1"
+  },
+  "keywords": [],
+  "author": "Antigravity",
+  "license": "MIT",
+  "type": "commonjs",
+  "devDependencies": {
+    "@types/node": "^25.0.10",
+    "@types/turndown": "^5.0.6",
+    "ts-node": "^10.9.2",
+    "typescript": "^5.9.3"
+  },
+  "dependencies": {
+    "axios": "^1.13.3",
+    "chalk": "^4.1.2",
+    "cheerio": "^1.2.0",
+    "commander": "^14.0.2",
+    "dotenv": "^17.2.3",
+    "gpt-3-encoder": "^1.1.4",
+    "openai": "^6.16.0",
+    "ora": "^5.4.1",
+    "turndown": "^7.2.2"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/UditAkhourii/contextmd.git"
+  }
+}

package/src/crawler.ts

@@ -0,0 +1,83 @@
+
+import axios from 'axios';
+import * as cheerio from 'cheerio';
+import { URL } from 'url';
+
+export interface Page {
+    url: string;
+    content: string; // HTML
+    title: string;
+}
+
+export class Crawler {
+    private visited = new Set<string>();
+    private queue: string[] = [];
+    private baseUrl: string;
+    private domain: string;
+
+    constructor(baseUrl: string) {
+        this.baseUrl = baseUrl;
+        this.domain = new URL(baseUrl).hostname;
+        this.queue.push(baseUrl);
+    }
+
+    private normalizeUrl(url: string, currentUrl: string): string | null {
+        try {
+            const absolute = new URL(url, currentUrl);
+            // Only keep http(s)
+            if (!['http:', 'https:'].includes(absolute.protocol)) return null;
+            // Stay on domain
+            if (absolute.hostname !== this.domain) return null;
+            // Remove hash
+            absolute.hash = '';
+            return absolute.toString();
+        } catch (e) {
+            return null;
+        }
+    }
+
+    async crawl(maxPages: number = 500, onUrlFound?: (url: string) => void): Promise<Page[]> {
+        const pages: Page[] = [];
+        
+        while (this.queue.length > 0 && pages.length < maxPages) {
+            const url = this.queue.shift()!;
+            
+            if (this.visited.has(url)) continue;
+            this.visited.add(url);
+            
+            if (onUrlFound) onUrlFound(url);
+
+            try {
+                const { data, headers } = await axios.get(url, {
+                    headers: { 'User-Agent': 'AgenticDocsConverter/1.0' },
+                    timeout: 10000 
+                });
+
+                const contentType = headers['content-type'] || '';
+                if (!contentType.includes('text/html')) continue;
+
+                const $ = cheerio.load(data);
+                const title = $('title').text() || url;
+                
+                // Extract links
+                $('a').each((_, element) => {
+                    const href = $(element).attr('href');
+                    if (href) {
+                        const normalized = this.normalizeUrl(href, url);
+                        if (normalized && !this.visited.has(normalized) && !this.queue.includes(normalized)) {
+                            this.queue.push(normalized);
+                        }
+                    }
+                });
+
+                pages.push({ url, content: data, title });
+
+            } catch (error) {
+                // console.error(`Failed to crawl ${url}: ${(error as Error).message}`);
+                // Continue despite errors
+            }
+        }
+        
+        return pages;
+    }
+}

package/src/index.ts

@@ -0,0 +1,84 @@
+#!/usr/bin/env node
+import { Command } from 'commander';
+import ora from 'ora';
+import chalk from 'chalk';
+import * as fs from 'fs/promises';
+import * as path from 'path';
+import 'dotenv/config'; // Load .env if present
+
+import { Crawler } from './crawler';
+import { Processor } from './processor';
+
+const program = new Command();
+
+program
+    .name('contextmd')
+    .description('Convert any documentation into an Agentic AI ready context file')
+    .argument('<url>', 'Base URL of the documentation to convert')
+    .option('-k, --key <key>', 'OpenAI API Key (or set OPENAI_API_KEY env var)')
+    .option('-o, --output <path>', 'Output file path', 'context.md')
+    .option('-l, --limit <number>', 'Max pages to crawl', '100')
+    .action(async (url, options) => {
+        try {
+            console.log(chalk.bold.cyan('\n🚀 ContextMD - Agentic AI Context Generator\n'));
+
+            const apiKey = options.key || process.env.OPENAI_API_KEY;
+
+            if (!apiKey) {
+                console.error(chalk.red('❌ Error: OpenAI API Key is required. Provide it via -k flag or OPENAI_API_KEY env var.'));
+                process.exit(1);
+            }
+
+            const crawler = new Crawler(url);
+            const processor = new Processor(apiKey);
+
+            const spinner = ora('Initializing crawler...').start();
+
+            // 1. Crawl
+            spinner.text = `Crawling ${url}...`;
+            const pages = await crawler.crawl(parseInt(options.limit), (foundUrl) => {
+                spinner.text = `Crawling... Found: ${foundUrl}`;
+            });
+
+            spinner.succeed(chalk.green(`Crawling complete! Found ${pages.length} pages.`));
+
+            // 2. Process
+            const outputContent: string[] = [];
+
+            // Header for context.md
+            outputContent.push(`# Documentation Context\n\nGenerated by ContextMD from ${url} at ${new Date().toISOString()}\n\n---\n\n`);
+
+            const processSpinner = ora('Converting and refining pages with AI...').start();
+
+            // Process sequentially or in small batches to avoid Rate Limits
+            const batchSize = 5;
+            let processedCount = 0;
+
+            for (let i = 0; i < pages.length; i += batchSize) {
+                const batch = pages.slice(i, i + batchSize);
+                const results = await Promise.all(batch.map(async (page) => {
+                    const result = await processor.processPage(page);
+                    return result;
+                }));
+
+                outputContent.push(...results);
+                processedCount += batch.length;
+                processSpinner.text = `Processing pages... (${Math.min(processedCount, pages.length)}/${pages.length})`;
+            }
+
+            processSpinner.succeed(chalk.green('Conversion complete!'));
+
+            // 3. Write
+            const outputPath = path.resolve(process.cwd(), options.output);
+            await fs.writeFile(outputPath, outputContent.join('\n'));
+
+            console.log(chalk.bold.green(`\n✅ Success! Agentic context written to: ${outputPath}`));
+            console.log(chalk.dim(`\nUsage tip: Give this file to your Agent/LLM to fully understand "${url}".\n`));
+
+        } catch (error) {
+            console.error(chalk.red('\n❌ Fatal Error:'), (error as Error).message);
+            process.exit(1);
+        }
+    });
+
+program.parse(process.argv);

package/src/processor.ts

@@ -0,0 +1,80 @@
+
+import TurndownService from 'turndown';
+import { OpenAI } from 'openai';
+import * as cheerio from 'cheerio';
+import { Page } from './crawler';
+
+export class Processor {
+    private openai: OpenAI;
+    private turndown: TurndownService;
+
+    constructor(apiKey: string) {
+        this.openai = new OpenAI({ apiKey });
+        this.turndown = new TurndownService({
+            headingStyle: 'atx',
+            codeBlockStyle: 'fenced'
+        });
+    }
+
+    private cleanHtml(html: string): string {
+        const $ = cheerio.load(html);
+
+        // Remove clutter
+        $('script').remove();
+        $('style').remove();
+        $('nav').remove();
+        $('footer').remove();
+        $('iframe').remove();
+        $('noscript').remove();
+        $('[role="navigation"]').remove();
+        $('.nav').remove();
+        $('.footer').remove();
+        $('.sidebar').remove(); // Risk: might remove content if class naming is bad, but usually safe for docs sidebar
+
+        // Get main content if possible
+        const main = $('main').html() || $('article').html() || $('body').html() || '';
+        return main;
+    }
+
+    async processPage(page: Page): Promise<string> {
+        // 1. Clean HTML
+        const cleanedHtml = this.cleanHtml(page.content);
+
+        // 2. Convert to Markdown
+        let markdown = this.turndown.turndown(cleanedHtml);
+
+        // 3. Enhance with LLM
+        // We use a cheap fast model for basic formatting/cleanup
+        try {
+            const response = await this.openai.chat.completions.create({
+                model: 'gpt-4o-mini', // Cost effective
+                messages: [
+                    {
+                        role: 'system',
+                        content: `You are an expert technical writer optimizing documentation for AI Agents.
+                        Your task is to rewrite the provided documentation Markdown to be:
+                        1. Extremely high-density and concise.
+                        2. Optimized for retrieval (keywords, clear logic).
+                        3. Stripped of conversational filler ("In this tutorial you will...").
+                        4. Strictly preserving ALL code blocks and technical constraints.
+                        5. Formatted with clear headers.
+                        
+                        Input is a raw scrape. Fix broken markdown if any.
+                        Return ONLY the refined markdown.`
+                    },
+                    {
+                        role: 'user',
+                        content: `URL: ${page.url}\nTitle: ${page.title}\n\nContent:\n${markdown}`
+                    }
+                ],
+                temperature: 0.1,
+            });
+
+            return `## Source: [${page.title}](${page.url})\n\n${response.choices[0].message.content || markdown}\n\n---\n\n`;
+
+        } catch (e) {
+            // Fallback to raw markdown if OpenAI fails
+            return `## Source: [${page.title}](${page.url})\n\n${markdown}\n\n---\n\n`;
+        }
+    }
+}

package/tsconfig.json

@@ -0,0 +1,15 @@
+
+{
+  "compilerOptions": {
+    "target": "es2020",
+    "module": "commonjs",
+    "outDir": "./dist",
+    "rootDir": "./src",
+    "strict": true,
+    "esModuleInterop": true,
+    "skipLibCheck": true,
+    "forceConsistentCasingInFileNames": true
+  },
+  "include": ["src/**/*"],
+  "exclude": ["node_modules"]
+}