doc-fetch 1.0.0__tar.gz

doc_fetch-1.0.0/PKG-INFO

@@ -0,0 +1,219 @@
+Metadata-Version: 2.4
+Name: doc-fetch
+Version: 1.0.0
+Summary: Dynamic documentation fetching CLI that converts entire documentation sites to single markdown files for AI/LLM consumption
+Home-page: https://github.com/AlphaTechini/doc-fetch
+Author: AlphaTechini
+Author-email: AlphaTechini <rehobothokoibu@gmail.com>
+License: MIT
+Project-URL: Homepage, https://github.com/AlphaTechini/doc-fetch
+Project-URL: Repository, https://github.com/AlphaTechini/doc-fetch
+Project-URL: Documentation, https://github.com/AlphaTechini/doc-fetch#readme
+Keywords: documentation,ai,llm,markdown,crawler,security
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Documentation
+Classifier: Topic :: Software Development :: Documentation
+Classifier: Topic :: Utilities
+Requires-Python: >=3.7
+Description-Content-Type: text/markdown
+Dynamic: author
+Dynamic: home-page
+Dynamic: requires-python
+
+# DocFetch - Dynamic Documentation Fetcher 📚
+
+**Transform entire documentation sites into AI-ready, single-file markdown with intelligent LLM.txt indexing**
+
+Most AIs can't navigate documentation like humans do. They can't scroll through sections, click sidebar links, or explore related pages. **DocFetch solves this fundamental problem** by converting entire documentation sites into comprehensive, clean markdown files that contain every section and piece of information in a format that LLMs love.
+
+## 🚀 Why DocFetch is Essential for AI Development
+
+### 🤖 **AI/LLM Optimization**
+- **Single-file consumption**: No more fragmented context across multiple pages
+- **Clean, structured markdown**: Perfect token efficiency for LLM context windows  
+- **Intelligent LLM.txt generation**: AI-friendly index with semantic categorization
+- **Noise removal**: Automatically strips navigation, headers, footers, ads, and buttons
+
+### ⚡ **Developer Productivity**
+- **One command automation**: Replace hours of manual copy-pasting with a single CLI command
+- **Complete documentation access**: Give your AI agents full access to official documentation
+- **Consistent formatting**: Uniform structure across different documentation sites
+- **Version control friendly**: Markdown files work perfectly with Git
+
+### 🎯 **Smart Content Intelligence**
+- **Automatic page classification**: Identifies APIs, guides, references, and examples
+- **Semantic descriptions**: Generates concise, relevant descriptions for each section
+- **URL preservation**: Maintains original source links for verification
+- **Adaptive content extraction**: Works with diverse documentation site structures
+
+### 🔧 **Production Ready**
+- **Concurrent fetching**: Fast downloads with configurable concurrency
+- **Respectful crawling**: Honors robots.txt and includes rate limiting
+- **Cross-platform**: Works on Windows, macOS, and Linux
+- **Multiple installation options**: NPM, Go install, or direct binary download
+
+## 📦 Installation
+
+### NPM (Recommended for JavaScript/Node.js developers)
+```bash
+npm install -g doc-fetch
+```
+
+### Go (For Go developers)
+```bash
+go install github.com/AlphaTechini/doc-fetch/cmd/docfetch@latest
+```
+
+### Direct Binary Download
+Visit [Releases](https://github.com/AlphaTechini/doc-fetch/releases) and download your platform's binary.
+
+## 🎯 Usage
+
+### Basic Usage
+```bash
+# Fetch entire documentation site to single markdown file
+doc-fetch --url https://golang.org/doc/ --output ./docs/golang-full.md
+
+# With LLM.txt generation for AI optimization
+doc-fetch --url https://react.dev/learn --output docs.md --llm-txt
+```
+
+### Advanced Usage
+```bash
+# Comprehensive documentation fetch with all features
+doc-fetch \
+  --url https://docs.example.com \
+  --output ./internal/docs.md \
+  --depth 4 \
+  --concurrent 10 \
+  --llm-txt \
+  --user-agent "MyBot/1.0"
+```
+
+### Command Options
+| Flag | Short | Description | Default |
+|------|-------|-------------|---------|
+| `--url` | `-u` | Base URL to fetch documentation from | **Required** |
+| `--output` | `-o` | Output file path | `docs.md` |
+| `--depth` | `-d` | Maximum crawl depth | `2` |
+| `--concurrent` | `-c` | Number of concurrent fetchers | `3` |
+| `--llm-txt` | | Generate AI-friendly llm.txt index | `false` |
+| `--user-agent` | | Custom user agent string | `DocFetch/1.0` |
+
+## 📁 Output Files
+
+When using `--llm-txt`, DocFetch generates two files:
+
+### `docs.md` - Complete Documentation
+```markdown
+# Documentation
+
+This file contains documentation fetched by DocFetch.
+
+---
+
+## Getting Started
+
+This guide covers installation, setup, and first program...
+
+---
+
+## Language Specification
+
+Complete Go language specification and syntax...
+```
+
+### `docs.llm.txt` - AI-Friendly Index
+```txt
+# llm.txt - AI-friendly documentation index
+
+[GUIDE] Getting Started
+https://golang.org/doc/install
+Covers installation, setup, and first program.
+
+[REFERENCE] Language Specification  
+https://golang.org/ref/spec
+Complete Go language specification and syntax.
+
+[API] net/http
+https://pkg.go.dev/net/http
+HTTP client/server implementation.
+```
+
+## 🌟 Real-World Examples
+
+### Fetch Go Documentation
+```bash
+doc-fetch --url https://golang.org/doc/ --output ./docs/go-documentation.md --depth 4 --llm-txt
+```
+
+### Fetch React Documentation  
+```bash
+doc-fetch --url https://react.dev/learn --output ./docs/react-learn.md --concurrent 10 --llm-txt
+```
+
+### Fetch Your Own Project Docs
+```bash
+doc-fetch --url https://your-project.com/docs/ --output ./internal/docs.md --llm-txt
+```
+
+## 🤖 How LLM.txt Supercharges Your AI
+
+The generated `llm.txt` file acts as a **semantic roadmap** for your AI agents:
+
+1. **Precise Navigation**: Agents can query specific sections without scanning entire documents
+2. **Context Awareness**: Know whether they're looking at an API reference vs. a tutorial
+3. **Efficient Retrieval**: Jump directly to relevant content based on query intent
+4. **Source Verification**: Always maintain links back to original documentation
+
+**Example AI Prompt Enhancement:**
+```
+Instead of: "What does the net/http package do?"
+Your AI can now: "Check the [API] net/http section in llm.txt for HTTP client/server implementation details"
+```
+
+## 🏗️ How It Works
+
+1. **Link Discovery**: Parses the base URL to find all internal documentation links
+2. **Content Fetching**: Downloads all pages concurrently with respect for robots.txt  
+3. **HTML Cleaning**: Removes non-content elements (navigation, headers, footers, etc.)
+4. **Markdown Conversion**: Converts cleaned HTML to structured markdown
+5. **Intelligent Classification**: Categorizes pages as API, GUIDE, REFERENCE, or EXAMPLE
+6. **Description Generation**: Creates concise, relevant descriptions for each section
+7. **Single File Output**: Combines all documentation into one comprehensive file
+8. **LLM.txt Generation**: Creates AI-friendly index with semantic categorization
+
+## 🚀 Future Features
+
+- **Incremental updates**: Only fetch changed pages on subsequent runs
+- **Custom selectors**: Allow users to specify content areas for different sites  
+- **Multiple formats**: Support PDF, JSON, and other output formats
+- **Token counting**: Estimate token usage for LLM context planning
+- **Advanced classification**: Machine learning-based page type detection
+
+## 💡 Why This Exists
+
+Traditional documentation sites are designed for **human navigation**, not **AI consumption**. When working with LLMs, you often need to manually copy-paste multiple sections or provide incomplete context. DocFetch automates this process, giving your AI agents complete access to documentation without the manual overhead.
+
+**Stop wasting time copying documentation. Start building AI agents with complete knowledge.**
+
+## 🤝 Contributing
+
+Contributions are welcome! Please open an issue or pull request on GitHub.
+
+## 📄 License
+
+MIT License
+
+---
+
+**Built with ❤️ for AI developers who deserve better documentation access**

doc_fetch-1.0.0/README.md

@@ -0,0 +1,188 @@
+# DocFetch - Dynamic Documentation Fetcher 📚
+
+**Transform entire documentation sites into AI-ready, single-file markdown with intelligent LLM.txt indexing**
+
+Most AIs can't navigate documentation like humans do. They can't scroll through sections, click sidebar links, or explore related pages. **DocFetch solves this fundamental problem** by converting entire documentation sites into comprehensive, clean markdown files that contain every section and piece of information in a format that LLMs love.
+
+## 🚀 Why DocFetch is Essential for AI Development
+
+### 🤖 **AI/LLM Optimization**
+- **Single-file consumption**: No more fragmented context across multiple pages
+- **Clean, structured markdown**: Perfect token efficiency for LLM context windows  
+- **Intelligent LLM.txt generation**: AI-friendly index with semantic categorization
+- **Noise removal**: Automatically strips navigation, headers, footers, ads, and buttons
+
+### ⚡ **Developer Productivity**
+- **One command automation**: Replace hours of manual copy-pasting with a single CLI command
+- **Complete documentation access**: Give your AI agents full access to official documentation
+- **Consistent formatting**: Uniform structure across different documentation sites
+- **Version control friendly**: Markdown files work perfectly with Git
+
+### 🎯 **Smart Content Intelligence**
+- **Automatic page classification**: Identifies APIs, guides, references, and examples
+- **Semantic descriptions**: Generates concise, relevant descriptions for each section
+- **URL preservation**: Maintains original source links for verification
+- **Adaptive content extraction**: Works with diverse documentation site structures
+
+### 🔧 **Production Ready**
+- **Concurrent fetching**: Fast downloads with configurable concurrency
+- **Respectful crawling**: Honors robots.txt and includes rate limiting
+- **Cross-platform**: Works on Windows, macOS, and Linux
+- **Multiple installation options**: NPM, Go install, or direct binary download
+
+## 📦 Installation
+
+### NPM (Recommended for JavaScript/Node.js developers)
+```bash
+npm install -g doc-fetch
+```
+
+### Go (For Go developers)
+```bash
+go install github.com/AlphaTechini/doc-fetch/cmd/docfetch@latest
+```
+
+### Direct Binary Download
+Visit [Releases](https://github.com/AlphaTechini/doc-fetch/releases) and download your platform's binary.
+
+## 🎯 Usage
+
+### Basic Usage
+```bash
+# Fetch entire documentation site to single markdown file
+doc-fetch --url https://golang.org/doc/ --output ./docs/golang-full.md
+
+# With LLM.txt generation for AI optimization
+doc-fetch --url https://react.dev/learn --output docs.md --llm-txt
+```
+
+### Advanced Usage
+```bash
+# Comprehensive documentation fetch with all features
+doc-fetch \
+  --url https://docs.example.com \
+  --output ./internal/docs.md \
+  --depth 4 \
+  --concurrent 10 \
+  --llm-txt \
+  --user-agent "MyBot/1.0"
+```
+
+### Command Options
+| Flag | Short | Description | Default |
+|------|-------|-------------|---------|
+| `--url` | `-u` | Base URL to fetch documentation from | **Required** |
+| `--output` | `-o` | Output file path | `docs.md` |
+| `--depth` | `-d` | Maximum crawl depth | `2` |
+| `--concurrent` | `-c` | Number of concurrent fetchers | `3` |
+| `--llm-txt` | | Generate AI-friendly llm.txt index | `false` |
+| `--user-agent` | | Custom user agent string | `DocFetch/1.0` |
+
+## 📁 Output Files
+
+When using `--llm-txt`, DocFetch generates two files:
+
+### `docs.md` - Complete Documentation
+```markdown
+# Documentation
+
+This file contains documentation fetched by DocFetch.
+
+---
+
+## Getting Started
+
+This guide covers installation, setup, and first program...
+
+---
+
+## Language Specification
+
+Complete Go language specification and syntax...
+```
+
+### `docs.llm.txt` - AI-Friendly Index
+```txt
+# llm.txt - AI-friendly documentation index
+
+[GUIDE] Getting Started
+https://golang.org/doc/install
+Covers installation, setup, and first program.
+
+[REFERENCE] Language Specification  
+https://golang.org/ref/spec
+Complete Go language specification and syntax.
+
+[API] net/http
+https://pkg.go.dev/net/http
+HTTP client/server implementation.
+```
+
+## 🌟 Real-World Examples
+
+### Fetch Go Documentation
+```bash
+doc-fetch --url https://golang.org/doc/ --output ./docs/go-documentation.md --depth 4 --llm-txt
+```
+
+### Fetch React Documentation  
+```bash
+doc-fetch --url https://react.dev/learn --output ./docs/react-learn.md --concurrent 10 --llm-txt
+```
+
+### Fetch Your Own Project Docs
+```bash
+doc-fetch --url https://your-project.com/docs/ --output ./internal/docs.md --llm-txt
+```
+
+## 🤖 How LLM.txt Supercharges Your AI
+
+The generated `llm.txt` file acts as a **semantic roadmap** for your AI agents:
+
+1. **Precise Navigation**: Agents can query specific sections without scanning entire documents
+2. **Context Awareness**: Know whether they're looking at an API reference vs. a tutorial
+3. **Efficient Retrieval**: Jump directly to relevant content based on query intent
+4. **Source Verification**: Always maintain links back to original documentation
+
+**Example AI Prompt Enhancement:**
+```
+Instead of: "What does the net/http package do?"
+Your AI can now: "Check the [API] net/http section in llm.txt for HTTP client/server implementation details"
+```
+
+## 🏗️ How It Works
+
+1. **Link Discovery**: Parses the base URL to find all internal documentation links
+2. **Content Fetching**: Downloads all pages concurrently with respect for robots.txt  
+3. **HTML Cleaning**: Removes non-content elements (navigation, headers, footers, etc.)
+4. **Markdown Conversion**: Converts cleaned HTML to structured markdown
+5. **Intelligent Classification**: Categorizes pages as API, GUIDE, REFERENCE, or EXAMPLE
+6. **Description Generation**: Creates concise, relevant descriptions for each section
+7. **Single File Output**: Combines all documentation into one comprehensive file
+8. **LLM.txt Generation**: Creates AI-friendly index with semantic categorization
+
+## 🚀 Future Features
+
+- **Incremental updates**: Only fetch changed pages on subsequent runs
+- **Custom selectors**: Allow users to specify content areas for different sites  
+- **Multiple formats**: Support PDF, JSON, and other output formats
+- **Token counting**: Estimate token usage for LLM context planning
+- **Advanced classification**: Machine learning-based page type detection
+
+## 💡 Why This Exists
+
+Traditional documentation sites are designed for **human navigation**, not **AI consumption**. When working with LLMs, you often need to manually copy-paste multiple sections or provide incomplete context. DocFetch automates this process, giving your AI agents complete access to documentation without the manual overhead.
+
+**Stop wasting time copying documentation. Start building AI agents with complete knowledge.**
+
+## 🤝 Contributing
+
+Contributions are welcome! Please open an issue or pull request on GitHub.
+
+## 📄 License
+
+MIT License
+
+---
+
+**Built with ❤️ for AI developers who deserve better documentation access**

doc_fetch-1.0.0/cmd/docfetch/main.go

@@ -0,0 +1,54 @@
+package main
+
+import (
+	"flag"
+	"log"
+	"strings"
+
+	"github.com/AlphaTechini/doc-fetch/pkg/fetcher"
+)
+
+func main() {
+	url := flag.String("url", "", "Base URL to fetch documentation from")
+	output := flag.String("output", "docs.md", "Output file path")
+	depth := flag.Int("depth", 2, "Maximum crawl depth")
+	concurrent := flag.Int("concurrent", 3, "Concurrent fetchers")
+	userAgent := flag.String("user-agent", "DocFetch/1.0", "Custom user agent")
+	llmTxt := flag.Bool("llm-txt", false, "Generate llm.txt index file")
+
+	flag.Parse()
+
+	if *url == "" {
+		log.Fatal("Error: URL is required\nUsage: doc-fetch --url <base-url> --output <file-path>")
+	}
+
+	// Validate configuration for security
+	config := fetcher.Config{
+		BaseURL:         *url,
+		OutputPath:      *output,
+		MaxDepth:        *depth,
+		Workers:         *concurrent,
+		UserAgent:       *userAgent,
+		GenerateLLMTxt:  *llmTxt,
+	}
+
+	if err := fetcher.ValidateConfig(&config); err != nil {
+		log.Fatalf("Configuration error: %v", err)
+	}
+
+	err := fetcher.Run(config)
+	if err != nil {
+		log.Fatalf("Failed to fetch documentation: %v", err)
+	}
+
+	log.Printf("Documentation successfully saved to %s", *output)
+	if *llmTxt {
+		llmTxtPath := *output
+		if strings.HasSuffix(*output, ".md") {
+			llmTxtPath = strings.TrimSuffix(*output, ".md") + ".llm.txt"
+		} else {
+			llmTxtPath = *output + ".llm.txt"
+		}
+		log.Printf("LLM.txt index generated: %s", llmTxtPath)
+	}
+}

doc_fetch-1.0.0/doc_fetch/__init__.py

@@ -0,0 +1,6 @@
+"""
+DocFetch - Dynamic documentation fetching CLI for AI/LLM consumption.
+
+This package provides a Python wrapper around the Go-based DocFetch binary,
+enabling easy installation and usage via pip.
+"""

doc_fetch-1.0.0/doc_fetch/__main__.py

@@ -0,0 +1,7 @@
+"""
+Module entry point for doc-fetch.
+"""
+from .cli import main
+
+if __name__ == "__main__":
+    main()

doc_fetch-1.0.0/doc_fetch/cli.py

@@ -0,0 +1,113 @@
+#!/usr/bin/env python3
+"""
+DocFetch CLI wrapper for Python.
+
+This module provides a Python interface to the Go-based DocFetch binary.
+It handles downloading the appropriate binary for your platform and
+executing it with the provided arguments.
+"""
+
+import os
+import sys
+import subprocess
+import platform
+from pathlib import Path
+
+# Get the directory where this script is located
+SCRIPT_DIR = Path(__file__).parent
+BIN_DIR = SCRIPT_DIR / "bin"
+BINARY_NAME = None
+
+
+def get_binary_name():
+    """Get the appropriate binary name for the current platform."""
+    system = platform.system().lower()
+    machine = platform.machine().lower()
+    
+    # Map machine architectures
+    arch_map = {
+        'x86_64': 'amd64',
+        'amd64': 'amd64', 
+        'arm64': 'arm64',
+        'aarch64': 'arm64'
+    }
+    
+    arch = arch_map.get(machine, 'amd64')
+    
+    if system == 'windows':
+        return f'doc-fetch_windows_{arch}.exe'
+    elif system == 'darwin':
+        return f'doc-fetch_darwin_{arch}'
+    else:  # linux and others
+        return f'doc-fetch_linux_{arch}'
+
+
+def download_binary():
+    """Download the appropriate binary from GitHub releases."""
+    import urllib.request
+    import ssl
+    
+    binary_name = get_binary_name()
+    binary_path = BIN_DIR / binary_name
+    
+    # Create bin directory if it doesn't exist
+    BIN_DIR.mkdir(exist_ok=True)
+    
+    # URL for the binary
+    url = f"https://github.com/AlphaTechini/doc-fetch/releases/download/v1.0.0/{binary_name}"
+    
+    print(f"📥 Downloading doc-fetch binary for {platform.system()} {platform.machine()}...")
+    print(f"   URL: {url}")
+    
+    try:
+        # Create SSL context to handle certificates
+        ssl_context = ssl.create_default_context()
+        
+        # Download the binary
+        with urllib.request.urlopen(url, context=ssl_context) as response:
+            with open(binary_path, 'wb') as f:
+                f.write(response.read())
+        
+        # Make executable on Unix-like systems
+        if platform.system() != 'Windows':
+            os.chmod(binary_path, 0o755)
+        
+        print("✅ Binary downloaded successfully!")
+        return binary_path
+        
+    except Exception as e:
+        print(f"❌ Failed to download binary: {e}")
+        print("💡 Please ensure you have internet access and can reach GitHub.")
+        sys.exit(1)
+
+
+def main():
+    """Main entry point for the doc-fetch CLI."""
+    global BINARY_NAME
+    
+    # Get binary path
+    binary_name = get_binary_name()
+    binary_path = BIN_DIR / binary_name
+    
+    # Download binary if it doesn't exist
+    if not binary_path.exists():
+        binary_path = download_binary()
+    
+    # Execute the binary with all arguments
+    try:
+        result = subprocess.run([str(binary_path)] + sys.argv[1:], check=False)
+        sys.exit(result.returncode)
+    except FileNotFoundError:
+        print("❌ doc-fetch binary not found!")
+        print("💡 This shouldn't happen. Please reinstall the package.")
+        sys.exit(1)
+    except KeyboardInterrupt:
+        print("\n⚠️  Interrupted by user")
+        sys.exit(130)
+    except Exception as e:
+        print(f"❌ Failed to execute doc-fetch: {e}")
+        sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()