doc-fetch-cli 1.1.1 → 1.1.2

package/CONTRIBUTING.md

@@ -1,274 +0,0 @@
-# Contributing to DocFetch
-
-Thank you for your interest in contributing to DocFetch! 🎉 This guide will help you get started.
-
-## 📋 Quick Overview
-
-1. **Create an issue first** - Always start by opening an issue to discuss your change
-2. **Wait for feedback** - Maintainers will respond and provide guidance
-3. **Fork and develop** - Once approved, fork the repo and make your changes
-4. **Submit a PR** - Open a pull request referencing the issue
-5. **Review process** - Maintainers will review and provide feedback
-6. **Merge** - Once approved, your contribution will be merged!
-
----
-
-## 🐛 Before You Start: Create an Issue
-
-**⚠️ IMPORTANT: Always create an issue before submitting a PR!**
-
-This helps us:
-- Avoid duplicate work
-- Discuss the best approach
-- Ensure your contribution aligns with project goals
-- Get early feedback from maintainers
-
-### Types of Issues
-
-#### 🐞 Bug Reports
-Include:
-- Clear description of the bug
-- Steps to reproduce
-- Expected vs actual behavior
-- Environment details (OS, Go version, DocFetch version)
-- Sample command that triggers the bug
-- Error messages or logs
-
-#### ✨ Feature Requests
-Include:
-- Clear description of the feature
-- Use case / problem it solves
-- Example usage
-- Any relevant links or references
-
-#### 📝 Documentation Improvements
-Include:
-- What needs improvement
-- Why it's needed
-- Suggested changes
-
-#### ⚡ Performance Improvements
-Include:
-- Current performance metrics
-- Proposed improvements
-- Benchmark results (if available)
-
----
-
-## 🚀 Development Setup
-
-### Prerequisites
-
-- Go 1.21 or later
-- Git
-- Make (optional, for running tests)
-
-### Fork and Clone
-
-```bash
-# Fork the repository on GitHub, then:
-git clone https://github.com/YOUR_USERNAME/doc-fetch.git
-cd doc-fetch
-
-# Add upstream remote
-git remote add upstream https://github.com/AlphaTechini/doc-fetch.git
-```
-
-### Build from Source
-
-```bash
-# Build the binary
-go build -o doc-fetch ./cmd/docfetch
-
-# Test it works
-./doc-fetch --help
-```
-
-### Run Tests
-
-```bash
-# Run all tests
-go test ./...
-
-# Run tests with coverage
-go test -cover ./...
-
-# Run specific package tests
-go test ./pkg/fetcher/...
-```
-
----
-
-## 💻 Making Changes
-
-### Branch Naming
-
-Use descriptive branch names:
-- `fix/content-extraction-bug`
-- `feat/add-pdf-support`
-- `docs/update-readme-examples`
-- `perf/improve-concurrent-fetching`
-
-### Code Style
-
-Follow Go best practices:
-- Run `go fmt` before committing
-- Run `go vet` to catch issues
-- Write clear, concise comments
-- Keep functions small and focused
-- Use meaningful variable names
-
-### Testing Requirements
-
-- Add tests for new features
-- Ensure existing tests pass
-- Include edge cases
-- Test with real documentation sites
-
-Example test:
-```go
-func TestContentExtraction(t *testing.T) {
-    doc := createTestDocument()
-    content := cleanContent(doc)
-    
-    if len(content) == 0 {
-        t.Error("Expected content to be extracted")
-    }
-    
-    if !strings.Contains(content, "expected text") {
-        t.Error("Expected content to contain specific text")
-    }
-}
-```
-
----
-
-## 📤 Submitting a Pull Request
-
-### PR Checklist
-
-Before submitting your PR, ensure:
-
-- [ ] You created an issue first and referenced it in the PR
-- [ ] Your code follows Go style guidelines
-- [ ] All tests pass (`go test ./...`)
-- [ ] You've added tests for new functionality
-- [ ] You've updated documentation if needed
-- [ ] Your commit messages are clear and descriptive
-- [ ] You've rebased on the latest main branch
-
-### PR Template
-
-When creating your PR, include:
-
-```markdown
-## Description
-Brief description of changes
-
-## Related Issue
-Fixes #123 (or "Related to #123")
-
-## Type of Change
-- [ ] Bug fix
-- [ ] New feature
-- [ ] Breaking change
-- [ ] Documentation update
-- [ ] Performance improvement
-- [ ] Refactoring
-
-## Testing
-Describe how you tested this:
-- [ ] Unit tests added/updated
-- [ ] Manual testing with real docs
-- [ ] Tested on: [list platforms]
-
-## Example Usage
-Show example command and output if applicable
-
-## Checklist
-- [ ] Code follows project guidelines
-- [ ] Self-review completed
-- [ ] Comments added where needed
-- [ ] Tests pass locally
-```
-
----
-
-## 🔍 Review Process
-
-1. **Automated Checks**: CI runs tests and linting
-2. **Maintainer Review**: At least one maintainer reviews
-3. **Feedback**: You may be asked to make changes
-4. **Approval**: Once approved, PR is merged
-5. **Release**: Changes included in next release
-
-Typical timeline: 3-7 days for review
-
----
-
-## 📖 Contribution Ideas
-
-Looking for ways to contribute? Here are some ideas:
-
-### Easy Wins
-- Fix typos in documentation
-- Add more examples to README
-- Improve error messages
-- Add unit tests for existing code
-
-### Intermediate
-- Add support for new documentation site formats
-- Improve content extraction selectors
-- Add progress indicators
-- Enhance LLM.txt generation
-
-### Advanced
-- Add PDF export support
-- Implement incremental updates
-- Add authentication support for private docs
-- Create plugin system for custom extractors
-
----
-
-## 🤝 Community Guidelines
-
-### Be Respectful
-- Treat everyone with respect
-- Welcome newcomers
-- Provide constructive feedback
-- Assume good intentions
-
-### Communication
-- Use clear, concise language
-- Explain your reasoning
-- Ask questions if unsure
-- Respond to feedback promptly
-
-### Collaboration
-- Work with maintainers, not against them
-- Be open to suggestions
-- Help other contributors
-- Share knowledge
-
----
-
-## 📜 License
-
-By contributing to DocFetch, you agree that your contributions will be licensed under the MIT License.
-
----
-
-## ❓ Questions?
-
-- **General questions**: Open a discussion on GitHub
-- **Bug reports**: Create an issue
-- **Feature requests**: Create an issue
-- **Quick questions**: Check existing issues/discussions first
-
----
-
-## 🙏 Thank You!
-
-Your contributions make DocFetch better for everyone. Whether it's a typo fix, a new feature, or better documentation - we appreciate your time and effort!
-
-Happy coding! 🚀

package/SECURITY.md

@@ -1,84 +0,0 @@
-# Security Policy
-
-## Security Features
-
-DocFetch includes several built-in security protections:
-
-### ✅ Path Traversal Protection
-- Output files can only be written within the current working directory
-- Relative paths (`../`) are blocked
-- Absolute paths outside the current directory are rejected
-
-### ✅ SSRF (Server-Side Request Forgery) Protection  
-- Only HTTP/HTTPS URLs are allowed
-- Private IP addresses (192.168.x.x, 10.x.x.x, etc.) are blocked
-- Localhost and loopback addresses are blocked
-- Internal network access is prevented
-
-### ✅ Rate Limiting
-- Maximum 10 requests per second to avoid overwhelming servers
-- Respectful crawling behavior
-
-### ✅ Input Validation
-- URL validation and sanitization
-- Output path validation
-- Parameter bounds checking (max depth: 10, max workers: 20)
-
-### ✅ Content Safety
-- HTML content is cleaned of scripts and dangerous elements
-- XSS patterns are filtered out
-- Only safe markdown is generated
-
-## Safe Usage Guidelines
-
-### Command Line Usage
-```bash
-# ✅ SAFE - relative path in current directory
-doc-fetch --url https://example.com --output docs.md
-
-# ✅ SAFE - subdirectory in current directory  
-doc-fetch --url https://example.com --output ./docs/site.md
-
-# ❌ BLOCKED - path traversal attempt
-doc-fetch --url https://example.com --output ../../etc/passwd
-
-# ❌ BLOCKED - absolute path outside current directory
-doc-fetch --url https://example.com --output /tmp/malicious.md
-```
-
-### URL Restrictions
-```bash
-# ✅ SAFE - public HTTPS site
-doc-fetch --url https://golang.org/doc/ --output docs.md
-
-# ❌ BLOCKED - private IP address
-doc-fetch --url http://192.168.1.1/admin --output docs.md
-
-# ❌ BLOCKED - localhost
-doc-fetch --url http://localhost:8080/api --output docs.md
-
-# ❌ BLOCKED - non-HTTP protocol
-doc-fetch --url file:///etc/passwd --output docs.md
-```
-
-## Reporting Security Issues
-
-If you discover a security vulnerability in DocFetch, please:
-
-1. **Do not disclose publicly** until it's been addressed
-2. Contact the maintainer directly at [your email]
-3. Provide detailed reproduction steps
-4. Allow reasonable time for patch development
-
-## Security Updates
-
-Security patches will be released as soon as possible after vulnerability confirmation. Users are encouraged to keep DocFetch updated to the latest version.
-
-## Dependencies Security
-
-DocFetch uses the following dependencies with known security track records:
-- `github.com/PuerkitoBio/goquery` - HTML parsing
-- `github.com/yuin/goldmark` - Markdown processing  
-- Standard Go libraries (`net/http`, `sync`, etc.)
-
-All dependencies are regularly audited and kept up-to-date.

package/cmd/docfetch/main.go

@@ -1,55 +0,0 @@
-package main
-
-import (
-	"flag"
-	"log"
-	"strings"
-
-	"github.com/AlphaTechini/doc-fetch/pkg/fetcher"
-)
-
-func main() {
-	url := flag.String("url", "", "Base URL to fetch documentation from")
-	output := flag.String("output", "docs.md", "Output file path")
-	depth := flag.Int("depth", 2, "Maximum crawl depth")
-	concurrent := flag.Int("concurrent", 3, "Concurrent fetchers")
-	userAgent := flag.String("user-agent", "DocFetch/1.0", "Custom user agent")
-	llmTxt := flag.Bool("llm-txt", false, "Generate llm.txt index file")
-
-	flag.Parse()
-
-	if *url == "" {
-		log.Fatal("Error: URL is required\nUsage: doc-fetch --url <base-url> --output <file-path>")
-	}
-
-	// Validate configuration for security
-	config := fetcher.Config{
-		BaseURL:         *url,
-		OutputPath:      *output,
-		MaxDepth:        *depth,
-		Workers:         *concurrent,
-		UserAgent:       *userAgent,
-		GenerateLLMTxt:  *llmTxt,
-	}
-
-	if err := fetcher.ValidateConfig(&config); err != nil {
-		log.Fatalf("Configuration error: %v", err)
-	}
-
-	// Use optimized high-performance fetcher
-	err := fetcher.RunOptimized(config)
-	if err != nil {
-		log.Fatalf("Failed to fetch documentation: %v", err)
-	}
-
-	log.Printf("Documentation successfully saved to %s", *output)
-	if *llmTxt {
-		llmTxtPath := *output
-		if strings.HasSuffix(*output, ".md") {
-			llmTxtPath = strings.TrimSuffix(*output, ".md") + ".llm.txt"
-		} else {
-			llmTxtPath = *output + ".llm.txt"
-		}
-		log.Printf("LLM.txt index generated: %s", llmTxtPath)
-	}
-}

package/doc_fetch/__init__.py

@@ -1,6 +0,0 @@
-"""
-DocFetch - Dynamic documentation fetching CLI for AI/LLM consumption.
-
-This package provides a Python wrapper around the Go-based DocFetch binary,
-enabling easy installation and usage via pip.
-"""

package/doc_fetch/__main__.py

@@ -1,7 +0,0 @@
-"""
-Module entry point for doc-fetch.
-"""
-from .cli import main
-
-if __name__ == "__main__":
-    main()

package/doc_fetch/cli.py

@@ -1,113 +0,0 @@
-#!/usr/bin/env python3
-"""
-DocFetch CLI wrapper for Python.
-
-This module provides a Python interface to the Go-based DocFetch binary.
-It handles downloading the appropriate binary for your platform and
-executing it with the provided arguments.
-"""
-
-import os
-import sys
-import subprocess
-import platform
-from pathlib import Path
-
-# Get the directory where this script is located
-SCRIPT_DIR = Path(__file__).parent
-BIN_DIR = SCRIPT_DIR / "bin"
-BINARY_NAME = None
-
-
-def get_binary_name():
-    """Get the appropriate binary name for the current platform."""
-    system = platform.system().lower()
-    machine = platform.machine().lower()
-    
-    # Map machine architectures
-    arch_map = {
-        'x86_64': 'amd64',
-        'amd64': 'amd64', 
-        'arm64': 'arm64',
-        'aarch64': 'arm64'
-    }
-    
-    arch = arch_map.get(machine, 'amd64')
-    
-    if system == 'windows':
-        return f'doc-fetch_windows_{arch}.exe'
-    elif system == 'darwin':
-        return f'doc-fetch_darwin_{arch}'
-    else:  # linux and others
-        return f'doc-fetch_linux_{arch}'
-
-
-def download_binary():
-    """Download the appropriate binary from GitHub releases."""
-    import urllib.request
-    import ssl
-    
-    binary_name = get_binary_name()
-    binary_path = BIN_DIR / binary_name
-    
-    # Create bin directory if it doesn't exist
-    BIN_DIR.mkdir(exist_ok=True)
-    
-    # URL for the binary
-    url = f"https://github.com/AlphaTechini/doc-fetch/releases/download/v1.0.0/{binary_name}"
-    
-    print(f"📥 Downloading doc-fetch binary for {platform.system()} {platform.machine()}...")
-    print(f"   URL: {url}")
-    
-    try:
-        # Create SSL context to handle certificates
-        ssl_context = ssl.create_default_context()
-        
-        # Download the binary
-        with urllib.request.urlopen(url, context=ssl_context) as response:
-            with open(binary_path, 'wb') as f:
-                f.write(response.read())
-        
-        # Make executable on Unix-like systems
-        if platform.system() != 'Windows':
-            os.chmod(binary_path, 0o755)
-        
-        print("✅ Binary downloaded successfully!")
-        return binary_path
-        
-    except Exception as e:
-        print(f"❌ Failed to download binary: {e}")
-        print("💡 Please ensure you have internet access and can reach GitHub.")
-        sys.exit(1)
-
-
-def main():
-    """Main entry point for the doc-fetch CLI."""
-    global BINARY_NAME
-    
-    # Get binary path
-    binary_name = get_binary_name()
-    binary_path = BIN_DIR / binary_name
-    
-    # Download binary if it doesn't exist
-    if not binary_path.exists():
-        binary_path = download_binary()
-    
-    # Execute the binary with all arguments
-    try:
-        result = subprocess.run([str(binary_path)] + sys.argv[1:], check=False)
-        sys.exit(result.returncode)
-    except FileNotFoundError:
-        print("❌ doc-fetch binary not found!")
-        print("💡 This shouldn't happen. Please reinstall the package.")
-        sys.exit(1)
-    except KeyboardInterrupt:
-        print("\n⚠️  Interrupted by user")
-        sys.exit(130)
-    except Exception as e:
-        print(f"❌ Failed to execute doc-fetch: {e}")
-        sys.exit(1)
-
-
-if __name__ == "__main__":
-    main()