doc-fetch-cli 1.1.0 → 1.1.2

package/INSTALLATION-FIX.md

@@ -0,0 +1,242 @@
+# DocFetch CLI Installation Fix
+
+**Issue**: "Binary not found" error when installing via NPM  
+**Date**: February 20, 2026  
+**Fixed in**: v1.1.2  
+
+---
+
+## 🐛 **Problem Analysis**
+
+### **Symptom 1: "Binary not found" error**
+```bash
+$ npm install -g doc-fetch-cli
+❌ doc-fetch binary not found!
+```
+
+**Root Cause**: The postinstall script wasn't copying the platform-specific binary to the expected location.
+
+### **Symptom 2: Command name confusion**
+```bash
+$ npm install -g doc-fetch-cli
+$ doc-fetch-cli --help    # ❌ Doesn't work
+$ doc-fetch --help        # ✅ Works
+```
+
+**This is actually CORRECT behavior!** Here's why:
+
+In NPM packages:
+- **Package name** (`doc-fetch-cli`): What you install
+- **Bin command** (`doc-fetch`): What you run
+
+This is standard practice. Examples:
+- `npm install -g nodemon` → run `nodemon`
+- `npm install -g typescript` → run `tsc`
+- `npm install -g doc-fetch-cli` → run `doc-fetch`
+
+---
+
+## 🔧 **What Was Fixed**
+
+### **Fix 1: Improved Binary Detection**
+
+Updated `bin/doc-fetch.js` to:
+- ✅ Try multiple possible binary locations
+- ✅ Detect platform and architecture correctly
+- ✅ Provide helpful error messages with troubleshooting steps
+- ✅ Support Linux (amd64/arm64), macOS, Windows
+
+**Before**:
+```javascript
+const binaryPath = path.join(__dirname, '..', 'doc-fetch');
+// Only checked one location, failed if binary wasn't there
+```
+
+**After**:
+```javascript
+const possiblePaths = [
+  path.join(packageDir, binaryName),        // Root directory
+  path.join(packageDir, 'bin', binaryName), // bin/ directory
+  // ... fallbacks
+];
+
+// Tries each location until found
+```
+
+### **Fix 2: Proper Postinstall Script**
+
+Updated `bin/postinstall.js` to:
+- ✅ Copy the correct platform-specific binary
+- ✅ Set executable permissions
+- ✅ Verify the binary works
+- ✅ Provide clear error messages if binary is missing
+
+**Key logic**:
+```javascript
+// Determine which binary to use for this platform
+if (platform === 'linux') {
+  expectedBinary = 'doc-fetch_linux_amd64';
+} else if (platform === 'darwin') {
+  expectedBinary = 'doc-fetch_darwin_amd64';
+}
+
+// Copy to expected location
+fs.copyFileSync(sourcePath, destPath);
+```
+
+### **Fix 3: Added .npmignore**
+
+Created `.npmignore` to ensure all necessary files are included in the NPM package:
+- ✅ Go binaries for all platforms
+- ✅ Bin wrapper scripts
+- ✅ Postinstall script
+- ❌ Excludes: source code, Python files, test files
+
+---
+
+## 📦 **How to Test the Fix**
+
+### **Clean Install Test**
+```bash
+# Uninstall completely
+npm uninstall -g doc-fetch-cli
+
+# Clear npm cache
+npm cache clean --force
+
+# Install fresh
+npm install -g doc-fetch-cli@latest
+
+# Test (note: command is 'doc-fetch' not 'doc-fetch-cli')
+doc-fetch --help
+```
+
+### **Expected Output**
+```
+🎉 DocFetch CLI installing...
+
+📦 Platform: linux x64
+📦 Expected binary: doc-fetch_linux_amd64
+
+✅ Binary installed: doc-fetch
+✅ Binary verified working
+
+✨ DocFetch CLI installed successfully!
+
+Usage:
+   doc-fetch --url https://docs.example.com --output docs.md
+
+Pro tip: Use --llm-txt flag to generate AI-friendly index files!
+```
+
+---
+
+## 🎯 **Platform Support**
+
+| Platform | Architecture | Binary Name | Status |
+|----------|-------------|-------------|--------|
+| Linux | x64 (amd64) | doc-fetch_linux_amd64 | ✅ Supported |
+| Linux | ARM64 | doc-fetch_linux_arm64 | ⚠️ Coming soon |
+| macOS | x64 (amd64) | doc-fetch_darwin_amd64 | ✅ Supported |
+| macOS | ARM64 (M1/M2) | doc-fetch_darwin_arm64 | ⚠️ Coming soon |
+| Windows | x64 (amd64) | doc-fetch_windows_amd64.exe | ✅ Supported |
+
+---
+
+## 🐛 **Troubleshooting**
+
+### **Error: "Binary not found"**
+
+**Solution 1**: Reinstall
+```bash
+npm uninstall -g doc-fetch-cli
+npm install -g doc-fetch-cli
+```
+
+**Solution 2**: Check what was installed
+```bash
+ls -la $(npm root -g)/doc-fetch-cli/
+```
+
+You should see:
+```
+-rwxr-xr-x  doc-fetch              # ← The actual binary
+-rwxr-xr-x  doc-fetch_linux_amd64  # ← Platform-specific binary
+drwxr-xr-x  bin/                   # ← Contains doc-fetch.js wrapper
+```
+
+**Solution 3**: Manual installation
+```bash
+# Download binary directly
+wget https://github.com/AlphaTechini/doc-fetch/releases/download/v1.1.1/doc-fetch_linux_amd64
+chmod +x doc-fetch_linux_amd64
+sudo mv doc-fetch_linux_amd64 /usr/local/bin/doc-fetch
+```
+
+### **Error: "Command not found: doc-fetch-cli"**
+
+**This is expected!** The command is `doc-fetch`, not `doc-fetch-cli`.
+
+```bash
+# Wrong ❌
+doc-fetch-cli --help
+
+# Correct ✅
+doc-fetch --help
+```
+
+### **Error: "Permission denied"**
+
+**Solution**: Fix permissions
+```bash
+# Find installation directory
+DOC_FETCH_DIR=$(npm root -g)/doc-fetch-cli
+
+# Fix permissions
+chmod +x $DOC_FETCH_DIR/doc-fetch
+chmod +x $DOC_FETCH_DIR/bin/doc-fetch.js
+```
+
+---
+
+## 📝 **For Future Releases**
+
+### **Publishing Checklist**
+
+1. Build all platform binaries:
+   ```bash
+   GOOS=linux GOARCH=amd64 go build -o doc-fetch_linux_amd64 ./cmd
+   GOOS=darwin GOARCH=amd64 go build -o doc-fetch_darwin_amd64 ./cmd
+   GOOS=windows GOARCH=amd64 go build -o doc-fetch_windows_amd64.exe ./cmd
+   ```
+
+2. Verify `.npmignore` includes:
+   - ✅ All platform binaries
+   - ✅ `bin/` directory
+   - ✅ `package.json` with correct `bin` field
+
+3. Test installation locally:
+   ```bash
+   npm pack                    # Create tarball
+   npm install -g ./doc-fetch-cli-*.tgz  # Install locally
+   doc-fetch --help            # Test
+   ```
+
+4. Publish:
+   ```bash
+   npm publish
+   ```
+
+---
+
+## 🔗 **Related Issues**
+
+- GitHub Issue: [#XX](https://github.com/AlphaTechini/doc-fetch/issues/XX)
+- NPM Package: https://www.npmjs.com/package/doc-fetch-cli
+- Documentation: https://github.com/AlphaTechini/doc-fetch/blob/main/README.md
+
+---
+
+**Last Updated**: February 20, 2026  
+**Version**: 1.1.2  
+**Status**: ✅ Fixed and tested

package/bin/doc-fetch.js

@@ -2,16 +2,51 @@
 const { spawn } = require('child_process');
 const path = require('path');
 const os = require('os');
+const fs = require('fs');
 
-// Determine binary path based on platform
-const binDir = path.join(__dirname, '..');
-const binaryName = os.platform() === 'win32' ? 'doc-fetch.exe' : 'doc-fetch';
-const binaryPath = path.join(binDir, binaryName);
+// Get the package installation directory
+const packageDir = path.join(__dirname, '..');
 
-// Check if binary exists
-if (!require('fs').existsSync(binaryPath)) {
+// Determine binary name based on platform
+const platform = os.platform();
+const arch = os.arch();
+let binaryName;
+
+if (platform === 'win32') {
+  binaryName = 'doc-fetch.exe';
+} else if (platform === 'darwin') {
+  binaryName = 'doc-fetch_darwin_amd64';
+} else {
+  // Linux and others
+  binaryName = arch === 'arm64' ? 'doc-fetch_linux_arm64' : 'doc-fetch_linux_amd64';
+}
+
+// Try multiple possible locations
+const possiblePaths = [
+  path.join(packageDir, binaryName),                    // Root directory
+  path.join(packageDir, 'bin', binaryName),             // bin/ directory  
+  path.join(packageDir, binaryName.replace('_linux_amd64', '')), // Fallback to generic name
+];
+
+// Find the binary
+let binaryPath = null;
+for (const testPath of possiblePaths) {
+  if (fs.existsSync(testPath)) {
+    binaryPath = testPath;
+    break;
+  }
+}
+
+if (!binaryPath) {
   console.error('❌ doc-fetch binary not found!');
-  console.error('💡 Please run: npm install doc-fetch');
+  console.error('');
+  console.error('💡 Troubleshooting steps:');
+  console.error('   1. Reinstall: npm uninstall -g doc-fetch-cli && npm install -g doc-fetch-cli');
+  console.error('   2. Check installation: ls -la $(npm root -g)/doc-fetch-cli/');
+  console.error('   3. Report issue: https://github.com/AlphaTechini/doc-fetch/issues');
+  console.error('');
+  console.error(`   Expected binary: ${possiblePaths[0]}`);
+  console.error(`   Platform: ${platform} ${arch}`);
   process.exit(1);
 }
 
@@ -24,8 +59,11 @@ const child = spawn(binaryPath, args, {
 
 child.on('error', (err) => {
   if (err.code === 'ENOENT') {
-    console.error('❌ doc-fetch binary not found!');
-    console.error('💡 Please run: npm install doc-fetch');
+    console.error('❌ Failed to execute doc-fetch binary');
+    console.error(`   Binary path: ${binaryPath}`);
+    console.error('   Error: Binary file may be corrupted or missing execute permissions');
+    console.error('');
+    console.error('💡 Try reinstalling: npm uninstall -g doc-fetch-cli && npm install -g doc-fetch-cli');
   } else {
     console.error('❌ Failed to start doc-fetch:', err.message);
   }

package/bin/postinstall.js

@@ -0,0 +1,88 @@
+#!/usr/bin/env node
+/**
+ * Post-install script for doc-fetch-cli
+ * Copies the correct platform-specific binary and sets up PATH
+ */
+
+const { execSync } = require('child_process');
+const path = require('path');
+const os = require('os');
+const fs = require('fs');
+
+console.log('🎉 DocFetch CLI installing...\n');
+
+const packageDir = path.join(__dirname, '..');
+const platform = os.platform();
+const arch = os.arch();
+
+// Determine which binary to use
+let binaryName;
+let expectedBinary;
+
+if (platform === 'win32') {
+  binaryName = 'doc-fetch.exe';
+  expectedBinary = 'doc-fetch_windows_amd64.exe';
+} else if (platform === 'darwin') {
+  binaryName = 'doc-fetch';
+  expectedBinary = 'doc-fetch_darwin_amd64';
+} else {
+  // Linux
+  binaryName = 'doc-fetch';
+  expectedBinary = arch === 'arm64' ? 'doc-fetch_linux_arm64' : 'doc-fetch_linux_amd64';
+}
+
+const sourcePath = path.join(packageDir, expectedBinary);
+const destPath = path.join(packageDir, binaryName);
+
+console.log(`📦 Platform: ${platform} ${arch}`);
+console.log(`📦 Expected binary: ${expectedBinary}\n`);
+
+// Check if the expected binary exists
+if (!fs.existsSync(sourcePath)) {
+  console.error(`⚠️  Warning: Expected binary not found: ${expectedBinary}`);
+  console.error('');
+  console.error('💡 This might be because:');
+  console.error('   1. The package was published without binaries');
+  console.error('   2. Your platform/architecture is not supported');
+  console.error('');
+  console.error('Supported platforms:');
+  console.error('   - Linux x64 (amd64)');
+  console.error('   - macOS x64 (amd64)');  
+  console.error('   - Windows x64 (amd64)');
+  console.error('');
+  console.error('💡 Workaround: Install from source');
+  console.error('   npm uninstall -g doc-fetch-cli');
+  console.error('   git clone https://github.com/AlphaTechini/doc-fetch.git');
+  console.error('   cd doc-fetch && go build -o doc-fetch ./cmd/docfetch');
+  console.error('   sudo cp doc-fetch /usr/local/bin/');
+  process.exit(1);
+}
+
+// Copy the binary to the expected location
+try {
+  fs.copyFileSync(sourcePath, destPath);
+  
+  // Make executable on Unix-like systems
+  if (platform !== 'win32') {
+    fs.chmodSync(destPath, 0o755);
+  }
+  
+  console.log(`✅ Binary installed: ${binaryName}`);
+} catch (error) {
+  console.error(`❌ Failed to install binary: ${error.message}`);
+  process.exit(1);
+}
+
+// Verify installation
+try {
+  const result = execSync(`"${destPath}" --help`, { encoding: 'utf8', stdio: ['pipe', 'pipe', 'ignore'] });
+  console.log('✅ Binary verified working\n');
+} catch (error) {
+  console.error('⚠️  Warning: Could not verify binary execution');
+  console.error(`   Error: ${error.message}\n`);
+}
+
+console.log('✨ DocFetch CLI installed successfully!\n');
+console.log('Usage:');
+console.log('   doc-fetch --url https://docs.example.com --output docs.md\n');
+console.log('Pro tip: Use --llm-txt flag to generate AI-friendly index files!\n');

package/package.json

@@ -1,18 +1,25 @@
 {
   "name": "doc-fetch-cli",
-  "version": "1.1.0",
+  "version": "1.1.2",
   "description": "Dynamic documentation fetching CLI that converts entire documentation sites to single markdown files for AI/LLM consumption",
   "bin": {
     "doc-fetch": "./bin/doc-fetch.js"
   },
   "scripts": {
-    "postinstall": "node ./bin/install.js"
+    "postinstall": "node ./bin/postinstall.js"
   },
   "repository": {
     "type": "git",
     "url": "https://github.com/AlphaTechini/doc-fetch.git"
   },
-  "keywords": ["documentation", "ai", "llm", "markdown", "crawler", "security"],
+  "keywords": [
+    "documentation",
+    "ai",
+    "llm",
+    "markdown",
+    "crawler",
+    "security"
+  ],
   "author": "AlphaTechini",
   "license": "MIT"
-}
+}

package/SECURITY.md

@@ -1,84 +0,0 @@
-# Security Policy
-
-## Security Features
-
-DocFetch includes several built-in security protections:
-
-### ✅ Path Traversal Protection
-- Output files can only be written within the current working directory
-- Relative paths (`../`) are blocked
-- Absolute paths outside the current directory are rejected
-
-### ✅ SSRF (Server-Side Request Forgery) Protection  
-- Only HTTP/HTTPS URLs are allowed
-- Private IP addresses (192.168.x.x, 10.x.x.x, etc.) are blocked
-- Localhost and loopback addresses are blocked
-- Internal network access is prevented
-
-### ✅ Rate Limiting
-- Maximum 10 requests per second to avoid overwhelming servers
-- Respectful crawling behavior
-
-### ✅ Input Validation
-- URL validation and sanitization
-- Output path validation
-- Parameter bounds checking (max depth: 10, max workers: 20)
-
-### ✅ Content Safety
-- HTML content is cleaned of scripts and dangerous elements
-- XSS patterns are filtered out
-- Only safe markdown is generated
-
-## Safe Usage Guidelines
-
-### Command Line Usage
-```bash
-# ✅ SAFE - relative path in current directory
-doc-fetch --url https://example.com --output docs.md
-
-# ✅ SAFE - subdirectory in current directory  
-doc-fetch --url https://example.com --output ./docs/site.md
-
-# ❌ BLOCKED - path traversal attempt
-doc-fetch --url https://example.com --output ../../etc/passwd
-
-# ❌ BLOCKED - absolute path outside current directory
-doc-fetch --url https://example.com --output /tmp/malicious.md
-```
-
-### URL Restrictions
-```bash
-# ✅ SAFE - public HTTPS site
-doc-fetch --url https://golang.org/doc/ --output docs.md
-
-# ❌ BLOCKED - private IP address
-doc-fetch --url http://192.168.1.1/admin --output docs.md
-
-# ❌ BLOCKED - localhost
-doc-fetch --url http://localhost:8080/api --output docs.md
-
-# ❌ BLOCKED - non-HTTP protocol
-doc-fetch --url file:///etc/passwd --output docs.md
-```
-
-## Reporting Security Issues
-
-If you discover a security vulnerability in DocFetch, please:
-
-1. **Do not disclose publicly** until it's been addressed
-2. Contact the maintainer directly at [your email]
-3. Provide detailed reproduction steps
-4. Allow reasonable time for patch development
-
-## Security Updates
-
-Security patches will be released as soon as possible after vulnerability confirmation. Users are encouraged to keep DocFetch updated to the latest version.
-
-## Dependencies Security
-
-DocFetch uses the following dependencies with known security track records:
-- `github.com/PuerkitoBio/goquery` - HTML parsing
-- `github.com/yuin/goldmark` - Markdown processing  
-- Standard Go libraries (`net/http`, `sync`, etc.)
-
-All dependencies are regularly audited and kept up-to-date.

package/cmd/docfetch/main.go

@@ -1,55 +0,0 @@
-package main
-
-import (
-	"flag"
-	"log"
-	"strings"
-
-	"github.com/AlphaTechini/doc-fetch/pkg/fetcher"
-)
-
-func main() {
-	url := flag.String("url", "", "Base URL to fetch documentation from")
-	output := flag.String("output", "docs.md", "Output file path")
-	depth := flag.Int("depth", 2, "Maximum crawl depth")
-	concurrent := flag.Int("concurrent", 3, "Concurrent fetchers")
-	userAgent := flag.String("user-agent", "DocFetch/1.0", "Custom user agent")
-	llmTxt := flag.Bool("llm-txt", false, "Generate llm.txt index file")
-
-	flag.Parse()
-
-	if *url == "" {
-		log.Fatal("Error: URL is required\nUsage: doc-fetch --url <base-url> --output <file-path>")
-	}
-
-	// Validate configuration for security
-	config := fetcher.Config{
-		BaseURL:         *url,
-		OutputPath:      *output,
-		MaxDepth:        *depth,
-		Workers:         *concurrent,
-		UserAgent:       *userAgent,
-		GenerateLLMTxt:  *llmTxt,
-	}
-
-	if err := fetcher.ValidateConfig(&config); err != nil {
-		log.Fatalf("Configuration error: %v", err)
-	}
-
-	// Use optimized high-performance fetcher
-	err := fetcher.RunOptimized(config)
-	if err != nil {
-		log.Fatalf("Failed to fetch documentation: %v", err)
-	}
-
-	log.Printf("Documentation successfully saved to %s", *output)
-	if *llmTxt {
-		llmTxtPath := *output
-		if strings.HasSuffix(*output, ".md") {
-			llmTxtPath = strings.TrimSuffix(*output, ".md") + ".llm.txt"
-		} else {
-			llmTxtPath = *output + ".llm.txt"
-		}
-		log.Printf("LLM.txt index generated: %s", llmTxtPath)
-	}
-}

package/doc_fetch/__init__.py

@@ -1,6 +0,0 @@
-"""
-DocFetch - Dynamic documentation fetching CLI for AI/LLM consumption.
-
-This package provides a Python wrapper around the Go-based DocFetch binary,
-enabling easy installation and usage via pip.
-"""

package/doc_fetch/__main__.py

@@ -1,7 +0,0 @@
-"""
-Module entry point for doc-fetch.
-"""
-from .cli import main
-
-if __name__ == "__main__":
-    main()