doc-fetch-cli 2.0.4 → 2.0.6

package/website/URL-STRUCTURE.md

@@ -0,0 +1,334 @@
+# DocFetch URL Structure & Information Architecture
+
+## 2026 SEO Best Practices for URL Hierarchy
+
+Google uses URL structure to understand:
+1. **Topical hierarchy** - How content relates
+2. **Site architecture** - Content organization
+3. **Context signals** - What the page is about
+
+## Current Structure (❌ Bad)
+
+```
+/blog/convert-docs-to-markdown-for-llm
+/blog/llm-txt-index-guide
+/blog/ai-agent-documentation-problem
+```
+
+**Problems:**
+- Flat structure (no hierarchy)
+- No topical clustering
+- Missed semantic signals
+
+## Target Structure (✅ Good)
+
+### Tier 1: Category Pages (Broad Topics)
+
+```
+/web3/                    - Web3 + AI integration
+/ai-infra/                - AI infrastructure
+/rag/                     - RAG systems
+/llm-tools/               - LLM developer tools
+```
+
+### Tier 2: Subcategory Pages (Specific Topics)
+
+```
+/web3/vector-databases/
+/web3/agent-economics/
+/ai-infra/cost-optimization/
+/ai-infra/security/
+/rag/context-preparation/
+/rag/retrieval-strategies/
+/llm-tools/documentation/
+/llm-tools/token-efficiency/
+```
+
+### Tier 3: Individual Articles (Long-tail)
+
+```
+/web3/vector-databases/hollowdb-review
+/web3/agent-economics/instant-rag-analysis
+/ai-infra/cost-optimization/multi-provider-routing
+/rag/context-preparation/convert-docs-to-markdown
+/rag/context-preparation/llm-txt-guide
+/llm-tools/documentation/docfetch-tutorial
+/llm-tools/token-efficiency/compression-techniques
+```
+
+## Implementation Plan
+
+### Phase 1: Restructure Existing Content
+
+**Current:** `/blog/convert-docs-to-markdown-for-llm`  
+**New:** `/rag/context-preparation/convert-docs-to-markdown`
+
+**Current:** `/blog/llm-txt-index-guide`  
+**New:** `/rag/context-preparation/llm-txt-guide`
+
+**Current:** `/blog/ai-agent-documentation-problem`  
+**New:** `/llm-tools/documentation/ai-agents-cant-read-docs`
+
+### Phase 2: Create Category Landing Pages
+
+Each category gets a landing page that:
+1. Introduces the topic
+2. Lists all articles in that category
+3. Links to subcategories
+4. Targets broad keywords
+
+Example: `/rag/context-preparation/`
+```markdown
+# RAG Context Preparation
+
+Complete guide to preparing, cleaning, and structuring context for Retrieval Augmented Generation systems.
+
+## Topics Covered
+
+### Documentation Conversion
+- [Convert Docs to Markdown for LLMs](/rag/context-preparation/convert-docs-to-markdown)
+- [LLM.txt Explained](/rag/context-preparation/llm-txt-guide)
+
+### Chunking Strategies
+- [Optimal Chunk Sizes for RAG](/rag/context-preparation/chunk-sizes)
+- [Semantic vs Fixed-Size Chunking](/rag/context-preparation/semantic-chunking)
+
+### Cleaning Techniques
+- [Remove HTML Noise from Documents](/rag/context-preparation/remove-html-noise)
+- [Code Block Preservation](/rag/context-preparation/preserve-code-blocks)
+```
+
+### Phase 3: Implement in SvelteKit
+
+#### File Structure
+
+```
+src/routes/
+├── rag/
+│   ├── +page.svelte              # /rag/ category page
+│   └── context-preparation/
+│       ├── +page.svelte          # /rag/context-preparation/ subcategory
+│       └── [slug]/
+│           └── +page.svelte      # /rag/context-preparation/[slug]
+├── llm-tools/
+│   ├── +page.svelte
+│   └── documentation/
+│       ├── +page.svelte
+│       └── [slug]/
+│           └── +page.svelte
+└── web3/
+    ├── +page.svelte
+    └── [subcategory]/
+        └── [slug]/
+            └── +page.svelte
+```
+
+#### Load Function Updates
+
+```typescript
+// src/routes/rag/context-preparation/[slug]/+page.ts
+export const load: PageLoad = async ({ params }) => {
+	const post = getPost(params.slug);
+	
+	return {
+		post,
+		category: 'RAG',
+		subcategory: 'Context Preparation',
+		breadcrumb: [
+			{ label: 'Home', href: '/' },
+			{ label: 'RAG', href: '/rag' },
+			{ label: 'Context Preparation', href: '/rag/context-preparation' },
+			{ label: post.title, href: `/rag/context-preparation/${post.slug}` }
+		]
+	};
+};
+```
+
+#### Breadcrumb Navigation
+
+Add structured breadcrumbs for UX + SEO:
+
+```svelte
+<nav class="breadcrumb" aria-label="Breadcrumb">
+	<ol>
+		{#each data.breadcrumb as crumb}
+			<li>
+				<a href={crumb.href}>{crumb.label}</a>
+			</li>
+		{/each}
+	</ol>
+</nav>
+```
+
+With Schema.org markup:
+
+```json
+{
+  "@context": "https://schema.org",
+  "@type": "BreadcrumbList",
+  "itemListElement": [
+    {
+      "@type": "ListItem",
+      "position": 1,
+      "name": "Home",
+      "item": "https://docfetch.dev"
+    },
+    {
+      "@type": "ListItem",
+      "position": 2,
+      "name": "RAG",
+      "item": "https://docfetch.dev/rag"
+    },
+    {
+      "@type": "ListItem",
+      "position": 3,
+      "name": "Context Preparation",
+      "item": "https://docfetch.dev/rag/context-preparation"
+    },
+    {
+      "@type": "ListItem",
+      "position": 4,
+      "name": "How to Convert Documentation",
+      "item": "https://docfetch.dev/rag/context-preparation/convert-docs-to-markdown"
+    }
+  ]
+}
+```
+
+## Internal Linking Strategy
+
+### Automated Contextual Linking
+
+Create a utility that auto-links keywords:
+
+```typescript
+// lib/auto-link.ts
+const keywordLinks = {
+	'llm.txt': '/rag/context-preparation/llm-txt-guide',
+	'RAG': '/rag',
+	'documentation conversion': '/rag/context-preparation/convert-docs-to-markdown',
+	'token efficiency': '/llm-tools/token-efficiency'
+};
+
+export function autoLinkContent(content: string): string {
+	let linked = content;
+	
+	for (const [keyword, url] of Object.entries(keywordLinks)) {
+		const regex = new RegExp(`\\b(${keyword})\\b`, 'gi');
+		// Don't link inside existing <a> tags
+		linked = linked.replace(regex, `<a href="${url}">$1</a>`);
+	}
+	
+	return linked;
+}
+```
+
+### Related Posts Algorithm
+
+Instead of hardcoded related posts, use tag-based matching:
+
+```typescript
+function findRelatedPosts(currentPost: Post, allPosts: Post[]): Post[] {
+	return allPosts
+		.filter(post => post.slug !== currentPost.slug)
+		.map(post => ({
+			...post,
+			relevanceScore: calculateRelevance(currentPost, post)
+		}))
+		.sort((a, b) => b.relevanceScore - a.relevanceScore)
+		.slice(0, 3);
+}
+
+function calculateRelevance(post1: Post, post2: Post): number {
+	let score = 0;
+	
+	// Same category: +3 points
+	if (post1.category === post2.category) score += 3;
+	
+	// Same subcategory: +5 points
+	if (post1.subcategory === post2.subcategory) score += 5;
+	
+	// Shared tags: +1 per tag
+	const sharedTags = post1.tags.filter(tag => post2.tags.includes(tag));
+	score += sharedTags.length;
+	
+	return score;
+}
+```
+
+## Redirect Strategy (Preserve SEO)
+
+When restructuring URLs, set up 301 redirects:
+
+```javascript
+// vercel.json or netlify.toml
+{
+  "redirects": [
+    {
+      "source": "/blog/convert-docs-to-markdown-for-llm",
+      "destination": "/rag/context-preparation/convert-docs-to-markdown",
+      "permanent": true
+    },
+    {
+      "source": "/blog/llm-txt-index-guide",
+      "destination": "/rag/context-preparation/llm-txt-guide",
+      "permanent": true
+    }
+  ]
+}
+```
+
+## URL Naming Conventions
+
+### ✅ DO
+
+- Use hyphens: `/context-preparation`
+- Lowercase only: `/llm-tools` not `/LLM-Tools`
+- Descriptive slugs: `/convert-docs-to-markdown`
+- Hierarchical: `/category/subcategory/article`
+- Remove stop words: `/rag/context-preparation` not `/rag/how-to-do-context-preparation`
+
+### ❌ DON'T
+
+- Underscores: `/context_preparation` (hard to read)
+- Uppercase: `/RAG/Context-Preparation` (case sensitivity issues)
+- Vague slugs: `/guide-1`, `/article-2`
+- Dates in URL: `/2026/02/21/...` (dates age, content doesn't)
+- Overly long: `/rag/context-preparation/how-to-convert-documentation-to-markdown-for-llms-complete-guide`
+
+## Target Keyword Mapping
+
+Map URLs to target keywords:
+
+| URL | Target Keyword | Search Volume |
+|-----|----------------|---------------|
+| `/rag/context-preparation/convert-docs-to-markdown` | convert documentation to markdown | 2,400/mo |
+| `/rag/context-preparation/llm-txt-guide` | llm.txt generator | 90/mo |
+| `/llm-tools/token-efficiency` | reduce LLM token costs | 720/mo |
+| `/web3/vector-databases/hollowdb-review` | hollowdb vector database | 320/mo |
+| `/ai-infra/cost-optimization` | AI infrastructure cost optimization | 1,200/mo |
+
+## Migration Checklist
+
+- [ ] Create new directory structure in `src/routes/`
+- [ ] Move existing blog posts to new locations
+- [ ] Update all internal links
+- [ ] Set up 301 redirects in `vercel.json`
+- [ ] Update sitemap.xml
+- [ ] Submit new sitemap to Google Search Console
+- [ ] Monitor 404 errors in Search Console
+- [ ] Update social sharing URLs
+- [ ] Test breadcrumb navigation
+- [ ] Verify canonical URLs point to new structure
+
+---
+
+**Why This Matters for SEO:**
+
+1. **Topical Authority** - Google sees deep coverage of each topic
+2. **Semantic Signals** - URL structure reinforces content themes
+3. **Internal Linking** - Natural link flow from category → article
+4. **User Experience** - Clear navigation, users understand where they are
+5. **Crawl Efficiency** - Googlebot understands site structure
+
+This is how you build a **content empire**, not just a blog.

package/website/WEBSITE-SUMMARY.md

@@ -0,0 +1,246 @@
+# DocFetch Website - Build Summary
+
+## What We Built
+
+A **classic, high-signal landing page** for DocFetch using SvelteKit - no AI-generated visual noise, just clean documentation-era aesthetics with god-like SEO.
+
+## Design Philosophy
+
+**Inspired by:**
+- Early GitHub pages (2010-2015 era)
+- Stripe documentation simplicity
+- Vercel's clean aesthetic
+- Classic technical documentation
+
+**Principles:**
+- Content-first layout
+- Clean typography over visual effects
+- Semantic HTML structure
+- Minimal dependencies (no CSS frameworks)
+- Fast load times (<1s on 3G)
+- Accessible by default
+
+## Technical Stack
+
+- **Framework**: SvelteKit 2.x
+- **Language**: TypeScript
+- **Styling**: Vanilla CSS (custom, no Tailwind/Bootstrap)
+- **Build Tool**: Vite
+- **Hosting**: Vercel-ready (also works on Netlify, static hosts)
+
+## Features Implemented
+
+### ✅ Core Pages
+- Single-page landing site with anchor navigation
+- Smooth scroll to sections
+- Sticky header with scroll state
+
+### ✅ Sections
+1. **Hero** - Value prop + command example + CTA buttons
+2. **Features** - 6 feature cards (AI/LLM optimized, LLM.txt, etc.)
+3. **Installation** - 4 install options (Python, Node.js, Go, Binary)
+4. **Usage** - Basic + advanced examples + options table
+5. **Examples** - Real-world use cases (Go, React, custom projects)
+6. **How It Works** - 6-step process visualization
+7. **LLM.txt Section** - Before/after comparison
+8. **Final CTA** - Conversion-focused close
+
+### ✅ SEO Optimization
+- Meta title & description
+- Open Graph tags (Twitter/Facebook/LinkedIn)
+- Twitter Card markup
+- JSON-LD structured data (SoftwareApplication schema)
+- Canonical URL
+- Semantic heading hierarchy
+- Mobile-responsive design
+- Fast first paint
+
+### ✅ Social Proof
+- NPM version badge
+- PyPI version badge
+- Go module badge
+- License badge
+- All from shields.io (auto-updating)
+
+### ✅ Developer Experience
+- Syntax-highlighted code blocks
+- Terminal-style command display
+- Copy-paste ready examples
+- Clear installation paths
+
+### ✅ Performance
+- Zero external CSS frameworks
+- No JavaScript animation libraries
+- Minimal bundle size
+- No web fonts (system fonts only)
+- SVG favicon (tiny, scalable)
+
+## File Structure
+
+```
+website/
+├── src/
+│   ├── routes/
+│   │   ├── +layout.svelte    # Root layout
+│   │   └── +page.svelte      # Main landing page
+├── static/
+│   ├── favicon.svg           # App icon
+│   ├── og.svg                # Social sharing image
+│   └── og.png                # Placeholder for PNG version
+├── package.json              # Dependencies + metadata
+├── svelte.config.js          # SvelteKit config
+├── vite.config.ts            # Vite config
+├── tsconfig.json             # TypeScript config
+├── README.md                 # Dev instructions
+├── DEPLOYMENT.md             # Deployment guide
+├── LAUNCH-CHECKLIST.md       # Pre-launch checklist
+└── WEBSITE-SUMMARY.md        # This file
+```
+
+## Color Palette
+
+- **Primary**: `#0066cc` (Blue - trust, professionalism)
+- **Primary Hover**: `#0052a3` (Darker blue)
+- **Background**: `#ffffff` (White - clean, classic)
+- **Secondary Background**: `#f8f9fa` (Light gray - section separation)
+- **Text Primary**: `#1a1a1a` (Near black - readability)
+- **Text Secondary**: `#4a4a4a` (Dark gray - supporting text)
+- **Text Muted**: `#6b7280` (Gray - tertiary info)
+- **Border**: `#e5e7eb` (Light gray - subtle dividers)
+- **Code Background**: `#1e1e1e` (Dark - VS Code style)
+- **Code Text**: `#d4d4d4` (Light gray - readable on dark)
+
+## Typography
+
+**Font Stack** (system fonts for speed):
+```css
+font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, 
+             Oxygen, Ubuntu, Cantarell, sans-serif;
+```
+
+**Monospace Stack**:
+```css
+font-family: 'SF Mono', Monaco, 'Cascadia Code', monospace;
+```
+
+## Responsive Breakpoints
+
+- **Desktop**: >768px (default)
+- **Mobile**: ≤768px (single column layouts, adjusted font sizes)
+
+## Accessibility Features
+
+- ✅ Proper heading hierarchy (h1 → h2 → h3)
+- ✅ Alt text on all images/badges
+- ✅ Color contrast meets WCAG AA
+- ✅ Keyboard navigable
+- ✅ Focus states visible
+- ✅ Semantic HTML elements
+- ✅ ARIA labels where needed
+
+## SEO Strategy
+
+### Target Keywords
+- "documentation fetcher"
+- "markdown converter"
+- "AI documentation tools"
+- "LLM context preparation"
+- "web scraper for docs"
+- "developer CLI tools"
+
+### Structured Data
+JSON-LD markup includes:
+- Software application type
+- Author information
+- Download URLs
+- Feature list
+- Aggregate rating (placeholder for reviews)
+- Programming languages supported
+
+### Social Sharing
+- Open Graph for Facebook/LinkedIn
+- Twitter Card for X/Twitter
+- Custom OG image (1200x630 SVG)
+
+## Deployment Options
+
+### Recommended: Vercel
+- Zero config deployment
+- Automatic HTTPS
+- Global CDN
+- Instant rollbacks
+- Preview deployments
+
+### Alternative: Netlify
+- Similar features to Vercel
+- Drag-and-drop deploys
+- Form handling (if needed later)
+
+### Self-Hosted
+- Static files only
+- Works with any HTTP server
+- nginx, Apache, Caddy, etc.
+
+## Performance Metrics (Expected)
+
+- **Lighthouse Performance**: 95-100
+- **First Contentful Paint**: <0.8s
+- **Time to Interactive**: <1.2s
+- **Total Bundle Size**: <100KB (gzipped)
+- **Requests**: <10
+
+## Next Steps
+
+### Immediate
+1. Test locally: `npm run dev`
+2. Build: `npm run build`
+3. Deploy to staging
+4. Review on multiple devices
+5. Fix any issues
+
+### Launch
+1. Deploy to production
+2. Configure custom domain
+3. Update main README with website link
+4. Announce launch
+
+### Post-Launch
+1. Add analytics (optional)
+2. Submit to search engines
+3. Monitor performance
+4. Gather user feedback
+
+## Maintenance
+
+### Monthly
+- Dependency updates
+- Broken link checks
+- Analytics review
+
+### Quarterly
+- Content refresh
+- Design audit
+- Competitor analysis
+
+## Credits
+
+**Built by**: AlphaTechini  
+**License**: MIT  
+**Repository**: https://github.com/AlphaTechini/doc-fetch
+
+---
+
+## Key Differentiators
+
+What makes this landing page special:
+
+1. **No AI Slop** - Clean, classic design without generic AI-generated visual noise
+2. **Developer-First** - Built by a developer, for developers
+3. **Performance Obsessed** - Every byte matters, no unnecessary dependencies
+4. **SEO Optimized** - God-like SEO with proper meta tags and structured data
+5. **Accessible** - Works for everyone, regardless of ability
+6. **Timeless** - Won't look dated in 6 months
+7. **Maintainable** - Simple codebase, easy to update
+8. **Fast** - Loads instantly even on slow connections
+
+This is what happens when you prioritize **substance over style** and **function over flash**.