mdcontext 0.0.1 → 0.2.0

package/research/frontmatter/COMMENTS-ARE-SKIPPED.md

@@ -0,0 +1,149 @@
+# Comments Are Skipped: The Format Problem
+
+**Date:** 2026-01-28
+**Status:** Critical insight - updates the thesis
+
+---
+
+## The Original Thesis
+
+Frontmatter in file headers → LLMs read first 20 lines → 94% token reduction.
+
+## The Problem
+
+LLMs already read the first 20 lines.
+
+**They skip the comments.**
+
+```typescript
+// ---
+// file: ./auth.ts
+// exports: [validateUser]
+// ---
+```
+
+LLM cognition: "Comment block → noise → skip → find code"
+
+**Frontmatter as comments is invisible.**
+
+---
+
+## The Evidence
+
+When I (Claude) read a file with frontmatter without being told to use it:
+
+1. Registered lines 1-8 as "comment header decoration"
+2. Skipped to line 10+ looking for actual code
+3. Never used the exports/imports metadata
+4. Read the full file to understand what it does
+
+**The data was there. I ignored it.**
+
+---
+
+## The Fix: Format, Not Behavior
+
+The problem isn't LLM read behavior. The problem is the format.
+
+### Comments = Skipped
+
+```typescript
+// ---
+// exports: [validateUser]
+// ---
+```
+
+Invisible. Dead on arrival.
+
+### Self-Announcing Header = Visible
+
+```typescript
+// --- FMM ---
+// exports: [validateUser]
+// ---
+```
+
+LLM sees `FMM` → pattern match → "this is metadata"
+
+### Code = Parsed
+
+```typescript
+export const __meta = {
+  exports: ["validateUser"],
+  imports: ["crypto"],
+  loc: 234
+};
+```
+
+LLM reads this as code. It's visible.
+
+### JSON = Queryable
+
+```json
+// .fmm/index.json
+{
+  "src/auth.ts": {
+    "exports": ["validateUser"],
+    "imports": ["crypto"],
+    "loc": 234
+  }
+}
+```
+
+LLM queries this before reading files. No comments to skip.
+
+---
+
+## The Updated Model
+
+| Format | Human Readable | LLM Visible | Recommendation |
+|--------|----------------|-------------|----------------|
+| Comment frontmatter | Yes | **No** | Keep for humans |
+| Code export | Yes | Yes | Bundler issues |
+| Manifest JSON | No | **Yes** | Add for LLMs |
+
+**Generate both:**
+- Inline comments → human readability
+- Manifest JSON → LLM queryability
+
+---
+
+## Implications for mdcontext
+
+mdcontext is about giving LLMs exactly what they need.
+
+**Lesson:** Format matters as much as content.
+
+- Markdown headers → LLMs parse these (structured)
+- Markdown prose → LLMs read this (content)
+- Code comments → LLMs skip this (noise)
+
+When designing LLM-readable formats:
+1. **Avoid comment syntax** - it signals "ignore me"
+2. **Use structured data** - JSON, YAML, code
+3. **Put it where LLMs look** - separate files, explicit markers
+
+---
+
+## The Meta Point
+
+We're generating markdown research docs.
+
+mdcontext exists to make markdown LLM-readable.
+
+The insight applies recursively:
+- **Structure** your markdown (headers, lists) → LLMs parse it
+- **Avoid** wall-of-text prose → LLMs skim it
+- **Use** explicit markers for key info → LLMs find it
+
+This document uses:
+- Headers for navigation
+- Tables for comparison
+- Code blocks for examples
+- Short paragraphs for scannability
+
+**Format is interface.**
+
+---
+
+*Captured: 2026-01-28*

package/research/frontmatter/LLM-CODE-NAVIGATION.md

@@ -0,0 +1,276 @@
+# LLM Code Navigation: The Case for Frontmatter
+
+**The 94% Solution**
+
+---
+
+## Abstract
+
+LLMs waste tokens reading entire files to understand what they do. By adding structured metadata (frontmatter) to the first 10 lines of source files, LLMs can triage and navigate codebases with **88-97% fewer tokens** while maintaining equivalent accuracy.
+
+This is not a developer tool. It's infrastructure for LLM cost reduction.
+
+---
+
+## The Problem
+
+When an LLM explores code, it follows a predictable pattern:
+
+```
+grep "thing" → find files → read files → understand code
+```
+
+The bottleneck is step 3. For every grep match, the LLM reads the entire file to understand:
+- What does this file export?
+- What does it depend on?
+- Is this the file I'm looking for?
+
+**Example:**
+```
+grep "validateUser" → 10 matches
+
+Read file 1 (400 lines) → wrong file
+Read file 2 (600 lines) → wrong file
+Read file 3 (200 lines) → this is it
+
+Total: 1,200 lines read to find the right context
+```
+
+Multiply this across every task, every file, every codebase. Tokens add up. Costs add up.
+
+---
+
+## The Solution
+
+**Frontmatter:** structured metadata in the first 10 lines of every source file.
+
+```typescript
+// ---
+// file: ./src/auth/session.ts
+// exports: [validateUser, createSession, destroySession]
+// imports: [crypto, ./database]
+// dependencies: [./types, ./config]
+// loc: 234
+// modified: 2026-01-27
+// ---
+
+import { createHash } from 'crypto';
+// ... rest of file
+```
+
+**The new workflow:**
+```
+grep "validateUser" → 10 matches
+
+Read first 15 lines of file 1 → exports: [UserService] → skip
+Read first 15 lines of file 2 → exports: [AuthMiddleware] → skip
+Read first 15 lines of file 3 → exports: [validateUser] → match!
+Read full file 3 (200 lines)
+
+Total: 245 lines read
+Savings: 80%
+```
+
+---
+
+## The Evidence
+
+Controlled experiments comparing LLM code navigation with and without frontmatter:
+
+### Experiment Results
+
+| Task | Control (no FMM) | FMM | Reduction |
+|------|------------------|-----|-----------|
+| Review recent changes | 1,824 lines | 65 lines | **96%** |
+| Refactor impact analysis | 2,800 lines | 345 lines | **88%** |
+| Architecture exploration | 7,135 lines | 180 lines | **97.5%** |
+
+**Test environment:** 244-file TypeScript codebase (81,732 total lines)
+
+### Quality Comparison
+
+| Metric | Control | FMM |
+|--------|---------|-----|
+| Files correctly identified | ✓ | ✓ |
+| Architecture diagrams produced | ✓ | ✓ |
+| Dependencies mapped | ✓ | ✓ |
+| Accuracy | Equivalent | Equivalent |
+
+**Same output. 94% fewer tokens.**
+
+---
+
+## Why It Works
+
+Frontmatter answers the three questions LLMs ask about every file:
+
+1. **What does this file do?** → `exports: [...]`
+2. **What does it depend on?** → `imports: [...]`, `dependencies: [...]`
+3. **How big is it?** → `loc: 234`
+
+With these answers in the first 15 lines, the LLM can triage without reading the full file.
+
+### The Triage Decision Tree
+
+```
+Read frontmatter (15 lines)
+    │
+    ├── Exports match what I'm looking for?
+    │   ├── Yes → Read full file
+    │   └── No → Skip (saved 200+ lines)
+    │
+    └── Dependencies relevant to my task?
+        ├── Yes → Read full file
+        └── No → Skip (saved 200+ lines)
+```
+
+---
+
+## The Economics
+
+### Per-Request Savings
+
+| Scenario | Without FMM | With FMM | Savings |
+|----------|-------------|----------|---------|
+| Simple lookup | 500 lines | 65 lines | 87% |
+| Refactoring task | 3,000 lines | 400 lines | 87% |
+| Architecture review | 7,000 lines | 200 lines | 97% |
+
+### At Scale
+
+Assuming:
+- 1,000 LLM coding requests/day
+- Average 2,000 lines read per request
+- $0.01 per 1K tokens (input)
+- ~4 chars per token
+
+**Without FMM:** 2M lines × 1000 requests = 2B lines/day = ~$5,000/day
+**With FMM (90% reduction):** ~$500/day
+
+**Annual savings: ~$1.6M** (per organization at this scale)
+
+---
+
+## The Crossover Point
+
+Frontmatter has overhead: ~8-10 lines per file for the metadata block.
+
+**FMM wins when:** `files_skipped × avg_file_size > frontmatter_overhead`
+
+| Codebase | Files | Avg LOC | Break-Even | FMM Value |
+|----------|-------|---------|------------|-----------|
+| Tiny | 4 | 30 | Skip 3+ files | Marginal |
+| Small | 50 | 100 | Skip 5+ files | Positive |
+| Medium | 200 | 200 | Skip 10+ files | Strong |
+| Large | 500+ | 300+ | Skip 15+ files | Massive |
+
+**Real codebases are medium-to-large. FMM wins by default.**
+
+---
+
+## The Adoption Path
+
+### What Doesn't Work
+
+- ❌ Manifest files (`.fmm/index.json`) - adds complexity
+- ❌ Discovery mechanisms - overengineered
+- ❌ CLAUDE.md hints - project-specific
+- ❌ New developer tooling - adoption friction
+
+### What Works
+
+The LLM workflow is:
+```
+grep → find files → READ files → understand
+```
+
+Frontmatter changes only the READ step:
+```
+grep → find files → READ FIRST 15 LINES → decide → maybe read rest
+```
+
+**The adoption path:**
+1. Codebases add frontmatter (`fmm generate src/`)
+2. LLM tools adopt "peek first" as default behavior
+
+No new tools for developers. No discovery layers. Just a behavior change in how LLMs read files.
+
+---
+
+## The Thesis
+
+**Frontmatter is infrastructure for LLM cost reduction.**
+
+Every codebase with frontmatter = cheaper to work with.
+Every LLM tool that peeks first = cheaper to run.
+
+The more codebases have frontmatter, the more pressure on LLM tools to optimize for it. The more tools optimize, the more value codebases get from adding it.
+
+**This is a coordination game with positive-sum economics.**
+
+---
+
+## Implementation
+
+### fmm (Frontmatter Matters)
+
+CLI tool to generate and maintain frontmatter:
+
+```bash
+# Add frontmatter to all TypeScript files
+fmm generate src/
+
+# Update existing frontmatter
+fmm update src/
+
+# Validate frontmatter is current (CI integration)
+fmm validate src/
+```
+
+**Supported languages:** TypeScript, JavaScript, Python, Rust, Go
+
+**Performance:** ~1,000 files/second on M1 Mac
+
+### Frontmatter Format
+
+```typescript
+// ---
+// file: ./relative/path.ts
+// exports: [namedExport1, namedExport2, DefaultExport]
+// imports: [external-package, ./local-dep]
+// dependencies: [./types, ./utils]
+// loc: 234
+// modified: 2026-01-27
+// ---
+```
+
+### Integration Points
+
+- **Pre-commit hooks:** Ensure frontmatter stays in sync
+- **CI validation:** `fmm validate` fails if frontmatter is stale
+- **Editor plugins:** Auto-update on save
+
+---
+
+## Conclusion
+
+LLMs reading code is expensive. Frontmatter makes it cheap.
+
+The evidence is clear: **88-97% token reduction** on real tasks, with equivalent accuracy.
+
+The path is simple: add frontmatter to codebases, change LLM read behavior to peek first.
+
+The economics do the rest.
+
+---
+
+## References
+
+- Experiment data: `fmm/research/exp13/`
+- fmm CLI: `github.com/mdcontext/fmm`
+- mdcontext: `github.com/mdcontext/mdcontext`
+
+---
+
+*Research conducted January 2026*
+*Stuart Robinson & Claude Opus 4.5*