mdcontext 0.0.1 → 0.1.0

package/docs/ROADMAP.md

@@ -0,0 +1,407 @@
+# Roadmap: @hw/mdcontext
+
+## Overview
+
+Build a token-efficient markdown analysis tool for LLM consumption. Each phase delivers testable functionality, building toward a complete system with parsing, semantic search, summarization, and analytics.
+
+## Phases
+
+- [ ] **Phase 1: Core Parsing** — Markdown AST extraction and structure
+- [ ] **Phase 2: Index & Storage** — Persistent indexes, file watching, caching
+- [ ] **Phase 3: Semantic Layer** — Embeddings, vector search
+- [ ] **Phase 4: Summarization** — Hierarchical compression, token optimization
+- [ ] **Phase 5: Analytics** — Performance metrics, query tracking
+- [ ] **Phase 6: Integration** — CLI, MCP server, HumanWork skills
+
+---
+
+## Phase 1: Core Parsing
+
+**Goal:** Extract structured data from markdown files.
+
+### 1.1: Project Setup
+
+- Initialize `packages/hw_mdcontext` in monorepo
+- TypeScript + Effect setup
+- Test infrastructure (vitest)
+- Basic CI integration
+
+**Deliverables:**
+
+- Package scaffolding
+- Build working
+- First test passing
+
+### 1.2: Markdown Parser
+
+- Integrate remark/unified
+- Parse to mdast (Markdown AST)
+- Handle frontmatter (YAML)
+- Handle GFM extensions (tables, task lists)
+
+**Deliverables:**
+
+- `parse(content: string): MdastRoot`
+- Frontmatter extraction
+- Unit tests for various markdown features
+
+### 1.3: Structure Extraction
+
+- Extract heading hierarchy
+- Identify sections (content between headings)
+- Extract code blocks with language tags
+- Extract links (internal, external, images)
+- Extract lists and tables
+
+**Deliverables:**
+
+- `extractStructure(ast): DocumentStructure`
+- Section tree with content
+- Link graph per document
+- Code block inventory
+
+### 1.4: Document Model
+
+- Define document schema (Effect Schema)
+- Section schema with metadata
+- Serialize/deserialize to JSON
+
+**Deliverables:**
+
+- `Document` type with full structure
+- `Section` type with bounds, content, metadata
+- JSON round-trip tests
+
+---
+
+## Phase 2: Index & Storage
+
+**Goal:** Persist parsed data, enable fast lookups, handle updates.
+
+### 2.1: Storage Interface
+
+- Define `MdStore` interface
+- In-memory implementation for testing
+- File-based implementation for persistence
+
+**Deliverables:**
+
+- `MdStore` interface (save, load, query)
+- `MemoryMdStore`
+- `FileMdStore` (JSON files in `.mdcontext/`)
+
+### 2.2: Document Indexing
+
+- Index documents by path
+- Index sections by heading
+- Index links (forward and back)
+- Incremental updates (changed files only)
+
+**Deliverables:**
+
+- Path → Document lookup
+- Heading → Section lookup
+- Backlink index
+- Change detection (mtime, hash)
+
+### 2.3: File Watching
+
+- Watch directory for changes
+- Debounce rapid changes
+- Incremental re-index
+- Configurable ignore patterns
+
+**Deliverables:**
+
+- `watch(dir, options): Effect<void>`
+- `.mdcontextignore` support
+- Debounce logic (default 500ms)
+
+### 2.4: Cache Management
+
+- Cache parsed documents
+- Cache structure indexes
+- Invalidation on file change
+- Size limits and eviction
+
+**Deliverables:**
+
+- LRU cache for documents
+- Index persistence to disk
+- Cache stats (hits, misses, size)
+
+---
+
+## Phase 3: Semantic Layer
+
+**Goal:** Enable meaning-based search via embeddings.
+
+### 3.1: Embedding Interface
+
+- Define `Embedder` interface
+- Pluggable backends (API, local)
+- Batch embedding support
+
+**Deliverables:**
+
+- `Embedder` interface
+- `embed(texts: string[]): Effect<Vector[]>`
+- Configuration for model selection
+
+### 3.2: OpenAI Embeddings
+
+- Implement OpenAI text-embedding-3-small
+- Rate limiting and retry logic
+- Cost tracking
+
+**Deliverables:**
+
+- `OpenAIEmbedder`
+- Automatic batching (max 8k tokens)
+- Cost per query metric
+
+### 3.3: Local Embeddings (Optional)
+
+- Python subprocess for sentence-transformers
+- Or ONNX runtime in Node
+- Fallback when API unavailable
+
+**Deliverables:**
+
+- `LocalEmbedder` (stretch goal)
+- Model download management
+
+### 3.4: Vector Index
+
+- Store embeddings with document/section IDs
+- Similarity search (cosine)
+- FAISS or hnswlib integration
+
+**Deliverables:**
+
+- `VectorIndex` interface
+- `search(query: Vector, k: number): Result[]`
+- Persistence to disk
+
+### 3.5: Semantic Search API
+
+- Text query → embed → search
+- Combine with structural filters
+- Rank and return results
+
+**Deliverables:**
+
+- `semanticSearch(query: string, options): SearchResult[]`
+- Filter by path pattern, heading level
+- Result with score, snippet, location
+
+---
+
+## Phase 4: Summarization
+
+**Goal:** Generate token-efficient summaries at multiple granularities.
+
+### 4.1: Token Counting
+
+- Accurate token counting (tiktoken or similar)
+- Budget management
+- Truncation strategies
+
+**Deliverables:**
+
+- `countTokens(text: string): number`
+- `truncateToTokens(text, limit): string`
+- Model-specific tokenizers (GPT-4, Claude)
+
+### 4.2: Section Summarization
+
+- Extract key points from section
+- Preserve structure indicators
+- Configurable compression ratio
+
+**Deliverables:**
+
+- `summarizeSection(section, options): Summary`
+- Key sentence extraction
+- Heading preservation
+
+### 4.3: Document Summarization
+
+- Hierarchical: summarize sections, then combine
+- TOC generation
+- Key topics extraction
+
+**Deliverables:**
+
+- `summarizeDocument(doc, options): DocSummary`
+- Multi-level output (100, 500, 2000 tokens)
+- Topic list
+
+### 4.4: Context Assembly
+
+- Build LLM-ready context from multiple sources
+- Priority-based inclusion
+- Token budget management
+
+**Deliverables:**
+
+- `assembleContext(sources, budget): string`
+- Source attribution
+- Overflow handling (truncate vs omit)
+
+---
+
+## Phase 5: Analytics
+
+**Goal:** Built-in observability for performance and usage.
+
+### 5.1: Metrics Foundation
+
+- Effect Metrics integration
+- Counter, Gauge, Histogram types
+- Metric naming conventions
+
+**Deliverables:**
+
+- Metrics layer setup
+- Standard metric types
+- Tagging (operation, status)
+
+### 5.2: Performance Metrics
+
+- Query latency (p50, p95, p99)
+- Index build time
+- Cache hit/miss rates
+- Embedding API latency
+
+**Deliverables:**
+
+- `mdcontext_query_duration_ms` histogram
+- `mdcontext_cache_hits_total` counter
+- `mdcontext_index_build_duration_ms` gauge
+
+### 5.3: Usage Metrics
+
+- Queries per time period
+- Token usage (input/output)
+- Most queried documents/sections
+- Search result click-through (if applicable)
+
+**Deliverables:**
+
+- `mdcontext_queries_total` counter
+- `mdcontext_tokens_used` counter
+- Query log with timestamps
+
+### 5.4: Reporting
+
+- Metrics export (Prometheus format)
+- Simple CLI report command
+- Alerting thresholds (optional)
+
+**Deliverables:**
+
+- `mdcontext metrics` CLI command
+- JSON and text output formats
+- Configurable retention
+
+---
+
+## Phase 6: Integration
+
+**Goal:** Make mdcontext usable from CLI, MCP, and HumanWork.
+
+### 6.1: CLI Tool
+
+- `mdcontext index <dir>` — build index
+- `mdcontext search <query>` — semantic search
+- `mdcontext context <path>` — LLM-ready summary
+- `mdcontext structure <path>` — show document structure
+
+**Deliverables:**
+
+- CLI with subcommands
+- Output formats (text, JSON)
+- Config file support
+
+### 6.2: Daemon Mode
+
+- `mdcontext daemon` — run as background service
+- HTTP/IPC API for queries
+- Auto-rebuild on changes
+
+**Deliverables:**
+
+- Daemon process management
+- Query API (REST or IPC)
+- Health check endpoint
+
+### 6.3: MCP Server
+
+- Expose tools for Claude integration
+- `md_search` — semantic search
+- `md_context` — get context for file/section
+- `md_structure` — document outline
+
+**Deliverables:**
+
+- MCP server implementation
+- Tool definitions
+- Claude Desktop/Code integration docs
+
+### 6.4: HumanWork Skills
+
+- `hw-md-search` — search markdown in .humanwork/
+- `hw-md-context` — get context for task/session
+- Integration with session-history
+
+**Deliverables:**
+
+- Skill definitions
+- Integration with existing HumanWork skills
+- Documentation
+
+---
+
+## Progress
+
+| Phase              | Status      | Plans | Completed |
+| ------------------ | ----------- | ----- | --------- |
+| 1. Core Parsing    | Not started | 4     | -         |
+| 2. Index & Storage | Not started | 4     | -         |
+| 3. Semantic Layer  | Not started | 5     | -         |
+| 4. Summarization   | Not started | 4     | -         |
+| 5. Analytics       | Not started | 4     | -         |
+| 6. Integration     | Not started | 4     | -         |
+
+**Total: 25 tasks across 6 phases**
+
+---
+
+## Dependencies
+
+```
+Phase 1 ─────────────────────────────────────────┐
+    │                                            │
+    ▼                                            │
+Phase 2 ──────────────┐                          │
+    │                 │                          │
+    ▼                 ▼                          │
+Phase 3          Phase 4                         │
+    │                 │                          │
+    └────────┬────────┘                          │
+             ▼                                   │
+         Phase 5 ◄───────────────────────────────┘
+             │
+             ▼
+         Phase 6
+```
+
+- Phase 2 depends on Phase 1 (need parser for indexing)
+- Phase 3 & 4 can parallel after Phase 2
+- Phase 5 spans all (analytics hooks added throughout)
+- Phase 6 integrates everything
+
+---
+
+_Created: 2025-01-18_

package/docs/test-links.md

@@ -0,0 +1,9 @@
+# Test Links
+
+This document links to other files.
+
+## Links
+
+- See [PROJECT.md](./PROJECT.md) for project overview
+- See [DESIGN.md](./DESIGN.md) for design details
+- See [ROADMAP.md](./ROADMAP.md) for roadmap

package/package.json

@@ -1,19 +1,78 @@
 {
   "name": "mdcontext",
-  "version": "0.0.1",
-  "description": "Intelligent markdown context extraction for LLMs. Coming soon.",
+  "version": "0.1.0",
+  "description": "Token-efficient markdown analysis tool for LLM consumption",
+  "type": "module",
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/mdcontext/mdcontext.git"
+  },
+  "publishConfig": {
+    "access": "public",
+    "provenance": true
+  },
+  "main": "./dist/index.js",
+  "types": "./dist/index.d.ts",
+  "bin": {
+    "mdcontext": "./dist/cli/main.js",
+    "mdcontext-mcp": "./dist/mcp/server.js"
+  },
   "keywords": [
     "markdown",
-    "context",
     "llm",
-    "semantic-search",
+    "ai",
+    "analysis",
+    "search",
     "embeddings"
   ],
-  "author": "alphab",
+  "author": "",
   "license": "MIT",
-  "repository": {
-    "type": "git",
-    "url": "https://github.com/mdcontext/mdcontext"
+  "engines": {
+    "node": ">=18.0.0"
+  },
+  "dependencies": {
+    "@effect/cli": "^0.73.0",
+    "@effect/platform": "^0.94.0",
+    "@effect/platform-node": "^0.104.0",
+    "@modelcontextprotocol/sdk": "^1.25.2",
+    "chokidar": "^4.0.3",
+    "effect": "^3.19.0",
+    "gray-matter": "^4.0.3",
+    "hnswlib-node": "^3.0.0",
+    "openai": "^6.16.0",
+    "remark": "^15.0.1",
+    "remark-gfm": "^4.0.0",
+    "remark-parse": "^11.0.0",
+    "tiktoken": "^1.0.18",
+    "unified": "^11.0.5",
+    "unist-util-visit": "^5.0.0"
+  },
+  "devDependencies": {
+    "@biomejs/biome": "^2.3.11",
+    "@changesets/changelog-github": "^0.5.2",
+    "@changesets/cli": "^2.29.8",
+    "@types/mdast": "^4.0.4",
+    "@types/node": "^22.10.0",
+    "tsup": "^8.5.1",
+    "typescript": "^5.7.2",
+    "vitest": "^2.1.8"
   },
-  "homepage": "https://mdcontext.com"
-}
+  "scripts": {
+    "postinstall": "node scripts/rebuild-hnswlib.js",
+    "build": "tsup src/cli/main.ts src/mcp/server.ts src/index.ts --format esm --dts",
+    "dev": "tsc --watch",
+    "test": "vitest run",
+    "test:all": "INCLUDE_EMBED_TESTS=true vitest run",
+    "test:rebuild": "REBUILD_TEST_INDEX=true vitest run",
+    "test:all:rebuild": "REBUILD_TEST_INDEX=true INCLUDE_EMBED_TESTS=true vitest run",
+    "test:watch": "vitest",
+    "typecheck": "tsc --noEmit",
+    "lint": "biome lint --write .",
+    "format": "biome format --write . && biome check --write .",
+    "check": "pnpm format && pnpm lint && pnpm typecheck",
+    "clean": "rm -rf dist",
+    "quality": "pnpm build && npx publint && npx attw --pack .",
+    "changeset": "changeset",
+    "release": "changeset publish"
+  }
+}

package/pnpm-workspace.yaml

@@ -0,0 +1,5 @@
+ignoredBuiltDependencies:
+  - '@parcel/watcher'
+  - esbuild
+  - hnswlib-node
+  - msgpackr-extract