ctxpkg 0.0.1

package/README.md

@@ -0,0 +1,282 @@
+# ctxpkg
+
+<p align="center">
+  <img src="docs/assets/banner.jpg" alt="ctxpkg banner" width="100%">
+</p>
+
+A package manager for AI agent context — manage, sync, and distribute documentation collections for AI-assisted development.
+
+> **The Vision:** Imagine an AI assistant that knows your context — your team's commit style, your company's security policies, your preferred patterns — without you explaining it every session.
+> [Read the story: **Context Stacking: How Sarah Automated Her Team's Brain**](docs/managing-ai-context-at-scale.md)
+
+## What is ctxpkg?
+
+Just as `npm` manages code dependencies, `ctxpkg` manages **context dependencies**.
+
+Stack documentation layers — from personal notes to team guidelines to project docs — into a unified knowledge base. Your AI agents search this indexed context instead of relying on stale training data or manual copy-paste.
+
+**Key capabilities:**
+
+- **Context Stacking** — Layer documentation from multiple sources (personal, team, project, global)
+- **Semantic Search** — Local vector + keyword search finds relevant content without dumping everything into prompts
+- **MCP Integration** — AI editors like Cursor and Claude Desktop can query your context directly
+- **Git-Native Distribution** — Index docs directly from any git repo (public or private) — no publishing required
+- **Bundle Any Source** — Export docs from Confluence, Notion, or any system to markdown, then package into distributable `.tar.gz` archives
+
+## Design Philosophy
+
+**Zero-friction adoption.** You probably already have documentation worth indexing — a folder of markdown notes, an Obsidian vault, your company's engineering wiki, or a repo full of ADRs and guides. ctxpkg works with what you have. Point it at existing files and start searching. No migration, no reformatting, no custom schemas required.
+
+**Low-risk investment.** Even if you decide ctxpkg isn't for you, any documentation you create remains useful. It's just markdown files with a simple manifest — humans can read it, other tools can consume it, and nothing is locked into a proprietary format. The worst case scenario is you end up with better-organized documentation.
+
+## Installation
+
+`npm i -g ctxpkg` or run command with `npx` prefix (`npx ctxpkg col init`)
+
+## Quick Start
+
+Get your AI agents access to your documentation in minutes:
+
+```bash
+# Initialize project config
+ctxpkg col init
+
+# Add your docs folder (requires manifest.json)
+ctxpkg col add docs ./docs/manifest.json
+
+# Index the documents
+ctxpkg col sync
+```
+
+Now configure your AI editor to use the ctxpkg MCP server:
+
+<details>
+<summary>🔧 Cursor</summary>
+
+Add to `~/.cursor/mcp.json`:
+
+```json
+{
+  "mcpServers": {
+    "ctxpkg": {
+      "command": "npx",
+      "args": ["-y", "ctxpkg", "mcp", "documents"]
+    }
+  }
+}
+```
+
+</details>
+
+<details>
+<summary>🤖 Claude Code</summary>
+
+Run this command:
+
+```bash
+claude mcp add ctxpkg -- npx -y ctxpkg mcp documents
+```
+
+</details>
+
+<details>
+<summary>⚡ Opencode</summary>
+
+Add to your Opencode configuration:
+
+```json
+{
+  "mcp": {
+    "ctxpkg": {
+      "type": "local",
+      "command": ["npx", "-y", "ctxpkg", "mcp", "documents"],
+      "enabled": true
+    }
+  }
+}
+```
+
+</details>
+
+**[See more AI editor setups](docs/setup-agents.md)** • **[Full tutorial: Getting Started](docs/getting-started.md)**
+
+## Documentation
+
+| Guide                                              | Description                                       |
+| -------------------------------------------------- | ------------------------------------------------- |
+| [AI Editor Setup](docs/setup-agents.md)            | Configure Cursor, Claude Code, Opencode, and more |
+| [Getting Started](docs/getting-started.md)         | First-time setup tutorial                         |
+| [CLI Reference](docs/cli-reference.md)             | Complete command documentation                    |
+| [Configuration](docs/configuration.md)             | Project config, global config, manifests          |
+| [How It Works](docs/how-it-works.md)               | Indexing pipeline, search algorithms              |
+| [MCP Server](docs/mcp-server.md)                   | AI editor integration and tools                   |
+| [AI Chat & Agent Mode](docs/ai-chat.md)            | Chat with docs, reduced-token MCP mode            |
+| [Agent Testing](docs/agent-testing.md)             | Validate agent performance with test suites       |
+| [Publishing Packages](docs/github-distribution.md) | Distribute docs via GitHub Releases               |
+
+## CLI Management Tools
+
+The CLI is primarily for managing your context collections. Most users will interact with ctxpkg through their AI editor via MCP.
+
+```bash
+# Collections — manage context packages
+ctxpkg col init                    # Initialize project
+ctxpkg col add <alias> <url>       # Add a collection
+ctxpkg col add -g <alias> <url>    # Add global collection
+ctxpkg col sync                    # Index documents
+ctxpkg col list                    # Show collections
+
+# MCP — AI editor integration (main use case)
+ctxpkg mcp docs                    # Start MCP server (tools mode)
+ctxpkg mcp agent                   # Start MCP server (agent mode)
+
+# Additional tools
+ctxpkg docs search "query"         # Direct search (testing)
+ctxpkg chat "question"             # AI-powered Q&A
+ctxpkg agent test tests.yaml       # Test agent performance
+ctxpkg daemon start                # Background service
+```
+
+See [CLI Reference](docs/cli-reference.md) for complete documentation.
+
+## Example: Context Stacking
+
+Layer context from multiple sources:
+
+```json
+{
+  "collections": {
+    "project-docs": {
+      "url": "file://./docs/manifest.json"
+    },
+    "team-standards": {
+      "url": "git+https://github.com/myorg/standards#main?manifest=manifest.json"
+    },
+    "react": {
+      "url": "git+https://github.com/facebook/react#v18.2.0?manifest=docs/manifest.json"
+    }
+  }
+}
+```
+
+**Git repositories are the easiest way to share documentation** — no publishing step required. Just point to a repo with a `manifest.json`:
+
+```bash
+# Add docs from any git repo (HTTPS or SSH)
+ctxpkg col add team-docs "git+https://github.com/myorg/docs#main?manifest=manifest.json"
+ctxpkg col add private-docs "git+ssh://git@github.com/myorg/private#main?manifest=manifest.json"
+```
+
+Add personal/global context available across all projects:
+
+```bash
+ctxpkg col add -g my-notes file:///Users/me/notes/manifest.json
+```
+
+## MCP Integration
+
+ctxpkg's primary purpose is giving AI agents access to your documentation through the **Model Context Protocol (MCP)**. Once configured, your AI assistant gains access to 8 document tools:
+
+- `search` - Semantic search across all your documentation
+- `search_batch` - Multiple queries in one call
+- `get_document` - Retrieve full document content
+- `get_section` - Get specific document sections
+- `get_outline` - Get document structure/outline
+- `find_related` - Find related documents
+- `list_collections` - List all indexed collections
+- `list_documents` - List all documents in collections
+
+### Agent Mode (Recommended for Chat)
+
+For reduced token costs in long conversations, use **Agent Mode**:
+
+```json
+{
+  "mcpServers": {
+    "ctxpkg-agent": {
+      "command": "npx",
+      "args": ["-y", "ctxpkg", "mcp", "agent"]
+    }
+  }
+}
+```
+
+This exposes a single `ask_documents` tool that uses an internal AI agent to search and synthesize answers. The calling agent sees only the final result, not intermediate search calls — reducing context overhead.
+
+See [MCP Server Documentation](docs/mcp-server.md) for complete details.
+
+## AI Chat & Agent Mode
+
+Chat with your documentation directly from the terminal, or use **Agent Mode** for reduced token costs in AI assistants.
+
+```bash
+# Configure your LLM
+ctxpkg config set llm.apiKey sk-...
+
+# One-shot question
+ctxpkg chat "How do I implement caching?" --use-case "Optimizing API performance"
+
+# Interactive session
+ctxpkg chat -i
+```
+
+**Agent Mode MCP** exposes a single `ask_documents` tool that uses an internal AI agent to search and synthesize answers. The calling agent sees only the final result, not intermediate search calls — reducing context overhead in long conversations.
+
+```json
+{
+  "mcpServers": {
+    "ctxpkg-agent": {
+      "command": "ctxpkg",
+      "args": ["mcp", "agent"]
+    }
+  }
+}
+```
+
+See [AI Chat & Agent Mode](docs/ai-chat.md) for details.
+
+## Distributing Internal Documentation
+
+ctxpkg can package documentation from any source — Confluence, Notion, SharePoint, or internal wikis — into distributable bundles that teams can share via internal systems.
+
+**Workflow:**
+
+1. **Export your docs as Markdown** — Use your platform's export tools or APIs to extract documentation
+2. **Add a manifest** — Create a `manifest.json` describing the collection:
+
+   ```json
+   {
+     "name": "company-knowledge-base",
+     "sources": [{ "pattern": "**/*.md" }]
+   }
+   ```
+
+3. **Create a bundle** — Package everything into a distributable archive:
+
+   ```bash
+   ctxpkg col pack --output knowledge-base-v1.tar.gz
+   ```
+
+4. **Distribute internally** — Host the bundle on internal file servers, S3, or artifact storage
+
+Teams can then add the bundle:
+
+```bash
+ctxpkg col add kb https://internal.example.com/bundles/knowledge-base-v1.tar.gz
+```
+
+This enables organizations to centralize and distribute institutional knowledge to AI agents across all teams, without requiring git repositories or public hosting.
+
+See [Publishing Packages](docs/github-distribution.md) for automated publishing with GitHub Actions.
+
+## Development
+
+```bash
+pnpm run test:lint   # Linting
+pnpm run test:unit   # Unit tests
+pnpm run build       # Build TypeScript
+```
+
+## License
+
+[GNU Affero General Public License v3.0 (AGPL-3.0)](LICENSE)

package/bin/cli.js

@@ -0,0 +1,8 @@
+#!/usr/bin/env node
+
+import { createProgram } from '../src/cli/cli.ts';
+
+const program = createProgram();
+
+// eslint-disable-next-line
+await program.parseAsync(process.argv);

package/bin/daemon.js

@@ -0,0 +1,7 @@
+#!/usr/bin/env node
+
+import { Daemon } from '../src/daemon/daemon.ts';
+
+const daemon = new Daemon();
+
+await daemon.start();

package/package.json

@@ -0,0 +1,70 @@
+{
+  "type": "module",
+  "main": "dist/exports.js",
+  "bin": {
+    "ctxpkg": "./bin/cli.js"
+  },
+  "files": [
+    "src",
+    "bin"
+  ],
+  "exports": {
+    ".": "./dist/exports.js"
+  },
+  "devDependencies": {
+    "@eslint/eslintrc": "3.3.3",
+    "@eslint/js": "9.39.2",
+    "@pnpm/find-workspace-packages": "6.0.9",
+    "@types/node": "25.0.8",
+    "@types/tar": "^6.1.13",
+    "@types/ws": "^8.18.1",
+    "@vitest/coverage-v8": "4.0.17",
+    "eslint": "9.39.2",
+    "eslint-config-prettier": "10.1.8",
+    "eslint-plugin-import": "2.32.0",
+    "eslint-plugin-prettier": "5.5.4",
+    "msw": "^2.12.7",
+    "prettier": "3.7.4",
+    "tsx": "^4.21.0",
+    "typescript": "5.9.3",
+    "typescript-eslint": "8.53.0",
+    "vitest": "4.0.17"
+  },
+  "name": "ctxpkg",
+  "version": "0.0.1",
+  "license": "AGPL-3.0",
+  "imports": {
+    "#root/*": "./src/*"
+  },
+  "dependencies": {
+    "@huggingface/transformers": "^3.8.1",
+    "@inquirer/prompts": "^8.2.0",
+    "@langchain/community": "^1.1.4",
+    "@langchain/core": "^1.1.13",
+    "@langchain/langgraph": "^1.0.15",
+    "@langchain/openai": "^1.2.2",
+    "@langchain/textsplitters": "^1.0.1",
+    "@modelcontextprotocol/sdk": "^1.25.2",
+    "@types/convict": "^6.1.6",
+    "better-sqlite3": "^12.6.0",
+    "chalk": "^5.6.2",
+    "commander": "^14.0.2",
+    "convict": "^6.2.4",
+    "dotenv": "^17.2.3",
+    "env-paths": "^3.0.0",
+    "knex": "^3.1.0",
+    "langchain": "^1.2.8",
+    "simple-git": "^3.30.0",
+    "sqlite-vec": "0.1.7-alpha.2",
+    "tar": "^7.5.2",
+    "ws": "^8.19.0",
+    "yaml": "^2.8.2",
+    "zod": "3"
+  },
+  "scripts": {
+    "test:lint": "eslint",
+    "build": "tsc --build",
+    "test:unit": "vitest --run --passWithNoTests",
+    "test": "pnpm run \"/^test:/\""
+  }
+}

package/src/agent/AGENTS.md

@@ -0,0 +1,249 @@
+# Agent — Agent Guidelines
+
+This document describes the agent module architecture for AI agents working on this codebase.
+
+## Overview
+
+The agent module provides a LangChain-based agent that uses document tools to search and synthesize information. It's designed to reduce token/context costs by consolidating multiple tool calls into a single, synthesized answer.
+
+## File Structure
+
+| File | Purpose |
+|------|---------|
+| `agent.ts` | Main agent implementation, factory, and retry logic |
+| `agent.types.ts` | TypeScript types and Zod schemas |
+| `agent.prompts.ts` | System prompts and templates |
+
+## Architecture
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│                        DocumentAgent                                │
+│  ┌───────────────────────────────────────────────────────────────┐  │
+│  │                    LangGraph React Agent                      │  │
+│  │                                                               │  │
+│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐   │  │
+│  │  │   search    │  │ get_section │  │ get_document, etc.  │   │  │
+│  │  └─────────────┘  └─────────────┘  └─────────────────────┘   │  │
+│  │                                                               │  │
+│  │  Uses configured LLM (OpenAI-compatible API)                  │  │
+│  └───────────────────────────────────────────────────────────────┘  │
+│                                                                     │
+│  Features:                                                          │
+│  • Verbose mode with step callbacks                                 │
+│  • Conversation history for multi-turn chat                         │
+│  • Collection filtering via system prompt                           │
+│  • Retry logic with exponential backoff                             │
+└─────────────────────────────────────────────────────────────────────┘
+```
+
+## Usage
+
+### Creating an Agent
+
+```typescript
+import { createDocumentAgent, getLLMConfigFromAppConfig } from '#root/agent/agent.ts';
+import { createClient } from '#root/client/client.ts';
+
+const client = await createClient({ mode: 'daemon' });
+const llmConfig = await getLLMConfigFromAppConfig();
+
+const agent = createDocumentAgent({
+  client,
+  llmConfig,
+  aliasMap: new Map([['docs', 'pkg:file://./docs/manifest.json']]),
+  collections: ['docs'],  // Optional: restrict to specific collections
+  onStep: (step) => console.log(step),  // Optional: verbose callbacks
+});
+
+// One-shot query (stateless)
+const response = await agent.ask(
+  'How do I implement streaming?',
+  'Building a chatbot that streams responses'
+);
+
+console.log(response.answer);
+console.log(response.sources);
+console.log(response.confidence);
+```
+
+### Multi-turn Conversation
+
+```typescript
+// First message
+const response1 = await agent.chat(
+  'What authentication methods are available?',
+  'Building a secure API'
+);
+
+// Follow-up (maintains conversation context)
+const response2 = await agent.chat(
+  'How do I implement the OAuth2 option?',
+  'Building a secure API'
+);
+
+// Clear history when starting new topic
+agent.clearHistory();
+```
+
+### Verbose Mode
+
+```typescript
+const agent = createDocumentAgent({
+  client,
+  llmConfig,
+  onStep: (step) => {
+    switch (step.type) {
+      case 'thinking':
+        console.log(`[thinking] ${step.content}`);
+        break;
+      case 'tool_call':
+        console.log(`[tool] ${step.toolName}`);
+        console.log(`  Input: ${JSON.stringify(step.toolInput)}`);
+        break;
+      case 'tool_result':
+        console.log(`[result] ${step.content}`);
+        break;
+      case 'error':
+        console.log(`[retry] ${step.content}`);
+        break;
+    }
+  },
+});
+```
+
+### Agent Response Format
+
+```typescript
+type AgentResponse = {
+  answer: string;           // Synthesized answer
+  sources: Array<{          // References used
+    collection: string;
+    document: string;
+    section?: string;
+  }>;
+  confidence: 'high' | 'medium' | 'low';
+  note?: string;            // Optional note
+};
+```
+
+## LLM Configuration
+
+The agent uses configuration from `config.ts`:
+
+```typescript
+llm: {
+  provider: string;    // OpenAI-compatible API base URL
+  model: string;       // Model identifier
+  apiKey: string;      // API key
+  temperature: number; // 0-2
+  maxTokens: number;   // Max response tokens
+}
+```
+
+Configure via CLI:
+
+```bash
+ctxpkg config set llm.apiKey sk-...
+ctxpkg config set llm.model gpt-4o
+ctxpkg config set llm.provider https://api.openai.com/v1
+```
+
+Or via environment variables:
+
+```bash
+export CTXPKG_LLM_API_KEY=sk-...
+export CTXPKG_LLM_MODEL=gpt-4o
+```
+
+## Agent Design
+
+### Tool Selection
+
+The agent uses LangGraph's React agent pattern with these tools:
+
+- `documents_search` — Semantic search across collections
+- `documents_list_documents` — Browse collection contents
+- `documents_get_outline` — Get document structure
+- `documents_get_section` — Get specific sections
+- `documents_get_document` — Get full documents
+- `documents_list_collections` — List available collections
+- `documents_search_batch` — Batch searches
+- `documents_find_related` — Find related content
+
+### Termination
+
+The agent stops when:
+
+1. It has synthesized a complete answer (JSON response)
+2. Maximum iterations reached (default: 15)
+3. No more relevant information to find
+
+### Response Parsing
+
+The agent is prompted to respond in JSON format. The parser:
+
+1. Looks for ```json code blocks
+2. Tries to parse the whole content as JSON
+3. Falls back to treating content as plain answer
+
+### Retry Logic
+
+The agent automatically retries on transient errors:
+
+- **Rate limits**: 429 errors
+- **Server errors**: 500, 502, 503, 504
+- **Network errors**: ECONNRESET, ETIMEDOUT
+
+Retry configuration:
+- Max retries: 3
+- Initial delay: 1000ms
+- Max delay: 30000ms
+- Backoff multiplier: 2x
+
+```typescript
+import { withRetry, isRetryableError } from '#root/agent/agent.ts';
+
+// Use retry logic for custom async operations
+const result = await withRetry(
+  () => someAsyncOperation(),
+  { maxRetries: 3, initialDelayMs: 1000, maxDelayMs: 30000, backoffMultiplier: 2 },
+  (attempt, error, delayMs) => console.log(`Retry ${attempt}: ${error.message}`)
+);
+```
+
+## Key Patterns
+
+### Lazy Config Loading
+
+Config is loaded dynamically to avoid circular imports:
+
+```typescript
+const getLLMConfigFromAppConfig = async (): Promise<LLMConfig> => {
+  const { config } = await import('#root/config/config.ts');
+  // ...
+};
+```
+
+### Tool Conversion
+
+Document tools are defined once and converted for LangChain:
+
+```typescript
+const toolDefinitions = createDocumentToolDefinitions({ client, aliasMap });
+const langchainTools = toLangchainTools(toolDefinitions);
+```
+
+### Collection Filtering
+
+Collections can be restricted via the `collections` option:
+
+```typescript
+const agent = createDocumentAgent({
+  client,
+  llmConfig,
+  collections: ['my-docs', 'api-docs'],  // Only search these
+});
+```
+
+This adds instructions to the system prompt telling the agent to pass these collections in all search calls.

package/src/agent/agent.prompts.ts

@@ -0,0 +1,66 @@
+/**
+ * System prompt for the document search agent
+ */
+export const AGENT_SYSTEM_PROMPT = `You are a documentation search agent. Your task is to find and synthesize information from technical documentation to answer user questions.
+
+## Guidelines
+
+1. **Start broad, then narrow**: Begin with a semantic search, then drill into specific sections or documents as needed.
+
+2. **Use the right tool for the job**:
+   - \`documents_search\` — Find relevant content across collections
+   - \`documents_list_documents\` — Browse what's available in a collection  
+   - \`documents_get_outline\` — Understand document structure before diving in
+   - \`documents_get_section\` — Get specific section content efficiently
+   - \`documents_get_document\` — Only when you need the full document
+
+3. **Stop when sufficient**: The user has provided a use case. Once you have enough information to address their specific use case, synthesize and respond. Don't over-research.
+
+4. **Cite sources**: Track which documents/sections you used.
+
+5. **Acknowledge uncertainty**: If you can't find sufficient information, say so.
+
+## Response Format
+
+When you have found sufficient information, respond with a JSON object in this exact format:
+
+\`\`\`json
+{
+  "answer": "Your synthesized answer here. Be clear, actionable, and include code examples when relevant.",
+  "sources": [
+    {"collection": "collection-name", "document": "document-id", "section": "Section Heading (if applicable)"}
+  ],
+  "confidence": "high|medium|low",
+  "note": "Optional note about limitations or suggestions for further reading"
+}
+\`\`\`
+
+Use "high" confidence when multiple sources agree or you found a direct answer.
+Use "medium" when the information is relevant but not comprehensive.
+Use "low" when you're extrapolating or the information is tangentially related.`;
+
+/**
+ * Format the collection restriction instruction
+ */
+export const formatCollectionRestriction = (collections: string[]): string => {
+  if (collections.length === 0) return '';
+  const collectionList = collections.map((c) => `"${c}"`).join(', ');
+  return `\n\n## Collection Restriction\nIMPORTANT: Only search within these collections: ${collectionList}. Always pass this list in the "collections" parameter of your search calls.`;
+};
+
+/**
+ * User prompt template
+ */
+export const formatUserPrompt = (query: string, useCase: string, collections?: string[]) => {
+  const collectionNote = collections?.length
+    ? `\n\nNote: Restrict your searches to these collections: ${collections.join(', ')}`
+    : '';
+
+  return `## Question
+${query}
+
+## Use Case
+${useCase}${collectionNote}
+
+Find the information needed to answer this question for the given use case. Search the documentation, then provide your synthesized answer in JSON format.`;
+};