smart-context-mcp 1.3.1 → 1.5.0

package/README.md

@@ -1,6 +1,6 @@
 # smart-context-mcp
 
-MCP server that reduces AI agent token usage by 90% with intelligent context compression.
+MCP server that reduces AI agent token usage by up to 90% with intelligent context compression (measured on this project).
 
 [![npm version](https://badge.fury.io/js/smart-context-mcp.svg)](https://www.npmjs.com/package/smart-context-mcp)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
@@ -42,6 +42,113 @@ npx smart-context-init --target .
 ```
 Restart your AI client. Done.
 
+---
+
+## 📊 Real Metrics
+
+**Production use on this project:**
+- ~7M tokens → ~800K tokens (approximately 89% reduction)
+- 1,500+ operations tracked
+- Compression ratios: 3x to 46x
+
+**Workflow savings:**
+- Debugging: ~85-90% reduction
+- Code Review: ~85-90% reduction
+- Refactoring: ~85-90% reduction
+- Testing: ~85-90% reduction
+- Architecture: ~85-90% reduction
+
+**Real adoption:**
+- Approximately 70-75% of complex tasks use devctx
+- Top tools: `smart_read` (850+), `smart_search` (280+), `smart_shell` (220+)
+- Non-usage: task too simple, no index built, native tools preferred
+
+---
+
+## 🚀 How to Invoke the MCP
+
+The MCP doesn't intercept prompts automatically. **You need to tell the agent to use it.**
+
+### Option 1: Use MCP Prompts (Easiest)
+
+In Cursor, type in the chat:
+
+```
+/prompt use-devctx
+
+[Your task here]
+```
+
+**Available prompts:**
+- `/prompt use-devctx` - Force devctx tools for current task
+- `/prompt devctx-workflow` - Full workflow (start → context → work → end)
+- `/prompt devctx-preflight` - Preflight only (build_index + smart_turn start)
+
+### Option 2: Explicit Instruction
+
+Just tell the agent directly:
+
+```
+Use smart_turn(start) to recover context, then [your task]
+```
+
+Or:
+
+```
+Use the MCP to review this code
+```
+
+### Option 3: Automatic (via Rules)
+
+The agent *should* use devctx automatically for complex tasks because:
+- ✅ `.cursorrules` is active in Cursor
+- ✅ `CLAUDE.md` is active in Claude Desktop (if you created it)
+- ✅ `AGENTS.md` is active in other clients (if you created it)
+
+**But it's not guaranteed** - the agent decides based on task complexity.
+
+### ⚡ Quick Reference
+
+| Scenario | Command |
+|----------|---------|
+| Start new task | `/prompt devctx-workflow` |
+| Continue previous task | `smart_turn(start) and continue` |
+| Force MCP usage | `/prompt use-devctx` |
+| First time in project | `/prompt devctx-preflight` |
+| Trust automatic rules | Just describe your task normally |
+
+---
+
+## 🚨 Agent Ignored devctx? → Paste This Next
+
+<table>
+<tr>
+<td width="100%" bgcolor="#FFF3CD">
+
+### 📋 Official Prompt (Copy & Paste)
+
+```
+Use smart-context-mcp for this task.
+Start with smart_turn(start), then use smart_context or smart_search before reading full files.
+End with smart_turn(end) if you make progress.
+```
+
+### ⚡ Ultra-Short
+
+```
+Use devctx: smart_turn(start) → smart_context → smart_turn(end)
+```
+
+</td>
+</tr>
+</table>
+
+**When:** Agent read large files with `Read`, used `Grep` repeatedly, or no devctx tools in complex task.
+
+**Why:** Task seemed simple, no index built, native tools appeared more direct, or rules weren't strong enough.
+
+---
+
 ## How it Works in Practice
 
 **The reality:** This MCP does not intercept prompts automatically. Here's the actual flow:
@@ -267,19 +374,50 @@ Get everything for a task in one call:
 
 Returns: relevant files + compressed content + symbol details + graph relationships
 
+**Smart pattern detection:** Automatically detects literal patterns (TODO, FIXME, /**, console.log, debugger) and prioritizes them in search.
+
 ### smart_summary
 
 Maintain task checkpoint:
 
 ```javascript
-// Save checkpoint
+// Save checkpoint (flat API - recommended)
+{ action: 'update', goal: 'Implement OAuth', status: 'in_progress', nextStep: '...' }
+
+// Or nested format (backward compatible)
 { action: 'update', update: { goal: 'Implement OAuth', status: 'in_progress', nextStep: '...' }}
 
 // Resume task
 { action: 'get' }
 ```
 
-Stores compressed task state (~100 tokens: goal, status, decisions, blockers), not full conversation.
+Stores compressed task state (~100 tokens: goal, status, decisions, blockers), not full conversation. Supports both flat and nested parameter formats.
+
+### smart_status
+
+Display current session context:
+
+```javascript
+{ format: 'detailed' }  // Full output with progress stats
+{ format: 'compact' }   // Minimal JSON
+```
+
+Shows goal, status, recent decisions, touched files, and progress. Updates automatically with each MCP operation.
+
+### smart_edit
+
+Batch edit multiple files:
+
+```javascript
+{
+  pattern: 'console.log',
+  replacement: 'logger.info',
+  files: ['src/a.js', 'src/b.js'],
+  mode: 'literal'  // or 'regex'
+}
+```
+
+Use `dryRun: true` for preview. Max 50 files per call.
 
 ## New Features
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "smart-context-mcp",
-  "version": "1.3.1",
+  "version": "1.5.0",
   "description": "MCP server that reduces agent token usage by 90% with intelligent context compression, task checkpoint persistence, and workflow-aware agent guidance.",
   "author": "Francisco Caballero Portero <fcp1978@hotmail.com>",
   "type": "module",
@@ -31,7 +31,8 @@
     "scripts/headless-wrapper.js",
     "scripts/init-clients.js",
     "scripts/report-metrics.js",
-    "scripts/report-workflow-metrics.js"
+    "scripts/report-workflow-metrics.js",
+    "scripts/report-adoption-metrics.js"
   ],
   "engines": {
     "node": ">=18"
@@ -60,7 +61,8 @@
     "eval:self": "node ./evals/harness.js --root=../.. --corpus=./evals/corpus/self-tasks.json",
     "eval:report": "node ./evals/report.js",
     "report:metrics": "node ./scripts/report-metrics.js",
-    "report:workflows": "node ./scripts/report-workflow-metrics.js"
+    "report:workflows": "node ./scripts/report-workflow-metrics.js",
+    "report:adoption": "node ./scripts/report-adoption-metrics.js"
   },
   "dependencies": {
     "@modelcontextprotocol/sdk": "^1.13.0",

package/scripts/report-adoption-metrics.js

@@ -0,0 +1,228 @@
+#!/usr/bin/env node
+
+/**
+ * Report adoption metrics - measures real MCP usage in non-trivial tasks
+ * 
+ * Usage:
+ *   npm run report:adoption
+ *   npm run report:adoption -- --days 7
+ *   npm run report:adoption -- --json
+ */
+
+import { withStateDb } from '../src/storage/sqlite.js';
+import { WORKFLOW_DEFINITIONS } from '../src/workflow-tracker.js';
+
+const parseArgs = (argv) => {
+  const args = {};
+  for (let i = 2; i < argv.length; i++) {
+    if (argv[i] === '--days' && argv[i + 1]) {
+      args.days = parseInt(argv[i + 1], 10);
+      i++;
+    } else if (argv[i] === '--json') {
+      args.json = true;
+    }
+  }
+  return args;
+};
+
+const formatPct = (value) => `${value.toFixed(1)}%`;
+const formatNumber = (value) => new Intl.NumberFormat('en-US').format(value);
+
+/**
+ * Classify if a session represents a non-trivial task
+ */
+const isNonTrivialTask = (sessionEvents, metricsEvents) => {
+  // Criteria 1: Multiple operations (≥5)
+  if (sessionEvents.length + metricsEvents.length >= 5) return true;
+
+  // Criteria 2: Large file reads (any file >500 lines)
+  const hasLargeFileRead = metricsEvents.some(
+    (m) => m.tool === 'Read' && m.raw_tokens > 1500 // ~500 lines
+  );
+  if (hasLargeFileRead) return true;
+
+  // Criteria 3: Multiple file reads (≥3)
+  const fileReads = metricsEvents.filter((m) => m.tool === 'Read' || m.tool === 'smart_read');
+  if (fileReads.length >= 3) return true;
+
+  // Criteria 4: Repeated searches (≥2)
+  const searches = metricsEvents.filter((m) => m.tool === 'Grep' || m.tool === 'smart_search');
+  if (searches.length >= 2) return true;
+
+  // Criteria 5: Workflow classification
+  const devctxTools = metricsEvents.filter((m) =>
+    ['smart_turn', 'smart_context', 'smart_search', 'smart_read', 'smart_shell'].includes(m.tool)
+  );
+  if (devctxTools.length > 0) return true;
+
+  return false;
+};
+
+/**
+ * Check if session used devctx tools
+ */
+const usedDevctx = (metricsEvents) => {
+  const devctxTools = ['smart_turn', 'smart_context', 'smart_search', 'smart_read', 'smart_shell', 'smart_read_batch'];
+  return metricsEvents.some((m) => devctxTools.includes(m.tool));
+};
+
+/**
+ * Calculate adoption metrics
+ */
+const calculateAdoptionMetrics = (days = 30) => {
+  return withStateDb((db) => {
+    const cutoff = new Date(Date.now() - days * 24 * 60 * 60 * 1000).toISOString();
+
+    // Get all sessions since cutoff
+    const sessions = db
+      .prepare(
+        `
+      SELECT session_id, snapshot_json, created_at
+      FROM sessions
+      WHERE created_at >= ?
+      ORDER BY created_at DESC
+    `
+      )
+      .all(cutoff);
+
+    const results = {
+      totalSessions: sessions.length,
+      nonTrivialTasks: 0,
+      tasksWithDevctx: 0,
+      adoptionRate: 0,
+      byWorkflow: {},
+      toolUsage: {},
+    };
+
+    // Initialize workflow stats
+    Object.keys(WORKFLOW_DEFINITIONS).forEach((type) => {
+      results.byWorkflow[type] = {
+        total: 0,
+        withDevctx: 0,
+        adoptionRate: 0,
+      };
+    });
+
+    // Analyze each session
+    sessions.forEach((session) => {
+      const snapshot = JSON.parse(session.snapshot_json || '{}');
+      const sessionId = session.session_id;
+
+      // Get events for this session
+      const sessionEvents = db
+        .prepare('SELECT * FROM session_events WHERE session_id = ?')
+        .all(sessionId);
+
+      const metricsEvents = db
+        .prepare('SELECT * FROM metrics_events WHERE session_id = ?')
+        .all(sessionId);
+
+      // Check if non-trivial
+      if (!isNonTrivialTask(sessionEvents, metricsEvents)) {
+        return;
+      }
+
+      results.nonTrivialTasks++;
+
+      // Check if used devctx
+      const hasDevctx = usedDevctx(metricsEvents);
+      if (hasDevctx) {
+        results.tasksWithDevctx++;
+      }
+
+      // Track tool usage
+      metricsEvents.forEach((m) => {
+        results.toolUsage[m.tool] = (results.toolUsage[m.tool] || 0) + 1;
+      });
+
+      // Classify by workflow if possible
+      const goal = snapshot.goal || '';
+      let workflowType = null;
+
+      for (const [type, def] of Object.entries(WORKFLOW_DEFINITIONS)) {
+        if (def.pattern.test(goal)) {
+          workflowType = type;
+          break;
+        }
+      }
+
+      if (workflowType) {
+        results.byWorkflow[workflowType].total++;
+        if (hasDevctx) {
+          results.byWorkflow[workflowType].withDevctx++;
+        }
+      }
+    });
+
+    // Calculate rates
+    if (results.nonTrivialTasks > 0) {
+      results.adoptionRate = (results.tasksWithDevctx / results.nonTrivialTasks) * 100;
+    }
+
+    Object.keys(results.byWorkflow).forEach((type) => {
+      const stats = results.byWorkflow[type];
+      if (stats.total > 0) {
+        stats.adoptionRate = (stats.withDevctx / stats.total) * 100;
+      }
+    });
+
+    return results;
+  });
+};
+
+/**
+ * Format and print report
+ */
+const printReport = (metrics, days) => {
+  console.log(`\nAdoption Metrics (Last ${days} Days)`);
+  console.log('='.repeat(50));
+  console.log();
+
+  console.log(`Total Sessions: ${formatNumber(metrics.totalSessions)}`);
+  console.log(`Non-Trivial Tasks: ${formatNumber(metrics.nonTrivialTasks)}`);
+  console.log(`Tasks with devctx: ${formatNumber(metrics.tasksWithDevctx)}`);
+  console.log();
+
+  console.log(`Overall Adoption: ${formatPct(metrics.adoptionRate)}`);
+  console.log();
+
+  console.log('By Workflow:');
+  Object.entries(metrics.byWorkflow)
+    .filter(([, stats]) => stats.total > 0)
+    .sort((a, b) => b[1].adoptionRate - a[1].adoptionRate)
+    .forEach(([type, stats]) => {
+      const def = WORKFLOW_DEFINITIONS[type];
+      console.log(
+        `  ${def.name.padEnd(25)} ${formatPct(stats.adoptionRate).padStart(7)} (${stats.withDevctx}/${stats.total})`
+      );
+    });
+  console.log();
+
+  console.log('Top devctx Tools:');
+  const devctxTools = ['smart_turn', 'smart_context', 'smart_search', 'smart_read', 'smart_shell'];
+  Object.entries(metrics.toolUsage)
+    .filter(([tool]) => devctxTools.includes(tool))
+    .sort((a, b) => b[1] - a[1])
+    .slice(0, 5)
+    .forEach(([tool, count]) => {
+      console.log(`  ${tool.padEnd(20)} ${formatNumber(count)} uses`);
+    });
+  console.log();
+};
+
+// Main
+const args = parseArgs(process.argv);
+const days = args.days || 30;
+
+try {
+  const metrics = calculateAdoptionMetrics(days);
+
+  if (args.json) {
+    console.log(JSON.stringify(metrics, null, 2));
+  } else {
+    printReport(metrics, days);
+  }
+} catch (error) {
+  console.error('Error calculating adoption metrics:', error.message);
+  process.exit(1);
+}

package/src/config/ignored-paths.js

@@ -0,0 +1,21 @@
+export const IGNORED_DIRS = [
+  'node_modules',
+  '.git',
+  '.next',
+  'dist',
+  'build',
+  'coverage',
+  '.venv',
+  'venv',
+  '__pycache__',
+  '.terraform',
+  '.devctx',
+];
+
+export const IGNORED_FILE_NAMES = [
+  'pnpm-lock.yaml',
+  'package-lock.json',
+  'yarn.lock',
+  'bun.lockb',
+  'npm-shrinkwrap.json',
+];

package/src/cross-project.js

@@ -1,24 +1,12 @@
-/**
- * Cross-project context for monorepos and related projects.
- * 
- * Enables context sharing across multiple related codebases.
- */
-
 import fs from 'node:fs';
 import path from 'node:path';
 import { loadIndex } from './index.js';
 import { smartSearch } from './tools/smart-search.js';
 import { smartRead } from './tools/smart-read.js';
-import { projectRoot, setProjectRoot } from './utils/paths.js';
+import { projectRoot } from './utils/paths.js';
 
 const CROSS_PROJECT_CONFIG_FILE = '.devctx-projects.json';
 
-/**
- * Load cross-project configuration.
- * 
- * @param {string} root - Project root
- * @returns {object|null} Configuration or null if not found
- */
 export const loadCrossProjectConfig = (root = projectRoot) => {
   const configPath = path.join(root, CROSS_PROJECT_CONFIG_FILE);
   
@@ -34,12 +22,6 @@ export const loadCrossProjectConfig = (root = projectRoot) => {
   }
 };
 
-/**
- * Discover related projects from config.
- * 
- * @param {string} root - Project root
- * @returns {Array<object>} Array of { name, path, type, description }
- */
 export const discoverRelatedProjects = (root = projectRoot) => {
   const config = loadCrossProjectConfig(root);
   if (!config?.projects) return [];
@@ -68,13 +50,6 @@ export const discoverRelatedProjects = (root = projectRoot) => {
   return projects;
 };
 
-/**
- * Search across multiple related projects.
- * 
- * @param {string} query - Search query
- * @param {object} options - Search options
- * @returns {Promise<Array>} Results from all projects
- */
 export const searchAcrossProjects = async (query, options = {}) => {
   const {
     root = projectRoot,
@@ -125,18 +100,10 @@ export const searchAcrossProjects = async (query, options = {}) => {
   return results;
 };
 
-/**
- * Read files from multiple projects.
- * 
- * @param {Array<object>} fileRefs - Array of { project, file, mode }
- * @param {string} root - Project root
- * @returns {Promise<Array>} Read results
- */
 export const readAcrossProjects = async (fileRefs, root = projectRoot) => {
   const relatedProjects = discoverRelatedProjects(root);
   const projectMap = new Map(relatedProjects.map(p => [p.name, p]));
 
-  const originalRoot = projectRoot;
   const results = [];
 
   for (const ref of fileRefs) {
@@ -151,11 +118,10 @@ export const readAcrossProjects = async (fileRefs, root = projectRoot) => {
     }
 
     try {
-      setProjectRoot(project.path);
-
       const readResult = await smartRead({
         filePath: ref.file,
         mode: ref.mode || 'outline',
+        cwd: project.path,
       });
 
       results.push({
@@ -175,17 +141,9 @@ export const readAcrossProjects = async (fileRefs, root = projectRoot) => {
     }
   }
 
-  setProjectRoot(originalRoot);
   return results;
 };
 
-/**
- * Find symbol definitions across projects.
- * 
- * @param {string} symbolName - Symbol to find
- * @param {string} root - Project root
- * @returns {Promise<Array>} Symbol locations across projects
- */
 export const findSymbolAcrossProjects = async (symbolName, root = projectRoot) => {
   const relatedProjects = discoverRelatedProjects(root).filter(p => p.hasIndex);
   const results = [];
@@ -223,12 +181,6 @@ export const findSymbolAcrossProjects = async (symbolName, root = projectRoot) =
   return results;
 };
 
-/**
- * Get dependency graph across projects.
- * 
- * @param {string} root - Project root
- * @returns {object} Cross-project dependency graph
- */
 export const getCrossProjectDependencies = (root = projectRoot) => {
   const relatedProjects = discoverRelatedProjects(root).filter(p => p.hasIndex);
   const dependencies = {
@@ -276,12 +228,6 @@ export const getCrossProjectDependencies = (root = projectRoot) => {
   return dependencies;
 };
 
-/**
- * Get statistics about cross-project usage.
- * 
- * @param {string} root - Project root
- * @returns {object} Usage statistics
- */
 export const getCrossProjectStats = (root = projectRoot) => {
   const relatedProjects = discoverRelatedProjects(root);
   const deps = getCrossProjectDependencies(root);
@@ -306,12 +252,6 @@ export const getCrossProjectStats = (root = projectRoot) => {
   return stats;
 };
 
-/**
- * Create a sample cross-project configuration.
- * 
- * @param {string} root - Project root
- * @returns {object} Sample configuration
- */
 export const createSampleConfig = (root = projectRoot) => {
   return {
     version: '1.0',