mcp-memory-keeper 0.10.1 → 0.11.0

package/CHANGELOG.md

@@ -7,7 +7,47 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
-## [0.10.1] - 2025-01-11
+## [0.11.0] - 2025-12-10
+
+### Breaking Changes
+
+- **Node.js 18 support dropped** - Minimum required Node.js version is now 20.0.0
+  - Node.js 18 reached End-of-Life on April 30, 2025
+  - Users on Node.js 18 must upgrade to Node.js 20 or later
+  - Existing installations will continue working until updated
+
+### Fixed
+
+- **Installation fails on Node.js 24 (#28)** - Updated `better-sqlite3` dependency
+  - Upgraded from `^11.10.0` to `^12.1.0` to support Node.js 24 (LTS "Krypton")
+  - Prebuilt binaries now available for Node.js 20, 22, and 24
+  - Resolves `gyp ERR!` build failures on Node.js 24
+
+## [0.10.2] - 2025-09-16
+
+### Fixed
+
+- **Critical Token Limit Issue (#24)** - Fixed token overflow with includeMetadata
+  - Implemented dynamic token limit calculation based on actual content size
+  - Automatically adjusts item limits based on average item size in session
+  - More accurate token estimation (3.5 chars/token vs 4)
+  - Configurable via environment variables (MCP_MAX_TOKENS, MCP_TOKEN_SAFETY_BUFFER)
+  - Added tokenInfo to response metadata for transparency
+  - Resolves "MCP tool context_get response exceeds maximum allowed tokens" errors
+
+### Added
+
+- **Token Limit Management Module** (`utils/token-limits.ts`)
+  - Dynamic calculation of safe item limits
+  - Response overhead estimation
+  - Configurable token limits via environment
+  - Better visibility into token usage
+  - Proper TypeScript interfaces for context items
+  - Environment variable validation with bounds checking
+  - Safe JSON parsing with error handling
+  - Well-documented constants replacing magic numbers
+
+## [0.10.1] - 2025-07-11
 
 ### Fixed
 
@@ -63,7 +103,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Added recipes for common patterns in RECIPES.md
 - Added troubleshooting tips for new features
 
-## [0.10.0] - 2025-01-26
+## [0.10.0] - 2025-06-26
 
 ### Added
 
@@ -102,7 +142,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Enhanced validation for channel names
 - Backward compatible - existing items default to 'default' channel
 
-## [0.9.0] - 2025-01-20
+## [0.9.0] - 2025-06-21
 
 ### Changed (BREAKING)
 
@@ -125,7 +165,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - `context_get_shared` tool (use `context_get` instead)
 - Complex sharing mechanism that was causing inconsistencies
 
-## [0.8.4] - 2025-01-19
+## [0.8.4] - 2025-06-19
 
 ### Fixed
 
@@ -139,7 +179,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Feature flags system (planned)
 - Database migration system (planned)
 
-## [0.8.3] - 2025-01-19
+## [0.8.3] - 2025-06-19
 
 ### Added
 
@@ -160,7 +200,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - Automatic schema migration for existing databases to add the `working_directory` column
 
-## [0.8.0] - 2024-01-18
+## [0.8.0] - 2025-06-18
 
 ### Added
 
@@ -200,7 +240,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - New tables: `journal_entries`, `compressed_context`, `tool_events`
 - All 255 tests passing
 
-## [0.7.0] - 2024-01-18
+## [0.7.0] - 2025-06-18
 
 ### Added
 
@@ -224,7 +264,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Added comprehensive test coverage (30 new tests)
 - All 236 tests passing
 
-## [0.6.0] - 2024-01-17
+## [0.6.0] - 2025-06-17
 
 ### Added
 
@@ -247,7 +287,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Comprehensive test coverage for semantic search
 - All 206 tests passing
 
-## [0.5.0] - 2024-01-17
+## [0.5.0] - 2025-06-17
 
 ### Added
 
@@ -270,7 +310,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Added `knowledge-graph.ts` utility module
 - Comprehensive test coverage for graph operations
 
-## [0.4.2] - 2024-01-16
+## [0.4.2] - 2025-06-17
 
 ### Added
 
@@ -284,7 +324,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Git integration error handling
 - Session list date filtering
 
-## [0.4.1] - 2024-01-16
+## [0.4.1] - 2025-06-17
 
 ### Fixed
 
@@ -297,7 +337,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Improved error messages for better debugging
 - Enhanced validation for file paths
 
-## [0.4.0] - 2024-01-15
+## [0.4.0] - 2025-06-17
 
 ### Added
 
@@ -318,7 +358,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Created `git.ts` utility module
 - 97% test coverage maintained
 
-## [0.3.0] - 2024-01-14
+## [0.3.0] - 2025-06-17
 
 ### Added
 
@@ -347,7 +387,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Implemented streaming for large exports
 - Transaction support for atomic operations
 
-## [0.2.0] - 2024-01-13
+## [0.2.0] - 2025-06-17
 
 ### Added
 
@@ -377,7 +417,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Memory leak in file cache operations
 - Session switching race condition
 
-## [0.1.0] - 2024-01-12
+## [0.1.0] - 2025-06-17
 
 ### Added
 
@@ -405,12 +445,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## Development Releases
 
-### [0.1.0-beta.2] - 2024-01-11
+### [0.1.0-beta.2] - 2025-06-17
 
 - Fixed Windows path handling
 - Added Node.js 18+ compatibility
 
-### [0.1.0-beta.1] - 2024-01-10
+### [0.1.0-beta.1] - 2025-06-17
 
 - Initial beta release
 - Basic functionality testing

package/README.md

@@ -183,10 +183,36 @@ claude mcp add memory-keeper node /absolute/path/to/mcp-memory-keeper/dist/index
 
 ### Environment Variables
 
+#### Storage and Installation
+
 - `DATA_DIR` - Directory for database storage (default: `~/mcp-data/memory-keeper/`)
 - `MEMORY_KEEPER_INSTALL_DIR` - Installation directory (default: `~/.local/mcp-servers/memory-keeper/`)
 - `MEMORY_KEEPER_AUTO_UPDATE` - Set to `1` to enable auto-updates
 
+#### Token Limit Configuration
+
+- `MCP_MAX_TOKENS` - Maximum tokens allowed in responses (default: `25000`, range: `1000-100000`)
+  - Adjust this if your MCP client has different limits
+- `MCP_TOKEN_SAFETY_BUFFER` - Safety buffer percentage (default: `0.8`, range: `0.1-1.0`)
+  - Uses only this fraction of the max tokens to prevent overflows
+- `MCP_MIN_ITEMS` - Minimum items to return even if exceeding limits (default: `1`, range: `1-100`)
+  - Ensures at least some results are returned
+- `MCP_MAX_ITEMS` - Maximum items allowed per response (default: `100`, range: `10-1000`)
+  - Upper bound for result sets regardless of token limits
+- `MCP_CHARS_PER_TOKEN` - Characters per token ratio (default: `3.5`, range: `2.5-5.0`) **[Advanced]**
+  - Adjusts token estimation accuracy for different content types
+  - Lower values = more conservative (safer but returns fewer items)
+  - Higher values = more aggressive (returns more items but risks overflow)
+
+Example configuration for stricter token limits:
+
+```bash
+export MCP_MAX_TOKENS=20000        # Lower max tokens
+export MCP_TOKEN_SAFETY_BUFFER=0.7  # More conservative buffer
+export MCP_MAX_ITEMS=50             # Fewer items per response
+export MCP_CHARS_PER_TOKEN=3.0      # More conservative estimation (optional)
+```
+
 ### Claude Code (CLI)
 
 #### Configuration Scopes

package/dist/__tests__/integration/issue-token-limit-channel-query.test.js

@@ -0,0 +1,128 @@
+"use strict";
+/**
+ * Test for token limit issue with channel queries
+ *
+ * Reproduces the bug where context_get with limit: 50 still exceeds token limits
+ * when querying a channel with large items.
+ */
+Object.defineProperty(exports, "__esModule", { value: true });
+const globals_1 = require("@jest/globals");
+const database_js_1 = require("../../utils/database.js");
+const RepositoryManager_js_1 = require("../../repositories/RepositoryManager.js");
+const token_limits_js_1 = require("../../utils/token-limits.js");
+(0, globals_1.describe)('Token Limit with Channel Query Bug', () => {
+    let dbManager;
+    let repositories;
+    let sessionId;
+    (0, globals_1.beforeEach)(() => {
+        dbManager = new database_js_1.DatabaseManager({ filename: ':memory:' });
+        repositories = new RepositoryManager_js_1.RepositoryManager(dbManager);
+        const session = repositories.sessions.create({
+            name: 'Test Session',
+            defaultChannel: 'test-channel',
+        });
+        sessionId = session.id;
+    });
+    (0, globals_1.afterEach)(() => {
+        dbManager.close();
+    });
+    (0, globals_1.it)('should enforce token limits even when limit parameter is provided', () => {
+        // Create 50 large context items (each ~1000 chars)
+        const largeValue = 'x'.repeat(1000);
+        for (let i = 0; i < 50; i++) {
+            repositories.contexts.save(sessionId, {
+                key: `large-item-${i}`,
+                value: largeValue,
+                channel: 'outbound-call-center',
+                priority: 'normal',
+            });
+        }
+        // Query with limit: 50
+        const result = repositories.contexts.queryEnhanced({
+            sessionId,
+            channel: 'outbound-call-center',
+            limit: 50,
+            sort: 'updated_desc',
+            includeMetadata: false,
+        });
+        (0, globals_1.expect)(result.items.length).toBe(50);
+        // Check if response would exceed token limit
+        const tokenConfig = (0, token_limits_js_1.getTokenConfig)();
+        const { exceedsLimit, safeItemCount } = (0, token_limits_js_1.checkTokenLimit)(result.items, false, tokenConfig);
+        // Build the actual response structure
+        const response = {
+            items: result.items,
+            pagination: {
+                total: result.totalCount,
+                returned: result.items.length,
+                offset: 0,
+                hasMore: false,
+                nextOffset: null,
+                truncated: false,
+                truncatedCount: 0,
+            },
+        };
+        const responseJson = JSON.stringify(response, null, 2);
+        const actualTokens = Math.ceil(responseJson.length / tokenConfig.charsPerToken);
+        // The bug: even with limit: 50, the response can exceed token limits
+        if (actualTokens > tokenConfig.mcpMaxTokens) {
+            // BUG REPRODUCED: The response exceeds token limits
+            // Verify that checkTokenLimit correctly detected the issue
+            (0, globals_1.expect)(exceedsLimit).toBe(true);
+            (0, globals_1.expect)(safeItemCount).toBeLessThan(50);
+            (0, globals_1.expect)(actualTokens).toBeGreaterThan(tokenConfig.mcpMaxTokens);
+        }
+    });
+    (0, globals_1.it)('should respect token limits over user-provided limit parameter', () => {
+        // Create 100 large context items (each ~800 chars)
+        const largeValue = 'y'.repeat(800);
+        for (let i = 0; i < 100; i++) {
+            repositories.contexts.save(sessionId, {
+                key: `item-${i}`,
+                value: largeValue,
+                channel: 'test-channel',
+                priority: 'normal',
+            });
+        }
+        // Query with limit: 50 (user expectation)
+        const result = repositories.contexts.queryEnhanced({
+            sessionId,
+            channel: 'test-channel',
+            limit: 50,
+            sort: 'created_desc',
+            includeMetadata: false,
+        });
+        // Simulate the context_get handler logic
+        const tokenConfig = (0, token_limits_js_1.getTokenConfig)();
+        const { exceedsLimit, safeItemCount } = (0, token_limits_js_1.checkTokenLimit)(result.items, false, tokenConfig);
+        let actualItems = result.items;
+        let wasTruncated = false;
+        if (exceedsLimit && safeItemCount < result.items.length) {
+            actualItems = result.items.slice(0, safeItemCount);
+            wasTruncated = true;
+        }
+        // Build response
+        const response = {
+            items: actualItems,
+            pagination: {
+                total: result.totalCount,
+                returned: actualItems.length,
+                offset: 0,
+                hasMore: wasTruncated || actualItems.length < result.totalCount,
+                nextOffset: wasTruncated ? actualItems.length : null,
+                truncated: wasTruncated,
+                truncatedCount: wasTruncated ? result.items.length - actualItems.length : 0,
+            },
+        };
+        const responseJson = JSON.stringify(response, null, 2);
+        const actualTokens = Math.ceil(responseJson.length / tokenConfig.charsPerToken);
+        // Verify token limit is not exceeded
+        (0, globals_1.expect)(actualTokens).toBeLessThanOrEqual(tokenConfig.mcpMaxTokens);
+        // Verify truncation occurred if needed
+        if (exceedsLimit) {
+            (0, globals_1.expect)(wasTruncated).toBe(true);
+            (0, globals_1.expect)(actualItems.length).toBeLessThan(50);
+            (0, globals_1.expect)(actualItems.length).toBe(safeItemCount);
+        }
+    });
+});

package/dist/__tests__/integration/issue24-final-fix.test.js

@@ -0,0 +1,241 @@
+"use strict";
+// Final test to verify the fix for issue #24
+// Simulates the exact scenario: context_get with sessionId: "current" and includeMetadata: true
+var __createBinding = (this && this.__createBinding) || (Object.create ? (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    var desc = Object.getOwnPropertyDescriptor(m, k);
+    if (!desc || ("get" in desc ? !m.__esModule : desc.writable || desc.configurable)) {
+      desc = { enumerable: true, get: function() { return m[k]; } };
+    }
+    Object.defineProperty(o, k2, desc);
+}) : (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    o[k2] = m[k];
+}));
+var __setModuleDefault = (this && this.__setModuleDefault) || (Object.create ? (function(o, v) {
+    Object.defineProperty(o, "default", { enumerable: true, value: v });
+}) : function(o, v) {
+    o["default"] = v;
+});
+var __importStar = (this && this.__importStar) || (function () {
+    var ownKeys = function(o) {
+        ownKeys = Object.getOwnPropertyNames || function (o) {
+            var ar = [];
+            for (var k in o) if (Object.prototype.hasOwnProperty.call(o, k)) ar[ar.length] = k;
+            return ar;
+        };
+        return ownKeys(o);
+    };
+    return function (mod) {
+        if (mod && mod.__esModule) return mod;
+        var result = {};
+        if (mod != null) for (var k = ownKeys(mod), i = 0; i < k.length; i++) if (k[i] !== "default") __createBinding(result, mod, k[i]);
+        __setModuleDefault(result, mod);
+        return result;
+    };
+})();
+Object.defineProperty(exports, "__esModule", { value: true });
+const database_1 = require("../../utils/database");
+const RepositoryManager_1 = require("../../repositories/RepositoryManager");
+const os = __importStar(require("os"));
+const path = __importStar(require("path"));
+const fs = __importStar(require("fs"));
+// Mock functions from index.ts
+function estimateTokens(text) {
+    return Math.ceil(text.length / 4);
+}
+function calculateSize(value) {
+    return Buffer.byteLength(value, 'utf8');
+}
+function validatePaginationParams(params) {
+    const errors = [];
+    let limit = 25; // default
+    let offset = 0; // default
+    // Validate limit
+    if (params.limit !== undefined && params.limit !== null) {
+        const rawLimit = params.limit;
+        if (!Number.isInteger(rawLimit) || rawLimit <= 0) {
+            errors.push(`Invalid limit: expected positive integer, got ${typeof rawLimit} '${rawLimit}'`);
+        }
+        else {
+            limit = Math.min(Math.max(1, rawLimit), 100); // clamp between 1-100
+        }
+    }
+    // Validate offset
+    if (params.offset !== undefined && params.offset !== null) {
+        const rawOffset = params.offset;
+        if (!Number.isInteger(rawOffset) || rawOffset < 0) {
+            errors.push(`Invalid offset: expected non-negative integer, got ${typeof rawOffset} '${rawOffset}'`);
+        }
+        else {
+            offset = rawOffset;
+        }
+    }
+    return { limit, offset, errors };
+}
+describe('Issue #24 - Final Fix Verification', () => {
+    let dbManager;
+    let repositories;
+    let tempDbPath;
+    let testSessionId;
+    beforeEach(() => {
+        tempDbPath = path.join(os.tmpdir(), `test-issue24-final-${Date.now()}.db`);
+        dbManager = new database_1.DatabaseManager({
+            filename: tempDbPath,
+            maxSize: 10 * 1024 * 1024,
+            walMode: true,
+        });
+        repositories = new RepositoryManager_1.RepositoryManager(dbManager);
+        // Create test session (simulating "current" session)
+        const session = repositories.sessions.create({
+            name: 'Current Session',
+            description: 'Simulating the current active session',
+        });
+        testSessionId = session.id;
+        // Create many realistic items that would cause overflow
+        const largeContent = `
+This is a realistic context item saved during development.
+It contains implementation details, code snippets, notes, and documentation.
+The content is substantial to simulate real-world usage patterns.
+Developers often save detailed context about complex features, debugging sessions,
+architectural decisions, API documentation, and troubleshooting information.
+`.trim();
+        for (let i = 0; i < 200; i++) {
+            repositories.contexts.save(testSessionId, {
+                key: `context_${String(i).padStart(3, '0')}`,
+                value: `${largeContent}\n\nSpecific item ${i} notes and details.`,
+                category: ['task', 'decision', 'progress', 'note'][i % 4],
+                priority: ['high', 'normal', 'low'][i % 3],
+                channel: `channel-${i % 10}`,
+                metadata: JSON.stringify({
+                    index: i,
+                    timestamp: new Date().toISOString(),
+                    tags: ['dev', 'test', 'review'][i % 3],
+                }),
+            });
+        }
+    });
+    afterEach(() => {
+        dbManager.close();
+        try {
+            fs.unlinkSync(tempDbPath);
+            fs.unlinkSync(`${tempDbPath}-wal`);
+            fs.unlinkSync(`${tempDbPath}-shm`);
+        }
+        catch (_e) {
+            // Ignore
+        }
+    });
+    it('should handle context_get with includeMetadata: true without exceeding token limit', () => {
+        // Simulate the exact call from the issue:
+        // context_get(sessionId: "current", includeMetadata: true)
+        const args = {
+            sessionId: testSessionId, // Would be resolved from "current"
+            includeMetadata: true,
+            // No limit specified - should use new default of 30
+        };
+        // Simulate what the handler does
+        const includeMetadata = args.includeMetadata;
+        const rawLimit = undefined; // Not specified in the call
+        const rawOffset = undefined;
+        // Apply our fix: default limit is 30 when includeMetadata is true
+        const defaultLimit = includeMetadata ? 30 : 100;
+        const paginationValidation = validatePaginationParams({
+            limit: rawLimit !== undefined ? rawLimit : defaultLimit,
+            offset: rawOffset,
+        });
+        const { limit, offset } = paginationValidation;
+        console.log(`Using limit: ${limit} (includeMetadata: ${includeMetadata})`);
+        // Query with the calculated limit
+        const result = repositories.contexts.queryEnhanced({
+            sessionId: testSessionId,
+            includeMetadata: true,
+            limit: limit,
+            offset: offset,
+        });
+        console.log(`Retrieved ${result.items.length} of ${result.totalCount} items`);
+        // Transform items with metadata
+        const itemsWithMetadata = result.items.map(item => ({
+            key: item.key,
+            value: item.value,
+            category: item.category,
+            priority: item.priority,
+            channel: item.channel,
+            metadata: item.metadata ? JSON.parse(item.metadata) : null,
+            size: item.size || calculateSize(item.value),
+            created_at: item.created_at,
+            updated_at: item.updated_at,
+        }));
+        // Calculate metrics for token checking
+        const itemsForMetrics = itemsWithMetadata;
+        const jsonString = JSON.stringify(itemsForMetrics);
+        const estimatedTokens = estimateTokens(jsonString);
+        console.log(`Items token estimate: ${estimatedTokens}`);
+        // Build full response
+        const response = {
+            items: itemsWithMetadata,
+            pagination: {
+                total: result.totalCount,
+                returned: result.items.length,
+                offset: 0,
+                hasMore: result.totalCount > result.items.length,
+                nextOffset: result.items.length < result.totalCount ? result.items.length : null,
+                totalCount: result.totalCount,
+                page: 1,
+                pageSize: limit,
+                totalPages: Math.ceil(result.totalCount / limit),
+                hasNextPage: result.totalCount > result.items.length,
+                hasPreviousPage: false,
+                previousOffset: null,
+                totalSize: itemsWithMetadata.reduce((sum, item) => sum + (item.size || 0), 0),
+                averageSize: Math.round(itemsWithMetadata.reduce((sum, item) => sum + (item.size || 0), 0) / result.items.length),
+                defaultsApplied: { limit: true, sort: true },
+                truncated: false,
+                truncatedCount: 0,
+            },
+        };
+        const fullResponseJson = JSON.stringify(response, null, 2);
+        const finalTokens = estimateTokens(fullResponseJson);
+        console.log(`Final response tokens: ${finalTokens}`);
+        // Verify the fix works
+        expect(limit).toBe(30); // Our new default when includeMetadata is true
+        expect(result.items.length).toBeLessThanOrEqual(30);
+        expect(finalTokens).toBeLessThan(15000); // Our new conservative TOKEN_LIMIT
+        expect(finalTokens).toBeLessThan(25000); // MCP's actual limit
+        console.log('✅ Fix verified: Response stays well under token limit');
+    });
+    it('should still allow explicit higher limits but truncate if needed', () => {
+        // User explicitly requests more items
+        const args = {
+            sessionId: testSessionId,
+            includeMetadata: true,
+            limit: 100, // Explicitly requesting many items
+        };
+        const result = repositories.contexts.queryEnhanced({
+            sessionId: testSessionId,
+            includeMetadata: true,
+            limit: args.limit,
+        });
+        // With explicit limit, we get what was requested (up to 100)
+        expect(result.items.length).toBeLessThanOrEqual(100);
+        // But in the handler, if this exceeds tokens, it would be truncated
+        const itemsWithMetadata = result.items.map(item => ({
+            key: item.key,
+            value: item.value,
+            category: item.category,
+            priority: item.priority,
+            channel: item.channel,
+            metadata: item.metadata ? JSON.parse(item.metadata) : null,
+            size: item.size || calculateSize(item.value),
+            created_at: item.created_at,
+            updated_at: item.updated_at,
+        }));
+        const responseJson = JSON.stringify({ items: itemsWithMetadata }, null, 2);
+        const tokens = estimateTokens(responseJson);
+        console.log(`With explicit limit=100: ${result.items.length} items, ${tokens} tokens`);
+        // This would trigger truncation in the handler
+        if (tokens > 15000) {
+            console.log('⚠️ Would trigger truncation in handler');
+        }
+    });
+});